Image watermarking protection method based on diffusion image editing model

Abstract

The application provides a picture watermark protection method based on a diffusion picture editing model, acquires a trained picture watermark protection network, uses a gradient truncation sampling strategy to enable the gradient to be returned, and obtains the watermark embedding module, the diffusion picture editing model and the watermark extraction module contained therein by jointly training and updating the parameters of the watermark embedding module and the watermark extraction module, then inputs a specified watermark bit string and a picture to be protected into the watermark embedding module to obtain a watermark picture, inputs a specified editing instruction and the watermark picture into the diffusion picture editing model to obtain a generated edited picture, inputs the generated edited picture into the watermark extraction module to obtain an extracted watermark bit string, and compares the extracted watermark bit string with the specified watermark bit string, so that the watermark protection process of the picture to be protected is completed while the memory occupation is reduced.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to an image watermark protection method based on a diffusion image editing model. Background Technology

[0002] Recently, StableDiffusion (a deep learning text-to-image diffusion generation model released in 2022, primarily used to generate detailed images based on text descriptions) and ChatGPT (a chatbot pre-trained converter developed by OpenAI, launched in November 2022) have become quite popular, allowing users to easily and freely generate text, images, videos, and other content using these AI (artificial intelligence) tools. Image generation is a primary use of StableDiffusion, encompassing various scenarios including image generation, image editing, and personalized customization using a small number of images. Image editing refers to modifying images using text editing commands, including content and style modifications.

[0003] Watermarking, a fundamental method for image protection, refers to embedding watermark information into an image that is imperceptible to the human eye and can be effectively extracted after embedding. Deep learning-based watermarking methods use neural networks and gradient backpropagation strategies to update model parameters, jointly training the watermark embedding and extraction models end-to-end, achieving impressive performance. The trained watermark embedding model can embed watermark information imperceptible to the human eye to obtain a watermarked image, while the trained watermark extraction model can effectively extract the embedded watermark information from the watermarked image. In practical applications, watermarked images often suffer some degree of distortion; therefore, how to effectively extract the watermark even when the watermarked image is subjected to various types of distortion is a key focus in this field. Current watermarking methods mainly consider distortions such as those caused by JPEG (an image compression algorithm) and physical changes (e.g., during photography). These methods primarily simulate non-differentiable distortion processes as differentiable ones, then incorporate them into the joint training of the watermark embedding model and the extraction model. This ensures that, after training, the watermark information embedded in the watermark embedding model can effectively resist the distortions adapted during training. However, image editing involves more complex and significant distortions. Existing watermarking methods do not consider this type of distortion and cannot effectively resist it; that is, the embedded watermark information cannot be effectively extracted from edited images. Therefore, how to enable the watermark information embedded in images to effectively resist the distortion caused by image editing is a worthwhile area of ​​exploration and the focus of this invention. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image watermark protection method based on a diffusion image editing model.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] A watermarking protection method based on a diffusion image editing model includes the following steps:

[0007] S1. Obtain a trained image watermark protection network, the image watermark protection network including a watermark embedding module, a diffusion image editing model, and a watermark extraction module; during the training process of the image watermark protection network, the watermark embedding module, the diffusion image editing model, and the watermark extraction module are jointly trained using a gradient truncation sampling strategy;

[0008] S2. Input the watermark bit string specified by the image owner and the image to be protected together into the watermark embedding module to obtain the watermarked image;

[0009] S3. Input the editing instructions specified by the image editor along with the watermark image into the diffusion image editing model to obtain the generated edited image; the specified editing instructions are a sentence that can describe the image editing method;

[0010] S4. Input the generated edited image into the watermark extraction module to obtain the extracted watermark bit string. Compare the extracted watermark bit string with the specified watermark bit string to complete the watermark protection process of the image to be protected.

[0011] Preferably, during the training process of the image watermark protection network, a pre-trained large language model is used. The real text descriptions corresponding to the original images are input into the large language model, where each text description is a sentence that describes the image content. The large language model outputs editing instructions, and these instructions, along with the original images, are used to construct the training samples for the image watermark protection network in the form of {original image I}. t Edit command INS};

[0012] In each iteration of the image watermark protection network, a binary watermark bit string M is first randomly generated. t The binary watermark bit strings are arranged in rows to obtain a two-dimensional watermark bit matrix M. t_2 The original image and the watermark feature matrix M are combined. t_2 The images are input together into the image processing neural network to obtain the watermarked image I. wUsing the gradient truncation sampling strategy, in the N-step sampling process of the image editing in the diffusion image editing model, in the first (Nk) steps of the sampling process, the editing command and the watermark image I are... w The images are input together into the diffusion image editing model for gradient truncation to obtain an intermediate generated image. In the subsequent k steps of the sampling process, the intermediate generated image, the editing instructions, and the watermark image I are... w The gradients are then fed into the diffusion image editing model for gradient propagation, and then fed back to the watermark embedding module and the watermark extraction module to finally obtain the generated edited image I. g The generated edited image I g The input is given to the watermark extraction module to obtain the edited image I generated. g Extracted watermark bit string M p ;

[0013] The watermark image I is analyzed using pixel-constrained mean square error loss. w and the original image I t Constraints are applied, and the latent space mean square error loss is used to adjust the latent space of the watermarked image I. w and the original image I t Constraints are applied, and the sum of the pixel constraint mean square error loss and the latent space mean square error loss is used as the watermark embedding mean square error loss to update the network parameters of the watermark embedding module using gradient descent. The parameters of the diffusion image editing model remain unchanged. The mean square error loss of the image watermark extraction before editing is used to update the binary watermark bit string M. t and the watermarked image I w Extracted watermark bit string M pb Constraints are applied, and the mean squared error loss of the edited image watermark extraction is used to adjust the binary watermark bit string M. t and the extracted watermark bit string M p Constraints are applied, and the mean squared error loss of watermark extraction before editing and the mean squared error loss of watermark extraction after editing are added together as the mean squared error loss of watermark extraction. The network parameters of the watermark extraction module are then updated by gradient descent. Finally, the mean squared error loss of watermark embedding and the mean squared error loss of watermark extraction are added together as the total loss of the image watermark protection network.

[0014] The pixel constraint mean square error loss L enc_pixel The function form is:

[0015]

[0016] Where MSE is the mean squared error loss function; n is the total number of pixels in the image; I tiFor the original image I t The pixel value at position i; I wi Image with watermark I w The pixel value at the i-th position;

[0017] The hidden space mean square error loss L enc_latent The function form is:

[0018] L enc_latent =MSE(VAE(I t ),VAE(I w ))

[0019] VAE stands for Variational Autoencoder;

[0020] The watermark embedding mean square error loss L enc The function form is:

[0021] L enc =L enc_pixel +L enc_latent

[0022] The mean square error loss of the image watermark extraction before editing, L dec_before The function form is:

[0023] L dec_before =MSE(M t M pb )

[0024] The mean square error loss of watermark extraction of the edited image L dec_after The function form is:

[0025] L dec_after =MSE(M t M p )

[0026] The watermark extraction mean square error loss L dec The function form is:

[0027] L dec =L dec_before +L dec_after

[0028] The total loss of the image watermark protection network L total The function form is:

[0029] L total =L enc +L dec .

[0030] Preferably, the specific process of step S2 is as follows: randomly generate a binary watermark bit string, arrange the binary watermark bit string in rows to obtain a two-dimensional watermark bit matrix, then expand the size of the two-dimensional watermark bit matrix to be the same as the size of the color image to be protected to obtain a watermark feature matrix, and input the image to be protected and the watermark feature matrix together into an image processing neural network to obtain a watermarked image.

[0031] Preferably, the large language model adopts the ChatGPT model.

[0032] Preferably, the original image is derived from an image-edit instruction pair dataset.

[0033] Preferably, the watermark embedding module uses a U-Net network.

[0034] Preferably, the diffusion image editing model adopts the InstructPix2Pix model.

[0035] Preferably, the image processing neural network uses a U-Net network.

[0036] Preferably, the size expansion method is through deconvolution or upsampling interpolation.

[0037] Preferably, the watermark extraction module uses a ResNet network.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] Traditional watermarking methods typically only consider minor distortions in images like JPEG during training, failing to effectively resist significant image editing, resulting in severe distortion of the image to be protected. In contrast, this invention proposes an image watermarking protection method based on a diffusion image editing model. Unlike traditional methods, this invention innovatively treats the image editing sampling process of the diffusion image editing model as a distortion process, integrating it into the watermark embedding and extraction modules. During the training of the image watermarking protection network, the image editing process is incorporated, and a gradient truncation sampling strategy allows for joint training of the watermark embedder and watermark extractor, reducing memory usage and improving performance. This invention can also effectively resist these image editing behaviors, distinguish the authenticity of images, trace back to protect the rights of the owner, and achieve source tracing and deepfake detection. Furthermore, this invention focuses on protecting images processed by the diffusion image editing model, including but not limited to: protecting the people in the image when maliciously altering them to create fake news; and protecting the composition, style, and inherent design of the original image when using style transfer to forge artwork, demonstrating broad application prospects. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of the image watermarking protection method based on the diffusion image editing model;

[0041] Figure 2 This is a schematic diagram illustrating the training process of an image watermarking protection method based on a diffusion image editing model.

[0042] Figure 3 This is a schematic diagram of the gradient truncation sampling strategy for image watermarking protection based on the diffusion image editing model.

[0043] Figure 4 This is a schematic diagram showing the change of evaluation index with the number of iterations in an embodiment of the image watermarking protection method based on the diffusion image editing model. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0046] In a preferred embodiment of the present invention, such as Figure 1 As shown, an image watermark protection method based on a diffusion image editing model is provided. The specific steps are as follows:

[0047] S1. Obtain a trained image watermark protection network, the image watermark protection network including a watermark embedding module, a diffusion image editing model, and a watermark extraction module; during the training process of the image watermark protection network, the watermark embedding module, the diffusion image editing model, and the watermark extraction module are jointly trained using a gradient truncation sampling strategy;

[0048] S2. Input the watermark bit string specified by the image owner and the image to be protected together into the watermark embedding module to obtain the watermarked image;

[0049] S3. Input the editing instructions specified by the image editor along with the watermark image into the diffusion image editing model to obtain the generated edited image; the specified editing instructions are a sentence that can describe the image editing method;

[0050] S4. Input the generated edited image into the watermark extraction module to obtain the extracted watermark bit string. Compare the extracted watermark bit string with the specified watermark bit string to complete the watermark protection process of the image to be protected.

[0051] It should be noted that, in the embodiments of the present invention, during the training process of the image watermark protection network, specifically, as follows: Figure 2 As shown, firstly, a pre-trained large language model is used, and the real text descriptions corresponding to the original images are input into the large language model. Each text description is a sentence that can describe the content of the image. The large language model outputs editing instructions, and these editing instructions, together with the original images, are used to construct training samples for the image watermark protection network {Original Image I}. t Editing command INS}; In each iteration of the image watermark protection network, first randomly generate a binary watermark bit string M t The binary watermark bit strings are arranged in rows to obtain a two-dimensional watermark bit matrix M. t_2 The original image and the watermark feature matrix M are combined. t_2 The images are input together into the image processing neural network to obtain the watermarked image I. w Using the gradient truncation sampling strategy, in the N-step sampling process of the image editing in the diffusion image editing model, in the first (Nk) steps of the sampling process, the editing command and the watermark image I are... w The images are input together into the diffusion image editing model for gradient truncation to obtain an intermediate generated image. In the subsequent k steps of the sampling process, the intermediate generated image, the editing instructions, and the watermark image I are... w The gradients are then fed into the diffusion image editing model for gradient propagation, and then fed back to the watermark embedding module and the watermark extraction module to finally obtain the generated edited image I. g The generated edited image I g The input is given to the watermark extraction module to obtain the edited image I generated. g Extracted watermark bit string M p The watermark image I is analyzed using pixel-constrained mean square error loss. w and the original image I t Constraints are applied, and the latent space mean square error loss is used to adjust the latent space of the watermarked image I. w and the original image I tConstraints are applied, and the sum of the pixel constraint mean square error loss and the latent space mean square error loss is used as the watermark embedding mean square error loss to update the network parameters of the watermark embedding module using gradient descent. The parameters of the diffusion image editing model remain unchanged. The mean square error loss of the image watermark extraction before editing is used to update the binary watermark bit string M. t and the watermarked image I w Extracted watermark bit string M pb Constraints are applied, and the mean squared error loss of the edited image watermark extraction is used to adjust the binary watermark bit string M. t and the extracted watermark bit string M p Constraints are applied, and the mean squared error loss of watermark extraction before editing and the mean squared error loss of watermark extraction after editing are added together as the mean squared error loss of watermark extraction. The network parameters of the watermark extraction module are then updated by gradient descent. Finally, the mean squared error loss of watermark embedding and the mean squared error loss of watermark extraction are added together as the total loss of the image watermark protection network.

[0052] The pixel constraint mean square error loss L enc_pixel The function form is:

[0053]

[0054] Where MSE is the mean squared error loss function; n is the total number of pixels in the image; I ti For the original image I t The pixel value at position i; I wi Image with watermark I w The pixel value at the i-th position;

[0055] The hidden space mean square error loss L enc_latent The function form is:

[0056] L enc_latent =MSE(VAE(I t ),VAE(I w ))

[0057] VAE stands for Variational Autoencoder;

[0058] The watermark embedding mean square error loss L enc The function form is:

[0059] L enc =L enc_pixel +L enc_latent

[0060] The mean square error loss of the image watermark extraction before editing, L dec_before The function form is:

[0061] L dec_before =MSE(M t M pb )

[0062] The mean square error loss of watermark extraction of the edited image L dec_after The function form is:

[0063] L dec_after =MSE(M t M p )

[0064] The watermark extraction mean square error loss L dec The function form is:

[0065] L dec =L dec_before +L dec_after

[0066] The total loss of the image watermark protection network L total The function form is:

[0067] L total =L enc +L dec .

[0068] It should be noted that the original data in this embodiment of the invention comes from the image-editing instruction pair dataset. The implementation of this dataset belongs to the prior art, and for details, please refer to the prior art literature: Brooks T, Holynski A, Efros AA. Instructpix2pix: Learning to follow image editing instructions[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:18392-18402. In the image-editing instruction pair dataset, each training sample is in the form of several images and the corresponding real text descriptions. Several images are used in the training process of the above-mentioned watermark image protection network, and the real text descriptions corresponding to the images are used as the input of the above-mentioned large language model. The large language model outputs the editing instructions corresponding to the images, which will not be elaborated further.

[0069] It should be noted that the number of training samples in the fine-tuning dataset of this invention can be selected according to the actual situation, and is not limited in this invention embodiment.

[0070] It should be noted that, in this embodiment of the invention, the image editing sampling process of the diffusion image editing model is creatively regarded as a distortion process and integrated into the watermark embedding module and the watermark extraction module. Specifically, during the training process of the above-mentioned image watermark protection network, the parameters of the diffusion image editing model are fixed and not updated. However, since the diffusion image editing model is located between the watermark embedding module and the watermark extraction module, gradient propagation is required to optimize the watermark embedding module and the watermark extraction module. However, since the image editing sampling process of the diffusion image editing model requires N steps, that is, the input data needs to be repeatedly input into the above-mentioned diffusion image editing model, it requires a large amount of GPU memory. Therefore, in this embodiment of the invention, a gradient truncation sampling strategy is adopted. During the N-step sampling process of the image editing in the diffusion image editing model, the editing instructions and the watermark image are input into the diffusion image editing model together. The first (Nk) steps of the sampling process are gradient truncation to obtain an intermediate generated image. The intermediate generated image, the editing instructions, and the watermark image are then input into the diffusion image editing model together. In the last k steps of the sampling process, gradient flow is performed, and the gradient is passed back to the watermark embedding module and the watermark extraction module to finally obtain the generated edited image. Throughout the entire sampling process, the watermark embedding module, the diffusion image editing model, and the watermark extraction module can be jointly trained. During the entire optimization and parameter update process, the gradient can be passed from back to front to optimize the parameters of the watermark embedding module and the watermark extraction module, greatly reducing the memory usage. At the same time, considering that the first part of the sampling process of the diffusion image editing model forms some basic outlines of the image and the quality of the generated image is also very low, the last k steps of image detail optimization are more important and the quality of the generated image is also higher. Therefore, the gradient truncation strategy adopted in this invention is also very reasonable.

[0071] It should be noted that in this embodiment of the invention, considering that most diffusion image editing models are based on StableDiffusion, its implementation is existing technology. For details, please refer to the existing technical literature: Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022:10684-10695. The aforementioned StableDiffusion operates on the image in the latent space. The latent space can be understood as transforming the original image through a series of convolutional layers into a smaller feature image. The feature image is generally reduced to 1 / 8 of the original image. That is, a 512*512 image, after passing through a variational autoencoder (VAE), will become a 64*64 feature image. The implementation of the latent space is existing technology. For details, please refer to the existing technical literature: Esser P, Rombach R, Ommer B. Taming transformers for high-resolution image. synthesis[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:12873-12883. Therefore, in this embodiment of the invention, the latent space mean square error loss L is constructed by combining the idea of ​​latent space. enc_latent .

[0072] It should be noted that, in this embodiment of the invention, step S2 specifically involves: randomly generating a binary watermark bit string as embedded watermark information, wherein the binary watermark bit string is a vector of size (1, K), where 1 represents the number of feature channels; arranging the binary watermark bit string row-wise to obtain a vector of size K. A two-dimensional watermark bit matrix is ​​obtained. Using deconvolution layer operations or upsampling interpolation operations, the size of the two-dimensional watermark bit matrix is ​​expanded to the same size as the color image to be protected, resulting in a watermark feature matrix. The image to be detected, with a size of 3×H×H, and the watermark feature matrix, with a size of (1, H, H), are horizontally concatenated along the first channel dimension to obtain an image watermark mixing matrix of size (4, H, H). This image watermark mixing matrix is ​​then input into the image processing neural network to obtain the watermarked image of size (3, H, H). In this embodiment of the invention, K = 4 is preferred.

[0073] It should be noted that the large language model used in the embodiments of the present invention can be any open-source and general-purpose model, and can be selected according to the actual situation. Any model that can be used to generate editing instructions is acceptable. In the embodiments of the present invention, the ChatGPT model is specifically used, and its implementation method belongs to the prior art. For details, please refer to the prior art literature: Ouyang L, WuJ, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in Neural Information Processing Systems,2022,35:27730-27744., which will not be elaborated further.

[0074] It should be noted that the watermark embedding module used in the embodiments of the present invention can be from any open-source and general model, and can be selected according to the actual situation. In the embodiments of the present invention, U-Net is specifically used, and its implementation method belongs to the prior art. For details, please refer to the prior art literature: Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medical ImageComputing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference,Munich,Germany,October 5-9,2015,Proceedings,Part III 18.SpringerInternational Publishing,2015:234-241., which will not be elaborated further.

[0075] It should be noted that the diffusion image editing model used in the embodiments of the present invention can be any open-source and general-purpose model, which can be selected according to the actual situation. Any model that can be used for image description tasks after fine-tuning can be used. In the above diffusion image editing model, the input generally includes the image to be edited and the editing instructions that the user can arbitrarily specify. The N-step sampling process of image editing in the above diffusion image editing model includes the number of sampling and denoising steps and the text guidance weight parameter. The two parameters often have a certain adjustment range. Therefore, during the training of the diffusion image editing model, they can be randomly selected to simulate real application scenarios as much as possible to improve robustness. In the embodiments of the present invention, the InstructPix2Pix model is specifically used. Its implementation method belongs to the prior art. For details, please refer to the prior art literature: Brooks T, Holynski A, Efros AA. Instructpix2pix: Learning to follow image editing instructions[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:18392-18402., which will not be elaborated here.

[0076] It should be noted that the image processing neural network used in the embodiments of the present invention can be any open-source and general-purpose model, which can be selected according to the actual situation. Any model that can be used for image description tasks after fine-tuning can be used. In the embodiments of the present invention, the UNet network is specifically used. Its implementation method belongs to the prior art. For details, please refer to the prior art literature: Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference,Munich,Germany,October5-9,2015,Proceedings,Part III 18.Springer International Publishing,2015:234-241., which will not be elaborated here.

[0077] It should be noted that the watermark extraction module used in the embodiments of the present invention can be any open-source and general-purpose model, and can be selected according to the actual situation. In this embodiment of the present invention, ResNet is specifically used, and its implementation method belongs to the prior art. For details, please refer to the prior art literature: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778., which will not be elaborated further.

[0078] To verify the effectiveness of this invention, the verification results using the method proposed in the embodiments of this invention are given below. The specific steps are as described above and will not be repeated here; the main focus is on the technical effects and implementation details. In the embodiments of this invention, a reasonable threshold 'a' is set.

[0079] The extracted watermark bit string M p and the specified binary watermark bit string M t The comparison is performed, and if the bit error rate (BER) is less than a, then the generated edited image I is considered acceptable. g It is from the original image I t Obtained through editing and conversion.

[0080] As shown in Table 1, BER (Bit Error Rate) represents the bit error rate, which is the ratio of the number of predicted erroneous bits to the total number of bits in each image. A lower BER value indicates a higher watermark extraction rate. PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) measure the similarity between the watermarked image and the original image; higher values ​​for both indicate higher visual quality of the watermarked image. Generally, a BER below 10%, PSNR above 30, and SSIM above 0.9 indicate that the watermark embedding and extraction processes are relatively effective. Therefore, the experimental results confirm the effectiveness of this scheme.

[0081] Table 1

[0082] 5.3 35.49 0.98

[0083] Furthermore, in this embodiment of the invention, the robustness of the above-mentioned watermark protection network was further tested, that is, when the generated edited image I... g After some disturbances, can the method proposed in this embodiment of the invention effectively detect and accurately identify the watermark? As shown in Table 2, it can be seen that the watermark protection network of this embodiment of the invention has good robustness to most disturbances.

[0084] Table 2

[0085]

[0086] Finally, this embodiment of the invention provides schematic diagrams showing the curve changes of three evaluation indicators—watermark extraction error rate, PSNR, and SSIM—during the training process of the watermark protection network. The horizontal axis of all three diagrams represents the number of training steps, from 0 to 20000, while the vertical axes represent the watermark extraction error rate, PSNR, and SSIM, respectively. It can be seen that the method proposed in this embodiment of the invention can effectively reduce the watermark extraction error rate and improve the visual quality indicators PSNR and SSIM during the training process.

[0087] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for image watermark protection based on a diffusion image editing model, characterized in that, Includes the following steps: S1. Obtain a trained image watermark protection network, the image watermark protection network including a watermark embedding module, a diffusion image editing model, and a watermark extraction module; During the training process of the image watermark protection network, the watermark embedding module, the diffusion image editing model, and the watermark extraction module are jointly trained using a gradient truncation sampling strategy. S2. Input the watermark bit string specified by the image owner and the image to be protected together into the watermark embedding module to obtain the watermarked image; S3. Input the editing instructions specified by the image editor along with the watermark image into the diffusion image editing model to obtain the generated edited image; the specified editing instructions are a sentence that can describe the image editing method; S4. Input the generated edited image into the watermark extraction module to obtain the extracted watermark bit string, compare the extracted watermark bit string with the specified watermark bit string to complete the watermark protection process of the image to be protected. During the training process of the image watermark protection network, a pre-trained large language model is used. Real text descriptions corresponding to the original images are input into the large language model, where each text description is a sentence that describes the image content. The large language model outputs editing instructions, and these instructions, along with the original images, are used to construct the training samples for the image watermark protection network in the form of {original image}. Edit command INS}; In each iteration of the image watermark protection network, a binary watermark bit string is first randomly generated. The binary watermark bit strings are arranged in rows to obtain a two-dimensional watermark bit matrix. The original image and the watermark feature matrix are combined. The images are input together into the image processing neural network to obtain the watermarked image. Using the gradient truncation sampling strategy, in the image editing of the diffusion image editing model... During the sampling process, the first step During the sampling process, the editing instructions and the watermark image are... The images are input together into the diffusion image editing model for gradient truncation to obtain the intermediate generated image. During the sampling process, the intermediate generated image, the editing instructions, and the watermark image are... The images are then fed into the diffusion image editing model for gradient propagation. The gradients are then passed back to the watermark embedding module and the watermark extraction module to finally obtain the generated edited image. The generated edited image The input is given to the watermark extraction module to obtain the generated edited image. Extracted watermark bit string ; The watermark image is analyzed using pixel-constrained mean square error loss. and the original image Constraints are applied to the watermarked image in the latent space using the latent space mean square error loss. and the original image Constraints are applied, and the sum of the pixel constraint mean square error loss and the latent space mean square error loss is used as the watermark embedding mean square error loss to update the network parameters of the watermark embedding module using gradient descent. The parameters of the diffusion image editing model remain unchanged. The mean square error loss of the image watermark extraction before editing is used to update the binary watermark bit string. and the watermarked image Extracted watermark bit string Constraints are applied to the binary watermark bit string using the mean squared error loss extracted from the edited image watermark. and the extracted watermark bit string Constraints are applied, and the mean squared error loss of watermark extraction before editing and the mean squared error loss of watermark extraction after editing are added together as the mean squared error loss of watermark extraction. The network parameters of the watermark extraction module are then updated by gradient descent. Finally, the mean squared error loss of watermark embedding and the mean squared error loss of watermark extraction are added together as the total loss of the image watermark protection network. The pixel constraint mean square error loss The function form is: ； in, Let the mean squared error loss function be used. This represents the total number of pixels in the image. Original image The Middle The pixel value at each position; Watermarked image The Middle The pixel value at each position; The hidden space mean square error loss The function form is: ； in, It is a variational autoencoder; The watermark embedding mean square error loss The function form is: ； The mean square error loss of image watermark extraction before editing The function form is: ； The mean square error loss of watermark extraction in the edited image The function form is: ； The mean square error loss of watermark extraction The function form is: ； The total loss of the image watermark protection network The function form is:

2. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The specific process of step S2 is as follows: randomly generate a binary watermark bit string, arrange the binary watermark bit string in rows to obtain a two-dimensional watermark bit matrix, then expand the size of the two-dimensional watermark bit matrix to be the same as the size of the color image to be protected to obtain a watermark feature matrix, and input the image to be protected and the watermark feature matrix together into an image processing neural network to obtain a watermarked image.

3. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The large language model used is the ChatGPT model.

4. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The original image and the actual text description are derived from an image-editing instruction pair dataset.

5. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The watermark embedding module uses a U-Net network.

6. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The diffusion image editing model uses the InstructPix2Pix model.

7. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The image processing neural network uses the U-Net network.

8. The image watermark protection method based on a diffusion image editing model as described in claim 2, characterized in that, The size expansion method is through deconvolution or upsampling interpolation.

9. The image watermark protection method based on a diffusion image editing model as described in claim 1, characterized in that, The watermark extraction module uses the ResNet network.

Images