An image watermarking system and method for style transfer deep editing

Abstract

The application discloses an image watermarking system and method for style transfer deep editing, and the system comprises a style transfer deep editing module, which is used for extracting original image features and performing style transfer deep editing; after the features are extracted by using a watermark encoding submodule, the watermark features and the image features are fused in an image reconstruction stage, and the style transfer task and the watermark embedding task of the image are simultaneously completed; a discriminator module is used for judging the quality of the output image of the style transfer deep editing module, and the visual effect of the generated image is improved through adversarial training; a watermark extraction module uses a pre-constructed convolutional neural network to decouple the host image features and the watermark features, and uses a reconstruction loss to guide the extraction and recovery of the watermark. According to the style transfer task, the encoding and embedding of the watermark are completed in the process of style transfer deep editing, so that the user cannot bypass the embedding process of the watermark, and the network supervision is assisted, and the use of the style transfer deep editing model is standardized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image watermarking system and method for style transfer depth editing. Background Technology

[0002] Style transfer can be defined as how to transform a given scene into another possible representation of that scene, given one possible representation. In recent years, with the development of deep learning, a large number of fake images generated by deep editing using style transfer have circulated online, posing potential risks and challenges to national and social security and network environment monitoring. One representative method is pix2pix based on the U-Net network. This method is the first general framework for the style transfer problem. After training on a pairwise training set of input and output images, the pix2pix generator network can directly transform the style of the input image into the style of the output image.

[0003] Currently, there are various methods for detecting forged images generated through deep editing. Most passive detection methods identify forgeries by detecting distortions and artifacts in the forged image. However, this approach is increasingly unable to handle more complex and realistic generation algorithms, and its generalization ability is often poor, exhibiting weak transferability across different datasets. Active detection methods, such as watermarking, typically encode and embed watermarks before image transmission and then verify image tampering by extracting the watermark after transmission. However, this method still cannot prevent malicious tampering, as perpetrators can modify the original image and then embed the watermark to mislead detectors, making effective monitoring difficult. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an image watermarking method for style transfer deep editing. This method aims to embed the watermark during the style transfer deep editing process, making it impossible for users to bypass the embedding process and aiding network monitoring. First, this invention designs an image watermarking encoding module based on a convolutional neural network to extract the encoded features of the image watermark. Then, based on modifications to the U-Net generator from pix2pix, the watermark features and input image features are concatenated and subsequently reconstructed during the image reconstruction stage, outputting an image with the embedded watermark and completed style transfer. Next, a watermark extractor based on a convolutional network is used to extract features from the output image and reconstruct the watermark, using reconstruction loss to guide watermark extraction and restoration. Finally, a distortion simulation module is used for adversarial training to further improve the robustness of the watermark. The specific technical solution of this invention is as follows:

[0005] This invention provides an image watermarking system for style transfer depth editing, comprising: a style transfer depth editing module, a discriminator module, a watermark extraction module, and a distortion simulation module, wherein...

[0006] The style transfer deep editing module includes a style transfer generator module, a watermark encoding module, and a feature fusion module. The style transfer generator module, based on the U-Net generation architecture, takes the image to be edited as input and has a symmetrical encoder-decoder structure. The encoder extracts features from the image, converting it into high-dimensional features, and the decoder upsamples and reconstructs the image. Skip connections are used in corresponding stages of the encoder and decoder to share feature information, preserving the image feature information of the intermediate layers during image reconstruction. The watermark encoding module, based on a convolutional neural network, extracts features from the input image to be embedded with a watermark, converting the watermark image into high-dimensional features and preserving the output watermark feature information. The feature fusion module fuses the image feature information and watermark feature information from the intermediate layers, transforms the fused features, and inputs them into subsequent reconstruction layers to complete the style transfer and watermark embedding of the image.

[0007] The discriminator module is based on the PatchGAN network and is used to determine the quality of the image output by the style transfer deep editing module.

[0008] The watermark extraction module is based on a convolutional neural network and takes the output image of the style transfer deep editing module as input to extract watermark image features.

[0009] The distortion simulation module is based on a convolutional neural network. It transforms the output image of the style transfer deep editing module and inputs the output image into the watermark extraction module. It uses adversarial training to improve the robustness of the watermark in the face of unknown post-processing operations.

[0010] Furthermore, the style transfer generator module is built based on U-Net, and the watermark encoding module uses a 5×5 transposed convolution with padding of 2 and the ReLU activation function;

[0011] After the feature fusion module performs the splicing operation, it uses a 1×1 convolutional layer and the ReLU activation function to reduce the feature dimensionality.

[0012] The watermark extraction module is built on a convolutional neural network. First, a 4-layer convolutional network module with 16 channels is used to decouple the features of the host image and the watermark image. Each convolutional network module uses ReLU as the activation function. Then, a downsampling module based on convolutional network-average pooling layer-ReLU activation layer is used to extract the watermark image.

[0013] The distortion simulation module consists of two convolutional networks. The first convolutional network changes the number of input feature channels from 3 to 16, and the second convolutional network restores the number of channels from 16 to 3. LeakyReLU is used as the activation function between the two convolutional networks.

[0014] Furthermore, the style transfer deep editing module, watermark extraction module, and distortion simulation module output dimensions are three channels.

[0015] Furthermore, both the image to be edited and the image to be embedded with the watermark are 3-channel images, with the image to be edited being 256×256 pixels and the image to be embedded with the watermark being 64×64 pixels.

[0016] This invention also provides an image watermarking method for style transfer-oriented depth editing, which uses the above-mentioned image watermarking system for image editing and includes the following steps:

[0017] S1: Construct the training dataset;

[0018] S1-1: Construct a style transfer training dataset by using paired image pairs {A,B} as training data, where A and B are different descriptions of the same scene;

[0019] S1-2: Construct a watermark image training dataset, collect images to be embedded with watermarks as training data, and preprocess the images to be embedded with watermarks into black and white watermark images w with a size of 64×64, a foreground color of white and a background color of black.

[0020] S2: Input the training dataset into the style transfer deep editing module and output a deep-edited image;

[0021] S2-1: Use the image obtained in step S1-1 to perform style transfer training on {A, B}, with the transfer direction being A→B. Input the image to be transformed, A, into the style transfer generator module, and output the image feature size as C. A *H A *W A C A H A W A These represent the number of output channels, length, and width, respectively.

[0022] S2-2: Input the black and white watermark image w obtained in step S1-2 into the watermark encoding module to obtain the high-dimensional features of the watermark image, and output the size of the watermark features as C. w *H A *W A C w H A W A These represent the number of output channels, length, and width, respectively.

[0023] S2-3: The image features C obtained in step S2-1 are... A *H A *W A The watermark feature C obtained from S2-2 w *H A *W A The input feature fusion module performs a concatenation operation to obtain (C) A +C w )*H A *W A After fusing the features, a 1×1 convolutional network is used to reduce the dimensionality to C. A *H A *W A The image is input into the subsequent reconstruction layer to complete the style transfer, and the output image B′ after style transfer is completed. B′ is the transferred image generated after embedding the image watermark, which is a 256×256 image with 3 channels.

[0024] S3: Input the image B′ obtained in S2-3 into the discriminator module. The network output of the discriminator module is an N×N matrix. Each element in the matrix has only two values: real or fake, corresponding to a local region in the input image. Finally, the N×N output results are averaged to serve as the discriminator module's final judgment on the input image.

[0025] S4: Input the image B′ obtained in S2-3 into the watermark extraction module for watermark extraction. First, use the convolutional neural network module to decouple the watermark image features and the host image features, and then reconstruct the watermark image to output a 3-channel 64×64 watermark image w′.

[0026] S5: Repeat steps S2-S4 until the loss function converges and the quality of the style transfer generated image B′ meets the given requirements.

[0027] S6: Watermark robustness adversarial training, adding a distortion simulation module, and simultaneously training and updating parameters for the style transfer deep editing module, discriminator module, and watermark extraction module;

[0028] S6-1: Complete image style transfer and watermark embedding according to step S2 to obtain the generated image B″, and input B″ into the discriminator module;

[0029] S6-2: Input image B″ into the distortion simulation module. The distortion simulation module generates perturbations through a convolutional neural network to simulate the distortion or post-processing operations encountered by the image during actual transmission, and outputs a distortion simulation sample B″. adv , B″ adv The image size is a 256×256 image with 3 channels;

[0030] S6-3: Transform the distorted simulation sample B″adv Image B″ is input into the watermark extraction module, and the watermark image w″ is extracted from each image. adv and w″;

[0031] S7: Repeat step S6 until the loss function converges and the quality of the generated image B″ and the extracted watermark image w″ meets the given requirements.

[0032] S8: Testing and Use;

[0033] S8-1: Input the image to be style-transfer deep editing and the watermark image to be embedded into the style-transfer deep editing module obtained in step S7, and output the deep-edited image with embedded watermark;

[0034] S8-2: Input the depth-edited image obtained in S8-1 into the watermark extraction module obtained in S7, and output the watermark image.

[0035] Furthermore, in steps S2 and S3, the generator and discriminator in GAN are trained alternately to improve the image quality generated by the style transfer deep editing module. The training objective of the style transfer deep editing module is that the discriminator module cannot determine whether the input image is a real image or a deep-edited image. The training objective of the discriminator module is to effectively distinguish whether the input image is real or not. The two ultimately reach Nash equilibrium through a zero-sum game.

[0036] Furthermore, the loss function of the style transfer deep editing module in step S2 is:

[0037]

[0038] in,

[0039]

[0040]

[0041]

[0042] In the formula, x is the input image of the style transfer deep editing module, y is the ground truth image corresponding to x, z is the noise input to the style transfer generator module, G(x,z) represents the generated image, w is the ground truth image of the watermark image, w' is the watermark image extracted by the watermark extraction module, and λ and β are hyperparameters. Let D(x,y) be the adversarial training function for the conditional generator network, and let D(x,y) be the output of the discriminator module. An optimization function to improve the quality of generated images. This represents the loss of the watermark image.

[0043] Furthermore, the loss function of the style transfer deep editing module in step S6 is:

[0044]

[0045] in,

[0046]

[0047]

[0048]

[0049]

[0050] In the formula, λ, β, and γ are hyperparameters, G(x,z) is the generated image from the depth editing module, and y′ adv Here, w represents the distortion adversarial sample generated by the distortion simulation module, w″ represents the ground truth value of the watermark image, and w″ represents the watermark extracted from G(x,z) by the watermark extraction module. adv For the watermark extraction module, from y′ adv The watermark extracted from it; This is to address the watermark distortion caused by adding the distortion simulation module; This is to account for the distortion in the generated image caused by adding the distortion simulation module.

[0051] Furthermore, the loss function of the watermark extraction module in step S4 is:

[0052]

[0053] In the formula, Dec represents the watermark extraction module, w is the true value of the watermark image, and w' is the watermark image extracted by the watermark extraction module.

[0054] The loss function of the watermark extraction module in step S6 is:

[0055]

[0056] In the formula, Dec represents the watermark extraction module, w is the ground truth of the watermark image, and w″ is the watermark image extracted by the watermark extraction module from G(x,z). adv For the watermark extraction module, from y′ adv The watermark image was extracted.

[0057] Furthermore, the loss function of the distortion simulation module in step S6 is:

[0058]

[0059] In the formula, G advFor the distortion simulation module, α1 and α2 are hyperparameters. α1 controls the distortion intensity generated by the distortion simulation module, and α2 controls the intensity of information loss generated by the distortion simulation module.

[0060] The beneficial effects of this invention are as follows:

[0061] 1. The image watermarking method for style transfer deep editing of the present invention can encode and embed watermarks while completing the style transfer task, so that users of the generated model cannot deliberately avoid embedding watermark information, which can assist in the supervision of the use of deep editing models.

[0062] 2. The network architecture and loss function designed in this invention can embed watermarks without significantly affecting the image generation quality, and can extract watermark information relatively completely;

[0063] 3. This invention applies adversarial training to further enhance the robustness of the watermark against unknown distortions, enabling the watermark to cope with various common post-processing operations. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:

[0065] Figure 1 This is an architecture diagram of the image watermarking method for style transfer-oriented depth editing according to the present invention;

[0066] Figure 2 This is a schematic diagram of the training process of the image watermarking method for style transfer-oriented depth editing of the present invention;

[0067] Figure 3 This is a schematic diagram of feature fusion in the style transfer deep editing module of this invention;

[0068] Figure 4 This is a schematic diagram illustrating the testing process of the image watermarking method for style transfer-oriented depth editing according to the present invention. Detailed Implementation

[0069] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0070] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0071] This invention provides an image watermarking system for style transfer-based depth editing, such as... Figure 1 As shown, it includes: a style transfer deep editing module, a discriminator module, a watermark extraction module, and a distortion simulation module, among which...

[0072] The style transfer deep editing module includes a style transfer generator module, a watermark encoding module, and a feature fusion module. The input is the image to be edited and the image to be embedded with watermark. The watermark encoding module extracts watermark features, and in the image reconstruction stage, the watermark features and the image features extracted by the style transfer generator module are spliced ​​together before subsequent image reconstruction is performed, thus completing the image style transfer task and watermark embedding task at the same time.

[0073] The style transfer generator module is based on the U-Net generation architecture. The input is the image to be edited. It is a symmetrical encoder-decoder structure. The encoder extracts features from the original image and converts the image into high-dimensional features. The decoder then upsamples the image to reconstruct it. Skip connections are used in the corresponding stages of the encoder and decoder to share feature information and retain the image feature information of the intermediate layers in the image reconstruction stage.

[0074] The watermark encoding module extracts features from the input image to be embedded with watermark based on a convolutional neural network, converts it into high-dimensional features, and retains the watermark feature information of its output.

[0075] The feature fusion module is used to fuse the image intermediate layer feature information and watermark feature information, and after transforming the fused features, it is input into the subsequent reconstruction layer to continue to complete the image reconstruction.

[0076] The discriminator module is a PatchGAN-based network used to judge the quality of the output image of the style transfer deep editing module. Based on previous research on style transfer tasks and generative adversarial networks (GANs), the PatchGAN discriminator can effectively penalize the structural differences between the output image and the target image.

[0077] The watermark extraction module takes the output image of the style transfer deep editing module as input, uses a pre-built convolutional neural network to extract image features, and uses reconstruction loss to guide the extraction and restoration of watermarks, which can extract watermark information relatively completely from the transferred image.

[0078] The distortion simulation module transforms the output image of the style transfer deep editing module through a convolutional neural network and inputs the output image into the watermark extraction module. It uses adversarial training to improve the robustness of the watermark in the face of unknown post-processing operations.

[0079] Furthermore, the style transfer generator module is built based on U-Net, the watermark encoding module uses a 5×5 transposed convolution with padding of 2 and the ReLU activation function, the feature fusion module performs feature dimensionality reduction using a 1×1 convolutional layer and the ReLU activation function after concatenation operation, the watermark extraction module is built based on a convolutional neural network, first using a 4-layer convolutional network module with 16 channels to decouple the features of the host image and the watermark image, each convolutional network module uses ReLU as the activation function, and then a downsampling module based on convolutional network-average pooling layer-ReLU activation layer is used to extract the watermark image; the distortion simulation module consists of two convolutional network layers, the first convolutional network changes the input feature channels from 3 to 16, the second convolutional network restores the number of channels from 16 to 3, and LeakyReLU is used as the activation function between the two convolutional networks.

[0080] Preferably, the feature fusion of the feature fusion module is completed in the penultimate layer of the U-Net upsampling stage.

[0081] Furthermore, the style transfer deep editing module, watermark extraction module, and distortion simulation module output dimensions are three channels.

[0082] Furthermore, both the image to be edited and the image to be embedded with the watermark are 3-channel images, with the image to be edited being 256×256 pixels and the image to be embedded with the watermark being 64×64 pixels.

[0083] This invention also provides an image watermarking method for style transfer depth editing, such as... Figure 2 As shown, it includes the following steps:

[0084] S1: Construct the training dataset;

[0085] S1-1: Constructing a style transfer training dataset requires building image pairs {A,B} as training data, where A and B are different descriptions of the same scene, such as {labeled map, real photo}.

[0086] S1-2: Construct a training dataset for watermark images. This method collects the school badges of different universities as watermark images to be embedded. In order to ensure the quality of style transfer images, the original school badge images need to be preprocessed into images with a size of 64×64, with the foreground color being white and the background color being black, i.e., black and white watermark images w.

[0087] S2: Complete in-depth style transfer editing, such as... Figure 3 As shown;

[0088] S2-1: Use the images obtained in step S1-1 to perform style transfer training on {A, B}, with the transfer direction being A→B. Input the image to be transformed, A, into the U-Net-based style transfer generator module. Image B is used as the ground truth in the subsequent loss function calculation to guide the generation training. The size of the output image feature is C. A *H A *W A C A H A W A These represent the number of output channels, length, and width, respectively.

[0089] S2-2: Input the black and white watermark image w obtained in step S1-2 into the watermark encoding module to obtain the high-dimensional features of the watermark image, and output the size of the watermark features as C. w *H A *W A C w H A W A These represent the number of output channels, length, and width, respectively.

[0090] S2-3: The image features C obtained in step S2-1 are... A *H A *W A The watermark feature C obtained from S2-2 w *H A *W A The input is fed into the feature fusion module for concatenation, resulting in (C) A +C w )*H A *W A After fusing the features, a 1×1 convolutional network is used to reduce the dimensionality to C. A *H A *W A The image is input into the subsequent reconstruction layer to complete the style transfer, and the output image B′ after style transfer is completed. B′ is the transferred image generated after embedding the image watermark. The size of B′ and B is the same, which is still a 256×256 image with 3 channels.

[0091] S3: Input the image B′ obtained from S2-3 into the discriminator module. The network output of this module is an N×N matrix. Each element in the matrix has only two values: real or fake, corresponding to a local region in the input image. Finally, the N×N output results are averaged to serve as the discriminator module's final judgment on the input image.

[0092] S4: Input the image B′ obtained in S2-3 into the watermark extraction module for watermark extraction. First, use the convolutional neural network module to decouple the features of the watermark image and the features of the host image, and then reconstruct the watermark image to output a 3-channel 64×64 watermark image w′.

[0093] S5: Repeat steps S2-S4 until the loss function converges and the quality of the style transfer generated image B′ and the watermark w′ are relatively stable.

[0094] S6: Watermark robustness adversarial training, adding a distortion simulation module, and simultaneously training and updating parameters for the style transfer deep editing module, discriminator module, and watermark extraction module;

[0095] S6-1: Complete image style transfer and watermark embedding according to step S2 to obtain generated image B″. In order to prevent the quality of generated image B″ from degrading after training with the distortion simulation module, input B″ into the discriminator module and constrain the quality of image B″ through adversarial training with the style transfer deep editing module.

[0096] S6-2: Input the image B″ from S6-1 into the distortion simulation module. The distortion simulation module generates perturbations through a convolutional neural network to simulate the unknown distortions or various post-processing operations that the image may encounter during actual transmission, and outputs the distortion simulation sample B″. adv , B″ adv The image size remains a 256×256 image with 3 channels;

[0097] S6-3: Transform the distorted simulation sample a″ adb The watermark images w″ extracted from the B″ image of S6-1 are input into the watermark extraction module. adv and w″;

[0098] S7: Repeat step S6 until the loss function converges and the quality of the generated image B″ and the extracted watermark image w″ is stable.

[0099] S8: Testing and use, such as Figure 4 As shown;

[0100] S8-1: Input the image to be style-transfer deep editing and the watermark image to be embedded into the style-transfer deep editing module obtained in S7, and output the deep-edited image with embedded watermark.

[0101] S8-2: Input the depth-edited image obtained in S8-1 into the watermark extraction module obtained in S7, and output the watermark image.

[0102] Furthermore, in steps S2 and S3, the generator and discriminator in GAN are trained alternately to improve the image quality generated by the deep editing module. The training goal of the deep editing module is to generate images that are realistic enough so that the discriminator module cannot determine whether the input image is a real image or a deep-edited image. The training goal of the discriminator module is to effectively distinguish whether the input image is real or not. The two ultimately reach Nash equilibrium through a zero-sum game.

[0103] Further, in step S5, the network parameters of the discriminator module, the watermark extraction module, and the style transfer deep editing module are updated sequentially until the loss function converges; in step S7, the network parameters of the discriminator module, the watermark extraction module, the distortion simulation module, and the style transfer deep editing module are updated sequentially until the loss function converges.

[0104] Furthermore, the loss function of the style transfer deep editing module in step S2 is:

[0105]

[0106] in,

[0107]

[0108]

[0109]

[0110] Where x is the input image of the style transfer deep editing module, y is the ground truth image corresponding to x, z is the noise input to the style transfer generator module to improve the diversity of generated images, G(x,z) represents the generated image, w is the ground truth of the watermark image, w' is the watermark image extracted by the watermark extraction module, and λ and β are hyperparameters. Let D(x,y) be the adversarial training function of the conditional generator network, and let D(x,y) be the output of the discriminator module. The optimization objective of the discriminator module is to maximize... The style transfer deep editing module should, as far as possible, enable... They shrink, and through adversarial training, they reach Nash equilibrium; An optimization function to further improve the quality of generated images. Choosing the L1 distance reduces the blurriness of the depth-edited output image, making the generated image as close as possible to the true value of the image while confusing the discriminator; The loss representing the watermark image is modeled using L2 distance to model the difference between the extracted watermark and the ground truth watermark, so that the watermark image embedded in the depth editing module can be extracted as completely as possible.

[0111] Furthermore, in step S6, after adding the distortion simulation module for joint training, the loss function of the style transfer deep editing module is adjusted as follows:

[0112]

[0113] in,

[0114]

[0115]

[0116]

[0117]

[0118] in, and Consistent with the above, λ, β, and γ are hyperparameters, G(x,z) is the generated image from the depth editing module, and y′ adv Here, w represents the distortion adversarial sample generated by the distortion simulation module, w″ represents the ground truth of the watermark image, and w″ represents the watermark image extracted from G(x,z) by the watermark extraction module. adv For the watermark extraction module, from y′ adv The watermark image extracted from it; To mitigate watermark distortion caused by the addition of the distortion simulation module, L2 distance is used to measure watermark distortion, enabling the watermark embedded in the generation module to effectively resist distortion. To mitigate the image distortion introduced by the distortion simulation module, L2 distance is used to measure the intensity of the distortion, thus constraining the style transfer network to generate more robust images.

[0119] Furthermore, the loss function of the watermark extraction module in step S4 is:

[0120]

[0121] Where Dec represents the watermark extraction module, w is the ground truth of the watermark image, and w' is the watermark image extracted by the watermark extraction module. The watermark extraction module extracts relatively complete watermark information by using L2 distance constraints, ensuring the visual quality of the extracted watermark image.

[0122] Furthermore, in step S6, after adding the distortion simulation module for joint training, the loss function of the watermark extraction module is adjusted as follows:

[0123]

[0124] Where Dec represents the watermark extraction module, w is the ground truth of the watermark image, and w″ is the watermark image extracted from G(x,z) by the watermark extraction module. adv For the watermark extraction module, from y′ adv The watermarked image extracted from it, G(x,z) and y′ adv The definition remains consistent with the above, further enhancing the robustness of watermarks against unknowable distortions.

[0125] Furthermore, in step S6, the loss function of the distortion simulation module is:

[0126]

[0127] Among them, G adv For the distortion simulation module, G(x,z),y ′ adv ,w,w″ adv The meaning remains consistent with the previous definition. α1 and α2 are hyperparameters. α1 controls the distortion intensity generated by the distortion simulation module, and α2 controls the intensity of information loss generated by the distortion simulation module. During training, the ratio of α1 to α2 needs to be carefully considered. If the intensity of the distortion perturbation is too high, the training process will be slow and the watermark's adaptability to distortion will be reduced. If the intensity of the distortion perturbation is too low, the watermark will not be robust enough to deal with unknown distortions.

[0128] To verify the effectiveness and practicality of this invention, facades (over 500 pairs) were used as the training dataset. The model was trained according to steps S1-S7, with Adam as the optimizer. The initial learning rate was set to 0.0002, and the two beam parameters were set to β1 = 0.5 and β2 = 0.999. 400 pairs of images were used to train the model, and 50 pairs were used to test it. A total of 200 iterations were performed. The first 100 iterations only trained the deep editing module, watermark extraction module, and discriminator module. The last 100 iterations added a distortion simulation network for joint training.

[0129] The model was evaluated using a test set of 50 image pairs. The trained model was tested according to step S8 above. The quality of the generated images was evaluated using the no-reference image evaluation methods FID, NIQE, PIQE, and BRIQSUE, with scores of 148.879, 51.314, 4.643, and 0.496, respectively. The style deep editing model without watermark embedding (the style transfer deep editing module only includes the style transfer generator module) scored 154.035, 61.405, 5.495, and 0.506 under the four methods. The present invention outperformed the control group in FID, NIQE, and PIQE, and its score under BRIQSUE was very close to that of the control group, indicating that the present invention can embed watermarks without significantly affecting the quality of the generated images. Following step S8, the extracted watermark was evaluated. The mean square error (MSE) and structural similarity (SSIM) were 0.000722 and 0.943, respectively. Furthermore, the extracted watermark image icon exhibited high visual quality, indicating that the present invention can embed and extract watermarks relatively completely. Following step S8, the image obtained in S8-1 underwent post-processing operations such as adjusting brightness, hue, contrast, saturation, Gaussian noise, and Gaussian blur to simulate image distortion in real-world applications. The post-processed image was then input into the watermark extraction module. The output watermark still maintained high visual quality and integrity, demonstrating the high robustness of the watermark embedded by the present invention. In conclusion, the present invention is effective and feasible.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image watermarking system for style transfer depth editing, characterized in that, include: The module includes a style transfer deep editing module, a discriminator module, a watermark extraction module, and a distortion simulation module. The style transfer deep editing module includes a style transfer generator module, a watermark encoding module, and a feature fusion module. The style transfer generator module, based on the U-Net generation architecture, takes the image to be edited as input and has a symmetrical encoder-decoder structure. The encoder extracts features from the image, converting it into high-dimensional features, and the decoder upsamples and reconstructs the image. Skip connections are used in corresponding stages of the encoder and decoder to share feature information, preserving the image feature information of the intermediate layers during image reconstruction. The watermark encoding module, based on a convolutional neural network, extracts features from the input image to be embedded with a watermark, converting the watermark image into high-dimensional features and preserving the output watermark feature information. The feature fusion module fuses the image feature information and watermark feature information from the intermediate layers, transforms the fused features, and inputs them into subsequent reconstruction layers to complete the style transfer and watermark embedding of the image. The discriminator module is based on the PatchGAN network and is used to determine the quality of the image output by the style transfer deep editing module. The watermark extraction module is based on a convolutional neural network and takes the output image of the style transfer deep editing module as input to extract watermark image features. The distortion simulation module is based on a convolutional neural network. It transforms the output image of the style transfer deep editing module and inputs the output image into the watermark extraction module. It uses adversarial training to improve the robustness of the watermark in the face of unknown post-processing operations.

2. The image watermarking system for style transfer-oriented depth editing according to claim 1, characterized in that, The style transfer generator module is built on U-Net, and the watermark encoding module uses a 5×5 transposed convolution with padding of 2 and the ReLU activation function. After the feature fusion module performs the splicing operation, it uses a 1×1 convolutional layer and the ReLU activation function to reduce the feature dimensionality. The watermark extraction module is built on a convolutional neural network. First, a 4-layer convolutional network module with 16 channels is used to decouple the features of the host image and the watermark image. Each convolutional network module uses ReLU as the activation function. Then, a downsampling module based on convolutional network-average pooling layer-ReLU activation layer is used to extract the watermark image. The distortion simulation module consists of two convolutional networks. The first convolutional network changes the number of input feature channels from 3 to 16, and the second convolutional network restores the number of channels from 16 to 3. LeakyReLU is used as the activation function between the two convolutional networks.

3. An image watermarking system for style transfer-oriented depth editing according to claim 1 or 2, characterized in that, The style transfer deep editing module, watermark extraction module, and distortion simulation module output three channels.

4. An image watermarking system for style transfer-oriented depth editing according to claim 1 or 2, characterized in that, Both the image to be edited and the image to be embedded with the watermark are 3-channel images. The image to be edited is 256×256 pixels, and the image to be embedded with the watermark is 64×64 pixels.

5. An image watermarking method for style transfer depth editing, characterized in that, Image editing using the image watermarking system according to any one of claims 1-4 includes the following steps: S1: Construct the training dataset; S1-1: Construct a style transfer training dataset by using paired image pairs {A,B} as training data, where A and B are different descriptions of the same scene; S1-2: Construct a watermark image training dataset, collect images to be embedded with watermarks as training data, and preprocess the images to be embedded with watermarks into black and white watermark images w with a size of 64×64, a foreground color of white and a background color of black. S2: Input the training dataset into the style transfer deep editing module and output a deep-edited image; S2-1: Use the image obtained in step S1-1 to perform style transfer training on {A, B}, with the transfer direction being A→B. Input the image to be transformed, A, into the style transfer generator module, and output the image feature size as C. A *H A *W A C A H A W A These represent the number of output channels, length, and width, respectively. S2-2: Input the black and white watermark image w obtained in step S1-2 into the watermark encoding module to obtain the high-dimensional features of the watermark image, and output the size of the watermark features as C. w *H A *W A C w H A W A These represent the number of output channels, length, and width, respectively. S2-3: The image features C obtained in step S2-1 are... A *H A *W A The watermark feature C obtained from S2-2 w *H A *W A The input feature fusion module performs a concatenation operation to obtain (C) A +C W )*H A *W A After fusing the features, a 1×1 convolutional network is used to reduce the dimensionality to C. A *H A *W A The image is input into the subsequent reconstruction layer to complete the style transfer, and the output image B′ after style transfer is completed. B′ is the transferred image generated after embedding the image watermark, which is a 256×256 image with 3 channels. S3: Input the image B′ obtained in S2-3 into the discriminator module. The network output of the discriminator module is an N×N matrix. Each element in the matrix has only two values: real or fake, corresponding to a local region in the input image. Finally, the N×N output results are averaged to serve as the discriminator module's final judgment on the input image. S4: Input the image B′ obtained in S2-3 into the watermark extraction module for watermark extraction. First, use the convolutional neural network module to decouple the watermark image features and the host image features, and then reconstruct the watermark image to output a 3-channel 64×64 watermark image w′. S5: Repeat steps S2-S4 until the loss function converges and the quality of the style transfer generated image B′ meets the given requirements. S6: Watermark robustness adversarial training, adding a distortion simulation module, and simultaneously training and updating parameters for the style transfer deep editing module, discriminator module, and watermark extraction module; S6-1: Complete image style transfer and watermark embedding according to step S2 to obtain the generated image B″, and input B″ into the discriminator module; S6-2: Input image B″ into the distortion simulation module. The distortion simulation module generates perturbations through a convolutional neural network to simulate the distortion or post-processing operations encountered by the image during actual transmission, and outputs a distortion simulation sample B″. adv , B″ adv The image size is a 256×256 image with 3 channels; S6-3: Transform the distorted simulation sample B″ adv Image B″ is input into the watermark extraction module, and the watermark image w″ is extracted from each image. adv and w″; S7: Repeat step S6 until the loss function converges and the quality of the generated image B″ and the extracted watermark image w″ meets the given requirements. S8: Testing and Use; S8-1: Input the image to be style-transfer deep editing and the watermark image to be embedded into the style-transfer deep editing module obtained in step S7, and output the deep-edited image with embedded watermark; S8-2: Input the depth-edited image obtained in S8-1 into the watermark extraction module obtained in S7, and output the watermark image.

6. The image watermarking method for style transfer-oriented depth editing according to claim 5, characterized in that, In steps S2 and S3, the generator and discriminator in GAN are trained alternately to improve the image quality generated by the style transfer deep editing module. The training objective of the style transfer deep editing module is that the discriminator module cannot determine whether the input image is a real image or a deep-edited image. The training objective of the discriminator module is to effectively distinguish whether the input image is real or not. The two ultimately reach Nash equilibrium through a zero-sum game.

7. The image watermarking method for style transfer-oriented depth editing according to claim 6, characterized in that, The loss function of the style transfer deep editing module in step S2 is: in, In the formula, x is the input image of the style transfer deep editing module, y is the ground truth image corresponding to x, z is the noise input to the style transfer generator module, G(x,z) represents the generated image, w is the ground truth image of the watermark image, w' is the watermark image extracted by the watermark extraction module, and λ and β are hyperparameters. Let D(x,y) be the adversarial training function for the conditional generator network, and let D(x,y) be the output of the discriminator module. An optimization function to improve the quality of generated images. This represents the loss of the watermark image.

8. The image watermarking method for style transfer-oriented depth editing according to claim 7, characterized in that, The loss function of the style transfer deep editing module in step S6 is: in, In the formula, λ, β, and γ are hyperparameters, G(x,z) is the generated image from the depth editing module, and y′ adv Here, w represents the distortion adversarial sample generated by the distortion simulation module, w″ represents the ground truth value of the watermark image, and w″ represents the watermark extracted from G(x,z) by the watermark extraction module. adv For the watermark extraction module, from y′ adv The watermark extracted from it; This is to address the watermark distortion caused by adding the distortion simulation module; This is to account for the distortion in the generated image caused by adding the distortion simulation module.

9. The image watermarking method for style transfer-oriented depth editing according to claim 8, characterized in that, The loss function of the watermark extraction module in step S4 is: In the formula, Dec represents the watermark extraction module, w is the true value of the watermark image, and w' is the watermark image extracted by the watermark extraction module. The loss function of the watermark extraction module in step S6 is: In the formula, Dec represents the watermark extraction module, w is the ground truth of the watermark image, and w″ is the watermark image extracted by the watermark extraction module from G(x,z). adv For the watermark extraction module, from y′ adv The watermark image was extracted.

10. The image watermarking method for style transfer-oriented depth editing according to claim 9, characterized in that, The loss function of the distortion simulation module in step S6 is: In the formula, G adv For the distortion simulation module, α1 and α2 are hyperparameters. α1 controls the distortion intensity generated by the distortion simulation module, and α2 controls the intensity of information loss generated by the distortion simulation module.

Images