Adaptive robust watermark embedding method and system based on deep neural network

Abstract

The present application relates to the cross field of deep learning and robust watermark, and relates to an adaptive robust watermark embedding method and system based on a deep neural network, which comprises the following steps: randomly selecting a carrier image and generating watermark sequence information; preprocessing the carrier image and the watermark sequence; encoding the preprocessed carrier image and watermark sequence information, and outputting a modification ratio map; performing Gaussian filtering on the carrier image, calculating the maximum value of the modification allowed when embedding the watermark sequence information, and outputting a threshold map; calculating a feature noise template mapped by the watermark information, and generating a watermark image; adding noise to the watermark image and performing image attack processing operations to generate an attacked watermark image; receiving the attacked watermark image, and outputting a predicted watermark information sequence. The present application can adaptively embed watermark information, has a large modification amount for complex texture regions of the carrier image and a small modification amount for smooth regions, has better visual quality, and has strong robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross field of deep learning and robust watermark, in particular to an adaptive robust watermark embedding method and system based on deep neural network. BACKGROUND

[0002] The popularity of the Internet has promoted the further development of digital multimedia, and images and videos have become one of the most important carriers for information transmission in the Internet era. However, while digital multimedia is booming, the problem of image and video copyright infringement is becoming more and more serious. It is urgent to protect the intellectual property rights and copyright of authors by copyright authentication of stolen images and video works.

[0003] Robust watermarking technology is a technology that embeds watermark information in a carrier image in a visually imperceptible manner. The watermarked image can still reliably extract the watermark message under noise interference, thereby achieving the function of copyright authentication. The two key indicators for measuring the performance of a robust watermarking system are robustness and visual imperceptibility. Robustness means that the watermark image can be completely extracted after being subjected to certain noise attacks and image processing operations. Visual imperceptibility means that the watermarked image and the original carrier image are difficult to distinguish.

[0004] In the past few years, with the great success of deep learning in the field of computer vision and pattern recognition, a series of deep learning-based digital watermarking methods have emerged. The deep learning-based method often trains the model in an end-to-end manner. The encoder network receives the carrier image and the watermark information and outputs the watermarked image. The decoder network receives the watermarked image after noise attack and tries to recover the watermark information. However, the existing deep learning-based robust watermarking method does not explicitly limit the amount of image modification during the generation of the watermarked image. The generated watermarked image occasionally has obvious noise, so the visual quality needs to be improved. On the other hand, the robustness of the existing method to other noise such as Moire fringe noise needs to be improved. SUMMARY

[0005] The present application provides an adaptive robust watermark embedding method and system based on deep neural network. Compared with the existing deep learning-based robust watermarking scheme, the present application can achieve high decoding accuracy under various common noise attacks while realizing adaptive watermark information embedding, improving the visual quality of the watermarked image and enhancing the imperceptibility of the watermarked image.

[0006] The method of the present application adopts the following technical solution: an adaptive robust watermark embedding method based on deep neural network, comprising the following steps:

[0007] S1, randomly select a carrier image I c and generate a random 0-1 watermark sequence information Msg as a training data set;

[0008] S2, pre-process the carrier image and the watermark sequence information;

[0009] S3, encode the pre-processed carrier image and the watermark sequence information, and output a modification ratio map M map ;

[0010] S4, Gaussian filter the carrier image, and calculate the maximum value allowed to modify when embedding the corresponding watermark sequence information, and output a threshold map T map according to the calculation result;

[0011] S5, calculate a feature noise template N pat mapped by the watermark sequence information, and superimpose the feature noise template on the carrier image I c to generate a watermark image I w ;

[0012] S6, perform corresponding image attack processing operations and add noise to the watermark image I w to generate an attacked watermark image I' w ;

[0013] S7, decode the attacked watermark image I' w and output the predicted watermark information sequence Msg p .

[0014] In a preferred embodiment, the pre-processing of step S2 includes the following steps:

[0015] S21, convert the watermark sequence information into a one-dimensional tensor through full connection, reshape it into a 3-dimensional image tensor, and then up-sample the image tensor so that the watermark sequence information becomes a 3-dimensional watermark tensor with the same dimension size as the carrier image;

[0016] S22, superimpose the watermark tensor obtained in step S21 and the carrier image according to the channel, as the input of the encoder.

[0017] In a preferred embodiment, the generation process of the threshold map T map in step S4 includes the following steps:

[0018] S41, Gaussian smoothing filter the carrier image I c to obtain a filtered carrier image I g ;

[0019] S42, use the Sobel operator in the horizontal direction on the Gaussian smoothing filtered carrier image I g to calculate the maximum value of the gradient in the horizontal direction Kh and K v first order difference in vertical direction, and sum as a threshold map,

[0020] In the preferred embodiment, the generation process of the watermark image in step S5 includes the following steps:

[0021] S51, matrix dot product of the modification ratio map and the threshold map to generate a feature noise template N pat ;

[0022] S52, add the feature noise template mapped from the watermark sequence information to the watermark image to generate a watermark image I w : I w = I c +N pat .

[0023] The system of the present application adopts the following technical scheme: an adaptive robust watermark embedding system based on a deep neural network, comprising:

[0024] a data set input module for randomly selecting a carrier image I c and generating a random 0-1 watermark sequence information Msg as a training data set;

[0025] a preprocessing module for preprocessing the input data set, converting the watermark sequence information into a tensor matrix of the same size as the carrier image;

[0026] an encoder module for generating a modification ratio map according to the carrier image and the watermark sequence information, the modification ratio map indicating the size of the pixel value modification required when embedding the watermark sequence information;

[0027] a threshold map generation module for calculating the texture complexity of the carrier image I c , and taking the texture complexity as a threshold map, the threshold map indicating the maximum modification amount that does not cause the human eye to perceive when embedding the watermark sequence information;

[0028] a watermark image generation module for integrating the threshold map and the modification ratio map, generating a feature noise template, and adding it to the carrier image I c to finally generate a watermark image;

[0029] a noise attack module for simulating image attack processing operations and adding noise to the watermark image to generate a corresponding attacked watermark image I′ w ;

[0030] a decoder module for decoding the attacked watermark image I′ w to output a predicted watermark information sequence Msg p .

[0031] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0032] 1、The present application adopts a self-adaptive embedding method, and the feature noise template obtained by mapping the watermark information has self-adaptability, and the intensity is high in the area with high texture complexity and is low in the smooth area of the image, so that the main modification amount of the carrier image is concentrated in the texture complex area, and the visual quality of the image is better, and the generated watermark image has no obvious noise points.

[0033] 2、The present application considers the influence of Moire fringe noise on decoding accuracy, introduces Moire fringe noise into the training of the decoder, so that the decoder has stronger robustness to Moire fringe noise, and has better robustness under the condition of pirated shooting images.

[0034] 3、The trained model of the present application well balances the robustness and visual quality, and has very high decoding accuracy while having good visual quality, and can resist various noises in the printing and shooting process.

[0035] 4、The present application has good time performance, and the robust watermark model training has low time complexity, short training time and easy convergence of the network. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is the model training flowchart of the embedding method in the embodiment of the present application;

[0037] Figure 2 is the overall framework diagram of the embedding system in the embodiment of the present application. DETAILED DESCRIPTION

[0038] In general, the present application adaptively maps the watermark information to be embedded into a feature noise template through a deep neural network, and then adds it to the carrier image; at the same time, a neural network decoder is trained, so that the decoder can recognize the original feature noise template and decode and restore the original watermark information even if the watermark image is attacked and disturbed by other noises. The feature noise template obtained by the present application has self-adaptability, and the noise intensity is high in the area with high texture complexity and is low in the smooth area, so that the visual quality of the watermark image generated by the present application is good. The present application will be described in further detail in combination with the embodiments and the drawings, but the embodiments of the present application are not limited thereto.

[0039] Embodiment 1

[0040] The embodiment is based on a deep neural network adaptive robust watermark embedding method, which mainly relates to the following technical features: 1) watermark adaptive embedding, mapping the watermark information sequence into a feature noise template, the intensity of the feature noise template is large for the texture complex area of the carrier image, and the intensity is small for the texture smooth area, so that the visual quality of the watermark image generated after adding to the carrier image is high. 2) Simulate the noise existing in the real world and introduce it into the training of the decoder to improve the robustness of the watermark model.

[0041] As shown in Figure 1 The adaptive robust watermark embedding method of the embodiment has the following specific steps:

[0042] S1, the reading of the carrier image and the generation of the random watermark sequence information are completed through the data set input module. First, 25000 images are randomly selected from the Mirflickr data set for normalization processing as the carrier image I c , Among them, H, W and C respectively represent the height, width and channel number of the carrier image, and in the embodiment, H=400, W=400 and C=3. At the same time, a 0-1 watermark sequence information Msg is randomly generated as training data set, Msg∈{0,1} L , and L represents the length of the watermark sequence information.

[0043] S2, the preprocessing module is used to complete the preprocessing operation of the carrier image and the watermark sequence information, and the preprocessing result is used as the input of the watermark encoder. The specific preprocessing steps include:

[0044] S21, the watermark sequence information is changed into a one-dimensional tensor through full connection, and is reshaped into a 3-dimensional image tensor. Then the image tensor is upsampled, so that the watermark sequence information is converted into a 3-dimensional watermark tensor with the same dimension (HWC) size as the carrier image.

[0045] S22, the watermark tensor obtained in step S21 is superimposed with the carrier image according to the channel, and used as the input of the encoder.

[0046] S3, the carrier image and the watermark sequence information after preprocessing are received by the encoder module for encoding, and the modification ratio map M map is output. The encoder module adopts the structure of Unet, and the last layer adopts the tanh function activation, so that the value of the modification ratio map output by the corresponding decoder is normalized to [-1, 1], and the value of the modification ratio map represents the proportion of the pixel value required to embed the corresponding watermark information.

[0047] S4, receiving the carrier image as input by the threshold map generation module, performing Gaussian filtering, and calculating the maximum value allowed to modify when embedding the corresponding watermark sequence information, outputting a threshold map T according to the calculation result map . The threshold map T map is used to indicate the maximum modification amount that does not cause the human eye to perceive when embedding the watermark sequence information, and its generation process is as follows:

[0048] S41, performing Gaussian smoothing filtering on the carrier image I c to obtain the filtered carrier image I g . The Gaussian smoothing filtering uses a Gaussian filter kernel, the kernel length is k=5, and the variance sigma=0.8.

[0049] S42, calculating the first-order difference in the horizontal direction (K g ) and the vertical direction (K h ) using the Sobel operator on the carrier image I v filtered by the Gaussian smoothing filter, and summing to obtain the threshold map T map ,

[0050] S5, completing the generation of the watermark image by the watermark image generation module, first calculating the feature noise template N pat mapped by the watermark sequence information, and then superimposing the feature noise template on the carrier image I c to generate the watermark image I w . The specific generation process of the watermark image is as follows:

[0051] S51, performing matrix point multiplication on the modification ratio map and the threshold map to generate the feature noise template: N pat = ε*M map *T map .

[0052] Wherein, ε is a control embedding watermark strength factor. It should be noted that in the early stage of training, ε is set to a larger value, and after training for a certain time, ε needs to be gradually reduced. In this embodiment, ε is initialized to 0.8, and after training for 120000 steps, it is gradually linearly decayed to 0.25.

[0053] S52, adding the feature noise template N pat mapped by the watermark sequence information to the carrier image I c to generate the watermark image: I w =I c +N pat .

[0054] S6, completing the noise attack on the watermark image I wThe corresponding image attack processing operation and adding noise are performed to generate the attacked watermark image I' w The noise attack process is specifically as follows:

[0055] S61, the watermark image I w The perspective transformation attack is performed, and in the training process, the coordinate offset allowed by the perspective transformation is limited in a certain range, and the limit formula is as follows:

[0056] |x-x' | < W*τ, |y-y' | < H*τ

[0057] Where (x, y) represents the original coordinates of the image, (x', y') represents the coordinates after the perspective transformation, H is the height of the image, W is the width of the image, and τ represents the maximum value of the allowed coordinate offset. In this embodiment, H = 400, W = 400, and τ = 0.1. This limit formula indicates that the maximum allowed coordinate offset of the perspective transformation is 40. The perspective transformation matrix is generated under this limit condition, and the perspective transformation operation is performed on the watermark image.

[0058] S62, the watermark image I w The Moire fringe noise attack is performed. According to the grating equation of the Moire fringe, the Moire fringe noise M of the same dimension (HWC) as the watermark image is generated, and then the Moire fringe noise is added to the watermark image in an α fusion manner, as shown in the following formula: I' w = (1-α)I w +αM, in this embodiment, α = 0.3.

[0059] S63, the watermark image I w The Jpeg compression attack is performed. The image is compressed by simulating the standard Jpeg compression process. First, the image is converted from RGB mode to YCbCr mode, then the image is divided into 8*8 small blocks, and discrete cosine transform (DCT) is performed, and then quantization operation is performed according to the following formula:

[0060]

[0061] Where Q i,j is the standard quantization table, S i,j is the DCT coefficient sub-block, and β is a constant. In this embodiment, β is randomly selected within the range [0.1, 1.1]. After quantization, inverse quantization, inverse discrete cosine transform is performed in turn, and the transform mode is RGB, and the Jpeg compression attack is completed.

[0062] S64, the watermark image I w The color transformation attack is performed. The brightness and contrast of the watermark image are offset, and the changes of color difference, brightness and contrast in the image transmission process are simulated. The offset formula is: bIw + c, in this embodiment, b is randomly selected in the range [0.6, 1.4], and c is randomly selected in the range [-0.2, 0.2].

[0063] S65, add Gaussian noise N(mu, sigma) to the watermark image 2 , in this embodiment, mu is set to 0, and sigma is randomly selected in the range [0, 0.18].

[0064] wherein all the noise has a small intensity at the beginning of training, and the intensity remains unchanged after the training step increases to a certain value. In this embodiment, the training is set to gradually increase the intensity of various noise from the 0th step to the 15000th step, and the intensity remains unchanged within a certain range after the 15000th step.

[0065] S7, receiving the attacked watermark image I' through the decoder w as input, decoding and outputting the predicted watermark information sequence Msg p , wherein the decoder is composed of 8 convolutional layers and 2 fully connected layers.

[0066] S8, calculate the information prediction error loss and the image reconstruction loss, and backward propagate to update the encoder and decoder network parameters, the specific process is as follows.

[0067] S81, calculate the information prediction error loss, the information prediction error loss adopts sigmoid cross entropy function, and its definition is as follows:

[0068]

[0069] wherein I i , j = Msg, I' i , j = sigmoid(Msg p ), N represents the size of the batch sample, and L represents the length of the embedded bit sequence.

[0070] S82, calculate the image visual quality loss, the visual quality loss includes mean square error loss L mse and image perception similarity difference L lpips , which are defined as follows:

[0071]

[0072] L lpips = calculate_Lpips(I w , I c )

[0073] Wherein, Lpips is a visual quality index learned by a neural network, N represents the size of a batch sample, H is the height of a carrier image, W is the width of the carrier image, I c is the carrier image, and I w is the watermark image.

[0074] S83, combine the information prediction error loss, mean square error loss and image perceptual similarity difference according to a certain weight as a total loss. The calculation formula of the total loss is as follows:

[0075] L=λ1L ce +λ2L mse +λ3L lpips

[0076] In this embodiment, λ1=3, λ2=0.5, and λ3=2.4.

[0077] S84, calculate the gradient according to the total loss L and propagate it backward. In this embodiment, the optimizer is an Adam optimizer, and the learning rate is set to 0.0001.

[0078] S9, loop the process of S1-S8 until the neural network converges, and in this embodiment, a total of 360000 steps are trained.

[0079] The visual quality performance and robustness test evaluation results of the robust watermark model are shown in Table 1. From the test evaluation results, it can be seen that the present application can balance the visual quality and robustness, and both high decoding accuracy and good visual quality are ensured.

[0080]

[0081] Table 1: Robust watermark performance of the present application

[0082] Embodiment 2

[0083] This embodiment and embodiment 1 are based on the same inventive concept, and are a self-adaptive robust watermark embedding system based on a deep neural network, which includes the following modules:

[0084] A data set input module is used to randomly select a carrier image I c and generate a random 0-1 watermark sequence information Msg as a training data set;

[0085] A preprocessing module is used to preprocess the input data set, and convert the watermark sequence information into a tensor matrix with the same size as the carrier image;

[0086] An encoder module is used to generate a modification ratio map according to the carrier image and the watermark sequence information, and the modification ratio map is used to indicate the size of the pixel value modification required when embedding the watermark sequence information;

[0087] a threshold map generation module for calculating the texture complexity of the carrier image I c and taking the texture complexity as a threshold map, which is used to indicate the maximum modification amount that does not cause the human eye to perceive when embedding the watermark sequence information;

[0088] a watermark image generation module for integrating the threshold map and the modification ratio map, generating a feature noise template, and adding the feature noise template to the carrier image I c to finally generate a watermark image;

[0089] a noise attack module for simulating image attack processing operations and adding noise to the watermark image to generate a corresponding attacked watermark image I' w ;

[0090] a decoder module for decoding the attacked watermark image I' w and outputting a predicted watermark information sequence Msg p ;

[0091] a loss function calculation module for calculating the reconstruction loss of the generated watermark image and the original carrier image and the prediction error loss of the watermark sequence information, and performing back propagation to update the network weights of the encoder and the decoder so that the neural network gradually converges.

[0092] In the threshold map generation module, the generation process of the threshold map T map is as follows: performing Gaussian smoothing filtering on the carrier image I c to obtain a filtered carrier image I g ; using a Sobel operator to perform first-order difference in the horizontal direction K h and the vertical direction K v on the carrier image I g that has been subjected to Gaussian smoothing filtering, and summing the results as the threshold map,

[0093] In the watermark image generation module, the modification ratio map and the threshold map are matrix-point multiplied to generate a feature noise template N pat ; and the feature noise template mapped from the watermark sequence information is added to the watermark image to generate a watermark image I w .

[0094] The modules of the embodiment are used to implement the steps of the robust watermark embedding method of embodiment 1, and the specific process will not be described again.

[0095] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. Adaptive robust watermark embedding method based on deep neural network, characterized by: The following steps are involved: S1. Randomly select carrier image I c And generate random 0-1 watermark sequence information Msg as training data set; S2, preprocessing the carrier image and watermark sequence information; S3, encode the pre-processed carrier image and watermark sequence information, and output the modified ratio map M map ; S4, perform Gaussian filtering on the carrier image, and calculate the maximum value allowed to be modified when embedding the corresponding watermark sequence information, and output the threshold map T according to the calculation result map ; S5. Calculate the characteristic noise template N mapped by the watermark sequence information pat , superimpose the characteristic noise template onto the carrier image I c Generate watermark image I w ; S6, watermark image I w Perform corresponding image attack processing operations and add noise to generate the attacked watermark image I′ w ; S7, the attacked watermark image I' w Decode and output the predicted watermark information sequence Msg p ; Step S6 includes the following steps: S61, perform perspective transformation attack on the watermark image; S62, perform a moire noise attack on the watermark image, and add the generated moire noise M to the carrier image in an α-fusion manner, I′ w =(1-α)I w +αM; S63, perform JPEG compression attack on the watermark image; S64, performing a color shift attack on the watermark image, shifting the brightness and contrast of the watermark image, and simulating the changes in color difference, brightness, and contrast during image transmission; S65. Add Gaussian noise to the image.

2. The adaptive robust watermark embedding method according to claim 1, characterized in that: The following steps are also included: S8. Calculate the information prediction error loss and image reconstruction loss, and back-propagate to update the network parameters of the encoder and decoder; S9, loop the process of S1-S8 until the neural network converges.

3. The adaptive robust watermark embedding method according to claim 1, characterized in that The pre-processing of step S2 includes the following steps: S21, converting the watermark sequence information into a one-dimensional tensor through full connection, and reshaping it into a three-dimensional image tensor, and then upsampling the image tensor so that the watermark sequence information becomes a three-dimensional watermark tensor of the same size as the carrier image; S22. Superimpose the watermark tensor obtained in step S21 and the carrier image according to the channel as the input of the encoder.

4. The adaptive robust watermark embedding method according to claim 3, characterized in that: The encoder adopts the Unet structure, and the last layer uses the tanh function activation.

5. The adaptive robust watermarking method according to claim 1, characterized in that: The threshold map T in step S4 map The generation process includes the following steps: S41, carrier image I c Perform Gaussian smoothing filtering to obtain the filtered carrier image I g ; S42, the carrier image I after Gaussian smoothing filtering g Use Sobel operator to calculate K in the horizontal direction h and K in the vertical direction v The first-order difference of and sum as the threshold map, 6. The adaptive robust watermarking method according to claim 1, characterized in that: The process of generating the watermark image in step S5 includes the following steps: S51, do matrix dot product of the modified scale map and the threshold map to generate a characteristic noise template N pat ; S52, adding the characteristic noise template mapped by the watermark sequence information to the watermark image to generate the watermark image I w :I w =I c +N pat .

7. Adaptive robust watermark embedding system based on deep neural network, characterized by: Includes the following modules: Dataset input module, used to randomly select carrier image I c And generate random 0-1 watermark sequence information Msg as training data set; The preprocessing module is used to preprocess the input data set and convert the watermark sequence information into a tensor matrix of the same size as the carrier image; An encoder module is used to generate a modification ratio map based on the carrier image and the watermark sequence information, wherein the modification ratio map is used to indicate the ratio of pixel values ​​that need to be modified when embedding the watermark sequence information; Threshold map generation module, used to calculate the carrier image I c The texture complexity is used as a threshold map, wherein the threshold map is used to indicate the maximum modification amount that is not perceived by human eyes when embedding watermark sequence information; The watermark image generation module is used to integrate the threshold map and the modified scale map to generate a characteristic noise template and add it to the carrier image I c Finally, the watermark image is generated; The noise attack module is used to simulate the image attack processing operation and add noise to the watermark image to generate the corresponding attacked watermark image I′ w ; Decoder module, used to decode the attacked watermark image I′ w Decode and output the predicted watermark information sequence Msg p ; The process of generating the attacked watermark image by the noise attack module includes: Perform perspective transformation attacks on watermarked images; Perform a moire noise attack on the watermark image, and add the generated moire noise M to the carrier image in an α-fusion manner, I′ w =(1-α)I w +αM; Conduct JPEG compression attacks on watermarked images; Perform color transformation attacks on watermarked images, shift the brightness and contrast of watermarked images, and simulate the changes in color difference, brightness, and contrast during image transmission; Add Gaussian noise to the image.

8. The adaptive robust watermark embedding system according to claim 7, characterized in that: Also includes: The loss function calculation module is used to calculate the reconstruction loss of the generated watermark image and the original carrier image, as well as the prediction error loss of the watermark sequence information, and backpropagate to update the network weights of the encoder and decoder so that the neural network gradually converges.

9. The adaptive robust watermark embedding system according to claim 7, characterized in that: In the threshold map generation module, the threshold map T map The generation process is as follows: for the carrier image I c Perform Gaussian smoothing filtering to obtain the filtered carrier image I g ; The carrier image I after Gaussian smoothing filtering g Use Sobel operator to calculate K in the horizontal direction h and K in the vertical direction v The first-order difference of and sum as the threshold map, In the watermark image generation module, the modified scale map and the threshold map are matrix dot product to generate the characteristic noise template N pat ; Add the characteristic noise template mapped by the watermark sequence information to the watermark image to generate the watermark image I w .

Images