Deep watermarking method for immune geometric distortion

Abstract

The invention discloses a deep watermarking method for immune geometric distortion, and the method comprises the steps: firstly constructing a watermark model in combination with a Swinin-Transform and a deformable convolutional network (DCN), and enabling the watermark model to comprise a watermark encoder and a watermark decoder; simulating styles of different types of non-geometric distortion by using a style migration network to form a distortion style combined noise layer; and applying the distortion style combined noise layer to the training of the constructed watermark model, enhancing the multiple robustness of the watermark model to various non-geometric distortions, and improving the generalization ability of the robustness. According to the method, the image copyright protection and image leakage prevention functions can be realized while the watermark embedding is ensured not to influence the image quality, so that the intellectual property and security protection of the image can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a deep watermarking method immune to geometric distortion. Background Art

[0002] Image watermarking technology can effectively prevent copyright infringement and leaks, and has always received widespread attention from academia and industry. Traditional watermarking schemes embed watermark information into images in a manually designed pattern, and can design watermark extraction methods accordingly based on the embedding method to restore the watermark information, thereby achieving subsequent copyright authentication and leak tracing. However, traditional watermarking schemes rely on specific manually designed patterns, and it is difficult to fully include the robustness to deal with various distortions that may be encountered during image transmission. Therefore, there is still a lot of room for improvement in performance. In recent years, with the development of deep learning technology, many deep learning-based image watermarking methods have been continuously proposed, and their performance is significantly better than that of previous traditional image watermarking methods. The existing deep watermarking scheme framework mainly includes a watermark encoder, a noise layer, and a watermark decoder. The watermark encoder needs to embed the watermark message into the image in a way that is invisible to the human eye, and try to ensure that the visual quality of the image is not reduced due to the embedding of the watermark; the noise layer usually appears between the watermark encoder and the decoder. It simulates the distortion that may occur in the image during channel transmission and applies noise to the watermark image, thereby adding interference to the training process to help the model improve its robustness to related distortions through training. By using various noise layers, the model can flexibly obtain various corresponding robustness; finally, the watermark decoder needs to process the received watermarked image that may have been interfered with and accurately extract the watermark message therein.

[0003] However, the current deep watermarking schemes still have some limitations. For watermarking schemes, the most important thing is to have strong robustness to various image distortions. Image distortion can generally be divided into two categories: 1) geometric distortion, such as image rotation and affine transformation; 2) non-geometric distortion, such as various image noise and filtering. For geometric distortion, this type of distortion will cause serious asynchronous problems, resulting in a serious decrease in the accuracy of watermark extraction. In recent years, convolutional neural networks (CNNs) have been the mainstream structure for building watermark encoders and decoders. However, the relatively fixed internal mechanism of convolutional neural networks lacks flexibility and cannot adapt well to the various spatial distortions caused by geometric distortion to images, resulting in a large number of existing deep watermarking schemes based on convolutional neural networks being unable to obtain good robustness to geometric distortion. For non-geometric distortion, existing deep watermarking schemes mainly rely on mixed noise layers to simultaneously achieve multiple robustness to multiple non-geometric distortions. The mixed noise layer contains multiple noise layers. During the training process, one of them will be randomly selected for each group of training samples as the noise layer corresponding to the group of training samples. However, the mixed noise layer provides limited knowledge for model training, which is limited to the characteristics of the limited types of distortions that it contains and is directly simulated. Therefore, the watermark encoder and decoder trained accordingly perform poorly under more variable distortion conditions in actual scenarios and cannot demonstrate robust generalization capabilities well.

[0004] In practical applications, robustness to geometric distortion and generalization ability to non-geometric distortion are both crucial, so existing deep watermarking schemes need to be further improved in both aspects. Summary of the invention

[0005] The purpose of the present invention is to provide a deep watermark method immune to geometric distortion, which can ensure that watermark embedding does not affect image quality while also achieving image copyright protection and image leakage prevention functions.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A deep watermarking method immune to geometric distortion, the method comprising:

[0008] Step 1: construct a watermark model by combining the image feature transformation processing model Swin-Transformer with a sliding window and the deformable convolutional network DCN, wherein the watermark model includes a watermark encoder and a watermark decoder;

[0009] Step 2: Use the style transfer network to simulate different types of non-geometric distortion styles to form a distortion style combination noise layer;

[0010] Step 3: Apply the distortion style combined noise layer to the training of the constructed watermark model to enhance the multiple robustness of the watermark model to various non-geometric distortions and improve the generalization ability of the robustness.

[0011] It can be seen from the technical solution provided by the present invention that the above method can ensure that watermark embedding does not affect image quality while also achieving image copyright protection and image leakage prevention functions, thereby helping to strengthen the protection of image intellectual property rights and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic flow chart of a deep watermarking method immune to geometric distortion provided by an embodiment of the present invention;

[0014] Figure 2 It is a schematic diagram of the model structure of the watermark encoder and the watermark decoder described in the embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] like Figure 1 The figure is a schematic flow chart of a deep watermarking method immune to geometric distortion provided by an embodiment of the present invention, the method comprising:

[0017] Step 1: construct a watermark model by combining the image feature transformation processing model Swin-Transformer with a sliding window and the deformable convolutional network DCN (Deformable Convolutional Network), wherein the watermark model includes a watermark encoder and a watermark decoder;

[0018] In this step, if Figure 2The figure shows a schematic diagram of the model structure of the watermark encoder and the watermark decoder according to an embodiment of the present invention. The watermark encoder takes the original carrier image and the watermark message as input. After obtaining the input, the watermark information is expanded by the linear layer and the deconvolution layer, and redundancy is introduced to improve the robustness of the watermark. The original carrier image is preprocessed by the convolution layer I (specifically, a convolutional neural network CNN layer combined with LayerNorm operation and PRelu operation). The processed input features are further processed by 4 squeeze-excitation blocks (i.e., SE blocks), and then cascaded and processed by another convolution layer I to obtain a shallow feature map I. m ; Then, four Swin-Transformer blocks are used to fully couple the carrier image features and watermark information features to obtain the deep global features I s ; Through multiple expansion blocks, I s It will be merged with the sub-graphs multiple times. m cascaded and processed by multiple different convolutional layers I to perform I s with I m To improve the visual quality of the image embedded with watermark information, the feature map after the above operation will be cascaded with the original carrier image for many times and processed by convolution layer I, and finally a convolutional neural network CNN layer will output the watermarked image;

[0019] The watermarked image propagates through the channel and finally reaches the watermark decoder, which extracts the watermark information. Specifically, first, the distorted watermarked image is preprocessed by four deformable convolution blocks (using a deformable convolution network DCN layer to replace the main CNN layer in the above SE block) and a deformable convolution layer (a deformable convolution network DCN layer combined with SwitchNorm operation and PRelu operation); then, four Swin-Transformer blocks extract watermark information features from the deeper dimensions of the watermarked image. For each feature map processed by the Swin-Transformer block, it will be cascaded with the features that have not been processed in the deep dimension after the sub-graph fusion operation, and processed by a convolution layer II (a CNN layer combined with SwitchNorm operation and PRelu operation) to fully fuse the deep and shallow features; then, a pooling layer condenses the watermark features extracted by the previous operation; finally, a linear layer outputs the extracted watermark information.

[0020] In the specific implementation, the image feature transformation processing model Swin-Transformer with a sliding window and the deformable convolutional network DCN are the main components of the watermark model structure of the present invention. Swin-Transformer has a flexible self-attention mechanism, which improves its ability to adaptively process image features, thereby flexibly adjusting the degree of attention to different positions of the image during the processing according to the actual situation of the watermarked image. Therefore, using Swin-Transformer to build a watermark model helps to flexibly pay attention to changes in the watermark embedding position when geometric distortion occurs, effectively track the area where the watermark is embedded and perform corresponding watermark extraction, so as to improve the accuracy of watermark extraction under geometric distortion.

[0021] At the same time, the grid sampling position of the deformable convolutional network DCN has a free additional offset, so adding DCN helps further help the watermark model improve its ability to adaptively cope with geometric distortion.

[0022] Step 2: Use the style transfer network to simulate different types of non-geometric distortion styles to form a distortion style combination noise layer;

[0023] In this step, a distortion style transfer network is trained to simulate various non-geometric image distortion styles, including an image encoder, an image decoder, and multiple distortion style layers. The specific process is as follows:

[0024] To construct the distorted style combination noise layer, we first need to complete the distorted style transfer network N S Training, distortion style transfer network N S Including image encoder E I , Image decoder D I And multiple distortion style layers T is the set of each distortion style layer; the image encoder E I Consists of 6 residual connection blocks ResBlock; Distortion style layer Both are composed of 2 ConvINRelu layers (i.e., a CNN combined with InstanceNorm and Relu operations); the image decoder D I It consists of 6 ResBlocks and a convolutional neural network (CNN) layer;

[0025] Distortion Style Transfer Network N S The training includes two branches, namely the image content branch and the distortion style branch. Assuming that there are n types of distortion styles that need to be simulated, first obtain the original image The corresponding image after the original image is interfered with by various distortions The image pair Where 1 represents the i-th distortion among all n distortions; Represents the set of all original images under the first distortion condition; Represents the set of corresponding images of all original images after the i-th distortion interference;

[0026] For the image content branch, the image encoder obtains the original image And extract its image features F I , then the image feature F I will be directly passed to the image decoder to obtain As the input image restored by the model; in the distortion style branch, for the i-th distortion, the corresponding distortion style layer simulates its style Will be inserted between the image encoder and the image decoder, the distortion style layer Get the image feature F output by the image encoder I And process it to obtain the image features after incorporating the corresponding distortion style features It will be handed over to the image decoder to obtain an image with the corresponding i-th distortion style

[0027] To train the distorted style transfer network Ns, the image content branch and the distorted style branch will be trained alternately, and different loss functions need to be designed for the two branches. For the image content branch, the only training loss is the original image Input image restored with the model The mean square error loss between For the distorted style branch, its loss function Including distorted images With distortion style image The mean square error loss between And two perceptual losses evaluated by the pre-trained classic neural network architecture VGG network, namely:

[0028]

[0029] and

[0030] Where V l and G(*) respectively represent the feature map and Gram matrix calculated by the lth layer of the publicly released pre-trained VGG network;

[0031] Loss Function and Constrain at the pixel level and feature level respectively With distortion style image The difference between the two, and the loss function It is used to constrain the changes in image content caused by style transfer;

[0032] Therefore, the loss function of the distorted style branch is The calculation expression is:

[0033]

[0034] Among them, α, β, and γ represent the loss function In the loss function The weights used in the calculation process; in the specific implementation, α, β, and γ are 1, 0.01, and 0.0001 respectively.

[0035] After completing the distortion style transfer network N S After training, different distortion style layers The different distortion styles simulated will be classified according to the type of non-geometric distortion they belong to, including noise, filtering and image compression. For the noise type, Gaussian noise, Poisson noise and salt and pepper noise are selected; for the filtering type, Gaussian filtering, median filtering and mean filtering are selected; for the image compression type, JPEG compression is selected.

[0036] Through the linear fusion method of style, the distortion styles belonging to the same category will be fused to obtain a style transfer layer representing the overall style of this type of distortion, including the noise style transfer layer Filter style transfer layer and image compression style transfer layer These three distortion style transfer layers will form the distortion style combination noise layer together with the image encoder and image decoder;

[0037] The input image first passes through the image encoder EI to extract image features; then passes through the noise style transfer layer randomly selected for each training sample batch Filter style transfer layer and image compression style transfer layer A distortion style is applied in the image decoder DI; finally, a watermarked image is obtained after interference by the distortion style combined noise layer.

[0038] Step 3: Apply the distortion style combined noise layer to the training of the constructed watermark model to enhance the multiple robustness of the watermark model to various non-geometric distortions and improve the generalization ability of the robustness.

[0039] In this step, the watermark encoder and watermark decoder are trained in an end-to-end training mode. The watermark encoder E obtains the original watermark message m and the carrier image I c , and output the watermarked image I w , containing watermark image I wIt will be processed by the distorted style combined noise layer to obtain the distorted watermarked image I no ; Distorted watermarked image I no It will be processed by the watermark decoder D and output the extracted watermark message mex;

[0040] Loss function of watermark encoder and watermark decoder during training It is expressed as:

[0041]

[0042] in is the original watermark message m and the extracted watermark message m ex The mean square error loss between is the carrier image I c With watermarked image I w The mean square error loss between is the carrier image I c With watermarked image I w Structural Similarity (SSIM) loss between them; Used to improve the accuracy of watermark extraction, and Used to improve the visual quality of watermarked images;

[0043] λ 1 ,λ 2 With λ 3 Respectively and In the loss function The weights in the calculation process; in actual use, the present invention sets λ 1 ,λ 2 With λ 3 The values ​​are 3, 1, 1.

[0044] In the specific implementation, by selecting geometric distortion as the noise layer to participate in the watermark model training, the watermark model can be immune to the interference caused by geometric distortion; by selecting the distortion style combination noise layer to participate in the watermark model training, the watermark model can obtain multiple robustness to non-geometric distortion, and such robustness can be generalized to various situations where non-geometric distortion occurs.

[0045] After the watermark model training is completed, the watermark encoder is used to embed watermark information in the image that needs to be protected by copyright. When the copyright ownership needs to be verified, the watermark decoder is used to extract the watermark information in the image to verify the copyright ownership.

[0046] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0047] The effectiveness of the method described in the embodiment of the present invention is verified by a specific example below. This example is comprehensively compared with existing deep watermarking schemes from two aspects: robustness and visual quality of watermarked images. The baseline methods include SSL (Watermarking images in self-supervised latent space, a deep learning image watermarking scheme in self-supervised latent space), MBRS (MBRS: Enhancing Robustness of DNN-based Watermarking by Mini-Batch of Real and Simulated JPEG Compression, a deep learning image watermarking scheme that improves robustness by alternating between real or simulated JPEG compression during model training) and CIN (Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms, a deep learning image watermarking scheme that combines reversible and irreversible mechanisms).

[0048] This example uses 5,000 images from COCO 2017, the most commonly used public image dataset, for watermark model training, and another 500 images from COCO 2017 are used for watermark model performance evaluation.

[0049] For the visual quality of the watermarked image, the indicators used are the peak signal-to-noise ratio (PSNR) and the structural similarity value (SSIM) used to evaluate the difference between the watermarked image and the original image. The higher their values, the better the visual quality of the watermarked image, as shown in Table 1:

[0050] Table 1 Comparison of visual quality of watermarked images in different schemes

[0051]

[0052] Judging from the average results calculated on 500 test images, the solution proposed in the present invention can achieve significantly the highest PSNR value and SSIM value under both geometric distortion conditions and non-geometric distortion conditions, fully demonstrating the obvious advantages of the present invention over previous solutions in terms of the visual quality of watermarked images.

[0053] For robustness, the evaluation indicator used is the bit accuracy rate BCR of watermark extraction under various distortion conditions. The higher the BCR, the stronger the robustness when distortion occurs, as shown in Table 2:

[0054] Table 2 Comparison of watermark extraction bit accuracy of different schemes under various distortion conditions

[0055]

[0056]

[0057] From the average results calculated on 500 test images, the scheme proposed by the present invention can maintain 100.00% extraction accuracy for the three geometric distortions of shearing, rotation, and affine transformation used in the test, which is at least 10% higher than the BCR of the previous scheme, which fully proves that the present invention has strong robustness against geometric distortion and can effectively be immune to geometric distortion. For the remaining wide range of non-geometric distortions in the test, the scheme of the present invention always maintains a watermark extraction accuracy of more than 95.00%, which is significantly higher than the accuracy that can be achieved by the previous deep watermarking scheme. These experimental results fully demonstrate the robustness of the present invention to various non-geometric distortions. This good robustness can be generalized to various non-geometric distortion conditions in experimental tests. The watermark model of the present invention is immune to geometric distortion, that is, under most geometric distortion conditions, the watermark model of the present invention can maintain 100% correct watermark information extraction.

[0058] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A deep watermarking method immune to geometric distortion, characterized in that: The method comprises: Step 1: construct a watermark model by combining the image feature transformation processing model Swin-Transformer with a sliding window and the deformable convolutional network DCN, wherein the watermark model includes a watermark encoder and a watermark decoder; Step 2: Use the style transfer network to simulate different types of non-geometric distortion styles to form a distortion style combination noise layer; Step 3: Apply the distortion style combined noise layer to the training of the constructed watermark model to enhance the multiple robustness of the watermark model to various non-geometric distortions and improve the generalization ability of the robustness.

2. The deep watermarking method immune to geometric distortion according to claim 1, characterized in that: In step 1, The watermark encoder takes the original carrier image and the watermark message as input. After obtaining the input, the watermark information is expanded by the linear layer and the deconvolution layer, and redundancy is introduced to improve the robustness of the watermark. The original carrier image is preprocessed by the convolution layer I. The features of the processed input are further processed by the four squeeze-excitation blocks, and then cascaded and processed by another convolution layer I to obtain the shallow feature map I m ; Then, four Swin-Transformer blocks are used to fully couple the carrier image features and watermark information features to obtain the deep global features I s ; Through multiple expansion blocks, I s It will be merged with the sub-graphs multiple times. m cascaded and processed by multiple different convolutional layers I to perform I s with I m After the above operation, the feature map will be cascaded with the original carrier image for many times and processed by the convolution layer I, and finally a convolutional neural network CNN layer will output the watermarked image; The watermarked image propagates through the channel and finally reaches the watermark decoder, which extracts the watermark information. Specifically, the distorted watermarked image is first preprocessed by four deformable convolution blocks and one deformable convolution layer. Then, four Swin-Transformer blocks extract the watermark information features from the deeper dimensions of the watermarked image. The feature map obtained after each Swin-Transformer block is cascaded with the features that have not been processed in the deep dimension after the sub-graph fusion operation, and processed by a convolution layer II to fully fuse the deep and shallow features. Afterwards, a pooling layer condenses the watermark features extracted by the previous operation; finally, a linear layer outputs the extracted watermark information.

3. The deep watermarking method immune to geometric distortion according to claim 1, characterized in that: In step 2, a distortion style transfer network is trained to simulate various non-geometric image distortion styles, including an image encoder, an image decoder, and multiple distortion style layers. The specific process is as follows: To construct the distorted style combination noise layer, we first need to complete the distorted style transfer network N S Training, distortion style transfer network N S Including image encoder E I , Image decoder D I And multiple distortion style layers T is the set of each distortion style layer; the image encoder E I Consists of 6 residual connection blocks ResBlock; Distortion style layer Both consist of 2 ConvINRelu layers; image decoder D I It consists of 6 ResBlocks and one CNN layer; Distortion Style Transfer Network N S The training includes two branches, namely the image content branch and the distortion style branch. Assuming that there are n types of distortion styles that need to be simulated, first obtain the original image The corresponding image after the original image is interfered with by various distortions The image pair Where i represents the i-th distortion case among all n distortions; Represents the set of all original images under the i-th distortion condition; Represents the set of corresponding images of all original images after the i-th distortion interference; For the image content branch, the image encoder obtains the original image And extract its image features F I , then the image feature F I will be directly passed to the image decoder to obtain As the input image of the model restoration; in the distortion style branch, for the i-th distortion, the corresponding distortion style layer simulates its style Will be inserted between the image encoder and the image decoder, the distortion style layer Get the image feature F output by the image encoder I And process it to obtain the image features after incorporating the corresponding distortion style features It will be handed over to the image decoder to obtain an image with the corresponding i-th distortion style To train the distorted style transfer network N S , the image content branch and the distorted style branch will be trained alternately, and different loss functions need to be designed for the two branches. For the image content branch, the only training loss is the original image Input image restored with the model The mean square error loss between For the distorted style branch, its loss function Including distorted images With distortion style image The mean square error loss between And two perceptual losses evaluated by the pre-trained classic neural network architecture VGG network, namely: and Where V l and G(*) respectively represent the feature map and Gram matrix calculated by the lth layer of the publicly released pre-trained VGG network; Loss Function and Constrain at the pixel level and feature level respectively With distortion style image The difference between the two, and the loss function It is used to constrain the changes in image content caused by style transfer; Therefore, the loss function of the distorted style branch is The calculation expression is: Among them, α, β, and γ represent the loss function In the loss function The weight in the calculation process; After completing the distortion style transfer network N S After training, different distortion style layers The different distortion styles simulated will be classified according to the type of non-geometric distortion they belong to, including noise, filtering and image compression. For the noise type, Gaussian noise, Poisson noise and salt and pepper noise are selected; for the filtering type, Gaussian filtering, median filtering and mean filtering are selected; for the image compression type, JPEG compression is selected. Through the linear fusion method of style, the distortion styles belonging to the same category will be fused to obtain a style transfer layer representing the overall style of this type of distortion, including the noise style transfer layer Filter style transfer layer and image compression style transfer layer These three distortion style transfer layers will form the distortion style combination noise layer together with the image encoder and image decoder; The input image first passes through the image encoder E I Extract image features; then pass through a noise style transfer layer randomly selected for each training sample batch Filter style transfer layer and image compression style transfer layer A distortion style is applied in; finally, it passes through the image decoder D I The watermarked image is obtained after being interfered by the distorted style combined noise layer.

4. The deep watermarking method immune to geometric distortion according to claim 1, characterized in that: In step 3, the watermark encoder and watermark decoder are trained using an end-to-end training method. The watermark encoder E obtains the original watermark message m and the carrier image I c , and output the watermarked image I w , containing watermark image I w It will be processed by the distorted style combined noise layer to obtain the distorted watermarked image I no ; Distorted watermarked image I no will be processed by the watermark decoder D and output the extracted watermark message m ex ; Loss function of watermark encoder and watermark decoder during training It is expressed as: in is the original watermark message n and the extracted watermark message m ex The mean square error loss between is the carrier image I c With watermarked image I w The mean square error loss between is the carrier image I c With watermarked image I w The structural similarity loss between them; Used to improve the accuracy of watermark extraction, and Used to improve the visual quality of watermarked images; λ1, λ2 and λ3 represent and In the loss function The weight in the calculation process; After the watermark model training is completed, the watermark encoder is used to embed watermark information in the image that needs to be protected by copyright. When the copyright ownership needs to be verified, the watermark decoder is used to extract the watermark information in the image to verify the copyright ownership.

Images