Single-stage training method of end-to-end digital watermark model, image watermark identification method and related equipment

Abstract

The embodiment of the invention provides a single-stage training method of an end-to-end digital watermark model, an image watermark identification method and related equipment. The training method comprises the following steps: positively inputting an image and watermark information to be embedded into a convolutional neural network and a first reversible neural network model; performing fusion processing; forward inputting into a second reversible neural network model, and adding noise in an actual scene based on second reversible neural network simulation to obtain a watermark image with noise; performing reverse input to the second reversible neural network model, performing reverse fusion processing, and performing reverse input to the first reversible neural network model and the convolutional neural network to obtain extracted watermark information and a recovered image; and training based on a first difference between the watermark information to be embedded and the extracted watermark information, a second difference between the first image and the watermark image with noise, and a third difference between the first image and the restored image to obtain application parameters of the trained end-to-end digital watermark model.

Description

Technical Field

[0001] The present application relates to the field of multimedia content security technology, and in particular to a single-stage training method for an end-to-end digital watermark model, an image watermark identification method, and related equipment. Background Art

[0002] Copyright protection and content authentication of digital media have become an increasingly prominent issue. With the widespread dissemination and easy-to-edit nature of digital images and videos, traditional security measures have been unable to meet the growing demand for copyright protection. Digital watermarking technology embeds imperceptible information markers in digital media to covertly identify media content, thereby providing a means of tracking and verification for copyright owners. However, the application of this technology in reality faces multiple challenges, especially in complex and changing noise environments. How to maintain the stability and reliability of watermarks has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the purpose of this application is to propose a single-stage training method and related equipment for an end-to-end digital watermark model.

[0004] Based on the above objectives, the present application provides a single-stage training method for an end-to-end digital watermark model, including:

[0005] Acquire a first image and watermark information to be embedded; the first image is an original image;

[0006] Forward inputting the first image and the watermark information to be embedded into a convolutional neural network to obtain multiple frequency domains of the first image, and determining a target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene; performing diffusion processing on the watermark information and preprocessing the first image to obtain a second image;

[0007] Forward inputting the watermark information to be embedded and the second image into the first reversible neural network model, embedding the watermark information to be embedded into the second image, and obtaining a third image;

[0008] Performing fusion processing on the third image to scale a partial area of ​​the third image according to an intensity factor to obtain a fourth image;

[0009] Forward inputting the fourth image into the second reversible neural network model, adding noise in an actual scene simulated by the second reversible neural network to the fourth image, so as to obtain a watermark image with noise;

[0010] Reversely inputting the noisy watermark image into the second reversible neural network model to perform non-differentiable noise removal processing;

[0011] Perform reverse fusion processing on the watermark image after non-differentiable noise removal;

[0012] Reversely inputting the watermark image after the reverse fusion process into the first reversible neural network model to extract the watermark information;

[0013] Reversely input the watermark image after watermark information extraction into the convolutional neural network, extract the watermark information of the target embedding frequency domain, and obtain the extracted watermark information and the restored image;

[0014] Training the convolutional neural network, the first reversible neural network model, and the second reversible neural network model based on a first difference between the watermark information to be embedded and the extracted watermark information, a second difference between the first image and the watermark image with noise, and a third difference between the first image and the restored image;

[0015] Return to the step of forward inputting the first image and the watermark information to be embedded into the convolutional neural network, perform iterative training until the first difference, the second difference and the third difference converge, and obtain application parameters of the trained end-to-end digital watermark model; wherein the application parameters include parameters of the trained convolutional neural network, parameters of the trained first reversible neural network model and parameters of the trained second reversible neural network model.

[0016] In some embodiments, the multiple frequency domains of the first image are obtained by performing wavelet transform on the first image; the multiple frequency domains include a first frequency domain, a second frequency domain, a third frequency domain and a fourth frequency domain, wherein the values ​​of the first frequency domain, the second frequency domain, the third frequency domain and the fourth frequency domain decrease in sequence;

[0017] The determining, according to the noise type in the actual scene, a target embedding frequency domain of the watermark information in the first image comprises: in response to determining that the noise type includes differentiable noise, determining the first frequency domain as a target frequency domain; or

[0018] In response to determining that the noise type includes non-differentiable noise, determining the second frequency domain and the fourth frequency domain as target frequency domains; or

[0019] In response to determining that the noise type includes differentiable noise and non-differentiable noise, the fourth frequency domain is determined to be a target frequency domain.

[0020] In some embodiments, the noise type in the actual scene includes differentiable noise and non-differentiable noise; adding the noise in the actual scene simulated based on the second reversible neural network to the fourth image to obtain a watermark image with noise includes:

[0021] Performing color space conversion, discrete cosine transform processing, quantization processing and encoding on the fourth image to obtain a fifth image with non-differentiable noise;

[0022] adding differentiable noise to the fifth image to obtain a watermark image with mixed noise;

[0023] The diffusing the watermark information and preprocessing the first image include:

[0024] Performing wavelet transform processing on the first image to decompose the first image into multiple frequency domains;

[0025] The watermark information after the diffusion process is embedded in the target embedding frequency domain in the multiple frequency domains.

[0026] In some embodiments, the color space conversion process includes: converting the fourth image from the RGB color space to the YUV color space, and the conversion formula includes: Y=0.299R+0.587G+0.114B; wherein Y is brightness; R is the value of the red channel, G is the value of the green channel, and B is the value of the blue channel; U is chromaticity, and V is chromaticity; U=-0.14713R-0.28886G+0.436B+128; V=0.615R-0.51499G-0.10001B+128.

[0027] In some embodiments, the discrete cosine transform process includes: dividing the fourth image after the spatial transformation process into a plurality of image blocks; performing a discrete cosine transform process on each image block to transform the data of the image block from the spatial domain to the frequency domain; the discrete cosine transform process is performed by formula , where F(u,v) is the discrete cosine transform coefficient in the frequency domain, f(x,y) is the pixel value in the spatial domain, and α(u) and α(v) are normalization coefficients.

[0028] In some embodiments, the quantization process is performed by the formula Conduct, where F q (u,v) is the quantized discrete cosine transform coefficient, and Q(u,v) is the quantization step size in the quantization table.

[0029] In some embodiments, the fusing process of the third image comprises:

[0030] Segmenting the third image in a first direction to obtain a first sub-image and a second sub-image; the first direction is perpendicular to the plane where the third image is located;

[0031] Performing scaling processing on a partial area of ​​the first sub-image according to the intensity factor to obtain a processed first sub-image;

[0032] The processed first sub-image is fused with the second sub-image.

[0033] In some embodiments, the step of acquiring multiple frequency domains of the first image and determining the target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene; performing diffusion processing on the watermark information and preprocessing the first image to obtain the second image;

[0034] Embedding the watermark information to be embedded into the second image to obtain the third image comprises: forward inputting the watermark information to be embedded and the second image into a first reversible neural network model, and outputting the third image; the first reversible neural network model comprises alternatingly arranged InvertibleConv1x1 layers and InvArch layers; the InvertibleConv1x1 layer realizes information exchange between channels through 1x1 convolution; the InvArch layer comprises a plurality of dense blocks for splitting, extracting and fusing feature maps;

[0035] The step of performing color space conversion, discrete cosine transform processing, quantization processing and encoding on the fourth image to obtain a fifth image with non-differentiable noise includes: forward inputting the fourth image into a second reversible neural network model and outputting a fifth image.

[0036] In some embodiments, the first difference comprises a bit error rate between the watermark information to be embedded and the extracted watermark information;

[0037] The second difference comprises a pixel mean square error between the first image and the noisy watermarked image;

[0038] The third difference comprises a pixel mean square error between the first image and the restored image;

[0039] The loss functions for training include: L T =λ I I I +λ W L W +λ R L R ; Among them, λ I , W and λ R are weight coefficients; L I is the image loss function; L W is the watermark information loss function; L R is the image reconstruction loss function.

[0040] The present application also provides an image watermark identification method, including:

[0041] Obtain an original image, target watermark information, and an image to be extracted watermark information; wherein the image to be extracted watermark information has the same image content as the original image:

[0042] Inputting the image to be watermarked into the end-to-end digital watermark model to obtain predicted watermark information;

[0043] It is determined whether the predicted watermark information is the same as the target watermark information.

[0044] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0045] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the methods described above.

[0046] An embodiment of the present application further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute any of the methods described above.

[0047] As can be seen from the above, the single-stage training method of the end-to-end digital watermark model provided by the present application obtains a first image and watermark information to be embedded; the first image is the original image; the first image and the watermark information to be embedded are forwardly input into the convolutional neural network to obtain multiple frequency domains of the first image, and the target embedding frequency domain of the watermark information in the first image is determined according to the noise type in the actual scene; the watermark information is diffused and the first image is preprocessed to obtain a second image; the watermark information to be embedded and the second image are forwardly input into the first reversible neural network model, and the watermark information to be embedded is embedded into the second image to obtain a third image; the third image is fused to scale a part of the third image according to the intensity factor to obtain a fourth image; the fourth image is forwardly input into the second reversible neural network model, and the noise in the actual scene simulated by the second reversible neural network is added to the fourth image to obtain a watermarked image with noise; the watermarked image with noise is reversely input into the second reversible neural network model to perform non-differentiable noise removal processing; The watermark image is subjected to reverse fusion processing; the watermark image subjected to reverse fusion processing is reversely input into the first reversible neural network model to extract watermark information; the watermark image subjected to watermark information extraction is reversely input into the convolutional neural network to extract watermark information in the target embedding frequency domain to obtain the extracted watermark information and the restored image; based on the first difference between the watermark information to be embedded and the extracted watermark information, the second difference between the first image and the watermark image with noise and the third difference between the first image and the restored image, the convolutional neural network, the first reversible neural network model and the second reversible neural network model are trained; the step of forward inputting the first image and the watermark information to be embedded into the convolutional neural network is returned to perform iterative training until the first difference, the second difference and the third difference converge to obtain the application parameters of the trained end-to-end digital watermark model; wherein the application parameters include the parameters of the trained convolutional neural network, the parameters of the trained first reversible neural network model and the parameters of the trained second reversible neural network model; the method can effectively resist the attack of various types of noise in real scenes and improve the robustness of the digital watermark. In addition, the most appropriate embedding domain can be selected according to different noise types, which can improve the imperceptibility of digital watermarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of a single-stage training method for an end-to-end digital watermark model according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a model architecture of a single-stage training method for an end-to-end digital watermark model according to an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a process of generating a watermark image when the end-to-end digital watermark model of an embodiment of the present application is used;

[0052] Figure 4 This is a schematic diagram of the comparison of the original image, the watermark image and the watermark image with mixed noise in the embodiment of the present application; wherein the first row is the original image, the second row is the image with the watermark added, and the third row is the noise image;

[0053] Figure 5 A schematic diagram of the process of an image watermark identification method according to an embodiment of the present application;

[0054] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0056] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0057] The two key factors to measure the performance of digital watermarking algorithms are robustness and imperceptibility. Robustness refers to the ability of the watermark information embedded in the image to survive various distortions. Imperceptibility refers to the fact that the image after the watermark is embedded is almost the same as the original image. Specifically, digital watermarks must be able to resist various intentional or unintentional signal processing attacks, including but not limited to compression, filtering, geometric transformation, and noise superposition. These factors may affect the detectability of the watermark and the integrity of the content. Therefore, researchers are committed to developing more advanced watermarking algorithms to improve their robustness under noise interference and ensure that the watermark information can be accurately extracted under extreme conditions.

[0058] In order to deal with the problem of various types of noise attacks, a common method is to use a noise layer to simulate various attack types and train it through the "encoder-noise layer-decoder" structure to make it resistant to various simulated noises. In addition, it is also possible to add additional decoders and use noise discriminators to decode according to different noise types.

[0059] In recent years, deep learning technology has achieved rapid development and its application in the field of digital watermarking has become increasingly widespread. In a related technology, a watermark image is obtained by embedding a watermark into a carrier image, and a watermark is extracted from the watermark image. Here, the watermark is a bit information sequence. The method is composed of a reversible module including two branches of a carrier image and a watermark, and a noise layer. The forward process embeds the watermark into the carrier image to synthesize the watermark image, and the watermark image is not much different from the original carrier image; the reverse process separates and extracts the watermark from a single watermark image. In addition, the method also includes related processing of the bit information sequence. In another related technology, a carrier image and a watermark to be embedded are obtained; diffusion processing is performed on the carrier image and the watermark to be embedded; the diffused carrier image and the watermark to be embedded are input into an invertible neural network (INN) to obtain a first image, which is an image after the watermark is embedded; watermark information is separated from the first image based on the first image and the diffused carrier image; a target image is obtained based on the watermark information and the diffused carrier image, which is an image after the watermark to be embedded is embedded into the carrier image. These methods have the problem of poor watermark performance when differentiable and non-differentiable noises are superimposed.

[0060] Based on this, the embodiment of the present application provides an end-to-end single-stage training digital watermarking method and related equipment, the carrier image is preprocessed by the image preprocessing module (IPM), the watermark information to be embedded is preprocessed by the watermark diffusion module (WDM), and then the preprocessed image and watermark information are fed into the watermark embedding module (WEM) to obtain the watermark image; then the watermark image is passed through the fusion module to monitor the overall characteristics of the image to ensure that the overall quality and characteristics of the image after the watermark is embedded remain stable, and then the watermark image is passed through the non-differentiable noise module (NNPM) and the differentiable noise pool to obtain a watermarked noise image. In this way, the embedding of the watermark and the simulation of the non-differentiable noise are realized by the reversible neural network (INN), so that the successful extraction of the watermark can be achieved when the differentiable and non-differentiable noise are superimposed. It can effectively resist the attack of various types of noise in real scenes and improve the robustness of the digital watermark. To a certain extent, it solves the problem of maintaining the stability and reliability of the watermark in a complex and changeable noise environment.

[0061] Figure 1 A flow chart of a single-stage training method for an exemplary end-to-end digital watermark model according to an embodiment of the present application is shown.

[0062] The single-stage training method of the end-to-end digital watermark model provided in the embodiment of the present application may include:

[0063] S110, obtaining a first image and watermark information to be embedded; the first image is an original image;

[0064] S120, forward inputting the first image and the watermark information to be embedded into a convolutional neural network to obtain multiple frequency domains of the first image, and determining a target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene; performing diffusion processing on the watermark information and preprocessing on the first image to obtain a second image;

[0065] S130, forward inputting the watermark information to be embedded and the second image into the first reversible neural network model, embedding the watermark information to be embedded into the second image, and obtaining a third image;

[0066] S140, performing fusion processing on the third image to scale a partial area of ​​the third image according to an intensity factor to obtain a fourth image;

[0067] S150, forward inputting the fourth image into the second reversible neural network model, adding noise in an actual scene simulated by the second reversible neural network to the fourth image, to obtain a watermark image with noise;

[0068] S210, reversely inputting the noisy watermark image into the second reversible neural network model to perform non-differentiable noise removal processing;

[0069] S220, performing reverse fusion processing on the watermark image after the non-differentiable noise removal processing;

[0070] S230, reversely inputting the watermark image after the reverse fusion process into the first reversible neural network model to extract watermark information;

[0071] S240, reversely inputting the watermark image after watermark information extraction into the convolutional neural network, extracting the watermark information of the target embedded in the frequency domain, and obtaining the extracted watermark information and the restored image;

[0072] S250, training the convolutional neural network, the first reversible neural network model, and the second reversible neural network model based on a first difference between the watermark information to be embedded and the extracted watermark information, a second difference between the first image and the watermark image with noise, and a third difference between the first image and the restored image;

[0073] S260, returning to the step of forward inputting the first image and the watermark information to be embedded into the convolutional neural network, performing iterative training until the first difference, the second difference and the third difference converge, and obtaining application parameters of the trained end-to-end digital watermark model; wherein the application parameters include parameters of the trained convolutional neural network, parameters of the trained first reversible neural network model and parameters of the trained second reversible neural network model.

[0074] In step S110, usually, the watermark information to be embedded can be a hidden watermark, that is, an imperceptible and invisible watermark. The watermark information to be embedded can be in the form of a digital sequence, a matrix form, or an image form, which can be determined according to the needs. Usually, the first image can have multiple frequency domains. In some embodiments, the multiple frequency domains can include a first frequency domain, a second frequency domain, a third frequency domain and a fourth frequency domain, wherein the values ​​of the first frequency domain, the second frequency domain, the third frequency domain and the fourth frequency domain decrease in sequence. Usually, the first frequency domain can be a high-frequency region (HH band), the second region can be a medium-frequency region (HL band), the third frequency domain can be a medium-low frequency region (LH band), and the fourth frequency domain can be a low-frequency region (LL band).

[0075] In step S120, generally, the types of noise in the actual scene may include differentiable noise, non-differentiable noise, and mixed noise containing both differentiable noise and non-differentiable noise. Differentiable noise may be, for example, salt and pepper noise, Gaussian noise, Poisson noise, and multiplicative noise. Non-differentiable noise may be, for example, JPEG compression.

[0076] In some embodiments, determining the target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene may include: in response to determining that the noise type includes differentiable noise, determining the first frequency domain as the target frequency domain. Through the sensitivity of the high-frequency region to small disturbances and the characteristics of differentiable noise, the influence of the frequency domain on the watermark can be effectively minimized through optimization technology, so that the watermark has good robustness.

[0077] In some embodiments, determining the target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene may include: in response to determining that the noise type includes non-differentiable noise, determining the second frequency domain and the fourth frequency domain as target frequency domains. By embedding the watermark into the medium and low frequency regions, it is possible to have a better tolerance for nonlinear distortions such as compression, so that the watermark can maintain a high degree of recognition and integrity even after undergoing common image processing operations. In this way, a more robust effect can be achieved.

[0078] In some embodiments, determining the target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene may include: in response to determining that the noise type includes differentiable noise and non-differentiable noise, determining the fourth frequency domain as the target frequency domain. In this way, the integrity and robustness of the watermark can be maintained to a certain extent.

[0079] In some embodiments, the diffusing the watermark information and preprocessing the first image include:

[0080] The first image is subjected to wavelet transform processing to decompose the first image into multiple frequency domains. That is, the multiple frequency domains of the first image are obtained by performing wavelet transform processing on the first image.

[0081] The watermark information after diffusion processing is embedded in the target embedding frequency domain in the multiple frequency domains. Figure 2 As shown, it should be understood that the multiple frequency domains of the first image are obtained, and the target embedding frequency domain of the watermark information in the first image is determined according to the noise type in the actual scene; the watermark information is diffused, and the watermark information is diffused. The first image is pre-processed to obtain the second image, which can be achieved by: forward inputting the first image and the watermark information to be embedded into the convolutional neural network.

[0082] like Figure 3 As shown, forward propagation is performed in the convolutional neural network: the first image is subjected to wavelet transform processing, down-sampling processing, linear processing, normalization enhancement of feature expression capability processing, and up-sampling processing; the watermark information is linearly transformed and reshaped, connected with the feature map obtained by the up-sampling processing and Haar transform, and the second image is output / obtained.

[0083] In practical applications, the convolutional neural network can be a watermark diffusion module (WDM) and an image preprocessing module (IPM). The module realizes the preprocessing and watermark diffusion of the original image. The original image is decomposed into different frequency domains using wavelet transform, and then downsampled through the downsampling convolution layer (DownConv), followed by the application of linear layer and batch normalization layer, and the ReLU activation function is used to enhance the feature expression ability. Subsequently, the image resolution is gradually restored through the upsampling convolution layer (UpConvt), and the message tensor is linearly transformed and reshaped, connected with the upsampled feature map and Haar transform to achieve effective diffusion of watermark information.

[0084] In step S130, it should be understood that embedding the watermark information to be embedded into the second image to obtain the third image can be achieved by: forward inputting the watermark information to be embedded and the second image into the first reversible neural network model.

[0085] It should be understood that, in this step, embedding the watermark information can be understood as a full-spread form, that is, encoding the watermark information in the entire image (eg, the second image).

[0086] In some embodiments, such as Figure 2As shown, the first reversible neural network model (such as the WEM watermark embedding module) may include an operation list of the InvertibleConv1x1 (reversible convolution block) layer and the InvArch layer, and the two layers are arranged alternately to support forward and backward propagation. In the forward operation, the first reversible neural network model (such as the reversible module) sequentially executes these layers to embed watermark information in the second image to obtain a third image. Among them, the InvertibleConv1x1 layer can realize information exchange between channels through 1x1 convolution, keeping the spatial size unchanged; and the InvArch layer uses three dense blocks (DenseBlock) to split and deeply process the feature map, each dense block is composed of multiple convolution layers to extract and fuse features. That is, the InvertibleConv1x1 layer realizes information exchange between channels through 1x1 convolution; the InvArch layer includes multiple dense blocks for splitting, extracting and fusing feature maps. This network hierarchical structure can achieve the concealment and robustness of the watermark to a certain extent, while achieving lossless recovery of the watermark information, providing an innovative reversible processing solution for digital watermarking technology.

[0087] In practical applications, for the WEM watermark embedding module, the input can be an image and a watermark message. Among them, the image (Image): is an image that has been preprocessed or has been embedded with some information, and its dimension is [b, 1, 128, 128] / [batch size, number of channels, height, width]. The message (Message): is the watermark information to be embedded, which can be a binary sequence, and its dimension is [b, 256] / [batch size, message length]. The output can be a processed image (Processed Image) or an extracted message (Extracted Message): If it is in embedding mode, the output is an image that has been embedded with a watermark, and its dimension is consistent with the input image. If it is in extraction mode, the output will be the watermark information extracted from the image, and its dimension is consistent with the input information. Among them, during training, forward propagation (forward pass) or backward propagation (backward pass) can be performed for inverse operations. The forward method of WEM determines whether to perform forward or backward operations according to the rev parameter. This model mainly consists of two types of layers: InvertibleConv1x1 and InvArch. They are applied alternately, and each operation can perform forward propagation and backpropagation to perform the inverse operation.

[0088] The processing flow may include: module initialization, creating a module list self.operations through configuration (opt). According to the number of block_num (reversible blocks: 16), loop adding InvertibleConv1x1 layers (if use_conv1x1 is true) and InvArch layers to self.operations. Forward propagation (forward method): If the rev parameter is false (indicating forward operation, i.e. embedding watermarks), each operation layer is applied to the input data x in the order of addition. If the rev parameter is true (indicating reverse operation, i.e. extracting watermarks), each operation layer is applied to the input data x in the reverse order of addition. Returns the data processed by all operation layers. InvertibleConv1x1 layer: Contains a reversible 1x1 convolution operation. The number of input and output channels is determined by split1_img+split2_repeat. (In the context of reversible blocks, the input feature map is usually split into two parts, split1_img represents the number of channels in the first part; split2_repeat represents the number of channels used to pass the additional information watermark.) The number of input and output neurons for each channel is equal to the height of the image multiplied by the width = H×W (since it is a 1x1 convolution, the spatial size remains unchanged). InvArch layer: contains three DenseBlocks for feature processing of different parts of the image. The input feature map is divided into two parts, and the length is determined by split_len1 and split_len2. (split_len1 and split_len2 are parameters used to define the size of the input feature map split. The feature map splitting here refers to dividing the input feature map (usually the output of a layer in the middle of the convolutional neural network) into two parts in the channel dimension so that they can be processed independently.) Each DenseBlock accepts a part of the feature map and processes it, and then combines it with the other part. The number of output neurons depends on the size of the feature map after segmentation and the number of channels. DenseBlock: Each DenseBlock is a densely connected block composed of multiple convolutional layers for extracting and fusing features. The number of input and output neurons depends on the configuration of the block and the size of the input feature map.

[0089] In some embodiments, in step S140, the performing fusion processing on the third image may include:

[0090] The third image is segmented in a first direction to obtain a first sub-image and a second sub-image; the first direction is perpendicular to the plane where the third image is located. Generally, the first direction can be understood as the thickness direction of the image, or the direction perpendicular to the bottom surface with the plane where the image is located as the bottom surface.

[0091] A partial area of ​​the first sub-image is scaled according to the intensity factor to obtain a processed first sub-image. The intensity factor S is a parameter used to control the intensity of the hidden information. It can determine the degree of influence of the hidden information on the output image. The selection of the intensity factor S is usually based on experiments and application requirements. Usually, the optimal S value can be determined by adjusting the value of S and observing its influence on the image quality and information extraction effect. In addition, the value of S can also be adjusted according to the characteristics of the image and the application scenario of the watermark (such as copyright protection, content authentication, etc.). It can avoid the S value being too large, affecting the naturalness of the image; it can also avoid the S value being too small, the hidden information is not significant enough, and it is difficult to be effectively extracted.

[0092] The processed first sub-image is fused with the second sub-image.

[0093] Through this fusion process, the visual quality of the obtained third image can be maintained after the watermark information is embedded in the second image. By segmenting the second image and selectively hiding the information, the hidden storage and subsequent extraction of information can be achieved without significantly affecting the image quality. In this way, the overall characteristics of the image can be monitored and the overall quality and characteristics of the image can be kept stable after the watermark is embedded.

[0094] In practice, in the forward propagation (not rev) of the fusion process, the input encoded_img is the encoded image. The self.split1_img parameter represents the split point, which divides the image into two parts, where the front part is part of the image and the back part is the hidden message. The message part is extracted from the encoded_img and stored in the variable msg. Use cover_down and self.strength_factor to hide the message. (cover_down is the cover image, and self.strength_factor is the strength factor of information hiding.) The output image out with the hidden message is returned. In the backward propagation (rev) of the fusion process: for information extraction. It copies the input encoded_img to form the output out. This operation does not change the image, but only copies the encoded_img once to extract the hidden message.

[0095] In step S150, in some embodiments, the noise type in the actual scene includes differentiable noise and non-differentiable noise. The step of obtaining a watermark image with noise in the fourth image based on the noise in the actual scene simulated by the second reversible neural network may include:

[0096] Performing color space conversion, discrete cosine transform processing, quantization processing and encoding on the fourth image to obtain a fifth image with non-differentiable noise;

[0097] Adding differentiable noise to the fifth image, to obtain a watermark image with mixed noise.

[0098] In some embodiments, the color space conversion process may include: converting the fourth image from the RGB color space to the YUV color space, and the conversion formula includes: Y=0.299R+0.587G+0.114B; wherein Y is brightness; R is the value of the red channel, G is the value of the green channel, and B is the value of the blue channel; U is chromaticity, and V is chromaticity; U=-0.14713R-0.28886G+0.436B+128; V=0.615R-0.51499G-0.10001B+128. In this way, the color space conversion of the fourth image can be achieved.

[0099] In some embodiments, the discrete cosine transform (DCT) processing may include: dividing the fourth image after the spatial transformation processing into a plurality of image blocks (for example, into 8x8 image blocks); performing a discrete cosine transform processing on each image block to convert the data of the image block from the spatial domain to the frequency domain; the discrete cosine transform processing is performed by formula , where F(u,v) is the discrete cosine transform coefficient in the frequency domain, f(x,y) is the pixel value in the spatial domain, and α(u) and α(v) are normalization coefficients. u and v are the frequency domain indexes in the DCT transform, and are the positions of the frequency domain coefficients after the transform. In the 8x8 image blocks obtained by segmenting the image, each element of each block is represented by a pair of indexes (u,v), where u and v vary from 0 to 7, respectively, indicating 8 possible horizontal and vertical frequency positions. x and y are the pixel position indexes in the spatial domain, which are used to indicate the positions of pixels in the original image. In the 8x8 image blocks obtained by segmenting the image, x and y also vary from 0 to 7, respectively, indicating the horizontal and vertical positions of 8 pixels.

[0100] In some embodiments, the quantization process can be performed by the formula Conduct, where F q (u,v) is the quantized discrete cosine transform DCT coefficient. The original discrete cosine transform DCT coefficient F(u,v) is converted into Fq(u,v) through the quantization process. Q(u,v) is the quantization step in the quantization table. After quantization, the discrete cosine transform DCT coefficient can be quantized so that the coefficient of the high-frequency part becomes zero or close to zero, thereby achieving a compression effect.

[0101] In this way, the subsequent inverse operation on the non-differentiable noise can be realized, which facilitates the extraction of the watermark in the image embedded with the non-differentiable noise.

[0102] In some of the embodiments, the color space conversion, discrete cosine transform, quantization and encoding of the fourth image to obtain the fifth image with non-differentiable noise can be achieved through a second reversible neural network model (e.g., a reversible block), and may include: forward inputting the fourth image into the second reversible neural network model, and outputting the fifth image.

[0103] In some possible embodiments, the structure of the second reversible neural network model (e.g., a reversible block) may include three sub-networks F, G, and H, which work together to achieve reversible transformation of data. In the forward propagation stage (i.e., after forward input), the input tensor x (e.g., the fourth image) can be divided into two parts, x1 and x2. Among them, sub-networks F and G can process the divided tensors, while sub-network H is used to generate a scaling factor s1 that controls the reversible operation. In sub-network H, the LeakyReLU activation function and the clamp parameter can ensure the stability of the scaling factor s1 and the control of the output. Finally, through a series of operations, including the conversion of the F and G networks and the scaling factors generated by the H network, the reversible transformation of the input tensor can be achieved.

[0104] In some embodiments, multiple differentiable noises may be added to the fifth image. For example, several differentiable noises may be added to the fifth image through a differentiable noise pool.

[0105] In practical applications, reversible blocks may include: 1. Weight initialization functions, including initialize_weights: Use Kaiming initialization method (He initialization) to initialize the weights of the convolutional and linear layers in the network, and initialize the weights and bias of the batch normalization layer. initialize_weights_xavier: Use Xavier initialization method to initialize the weights of the convolutional and linear layers in the network, as well as the batch normalization layer. 2. Layer construction functions, including make_layer: Create a serialized list of neural network layers based on a given block (block) and number of layers (n_layers). 3. Residual blocks, including ResidualBlock_noBN: Define a residual block without batch normalization (BN), containing two convolutional layers, and using ReLU activation function. 4. Optical flow transformation functions, including flow_warp: Perform optical flow transformation based on the input feature map (x) and optical flow (flow) using the selected interpolation mode (interp_mode) and padding mode (padding_mode). 5. Dense blocks and convolution blocks, including DenseBlock: defines a dense block containing multiple convolutional layers and LeakyReLU activation functions, using the DenseNet architecture. ConvBlock: defines a convolutional block containing multiple convolutional layers and LeakyReLU activation functions. 6. Subnet construction functions, including subnet: returns a constructor for creating DenseBlock or ConvBlock based on the provided network structure (net_structure). 7. Reversible network blocks, including InvBlockExp: defines a reversible network block containing three subnetworks (F, G, H) that implement reversible forward and backward propagation. This block can be used to build a reversible neural network architecture. The InvBlockExp class defines a reversible network block, which is a key component when building a reversible neural network. The reversible network block allows the forward propagation and backpropagation of the network to be performed with constant time complexity because they do not need to store or calculate gradient information.

[0106] In some possible embodiments, the reversible block can be an InvBlockExp class, and the class attributes include, channel_num: the number of channels of the input tensor. channel_split_num: the number of channels to be split in the reversible block. clamp: used to control the scaling factor of the activation function output in the reversible block. The initialization method (__init__) includes receiving channel_num and channel_split_num as inputs to determine how the tensor is split. subnet_constructor: a function used to create a subnetwork based on the provided network structure. The subnet function is used here, which can return a constructor for creating a DenseBlock or ConvBlock. F, G, H: three subnetworks, respectively used for different operations in the reversible block. F and G are used for state conversion, while H is used to generate the scaling factor s. Forward propagation (forward) includes: x: input tensor. rev: Boolean value indicating whether to perform backpropagation. The tensor x is split into x1 and x2 based on channel_split_num. If rev is False (forward propagation): y1 is calculated as x1 plus the result of passing the F network through x2. s is computed as the scaled and activated output of H(y1). y2 is computed as x2 times exp(s) plus the result of passing y1 through the G network. If rev is True (backward propagation): s is computed as the scaled and activated output of H(x1). y2 is computed as x2 minus the result of passing x1 through the G network, divided by exp(s). y1 is computed as x1 minus the result of passing y2 through the F network. Jacobian Calculates the sum of the Jacobian determinants of the reversible blocks, used to stabilize backpropagation during training. jac: Sum of Jacobian determinants. If rev is False, jac is the sum of s; if rev is True, it is the sum of -s.

[0107] The single-stage training method of the end-to-end digital watermark model includes extracting watermark information from a noisy watermark image, which may include inversely processing (i.e., reversely operating) each step of obtaining a noisy watermark image, including the above-mentioned steps S110 to S150, i.e., performing the process of steps S210 to S260 to extract the watermark information therein.

[0108] In some embodiments, the watermark information extraction process is performed on the watermark image with noise to extract the embedded watermark information (that is, extract the watermark information in the watermark image with noise), that is, the process of steps S210 to S260 is performed, which can be understood as:

[0109] The watermark image with mixed noise is subjected to a differentiable noise removal process to obtain a fifth image with non-differentiable noise. Generally, the denoising process may be the reverse process of the aforementioned adding differentiable noise.

[0110] The fifth image (i.e., the watermark image after the differentiable noise removal process) is reversely input into the second reversible neural network model to perform non-differentiable noise removal process; and the fourth image is output.

[0111] The fourth image (ie the watermark image after the non-differentiable noise removal process) is subjected to a reverse fusion process to obtain a third image. Generally, the reverse fusion process may be a reverse process of the aforementioned fusion process.

[0112] The third image (i.e., the watermark image after the reverse fusion process) is reversely input into the first reversible neural network model to extract the watermark information, and the watermark information embedded in the overall image (i.e., the second image) and the second image are output.

[0113] The second image (i.e., the watermark image after watermark information extraction) is inputted into the convolutional neural network in reverse order to extract the watermark information of the target embedded frequency domain, and the watermark information and the first image are outputted.

[0114] In some embodiments, in step S210, after the fifth image is reversely input into the second reversible neural network model, back propagation is performed in the second reversible neural network model (e.g., a reversible block), and the input tensor (i.e., the fifth image) is reversely operated using subnetworks F and G, and the scaling factor s previously calculated by the LeakyReLU activation function and the clamp parameter to restore the original data and obtain the fourth image. This process can not only maintain the integrity of the data, but also improve the computational efficiency of the network.

[0115] In some embodiments, the process of obtaining the fourth image from the fifth image may include: inverse quantization processing and inverse discrete cosine transform processing. The inverse quantization processing may be performed by the formula F(u, v)=F q (u, v) × Q (u, v); the inverse quantization process can restore the quantized discrete cosine transform DCT coefficients to the original discrete cosine transform DCT coefficients. The inverse discrete cosine transform can be obtained by the formula The inverse discrete cosine transform (IDCT) can convert the frequency domain signal transformed by discrete cosine transform (DCT) back to the spatial domain, thus restoring the image.

[0116] In some possible embodiments, the reversible block can also calculate the sum of the determinants of the Jacobian matrix, which provides a guarantee for the stability of back propagation during the training of the model (e.g., the second reversible neural network model). This feature provides significant advantages when building reversible neural networks, especially in application scenarios that require efficient computing and data recovery. Through the subnet constructor, different network structures can be flexibly adapted to achieve highly customized network design.

[0117] In some of the embodiments, in step S230, the third image (i.e., the watermark image after reverse fusion processing) is reversely input into the first reversible neural network model (e.g., a reversible module), and the watermark can be extracted by the reverse operation of the aforementioned forward input.

[0118] In some embodiments, in step S240, the second image (i.e., the watermark image after watermark information extraction) is reversely input into a convolutional neural network (e.g., a diffusion module), and the watermark information and the first image are output. Specifically, the original embedded watermark information can be gradually restored and extracted by using inverse wavelet transform and a series of upsampling and downsampling convolution layers, in conjunction with batch normalization and ReLU activation function. At this point, the complete watermark information is extracted.

[0119] In some embodiments, in step S250, the first difference may include a first similarity between a numerical value corresponding to the extracted watermark information and a numerical value corresponding to the watermark information to be embedded. Usually, the watermark information may be in a numerical form, such as a numerical value in the form of 0101... The first similarity may be a bit error rate. That is, the first difference includes a bit error rate between the watermark information to be embedded and the extracted watermark information.

[0120] In some embodiments, in step S250, the second difference may include a second similarity between the first image and the watermark image with noise. The second similarity may be a peak signal-to-noise ratio. The second similarity may also be a pixel mean square error. That is, the second difference includes a pixel mean square error between the first image and the watermark image with noise.

[0121] In some embodiments, in step S250, the third difference may include a third similarity between the first image and the restored image. The third similarity may be a pixel mean square error. The third difference may include a pixel mean square error between the first image and the restored image. In some embodiments, the loss function for training may include: L T =λ I I I +λ W L W +λ R LR ; Among them, λ I , W and λ R are weight coefficients; L I is the image loss function; L W is the watermark information loss function; L R is the image reconstruction loss function. Among them, L I =||I m -WI m || 2 , I m is the first image, WI m is a watermark image with noise. W =||W m -RW m || 2 , W m is the watermark information to be embedded, RW m is the extracted watermark information. R =||I m -RI m || 2 , R.I. m To restore the image.

[0122] In practical applications, the following specific steps may be included in the training phase:

[0123] Initialize the neural network phase: set network parameters, including optimizer (Adam), learning rate (0.1), and number of cycles (epoch: 50).

[0124] Data preparation: Prepare a training dataset, including original images and corresponding watermark information.

[0125] Training loop (i.e. iterative training): For each training cycle (i.e. each round of training), the neural network can perform the same training steps. Each round can include the following steps: Set the network to training mode, and the neural network performs gradient calculation and back propagation after forward propagation. Through the data loader, obtain image and watermark data in batches, and the number of images in each batch can be set to 16, 32 or 48, etc. In each round, the neural network is trained according to the loss function, that is, the loss function value is reduced, and the training is terminated when the loss function value can no longer be reduced.

[0126] In some embodiments, in step S260, the first difference, the second difference and the third difference converge, which can be understood as training until the value of the loss function cannot be reduced.

[0127] Application Examples

[0128] By adopting the method of the embodiment of the present application, watermark information (watermark information in the form of a digital sequence) is embedded in the original image, and noise containing only slight noise (Gaussian noise), only non-differentiable noise (JPEG compression), and a superposition of differentiable (salt and pepper noise and Gaussian noise) and non-differentiable noise (JPEG) is embedded, respectively, to obtain a watermark image with corresponding noise, and the watermark information is extracted.

[0129] The experimental results are shown in Table 1, Table 2, Table 3 and Figure 4 As shown. Among them, Figure 4 The first row is the original image, the second row is the watermarked image, and the third row is the noise image with superimposed noise. Table 1-3 shows the robustness and imperceptibility. Among them, robustness corresponds to ACC; imperceptibility corresponds to PSNR.

[0130] Table 1 Performance of only differentiable noise (Gaussian noise)

[0131]

[0132] Table 2 Performance of non-differentiable noise only (JPEG)

[0133]

[0134] Table 3 Superposition of differentiable noise and non-differentiable noise

[0135]

[0136] Result analysis:

[0137] According to Table 1, Table 2 and Table 3, it can be seen that when the watermark is attacked by three different types of noise, the method of the present application has good robustness and imperceptibility. In the three cases of only differentiable noise (Gaussian noise), only non-differentiable noise (JPEG compression), and the superposition of differentiable (salt and pepper noise and Gaussian noise) and non-differentiable noise (JPEG), the method of the present application can extract the watermark information well without affecting the carrier image. It can be seen from this series of experimental data that when only affected by differentiable noise, embedding the digital watermark into the high-frequency area (HH band) has the best performance. In the face of situations involving non-differentiable noise, such as JPEG compression, the experimental results show that embedding the watermark into the medium and low frequency areas can bring more robust results.

[0138] Depend on Figure 4It can be seen that the first row is the original image, the second row is the watermark image, and the third row is the noise image. By comparing the images, it can be seen that there is almost no difference between the watermark image in the second row and the original image in the first row to the naked eye. The noise image in the third row can still successfully extract the watermark even after suffering from severe noise interference. According to the signal-to-noise ratio, it can be concluded that the watermark using the method of the present application has good imperceptibility. According to the noise image in the third row and the ACC value in Table 3, it can be seen that after such a large degree of noise interference, the method of the present application can still successfully extract the watermark, indicating that the method of the present application has good robustness.

[0139] The method of the embodiment of the present application provides important guidance for image watermarking: when designing a digital watermarking algorithm, the most appropriate embedding strategy should be selected according to the expected noise environment and characteristics. For differentiable noise, the high-frequency area of ​​the image is used; while for non-differentiable noise, the strategy should shift to utilizing the robustness of the medium and low frequency areas. This flexible adaptability is crucial to ensuring the effectiveness of digital watermarking in a variety of practical applications.

[0140] The embodiment of the present application uses an end-to-end reversible digital watermark model. The end-to-end design means that the model directly generates an image embedded with a watermark from the input original image, omitting multiple separation steps in the traditional process and improving efficiency. At the same time, reversibility ensures the accurate extraction of the watermark and maintains the integrity of the watermark even under extreme conditions. This end-to-end reversible model not only simplifies the process of watermark embedding and extraction, but also improves the generalization ability of the model and its robustness to noise attacks by learning directly from the data. Through an end-to-end training method, the model can automatically adjust parameters to optimize performance and achieve more efficient and reliable digital watermark protection.

[0141] The method of the embodiment of the present application mainly includes the following four aspects: First, the original image and watermark information are taken as input through the image preprocessing module (IPM) and the watermark diffusion module (WDM). Through the IPM, we divide the image into 4 frequency domains and can select the appropriate embedding domain for embedding according to the actual noise type. Secondly, the watermark information and the preprocessed image and watermark information are passed through the watermark embedding module (WEM), and forward propagation and reverse propagation can be performed. In the forward propagation, the output is the watermark image, and the reverse propagation output is the restored image and the extracted watermark. Then the watermark image is passed through the fusion module (FM), which is used to monitor the overall characteristics of the image to ensure that the overall quality and characteristics of the image remain stable after the watermark is embedded. Finally, the watermark image is passed through the non-differentiable noise processing module (NNPM) to simulate non-differentiable noise, and then through the differentiable noise pool, finally obtaining a noisy image containing a watermark. The method of the embodiment of the present application can be used through NNPM to extract watermarks even for images containing non-differentiable noise, thereby improving the robustness of the watermark to non-differentiable noise and noise superposition.

[0142] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0143] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0144] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0145] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0146] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0147] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0148] Based on the same inventive concept, corresponding to the single-stage training method of the end-to-end digital watermark model of any of the above-mentioned embodiment methods, the present application also provides an image watermark identification method.

[0149] like Figure 5 As shown, the image watermarking method may include:

[0150] S310, acquiring an original image, target watermark information, and an image to be extracted watermark information; wherein the image to be extracted watermark information has the same image content as the original image;

[0151] S320, inputting the image to be watermarked into the end-to-end digital watermark model to obtain predicted watermark information; wherein the end-to-end digital watermark model is trained using the single-stage training method for the end-to-end digital watermark model as described in any of the preceding items;

[0152] S330: Determine whether the predicted watermark information is the same as the target watermark information.

[0153] In some embodiments, in step S310, the image to be extracted with watermark information may be a target image suspected to have target watermark information and having the same image content as the original image. The target image may be identified by naked eyes and determined to have the same image content as the original image. Alternatively, after the naked eyes identify the same, the pixel mean square error is calculated, and step S320 is performed only when the pixel mean square error is less than a threshold.

[0154] In some of the embodiments, S320 may be understood as reversely inputting the image from which the watermark information is to be extracted into the end-to-end digital watermark model to extract the watermark information and obtain predicted watermark information.

[0155] In some embodiments, in step S330, determining whether the predicted watermark information is the same as the target watermark information can be determined by comparing whether the bit error rate of the predicted watermark information and the target watermark information is less than a preset threshold. Specifically, when the bit error rate of the predicted watermark information and the target watermark information is less than or equal to the preset threshold, it can be determined that the predicted watermark information is the same as the target watermark information; thereby determining that the image to be extracted watermark information has the target watermark information. When the bit error rate of the predicted watermark information and the target watermark information is greater than the preset threshold, it can be determined that the predicted watermark information is different from the target watermark information; thereby determining that the image to be extracted watermark information does not have the target watermark information.

[0156] It should be understood that, generally, when the image from which the watermark information is to be extracted has the target watermark information, the watermark information in the image from which the watermark information is to be extracted is added by the end-to-end digital watermark model trained by the aforementioned single-stage training method of the end-to-end digital watermark model. That is, the end-to-end digital watermark model trained by the single-stage training method of the end-to-end digital watermark model in the aforementioned embodiment can be used alone to add watermark information when used. The specific process of adding watermark information can be as follows: Figure 3 As shown, it can also be used alone to decode a target image with watermark information. Of course, if the corresponding watermark information can be decoded, then the target image with watermark information itself is watermarked using an end-to-end digital watermark model trained using a single-stage training method of an end-to-end digital watermark model.

[0157] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the single-stage training method of the end-to-end digital watermark model or the image watermark identification method described in any of the above embodiments is implemented.

[0158] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0159] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0160] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0161] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0162] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0163] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0164] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0165] The electronic device of the above embodiment is used to implement the single-stage training method of the corresponding end-to-end digital watermark model or the image watermark identification method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0166] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the single-stage training method of the end-to-end digital watermark model or the image watermark identification method as described in any of the above embodiments.

[0167] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0168] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the single-stage training method of the end-to-end digital watermark model or the image watermark identification method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0169] Based on the same inventive concept, corresponding to the single-stage training method of the end-to-end digital watermark model or the image watermark identification method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the single-stage training method of the end-to-end digital watermark model or the image watermark identification method. Corresponding to the execution subject corresponding to each step in each embodiment of the single-stage training method of the end-to-end digital watermark model or the image watermark identification method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0170] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the single-stage training method of the end-to-end digital watermark model or the image watermark identification method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0171] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0172] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0173] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0174] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A single-stage training method for an end-to-end digital watermark model, characterized in that: include: Acquire a first image and watermark information to be embedded; The first image is an original image; Forward inputting the first image and the watermark information to be embedded into a convolutional neural network to obtain multiple frequency domains of the first image, and determining a target embedding frequency domain of the watermark information in the first image according to the noise type in the actual scene; Performing diffusion processing on the watermark information and preprocessing the first image to obtain a second image; Forward inputting the watermark information to be embedded and the second image into the first reversible neural network model, embedding the watermark information to be embedded into the second image, and obtaining a third image; Performing fusion processing on the third image to scale a partial area of ​​the third image according to an intensity factor to obtain a fourth image; Forward inputting the fourth image into the second reversible neural network model, adding noise in an actual scene simulated by the second reversible neural network to the fourth image, so as to obtain a watermark image with noise; Reversely inputting the noisy watermark image into the second reversible neural network model to perform non-differentiable noise removal processing; Perform reverse fusion processing on the watermark image after non-differentiable noise removal; Reversely inputting the watermark image after the reverse fusion process into the first reversible neural network model to extract the watermark information; Reversely input the watermark image after watermark information extraction into the convolutional neural network, extract the watermark information of the target embedding frequency domain, and obtain the extracted watermark information and the restored image; Training the convolutional neural network, the first reversible neural network model, and the second reversible neural network model based on a first difference between the watermark information to be embedded and the extracted watermark information, a second difference between the first image and the watermark image with noise, and a third difference between the first image and the restored image; Return to the step of forward inputting the first image and the watermark information to be embedded into the convolutional neural network, perform iterative training until the first difference, the second difference and the third difference converge, and obtain application parameters of the trained end-to-end digital watermark model; wherein the application parameters include parameters of the trained convolutional neural network, parameters of the trained first reversible neural network model and parameters of the trained second reversible neural network model.

2. The single-stage training method of the end-to-end digital watermark model according to claim 1, characterized in that: The multiple frequency domains of the first image are obtained by performing wavelet transform on the first image; the multiple frequency domains include a first frequency domain, a second frequency domain, a third frequency domain and a fourth frequency domain, wherein the values ​​of the first frequency domain, the second frequency domain, the third frequency domain and the fourth frequency domain decrease in sequence; The determining, according to the noise type in the actual scene, a target embedding frequency domain of the watermark information in the first image comprises: in response to determining that the noise type includes differentiable noise, determining the first frequency domain as a target frequency domain; or In response to determining that the noise type includes non-differentiable noise, determining the second frequency domain and the fourth frequency domain as target frequency domains; or In response to determining that the noise type includes differentiable noise and non-differentiable noise, the fourth frequency domain is determined to be a target frequency domain.

3. The single-stage training method of the end-to-end digital watermark model according to claim 1, characterized in that: The noise types in the actual scenario include differentiable noise and non-differentiable noise; Adding noise in an actual scene simulated by the second reversible neural network to the fourth image to obtain a watermark image with noise includes: Performing color space conversion, discrete cosine transform processing, quantization processing and encoding on the fourth image to obtain a fifth image with non-differentiable noise; adding differentiable noise to the fifth image to obtain a watermark image with mixed noise; The diffusing the watermark information and preprocessing the first image include: Performing wavelet transform processing on the first image to decompose the first image into multiple frequency domains; The watermark information after the diffusion process is embedded in the target embedding frequency domain in the multiple frequency domains.

4. The single-stage training method for an end-to-end digital watermark model according to claim 3, characterized in that: The color space conversion process includes: converting the fourth image from the RGB color space to the YUV color space, and the conversion formula includes: Y=0.299R+0.587G+0.114B; wherein Y is brightness; R is the value of the red channel, G is the value of the green channel, and B is the value of the blue channel; U is chromaticity, and V is chromaticity; U=-0.14713R-0.28886G+0.436B+128; V=0.615R-0.51499G-0.10001B+128 The discrete cosine transform process includes: dividing the fourth image after the spatial transformation process into a plurality of image blocks; performing discrete cosine transform process on each image block to transform the data of the image block from the spatial domain to the frequency domain; the discrete cosine transform process is performed by formula Where, F(u,v) is the discrete cosine transform coefficient in the frequency domain, f(x,y) is the pixel value in the spatial domain, α(u) and α(v) are normalization coefficients respectively; The quantization process is performed by Conduct, where F q (u,v) is the quantized discrete cosine transform coefficient, and Q(u,v) is the quantization step size in the quantization table.

5. The single-stage training method of the end-to-end digital watermark model according to claim 3, characterized in that: The performing fusion processing on the third image comprises: Segmenting the third image in a first direction to obtain a first sub-image and a second sub-image; the first direction is perpendicular to the plane where the third image is located; Performing scaling processing on a partial area of ​​the first sub-image according to the intensity factor to obtain a processed first sub-image; The processed first sub-image is fused with the second sub-image.

6. The single-stage training method for an end-to-end digital watermark model according to claim 3, characterized in that: The step of acquiring multiple frequency domains of the first image, and determining a target embedding frequency domain of the watermark information in the first image according to a noise type in an actual scene; Performing diffusion processing on the watermark information and preprocessing the first image to obtain a second image; Embedding the watermark information to be embedded into the second image to obtain the third image comprises: forward inputting the watermark information to be embedded and the second image into a first reversible neural network model, and outputting the third image; the first reversible neural network model comprises alternatingly arranged InvertibleConv1x1 layers and InvArch layers; the InvertibleConv1x1 layer realizes information exchange between channels through 1x1 convolution; the InvArch layer comprises a plurality of dense blocks for splitting, extracting and fusing feature maps; The step of performing color space conversion, discrete cosine transform processing, quantization processing and encoding on the fourth image to obtain a fifth image with non-differentiable noise includes: forward inputting the fourth image into a second reversible neural network model and outputting a fifth image.

7. The single-stage training method for an end-to-end digital watermark model according to claim 3, characterized in that: The first difference comprises a bit error rate between the watermark information to be embedded and the extracted watermark information; The second difference comprises a pixel mean square error between the first image and the noisy watermarked image; The third difference comprises a pixel mean square error between the first image and the restored image; The loss functions for training include: L T =λ I I I +λ W L W +λ R L R ; Among them, λ I , W and λ R are weight coefficients; L I is the image loss function; L W is the watermark information loss function; L R is the image reconstruction loss function.

8. A method for identifying an image watermark, characterized in that: include: Acquire an original image, target watermark information and an image to be extracted watermark information; wherein the image to be extracted watermark information has the same image content as the original image; Inputting the image to be extracted with watermark information into an end-to-end digital watermark model to obtain predicted watermark information; wherein the end-to-end digital watermark model is trained using the single-stage training method of the end-to-end digital watermark model according to any one of claims 1 to 7; It is determined whether the predicted watermark information is the same as the target watermark information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 or claim 7 when executing the program.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute any one of claims 1 to 6 or the method according to claim 7.

Images