Training method of image watermark processing network and watermark embedding method

Abstract

The invention relates to a training method of an image watermark processing network and a watermark embedding method, and the training method of the image watermark processing network comprises the steps: in a training stage, a disturbance layer between an encoder network and a decoder network is based on preset pixel transformation and / or preset coordinate transformation; adding disturbance to an image output by the encoder network to train the encoder network and the decoder network; and based on a preset residual information loss function, training the encoder network and the decoder network to obtain a trained image watermark processing network; the residual information loss function is constructed based on mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image. The robustness of image disturbance can be improved, and the risk of identity information leakage can be reduced at the same time.

Description

Technical Field

[0001] The present application relates to the field of image watermark processing, and in particular to a training method of an image watermark processing network and a watermark embedding method. Background Art

[0002] At present, with the development and promotion of generative artificial intelligence technology, people can use generative artificial intelligence technology to generate high-quality images with commercial value. Among them, in order to achieve copyright protection and traceability and detection of generated content, image watermarking technology is usually used to help verify image copyright. First, the encoder network is used to embed unique watermark information into the original image to generate a watermarked image. In the watermark verification stage, the watermarked image is input into the decoder network to extract the watermark information. Finally, the verification function is used to check whether the extracted watermark information matches the original watermark information. Image watermarking technology can also enable image creators to embed and verify corresponding secret watermarks for different distribution channels to track abnormal distribution behavior.

[0003] Current watermark embedding and verification, when dealing with adversarial attacks, often improves robustness against operations such as rotation and scaling by adding watermarks in the transform domain. However, this method often fails when facing adaptive or higher-level attacks, and cannot protect the identity information carried in the watermarked image from being leaked.

[0004] Currently, no effective solution has been proposed to address the problems of low robustness of image watermark processing and high risk of identity information leakage in related technologies. Summary of the invention

[0005] In this embodiment, a training method for an image watermark processing network and a watermark embedding method are provided to solve the problems of low robustness of image watermark processing and high risk of identity information leakage in related technologies.

[0006] In a first aspect, a training method for an image watermark processing network is provided in this embodiment, wherein the image watermark processing network includes an encoder network and a decoder network, and the method includes:

[0007] In the training phase, a perturbation layer between the encoder network and the decoder network adds perturbations to the image output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation to train the encoder network and the decoder network; and

[0008] Based on a preset residual information loss function, the encoder network and the decoder network are trained to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image.

[0009] In some embodiments, the method further comprises:

[0010] Perturbation modeling is performed based on the disturbance added in the perturbation layer to obtain a perturbation certification radius corresponding to the disturbance.

[0011] In some of the embodiments: where the perturbation comprises a perturbation added by a pixel transformation based on Gaussian noise, the perturbation modeling comprises modeling for Gaussian noise perturbation;

[0012] In the case where the perturbation comprises a perturbation added based on an affine transformation, the perturbation modeling comprises modeling for the affine transformation perturbation.

[0013] In some embodiments, the method further comprises:

[0014] Get the watermark image to be verified;

[0015] The multiple interferences obtained by sampling are added to the watermark image to be verified respectively, and based on sample voting, the watermark verification results corresponding to the multiple interferences are voted to verify the robustness of the image watermark processing network.

[0016] In some of the embodiments, the residual information loss function is used to maximize the mutual information between the watermark image and the watermark information, and minimize the mutual information between the watermark residual image and the watermark information.

[0017] In some of the embodiments, the residual information loss function is constructed by approximate representation of the mutual information between the watermark image and the watermark information and the mutual information between the watermark residual image and the watermark information; the approximate representation is calculated based on a neural estimator.

[0018] In a second aspect, a watermark embedding method is provided in this embodiment, including:

[0019] Get the image and watermark information to be embedded;

[0020] Based on the training method of the image watermark processing network described in the first aspect above, the trained encoder network performs watermark embedding processing on the image to be embedded and the watermark information to obtain a watermark embedded image.

[0021] In a third aspect, a watermark verification method is provided in this embodiment, including:

[0022] Obtain the watermark image to be extracted;

[0023] Based on the training method of the image watermark processing network described in the first aspect above, the decoder network obtained by training performs watermark extraction processing on the watermark image to be extracted to obtain a restored image and extract watermark information;

[0024] The extracted watermark information is compared with the original watermark information corresponding to the restored image to obtain a watermark verification result.

[0025] In a fourth aspect, in this embodiment, a training device for an image watermark processing network is provided, including: the image watermark processing network includes an encoder network and a decoder network, and the training device includes: a first training module and a second training module; wherein:

[0026] The first training module is used to add perturbations to the images output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation at a perturbation layer between the encoder network and the decoder network during a training phase, so as to train the encoder network and the decoder network;

[0027] The second training module is used to train the encoder network and the decoder network based on a preset residual information loss function to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image.

[0028] In a fifth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the training method of the image watermark processing network described in the first aspect, or the watermark embedding method described in the second aspect, or the watermark verification method described in the third aspect is implemented.

[0029] Compared with the related art, the present embodiment provides a training method and a watermark embedding method for an image watermark processing network. The training method for the image watermark processing network includes, in the training stage, adding perturbations to the image output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation at the perturbation layer between the encoder network and the decoder network to train the encoder network and the decoder network; and, based on a preset residual information loss function, training the encoder network and the decoder network to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is the residual image of the watermark image and the corresponding original image. It can improve the robustness to image perturbations and reduce the risk of identity information leakage at the same time.

[0030] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 is a hardware structure block diagram of a terminal of the training method of the image watermark processing network of this embodiment;

[0033] Figure 2 is a flow chart of the training method of the image watermark processing network of this embodiment;

[0034] Figure 3 It is a schematic diagram of image comparison;

[0035] Figure 4 is a training diagram of identity information leakage protection in this embodiment;

[0036] Figure 5 It is a schematic diagram of the watermark processing framework of the existing related technology;

[0037] Figure 6 is a flow chart of the watermark embedding method of this embodiment;

[0038] Figure 7 is a flow chart of the watermark verification method of this embodiment;

[0039] Figure 8 is a flowchart of a method for training an image watermark processing network in some of the embodiments;

[0040] Fig. 9 is a schematic diagram of the composition of an image watermark processing network of this embodiment;

[0041] Fig.10 It is a structural block diagram of the training device of the image watermark processing network of this embodiment. DETAILED DESCRIPTION

[0042] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the training method of the image watermark processing network of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0045] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the training method of the image watermark processing network in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0047] In this embodiment, a training method for an image watermark processing network is provided. Figure 2 is a flow chart of the training method of the image watermark processing network of this embodiment. Figure 2 As shown, the process includes the following steps:

[0048] Step S210, during the training phase, in the perturbation layer between the encoder network and the decoder network, perturbations are added to the image output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation to train the encoder network and the decoder network.

[0049] In this step, the image watermark processing network is trained based on the random smoothing strategy. Among them, random smoothing, as a widely adopted defense strategy, can provide a provable robustness guarantee for the classifier to protect the classifier from adversarial attacks. The characteristics of this random smoothing are that it is applicable to various classifier architectures and can maintain high accuracy in large models and large-scale data sets. In this embodiment, considering the applicability of random smoothing to the watermark decoder network, the network training will be based on the principle of random smoothing. The basic principle of random smoothing is to add noise in the training and inference stages of the classifier, so as to more effectively smooth the decision boundary and reduce the steep transition area used by adversarial samples.

[0050] In this step, disturbance is added to the perturbation layer between the encoder network and the decoder network, and the encoder network and the decoder network are trained so that the trained image watermark processing network can have the ability to resist image perturbations.

[0051] Here, we first explain the mathematical modeling involved in watermarking. Let X represent the set of all possible original images (i.e., the set of images without watermarks), T represent the set of all possible watermark information, and W represent the set of all watermark images. For each original image x∈X and watermark information t∈T, there is a corresponding watermark image w∈W. Let V=v=<i,j> Represents the coordinate space of the image, where i and j represent the x-axis and y-axis coordinates of the pixels in the image respectively. A coordinate watermark image can be defined as The pixel used to represent the watermark image w is placed at the coordinate v. Introduce a decoder model D: Then for any watermark image w, the decoder network extracts the estimated watermark information and expresses the estimated watermark information as: In the watermark verification phase, the verification function Verify:T×T×R→{0,1} can be used, where R represents a real number set. D ,t,τ) is used to evaluate the decoded watermark information t D The matching degree between the original watermark information t. If t D When the corresponding relationship between t and t is greater than or equal to the preset threshold τ, the value of the verification function is 1, indicating that the verification is successful, otherwise it is 0, indicating that the verification fails.

[0052] The watermark verification model can also be defined as The watermark verification model combines the two processes of watermark decoding and watermark information verification. First, the decoder network D is applied to the watermark image w to extract the decoded information. After that, the decoded watermark information is combined with the original watermark information t and the predefined tolerance threshold τ for evaluation through the verification function Verify. Therefore, the watermark verification model h evaluates the consistency between the decoded watermark information and the original watermark information t taking into account the threshold τ, which can be specifically expressed as:

[0053]

[0054] If the output value of formula (1) is 1, it means that the decoded watermark information successfully matches the original watermark information, and if the output value is 0, it means that the decoded watermark information does not match the original watermark information.

[0055] The pixel transformation of an image may be to modify only the pixel values ​​of the image, such as applying Gaussian noise to the image (Gaussian transformation). The coordinate transformation of an image requires the coordinate transformation of the pixels in the image, such as moving the pixel at coordinate a in the image to coordinate b. Figure 3 is a schematic diagram of image comparison. Figure 3 As shown, the upper left image is the original image without disturbance, and the lower left image is the coordinate vector field corresponding to the original image; the upper middle image is the image with Gaussian noise applied (pixel transformation), and the lower middle image is the coordinate vector field corresponding to the image with Gaussian noise applied; the upper right image is the image with affine transformation (coordinate transformation), and the lower right image is the coordinate vector field corresponding to the image with affine transformation.

[0056] In this step, watermark verification is regarded as a binary classification model, and the authentication robustness framework of image classification is applied. The perturbation of the watermark image can be converted to pixel space and coordinate space to ensure the authentication robustness of various operations in each space. Since the perturbation operation on the watermark image will affect the pixel space or coordinate space, the perturbation of the watermark image is essentially adjusting the pixel transformation or coordinate transformation of the watermark image. By utilizing pixel transformation or coordinate transformation, a new transformation smoothed watermark verification model can be constructed from the basic watermark verification model in the related art. When the input of the new transformation smoothed watermark verification model (i.e. ) is disturbed by pixel transformation or coordinate transformation, the new transformation smoothing watermark verification model predicts the category with the highest probability of returning (the category includes output value 1 and output value 0, representing watermark verification success and failure, respectively).

[0057] During the training process, we can introduce perturbations only in the perturbation layer between the encoder network and the decoder network based on robust watermark training. After receiving the watermark image w, the encoder network processes and outputs it, and obtains the corresponding pixel value and pixel coordinate v. Then, by applying it to the pixel change The watermark image w is perturbed by the transformation of θ(v,r) or the transformation of the coordinate transformation θ(v,r). Finally, the new image with the perturbation is input into the decoder network to train and update the encoder network and the decoder network.

[0058] Step S220, based on a preset residual information loss function, the encoder network and the decoder network are trained to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is the residual image of the watermark image and the corresponding original image. There is no order of execution between step S210 and step S220.

[0059] Specifically, in the process of embedding watermark information, the original watermark information t is processed by the encoder network to achieve watermark embedding, thereby outputting the watermark image w. Ideally, based on the information bottleneck (IB) principle, the watermark image w should retain all the information from the watermark information t and the clean original image x. Therefore, in this step, it is necessary to ensure that the watermark image w retains all the information of the watermark information t, that is, the sufficiency of w for t, which involves the task of maximizing the mutual information between w and t; at the same time, it is necessary to ensure that the watermark residual image z does not contain any information about t, which involves the task of minimizing the mutual information between z and t. By combining these two tasks, the residual information target can be set as:

[0060] maxI(w;t)-I(z;t) (2)

[0061] Here is a brief description of mutual information. For variables A and B, the mutual information I(A; B) between the two is defined as:

[0062] I(A;B)=∫p(A,B)log p(A,B)p(A)p(B)dAdB (3)

[0063] Since research shows that mutual information cannot be directly calculated in high-dimensional space, a neural estimator can be used to approximate the calculation in this step. In order to process the mutual information estimation in equation (3), the following theorem can be introduced:

[0064]

[0065] in, Both represent probability distribution, f KL represents the KL divergence.

[0066] Based on this, the residual information loss function can be defined as:

[0067]

[0068] in, represents the expected probability statistics, E represents the parameters of the encoder network, and The probability of extracting identity information (via watermark information t) from w and z respectively. and Both can be parameterized by the decoder network D, and the expected value is approximated by a batch of samples. Due to the range and asymmetry of the KL divergence, the range of the estimated value of the mutual information is [0, +∞]. By optimizing the estimated value of the mutual information, the parameters of the encoder network are updated. Based on the trained encoder network, the identity information in w will be retained to the greatest extent, while the identity information in z will approach 0.

[0069] Figure 4 FIG. 1 is a schematic diagram of the training for identity information leakage protection in this embodiment. Figure 4 As shown in the figure, after the watermark information t and the original image x are processed by the encoder network E, the watermark image w will be output. d After adding perturbations to the watermark image w, the new image with perturbations is input into the decoder network D. The decoder network D extracts the watermark information p(t|w) from the watermark image w, and extracts the watermark information p(t|z) from the watermark residual image z. The residual information loss function (L RIL ), and the encoder network and the decoder network are trained according to the value of the residual information loss function. The trained image watermark processing network can finally have the identity information leakage protection function.

[0070] Figure 5 FIG. 1 is a schematic diagram of a watermark processing framework of existing related technologies. Figure 5 As shown in the figure, in the existing related technologies, the original image and watermark information are processed by the encoder network to obtain the watermark image, and the watermark image is processed by the decoder network to output the estimated watermark information. In this process, there is a threat of identity leakage, and the robustness to image perturbations cannot be guaranteed. Therefore, there is a hidden danger of exposing personal or institutional information, which weakens the effectiveness of watermarks as a copyright protection tool.

[0071] In this regard, this embodiment, based on the above steps, enhances the robustness of the image watermarking technology in a random smoothing manner, and achieves effective protection against two types of interference: pixel transformation u and coordinate transformation r. Based on the strategy of reducing the loss of residual information, with the goal of reducing the mutual information between the watermark residual image and the watermark information, a residual information loss function is constructed to reduce the probability of identity information leakage.

[0072] Through the above steps S210 to S220, in the training stage, the perturbation layer between the encoder network and the decoder network adds perturbations to the image output by the encoder network based on the preset pixel transformation and / or the preset coordinate transformation to train the encoder network and the decoder network; and, based on the preset residual information loss function, the encoder network and the decoder network are trained to obtain the trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is the residual image of the watermark image and the corresponding original image. It can improve the robustness to image perturbations and reduce the risk of identity information leakage at the same time.

[0073] In one embodiment, the training method of the image watermark processing network may further include:

[0074] Perturbation modeling is performed based on the perturbation added in the perturbation layer, and a perturbation certification radius corresponding to the perturbation is obtained.

[0075] The perturbations of pixel transformation and coordinate transformation can be modeled separately. The perturbations of the image can include pixel-level And the modification of the coordinate level (θ). Given the pixels of image w, the perturbation transformation can be expressed mathematically as:

[0076]

[0077] Where w is the watermark image without disturbance, and w′ is the watermark image with disturbance.

[0078] According to equation (6), if the coordinates of the pixels in the image are not changed, the change of the perturbed image is only related to the pixel value. If the pixel coordinates are transformed, a new pixel position (v′) will be generated, which may not be accurately aligned with the original image coordinates.

[0079] The interpolation function f can be introduced I (w,v′) is used to represent the pixel value estimation of the target position based on the neighboring pixel values. The coordinate transformation expression can be:

[0080]

[0081] For Gaussian noise, Gaussian noise is added to the image w pixels as a disturbance, which can be modeled as:

[0082]

[0083] Among them, α~N(0,σ 2 I) represents a Gaussian distribution with a mean of 0 and a variance of σ. Formula (8) represents the generation and injection of noise for each pixel. The noise is N(0,σ 2 I) Consistency.

[0084] For affine transformation, the image coordinate v can be relocated to v′ by combining linear transformations such as scaling, rotation, shearing, and translation, which can be mathematically expressed as θ F (v,β), so it can be modeled as:

[0085] θ F (v,β)=(β1i+β2j+β3,β4i+β5j+β6),β~N(0,σ2I) (9)

[0086] Among them, N(0,σ2I) means that each component in β (which can be understood as different parameters of the transformation) is independently distributed according to a Gaussian distribution with mean 0 and variance σ.

[0087] In addition, corresponding smooth watermark verification can be performed for pixel transformation and coordinate transformation respectively. To perform pixel transformation with parameter U on the pixel, where U is the parameter set of pixel transformation, let is any basic watermark authentication model. Then take random variables ε~P from U ∈ , define the pixel transformation smoothing watermark verification model for:

[0088]

[0089] In formula (10), P represents the probability of verification success or failure. Formula (10) shows that in the case of disturbance The most likely output category. y represents the true result of the verification, which is equivalent to the true label of the binary classification model.

[0090] For coordinate transformation, let θ:V×M→V be the coordinate transformation, where M is the parameter set of coordinate transformation, fI:W×V→W is the interpolation function, is any basic watermark verification model. Take random variables ρ~P from M ρ , the coordinate transformation can be used to smooth the watermark verification model Build as:

[0091]

[0092] In particular, in this embodiment, pixel transformation and coordinate transformation can be proposed based on more common noise types. For example, pixel transformation is proposed based on Gaussian distribution and uniform distribution. and coordinate transformation θ.

[0093] Among them, Gaussian noise only changes the image pixel value without affecting the pixel coordinates. is based on Gaussian noise ε~N(0,σ 2 I) pixel transformation. It is a watermark verification model simplified from the basic watermark verification model h, and assumes satisfy So Will apply to all ∥δ G ∥2 <R G , where R G is the certification radius for Gaussian noise perturbation:

[0094] R G =σΦ -1 (p A ) (12)

[0095] represents the noise addition of the pixel. Therefore, δ G Represents the pixel-level distance between the original image and the perturbed image. Φ -1 Represents the inverse function of the standard normal distribution. ||δ G ||2 As long as the added noise is less than R in equation (12) G , It can resist the added Gaussian noise disturbance. When the noise level σ is high, or p A When it is larger, the authentication radius R G In this way, the authentication radius for the disturbance of Gaussian noise can be determined.

[0096] Additionally, for affine transformation, affine transformation only changes the image coordinates, and the affine transformation parameters have six components (ie, β = [β1, ..., β6]), applicable to the coordinates. Let θ F :V×R→V is a coordinate transformation based on Gaussian noise ρ~N(0,σ2I). Assume is the smoothed watermark verification model of the base model, and assumes Satisfy P(h(f I (θ F (v,ρ)),t,τ))≤p A ,but Applicable to all ||δ F||2 <R F , where R F is the certification radius for affine transformation perturbations:

[0097] R F =σΦ -1 (p A ) (13)

[0098] θ F It represents the linear transformation of the parameter table β to the x-axis and y-axis, that is, v =<i,j> δ F Represents the distance of the coordinate transformation coefficient, denoted as It only needs to satisfy the equation (13) about δ F conditions, It can resist affine perturbations. In this way, the authentication radius of affine perturbations can be determined.

[0099] Therefore, this embodiment models different image disturbances, thereby obtaining a mathematically accurate authentication radius corresponding to the disturbance in the verification phase. The authentication radius can be used to provide a theoretical robustness guarantee. Compared with the empirical method used in the prior art, this embodiment can ensure that the classification result of the output watermark verification does not change under a specific range of disturbances in the input image. The maximum radius.

[0100] In one embodiment, when the perturbation includes a perturbation added by a pixel transformation based on Gaussian noise, the perturbation modeling includes modeling for Gaussian noise perturbation; when the perturbation includes a perturbation added by an affine transformation, the perturbation modeling includes modeling for affine change perturbation.

[0101] That is, mathematical modeling can be performed for different disturbances to determine the authentication radius corresponding to different disturbances, so as to provide theoretical robustness guarantees for different image disturbances.

[0102] In addition, in one embodiment, the training method of the above-mentioned image watermark processing network may also include:

[0103] Obtain a watermark image to be verified; add multiple interferences obtained by sampling to the watermark image to be verified respectively, and vote on the watermark verification results corresponding to the multiple interferences based on sample voting to verify the robustness of the image watermark processing network.

[0104] Specifically, in the watermark verification stage, for the watermark image to be verified, the sampled multiple interferences can be added to the watermark image to be verified (for example, 100,000 times of Gaussian noise addition), and multiple samples corresponding to the multiple interferences are extracted. Then, based on the trained image watermark processing network, the watermark verification results y corresponding to the multiple samples are calculated.A The initial guess value of p is calculated based on multiple samples. A (p A The value of y A Finally, the robustness of the image watermark processing network is verified based on sample voting on multiple samples, and a robust prediction result is output.

[0105] Through the above process, this embodiment can verify the robustness of the image watermark processing network based on multiple sample samplings.

[0106] Additionally, in one embodiment, the residual information loss function is used to maximize the mutual information between the watermark image and the watermark information, and minimize the mutual information between the watermark residual image and the watermark information.

[0107] Among them, by maximizing the mutual information between the watermark image and the watermark information, and minimizing the mutual information between the watermark residual image and the watermark information, a strategy of reducing the loss of residual information is adopted to further reduce the risk of identity information leakage, thereby improving the security of identity information protection during watermark image processing. Thus, by finding a balance between robustness and identity leakage, the possibility of identity information leakage is reduced while achieving higher-precision watermark verification.

[0108] This embodiment also provides a watermark embedding method. Figure 6 is a flow chart of the watermark embedding method of this embodiment. Figure 6 As shown, the process includes the following steps:

[0109] Step S601, obtaining the image to be embedded and watermark information.

[0110] Step S602: Based on the training method of the image watermark processing network provided in any of the above embodiments, the trained encoder network performs watermark embedding processing on the image to be embedded and the watermark information to obtain a watermark embedded image.

[0111] Through the above steps S601 to S602, watermark embedding processing is performed based on the image watermark processing network with the dual functions of verifiable robustness guarantee and identity leakage protection, which can improve the watermark embedded image's ability to resist interference and reduce the risk of identity leakage in the subsequent watermark extraction process.

[0112] In addition, this embodiment also provides a watermark verification method. Figure 7 FIG. 1 is a flow chart of the watermark verification method of this embodiment. Figure 7 As shown, the watermark verification method may include:

[0113] Step S701, obtaining the watermark image to be extracted;

[0114] Step S702, based on the training method of the image watermark processing network provided in any of the above embodiments, the decoder network obtained by training performs watermark extraction processing on the watermark image to be extracted to obtain a restored image and extract watermark information;

[0115] Step S703, comparing the extracted watermark information with the original watermark information corresponding to the restored image to obtain a watermark verification result.

[0116] Through the above steps S701 to S703, watermark extraction processing is performed based on the image watermark processing network with the dual functions of verifiable robustness guarantee and identity leakage protection, which can improve the watermark extraction process's ability to resist interference and reduce the risk of identity leakage.

[0117] Next, the application scenario of watermark processing is explained. If an artificial intelligence company C promises to embed a watermark in the generated image it provides, it means that the company C will integrate the corresponding watermark algorithm in all images generated by the service platform it provides. Therefore, the company C can train the image watermark processing network obtained based on the training method of the image watermark processing network provided in the above embodiment to embed the watermark. When the user uses the artificial intelligence service of company C, he can request to generate the corresponding image by entering a specific text prompt. After that, the image generated according to these prompts will be sent to the watermark encoder together with the relevant watermark information, and finally returned to the user in the form of an image (watermark image) containing an invisible watermark. If the user chooses to share these generated watermark images to various social media platforms on the Internet, the shared watermark images may be subject to information modification due to the technical processing flow of the platform or the attack behavior of a third party, and then continue to spread on the network. On the one hand, the traceable watermark information in the image will be destroyed, making it impossible for subsequent detection and tracing to obtain reliable results. On the other hand, the attacker may use the residual image of the watermark image to obtain the identity information in the watermark information, resulting in the leakage of identity information.

[0118] In this regard, trusted third-party organizations such as social media platforms and other related forums can extract watermark information by deploying the training method of the image watermark processing network provided by the above embodiment, and the trained image watermark processing network. Not only can the accuracy of watermark verification and traceability results be ensured under a certain degree of attack and processing (based on the limit determined by the authentication radius), but it can also prevent attackers from stealing identity information through watermark residual images, reducing the risk of identity leakage.

[0119] Therefore, it can provide reliable guarantees for the source and authenticity of images, thereby helping the public to more accurately distinguish the authenticity of images, reduce the spread of false information and the leakage of identity privacy, and also provide effective technical support for content supervision in cyberspace.

[0120] Figure 8 is a flowchart of a method for training an image watermark processing network in some of the embodiments, such as Figure 8 As shown, the training method of the image watermark processing network includes the following steps:

[0121] Step S801, during the training phase, at a perturbation layer between the encoder network and the decoder network, perturbations are added to the image output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation to train the encoder network and the decoder network.

[0122] Step S802, based on the preset residual information loss function, the encoder network and the decoder network are trained to obtain the trained image watermark processing network. The residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is the residual image of the watermark image and the corresponding original image; wherein the residual information loss function is used to maximize the mutual information between the watermark image and the watermark information, and minimize the mutual information between the watermark residual image and the watermark information; the residual information loss function is: the mutual information between the watermark image and the watermark information and the mutual information between the watermark residual image and the watermark information are constructed by approximate representation; the approximate representation is calculated based on the neural estimator.

[0123] Step S803, in the watermark verification stage, obtaining a watermark image to be verified.

[0124] Step S804, adding the sampled multiple interferences to the watermark image to be verified respectively, and voting on the watermark verification results corresponding to the multiple interferences based on sample voting to verify the robustness of the image watermark processing network.

[0125] Step S805, after completing the training, perturbation modeling is performed based on the perturbation added in the perturbation layer to obtain a perturbation certification radius corresponding to the perturbation. Wherein, when the perturbation includes perturbation added based on Gaussian noise pixel transformation, the perturbation modeling includes modeling for Gaussian noise perturbation; when the perturbation includes perturbation added based on affine transformation, the perturbation modeling includes modeling for affine change perturbation.

[0126] Steps S801 to S805 use random smoothing to enhance the robustness of the image watermarking technology and achieve effective protection against two types of interference: pixel transformation and coordinate transformation. Based on the strategy of reducing the loss of residual information, the residual information loss function is constructed with the goal of reducing the mutual information between the watermark residual image and the watermark information, so as to reduce the probability of identity information leakage.

[0127] Among them, for the protection against identity leakage, the watermark identity leakage of the watermark residual image is reduced by specifying the mutual information target. The following table is a comparative analysis of identity leakage on two different data sets (COCO data set and CelebA data set) and two different watermark processing networks (with identity leakage protection and without identity leakage protection). Among them, the "Forged bit accuracy" in the table indicates the success rate of the attacker using the watermark residual image to imitate the watermark identity information, which reflects the degree of leakage of the watermark identity information. Based on the table below, it can be seen that the network using identity protection (W / RIL) is better than the one without identity leakage protection. The degree of identity leakage of network watermarks is significantly reduced.

[0128]

[0129] Among them, the column corresponding to "Clean vanilla" indicates the imitation accuracy under the original watermark training method; the column corresponding to "Empirically robust" indicates the imitation accuracy under the watermark training method using the empirical method; the column corresponding to "W-RS: Gaussian noise" indicates the imitation accuracy under Gaussian noise using the verifiable robust module; the column corresponding to "W-RS:Affine" indicates the imitation accuracy under affine transformation using the verifiable robust module. Imitation accuracy reflects the degree of identity information leakage, so the lower the accuracy, the more effective the identity privacy protection.

[0130] In general, the present embodiment introduces random smoothing to ensure the robustness of the image watermark against disturbances of pixel transformation and / or coordinate transformation, and models the image disturbance as a combination of pixel and coordinate transformation, thereby providing provable robustness guarantees in the respective pixel space and coordinate space. In addition, based on the theoretical analysis of mutual information, the identity information of the watermark image is fully retained, and the watermark information of the residual watermark image is reduced. Variational reasoning is further used to derive the residual information loss, so as to more objectively fit the mutual information to enhance the identity leakage protection in the robust watermark.

[0131] Fig. 9 FIG. 1 is a schematic diagram of the composition of an image watermark processing network of this embodiment. Fig. 9As shown, the graphic watermark processing network includes an encoder network and a decoder network. Based on training, an identity leakage protection module is introduced to provide identity leakage protection, and a verifiable robustness module is introduced to provide robustness guarantee.

[0132] In this embodiment, a training device for an image watermark processing network is also provided, which is used to implement the above embodiments and preferred implementations, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0133] Fig.10 : is a structural block diagram of the training device 10 of the image watermark processing network of this embodiment, as shown in Fig.10 As shown, the training device 10 of the image watermark processing network includes: the image watermark processing network includes an encoder network and a decoder network, and the training device 10 of the image watermark processing network includes: a first training module 12 and a second training module 14; wherein:

[0134] The first training module 12 is used to add perturbations to the image output by the encoder network at the perturbation layer between the encoder network and the decoder network during the training phase, based on a preset pixel transformation and / or a preset coordinate transformation, so as to train the encoder network and the decoder network; the second training module 14 is used to train the encoder network and the decoder network based on a preset residual information loss function to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image.

[0135] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0136] In one of the embodiments, the training device 10 for the image watermark processing network may further include: a radius authentication module; the radius authentication module is used to perform perturbation modeling based on the perturbation added to the perturbation layer to obtain a perturbation authentication radius corresponding to the perturbation.

[0137] Additionally, in one embodiment, when the perturbation includes a perturbation added by a pixel transformation based on Gaussian noise, the perturbation modeling includes modeling for Gaussian noise perturbation; when the perturbation includes a perturbation added by an affine transformation, the perturbation modeling includes modeling for affine change perturbation.

[0138] In addition, in one embodiment, the training device 10 of the above-mentioned image watermark processing network may also include: a verification module; the verification module is used to obtain the watermark image to be verified; the multiple interferences obtained by sampling are added to the watermark image to be verified respectively, and based on sample voting, the watermark verification results corresponding to the multiple interferences are voted to verify the robustness of the image watermark processing network.

[0139] In one embodiment, the residual information loss function is used to maximize the mutual information between the watermark image and the watermark information, and minimize the mutual information between the watermark residual image and the watermark information.

[0140] In one embodiment, the residual information loss function is: an approximate representation of the mutual information between the watermark image and the watermark information and the mutual information between the watermark residual image and the watermark information is constructed; the approximate representation is calculated by a neural estimator.

[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0142] In addition, in combination with the training method, watermark embedding method, and watermark verification method of the image watermark processing network provided in the above embodiments, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the training methods, watermark embedding methods, or watermark verification methods of the image watermark processing network in the above embodiments is implemented.

[0143] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0145] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A training method for an image watermark processing network, characterized in that: The image watermark processing network includes an encoder network and a decoder network, and the method includes: In the training phase, a perturbation layer between the encoder network and the decoder network adds perturbations to the image output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation to train the encoder network and the decoder network; and Based on a preset residual information loss function, the encoder network and the decoder network are trained to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image.

2. The training method for image watermark processing network according to claim 1, characterized in that: The method further comprises: Perturbation modeling is performed based on the disturbance added in the perturbation layer to obtain a perturbation certification radius corresponding to the disturbance.

3. The training method of the image watermark processing network according to claim 2, characterized in that: In the case where the perturbation comprises a perturbation added by a pixel transformation based on Gaussian noise, the perturbation modeling comprises modeling for Gaussian noise perturbation; In the case where the perturbation comprises a perturbation added based on an affine transformation, the perturbation modeling comprises modeling for the affine transformation perturbation.

4. The training method for image watermark processing network according to claim 1, characterized in that: The method further comprises: Get the watermark image to be verified; The multiple interferences obtained by sampling are added to the watermark image to be verified respectively, and based on sample voting, the watermark verification results corresponding to the multiple interferences are voted to verify the robustness of the image watermark processing network.

5. The training method of an image watermark processing network according to any one of claims 1 to 4, characterized in that: The residual information loss function is used to maximize the mutual information between the watermark image and the watermark information, and minimize the mutual information between the watermark residual image and the watermark information.

6. The training method for image watermark processing network according to claim 5, characterized in that: The residual information loss function is constructed by: constructing an approximate representation of the mutual information between the watermark image and the watermark information and the mutual information between the watermark residual image and the watermark information; the approximate representation is calculated based on a neural estimator.

7. A watermark embedding method, characterized in that: include: Obtain the image and watermark information to be embedded; Based on the training method of the image watermark processing network according to any one of claims 1 to 6, the trained encoder network performs watermark embedding processing on the image to be embedded and the watermark information to obtain a watermark embedded image.

8. A watermark verification method, characterized in that: include: Obtain the watermark image to be extracted; Based on the training method of the image watermark processing network according to any one of claims 1 to 6, the decoder network obtained by training performs watermark extraction processing on the watermark image to be extracted to obtain a restored image and extract watermark information; The extracted watermark information is compared with the original watermark information corresponding to the restored image to obtain a watermark verification result.

9. A training device for an image watermark processing network, characterized in that: include: The image watermark processing network includes an encoder network and a decoder network, and the training device includes: a first training module and a second training module; wherein: The first training module is used to add perturbations to the images output by the encoder network based on a preset pixel transformation and / or a preset coordinate transformation at a perturbation layer between the encoder network and the decoder network during a training phase, so as to train the encoder network and the decoder network; The second training module is used to train the encoder network and the decoder network based on a preset residual information loss function to obtain a trained image watermark processing network; the residual information loss function is constructed based on the mutual information between the watermark residual image and the watermark image; the watermark residual image is a residual image of the watermark image and the corresponding original image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the training method of the image watermark processing network described in any one of claims 1 to 6 are implemented, or the steps of the watermark embedding method described in claim 7 are implemented, or the steps of the watermark verification method described in claim 8 are implemented.

Images