A method for processing digital work data

Abstract

The present invention relates to the field of digital work data processing, and discloses a digital work data processing method, including: generating a robust watermark and a corresponding hidden prompt signal through an adversarial optimization algorithm, wherein the hidden prompt signal is used to guide the model to hide the watermark during the process of a diffusion model generating an image work; implanting the generated robust watermark in the intermediate step of the reverse diffusion stage of the diffusion model generating the image work; in the remaining reverse diffusion steps after the watermark is implanted, using text prompts and prompt signals to jointly guide the diffusion model to generate an image work that hides the robust watermark and meets semantic conditions, and reducing the influence of the watermark on the quality and visual effect of the final generated image; when it is necessary to verify the source and ownership of the generated image digital work, verifying the existence of the robust watermark through the reverse process of the reverse diffusion of the diffusion model. The present invention can realize the secret implantation of a robust watermark in the generated content without retraining the model.

Description

Technical Field

[0001] The present invention relates to the technical field of digital work data processing, and more specifically, to a digital work data processing method. Background Art

[0002] Diffusion Models (DMs) are revolutionizing content creation for digital works and generating stunning photorealistic images in different fields. Diffusion models are generative models that gradually transform noise into images through an iterative process. Such models are used to generate photorealistic images, such as text-to-image diffusion models, which can generate corresponding images based on text descriptions. The emergence of text-to-image diffusion models, coupled with personalized generation techniques, has enabled almost anyone to create highly specific content. However, it has also raised concerns about authenticity and ownership, including the risk of plagiarism and the potential misuse of images of public figures. As a result, there is a growing advocacy for robust mechanisms to verify the origin of generated content. Watermarking technology provides a proactive way to verify the origin of generated content. This technology embeds imperceptible secret information in the generated content. This information serves as a unique identifier to confirm the source of the image while being unnoticeable to the human eye. The information also needs to be robust enough to withstand the distortion that may occur during online sharing.

[0003] Existing watermarking techniques face great challenges in balancing stealth and robustness. Traditional post-processing methods use empirical methods to identify invisible and robust watermarks and embed them into the generated image. These methods passively achieve stealth by limiting the watermark strength, which compromises robustness. Conversely, stronger watermarks enhance robustness but also introduce visible artifacts in the generated image. Recent advances in intrinsic watermarking techniques for diffusion models expect that these methods can allow generative models to learn this balance and directly generate watermark content. However, these methods often require expensive model retraining or cause unexpected semantic changes in the generated image. Summary of the invention

[0004] The purpose of the present invention is to propose a digital work data processing method, which can secretly embed a robust watermark in the generated content without retraining the model.

[0005] To achieve the above-mentioned purpose, the present invention proposes a digital work data processing method for implanting a hidden watermark in the process of generating an image digital work using a diffusion model. The method comprises:

[0006] Generate a robust watermark and a corresponding hidden prompt signal through an adversarial optimization algorithm, wherein the hidden prompt signal is used to guide the model to hide the watermark during the process of generating image works by the diffusion model;

[0007] Implanting the generated robust watermark in the intermediate step of the reverse diffusion stage of the diffusion model generating the image work;

[0008] In the remaining reverse diffusion steps after the watermark is implanted, the text prompt and the prompt signal are used to jointly guide the diffusion model to generate an image work that hides the robust watermark and meets the semantic conditions, and reduce the impact of the watermark on the quality and visual effect of the final generated image;

[0009] When it is necessary to verify the source and ownership of the generated digital image work, the existence of the robust watermark is verified through the inverse process of the inverse diffusion of the diffusion model.

[0010] Optionally, the generating of a robust watermark and a corresponding hidden prompt signal by using an adversarial optimization algorithm includes:

[0011] Randomly initialize the watermark: watermark w i It starts from a random value to ensure that each generated watermark is unique;

[0012] Hide prompt signal initialization: Hide prompt signal w p Initialized to NULL or a baseline value, indicating no additional guidance signal;

[0013] Select sample: randomly select a sample x0 from the data set, which is the image to be generated by the diffusion model;

[0014] Obtain noise representation: For the selected sample x0, calculate its random noise representation x at the watermark embedding point through the forward process of the diffusion model t ;

[0015] Perform adversarial optimization:

[0016] Alternately optimize watermark w i and optimization prompt signal w p ,in,

[0017] Optimize watermark i : At a fixed prompt signal w p Next, optimize the watermark w i To minimize the image preservation loss function and the constraint loss function, and at the same time minimize w i The norm of maximizes the strength of the watermark;

[0018] Optimize the prompt signal w p :In the fixed watermark w i Next, optimize the prompt signal w p To minimize the image preservation loss function and the constraint loss function;

[0019] Repeat the alternating optimization steps for iterative optimization until the maximum number of iterations is reached or the value of the loss function is lower than the set threshold, and output the optimized watermark w i and prompt signal w p , that is, the robust watermark and the corresponding hidden prompt signal are obtained.

[0020] Optionally, the image preservation loss function is:

[0021] ,

[0022] in, ,

[0023]

[0024] In the formula, The loss function is retained for the image, which is used to measure the difference between the watermarked image and the original image; MES stands for mean square error; is the original, unwatermarked image; It is the final image generated by the inverse process of the diffusion model under the guidance of watermark implantation and prompt signal; is the noise representation at the watermark implantation point, i.e., the state after the watermark is implanted at an intermediate step t in the diffusion process; is the cumulative acceptance rate at the current time step t, which is used to control the noise addition and removal process; represents the embedding of the text prompt p; Indicates that at a given watermark implantation point , time step t, embedded text prompt p and prompt signal w p When , the model predicts the noise; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the reverse diffusion process, thereby restoring a clear image; Indicates that at a given watermark implantation point , time step t and prompt signal w p The noise of the model prediction is generated by considering the watermark and the prompt signal; Indicates that at a given watermark implantation point , the noise in the model prediction at time step t and text prompt p; is the embedding representation of the empty text prompt, that is, the embedding vector when no text prompt is provided; Indicates that at a given watermark implantation point , time step t, and an empty text prompt, the noise predicted by the model, this noisy prediction is generated without additional guidance.

[0025] Optionally, the constraint loss function is:

[0026] ,

[0027] in, is the image constraint loss function, which is used to prevent the prompt signal from having too much influence on the image generation process; Indicates that at a given watermark implantation point , time step t and prompt signal w p When , the model predicts the noise; Indicates that at a given watermark implantation point , time step t, and an empty text prompt.

[0028] Optionally, the objective function of the prompt signal optimization in the adversarial optimization algorithm is:

[0029] ,

[0030] in, is the optimization loss function of the prompt signal; α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss.

[0031] Optionally, the objective function of watermark optimization in the adversarial optimization algorithm is:

[0032] ,

[0033] in, is the optimized loss function of the watermark, α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss; λ is a hyperparameter used to control the regularization term of watermark strength; is the norm of the watermark, which is used to maximize the strength of the watermark.

[0034] Optionally, the generated robust watermark is implanted in an intermediate step of the reverse diffusion stage of the diffusion model generating the image work, including:

[0035] In the multiple steps of the reverse diffusion of the generation process of the diffusion model, an intermediate step is determined as the best watermark implantation point. The best watermark implantation point is the diffusion stage where the basic semantics of the image has been formed but the details have not been fully determined.

[0036] The intermediate diffusion state x corresponding to the selected watermark implantation point t The watermark wi is implanted in the equation:

[0037] ,

[0038] in, is the intermediate diffusion state after watermarking, I is the watermarking embedding function, which is used to embed the watermark into the frequency domain, and x t is the noise representation of the watermark implantation point, i.e., the state of an intermediate step in the diffusion process, w i For the generated watermark, M is the coverage area of ​​the watermark, which defines the position and range of the watermark in the frequency domain.

[0039] Optionally, the method for selecting the best watermark implantation point includes:

[0040] By analyzing the changes of predicted noise in different diffusion steps, it is determined that the watermark implantation point should be selected after the basic semantics of the image is formed and before the image details are formed, which has the least impact on the image semantics and quality.

[0041] By analyzing the sensitivity of the prediction noise to the frequency domain interference in different diffusion steps, an implantation point is selected so that the addition of the watermark has the least impact on the average value of the prediction noise;

[0042] The selected implantation point needs to balance the robustness and concealment of the watermark, ensuring that the watermark remains detectable under various image operations and transformations while having minimal impact on the visual quality of the image.

[0043] Optionally, in the remaining reverse diffusion step after the watermark is implanted, the text prompt and the prompt signal are used together to guide the diffusion model to generate an image work that hides the robust watermark and satisfies the semantic condition, including:

[0044] In the remaining back-diffusion generation step after watermarking, the optimized cue signal w is used. p The continuous guidance diffusion model hides the implanted robust watermark, and the prompt signal w p Combined with the original text hint p, it ensures that the generated image not only conforms to the text description but also hides the watermark;

[0045] After the watermark implantation point, the adjusted generating function is used to generate an image with a hidden watermark, and the generating function is:

[0046]

[0047]

[0048] in, is a generating function used to generate the state of the previous step from the current step; is the previous state generated, is the status of the current step; The final image predicted from the noise latent state by the inverse process for the case without watermark implantation; The final image predicted from the watermark noise latent state by the inverse process for the watermarking case; Indicates that no watermark is implanted in a given state , the noise in the model prediction at time step t and text prompt p; Indicates that at a given watermark implantation point , time step t, text prompt p and prompt signal w p When , the model predicts the noise; Indicates the time step of the watermark implantation point; T is the total number of time steps; is the cumulative acceptance rate at the current time step t; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the inverse diffusion process, thereby restoring a clear image.

[0049] Optionally, verifying the existence of the robust watermark by an inverse process of inverse diffusion of the diffusion model includes:

[0050] The transformed watermark image Reverse to the watermark embedding step t injection , to retrieve the intermediate diffusion state diffusion ;

[0051] The intermediate state obtained from the reverse Extract the watermark information w′ in the frequency space;

[0052] The similarity between the original watermark w and the extracted watermark w′ is calculated by the following formula:

[0053] ,

[0054] In the formula, represents the L1 distance between the original watermark w and the extracted watermark w′, Indicates the size of the area covered by the watermark. is the value of the original watermark in region M, To extract the value of the watermark in area M;

[0055] The calculated distance is compared with the preset threshold τ. If D(w,w′)≤τ, it is determined that the watermark still exists.

[0056] The beneficial effects of the present invention are:

[0057] The present invention aims to provide a robust and covert watermark for the content generated by the diffusion model for authentication, ownership protection and reduction of potential abuse. Firstly, the present invention uses an adversarial optimization algorithm to generate a robust watermark and a hidden prompt signal, and implants the watermark in the intermediate step of the reverse generation process of the diffusion model to ensure the concealment of the watermark and the image quality. By alternately optimizing the watermark and the prompt signal, the concealment and robustness of the watermark are balanced, and the image preservation loss and the constraint loss are used to guide the optimization process. After that, the optimized prompt signal is used to guide the model to hide the watermark during the generation process, reducing the impact of the watermark on the final image. The intermediate state of the watermark implantation point can be retrieved through the reverse diffusion process, and the watermark information can be extracted. Then, the existence and integrity of the watermark can be verified using a metric such as the L1 distance. The watermark generated by the method of the present invention can still be detected even after the image has been significantly tampered with and transformed; at the same time, the watermark is implanted in a way that makes it invisible to the human eye and does not affect the visual quality of the image; the method does not require retraining of the diffusion model, reducing the computational cost and time; the watermark implantation and hiding process does not interfere with the semantic content of the image, maintaining consistency with the original text prompt; and the watermark verification process is relatively fast because it only needs to be reversed to a specific watermark implantation point, rather than the entire generation process. The scheme is applicable to a variety of diffusion models, including potential and image diffusion models, and has good versatility. The method can help creators establish ownership, prevent unauthorized use, and may provide technical support for legal copyright protection.

[0058] The system of the present invention has other characteristics and advantages, which will be apparent from the drawings incorporated herein and the following detailed description, or will be described in detail in the drawings incorporated herein and the following detailed description, which together serve to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, in which like reference numerals generally represent like components.

[0060] Figure 1 A flow chart showing the steps of a method for processing digital work data according to the present invention is shown. DETAILED DESCRIPTION

[0061] Generating content watermarks is an important tool for authentication, ownership protection, and mitigation of potential abuse. Existing watermarking methods face the challenge of balancing robustness and concealment. Traditional methods empirically inject watermarks that are both invisible and robust, and passively achieve concealment by limiting the watermark strength, which reduces robustness.

[0062] The present invention proposes a digital work data processing method, which can be applied to a text-to-image diffusion model, which supports additional hint signals to guide the generation process. The method actively achieves hiding by explicitly introducing a watermark hiding process, thereby allowing for the embedding of stronger watermarks. Specifically, a robust watermark is implanted in an intermediate diffusion state, and then the diffusion model is guided to hide the watermark in the final generated image, and an adversarial optimization algorithm is used to generate the best hidden hint guidance signal for each watermark. Hint embedding is optimized to minimize artifacts in the generated image, while the watermark is optimized to achieve maximum strength, and the watermark can be verified by reversing the generation process. Experiments on various diffusion models show that the watermark can still be verified even when the image is severely tampered with, and shows higher concealment than other state-of-the-art robust watermarking methods.

[0063] The present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0064] First, the working principle of the existing diffusion model is described:

[0065] Diffusion models are a type of generative model that generates data (such as images) by simulating the physical diffusion process. The working principle of this model is based on a simple intuition: if a random noise process can gradually transform meaningful data into meaningless noise, then in theory, this process can be reversed to recover meaningful data from the noise. Here is the basic working principle of the diffusion model:

[0066] (1) Forward Process

[0067] initialization:

[0068] Start with a simple data distribution, usually Gaussian (i.e. noise).

[0069] Add noise gradually:

[0070] The data is gradually converted into noise by adding noise step by step. Each step introduces more randomness into the data according to a predetermined noise scheme until it is completely converted into noise.

[0071] Markov Chain:

[0072] This process can be viewed as a Markov chain, where the state at each step depends only on the state at the previous step and is independent of all previous states.

[0073] (2) Reverse Process

[0074] Initialize the noise:

[0075] It starts from a pure noise state, which is the end point of the forward diffusion process.

[0076] Remove noise step by step:

[0077] The goal of the inverse process is to recover the original data from the noise. This is achieved by removing the noise step by step, and each step will reduce the effect of the noise according to the inverse diffusion probability distribution learned by the model.

[0078] Prediction noise:

[0079] At each step, the model needs to predict the noise of the current step and remove this noise from the current state, thereby gradually recovering the structure of the data.

[0080] Iterative Optimization:

[0081] The reverse diffusion process usually requires iterative optimization, and each step updates the current state based on the model's understanding of the data distribution until it converges to meaningful data.

[0082] (3) Training process

[0083] Data distribution estimation:

[0084] The training process of a diffusion model involves estimating the data distribution, which is usually done by maximizing the log-likelihood of the data.

[0085] Noise prediction network:

[0086] A neural network is trained to predict the noise at each step. This network is usually a deep learning model such as U-Net, which learns to predict the complete noise pattern from partial noise data.

[0087] Iterative refinement:

[0088] Through iterative training, the model gradually learns how to predict more accurately and remove noise, thereby improving the quality of generated data

[0089] Diffusion models have attracted much attention due to their ability to generate high-quality samples, especially in the field of image generation. They are able to generate detailed and realistic images, and have also shown potential in processing sequential data such as text, time series, etc. The success of diffusion models is attributed to their ability to gradually and finely restore the data structure, thereby maintaining the diversity and details of the data during the generation process.

[0090] The method of the present invention is further explained below by a specific embodiment.

[0091] like Figure 1 As shown, this embodiment provides a digital work data processing method for implanting a hidden watermark in the process of generating an image digital work by a diffusion model. The method includes:

[0092] S1: Generate a robust watermark and a corresponding hidden prompt signal through an adversarial optimization algorithm, wherein the hidden prompt signal is used to guide the model to hide the watermark during the process of generating image works by the diffusion model;

[0093] In this step, the generation of a robust watermark and a corresponding hidden prompt signal by using an adversarial optimization algorithm includes:

[0094] Randomly initialize the watermark: watermark w i It starts from a random value to ensure that each generated watermark is unique;

[0095] Hide prompt signal initialization: Hide prompt signal w p Initialized to NULL or a baseline value, indicating no additional guidance signal;

[0096] Select sample: randomly select a sample x0 from the data set, which is the image to be generated by the diffusion model;

[0097] Obtain noise representation: For the selected sample x0, calculate its random noise representation x at the watermark embedding point through the forward process of the diffusion model t ;

[0098] Perform adversarial optimization:

[0099] Alternately optimize watermark w i and optimization prompt signal w p ,in,

[0100] Optimize watermark i : At a fixed prompt signal w p Next, optimize the watermark w i To minimize the image preservation loss function and the constraint loss function, and at the same time minimize w i The norm of maximizes the strength of the watermark;

[0101] Optimize the prompt signal w p :In the fixed watermark w i Next, optimize the prompt signal w p To minimize the image preservation loss function and the constraint loss function;

[0102] Repeat the alternating optimization steps for iterative optimization until the maximum number of iterations is reached or the value of the loss function is lower than the set threshold, and output the optimized watermark w i and prompt signal w p , that is, the robust watermark and the corresponding hidden prompt signal are obtained.

[0103] Wherein, the image preservation loss function is:

[0104] ,

[0105] in, ,

[0106]

[0107] In the formula, The loss function is retained for the image, which is used to measure the difference between the watermarked image and the original image; MES stands for mean square error; is the original, unwatermarked image; It is the final image generated by the inverse process of the diffusion model under the guidance of watermark implantation and prompt signal; is the noise representation at the watermark implantation point, i.e., the state after the watermark is implanted at an intermediate step t in the diffusion process; is the cumulative acceptance rate at the current time step t, which is used to control the noise addition and removal process; represents the embedding of the text prompt p; Indicates that at a given watermark implantation point , time step t, embedded text prompt p and prompt signal w p When , the model predicts the noise; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the reverse diffusion process, thereby restoring a clear image; Indicates that at a given watermark implantation point , time step t and prompt signal w p The noise of the model prediction is generated by considering the watermark and the prompt signal; Indicates that at a given watermark implantation point , the noise in the model prediction at time step t and text prompt p; is the embedding representation of the empty text prompt, that is, the embedding vector when no text prompt is provided; Indicates that at a given watermark implantation point , time step t, and an empty text prompt, the noise predicted by the model, this noisy prediction is generated without additional guidance.

[0108] The constraint loss function is:

[0109] ,

[0110] in, is the image constraint loss function, which is used to prevent the prompt signal from having too much influence on the image generation process; Indicates that at a given watermark implantation point , time step t and prompt signal w p When , the model predicts the noise; Indicates that at a given watermark implantation point , time step t, and an empty text prompt.

[0111] In this embodiment, the objective function of prompt signal optimization in the adversarial optimization algorithm is:

[0112] ,

[0113] in, is the optimization loss function of the prompt signal; α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss.

[0114] The objective function of watermark optimization in the adversarial optimization algorithm is:

[0115] ,

[0116] in, is the optimized loss function of the watermark, α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss; λ is a hyperparameter used to control the regularization term of watermark strength; is the norm of the watermark, which is used to maximize the strength of the watermark.

[0117] S2: implanting the generated robust watermark in the intermediate step of the reverse diffusion stage of the diffusion model to generate the image work;

[0118] This step specifically includes:

[0119] In the multiple steps of the reverse diffusion of the generation process of the diffusion model, an intermediate step is determined as the best watermark implantation point. The best watermark implantation point is the diffusion stage where the basic semantics of the image has been formed but the details have not been fully determined.

[0120] The intermediate diffusion state x corresponding to the selected watermark implantation point t The watermark wi is implanted in the equation:

[0121] ,

[0122] in, is the intermediate diffusion state after watermarking, I is the watermarking embedding function, which is used to embed the watermark into the frequency domain, and x t is the noise representation of the watermark implantation point, i.e., the state of an intermediate step in the diffusion process, w i For the generated watermark, M is the coverage area of ​​the watermark, which defines the position and range of the watermark in the frequency domain.

[0123] In this embodiment, the method for selecting the best watermark implantation point includes:

[0124] By analyzing the changes of predicted noise in different diffusion steps, it is determined that the watermark implantation point should be selected after the basic semantics of the image is formed and before the image details are formed, which has the least impact on the image semantics and quality.

[0125] By analyzing the sensitivity of the prediction noise to the frequency domain interference in different diffusion steps, an implantation point is selected so that the addition of the watermark has the least impact on the average value of the prediction noise;

[0126] The selected implantation point needs to balance the robustness and concealment of the watermark, ensuring that the watermark remains detectable under various image operations and transformations while having minimal impact on the visual quality of the image.

[0127] Specifically, choosing the best watermark implantation point is a key step because it affects the concealment and robustness of the watermark and its impact on image quality. Factors to consider include:

[0128] (1) The implantation point should be selected after the basic semantics and structure of the image have been formed to avoid the watermark having a significant impact on the image content.

[0129] (2) Watermarks are usually implanted in the intermediate steps of the diffusion process, which should be before the details of the image are formed so that there are enough steps to hide the watermark.

[0130] (3) It is necessary to analyze the sensitivity of the prediction noise to the frequency domain interference in different diffusion steps and select an implantation point so that the addition of the watermark has the least impact on the average value of the prediction noise.

[0131] (4) The selected implantation points should be able to maintain the visual quality of the image and avoid introducing too many artifacts.

[0132] (5) The choice of implantation point should also take into account the robustness of the watermark to various possible image operations and transformations in subsequent steps.

[0133] Based on the above considerations, the best watermark implantation point can be selected by the following methods:

[0134] (1) Experimentally analyze the semantics and quality changes of images in different steps to determine the optimal watermark implantation point.

[0135] (2) Use sensitivity analysis to evaluate the impact of noise on the image at different steps and find the optimal time to implant the watermark.

[0136] (3) For each candidate implantation point, test the consistency between the image after watermark implantation and the original text prompt, and select the best point that maintains semantic consistency.

[0137] (4) Perform a robustness test on each candidate implantation point, including various processing and transformations of the image, and select the point that can still maintain the integrity of the watermark under these conditions.

[0138] (5) Use optimization algorithms (such as genetic algorithms, simulated annealing, etc.) to find the best watermark implantation point and search for the best solution through iteration.

[0139] (6) Perform statistical analysis on multiple candidate implantation points, compare their performance, and select the point that achieves the best balance between concealment and robustness.

[0140] In summary, choosing the best watermark implantation point requires comprehensive consideration of the image's semantics, quality, robustness, and sensitivity to prediction noise. Through experimental analysis, sensitivity analysis, and optimization algorithms, an optimal implantation point can be determined so that the watermark can be effectively hidden while maintaining robustness under various conditions, while minimizing the impact on image quality. This approach helps to achieve a robust and covert watermark for generated content without sacrificing image quality.

[0141] S3: in the remaining reverse diffusion steps after the watermark is implanted, the text prompt and the prompt signal are used to jointly guide the diffusion model to generate an image work that hides the robust watermark and meets the semantic conditions, and reduce the impact of the watermark on the quality and visual effect of the final generated image;

[0142] This step specifically includes:

[0143] In the remaining back-diffusion generation step after watermarking, the optimized cue signal w is used. p The continuous guidance diffusion model hides the implanted robust watermark, and the prompt signal w p Combined with the original text hint p, it ensures that the generated image not only conforms to the text description but also hides the watermark;

[0144] After the watermark implantation point, the adjusted generating function is used to generate an image with a hidden watermark, and the generating function is:

[0145]

[0146]

[0147] in, is a generating function used to generate the state of the previous step from the current step; is the previous state generated, is the status of the current step; The final image predicted from the noise latent state by the inverse process for the case without watermark implantation; The final image predicted from the watermark noise latent state by the inverse process for the watermarking case; Indicates that no watermark is implanted in a given state , the noise in the model prediction at time step t and text prompt p; Indicates that at a given watermark implantation point , time step t, text prompt p and prompt signal w p When , the model predicts the noise; Indicates the time step of the watermark implantation point; T is the total number of time steps; is the cumulative acceptance rate at the current time step t; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the inverse diffusion process, thereby restoring a clear image.

[0148] Among them, the text prompt is a descriptive text provided by the user to the diffusion model, which is used to guide the model to generate an image matching the text description. In the text-to-image diffusion model, the text prompt serves as conditional information to help the model understand the semantic content of the image to be generated; the prompt signal is an optimization signal based on the text prompt. The prompt signal is a numerical vector or an embedded vector, and can provide additional guiding information when the model generates an image, which is used to minimize the impact of the watermark on the final image during the image generation process while maintaining the robustness of the watermark. The prompt signal and the text prompt jointly guide the model to generate an image with a hidden watermark.

[0149] S4: When it is necessary to verify the source and ownership of the generated digital image work, the existence of the robust watermark is verified through the inverse process of the inverse diffusion of the diffusion model.

[0150] In this step, verifying the existence of the robust watermark through the inverse process of reverse diffusion of the diffusion model includes:

[0151] The transformed watermark image Reverse to the watermark embedding step t injection , to retrieve the intermediate diffusion state diffusion ;

[0152] The intermediate state obtained from the reverse Extract the watermark information w′ in the frequency space;

[0153] The similarity between the original watermark w and the extracted watermark w′ is calculated by the following formula:

[0154] ,

[0155] In the formula, represents the L1 distance between the original watermark w and the extracted watermark w′, Indicates the size of the area covered by the watermark. is the value of the original watermark in region M, To extract the value of the watermark in area M;

[0156] The calculated distance is compared with the preset threshold τ. If D(w,w′)≤τ, it is determined that the watermark still exists.

[0157] Specifically, the watermark verification process retrieves and extracts the watermark information through the reverse diffusion step and frequency domain analysis, and then measures their similarity by calculating the L1 distance between the original watermark and the extracted watermark. If this distance is less than or equal to the preset threshold, the watermark verification is considered successful, indicating that the existence and integrity of the watermark in the image have been confirmed. This method can effectively verify the robustness and concealment of the watermark after the image has been processed and transformed, which is crucial for protecting the copyright of digital content and preventing abuse.

[0158] It should be noted that both watermark implantation and watermark verification in this method involve the reverse diffusion process of the diffusion model, but they occur at different stages of the reverse diffusion. Among them, watermark implantation is to implant the watermark in the reverse diffusion process, while watermark verification is to verify the watermark in the reverse process of reverse diffusion, that is, from the final image to the state of the watermark implantation point. Such a design allows the effective implantation and verification of watermarks without damaging the image quality.

[0159] The beneficial effects of the present invention are as follows:

[0160] (1) Enhanced robustness: The watermark generated by this method can still be detected even after the image has been significantly tampered with and transformed.

[0161] (2) Improved stealth: Watermarks are embedded in such a way that they are invisible to the human eye and do not affect the visual quality of the image.

[0162] (3) No need for model retraining: The method of the present invention does not require retraining of the diffusion model, thus reducing computational cost and time.

[0163] (4) Maintaining semantic consistency: The watermark implantation and hiding process does not interfere with the semantic content of the image and maintains consistency with the original text cues.

[0164] (5) Fast Verification: The watermark verification process is relatively fast because it only requires reversing to a specific watermark implantation point rather than the entire generation process.

[0165] (6) Wide applicability: The scheme is applicable to a variety of diffusion models, including latent and image diffusion models, and has good versatility.

[0166] (7) Social and legal impact: As a watermarking method, it can help creators establish ownership, prevent unauthorized use, and may provide technical support for legal copyright protection.

[0167] Based on the above, the digital work data processing method of the present invention allows embedding a more robust watermark while maintaining the concealment of the watermark by introducing an additional watermark hiding process, that is, a robust watermark is implanted in the intermediate diffusion state of the diffusion model, and a prompt signal generated by an adversarial optimization algorithm is used to guide the model to hide the watermark. This method can achieve the robustness and concealment of the watermark while maintaining the high quality and semantic consistency of the generated image without retraining the model. It provides an effective watermark solution to protect the integrity and ownership of the content generated by the diffusion model without sacrificing image quality.

[0168] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A digital work data processing method for implanting a hidden watermark in the process of generating an image digital work using a diffusion model, characterized in that: The method comprises: Generate a robust watermark and a corresponding hidden prompt signal through an adversarial optimization algorithm, wherein the hidden prompt signal is used to guide the model to hide the watermark during the process of generating an image digital work by a diffusion model; Implanting the generated robust watermark in the intermediate step of the reverse diffusion stage of the diffusion model generating the image work; In the remaining reverse diffusion steps after the watermark is implanted, the text prompt and the prompt signal are used to jointly guide the diffusion model to generate an image work that hides the robust watermark and meets the semantic conditions, and reduce the impact of the watermark on the quality and visual effect of the final generated image; When it is necessary to verify the source and ownership of the generated digital image work, the existence of the robust watermark is verified by the inverse process of the inverse diffusion of the diffusion model; The method of generating a robust watermark and a corresponding hidden prompt signal by using an adversarial optimization algorithm includes: Randomly initialize the watermark: watermark w i It starts from a random value to ensure that each generated watermark is unique; Hide prompt signal initialization: Hide prompt signal w p Initialized to NULL or a baseline value, indicating no additional guidance signal; Select sample: randomly select a sample x0 from the data set, which is the image to be generated by the diffusion model; Obtain noise representation: For the selected sample x0, calculate its random noise representation x at the watermark embedding point through the forward process of the diffusion model t ; Perform adversarial optimization: Alternately optimize watermark w i and optimization prompt signal w p ,in, Optimize watermark i : At a fixed prompt signal w p Next, optimize the watermark w i To minimize the image preservation loss function and the constraint loss function, and at the same time minimize w i The norm of maximizes the strength of the watermark; Optimize the prompt signal w p :In the fixed watermark w i Next, optimize the prompt signal w p To minimize the image preservation loss function and the constraint loss function; Repeat the alternating optimization steps for iterative optimization until the maximum number of iterations is reached or the value of the loss function is lower than the set threshold, and output the optimized watermark w i and prompt signal w p , that is, the robust watermark and the corresponding hidden prompt signal are obtained.

2. The method for processing digital work data according to claim 1, characterized in that: The image preservation loss function is: ， in, , ， In the formula, The loss function is retained for the image, which is used to measure the difference between the watermarked image and the original image; MES stands for mean square error; is the original, unwatermarked image; It is the final image generated by the inverse process of the diffusion model under the guidance of watermark implantation and prompt signal; is the noise representation at the watermark implantation point, i.e., the state after the watermark is implanted at an intermediate step t in the diffusion process; is the cumulative acceptance rate at the current time step t, which is used to control the noise addition and removal process; represents the embedding of the text prompt p; Indicates that at a given watermark implantation point , time step t, embedded text prompt p and prompt signal w p When , the model predicts the noise; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the reverse diffusion process, thereby restoring a clear image; Indicates that at a given watermark implantation point , time step t and prompt signal w p The noise of the model prediction is generated by considering the watermark and the prompt signal; Indicates that at a given watermark implantation point , the noise in the model prediction at time step t and text prompt p; is the embedding representation of the empty text prompt, that is, the embedding vector when no text prompt is provided; Indicates that at a given watermark implantation point , time step t, and an empty text prompt, the noise predicted by the model, this noisy prediction is generated without additional guidance.

3. The method for processing digital work data according to claim 2, characterized in that: The constraint loss function is: ， in, is the image constraint loss function, which is used to prevent the prompt signal from having too much influence on the image generation process; Indicates that at a given watermark implantation point , time step t and prompt signal w p When , the model predicts the noise; Indicates that at a given watermark implantation point , time step t, and an empty text prompt.

4. The method for processing digital work data according to claim 2, characterized in that: The objective function of the prompt signal optimization in the adversarial optimization algorithm is: ， in, is the optimization loss function of the prompt signal; α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss.

5. The method for processing digital work data according to claim 4, characterized in that: The objective function of watermark optimization in the adversarial optimization algorithm is: ， in, is the optimized loss function of the watermark, α and β are hyperparameters used to balance the weights of image preservation loss and constraint loss; λ is a hyperparameter used to control the regularization term of watermark strength; is the norm of the watermark, which is used to maximize the strength of the watermark.

6. The method for processing digital work data according to claim 1, characterized in that: The robust watermark generated by implanting the generated robust watermark in the intermediate step of the reverse diffusion stage of the diffusion model generating the image work includes: In the multiple steps of the reverse diffusion of the generation process of the diffusion model, an intermediate step is determined as the best watermark implantation point. The best watermark implantation point is the diffusion stage where the basic semantics of the image has been formed but the details have not been fully determined. The intermediate diffusion state x corresponding to the selected watermark implantation point t The watermark wi is implanted in the equation: ， in, is the intermediate diffusion state after watermarking, I is the watermarking embedding function, which is used to embed the watermark into the frequency domain, and x t is the noise representation of the watermark implantation point, i.e., the state of an intermediate step in the diffusion process, w i For the generated watermark, M is the coverage area of ​​the watermark, which defines the position and range of the watermark in the frequency domain.

7. The method for processing digital work data according to claim 6, characterized in that: The best watermark implantation point selection methods include: By analyzing the changes of predicted noise in different diffusion steps, it is determined that the watermark implantation point should be selected after the basic semantics of the image is formed and before the image details are formed, which has the least impact on the image semantics and quality. By analyzing the sensitivity of the prediction noise to the frequency domain interference in different diffusion steps, an implantation point is selected so that the addition of the watermark has the least impact on the average value of the prediction noise; The selected implantation point needs to balance the robustness and concealment of the watermark, ensuring that the watermark remains detectable under various image operations and transformations while having minimal impact on the visual quality of the image.

8. The method for processing digital work data according to claim 1, characterized in that: In the remaining reverse diffusion step after the watermark is implanted, the text prompt and the prompt signal are used to jointly guide the diffusion model to generate an image work that hides the robust watermark and satisfies the semantic condition, including: In the remaining back-diffusion generation step after watermarking, the optimized cue signal w is used. p The continuous guidance diffusion model hides the implanted robust watermark, and the prompt signal w p Combined with the original text hint p, it ensures that the generated image not only conforms to the text description but also hides the watermark; After the watermark implantation point, the adjusted generating function is used to generate an image with a hidden watermark, and the generating function is: ； in, is a generating function used to generate the state of the previous step from the current step; is the previous state generated, is the status of the current step; The final image predicted from the noise latent state by the inverse process for the case without watermark implantation; The final image predicted from the watermark noise latent state by the inverse process for the watermarking case; Indicates that no watermark is implanted in a given state , the noise in the model prediction at time step t and text prompt p; Indicates that at a given watermark implantation point , time step t, text prompt p and prompt signal w p When , the model predicts the noise; Indicates the time step of the watermark implantation point; T is the total number of time steps; is the cumulative acceptance rate at the current time step t; and It is a coefficient used to adjust the noise prediction to ensure that the noise is correctly removed during the inverse diffusion process, thereby restoring a clear image.

9. The method for processing digital work data according to claim 8, characterized in that: Verifying the existence of the robust watermark through the inverse process of the inverse diffusion of the diffusion model includes: The transformed watermark image Reverse to the watermark embedding step t injection , to retrieve the intermediate diffusion state diffusion ; The intermediate state obtained from the reverse Extract the watermark information w′ in the frequency space; The similarity between the original watermark w and the extracted watermark w′ is calculated by the following formula: ， In the formula, represents the L1 distance between the original watermark w and the extracted watermark w′, Indicates the size of the area covered by the watermark. is the value of the original watermark in region M, To extract the value of the watermark in area M; The calculated distance is compared with the preset threshold τ. If D(w,w′)≤τ, it is determined that the watermark still exists.

Images