Conformal low-rank adaptive watermarking method based on text-to-image large model

Abstract

This invention provides a latent space consistency low-rank adaptive watermarking method based on a large model of text-based images. The method includes: encoding the original watermark into a watermark latent representation using a watermark encoder component; encoding the carrier image into a content latent representation using an encoder; performing discrete wavelet transform on the watermark latent representation and the content latent representation using a consistency embedding component to obtain a watermark-containing latent representation; adjusting the parameters of a denoising network using a watermark feature modulation network to obtain a watermark feature modulation denoising network; forward diffusing the watermark-containing latent representation to obtain watermark-containing initial noise; denoising the watermark-containing initial noise using the watermark feature modulation denoising network to obtain a watermark-containing image; extracting the decoded watermark from the watermark-containing image using a watermark decoder; verifying the consistency between the original watermark and the decoded watermark to achieve source tracing of the generated image. This invention implements an endogenous watermarking method based on a diffusion model, ensuring the quality of the generated image while maintaining robustness.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a latent space consistent low-rank adaptive watermarking method based on a large text-generated graph model. Background Technology

[0002] With the rapid development of image artificial intelligence technology, the stable diffusion model has been widely used in image generation, style transfer and customization due to its excellent generation quality.

[0003] Currently, one existing diffusion model watermarking method is a post-watermarking method. This method does not fully utilize the inherent characteristics of the diffusion model, nor does it adequately consider the similarity structure in the latent space and the characteristics of human visual perception, thus reducing the robustness of the watermark and causing degradation in the generation quality.

[0004] Therefore, how to design an endogenous watermarking method based on the diffusion model without compromising the model's generation capabilities, so that the watermark is implicitly injected into the image content without damaging the image quality, has become one of the urgent problems to be solved in diffusion model endogenous watermarking. Summary of the Invention

[0005] Embodiments of the present invention provide a latent space consistency low-rank adaptive watermarking method based on a large model of text-generated images, so as to effectively improve the quality of generated images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] A latent space consistency low-rank adaptive watermarking method based on a large model of text-generated graphs includes:

[0008] The original watermark is encoded into a watermark latent representation by a watermark encoder component, the carrier image is encoded into a content latent representation by an encoder, and the watermark latent representation and the content latent representation are synchronously subjected to discrete wavelet transform by a latent space consistency embedding component to obtain a watermark-containing latent representation.

[0009] The watermark feature modulation denoising network is obtained by adjusting the parameters of the denoising network through the watermark feature modulation network.

[0010] The watermarked latent representation is forward diffused to obtain the watermarked initial noise. The watermarked initial noise is then denoised by a watermark feature modulation denoising network to obtain the watermarked image. The decoded watermark is then extracted from the watermarked image by a watermark decoder component.

[0011] If the original watermark and the decoded watermark are verified to be consistent, then it is determined that the watermarked image and the carrier image are of the same origin.

[0012] Preferably, the process of encoding the original watermark into a watermark latent representation using a watermark encoder component, encoding the carrier image into a content latent representation using an encoder, and synchronously performing discrete wavelet transform on the watermark latent representation and the content latent representation using a latent space consistency embedding component to obtain a watermark-containing latent representation includes:

[0013] The original watermark s with a capacity of n bits is encoded into a watermark latent representation using a watermark encoder. The carrier image I of size M×M is encoded into a content latent representation by an encoder. :

[0014]

[0015]

[0016] Through consistency embedding components and Discrete wavelet transforms are performed separately, and directional energy-sensing weighted fusion processing is applied in the mid-to-low frequency range to obtain the mid-to-low frequency sub-band. :

[0017]

[0018] in, This indicates that a discrete wavelet transform is being performed. For mid-to-low frequency sub-bands, i represents the sub-band layer. Represented as an intensity control factor, for mid-to-low frequency subbands Perform inverse discrete wavelet transform to obtain the latent representation of the watermark. :

[0019]

[0020] in, This is the inverse discrete wavelet transform.

[0021] Preferably, the step of adjusting the parameters of the denoising network through the watermark feature modulation network to obtain the watermark feature modulation denoising network includes:

[0022] The original watermark s is mapped using a watermark feature modulation network to obtain the modulated watermark feature matrix. :

[0023]

[0024] in, This represents a watermark feature modulation network that modulates the watermark feature matrix. The size is r×r,

[0025] The modulated watermark feature matrix is ​​then processed by a low-rank adapter. By introducing a diffusion model for denoising, the fine-tuned parameters of the denoising network are obtained. :

[0026]

[0027] Where W represents the original denoising network parameters, A and B are the lower projection matrix and upper projection matrix of the low-rank adapter, and r is the dimension of the low-rank subspace. Indicates the weighting parameter;

[0028] go through After modulation and denoising, a watermark feature modulation and denoising network is obtained.

[0029] Preferably, the step of forward-diffusion to obtain initial watermark noise from the watermarked latent representation, denoising the initial watermark noise through a watermark feature modulation denoising network to obtain a watermarked image, and extracting the decoded watermark from the watermarked image using a watermark decoder component includes:

[0030] The watermarked potential representation The initial noise containing the watermark is obtained after forward diffusion. The initial noise containing the watermark The input to the watermark feature modulation and denoising network contains the initial noise of the watermark. After gradual denoising, the watermark-containing latent representation is obtained. Then, the watermark potential representation The input is fed into the decoder, and the decoder decodes it. Obtain watermarked image ;

[0031]

[0032] in, For decoders;

[0033] The watermarked image The input watermark decoder component, the watermark decoder component from the watermarked image Extract the decoded watermark with a capacity of n bits. ;

[0034]

[0035] in, Indicates the watermark decoder;

[0036] Verify the decoded watermark The consistency between the decoded watermark and the original watermark s is verified if the decoded watermark is true. If the watermark is consistent with the original watermark s, then the watermarked image is determined. It is of the same origin as the carrier image I.

[0037] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention employs a consistency embedding component to perform directional energy-aware weighted fusion in the low-frequency subband of the latent space to generate a watermarked latent representation; subsequently, a watermark low-rank adaptation component modulates and aligns the watermark information to achieve watermark injection into the diffusion model denoising process; a pre-trained watermark encoder component and a consistency embedding component again obtain the watermarked latent representation, and a watermarked image is generated through a diffusion process guided by modulation of the watermark; the watermark decoder component extracts the watermark from the watermarked image. The present invention implements an endogenous watermarking method based on a diffusion model, ensuring the quality of the generated image while maintaining robustness.

[0038] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram illustrating the implementation principle of a low-rank adaptive watermarking method based on a large text-generated graph model, as provided in an embodiment of the present invention.

[0041] Figure 2 The flowchart illustrates a low-rank adaptive watermarking method based on a large text-generated graph model, as provided in this embodiment of the invention.

[0042] Figure 3 This is a subjective comparison result view of a low-rank adaptive watermarking method based on a large text-generated image model, provided in an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0046] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0047] The implementation principle of the low-rank adaptive watermarking method based on latent space consistency of a large text-generated graph model provided by this invention is as follows: Figure 1 As shown, this method can solve the problem of the impact of endogenous watermarking on the semantics of generated images and the resulting degradation of the perceptual quality of generated images caused by existing diffusion models. The processing flow of this method is as follows: Figure 2 As shown, the processing steps include the following:

[0048] Step S10: During the watermark pre-training phase, the watermark encoder component encodes the original watermark into a watermark latent representation, and the encoder encodes the carrier image into a content latent representation.

[0049] By using a latent space consistency embedding component, the latent representation of the watermark and the latent representation of the image are simultaneously subjected to discrete wavelet transform. The watermark subject information is embedded into the low-frequency sub-band, and the watermark embedding intensity of the remaining sub-bands is adaptively adjusted through a directional energy weighting strategy to obtain the watermark-containing latent representation.

[0050] Step S20: In the watermark endogenous adaptation stage, the aforementioned watermark-containing latent representation is forward-dividated to obtain the initial watermark-containing noise. The parameters of the denoising network are adjusted through the watermark feature modulation network to obtain the watermark feature modulation denoising network. After the initial watermark-containing noise is denoised by the watermark feature modulation denoising network, the watermark-containing image is obtained. The decoded watermark is extracted from the watermark-containing image by the watermark decoder component.

[0051] By verifying the consistency between the original watermark and the decoded watermark, the source of the watermarked image can be traced. If the original watermark and the decoded watermark are consistent, it indicates that the watermarked image and the carrier image share the same origin.

[0052] The watermarked latent representation is forward diffused to obtain the watermarked initial noise. This watermarked initial noise is input into the denoising network of the text image large model. The low-rank adaptive watermarking uses the watermark feature modulation network and the low-rank adapter to fine-tune the text image large model denoising network. The watermarked initial noise is denoised to obtain the watermarked image. The watermark decoder component extracts the watermark from the watermarked image.

[0053] The encoder and decoder in the above steps are components of the stable diffusion model.

[0054] Specifically, step S10 includes: in the watermark pre-training stage, through the watermark encoder... The original watermark s with a capacity of n bits (e.g., 48 bits) is encoded into a watermark latent representation. via encoder Encode a carrier image I of size M×M (e.g., 512×512) into a content latent representation. :

[0055]

[0056]

[0057] The improved consistency embedding component will represent the latent representation of the original watermark. The latent representation of the content of carrier image I Fusion is performed at low and mid frequencies to obtain the watermarked potential representation. :

[0058]

[0059] in, (.) represents the latent space consistency fusion module, which is used for... and Discrete wavelet transforms are performed separately, and directional energy-sensing weighted fusion processing is applied in the mid-to-low frequency range to obtain the mid-to-low frequency sub-band. :

[0060]

[0061] in, This indicates that a discrete wavelet transform is being performed. For the mid-to-low frequency sub-band, i indicates that the sub-band layer includes low-frequency approximation coefficients LL, high-frequency horizontal detail coefficients LH, and high-frequency vertical detail coefficients HL. The intensity control factor is then used to process the fused subbands using inverse discrete wavelet transform to obtain the latent representation containing the watermark. :

[0062]

[0063] in, This is the inverse discrete wavelet transform.

[0064] Specifically, step S20 includes:

[0065] In the watermark endogenous adaptation stage, the original watermark s is mapped through a watermark feature modulation network to obtain the modulated watermark feature matrix. :

[0066]

[0067] in, This represents a watermark feature modulation network that modulates the watermark feature matrix. The size is r×r.

[0068] The modulated watermark feature matrix is ​​then processed by a low-rank adapter. Introducing a diffusion model for denoising, and fine-tuning the denoising network parameters, the specific process is as follows:

[0069]

[0070] Where W represents the original denoising network parameters, A and B are the lower projection matrix and upper projection matrix of the low-rank adapter, and r is the dimension of the low-rank subspace. This represents the weight parameter (e.g., 1). This represents the parameters of the denoising network after fine-tuning.

[0071] go through After modulation and denoising, a watermark feature modulation and denoising network is obtained.

[0072] The watermarked latent representation obtained in step S10 above The initial noise containing the watermark is obtained after forward diffusion. The initial noise containing the watermark The input is fed into the watermark feature modulation and denoising network described above. After gradual denoising, the watermark-containing latent representation is obtained. Then The input is fed into the decoder, and the decoder decodes it. Obtain watermarked image .

[0073]

[0074] in, For decoders.

[0075] Watermarked images The input watermark decoder component, the watermark decoder component from the watermarked image Extract the decoded watermark with a capacity of n bits. The above n can be 48;

[0076]

[0077] in, This indicates a watermark decoder.

[0078] Verify the decoded watermark The consistency between the decoded watermark and the original watermark s is verified if the decoded watermark is true. If the watermark is consistent with the original watermark s, then the watermarked image is determined. It is of the same origin as the carrier image I, thereby enabling the tracing of the source of the generated image.

[0079] To demonstrate the effectiveness of this invention, a series of experiments were conducted. The experiments used the commonly used COCO2017 dataset to train the watermarking model, including a watermark encoder and decoder. The watermark feature modulation network and low-rank adaptation component were fine-tuned using the Gustavosta-sample dataset. After preprocessing, 10,000 color images with a resolution of 512×512 from the COCO2017 dataset and 10,000 color images with a resolution of 512×512 from the Gustavosta-sample dataset were actually used. Both the training and inference phases of this experiment were completed on an NVIDIA RTX 4090 GPU.

[0080] To evaluate the effectiveness and overall performance of this invention, for 48-bit watermarked images, the FID metric was used to measure the distribution difference between the generated image and the real image; the CLIP (Contrastive Language-Image Pretraining) score was used to measure the semantic consistency between the generated content and the text prompts; and the DreamSim metric was used to measure the similarity between images. The extraction accuracy (ACC) metric was used to measure the robustness of the watermark on clean images and degraded images subjected to various attack scenarios. First, watermarks were added to clean images using different watermarking methods. Second, various common image processing operations were performed on the watermarked images. The applied distortions included Gaussian blur, color jitter, JPEG compression, Gaussian noise, and two forms of diffusion-based model-based reconstruction degradation (model versions SD1.5 and SD2.1). The images were degraded using different versions of the stable diffusion model through noise addition and denoising processes to simulate real-world scenarios of AI-based downstream tasks that could compromise the integrity of the watermark.

[0081] According to the objective results shown in Table 1, the FID (Fréchet Inception Distance) value of this invention is 24.81, the CLIP score is 0.3352, and the DreamSim value is as low as 0.1706. According to the objective results shown in Table 2, the extraction accuracy of this invention reaches above 0.9 under clean images and various image degradation conditions. The results indicate that this invention can effectively maintain the robustness of watermark embedding, ensure the overall quality of the generated image, and fully meet the requirements of endogenous watermarking for image quality and watermark robustness in source tracing scenarios.

[0082] Table 1. Objective results of image quality in this invention.

[0083] FID↓CLIP score↑ 24.810.3352 DreamSim↓ 0.1706

[0084] Table 2 Objective results of the extraction quality of this invention

[0085] ACC 0.9933 0.9913 0.9878 0.9835 0.9742 0.9535 0.9469

[0086] Figure 3 This is a subjective comparison result view of a low-rank adaptive watermarking method based on a large text-generated graph model provided in an embodiment of the present invention, such as... Figure 3As shown, under the same diffusion model and prompt text, the watermark-free image generated by the stable diffusion model, the watermarked image generated by existing comparison methods, and the watermarked image generated by this invention are compared. Compared with existing comparison methods, the watermarked image generated by this invention is still very close to the watermark-free image in terms of semantics and structure. Therefore, this invention can maintain the robustness of the endogenous watermark while ensuring good generation quality of the diffusion model and the generated image, verifying the effectiveness of the low-rank adaptive watermarking based on the text-generated image large model of this invention.

[0087] In summary, the embodiments of the present invention first combine the latent space consistency embedding component to generate a watermarked latent representation; then combine the watermark low-rank adaptation component to realize watermark injection in the diffusion model denoising process, so as to realize the endogenous watermark of the diffusion model while ensuring the quality of the generated image.

[0088] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0089] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0090] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0091] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A latent space consistency low-rank adaptive watermarking method based on a large model of text-generated graphs, characterized in that, include: The original watermark is encoded into a watermark latent representation by a watermark encoder component, the carrier image is encoded into a content latent representation by an encoder, and the watermark latent representation and the content latent representation are synchronously subjected to discrete wavelet transform by a latent space consistency embedding component to obtain a watermark-containing latent representation. The watermark feature modulation denoising network is obtained by adjusting the parameters of the denoising network through the watermark feature modulation network. The watermarked latent representation is forward diffused to obtain the watermarked initial noise. The watermarked initial noise is then denoised by a watermark feature modulation denoising network to obtain the watermarked image. The decoded watermark is then extracted from the watermarked image by a watermark decoder component. If the original watermark and the decoded watermark are verified to be consistent, then it is determined that the watermarked image and the carrier image are of the same origin.

2. The method according to claim 1, characterized in that, The process involves encoding the original watermark into a watermark latent representation using a watermark encoder component, encoding the carrier image into a content latent representation using an encoder, and simultaneously performing discrete wavelet transform on the watermark latent representation and the content latent representation using a latent space consistency embedding component to obtain the watermark-containing latent representation. This includes: The original watermark s with a capacity of n bits is encoded into a watermark latent representation using a watermark encoder. The carrier image I of size M×M is encoded into a content latent representation by an encoder. : Through consistency embedding components and Discrete wavelet transforms are performed separately, and directional energy-sensing weighted fusion processing is applied in the mid-to-low frequency range to obtain the mid-to-low frequency sub-band. : in, This indicates that a discrete wavelet transform is being performed. For mid-to-low frequency sub-bands, i represents the sub-band layer. Represented as an intensity control factor, for mid-to-low frequency subbands Perform inverse discrete wavelet transform to obtain the latent representation of the watermark. : in, This is the inverse discrete wavelet transform.

3. The method according to claim 2, characterized in that, The method of adjusting the parameters of the denoising network through the watermark feature modulation network to obtain the watermark feature modulation denoising network includes: The original watermark s is mapped using a watermark feature modulation network to obtain the modulated watermark feature matrix. : in, This represents a watermark feature modulation network that modulates the watermark feature matrix. The size is r×r, The modulated watermark feature matrix is ​​then processed by a low-rank adapter. By introducing a diffusion model for denoising, the fine-tuned parameters of the denoising network are obtained. : Where W represents the original denoising network parameters, A and B are the lower projection matrix and upper projection matrix of the low-rank adapter, and r is the dimension of the low-rank subspace. Indicates the weighting parameter; go through After modulation and denoising, a watermark feature modulation and denoising network is obtained.

4. The method according to claim 3, characterized in that, The process of forward-diffusion to obtain initial watermark noise from the latent watermark representation, denoising the initial watermark noise using a watermark feature modulation denoising network to obtain a watermarked image, and extracting the decoded watermark from the watermarked image using a watermark decoder component includes: The watermarked potential representation The initial noise containing the watermark is obtained after forward diffusion. The initial noise containing the watermark The input to the watermark feature modulation and denoising network contains the initial noise of the watermark. The watermark-containing latent representation was obtained through gradual noise reduction. Then, the watermark potential representation The input is fed into the decoder, and the decoder decodes it. Obtain watermarked image ; in, For decoders; The watermarked image The input watermark decoder component, the watermark decoder component from the watermarked image Extract the decoded watermark with a capacity of n bits. ; in, Indicates the watermark decoder; Verify the decoded watermark The consistency between the decoded watermark and the original watermark s, if the decoded watermark is proven... If the watermark is consistent with the original watermark s, then the watermarked image is determined. It is of the same origin as the carrier image I.

Images