Watermark generation method and device based on diffusion model, terminal equipment and medium
By modifying the noise addition method to the watermark noise sampling strategy during the image generation process of the diffusion model, the problem of low watermark addition performance when generating visual images of the diffusion model is solved, and high-quality watermark embedding and copyright protection are achieved.
Patent Information
- Application Number
- CN202411466631.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-18
AI Technical Summary
When generating visual images based on diffusion models, the watermark addition performance in the prior art is low, which may lead to a degradation of image quality.
By modifying the noise addition method of the diffusion model during image generation, the watermark noise sampling strategy is added by itself without fine-tuning the diffusion model.
It realizes the effective addition of watermarks to the visual image while maintaining the quality of image generation, enhances the concealment and robustness of the watermarks, and supports the copyright protection of the target watermark image corresponding to the target text information.
Smart Images
Figure CN119477653A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image generation, and in particular to a watermark generation method, device, terminal equipment and medium based on a diffusion model. Background Art
[0002] In the financial field, using artificial intelligence to generate visual images based on real text information can significantly improve the intuitiveness and readability of real text information. This technology enables complex financial data, insurance terms or market trends to be clearly communicated through images, etc., helping customers and decision makers to quickly understand key information. However, in order to prevent these visual images from being misappropriated by others and to protect the legitimate rights and interests of the company, it is crucial to embed watermarks in the process of generating images.
[0003] In current watermark generation technology, when using a diffusion model, additional training data is usually required to fine-tune the original model. This process is not only time-consuming, but may also affect the overall performance of the image generated by the diffusion model, leading to a decrease in the quality of the final visualized image. Therefore, how to efficiently embed watermarks while maintaining the quality of the generated image is a challenge that needs to be urgently addressed in the financial field. Summary of the invention
[0004] The main purpose of the embodiments of the present invention is to provide a watermark generation method, apparatus, terminal device and medium based on a diffusion model, aiming to solve the problem in the related art that the performance of adding watermarks in the process of generating visual images based on a diffusion model is low and may cause the quality of the visual images to deteriorate.
[0005] In a first aspect, an embodiment of the present invention provides a watermark generation method based on a diffusion model, comprising:
[0006] Obtaining target text information, and performing text encoding on the target text information to obtain a corresponding text encoding vector;
[0007] Generate a corresponding image encoding vector using a multimodal pre-trained model according to the text encoding vector;
[0008] Determine the watermark area corresponding to the watermark and the watermark noise sampling strategy corresponding to the watermark;
[0009] Image generation is performed using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
[0010] In a second aspect, an embodiment of the present invention provides a watermark generation device based on a diffusion model, comprising:
[0011] A text encoding module, used to obtain target text information, and perform text encoding on the target text information to obtain a corresponding text encoding vector;
[0012] A coding conversion module, used to generate a corresponding image coding vector according to the text coding vector using a multimodal pre-training model;
[0013] A data determination module, used to determine a watermark area corresponding to adding a watermark and a watermark noise sampling strategy corresponding to adding a watermark;
[0014] The watermark generation module is used to generate an image using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
[0015] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the watermark generation methods based on the diffusion model provided in the specification of the present invention are realized.
[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any diffusion model-based watermark generation method provided in the specification of the present invention.
[0017] The embodiment of the present invention provides a watermark generation method, device, terminal device and medium based on a diffusion model, the method comprising: obtaining target text information, and performing text encoding on the target text information to obtain a corresponding text encoding vector; generating a corresponding image encoding vector using a multimodal pre-training model according to the text encoding vector; determining a watermark region corresponding to adding a watermark and a watermark noise sampling strategy corresponding to adding a watermark; performing image generation using a diffusion model according to the watermark noise sampling strategy, the watermark region and the image encoding vector, and obtaining a target watermark image corresponding to the target text information. In this method, when an image corresponding to the target text information is generated based on a diffusion model, the noise addition method corresponding to the watermark region in the diffusion model during the image generation process can be modified to a watermark noise sampling strategy, without fine-tuning the diffusion model itself, so that a watermark can be added automatically during the image generation process. This not only meets the quality requirements of the generated image, but also can effectively add a watermark to the image generated based on the target text information. This solves the problem of low watermark adding performance in related technologies when generating visual images based on diffusion models, as well as the problem of visual image quality degradation caused by improper watermark adding. This method can effectively embed watermarks while maintaining the quality of image generation, providing strong support for the copyright protection of the target watermark image corresponding to the target text information. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 A schematic diagram of a flow chart of a watermark generation method based on a diffusion model provided by an embodiment of the present invention;
[0020] Figure 2 for Figure 1 A schematic flow chart of sub-step S103 of the watermark generation method based on the diffusion model;
[0021] Figure 3 A schematic diagram of a scenario flow of a watermark generation method based on a diffusion model provided by an embodiment of the present invention;
[0022] Figure 4 A schematic diagram of the module structure of a watermark generation device based on a diffusion model provided by an embodiment of the present invention;
[0023] Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] The embodiment of the present invention provides a watermark generation method, device, terminal device and medium based on a diffusion model. The watermark generation method based on a diffusion model can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The terminal device can be a server or a server cluster.
[0028] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0029] Please refer to Figure 1 , Figure 1 A schematic flow chart of a watermark generation method based on a diffusion model provided in an embodiment of the present invention.
[0030] like Figure 1 As shown, the watermark generation method based on the diffusion model includes steps S101 to S104.
[0031] Step S101: Obtain target text information, and perform text encoding on the target text information to obtain a corresponding text encoding vector.
[0032] Exemplarily, the target text information may be a description text for which a visual image needs to be generated in the financial field, or may be a description text for which a visual image needs to be generated in the insurance field.
[0033] For example, in the claims process in the insurance field, it may be necessary to generate a simulated image of the accident scene and then determine the description text of the accident scene as the target text information.
[0034] Exemplarily, the target text information is encoded according to a text encoding model to obtain a corresponding text encoding vector. The text encoding model may be TextEncoder, Word2Vec, sentence embedding model, or other pre-trained models in natural language processing.
[0035] It should be noted that this application does not impose any specific restrictions on the model structure of the text encoding model, and you can choose it according to actual needs.
[0036] Step S102: Generate a corresponding image coding vector based on the text coding vector using a multimodal pre-training model.
[0037] Exemplarily, the multimodal pre-trained model is able to process text data and image data, and map the text data and image data to the same semantic space, and then the multimodal pre-trained model will generate image features that match the text encoding vector based on the text encoding vector. Therefore, when processing text encoding vectors, the multimodal pre-trained model will usually generate image encoding vectors through different levels of feature extraction and fusion strategies.
[0038] It should be noted that the multimodal pre-training model can be CLIP (Contrastive Language Image Pretraining), DALL-E, ALIGN, etc. This application does not impose specific restrictions and can be selected according to actual needs.
[0039] Step S103: determine the watermark area corresponding to the watermark and the watermark noise sampling strategy corresponding to the watermark.
[0040] Exemplarily, the watermark area is a position where a watermark needs to be added in a visualized image when the visualized image is generated according to the target text information.
[0041] Exemplarily, the watermark area may be one or more continuous areas specified in the visualized image, or may be one or more discrete areas specified in the visualized image.
[0042] For example, in the process of generating a visual image and adding a watermark based on the target text information based on the diffusion model, the present application does not fine-tune the diffusion model, but modifies the noise adding method in the process of generating the visual image for the watermark area to be watermarked. Thus, the watermark noise sampling strategy corresponding to the watermarking is determined according to the needs or expert experience.
[0043] For example, when generating a visual image based on the target text information based on the diffusion model, the method of adding noise is random noise. However, in the present application, when generating the corresponding watermark area in the visual image based on the target text information, the method of adding noise is modified to noise based on uniform distribution, that is, the watermark noise sampling strategy is a noise sampling strategy based on uniform distribution.
[0044] In one embodiment, the watermark area corresponding to the watermark is determined, specifically, referring to Figure 2 , step S103 includes: sub-step S1031 to sub-step S1032.
[0045] Sub-step S1031, determining the watermark adding period and the corresponding starting position when adding the watermark.
[0046] Exemplarily, the watermark adding period is the interval at which the watermark appears when generating the target watermark image, and the starting position is the position where the watermark appears for the first time when generating the target watermark image. The watermark adding period can be selected as any positive integer that does not exceed the size of the target watermark image, and is generally selected as 2, 3, 4, etc.
[0047] Sub-step S1032: performing area selection in the target watermark image according to the starting position and the watermark adding period to obtain the watermark area corresponding to the watermark adding.
[0048] Exemplarily, the starting position is determined to be a position corresponding to the target watermark image, and then the corresponding position is also determined as the watermark area after each watermark adding cycle from the starting position.
[0049] For example, if the target watermark image is an N*N image, each row of the target watermark image is concatenated to obtain a 1*(N*N) image vector, so that after determining the position corresponding to the start position from the image vector, the corresponding position is also determined as the watermark area after each watermark adding cycle. For example, if the start position is the first pixel position in the target watermark image, the watermark adding cycle is 3, and the target watermark image is a 4*4 image, then after concatenating each row of the target watermark image, a 1*16 image vector can be obtained, and the start position is the first pixel position in the target watermark image, that is, the first position in the 1*16 image vector is the first watermark adding position, and after the watermark adding cycle is 3, the fifth position is the second watermark adding position, and the second watermark adding position is further separated by 3 watermark adding cycles to obtain the ninth position as the third watermark adding position, and so on, all the obtained watermark adding positions are converted to the target watermark image, and the result after converting the watermark adding position to the target watermark image is determined as the watermark area.
[0050] Specifically, by setting the watermark adding period and starting position, the distribution of the watermark can be better controlled, making it difficult to be detected in the image, thereby enhancing the concealment of the watermark. It also helps the systematic management of watermarks and facilitates the tracking and management of watermarks in the later stage.
[0051] In some embodiments, determining the watermark adding period includes: determining an initial adding period and initial text information; generating an initial generated image corresponding to the initial text information according to the diffusion model, and generating an initial watermark image corresponding to the initial text information according to the initial adding period and the diffusion model; calculating the image quality difference between the initial generated image and the initial watermark image, and adjusting the initial adding period according to the image quality difference until the watermark adding period is obtained.
[0052] Exemplarily, the initial adding period can be selected as any positive integer that does not exceed the target watermark image size, and is generally selected as 2, 3, 4, etc.
[0053] Exemplarily, an initial generated image corresponding to the initial text information is generated according to the diffusion model, and then the initial watermark adding area is determined according to the starting position and the initial adding period, and then the initial text information is encoded according to the text encoding model to obtain the corresponding text encoding result, and then the multimodal pre-training model is used to perform data conversion on the text encoding result to obtain the corresponding image encoding result. Thus, the noise adding method at the position corresponding to the initial watermark adding area in the image encoding result is modified to the watermark noise sampling strategy, and the noise adding method at the position not corresponding to the initial watermark adding area in the image encoding result remains unchanged, so as to generate the initial watermark image corresponding to the initial text information based on the diffusion model using the above noise adding method.
[0054] Exemplarily, the quality evaluation indexes include peak signal-to-noise ratio, structural similarity index, mean square error, etc. Thus, the difference between different quality evaluation indicators of the initial generated image and the initial watermark image is calculated, and the corresponding difference is determined as the image quality difference.
[0055] For example, the square of the difference of each pixel in the initial generated image is calculated, and then the average value is calculated to obtain the first mean square error corresponding to the initial generated image, and the square of the difference of each pixel in the initial watermark image is calculated and then the average value is calculated to obtain the second mean square error corresponding to the initial watermark image, and then the difference between the first mean square error and the second mean square error is calculated and determined as the image quality difference.
[0056] Exemplarily, when the image quality difference is less than or equal to the preset difference, the initial adding period is determined to be the watermark adding period; if the image quality difference is greater than the preset difference, the initial adding period is reduced, and a new initial watermark image corresponding to the initial text information is regenerated according to the reduced initial adding period. Next, the image quality difference between the new initial watermark image and the initially generated image is calculated, and this process is repeated until the image quality difference is less than or equal to the preset difference, at which time the reduced initial adding period is determined to be the watermark adding period.
[0057] Specifically, by selecting the watermark adding period based on the image quality difference method, the image generation quality can be ensured while effectively meeting the visual image watermarking requirements, providing a reliable solution for achieving high-quality image generation and effective watermark protection.
[0058] Step S104: Generate an image using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
[0059] Exemplarily, the noise adding method at the position corresponding to the watermark area in the image coding vector is modified to the watermark noise sampling strategy, and the noise adding method at the position not corresponding to the watermark area in the image coding result remains unchanged, thereby using the above-mentioned noise adding method to generate a target watermark image corresponding to the target text information based on the diffusion model.
[0060] In some embodiments, the method of generating an image using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information includes: determining the target noise sampling strategy corresponding to the noise processing of the image coding vector in the diffusion model; identifying the watermark addition position in the image coding vector according to the watermark area to obtain the target watermark position corresponding to the image coding vector; adjusting the noise addition method corresponding to the target watermark position in the image coding vector from the target noise sampling strategy to the watermark noise sampling strategy; performing noise processing on the image coding vector according to the watermark noise sampling strategy and the target noise sampling strategy to obtain a target coding vector; and generating an image of the target coding vector using the diffusion model to obtain the target watermark image corresponding to the target text information.
[0061] Exemplarily, a target noise sampling strategy corresponding to noise processing performed on an image coding vector in a diffusion model is determined. For example, the target noise sampling strategy is a noise sampling strategy that conforms to a normal distribution.
[0062] Exemplarily, the watermark adding position in the image coding vector is identified according to the watermark area to obtain the target watermark position corresponding to the image coding vector. For example, the watermark area includes (x1, y1), (x2, y2), (x3, y3), etc., then the position corresponding to the y1 position in the x1th coding vector in the image coding vector is determined as the target watermark position; the position corresponding to the y2 position in the x2th coding vector in the image coding vector is determined as the target watermark position; the position corresponding to the y3 position in the x3th coding vector in the image coding vector is determined as the target watermark position.
[0063] Exemplarily, the noise adding method corresponding to the target watermark position in the image coding vector is adjusted from the target noise sampling strategy to the watermark noise sampling strategy. For example, if the target noise sampling strategy is a random sampling strategy and the watermark noise sampling strategy is a normal distribution noise sampling strategy, then the noise adding method corresponding to the target watermark position in the image coding vector is a normal distribution noise sampling strategy, and the noise adding methods of other positions in the image coding vector except the target watermark position are still the normal distribution noise sampling strategy.
[0064] Exemplarily, the noise adding method corresponding to the target watermark position in the image coding vector is the watermark noise sampling strategy, and the noise adding method at other positions in the image coding vector except the target watermark position is still the target noise sampling strategy, so that the image coding vector is noise processed according to the method of adding noise at different positions in the image coding vector to obtain the target coding vector.
[0065] Exemplarily, the diffusion model is used to generate an image of the target coding vector to obtain a target watermark image corresponding to the target text information.
[0066] For example, Figure 3 As shown in the figure, first the target text information is passed through TextEncoder to obtain a text encoding vector, then the text encoding vector is converted into an image encoding vector through CLIP, and the image encoding vector is processed with noise according to the watermark noise sampling strategy and the target noise sampling strategy, so that Classifier-free is used to accelerate the generation of a high-resolution target watermark image. The formula is expressed as follows:
[0067] x t-1 =Denoise(∈(x t , t), x t , accelerator(y, x t ))+σ t ∈
[0068] Denois e represents the function of three variables, which is directly determined by the design of the diffusion model, ∈(x t, t) represents the original noise of the prediction (obtained by training the neural network) and x t represents the input of step t, accelerator(y, x t ) represents the accelerated generation part selected and adopted, which is also directly determined by the design scheme, and y represents the target text information. t ∈ is the noise term, where σ t It is directly determined by the design of the diffusion model (some generative models are also obtained by iterative optimization), ∈ is a standard normal distribution, and the noise term sampling method is modified by random sampling to select a new The noise addition method of ∈ becomes uniformly distributed according to U(0,1)
[0069] In some embodiments, after obtaining the target watermark image corresponding to the target text information, the method further includes: obtaining an image recognition model and determining the target resolution required for the input image when the image recognition model performs recognition; adjusting the target watermark image according to the target resolution to obtain a target input image; using the image recognition model to identify whether a watermark is added to the target input image, and obtaining a target recognition result corresponding to the target input image.
[0070] For example, since the watermark added to the target watermark image is an implicit watermark, the naked eye cannot directly see whether there is a watermark in the target watermark image. Therefore, it is necessary to use an image recognition model to recognize the target watermark image to determine whether a watermark is really added to the target watermark image.
[0071] Exemplarily, the resolution of the training data when the image recognition model is used for model training is used, and then the resolution corresponding to the training data is determined as the target resolution.
[0072] Exemplarily, according to the determined target resolution, the target watermark image is adjusted to obtain the target input image. For example, when the resolution of the target watermark image is greater than the target resolution, the target watermark image needs to be cropped to obtain the target input image. If the resolution of the target watermark image is less than the target resolution, the target watermark image needs to be expanded to obtain the target input image.
[0073] For example, when the resolution of the target watermark image is lower than the expected target resolution, the target watermark image needs to be pixel-expanded to adjust it to the desired target resolution. In this process, each pixel of the target watermark image will be expanded to the corresponding new position, and the unfilled area will be filled with blank pixels with a pixel value of 0, thereby generating a target input image that meets the target resolution.
[0074] For example, the original size of the target watermark image is 640x480 pixels, and the required target resolution is 1280x960 pixels. In this case, the target watermark image needs to be expanded on the target canvas in a certain order, for example, the target watermark image is placed in the upper left corner or upper right corner of the target canvas, and then all canvas areas not filled with the target watermark image are filled with pixels with a pixel value of 0, so as to obtain a target input image that meets the target resolution.
[0075] Exemplarily, the target input image is input into the image recognition model to identify whether a watermark is added, so as to obtain the target distribution information corresponding to the target input image. The target distribution information is a first probability corresponding to adding a watermark and a second probability corresponding to not adding a watermark, so as to determine the label corresponding to the maximum value of the first probability and the second probability as the target recognition result.
[0076] For example, when the first probability is greater than the second probability, the target recognition result is to add a watermark; when the first probability is less than the second probability, the target recognition result is not to add a watermark.
[0077] It should be noted that the image recognition model can be a deep learning model or a machine learning model, such as a support vector machine, random forest, etc. This application does not impose specific restrictions on the model structure of the image recognition model, and you can choose it according to actual needs.
[0078] Specifically, when the target watermark image is generated according to the target text information, the watermark is added in an implicit way, and the naked eye cannot confirm whether there is a watermark in the image. Therefore, the image recognition model can accurately identify whether the target watermark image is watermarked, which provides strong support for subsequent copyright protection.
[0079] In some embodiments, performing image adjustment on the target watermark image according to the target resolution to obtain the target input image includes: extracting data from the target watermark image according to the watermark area to obtain a watermark image area corresponding to the target watermark image; and performing image adjustment on the watermark image area according to the target resolution to obtain the target input image.
[0080] Exemplarily, information about the location of the watermark added in the target watermark image is extracted according to the watermark area, thereby obtaining a watermark image area corresponding to the target watermark image.
[0081] For example, if the watermark area includes (x1, y1), (x2, y2), (x3, y3), etc., the information of the pixel positions (x1, y1), (x2, y2), (x3, y3) in the target watermark image is extracted to obtain the pixel value p1 corresponding to (x1, y1), the pixel value p2 corresponding to (x2, y2), the pixel value p3 corresponding to (x3, y3), etc., so that the extracted information is sequentially combined according to the pixel positions to obtain the watermark image area. Alternatively, the extracted information is sequentially combined according to the acquisition order to obtain the watermark image area.
[0082] Exemplarily, the watermark image region is adjusted according to the target resolution to obtain the target input image.
[0083] For example, the original size of the watermark image area is 10x10 pixels, and the required target resolution is 20x20 pixels. In this case, the watermark image area needs to be expanded on the target canvas in a certain order, for example, the watermark image area is placed in the upper left corner or upper right corner of the target canvas, and then all canvas areas not filled with the watermark image area are filled with pixels with a pixel value of 0, so as to obtain a target input image that meets the target resolution.
[0084] Specifically, according to the above steps, only information that has been watermarked or may have been watermarked can be identified, thereby providing support for subsequently improving the accuracy of target identification results.
[0085] In some embodiments, obtaining an image recognition model includes: obtaining a historical text image and a historical watermark image corresponding to the historical text information, wherein the historical text image is an image generated according to the historical text information but does not contain a watermark, and the historical watermark image is an image containing a watermark generated according to the historical text information, the watermark area, and the watermark noise sampling strategy; cropping the historical text image according to the watermark area to obtain a first image; cropping the historical watermark image according to the watermark area to obtain a second image; adjusting the resolution of the first image according to the target resolution to obtain a first target image; adjusting the resolution of the second image according to the target resolution to obtain a second target image; setting a first label corresponding to the first target image and a second label corresponding to the second target image; determining target training data according to the first target image, the second target image, the first label, and the second label; determining an initial recognition model, and training the initial recognition model with the target training data to obtain the image recognition model.
[0086] Exemplarily, a historical text image and a historical watermark image corresponding to the historical text information are obtained, wherein the historical text image is an image generated based on the historical text information but does not contain a watermark, and the historical watermark image is an image containing a watermark generated based on the historical text information, the watermark area and the watermark noise sampling strategy.
[0087] For example, a historical text image is obtained when the historical text information is directly generated according to the diffusion model without noise modification, and a historical watermark image corresponding to the historical text information is obtained when a watermark is added during the image generation process according to the diffusion model using the watermark area and watermark noise sampling strategy.
[0088] Exemplarily, information of a corresponding position in a historical text image is extracted based on a watermark area to obtain a first image corresponding to the watermark area in the historical text image, and information of a watermarked position in a historical watermark image is extracted based on the watermark area to obtain a second image corresponding to the historical watermark image.
[0089] Exemplarily, the first image is adjusted in resolution according to the target resolution to obtain the first target image; and the second image is adjusted in resolution according to the target resolution to obtain the second target image;
[0090] Exemplarily, a first label corresponding to the first target image and a second label corresponding to the second target image are set; wherein the first label is not watermarked and the second label is watermarked, then the first target image, the second target image, the first label and the second label are determined as target training data.
[0091] Exemplarily, the initial recognition model is a generative adversarial network, and then the initial recognition model is trained using target training data, and cross entropy loss is used as the Loss function for iterative training. When the loss value of the corresponding Loss function under the initial recognition model is less than a preset value, an image recognition model is obtained; when the loss value of the corresponding Loss function under the initial recognition model is greater than or equal to the preset value, the initial recognition model is continued to be trained using the target training data until the corresponding recognition accuracy under the initial recognition model is greater than the preset value, and an image recognition model is obtained.
[0092] Specifically, through adversarial training, the image recognition model can better identify whether a watermark is added and improve its robustness to various interferences, thereby improving the accuracy and reliability of the recognition results of the image recognition model.
[0093] The embodiment of the present invention provides a watermark generation method, device, terminal device and medium based on a diffusion model, the method comprising: obtaining target text information, and performing text encoding on the target text information to obtain a corresponding text encoding vector; generating a corresponding image encoding vector using a multimodal pre-training model according to the text encoding vector; determining a watermark region corresponding to adding a watermark and a watermark noise sampling strategy corresponding to adding a watermark; performing image generation using a diffusion model according to the watermark noise sampling strategy, the watermark region and the image encoding vector, and obtaining a target watermark image corresponding to the target text information. In this method, when an image corresponding to the target text information is generated based on a diffusion model, the noise addition method corresponding to the watermark region in the diffusion model during the image generation process can be modified to a watermark noise sampling strategy, without fine-tuning the diffusion model itself, so that a watermark can be added automatically during the image generation process. This not only meets the quality requirements of the generated image, but also can effectively add a watermark to the image generated based on the target text information. This solves the problem of low watermark adding performance in related technologies when generating visual images based on diffusion models, as well as the problem of visual image quality degradation caused by improper watermark adding. This method can effectively embed watermarks while maintaining the quality of image generation, providing strong support for the copyright protection of the target watermark image corresponding to the target text information.
[0094] That is, the present application adds watermarks by modifying the noise adding method. Compared with the watermark generation method using fine-tuning, the present application reduces the complexity of watermark implementation. Compared with the traditional fine-tuning method, the present application avoids the use of a large amount of data, computing resources and time overhead. Moreover, it eliminates the unpredictable changes that may be introduced by fine-tuning. Furthermore, the present application chooses to modify the noise sampling method for watermark generation, providing a way to embed watermarks at a deeper level, making the watermark more difficult to detect and remove. At the same time, the watermark generation method of the present application also enhances the robustness of the watermark. Finally, the watermark area of the present application to add watermarks allows a variety of customized settings for the watermark, so the watermark generation method of the present application also increases the flexibility of adding watermarks.
[0095] See also Figure 4 , Figure 4A watermark generation device 200 based on a diffusion model is provided in an embodiment of the present application. The watermark generation device 200 based on a diffusion model includes a text encoding module 201, an encoding conversion module 202, a data determination module 203, and a watermark generation module 204, wherein the text encoding module 201 is used to obtain target text information and perform text encoding on the target text information to obtain a corresponding text encoding vector; the encoding conversion module 202 is used to generate a corresponding image encoding vector based on the text encoding vector using a multimodal pre-training model; the data determination module 203 is used to determine a watermark area corresponding to adding a watermark and a watermark noise sampling strategy corresponding to adding a watermark; the watermark generation module 204 is used to perform image generation using a diffusion model based on the watermark noise sampling strategy, the watermark area and the image encoding vector to obtain a target watermark image corresponding to the target text information.
[0096] In some implementations, the data determination module 203 performs the following steps in the process of determining the watermark area corresponding to the watermark to be added:
[0097] Determine the watermark adding period and the corresponding starting position when adding the watermark;
[0098] According to the starting position and the watermark adding period, a region is selected in the target watermark image to obtain the watermark region corresponding to the watermark adding.
[0099] In some implementations, the data determination module 203, during the process of determining the watermark adding period, performs:
[0100] Determine the initial adding cycle and initial text information;
[0101] Generating an initial generated image corresponding to the initial text information according to the diffusion model, and generating an initial watermark image corresponding to the initial text information according to the initial adding period and the diffusion model;
[0102] The image quality difference between the initial generated image and the initial watermark image is calculated, and the initial adding period is adjusted according to the image quality difference until the watermark adding period is obtained.
[0103] In some implementations, the watermark generation module 204, in the process of generating an image using a diffusion model according to the watermark noise sampling strategy, the watermark region, and the image coding vector to obtain a target watermark image corresponding to the target text information, performs:
[0104] Determining a target noise sampling strategy corresponding to noise processing performed on the image coding vector in the diffusion model;
[0105] Identify the watermark adding position in the image coding vector according to the watermark area, and obtain the target watermark position corresponding to the image coding vector;
[0106] Adjusting the noise adding method corresponding to the target watermark position in the image coding vector from the target noise sampling strategy to the watermark noise sampling strategy;
[0107] Performing noise processing on the image coding vector according to the watermark noise sampling strategy and the target noise sampling strategy to obtain a target coding vector;
[0108] The target coding vector is image-generated using the diffusion model to obtain the target watermark image corresponding to the target text information.
[0109] In some implementations, after obtaining the target watermark image corresponding to the target text information, the watermark generation module 204 further performs:
[0110] Obtaining an image recognition model, and determining a target resolution corresponding to an input image when the image recognition model performs recognition;
[0111] Adjusting the target watermark image according to the target resolution to obtain a target input image;
[0112] The image recognition model is used to identify whether the target input image is watermarked, and a target recognition result corresponding to the target input image is obtained.
[0113] In some implementations, the watermark generation module 204, during the process of adjusting the target watermark image according to the target resolution to obtain the target input image, performs:
[0114] Extracting data from the target watermark image according to the watermark area to obtain a watermark image area corresponding to the target watermark image;
[0115] The watermark image region is image adjusted according to the target resolution to obtain the target input image.
[0116] In some implementations, the watermark generation module 204 performs, during the process of obtaining the image recognition model:
[0117] Obtaining a historical text image and a historical watermark image corresponding to the historical text information, wherein the historical text image is an image generated according to the historical text information but does not contain a watermark, and the historical watermark image is an image containing a watermark generated according to the historical text information, the watermark area, and the watermark noise sampling strategy;
[0118] Cutting the historical text image according to the watermark area to obtain a first image;
[0119] Cutting the historical watermark image according to the watermark area to obtain a second image;
[0120] Adjusting the resolution of the first image according to the target resolution to obtain a first target image;
[0121] Adjusting the resolution of the second image according to the target resolution to obtain a second target image;
[0122] Setting a first label corresponding to the first target image and a second label corresponding to the second target image;
[0123] Determine target training data according to the first target image, the second target image, the first label, and the second label;
[0124] An initial recognition model is determined, and the target training data is used to perform model training on the initial recognition model to obtain the image recognition model.
[0125] In some implementations, the watermark generation apparatus 200 based on the diffusion model may be applied to a terminal device.
[0126] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the watermark generation device 200 based on the diffusion model described above can refer to the corresponding process in the aforementioned embodiment of the watermark generation method based on the diffusion model, and will not be repeated here.
[0127] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.
[0128] like Figure 5 As shown, the terminal device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.
[0129] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0130] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0131] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0132] The processor is used to run a computer program stored in the memory, and implement any one of the diffusion model-based watermark generation methods provided by the embodiments of the present invention when executing the computer program.
[0133] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:
[0134] Obtaining target text information, and performing text encoding on the target text information to obtain a corresponding text encoding vector;
[0135] Generate a corresponding image encoding vector using a multimodal pre-trained model according to the text encoding vector;
[0136] Determine the watermark area corresponding to the watermark and the watermark noise sampling strategy corresponding to the watermark;
[0137] Image generation is performed using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
[0138] In some implementations, the processor 301, in the process of determining the watermark area corresponding to adding the watermark, executes:
[0139] Determine the watermark adding period and the corresponding starting position when adding the watermark;
[0140] According to the starting position and the watermark adding period, a region is selected in the target watermark image to obtain the watermark region corresponding to the watermark adding.
[0141] In some implementations, the processor 301, during the process of determining the watermark adding period, executes:
[0142] Determine the initial adding cycle and initial text information;
[0143] Generating an initial generated image corresponding to the initial text information according to the diffusion model, and generating an initial watermark image corresponding to the initial text information according to the initial adding period and the diffusion model;
[0144] The image quality difference between the initial generated image and the initial watermark image is calculated, and the initial adding period is adjusted according to the image quality difference until the watermark adding period is obtained.
[0145] In some implementations, the processor 301, in the process of generating an image using a diffusion model according to the watermark noise sampling strategy, the watermark area, and the image coding vector to obtain a target watermark image corresponding to the target text information, executes:
[0146] Determining a target noise sampling strategy corresponding to noise processing performed on the image coding vector in the diffusion model;
[0147] Identify the watermark adding position in the image coding vector according to the watermark area, and obtain the target watermark position corresponding to the image coding vector;
[0148] Adjusting the noise adding method corresponding to the target watermark position in the image coding vector from the target noise sampling strategy to the watermark noise sampling strategy;
[0149] Performing noise processing on the image coding vector according to the watermark noise sampling strategy and the target noise sampling strategy to obtain a target coding vector;
[0150] The target coding vector is image-generated using the diffusion model to obtain the target watermark image corresponding to the target text information.
[0151] In some implementations, after obtaining the target watermark image corresponding to the target text information, the processor 301 further executes:
[0152] Obtaining an image recognition model, and determining a target resolution corresponding to an input image when the image recognition model performs recognition;
[0153] Adjusting the target watermark image according to the target resolution to obtain a target input image;
[0154] The image recognition model is used to identify whether the target input image is watermarked, and a target recognition result corresponding to the target input image is obtained.
[0155] In some implementations, the processor 301, during the process of adjusting the target watermark image according to the target resolution to obtain the target input image, executes:
[0156] Extracting data from the target watermark image according to the watermark area to obtain a watermark image area corresponding to the target watermark image;
[0157] The watermark image region is image adjusted according to the target resolution to obtain the target input image.
[0158] In some implementations, the processor 301, during the process of obtaining the image recognition model, executes:
[0159] Obtaining a historical text image and a historical watermark image corresponding to the historical text information, wherein the historical text image is an image generated according to the historical text information but does not contain a watermark, and the historical watermark image is an image containing a watermark generated according to the historical text information, the watermark area, and the watermark noise sampling strategy;
[0160] Cutting the historical text image according to the watermark area to obtain a first image;
[0161] Cutting the historical watermark image according to the watermark area to obtain a second image;
[0162] Adjusting the resolution of the first image according to the target resolution to obtain a first target image;
[0163] Adjusting the resolution of the second image according to the target resolution to obtain a second target image;
[0164] Setting a first label corresponding to the first target image and a second label corresponding to the second target image;
[0165] Determine target training data according to the first target image, the second target image, the first label, and the second label;
[0166] An initial recognition model is determined, and the target training data is used to perform model training on the initial recognition model to obtain the image recognition model.
[0167] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned embodiment of the watermark generation method based on the diffusion model, and will not be repeated here.
[0168] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any diffusion model-based watermark generation method provided in the description of the embodiment of the present invention.
[0169] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.
[0170] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0171] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0172] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A watermark generation method based on a diffusion model, characterized in that: The method comprises: Obtaining target text information, and performing text encoding on the target text information to obtain a corresponding text encoding vector; Generate a corresponding image encoding vector using a multimodal pre-trained model according to the text encoding vector; Determine the watermark area corresponding to the watermark and the watermark noise sampling strategy corresponding to the watermark; Image generation is performed using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
2. The method according to claim 1, characterized in that Determine the watermark area corresponding to the watermark to be added, including: Determine the watermark adding period and the corresponding starting position when adding the watermark; According to the starting position and the watermark adding period, a region is selected in the target watermark image to obtain the watermark region corresponding to the watermark adding.
3. The method according to claim 2, characterized in that The step of determining a watermark adding period includes: Determine the initial adding cycle and initial text information; Generating an initial generated image corresponding to the initial text information according to the diffusion model, and generating an initial watermark image corresponding to the initial text information according to the initial adding period and the diffusion model; The image quality difference between the initial generated image and the initial watermark image is calculated, and the initial adding period is adjusted according to the image quality difference until the watermark adding period is obtained.
4. The method according to claim 1, characterized in that: The step of generating an image by using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information includes: Determining a target noise sampling strategy corresponding to noise processing performed on the image coding vector in the diffusion model; Identify the watermark adding position in the image coding vector according to the watermark area, and obtain the target watermark position corresponding to the image coding vector; Adjusting the noise adding method corresponding to the target watermark position in the image coding vector from the target noise sampling strategy to the watermark noise sampling strategy; Performing noise processing on the image coding vector according to the watermark noise sampling strategy and the target noise sampling strategy to obtain a target coding vector; The target coding vector is image-generated using the diffusion model to obtain the target watermark image corresponding to the target text information.
5. The method according to any one of claims 1 to 4, characterized in that After obtaining the target watermark image corresponding to the target text information, the method further includes: Obtaining an image recognition model, and determining a target resolution corresponding to an input image when the image recognition model performs recognition; Adjusting the target watermark image according to the target resolution to obtain a target input image; The image recognition model is used to identify whether the target input image is watermarked, and a target recognition result corresponding to the target input image is obtained.
6. The method according to claim 5, characterized in that The step of adjusting the target watermark image according to the target resolution to obtain a target input image comprises: Extracting data from the target watermark image according to the watermark area to obtain a watermark image area corresponding to the target watermark image; The watermark image region is image adjusted according to the target resolution to obtain the target input image.
7. The method according to claim 5, characterized in that The obtaining of the image recognition model comprises: Obtaining a historical text image and a historical watermark image corresponding to the historical text information, wherein the historical text image is an image generated according to the historical text information but does not contain a watermark, and the historical watermark image is an image containing a watermark generated according to the historical text information, the watermark area, and the watermark noise sampling strategy; Cutting the historical text image according to the watermark area to obtain a first image; Cutting the historical watermark image according to the watermark area to obtain a second image; Adjusting the resolution of the first image according to the target resolution to obtain a first target image; Adjusting the resolution of the second image according to the target resolution to obtain a second target image; Setting a first label corresponding to the first target image and a second label corresponding to the second target image; Determine target training data according to the first target image, the second target image, the first label, and the second label; An initial recognition model is determined, and the target training data is used to perform model training on the initial recognition model to obtain the image recognition model.
8. A watermark generation device based on a diffusion model, characterized in that: include: A text encoding module, used to obtain target text information, and perform text encoding on the target text information to obtain a corresponding text encoding vector; A coding conversion module, used to generate a corresponding image coding vector using a multimodal pre-training model according to the text coding vector; A data determination module, used to determine a watermark area corresponding to adding a watermark and a watermark noise sampling strategy corresponding to adding a watermark; The watermark generation module is used to generate an image using a diffusion model according to the watermark noise sampling strategy, the watermark area and the image coding vector to obtain a target watermark image corresponding to the target text information.
9. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the watermark generation method based on the diffusion model according to any one of claims 1 to 7 when executing the computer program.
10. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the watermark generation method based on the diffusion model according to any one of claims 1 to 7.