Watermark information processing method, model training method and related equipment products

Abstract

The present application provides a watermark information processing method, a model training method and related equipment products, wherein the method includes: in response to a content generation instruction, obtaining a prompt word corresponding to the generation instruction; inputting the prompt word into a content generation model, so that the content generation model generates target digital media information according to the prompt word, wherein the target digital media information carries a preset watermark embedded by a target decoder of the content generation model; wherein the target decoder is obtained by fine-tuning and training the original decoder of the content generation model based on a preset editing layer and a watermark detector. The present application realizes the guarantee that the watermarked digital media information can still be accurately detected after a preset editing operation, thereby improving the resistance of the digital watermark to complex editing operations.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a watermark information processing method, a model training method, and related equipment products. Background Art

[0002] Digital watermarking is a technology used to protect digital media content (such as images, audio, video or text). It embeds invisible or imperceptible identification information in the original content to achieve copyright protection, content authentication and anti-counterfeiting purposes. Digital watermarking technology can help copyright holders track and identify illegally copied or distributed content.

[0003] Although traditional digital watermarking technology can achieve copyright protection and tampering detection of images to a certain extent, the effectiveness of image watermarking is severely tested when facing complex editing attacks. Traditional watermarking technology is difficult to adapt to high-intensity image transformation. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a watermark information processing method, a model training method and related equipment products, which ensure that the digital media information with a watermark can still be accurately detected after a preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, thereby providing a reliable data basis for the subsequent protection of the ownership of digital media information.

[0005] In a first aspect, an embodiment of the present application provides a watermark information processing method, comprising: in response to a content generation instruction, obtaining a prompt word corresponding to the generation instruction; inputting the prompt word into a content generation model so that the content generation model generates target digital media information according to the prompt word, and the target digital media information carries a preset watermark embedded by a target decoder of the content generation model; wherein the target decoder is obtained by fine-tuning an original decoder of the content generation model based on a preset editing layer and a watermark detector, and during the fine-tuning training process, the preset editing layer is used to perform a preset editing operation on the output information with a preset watermark output by the original decoder, and the watermark detector is used to extract the watermark information in the output information after the preset editing operation.

[0006] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the output information.

[0007] In one embodiment, the process of embedding a preset watermark through the target decoder of the content generation model includes: obtaining the denoised target information features obtained after the content generation model denoises the random noise according to the prompt word; reconstructing the information according to the target information features through the target decoder to generate initial digital media information, and embedding the preset watermark in the initial digital media information to obtain the target digital media information.

[0008] In one embodiment, before the prompt word is input into the content generation model so that the content generation model generates the target digital media information according to the prompt word, it also includes: counting the watermark extraction distribution information of the watermark detector on the sample digital media information, in which there is no watermark information; determining the bias feature of the watermark detector according to the distribution information, and determining the preset watermark according to the bias feature.

[0009] In one embodiment, the method further includes: in response to an ownership detection instruction for designated digital media information, using the watermark detector to extract a target watermark in the designated digital media information; comparing the target watermark with the preset watermark, and determining the ownership information of the designated digital media information according to the comparison result.

[0010] In a second aspect, an embodiment of the present application provides a method for fine-tuning a content generation model, comprising: determining an original decoder and a preset editing layer of a pre-trained content generation model; generating first digital media information with a preset watermark through the original decoder according to sample prompt words and random noise of the content generation model; generating second digital media information after performing a preset editing operation on the first digital media information through the preset editing layer; extracting a watermark in the second digital media information through a pre-trained watermark detector, and fine-tuning the original decoder of the content generation model with the goal of minimizing a first loss function between the extracted watermark and the preset watermark to obtain a fine-tuned target decoder.

[0011] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the first digital media information.

[0012] In one embodiment, the step of generating the first digital media information with a preset watermark through the original decoder according to the sample prompt words and random noise of the content generation model includes:

[0013] Obtaining sample prompt words and random noise of the content generation model;

[0014] De-noising the random noise according to the sample prompt word to obtain the sample information feature after de-noising;

[0015] The sample information feature is input into the original decoder, so that the original decoder generates first digital media information with a preset watermark according to the sample information feature.

[0016] In one embodiment, the method further includes: calculating a second loss function between the second digital media information and the first digital media information, and fine-tuning the original decoder with the goal of minimizing the first loss function and the second loss function to obtain a fine-tuned target decoder.

[0017] In one embodiment, the method further includes: fine-tuning the pre-trained watermark detector with the goal of minimizing the first loss function and the second loss function to obtain a fine-tuned watermark detector.

[0018] In one embodiment, the process of determining the pre-trained watermark detector includes: obtaining carrier information and sample watermarks; training a preset watermark model using the carrier information and the sample watermark to obtain a trained watermark model; and determining the watermark detector of the trained watermark model as the pre-trained watermark detector.

[0019] In a third aspect, an embodiment of the present application provides a watermark information processing device, including:

[0020] An acquisition module, configured to acquire, in response to a content generation instruction, a prompt word corresponding to the generation instruction;

[0021] A generation module, used for inputting the prompt word into a content generation model, so that the content generation model generates target digital media information according to the prompt word, wherein the target digital media information carries a preset watermark embedded by a target decoder of the content generation model;

[0022] Among them, the target decoder is obtained by fine-tuning the original decoder of the content generation model based on the preset editing layer and the watermark detector. During the fine-tuning training process, the preset editing layer is used to perform a preset editing operation on the output information with a preset watermark output by the original decoder, and the watermark detector is used to extract the watermark information in the output information after the preset editing operation.

[0023] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the output information.

[0024] In one embodiment, the generating module is used to reconstruct information according to the target information characteristics through a target decoder to generate initial digital media information, and embed the preset watermark in the initial digital media information to obtain the target digital media information.

[0025] In one embodiment, it also includes: a determination module, which is used to count the watermark extraction distribution information of the watermark detector on the sample digital media information before inputting the prompt word into the content generation model so that the content generation model generates the target digital media information according to the prompt word, and there is no watermark information in the sample digital media information; determine the bias characteristics of the watermark detector according to the distribution information, and determine the preset watermark according to the bias characteristics.

[0026] In one embodiment, it also includes: a detection module, which is used to respond to an ownership detection instruction for specified digital media information, use the watermark detector to extract the target watermark in the specified digital media information; compare the target watermark with the preset watermark, and determine the ownership information of the specified digital media information based on the comparison result.

[0027] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0028] at least one processor; and

[0029] a memory communicatively coupled to the at least one processor;

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0031] In a fifth aspect, an embodiment of the present application provides a cloud device, including:

[0032] at least one processor; and

[0033] a memory communicatively coupled to the at least one processor;

[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of the above aspects.

[0035] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the above aspects is implemented.

[0036] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0037] The watermark information processing method, model training method and related equipment products provided in the embodiments of the present application simulate different editing operations on digital media information by introducing a preset editing layer in the fine-tuning process, extracting the watermark in the output information after the preset editing operation through the watermark detector, and fine-tuning the original decoder of the content generation model based on the preset editing layer and the watermark detector to obtain the target decoder of the content generation model. In the process of generating digital media information, the target digital media information is generated by combining the content generation model with the prompt words, wherein the target decoder of the fine-tuned content generation model can embed the preset watermark in the target digital media information. In this way, by introducing the preset editing layer and the watermark detector in the process of fine-tuning the model, it is ensured that the target digital media information with the watermark can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, providing a reliable data basis for the subsequent protection of digital media information ownership. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0039] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of an application scenario of a watermark information processing system provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a flow chart of a method for fine-tuning a content generation model according to an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a watermark model training provided in an embodiment of the present application;

[0043] Figure 5A A schematic diagram of a fine-tuning training process of a content generation model provided in an embodiment of the present application;

[0044] Figure 5B A schematic diagram of a fine-tuning training process of a content generation model provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of a model fine-tuning training provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a process for processing watermark information provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of a process for processing watermark information of a commodity image according to an embodiment of the present application;

[0048] Fig. 9 A schematic diagram of the structure of a watermark information processing device provided in an embodiment of the present application;

[0049] Fig.10 A schematic diagram of the structure of a cloud device provided in an embodiment of the present application.

[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0052] The term "and / or" in this article is used to describe the association relationship of associated objects, specifically indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0054] In order to clearly describe the technical solution of the embodiment of the present application, the terms involved in the present application are first defined:

[0055] AI: Artificial Intelligence.

[0056] AIGC: Artificial Intelligence Generated Content, including but not limited to text, images, audio and video.

[0057] AI Editing: Use artificial intelligence technology to edit content, such as image style transfer, filter application, object recognition and removal, background replacement, etc.

[0058] SD: Stable Diffusion, a deep learning model for generating images.

[0059] Diffusion Model: Diffusion Model is a model for generating data, such as the SD model. The basic idea is to start with random noise and gradually "denoise" it to generate target data. A key feature of diffusion models is that they can generate high-quality and diverse samples, which makes them widely used in image generation, text generation and other fields.

[0060] VAE: Variational Autoencoder, variational autoencoder.

[0061] LDM: Latent Diffusion Model, a generative model, is a model that obtains intermediate representation results by step-by-step denoising in a latent representation space, and then decodes it into a complete generated image. Compared with the previous diffusion model based on pixel space, the latent representation space significantly reduces the amount of calculation. It combines the advantages of the diffusion model and the variational autoencoder (VAE) to generate high-quality images from text or other input modes.

[0062] DDIM: Denoising Diffusion Implicit Models, a technique for generating models, especially in image generation tasks. DDIM is a variant of the Diffusion Models that aims to improve sampling efficiency and generation quality.

[0063] DDIM Inversion: The process of inversely mapping a specific image to its representation in the latent space.

[0064] Watermark model hidden: an end-to-end training framework that can be used for data hiding. It uses deep neural networks to achieve data hiding. The goal of data hiding technology is to embed secret information into the carrier data to make it difficult to detect.

[0065] MSE: Mean Squared Error, is an indicator commonly used in statistics and signal processing to measure the difference between the predicted value and the true value.

[0066] Prompt: The prompt word of the large model is an input used to guide the model to generate a specific type of output. It tells the model what type of content should be generated, or in what style. In image generation, the prompt may be a text description of the required image. In AI editing, the prompt may be a description of the image editing process.

[0067] Digital watermark: Add some digital information that is not easily perceived by the human eye to digital media information (such as images, sounds, videos, etc.) in order to protect the copyright of digital media, prove the authenticity of products, track piracy or add additional information to products. The watermark information is embedded in the carrier file and does not affect the visibility and integrity of the original file.

[0068] BCE: Binary Cross-Entropy, binary cross entropy.

[0069] SGD: Stochastic Gradient Descent, stochastic gradient descent, is a commonly used optimization algorithm, widely used in the training of machine learning and deep learning models. The main goal of SGD is to iteratively update model parameters to minimize the loss function, thereby improving the performance of the model.

[0070] Adam: Adaptive Moment Estimation, is a widely used optimization algorithm, especially suitable for training deep learning models.

[0071] JPEG: Joint Photographic Experts Group, the name of an image compression standard.

[0072] JND loss: Just Noticeable Difference loss, which is a loss function commonly used in image processing tasks to simulate the sensitivity of the human visual system to image details.

[0073] Watson Vision Models: A suite of visual recognition tools commonly used for tasks such as image classification, object detection, and image analysis.

[0074] The watermark information processing method of the embodiment of the present application can be applied to any field involving watermark processing, such as AIGC model distribution tracing, AIGC synthetic image marking protection and other scenarios.

[0075] Artificial Intelligence (AI) is the study of the technology of using computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and tasks that usually require human intelligence to complete are performed by computer systems or machines. Artificial Intelligence technology is widely used in various fields.

[0076] With the rapid rise of artificial intelligence technology, AI editing tools, especially deep learning-based image generation technology such as diffusion models, are changing the way we create and consume visual content in unprecedented ways. These technologies can generate highly realistic images and even imitate the style of specific artists or synthesize non-existent scenes, greatly enriching the possibilities of creative expression. However, this also brings unprecedented challenges to the authenticity and copyright protection of images. With the widespread application of AI-generated images in social media, news reporting, advertising and other fields, how to distinguish between real and fake, original and counterfeit has become a problem that needs to be solved urgently.

[0077] In traditional watermarking technology, watermark information can be embedded in the sampled noise, and the original noise of the generated image can be obtained based on the reversibility of DDIM, and then the watermark information can be extracted. The disadvantage of this solution is that it cannot resist AI editing attacks. In addition, this solution is to plug a watermark embedding and extraction plug-in into the AIGC generated model, which will affect the reasoning speed of the original model. When the model itself is distributed for fine-tuning, the watermark plug-in needs to be distributed with it and is easy to be removed.

[0078] It can be seen that although traditional digital watermarking technology can achieve copyright protection and tampering detection of images to a certain extent, its effectiveness is severely tested when facing AI-edited diffusion model-generated images. This is because AI-generated images are often deeply modified at the feature level, and traditional watermark embedding and detection methods are difficult to adapt to such high-intensity image transformations. In addition, the images generated by the diffusion model are visually very similar to real images, making it difficult for existing passive detection technologies to accurately judge the authenticity of images.

[0079] Taking the e-commerce platform scenario as an example, merchants on the e-commerce platform can use AI big models to generate different product images to update the products. If others see that the product is selling well, they are likely to misappropriate the product image without authorization, and then use AI to edit the product image in an attempt to escape infringement. For example, using AI editing to remove the watermark information of the product image, traditional watermark technology cannot resist this kind of AI editing attack, and the product image will soon lose its ownership information, seriously infringing on the user's information security.

[0080] In a related embodiment, a fixed watermark model hidden can be used to fine-tune the decoder of the variational autoencoder VAE in the LDM generation model. By using the MSE loss constraint of the watermark information extracted by hidden and the randomly specified watermark information as well as the perceptual loss constraint of the generated images before and after fine-tuning the VAE, it is ensured that the image generated by LDM can extract the specified watermark information through hidden and minimize the impact on the quality of the generated image.

[0081] However, this scheme can only ensure that the images generated by the fine-tuned LDM model can extract the watermark information through hidden, but it cannot ensure the robustness of the generated images after undergoing editing operation attacks such as compression and scaling, especially it cannot cope with AI editing operation attacks. In addition, randomly specifying watermarks will also have false positive problems.

[0082] In order to solve at least one of the above problems, the embodiment of the present application provides a watermark information processing solution, which simulates different editing operations on digital media information by introducing a preset editing layer in the fine-tuning process, extracts the watermark in the output information after the preset editing operation through a watermark detector, and fine-tunes the original decoder of the content generation model based on the preset editing layer and the watermark detector to obtain the target decoder of the content generation model. In the process of generating digital media information, the random noise is denoised by the content generation model, and the target information features are generated in combination with the prompt words. After fine-tuning, the target decoder of the content generation model can generate the target digital media information with a preset watermark according to the target information features. In this way, by introducing the preset editing layer and the watermark detector in the model fine-tuning process, it is ensured that the target digital media information with a watermark can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, providing a reliable data basis for the subsequent protection of digital media information ownership.

[0083] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0084] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1A processor is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11, and the instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment, so as to ensure that the digital media information with watermarks can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, providing a reliable data basis for the subsequent protection of the ownership of digital media information.

[0085] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0086] Figure 2 A schematic diagram of an application scenario 200 of a watermark information processing system provided in an embodiment of the present application. Figure 2 As shown, the system includes: a server 210 and a terminal 220, wherein:

[0087] The server 210 may be a data platform that provides watermark information processing services, such as an e-commerce platform. In actual scenarios, an e-commerce platform may have multiple servers 210. Figure 2 In the figure, one server 210 is taken as an example.

[0088] The terminal 220 may be a computer, mobile phone, tablet or other device used by the user to log in to the e-commerce platform. There may also be multiple terminals 220. Figure 2 Two terminals 220 are taken as an example for illustration.

[0089] The terminal 220 and the server 210 can transmit information via the Internet, so that the terminal 220 can access the data on the server 210. The terminal 220 and / or the server 210 can be implemented by the electronic device 1.

[0090] The watermark information processing solution of the embodiment of the present application can be deployed on the server 210, can also be deployed on the terminal 220, or can be deployed partially on the server 210 and partially on the terminal 220. In actual scenarios, it can be selected based on actual needs, and this embodiment does not limit it.

[0091] When the watermark information processing solution is fully or partially deployed on the server 210 , a calling interface may be opened to the terminal 220 to provide algorithm support to the terminal 220 .

[0092] The method provided in the embodiment of the present application can be implemented by executing the corresponding software code by the electronic device 1, and can be implemented by exchanging data with the server. The electronic device 1 can be a local terminal device. When the method is run on the server, the method can be implemented and executed based on the cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0093] In a possible implementation, the method provided in the embodiment of the present application provides a graphical user interface through a terminal device, wherein the terminal device can be the local terminal device mentioned above, or can be the client device in the cloud interaction system mentioned above.

[0094] Please see Figure 3 , which is a fine-tuning method for the content generation model of an embodiment of the present application, which can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the watermark information processing application scenario shown in , it is used to fine-tune the content generation model to ensure that the digital media information with watermarks generated by the trained content generation model can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, providing a reliable data basis for the subsequent protection of the ownership of digital media information. In this embodiment, the terminal 220 is used as an execution end as an example, and the method includes the following steps:

[0095] Step 301: Determine a pre-trained watermark detector, an original decoder of a pre-trained content generation model, and a preset editing layer.

[0096] In this step, the watermark detector is used to extract watermark information from digital media information, which refers to various types of information stored, transmitted and presented in digital form. Such information includes but is not limited to text, images, audio, video and other multimedia content. The watermark model generally includes a watermark embedder and a watermark detector, and a pre-trained watermark detector can be obtained through a pre-trained watermark model. For example, the watermark model can be any one of the watermark models based on hidden, staga stamp (an invisible watermark technology) technology and other watermark models designed according to business scenarios.

[0097] The content generation model is used to generate target digital media information according to the user prompt words. The content generation model includes a decoder. The original decoder refers to the decoder in the content generation model that has not been fine-tuned. It is used to embed watermark information in the digital media information during the digital media information generation process. For example, the original decoder can be a variational autoencoder (VAE). Taking the digital media information as an image as an example, the content generation model can be an image generation model, such as an LDM model. The LDM model includes a variational autoencoder. In the inference stage, random sampling is performed in the latent space, and then multi-step noise prediction is performed. After obtaining the denoised features, the original decoder is decoded into the pixel space to generate the final target image.

[0098] The preset editing layer is used to perform editing operations on the watermarked digital media information output by the original decoder during the fine-tuning training process, such as compressing and cropping the watermarked image to try to remove the watermark, simulating the editing attacks that the watermarked image may be subjected to in actual scenarios.

[0099] Step 302: Generate first digital media information with a preset watermark through an original decoder according to the sample prompt words and random noise of the content generation model.

[0100] In this step, the sample prompt words are used as inputs of the content generation model in the fine-tuning training process to instruct the content generation model to generate the corresponding first digital media information. The sample prompt words can be as diverse as possible. For example, if the sample prompt word is "generate a puppy image with a watermark", the content generation model generates the first puppy image with a preset watermark through the original decoder based on the sample prompt word and the random noise image.

[0101] In one embodiment, before generating the first digital media information with a preset watermark through an original decoder according to the sample prompt words and random noise of the content generation model, the method further includes: extracting distribution information of the watermark on the sample digital media information by a statistical watermark detector, where no watermark information exists in the sample digital media information, determining a bias feature of the watermark detector according to the distribution information, and determining a preset watermark according to the bias feature.

[0102] In this embodiment, in order to ensure the stability of the digital watermark, a suitable preset watermark can be configured in a targeted manner according to the specific characteristics of the watermark detector. Taking the scene where the sample digital media information is an image as an example, assuming that the watermark information is a 01 string, such as 00110101, for any image (including images without watermarks), the watermark detector outputs a 01 string. In order to reduce false alarms, watermark extraction is performed on a certain number of images without watermarks. It can be found that when the watermark detector extracts watermarks, the probability of each bit extraction (i.e., watermark extraction distribution information) can be found. Taking an 8-bit watermark as an example, if the statistical watermark extraction distribution information is [0.85, 0.2, 0.55, 0.4, 0.9, 0.2, 0.3, 0.7], it means that the probability that the watermark detector predicts the first bit to be 1 on the image without watermark is 0.85, indicating that the bias feature of the watermark detector is that the first bit prediction result on the image without watermark is more biased to be 1. In order to improve the watermark detector's ability to distinguish between images with watermarks and images without watermarks, a preset watermark can be set in the opposite direction of the watermark detector's detection bias towards images without watermarks. For example, 0.5 is used as the dividing line, and according to the watermark extraction distribution information, the watermark information corresponding to the bit with a probability greater than 0.5 is determined to be 0, and the watermark information corresponding to the bit with a probability less than 0.5 is determined to be 1. In the scenario where the watermark extraction distribution information is [0.85, 0.2, 0.55, 0.4, 0.9, 0.2, 0.3, 0.7], the first bit of the preset watermark can be selected as 0, and so on. The preset watermark can be selected as 01010110 to ensure that the probability of collision between the preset watermark and the watermark extracted by the watermark detector from the image without watermark is the lowest.

[0103] Step 303: After performing a preset editing operation on the first digital media information through a preset editing layer, second digital media information is generated.

[0104] In this step, in order to make the watermarked digital media information generated by the fine-tuned content generation model resist attacks from various editing operations, a preset editing layer is introduced during the training process to simulate various editing operations on the first digital media information. Taking the aforementioned puppy image as an example, the preset editing operation is performed on the first puppy image through the preset editing layer to generate a second puppy image. The preset editing layer can be set based on actual needs, and different preset editing layers can be set for different application scenarios, so that the fine-tuned content generation model has stable watermark embedding and detection functions.

[0105] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the first digital media information.

[0106] In this embodiment, by introducing multiple editing operations in the preset editing layer, including but not limited to AI editing operations, geometric transformation operations, and custom operations, the flexibility and adaptability of the content generation model are further enhanced. By supporting AI editing operations, geometric transformation operations, and custom operations, the method provides a wealth of editing options, simulating as much as possible the editing attacks that digital watermarks may encounter in actual scenarios, so that the digital media information with digital watermarks generated by the fine-tuned content generation model can resist various editing attacks and improve the stability of the digital watermark.

[0107] Step 304: extract the watermark in the second digital media information through a watermark detector, and fine-tune the original decoder of the content generation model with the goal of minimizing the first loss function between the extracted watermark and the preset watermark to obtain a fine-tuned target decoder.

[0108] In this step, the first loss function is used to characterize the difference between the extracted watermark and the preset watermark. In the training process, the goal is to minimize the first loss function, which can improve the accuracy of the trained model. Taking the aforementioned puppy image as an example, the watermark in the second puppy image is extracted by the watermark detector during the fine-tuning training process, and then the original decoder of the content generation model is fine-tuned to obtain the fine-tuned target decoder with the goal of minimizing the first loss function between the extracted watermark and the preset watermark. Then, the original decoder of the content generation model is replaced with the target decoder. Since the target decoder learns a more stable watermark embedding method during the fine-tuning training based on the preset editing layer and the watermark detector, the image with a digital watermark generated by the fine-tuned content generation model can resist various editing attacks and improve the stability of the digital watermark. It is achieved to ensure that the digital media information with a watermark can still be accurately detected after the preset editing operation, improve the concealment and robustness of the watermark, make it difficult to be detected or removed, improve the resistance of the digital watermark to complex editing operations, and provide a reliable data basis for the subsequent digital media information ownership protection.

[0109] In one embodiment, the method further includes: calculating a second loss function between the second digital media information and the first digital media information, and fine-tuning the original decoder with the goal of minimizing the first loss function and the second loss function to obtain a fine-tuned target decoder.

[0110] In this embodiment, the second loss function is used to characterize the difference between the second digital media information and the first digital media information. During the training process, in order to ensure the content consistency of the digital media information before and after the watermark is embedded, the second loss function can be used to constrain the training. The original decoder is fine-tuned with the goal of minimizing the first loss function and the second loss function, so that the fine-tuned target decoder can generate watermarked digital media information with accurate content, thereby improving the accuracy of the target decoder.

[0111] In one embodiment, the method further includes: fine-tuning the pre-trained watermark detector with the goal of minimizing the first loss function and the second loss function to obtain a fine-tuned watermark detector.

[0112] In this embodiment, in addition to fine-tuning the decoder, the watermark detector can be further adjusted and optimized according to different application requirements, so that the fine-tuned watermark detector can adapt to different usage scenarios and requirements. This flexibility enables the system to maintain efficient watermark detection capabilities in a variety of environments. Specifically, by fine-tuning the pre-trained watermark detector with the goal of minimizing the first loss function and the second loss function, the detection accuracy of the watermark detector can be significantly improved. The optimization of the first loss function ensures the consistency of the watermark extracted by the model with the preset watermark, and the constraint of the second loss function ensures that the digital media information can maintain content consistency before and after the watermark is embedded. Under the constraints of the two, the fine-tuned watermark detector can more effectively identify and detect watermarks.

[0113] In one embodiment, the process of determining the pre-trained watermark detector in step 301 may specifically include: obtaining carrier information and sample watermarks. Using the carrier information and sample watermarks to train a preset watermark model to obtain a trained watermark model. Determining the watermark detector of the trained watermark model as the pre-trained watermark detector.

[0114] In this embodiment, a watermark detector can be obtained by training a watermark model. The carrier information refers to the carrier used to carry the sample watermark, such as a carrier image. During the training process, the sample watermark is embedded in the carrier image. The sample watermark can be a randomly generated watermark or a specifically selected watermark. A watermark model generally includes a watermark embedder and a watermark detector. By obtaining the carrier information and the sample watermark, and using this information to train the preset watermark model. During the training process, using the diverse carrier information and sample watermarks, the watermark model can learn watermark features of different types and forms. This diversity training enables the pre-trained watermark detector to maintain high detection performance when facing unknown or changing watermark styles. The pre-trained watermark detector can be further adjusted and optimized according to different application requirements so that it can adapt to different usage scenarios and requirements. This flexibility enables the system to maintain efficient watermark detection capabilities in a variety of environments.

[0115] like Figure 4 As shown in FIG. 1 , a schematic diagram of a watermark model training provided in an embodiment of the present application is shown. Taking the image watermark processing scenario as an example, the training sample includes watermark information and a carrier image. The watermark model includes a watermark embedder W e , watermark detector W d , and an optional noise layer, the noise layer is differentiable. Watermark model training includes two processes: watermark embedding and watermark extraction. At the beginning of training, the network weight information is initialized. The watermark information w1 can be randomly generated according to the watermark capacity. e The watermark information w1 is embedded in the carrier image I1 to generate the watermarked image I2. The watermarked image I2 is processed by the differentiable noise layer and then enters the watermark detector W. d , through the watermark detector W d Extract watermark information w2 from the watermarked image I2. The loss function includes the first loss function loss w And the second loss function loss I , where loss I Used to constrain the difference between the watermarked image I2 and the carrier image I1, loss w Used to constrain the difference between watermark information w2 and watermark information w1 to minimize loss I And loss w is the training target. The loss function includes but is not limited to the common BCE, MSE loss and perceptual loss for images, and the optimization method includes but is not limited to the common SGD, Adam, etc.

[0116] Optionally, the differentiable noise layer is a collection of differentiable attacks, such as scaling, cropping, approximate JPEG compression, etc. These attacks can be implemented in multiple ways. During the training process, for each batch (referring to a group of samples processed simultaneously during the training process), an attack can be randomly selected in the noise layer to attack it, so as to improve the detection accuracy of the watermark detector.

[0117] In one embodiment, step 302 specifically includes: obtaining sample prompt words and random noise of the content generation model. De-noising the random noise according to the sample prompt words to obtain sample information features after de-noising. Inputting the sample information features into the original decoder so that the original decoder generates the first digital media information with a preset watermark according to the sample information features.

[0118] In this embodiment, random noise is denoised according to the sample prompt word, ensuring that the digital media information finally generated can accurately reflect the content of the input prompt word. In the process of generating digital media information, the original decoder generates the first digital media information with a preset watermark according to the denoised sample information features. Taking the LDM-based image generation model as an example, the LDM model randomly samples in the latent space during the inference stage, and then performs multi-step noise prediction to obtain the denoised features, which are then decoded into the pixel space by the VAE decoder to generate an image that meets the prompt word and embed the preset watermark in the image.

[0119] like Figure 5A As shown in FIG. 1 , a schematic diagram of a fine-tuning training process of a content generation model provided in an embodiment of the present application is shown. First, a pre-trained watermark detector W is obtained. d , then fix the watermark detector W d , fine-tune the VAE decoder D vae , fixed VAE decoder D vae , fine-tune the watermark detector W d , the fine-tuning process is performed by alternating iterations, and the target decoder Df is finally produced. vae and the fine-tuned watermark detector W d f .

[0120] like Figure 5B As shown, it is a schematic diagram of a fine-tuning training process of a content generation model provided in an embodiment of the present application. Taking the image watermarking scenario as an example, for the LDM large model L to implement the watermarking function, the training process may include:

[0121] Step 1: Train the digital image watermark model and obtain the watermark detector W d For details, please refer to the above Figure 4 The corresponding description of the embodiment.

[0122] Step 2: Get the basic LDM model L and the VAE decoder D in the model L vae .

[0123] Step 3: Statistical watermark detector W d Extract the distribution information of the watermark on the image without watermark, and select the preset watermark w according to the watermark extraction distribution information s For details, please refer to the above description of preset watermarks.

[0124] Step 4: Fix the watermark detector W d , fine-tune the VAE decoder D vae .

[0125] Step 5: Fix the VAE decoder D vae , design preset editing layers, fine-tune watermark detector W d .

[0126] Step 6: Iterate Step 4 and Step 5 to obtain the fine-tuned target decoder D f vae , watermark detector W d f , and then replace the original VAE decoder D of the large model L vae The target decoder D f vae , get the LDM large model L with watermark function f (i.e., the fine-tuned content generation model) and the corresponding fine-tuned watermark detector W d f By iteratively fine-tuning D f vae and the watermark detector W d Higher robustness can be obtained. At this time, after the large model L f The generated images all contain a preset watermark w s , even after AI tools have been used to edit the image, the watermark detector W d f The watermark can still be detected, which improves the digital watermark's resistance to AI editing operations and provides an accurate data basis for the ownership information of the watermark image.

[0127] like Figure 6 As shown in FIG. 1 , a schematic diagram of a model fine-tuning training provided in an embodiment of the present application is shown. Taking the image watermarking scenario as an example, for the fixed watermark detector W in Step 4, d , fine-tune the VAE decoder D vae ,like Figure 6 As shown, z Tis the noise randomly sampled in the latent space. After denoising by a T-step denoising network (u-net, a convolutional neural network architecture for image segmentation) under the constraint of the sample prompt word P (Prompt), the denoised sample information feature z0 is obtained. At this time, the original decoder D of the large model L vae The image I without watermark can be obtained from z0 s , you can copy a copy of D vae As the target decoder D f vae The initialization value of z0 passes through the target decoder D f vae Get the embedded preset watermark w s Image I s f , during fine-tuning, constraint I s f After the AI ​​editing layer, it can be detected by the watermark detector W d f Propose watermark w e , through the second loss function loss I , Constraint I s f Close to I s , through the first loss function loss w Constraint s Close to w s Among them, loss I It can be obtained using the following formula:

[0128]

[0129] loss w It can be obtained using the following formula:

[0130] loss w =λ3*BCEloss(w s ,w e )

[0131] Among them, MSEloss represents image I s f With image I s The mean square error loss between the two. JNDloss represents the image I s f With image I s The perceptual loss between the two, including but not limited to the common perceptual loss and the loss combined with the Watson visual model. BCEloss represents the preset watermark w s With the extracted watermark w eλ1 represents the weight of JNDloss, λ2 represents the weight of MSEloss, and λ3 represents the weight of BCEloss. In actual scenarios, the weight values ​​of the above three losses can be set according to the business scenario requirements. If the business scenario focuses on robustness, the weight λ3 of the BCEloss part can be increased. If the business focuses on visual quality, the weight λ2 of MSEloss representing visual quality and the weight λ1 of JNDloss can be increased.

[0132] The above method configures the preset watermark information by statistically extracting the distribution information of the watermark detected by the watermark detector, so that the watermark detection result has a lower false positive. By introducing the AI ​​editing layer, the VAE decoder and watermark detector are fine-tuned by alternating optimization, so that the watermarked image generated by the trained content generation model can not only resist the attack of traditional editing operations, but also resist the AI ​​editing attack, ensuring the authenticity and copyright of the image. Different from the traditional plug-in type of adding watermarks, the function of embedding watermarks in this solution is integrated with some parameters of the AIGC model VAE, which has high concealment and does not affect the fine-tuning function and reasoning time of the large model denoising network.

[0133] Please see Figure 7 , which is a watermark information processing method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the watermark information processing application scenario shown in , it is used to generate digital media information with watermarks, so that the digital media information with watermarks can still be accurately detected after the preset editing operation, improve the concealment and robustness of the watermark, make it difficult to be detected or removed, improve the resistance of the digital watermark to complex editing operations, and provide a reliable data basis for the subsequent protection of the ownership of digital media information. This embodiment takes the terminal 220 as an example of the execution end, and the method includes the following steps:

[0134] Step 701: In response to a content generation instruction, obtain a prompt word corresponding to the generation instruction.

[0135] In this step, the content generation instruction can be triggered by the user. For example, if an e-commerce user wants to create a product image, he can input a prompt word into the large model and trigger the image generation instruction.

[0136] Step 702: Input the prompt word into the content generation model so that the content generation model generates target digital media information according to the prompt word, and the target digital media information carries a preset watermark embedded by a target decoder of the content generation model.

[0137] In this step, the target decoder is trained by the fine-tuning method of any of the aforementioned embodiments. Specifically, the target decoder is obtained by fine-tuning the original decoder of the content generation model based on the preset editing layer and the watermark detector. During the fine-tuning training process, the preset editing layer is used to perform a preset editing operation on the output information with a preset watermark output by the original decoder (such as the aforementioned first digital media information), and the watermark detector is used to extract the watermark information in the output information after the preset editing operation (such as the aforementioned second digital media information). The fine-tuning training process aims to minimize the difference between the extracted watermark and the preset watermark. The detailed fine-tuning training process can be found in the relevant description of the method in any of the aforementioned embodiments, which will not be repeated here.

[0138] During the inference process, the content generation model denoises the random noise according to the prompt words to obtain the denoised target information features, and generates the target digital media information carrying a preset watermark according to the target information features through the target decoder.

[0139] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the output information.

[0140] In this embodiment, the output information may refer to the first digital media information involved in the aforementioned embodiment. For related descriptions, reference may be made to the description of the preset editing layer in the aforementioned embodiment, which will not be repeated here.

[0141] In one embodiment, the process of embedding a preset watermark through a target decoder of a content generation model includes: obtaining denoised target information features after the content generation model denoises random noise according to a prompt word; reconstructing information according to the target information features through a target decoder to generate initial digital media information, and embedding a preset watermark in the initial digital media information to obtain target digital media information.

[0142] In this embodiment, the content generation model may be a diffusion model, such as an LDM large model, and the target decoder may be a target decoder D of the LDM large model L after fine-tuning training. f vae Taking the image generation process as an example, first, the content generation model denoises the random noise according to the prompt word to obtain the denoised target information features. The target decoder reconstructs the image using the target information features to generate the initial image, and then embeds a preset watermark in the generated initial image to obtain the final target image. The target image can not only resist attacks from traditional editing operations, but also resist attacks from AI editing, ensuring the authenticity and copyright of the image.

[0143] In one embodiment, before step 702, the method further includes: collecting watermark extraction distribution information of the watermark detector on the sample digital media information, and the sample digital media information does not contain watermark information. The bias feature of the watermark detector is determined according to the distribution information, and the preset watermark is determined according to the bias feature. For details, please refer to the description of the selection process of the preset watermark in the above embodiment, which will not be repeated here.

[0144] In one embodiment, the method further includes: in response to the ownership detection instruction for the specified digital media information, extracting the target watermark in the specified digital media information using a watermark detector, comparing the target watermark with a preset watermark, and determining the ownership information of the specified digital media information according to the comparison result.

[0145] In this embodiment, by extracting and comparing the watermark embedded in the digital media information, the ownership of the information can be accurately verified to ensure the clarity of copyright ownership. The watermark detector is trained by the fine-tuning training method of any of the aforementioned embodiments. The watermark detector is used to automatically extract and compare watermarks. Even after the digital media information is edited, compressed or transmitted by AI, the watermark can still be effectively extracted and identified, which not only simplifies the ownership verification process and effectively promotes the correct use and dissemination of digital media information, but also ensures the reliability of ownership information and enhances the security of information.

[0146] The above-mentioned watermark information processing method simulates different editing operations on digital media information by introducing a preset editing layer in the fine-tuning process, extracts the watermark in the output information after the preset editing operation through a watermark detector, and fine-tunes the original decoder of the content generation model based on the preset editing layer and the watermark detector to obtain the target decoder of the content generation model. In the process of generating digital media information, the content generation model is used to denoise the random noise, and the target information features are generated in combination with the prompt words. After fine-tuning, the target decoder of the content generation model can generate the target digital media information with the preset watermark according to the target information features. In this way, by introducing the preset editing layer and the watermark detector in the process of model fine-tuning, it is ensured that the target digital media information with the watermark can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of the digital watermark to complex editing operations, providing a reliable data basis for the subsequent protection of digital media information ownership.

[0147] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0148] Please see Figure 8 , which is a method for processing watermark information of a commodity image according to an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the watermark information processing application scenario shown in , it is used to generate a product image with a watermark. In this embodiment, the terminal 220 is used as an execution end, and the product image scenario in the e-commerce scenario is used as an example. The method includes the following steps:

[0149] Step 801: In response to a product image generation instruction, obtain a prompt word corresponding to the generation instruction, where the prompt word includes product information.

[0150] In this step, the product information includes but is not limited to product-related knowledge information, appearance information, product promotion information, product-related order information, product-related service information, and the like.

[0151] Step 802: Input the prompt word into the content generation model, so that the content generation model performs denoising on the random noise according to the prompt word, obtains the denoised target image features, and generates a target product image carrying a preset watermark according to the target image features through a target decoder. The target decoder is trained according to the fine-tuning method of the content generation model of any of the above embodiments.

[0152] The above-mentioned product watermark information processing method realizes the addition of preset watermarks to product images created by AI in e-commerce scenarios, ensuring that the target product image with the watermark can still be accurately detected after the preset editing operation, thereby improving the concealment and robustness of the watermark, making it difficult to be detected or removed, and improving the resistance of digital watermarks to complex editing operations, providing a reliable data basis for the subsequent protection of the ownership of product images.

[0153] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0154] Please see Fig. 9 , which is a watermark information processing device 900 of an embodiment of the present application, which can be applied to Figure 1 The electronic device 1 shown can be applied to Figure 2 In the watermark information processing application scenario shown in , it is used to generate digital media information with watermarks, so that the digital media information with watermarks can still be accurately detected after the preset editing operation, improve the concealment and robustness of the watermark, make it difficult to be detected or removed, improve the resistance of the digital watermark to complex editing operations, and provide a reliable data basis for the subsequent protection of the ownership of digital media information. The device includes: an acquisition module 901 and a generation module 902, and the functional principles of each module are as follows:

[0155] The acquisition module 901 is used to respond to the content generation instruction and acquire the prompt word corresponding to the generation instruction.

[0156] The generation module 902 is used to input the prompt word into the content generation model so that the content generation model generates target digital media information according to the prompt word, and the target digital media information carries a preset watermark embedded by a target decoder of the content generation model.

[0157] Among them, the target decoder is obtained by fine-tuning the original decoder of the content generation model based on the preset editing layer and the watermark detector. During the fine-tuning training process, the preset editing layer is used to perform a preset editing operation on the output information with a preset watermark output by the original decoder, and the watermark detector is used to extract the watermark information in the output information after the preset editing operation.

[0158] In one embodiment, the preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the output information.

[0159] In one embodiment, the generation module 902 is used to obtain the denoised target information features obtained after the content generation model denoises the random noise according to the prompt word; reconstruct the information according to the target information features through the target decoder to generate initial digital media information, and embed a preset watermark in the initial digital media information to obtain the target digital media information.

[0160] In one embodiment, the method further includes: a determination module, which is used to collect watermark extraction distribution information of the watermark detector on the sample digital media information before generating the target digital media information carrying the preset watermark according to the target information characteristics through the target decoder, and the watermark information does not exist in the sample digital media information. The bias characteristics of the watermark detector are determined according to the distribution information, and the preset watermark is determined according to the bias characteristics.

[0161] In one embodiment, the system further includes: a detection module, which is used to extract a target watermark in the specified digital media information using a watermark detector in response to an ownership detection instruction for the specified digital media information, compare the target watermark with a preset watermark, and determine the ownership information of the specified digital media information according to the comparison result.

[0162] For a detailed description of the watermark information processing device 900, please refer to the description of the relevant method steps in the above embodiment. The implementation principle and technical effect are similar and will not be repeated in this embodiment.

[0163] Fig.10 The following is a schematic diagram of a cloud device 100 provided in an exemplary embodiment of the present application. The cloud device 100 can be used to run the method provided in any of the above embodiments. Fig.10 As shown, the cloud device 100 may include: a memory 1004 and at least one processor 1005, Fig.10 A processor is taken as an example.

[0164] The memory 1004 is used to store computer programs and can be configured to store various other data to support operations on the cloud device 100. The memory 1004 can be an object storage service (OSS).

[0165] Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0166] The processor 1005 is coupled to the memory 1004 and is used to execute the computer program in the memory 1004 to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0167] Furthermore, if Fig.10 The cloud device also includes: a firewall 1001, a load balancer 1002, a communication component 1006, a power supply component 1003 and other components. Fig.10 Only some components are shown schematically, which does not mean that the cloud device only includes Fig.10 Components shown.

[0168] In one embodiment, the above Fig.10 The communication component 1006 in is configured to facilitate wired or wireless communication between the device where the communication component 1006 is located and other devices. The device where the communication component 1006 is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, LTE (Long Term Evolution, Long Term Evolution, referred to as LTE), 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 1006 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1006 also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (Radio Frequency Identification, referred to as RFID) technology, Infrared Data Association (Infrared Data Association, referred to as IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (bluetooth, referred to as BT) technology and other technologies to achieve.

[0169] In one embodiment, the above Fig.10 The power supply component 1003 provides power to various components of the device where the power supply component 1003 is located. The power supply component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0170] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0171] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0173] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0174] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM (Random Access Memory) memory, and may also include non-volatile storage NVM (Nonvolatile memory, NVM for short), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0175] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special computer.

[0176] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0177] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, clothing or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, clothing or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, clothing or device including the element.

[0178] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0180] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0181] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A watermark information processing method, characterized in that: include: In response to a content generation instruction, obtaining a prompt word corresponding to the generation instruction; Inputting the prompt word into a content generation model so that the content generation model generates target digital media information according to the prompt word, wherein the target digital media information carries a preset watermark embedded by a target decoder of the content generation model; Among them, the target decoder is obtained by fine-tuning the original decoder of the content generation model based on the preset editing layer and the watermark detector. During the fine-tuning training process, the preset editing layer is used to perform a preset editing operation on the output information with a preset watermark output by the original decoder, and the watermark detector is used to extract the watermark information in the output information after the preset editing operation.

2. The method according to claim 1, characterized in that The preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations and custom operations applied to the output information.

3. The method according to claim 1, characterized in that The process of embedding a preset watermark through a target decoder of the content generation model includes: Obtaining the denoised target information features obtained after the content generation model denoises the random noise according to the prompt word; The target decoder reconstructs the information according to the target information characteristics to generate initial digital media information, and embeds the preset watermark in the initial digital media information to obtain the target digital media information.

4. The method according to claim 1, characterized in that: Before inputting the prompt word into the content generation model so that the content generation model generates the target digital media information according to the prompt word, the method further includes: Counting watermark extraction distribution information of the watermark detector on sample digital media information, wherein no watermark information exists in the sample digital media information; The bias characteristics of the watermark detector are determined according to the distribution information, and the preset watermark is determined according to the bias characteristics.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: In response to an ownership detection instruction for designated digital media information, extracting a target watermark in the designated digital media information using the watermark detector; The target watermark is compared with the preset watermark, and the ownership information of the designated digital media information is determined according to the comparison result.

6. A method for fine-tuning a content generation model, characterized in that: include: Determine the original decoder and preset editing layer of the pre-trained content generation model; Generate first digital media information with a preset watermark through the original decoder according to the sample prompt words and random noise of the content generation model; After performing a preset editing operation on the first digital media information through the preset editing layer, generating second digital media information; The watermark in the second digital media information is extracted by a pre-trained watermark detector, and the original decoder of the content generation model is fine-tuned to obtain a fine-tuned target decoder with the goal of minimizing a first loss function between the extracted watermark and the preset watermark.

7. The method according to claim 6, characterized in that The preset editing layer includes one or more of artificial intelligence editing operations, geometric transformation operations, and custom operations applied to the first digital media information.

8. The method according to claim 6, characterized in that The step of generating the first digital media information with a preset watermark through the original decoder according to the sample prompt words and random noise of the content generation model comprises: Obtaining sample prompt words and random noise of the content generation model; De-noising the random noise according to the sample prompt word to obtain the sample information feature after de-noising; The sample information feature is input into the original decoder, so that the original decoder generates first digital media information with a preset watermark according to the sample information feature.

9. The method according to claim 6, characterized in that Also includes: A second loss function between the second digital media information and the first digital media information is calculated, and the original decoder is fine-tuned to obtain a fine-tuned target decoder with the goal of minimizing the first loss function and the second loss function.

10. The method according to claim 9, characterized in that Also includes: With the goal of minimizing the first loss function and the second loss function, the pre-trained watermark detector is fine-tuned to obtain a fine-tuned watermark detector.

11. The method according to claim 6, characterized in that The determination process of the pre-trained watermark detector includes: Get carrier information and sample watermark; Using the carrier information and the sample watermark to train a preset watermark model to obtain a trained watermark model; The watermark detector of the trained watermark model is determined as the pre-trained watermark detector.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

Images