Method and device for determining image generation model and image generation method and device

Abstract

The invention provides a method for determining an image generation model, and the method comprises the steps: obtaining a target watermark code which represents watermark information expected to be embedded in an image generated by the image generation model; acquiring a sample image; based on the sample image, a reconstructed image is generated through a target model, and the target model is configured to generate the reconstructed image according to noisy data related to the sample image; generating an extracted watermark code through a watermark extraction network based on the reconstructed image; determining watermark extraction loss based on the extracted watermark code and the target watermark code; determining image perception loss based on the reconstructed image and the sample image; and updating model parameters of the target model based on the watermark extraction loss and the image perception loss to determine an image generation model. The embodiment of the invention can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like, is helpful to avoid watermarking-free image leakage, and reduces the embezzlement risk of the image generation model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a method and apparatus for determining an image generation model, an image generation method and apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the vigorous development of artificial intelligence (AI) technology, it has been increasingly applied to different fields and played an important role. For example, the popularity of artificial intelligence generated content (AIGC) is currently high. It is predicted that by 2025, AIGC will account for 10% of all generated data in the world. The application of AIGC has largely liberated the productivity of many industries and accelerated the pace of informatization construction, but at the same time, the copyright issue of AIGC has also become a focus of controversy. Specifically, the inability to confirm rights and copyright disputes will be a major dilemma facing AIGC. Therefore, there is an urgent need for a technical solution for AIGC that can declare copyright and locate AI-generated content. Summary of the invention

[0003] In view of this, the present disclosure provides a method and apparatus for determining an image generation model, a picture generation method and apparatus, a computing device, a computer-readable storage medium, and a computer program product, which can alleviate, mitigate, or even eliminate the above-mentioned problems.

[0004] According to one aspect of the present disclosure, a method for determining an image generation model is provided, comprising: obtaining a target watermark code, wherein the target watermark code represents watermark information expected to be embedded in an image generated by the image generation model; obtaining a sample image; generating a reconstructed image based on the sample image through a target model, wherein the target model is configured to generate a reconstructed image based on noisy data associated with the sample image; generating an extracted watermark code based on the reconstructed image through a watermark extraction network; determining a watermark extraction loss based on the extracted watermark code and the target watermark code; determining an image perception loss based on the reconstructed image and the sample image; and updating model parameters of the target model based on the watermark extraction loss and the image perception loss to determine the image generation model.

[0005] In some embodiments, the watermark extraction network is determined by the following steps: obtaining an input image and an input watermark code; generating a watermarked image based on the input image and the input watermark code; extracting the watermark code through a target neural network based on the watermarked image; determining the watermark loss based on the extracted watermark code and the input watermark code; and updating the network parameters of the target neural network based on the watermark loss to determine the watermark extraction network.

[0006] In some embodiments, based on the watermarked image, extracting the watermark code through the target neural network includes: performing an image attack operation on the watermarked image to generate a processed watermarked image, the image attack operation including at least one of the following: image cropping, image compression, image noisy; based on the processed watermarked image, extracting the watermark code through the target neural network.

[0007] In some embodiments, generating a watermarked image based on an input image and an input watermark code includes: generating a watermarked image based on the input image and the input watermark code; generating an activated watermarked image based on the watermarked image through an activation function; generating a watermarked image based on a weighted sum of the input image and the activated watermarked image.

[0008] In some embodiments, the activation function is configured to convert the pixel value of each pixel in the watermark image into a preset interval, and the preset interval is symmetric about the zero point.

[0009] In some embodiments, updating the model parameters of the target model based on the watermark extraction loss and the image perception loss includes: determining a merging loss based on a weighted sum of the image perception loss and the watermark extraction loss; and updating the model parameters of the target model based on the merging loss.

[0010] In some embodiments, obtaining a target watermark code includes: obtaining a target watermark, the target watermark including a plurality of characters; encoding the plurality of characters in the target watermark; and concatenating the codes of the plurality of characters to generate the target watermark code.

[0011] In some embodiments, the target model includes an encoder, a diffusion network and a decoder, and wherein, based on the sample image, generating a reconstructed image through the target model includes: generating a latent space vector based on the sample image through the encoder; generating a reconstruction vector based on the latent space vector through the diffusion network, wherein the diffusion network is configured to generate a reconstruction vector based on noisy data related to the latent space vector; and generating a reconstructed image based on the reconstruction vector through the decoder.

[0012] In some embodiments, the encoder has fixed parameters, and wherein, based on the watermark extraction loss and the image perception loss, updating the model parameters of the target model to determine the image generation model includes: based on the watermark extraction loss and the image perception loss, updating the parameters of the diffusion network and the decoder to determine the image generation model.

[0013] According to another aspect of the present disclosure, there is provided an image generation method, comprising: acquiring starting image data, wherein the starting image data comprises noise data; generating an output image based on the starting image data through an image generation model, wherein the image generation model is determined by the method described in any one of the embodiments in the aforementioned aspects.

[0014] According to another aspect of the present disclosure, there is provided an apparatus for determining an image generation model, comprising: a first acquisition module, configured to acquire a target watermark code, wherein the target watermark code represents watermark information expected to be embedded in an image generated by the image generation model; a second acquisition module, configured to acquire a sample image; a reconstructed image generation module, configured to generate a reconstructed image through a target model based on the sample image, wherein the target model is configured to generate a reconstructed image based on noisy data associated with the sample image; a watermark extraction module, configured to generate an extracted watermark code through a watermark extraction network based on the reconstructed image; a first loss determination module, configured to determine a watermark extraction loss based on the extracted watermark code and the target watermark code; a second loss determination module, configured to determine an image perception loss based on the reconstructed image and the sample image; a parameter updating module, configured to update model parameters of the target model based on the watermark extraction loss and the image perception loss to determine the image generation model.

[0015] According to another aspect of the present disclosure, an image generating device is provided, comprising: an acquisition module configured to acquire starting image data, wherein the starting image data comprises noise data; a generation module configured to generate an output image based on the starting image data through an image generation model, wherein the image generation model is determined using the device according to the aforementioned aspects.

[0016] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute the method according to any one of the embodiments of the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the method according to any one of the embodiments in the aforementioned aspects is executed.

[0018] According to another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which implement the steps of the method described in any one of the embodiments of the aforementioned aspects when the computer instructions are executed by a processor.

[0019] Through the method for determining the image generation model provided by the present disclosure, a corresponding reconstructed image can be generated based on a sample image and using a target model, a watermark code can be extracted from the reconstructed image using a predetermined watermark extraction network, the image perception loss can be determined based on the reconstructed image and the sample image, the watermark extraction loss can be determined based on the extracted watermark code and the target watermark code, and the model parameters of the target model can be updated based on both the watermark extraction loss and the image perception loss to determine the image generation model. The image generation model determined by the above process can embed the target watermark code while generating the image. On the one hand, the embedded watermark code can be used to declare copyright and determine whether it is AI-generated content; on the other hand, compared with the watermark embedding method of first generating an image and then adding a watermark, the above method helps to avoid the leakage of watermark-free images and reduce the risk of the image generation model being stolen; on the other hand, since both the watermark extraction loss and the image perception loss are considered at the same time, the watermark embedding achieved by the above process can achieve the aforementioned copyright declaration effect without significantly affecting the quality of the generated image.

[0020] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 An exemplary scenario in which the technical solution of the present disclosure can be applied is schematically shown;

[0023] Figure 2A An exemplary flowchart of a method for determining an image generation model according to some embodiments of the present disclosure is schematically shown;

[0024] Figure 2B Schematically illustrates an exemplary block diagram for determining an architecture of an image generation model according to some embodiments of the present disclosure;

[0025] Figure 3A An exemplary flow chart of a method for determining a watermark extraction network according to some embodiments of the present disclosure is schematically shown;

[0026] Figure 3B Schematically illustrates an exemplary block diagram for determining the architecture of a watermark extraction network according to some embodiments of the present disclosure;

[0027] Figure 4 Schematically illustrates an exemplary network structure of a watermark network according to some embodiments of the present disclosure;

[0028] Figure 5 Schematically illustrates an exemplary encoding process of watermark information according to some embodiments of the present disclosure;

[0029] Figure 6 Schematically illustrates an exemplary training process of an image generation model according to some embodiments of the present disclosure;

[0030] Figure 7 An exemplary flowchart of an image generation method according to some embodiments of the present disclosure is schematically shown;

[0031] Figure 8 An exemplary block diagram of an apparatus for determining an image generation model according to some embodiments of the present disclosure is schematically shown;

[0032] Fig. 9 An exemplary block diagram of an image generating device according to some embodiments of the present disclosure is schematically shown;

[0033] Fig.10 An exemplary block diagram of a computing device according to some embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] Before describing the embodiments of the present disclosure in detail, some related concepts are first explained.

[0035] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0036] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0037] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0038] Artificial Intelligence Generated Content (AIGC) can be understood as a technology that uses artificial intelligence technology to implement content production. It can generate relevant content with appropriate generalization capabilities by learning and identifying existing data based on methods such as model training. In this field, models or applications such as ChatGPT (Chat Generative Pre-trained Transformer) and StableDiffusion have received widespread attention and application. These models or applications have largely liberated the productivity of multiple industries and accelerated the pace of informatization construction.

[0039] A watermark can be understood as additional information added to digital media (such as images, audio, video or documents). This additional information can be copyright information, owner identification, etc., or any information that you want to add. On this basis, an invisible watermark can refer to embedding the above additional information into digital media, making the embedded information invisible or difficult to detect, so as to avoid the watermark affecting the visual quality of the original media. In other words, the main purpose of an invisible watermark can be understood as providing a difficult-to-remove mark without interfering with the original content.

[0040] As mentioned above, the copyright issue of AIGC has become an issue that needs to be resolved or improved urgently. On the one hand, AI-generated images are at risk of leakage and abuse, so there is a need to determine whether the image is AI-generated and which organization or individual generated the image; on the other hand, the model used to generate the image itself is at risk of leakage and abuse, so there is a need to determine whether the model has been leaked and locate the organization or individual who has stolen the model. Through research and analysis, the applicant found that in the relevant technology, there is a solution for adding watermarks to AI-generated images. However, in this type of solution, an additional watermark is added to the generated image after the model generates the image. In this process, images that are not protected by watermarks in the middle are still at risk of leakage, and if the model that generates the image is leaked, it is impossible to prove whether the model has been stolen and by whom.

[0041] Based on the above considerations, the present disclosure provides a new technical solution to solve or alleviate the various problems mentioned above. The technical solution provided by the present disclosure will be described in detail below in conjunction with specific embodiments.

[0042] Figure 1 An example application scenario 100 to which the technical solution provided by the embodiments of the present disclosure can be applied is schematically shown.

[0043] like Figure 1 As shown, the scene 100 may include a server 110. The server 110 may be a single server or a server cluster, or a cloud server, and may be configured to execute various embodiments of the method for determining an image generation model described in the present disclosure. Exemplarily, a target model to be trained may be deployed on the server 110, which may automatically or under user control perform training of the target model based on data stored locally or from other devices (such as a database 120) by the methods described in the various embodiments of the present disclosure, and determine the image generation model. Alternatively or additionally, the server 110 may also be configured to execute various embodiments of the image generation method described in the present disclosure. Exemplarily, the server 110 may generate an image based on data automatically generated or provided by the user, using an image generation model determined by the method according to an embodiment of the present disclosure. Optionally, the generated image may be stored locally, stored in other devices (such as a database 120), or sent to other devices (such as a terminal device 130), etc. Optionally, the server 110 may also be a computing device of other types. Optionally, in addition to executing the technical solution provided by the present disclosure, the server 110 may also run other applications and store other data. For example, the server 110 may include multiple virtual hosts to run different application programs and provide different services.

[0044] Optionally, the scene 100 may also include a database 120. Exemplarily, the database 120 may be viewed as an electronic filing cabinet, i.e., a place for storing electronic files, and users may perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data that is stored together in a certain manner, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. Exemplarily, the database device 120 may be used to store data or files such as text and images. For example, the server 110 or the terminal device 130 may obtain the required data from the database 120, or may send (e.g., upload) the generated and updated data to the database 120.

[0045] Optionally, the scene 100 may also include a terminal device 130. The terminal device 130 includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc. The embodiments described in the present disclosure can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. Exemplarily, a user can access or use an application on the server 110 through the terminal device 130, send data to it, or receive data from it. For example, a user can instruct the server 110 to start executing a method for determining an image generation model, send sample data for determining an image generation model or a data identifier indicating sample data to the server 110, instruct the server 110 to generate an image using the determined image generation model, receive an image generated by the server 110, and send parameters required in the image generation process to the server 110 through the terminal device 130.

[0046] Two or more of the above-mentioned server 110, terminal device 130, and database 120 can communicate with each other via a network 140. The network 140 can be a wired network connected via cables, optical fibers, etc., or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, etc.

[0047] Figure 2A The following schematically shows an exemplary flow chart of a method 200A for determining an image generation model according to some embodiments of the present disclosure. For example, the method 200A may be executed by a computing device, which may be, for example, Figure 1 The server 110 or the terminal device 130 in the scene 100 shown. Figure 2A As shown, method 200A may include steps 210 to 270, as follows.

[0048] In step 210, a target watermark code may be obtained, wherein the target watermark code may represent watermark information that is expected to be embedded in an image generated by the image generation model. Exemplarily, the watermark information may be identification information of the model owner or user (such as the name, code, etc. of the owner), copyright statement information, or other information that is expected to be embedded in the image. The target watermark code may be a code generated based on the watermark information, such as a binary code, etc.

[0049] In step 220, a sample image may be obtained. Exemplarily, the sample image may be obtained from a local storage, an external storage device, or received from other devices such as a user terminal. Optionally, a public or self-owned image dataset may be used as a sample dataset, and each image in the sample dataset may be regarded as a sample image. Optionally, the image dataset may be constructed according to the requirements of a specific application scenario. For example, if it is desired to use an image generation model to generate images of a certain style or type, images of the corresponding style or type may be selected to construct the image dataset.

[0050] In step 230, a reconstructed image can be generated based on the sample image through the target model, wherein the target model can be configured to generate a reconstructed image based on noisy data related to the sample image. Exemplarily, the sample image can be input into the target model, and its output can be obtained as the reconstructed image. Exemplarily, the target model can be a neural network model to be trained, and it can have different network structures according to the application scenario requirements. For example, it can be constructed based on diffusion model structures such as diffusion model (DM), latent diffusion model (LDM), stable diffusion, etc., or it can also be constructed based on other existing or self-designed model structures. Exemplarily, a sample image or a processed sample image (such as a downsampled sample image) can be subjected to noise processing to obtain noisy data related to the sample image, and the image generated by the target model based on such noisy data can be used as the reconstructed image. In addition, optionally, the target model can also consider other data, such as guide text, guide image, etc., in the process of generating the reconstructed image. Exemplarily, the guidance text or guidance image may contain guidance information related to the image to be reconstructed and may be encoded as a vector and injected into the target model through, for example, an attention mechanism.

[0051] In step 240, an extracted watermark code may be generated based on the reconstructed image through a watermark extraction network. The watermark extraction network may be a pre-trained neural network that may receive an input image and output a watermark code extracted from the input image, and the watermark code may be, for example, a binary code or other forms of code.

[0052] In step 250, a watermark extraction loss may be determined based on the extracted watermark code and the target watermark code. The watermark extraction loss may be configured to characterize the degree of difference between the extracted watermark code and the target watermark code, so that it may be used to control the quality of watermark embedding in the reconstructed image. Optionally, an appropriate loss function may be selected according to specific application requirements to determine the watermark extraction loss. Exemplarily, the watermark extraction loss may be determined based on a loss function such as Binary Cross Entropy (BCE).

[0053] In step 260, the image perceptual loss can be determined based on the reconstructed image and the sample image. The image perceptual loss can be configured to characterize the degree of difference between the reconstructed image and the sample image, so that it can be used to control the image quality of the reconstructed image. Optionally, an appropriate loss function can be selected according to specific application requirements to determine the image perceptual loss. Exemplarily, the image perceptual loss can be determined based on the Watson-VGG perceptual loss, or the image perceptual loss can be determined using loss functions such as LPIPS, SSIM, etc. Optionally, different loss functions can be tried to determine the image perceptual loss, and the best loss function can be selected from them.

[0054] In step 270, the model parameters of the target model may be updated based on the watermark extraction loss and the image perception loss to determine the image generation model. Exemplarily, some or all of the model parameters of the target model may be updated to minimize the watermark extraction loss and the image perception loss as much as possible, or to minimize a combination of the two. Optionally, the model parameters may be updated based on the watermark extraction loss and the image perception loss based on a mechanism such as gradient descent.

[0055] It should be understood that steps 220 to 270 can be iteratively executed until a cutoff condition is reached. The cutoff condition may include at least one of the following: the number of iterations reaches a preset threshold number of times, the watermark extraction loss is less than a preset threshold, the image perception loss is less than a preset threshold, the combination of the watermark extraction loss and the image perception loss is less than a preset threshold, and all sample images in the sample data set have been traversed. After the cutoff condition is reached, illustratively, the image generation model can be determined based on the target model or its partial structure after parameter update. For example, the target model or its partial structure after parameter update can be used as the determined image generation model, or the updated parameters of the target model can be used as the model parameters of the determined image generation model, and the image generation model has a model structure that is the same as the target model or the same as a partial structure of the target model. The determined image generation model can be used to generate an image containing the target watermark encoding.

[0056] The image generation model determined by method 200A can embed the target watermark code while generating the image. On the one hand, this embedded watermark code can be used to declare copyright and determine whether it is AI-generated content. For example, the watermark code can be extracted from an image through a watermark extraction network. If the extracted watermark code is consistent with the target watermark code, the image can be determined to be an image generated using the above-mentioned image generation model, thereby determining the copyright of the image and confirming that the image is AI-generated content. On the other hand, in the watermark embedding method of first generating an image and then adding a watermark, there is a watermark-free image stage. This watermark-free image has a risk of leakage, and when the model for generating the watermark-free image is leaked, it is difficult to locate the theft of the model. In contrast, the image generation model determined by the above method 200A can directly generate a watermarked image, and there is no landed watermark-free image, which helps to avoid the leakage of the watermark-free image and helps to reduce the risk of the image generation model being stolen. On the other hand, since both watermark extraction loss and image perception loss are taken into consideration during the implementation of the above method, the determined image generation model can embed the target watermark code without significantly affecting the quality of the generated image, that is, achieving the invisible watermark effect mentioned above, thereby helping to achieve a balance between image quality and copyright statement.

[0057] To further facilitate understanding, Figure 2B Schematically shows an exemplary block diagram of an architecture 200B for determining an image generation model according to some embodiments of the present disclosure. Figure 2BAs shown, the sample image y can be input into the target model to obtain the reconstructed image y_o. The reconstructed image y_o can be input into the watermark extraction network to obtain the extracted watermark code m'_bits. According to the loss function 1, the watermark extraction loss L_m can be obtained based on the extracted watermark code m'_bits and the target watermark code m_bits. Exemplarily, assuming that the BCE loss function is used as the loss function 1, then L_m=BCE(m_bits,m'_bits)=-(m_bits log(m'_bits)+(1-m_bits)*log(1-m'_bits)). According to the loss function 2, the image perceptual loss L_i can be obtained based on the reconstructed image y_o and the sample image y. As mentioned above, the loss function 2 can be selected according to the specific application scenario. For example, assuming that the target model is a diffusion model, the Watson-VGG perceptual loss commonly used in the training process of such models can be used as the loss function 2. Subsequently, the combined loss L can be obtained based on the combination of the image perception loss Li and the watermark extraction loss Lm, and the model parameters of the target model can be updated according to L. The above operation can be iteratively performed based on different sample images in the sample data set until a cutoff condition is reached, such as L convergence, and the image generation model is determined based on the target model with the final parameter update.

[0058] In some embodiments, the aforementioned step 270 may include: determining a combined loss based on a weighted sum of the image perception loss and the watermark extraction loss; and updating the model parameters of the target model based on the combined loss. Exemplarily, at least one of the image perception loss and the watermark extraction loss may have a corresponding weight factor. By adjusting the weight factor, the proportion of the image perception loss and the watermark extraction loss in the combined loss may be adjusted, thereby adjusting the balance between the watermark strength and the image quality, so as to achieve a better model training effect and a better watermarked image generation effect. Exemplarily, refer to Figure 2B , the merged loss L can be determined as L=L_m+λ*L_i, where λ is a weight adjustment factor, which is used to control the weight of the image perception loss in the merged loss, wherein the smaller the value of λ, the higher the proportion of the watermark extraction loss, the stronger the watermark embedding, and the worse the quality of the generated image. Therefore, the best balance can be achieved between the watermark extraction loss L_m and the image perception loss L_i by adjusting the value of the factor λ. Optionally, the value of λ can be selected from [0.4, 0.05] to achieve a better balance. Alternatively or additionally, a weight adjustment factor can also be set for the watermark extraction loss. In addition, optionally, the above-mentioned weight adjustment factor can be a fixed value or can be dynamically adjusted. For example, the weight adjustment factor can be a preset function, and the weights of the image perception loss and the watermark extraction loss can be adaptively adjusted according to the model training process.

[0059] Figure 3A An exemplary flow chart of a method 300A for determining a watermark extraction network according to some embodiments of the present disclosure is schematically shown. Optionally, the method 300A may be performed by a device or apparatus that performs the method 200A, or may also be performed by other devices or apparatuses. As shown in the figure, the method 300A includes steps 310 to 350, which are specifically as follows.

[0060] In step 310, an input image and an input watermark code may be obtained. The input image may be any image. For example, any public or proprietary image set (such as MS-COCO, etc.) may be used as a sample set for determining a watermark extraction network, wherein each image may be used as the input image. Exemplarily, the input watermark code may be a randomly generated watermark code. Optionally, to be compatible with the determination process of the image generation model, the input watermark code may have the same length as the aforementioned target watermark code. For example, assuming that the target watermark code is a 128-bit binary code, the input watermark code may also be a 128-bit binary code. In addition, optionally, in different iteration rounds, the same input image and different input watermark codes may be used, or different input images and the same watermark code may be used, or different input images and different watermark codes may be used.

[0061] In step 320, a watermarked image may be generated based on the input image and the input watermark code. Exemplarily, the watermarked image may be generated by using a neural network, or by performing various linear and / or nonlinear operations on the input image and the input watermark code.

[0062] In step 330, the watermark code can be extracted based on the watermarked image through the target neural network. According to specific application requirements or application scenarios, the target neural network can adopt different network structures. The target neural network can be configured to output coded data based on the image input to the network, that is, the watermark code mentioned above.

[0063] In step 340, a watermark loss may be determined based on the extracted watermark code and the input watermark code. The watermark loss may be configured to characterize the degree of difference between the extracted watermark code and the input watermark code, and thus may be used to measure the ability of the watermark extraction network to extract the watermark code. Optionally, any applicable loss function may be selected as required to determine the watermark loss. For example, the watermark loss may be determined based on a loss function such as Binary Cross Entropy (BCE).

[0064] In step 350, the network parameters of the target neural network may be updated based on the watermark loss to determine the watermark extraction network. Exemplarily, all or part of the network parameters of the target neural network may be updated to minimize the watermark loss as much as possible. Alternatively, the network parameters of the target neural network may be updated based on the watermark loss based on a mechanism such as gradient descent.

[0065] It should be understood that steps 310 to 350 can be iteratively executed until a cutoff condition is reached. The cutoff condition may include at least one of the following: the number of iterations reaches a preset threshold number of times, and the watermark loss is less than a preset threshold. After the cutoff condition is reached, illustratively, a watermark extraction network can be determined based on the target neural network after parameter update. The determined watermark extraction network can be used in the aforementioned method 200A, and can be used to subsequently extract watermark codes from an image to determine whether the image is generated by AI, and to determine the copyright ownership of the image, such as determining whether the image is generated by the image generation model determined by method 200A.

[0066] In some embodiments, step 330 may include: performing an image attack operation on the watermarked image to generate a processed watermarked image, the image attack operation including at least one of the following: image cropping, image compression, image noise; based on the processed watermarked image, extracting the watermark code through the target neural network. Exemplarily, a set of image attack operations may be predetermined, and the set may include various image attack operations such as image cropping, image compression, and image noise. After the watermarked image is generated, one or more image attack operations may be randomly selected from the set or according to preset rules to process the watermarked image, so that the watermark is robust in common image processing operations (such as cropping, compression, etc.). This in turn helps to improve the robustness of the determined watermark extraction network.

[0067] In some embodiments, step 320 may include: generating a watermark image based on the input image and the input watermark code; generating a watermarked image based on the watermark image and the input image. Exemplarily, the watermark image may be superimposed on the input image, or based on a weighted combination of the two, to generate the watermarked image, or, in some embodiments, an activated watermark image may be generated based on the watermark image through an activation function; and then the watermarked image may be generated based on the weighted sum of the input image and the activated watermark image. Through the activation function, each pixel value in the watermark image may be mapped in a nonlinear manner, for example, to a preset interval. This may adjust the direction and degree of adjustment of each pixel value of the input image when the watermark image is superimposed on the input image. In addition, exemplarily, a weight factor may be set for at least one of the input image and the activated watermark image to control the proportion of the two in the watermarked image. Optionally, the weighted sum of the input image and the activated watermark image can be determined by a weight factor related to the activated watermark image, thereby allowing the image distortion caused by the activated watermark image being superimposed on the input image to be constrained by selecting a suitable weight factor, so as to better control image loss and improve the bit accuracy when extracting the watermark. This helps to obtain a more accurate and robust watermark extraction network.

[0068] In some embodiments, the activation function can be configured to convert the pixel value of each pixel in the watermark image to a preset interval, which is symmetric about the zero point. By selecting an activation function with a value range that is symmetric about the zero point, when the activated watermark image is superimposed on the input image, the degree of modification of the pixel value of the input image in the positive and negative directions can be controlled within the same range, thereby helping to achieve a better watermarking effect, and further helping to train a watermark extraction network with better performance. Optionally, a tanh function or the like can be selected as the above activation function.

[0069] For ease of understanding, Figure 3B Schematically shows an exemplary block diagram of an architecture 300B for determining a watermark extraction network according to some embodiments of the present disclosure. Figure 3BAs shown, an input image x obtained from a sample set (such as the aforementioned MS-COCO) and a randomly generated input watermark code m_bits can be input into a watermark embedding network W_E, and the watermark embedding network W_E can be configured to generate a watermarked image x_w based on the input image and the input watermark code according to the description of the relevant embodiments above, where x_w=W_E(x,m_bits). It should be understood that the input watermark code used in the training process of the watermark extraction network and the aforementioned target watermark code can be different codes. Subsequently, the watermarked image x_w can be input into an image processing module W_N, and the image processing module W_N can be configured to randomly select one or several image attack operations as described in the aforementioned embodiments to transform the watermarked image to obtain a processed watermarked image x_w_n, where x_w_a=Random(W_N(x_w)). Then, the processed watermarked image x_w_n can be input into a watermark extraction network W_D to extract the watermark code m'_bits. The watermark loss L can then be determined based on the input watermark code m_bits and the extracted watermark code m'_bits M Here, the loss function can use BCE or other loss functions, such as L M =BCE(m_bits,m'_bits)=-(m_bits*log(m'_bits))+(1-m_bits)*log(1-m'_bits)). It can be based on the watermark loss L M The network parameters of the watermark extraction network W_D are updated, and the network parameters of the watermark embedding network W_E may also be updated simultaneously. The above process may be iteratively performed until a cutoff condition is met, such as the watermark loss L converges. After the cutoff condition is met, the watermark extraction network W_D with updated parameters may be used as the watermark extraction network used in the aforementioned method 200A. Optionally, in Figure 3B In the training process shown, the image loss can be additionally determined based on the input image and the watermarked image, and the network parameters can be updated based on both the image loss and the watermark loss. However, considering that only the watermark extraction network W_D is used in the end, there is no need to pay attention to the impact of watermark coding embedding on image quality and the strength of watermark traces. Therefore, in order to improve training efficiency and simplify the training process, only watermark loss can be considered without considering image loss.

[0070] Further, Figure 4 Schematically illustrates an exemplary network structure of a watermark network 400 according to some embodiments of the present disclosure. Figure 4 As shown, the watermark network 400 may include a watermark embedding network W_E, an image processing module W_N, and a watermark extraction network W_D. Figure 4As shown, the watermark embedding network W_E can generate a watermark image M based on the input watermark code m_bits and the input image x, and then the watermark image M can be processed by the tanh function to obtain the activated watermark image tanh(M), and then the watermarked image is generated based on the weighted sum of the activated watermark image and the input image, that is, the watermarked image finally output by the watermark embedding network is W_E(x,m_bits)=a*tanh(M)+x, where an appropriate weight factor a can be selected to better control the image loss and improve the network training effect. Exemplarily, the value of the weight factor a can be determined in the interval [0.8,0.2].

[0071] More specifically, Figure 4 The watermark embedding network W_E shown can perform a convolution operation on the input image to obtain an intermediate representation 401, and copy and splice the input watermark code to obtain a message representation 402. The intermediate representation 401 and the message representation 402 can be spliced ​​together, and then a subsequent convolution operation is performed on the spliced ​​representation to obtain a watermark image, which can have the same dimension as the input image. The image processing module W_N can perform random image attack operations on the received watermarked image as described in the previous embodiment, such as image cropping, image compression, image noise, etc. The watermark extraction network W_D can process the processed watermarked image through multiple convolutional layers to obtain an intermediate representation 403, and then pooling can be applied to the intermediate representation 403, such as global spatial average pooling, to obtain a vector 404, which can have the same dimension as the input watermark code. Then a single linear layer can be applied to the vector 404 to obtain the extracted watermark code.

[0072] It should be understood that Figure 4 The network structure shown is merely exemplary, and different network structures may be selected or designed according to specific application requirements.

[0073] In some embodiments, the operation of obtaining the target watermark code in step 210 can be implemented by the following process: obtaining a target watermark including multiple characters; encoding multiple characters in the target watermark; splicing the codes of multiple characters to generate the target watermark code. Exemplarily, the target watermark can be a string with a preset number of bits, such as a 16-bit string. The number of characters in the string can be selected according to application requirements, and the present disclosure is not limited to this. Exemplarily, each character in the string can be encoded separately, for example, ASCII code or other encoding methods can be used to encode each character into a binary representation, and then the converted binary representations of each character can be spliced ​​in sequence to obtain the corresponding target watermark string. Through the above process, a watermark code suitable for use with the model can be quickly obtained based on the watermark string.

[0074] For ease of understanding, Figure 5 The exemplary encoding process 500 of watermark information according to some embodiments of the present disclosure is schematically shown. Figure 5 As shown, a string 510 can be obtained, which includes multiple characters, wherein each circle can represent a character. The multiple characters can be converted into a set of ASCII codes, and then converted into a set of binary representations 520, wherein each box can represent the binary representation corresponding to a character. For example, assuming that the string "AB" is used as the target watermark, the ASCII codes corresponding to the characters are 65 (A) and 66 (B), and the binary representations of 65 (A) and 66 (B) are 01000001 and 01000010, respectively. The binary representations corresponding to the characters can be spliced ​​together in order to obtain the target watermark code 530. Continuing with the aforementioned string "AB" as an example, the target watermark code corresponding to the string "AB" can be 0100000101000010.

[0075] In addition, the above process of obtaining the target watermark code based on the target watermark can also be applied to the acquisition of the input code. For example, the input watermark can be obtained (e.g., randomly generated), and the corresponding input watermark code can be obtained based on the above process. Alternatively, as mentioned above, optionally, the input watermark code can also be directly randomly generated.

[0076] The above process of obtaining the target watermark code based on the target watermark can be implemented with the help of a specific encoder M, that is, the character string m of the target watermark is input into the encoder M, and the output M(m) of the encoder M is obtained as the target watermark code m_bits, that is, m_bits=M(m). In addition, based on the opposite logic, the watermark code extracted from the image using the watermark extraction network can be reversely mapped to the corresponding watermark character string, so as to identify the watermark information represented by the extracted watermark code.

[0077] In some embodiments, the aforementioned target model may include an encoder, a diffusion network, and a decoder. In such an embodiment, the aforementioned step 230 may include: generating a latent space vector based on the sample image through an encoder; generating a reconstruction vector based on the latent space vector through a diffusion network, wherein the diffusion network is configured to generate a reconstruction vector based on noisy data related to the latent space vector; and generating a reconstructed image based on the reconstruction vector through a decoder. Exemplarily, noisy data related to the latent space vector can be obtained by adding different levels of noise to the latent space vector, and the diffusion network can be configured to generate a reconstruction vector based on such noisy data, for example, by noise prediction, and subtracting the predicted noise from the noisy data to obtain the reconstruction vector. Exemplarily, the encoder can be configured to downsample the sample image to generate the corresponding latent space vector; the decoder can be configured to upsample the reconstruction vector to generate the corresponding reconstructed image. Thus, the amount of data required to be processed by the diffusion network can be reduced, which helps to improve the overall model training efficiency and helps to improve the subsequent image generation efficiency.

[0078] In some embodiments, in order to further improve the training efficiency, the parameters of the encoder can be fixed. For example, the parameters of the encoder can be obtained in the pre-training process and used in the determination process of the image generation model described in the above embodiments. In such embodiments, when executing the aforementioned step 270, only the parameters of the diffusion network and the decoder can be updated based on the watermark extraction loss and the image perception loss, while the encoder parameters remain unchanged, and after the cutoff condition is reached, the image generation model can be determined based on both the diffusion network and the decoder.

[0079] For ease of understanding, Figure 6 Taking latent diffusion models (LDMs) as an example, the training process 600 of the image generation model according to the above embodiment is schematically shown. Figure 6As shown, LDMs mainly includes an encoder LDMs_E, a latent space diffusion model LDMs_Latent, and a decoder LDMs_D. As mentioned above, the training of LDMs_E can be completed at a very early step, and can be reused in different pre-training processes and has fixed parameters. Therefore, the following process is mainly used to train LDMs_Latent and LDMs_D. Exemplarily, an appropriate training set can be selected according to the actual scenario, and a watermark string m can be obtained as a target watermark, and the string m can be a 16-bit or other bit string. The watermark string m can be encoded into a binary code m_bits=(m_bits_1,...,m_bits_i...,m_bits_128) according to the encoding process described in the previous embodiment, m_i∈{0,1}. In each iteration round of the training process, a sample image y can be obtained in the training set and input into the encoder LDMs_E with fixed parameters, and a latent space vector z can be output, that is, z=LDMs_E(y), y∈h*w*c. Next, the latent space vector z is provided to LDMs_Latent for latent space vector reconstruction, and the reconstructed vector z' is output, i.e., z'=LDMs_Latent(z). The latent space vector z' is sent to the decoder LDMs_D for explicit space image reconstruction, and the reconstructed image is y_o, i.e., y_o=LDMs_D(z'). y_o is sent to the watermark extraction network W_D to extract the watermark information m'_bits, i.e., m'_bits=W_D(y_o). The watermark extraction network W_D can be based on the previous reference FIG. 3A to FIG. 4 The various embodiments described are predetermined. Then, the watermark extraction loss L_m and the image perception loss L_i can be calculated. In this embodiment, the BCE loss between m_bits and m'_bits can be used as the watermark extraction loss, that is, L_m=BCE(m_bits,m'_bits)=-(m_bits*log(m'_bits))+(1-m_bits)*log(1-m'_bits)); Watson-VGG perceptual loss can be used as the image perception loss L_i, or LPIPS, SSIM and other loss functions can be selected according to specific application requirements. Subsequently, the final loss function can be calculated as L=L_m+λ*L_i, where λ is a weight adjustment function. During optimization, the value of the factor λ can be adjusted to achieve the best balance between the watermark extraction loss L_m and the image perception loss L_i. L can be used as the optimization target, the training set can be traversed, and the above steps can be iteratively performed until L converges and the training is completed. After the training is completed, the trained LDMs_Latent and LDMs_D can be retained as the determined image generation model for subsequent image generation tasks.

[0080] It should be understood that although Figure 6LDMs are used, but the various embodiments described in the present disclosure are also applicable to different diffusion models. Those skilled in the art can apply the aforementioned method for determining an image generation model to models of different structures according to the description of the present disclosure.

[0081] Figure 7 The following schematically shows an exemplary flow chart of an image generation method 700 according to some embodiments of the present disclosure. Exemplarily, the method 700 may be executed by a computing device, which may be, for example, Figure 1 The server 110 or the terminal device 130 in the scenario 100 shown. Optionally, the execution subject of the method 700 may be the same as or different from the execution subject of the method 200A. Figure 7 As shown, method 700 may include steps 710 to 720, which are specifically as follows.

[0082] In step 710, starting image data may be obtained, wherein the starting image data includes noise data. In step 720, an output image may be generated based on the starting image data through an image generation model, wherein the image generation model may be determined by the method 200A described according to various embodiments above. The image generated by the method may contain an embedded target watermark code, and such watermark embedding may be used to declare copyright.

[0083] For example, Gaussian noise (or other noise data) can be used as the starting image data, and the dimension of the Gaussian noise can correspond to the image dimension used in the image generation model training process. For example, for a general diffusion model, its dimension can be equal to the dimension of the sample image used in method 200A; for a potential diffusion model, its dimension can be equal to the dimension of the latent space vector output by the encoder; and so on. For example, if using Figure 6 The model structure shown, and the determined image generation model includes LDMs_Latent and LDMs_D, then Gaussian noise n with appropriate dimensions can be sent to LDMs_Latent and LDMs_D, and an image y_o=LDMs_D(LDMs_Latent(n)) containing the target watermark code can be obtained. When copyright identification is required to determine whether the image is AI generated, the image can be sent to the watermark extraction network W_D to extract the watermark code m_bits=W_D(y_o). According to the encoding process described in the previous embodiment, the extracted watermark code is reversely mapped into a watermark string, thereby completing the extraction of the watermark information, that is, m=M'(m_bits), where M' represents the reverse mapping process of the mapping process M of encoding the watermark string into the watermark code.

[0084] It should be understood that the method 700 may have the same or similar features and advantages as the method 200A described above, and may have corresponding embodiments, which will not be described in detail for the sake of brevity.

[0085] Figure 8 An exemplary block diagram of an apparatus 800 for determining an image generation model according to some embodiments of the present disclosure is schematically shown. As shown in the figure, the apparatus 800 includes a first acquisition module 810, a second acquisition module 820, a reconstructed image generation module 830, a watermark extraction module 840, a first loss determination module 850, a second loss determination module 860 and a parameter update module 870.

[0086] Specifically, the first acquisition module 810 can be configured to: acquire a target watermark code, wherein the target watermark code represents the watermark information expected to be embedded in the image generated by the image generation model; the second acquisition module 820 can be configured to: acquire a sample image; the reconstructed image generation module 830 can be configured to: generate a reconstructed image based on the sample image through a target model, wherein the target model is configured to generate a reconstructed image based on noisy data related to the sample image; the watermark extraction module 840 can be configured to: generate an extracted watermark code based on the reconstructed image through a watermark extraction network; the first loss determination module 850 can be configured to: determine the watermark extraction loss based on the extracted watermark code and the target watermark code; the second loss determination module 860 can be configured to: determine the image perception loss based on the reconstructed image and the sample image; the parameter updating module 870 can be configured to: update the model parameters of the target model based on the watermark extraction loss and the image perception loss to determine the image generation model.

[0087] Fig. 9 An exemplary block diagram of an image generating device 900 according to some embodiments of the present disclosure is schematically shown. As shown in the figure, the device 900 includes an acquisition module 910 and a generation module 920.

[0088] Specifically, the acquisition module 910 can be configured to: acquire starting image data, wherein the starting image data includes noise data; the generation module 920 can be configured to: generate an output image based on the starting image data through an image generation model, wherein the image generation model is determined using the device 800.

[0089] It should be understood that the apparatus 800 and the apparatus 900 can be implemented in software, hardware or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0090] In addition, the device 800 can be used to implement the method 200A described above, and the device 900 can be used to implement the method 700 described above, and the relevant details have been described in detail above, and for the sake of brevity, they will not be repeated here. The device 800 and the device 900 can have the same features and advantages as described about the above methods.

[0091] Fig.10 Schematically shows an exemplary block diagram of a computing device 1000 according to some embodiments of the present disclosure. For example, it can be Figure 1 The server 110 or the terminal device 130 in.

[0092] As shown, the example computing device 1000 includes a processing system 1001, one or more computer-readable storage media 1002, and one or more I / O interfaces 1003 that are communicatively coupled to each other. Although not shown, the computing device 1000 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or more combinations of the following different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include such as control and data lines.

[0093] Processing system 1001 represents the functionality of performing one or more operations using hardware. Thus, processing system 1001 is illustrated as including hardware elements 1004 that may be configured as processors, functional blocks, etc. This may include implementing application specific integrated circuits in hardware or other logic devices formed using one or more semiconductors. Hardware elements 1004 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0094] The computer-readable storage medium 1002 is illustrated as including a memory / storage device 1005. The memory / storage device 1005 represents a memory / storage device associated with one or more computer-readable storage media. The memory / storage device 1005 may include a volatile storage medium (such as a random access memory (RAM)) and / or a non-volatile storage medium (such as a read-only memory (ROM), flash memory, an optical disk, a magnetic disk, etc.). The memory / storage device 1005 may include fixed media (e.g., RAM, ROM, a fixed hard drive, etc.) and removable media (e.g., flash memory, a removable hard drive, an optical disk, etc.). Exemplarily, the memory / storage device 1005 may be used to store the various watermark data, image data, etc. mentioned in the above embodiments. The computer-readable storage medium 1002 may be configured in various other ways as further described below.

[0095] One or more input / output interfaces 1003 represent functionality that allows a user to enter commands and information into the computing device 1000 and also allows information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., a visible or invisible wavelength (such as infrared frequency) can be used to detect motion that does not involve touch as a gesture), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a display or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, the input device can allow the user to input watermark information, provide sample images, indicate the start of model training, and the like, and the output device can allow the user to view the generated image, the determined model parameters, and the like.

[0096] The computing device 1000 also includes an image generation model determination application or image generation application 1006. The image generation model determination application or image generation application 1006 can be stored as computer program instructions in the memory / storage device 1005. The image generation model determination application or image generation application 1006 can be implemented together with the processing system 1001 or the like with respect to Figure 8 The device 800 described or related Fig. 9 The overall functions of each module of the device 900 are described.

[0097] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Typically, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," etc., as used herein, generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0098] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable storage media. Computer-readable storage media may include various media accessible by computing device 1000. By way of example and not limitation, computer-readable storage media may include “computer-readable storage media” and “computer-readable signal media”.

[0099] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0100] "Computer readable signal media" refers to signal bearing media that is configured to send instructions to the hardware of computing device 1000, such as via a network. Signal media typically can embody computer executable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave, data signal or other transmission mechanism. Signal media also includes any information transfer media. By way of example and not limitation, signal media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared and other wireless media.

[0101] As previously described, hardware elements 1004 and computer-readable storage media 1002 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements may include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0102] The aforementioned combination may also be used to implement various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1004. Computing device 1000 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 1004 of a processing system, a module may be implemented as a module executable by computing device 1000 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1000 and / or processing system 1001 to implement the techniques, modules and examples described herein.

[0103] The techniques described herein may be supported by these various configurations of computing device 1000 and are not limited to the specific examples of the techniques described herein.

[0104] It should be understood that, for the sake of clarity, embodiments of the present disclosure are described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality that is described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to a suitable unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0105] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the above-mentioned method for determining an image generation model or an image generation method is implemented.

[0106] The present disclosure provides a computer program product or a computer program, which includes computer executable instructions, which are stored in a computer-readable storage medium. A processor of a computing device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the computing device executes the method for determining an image generation model or the image generation method provided in the above various embodiments.

[0107] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0108] It is understandable that when practicing the technical solutions described in various embodiments of the present disclosure, the collection and processing of relevant data such as sample images should be strictly in accordance with the requirements of relevant national laws and regulations in actual application, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

Claims

1. A method for determining an image generation model, comprising: Acquire a target watermark code, wherein the target watermark code represents watermark information that is expected to be embedded in an image generated by the image generation model; Get a sample image; Based on the sample image, generating a reconstructed image by using a target model, wherein the target model is configured to generate the reconstructed image according to noisy data associated with the sample image; Based on the reconstructed image, generating an extracted watermark code through a watermark extraction network; determining a watermark extraction loss based on the extracted watermark code and the target watermark code; Determining image perception loss based on the reconstructed image and the sample image; as well as Based on the watermark extraction loss and the image perception loss, the model parameters of the target model are updated to determine the image generation model.

2. The method according to claim 1, wherein: The watermark extraction network is determined by the following steps: Get input image and input watermark code; generating a watermarked image based on the input image and the input watermark code; Based on the watermarked image, extracting the watermark code through a target neural network; determining a watermark loss based on the extracted watermark code and the input watermark code; as well as Based on the watermark loss, the network parameters of the target neural network are updated to determine the watermark extraction network.

3. The method according to claim 2, wherein: The extracting of watermark code based on the watermarked image through a target neural network comprises: performing an image attack operation on the watermarked image to generate a processed watermarked image, the image attack operation comprising at least one of the following: image cropping, image compression, image noise addition; and Based on the processed watermarked image, the watermark code is extracted through a target neural network.

4. The method according to claim 2, wherein: The step of generating a watermarked image based on the input image and the input watermark code comprises: generating a watermark image based on the input image and the input watermark code; Based on the watermark image, generating an activated watermark image through an activation function; and The watermarked image is generated based on a weighted sum of the input image and the activated watermark image.

5. The method according to claim 4, wherein: The activation function is configured to convert the pixel value of each pixel in the watermark image into a preset interval, and the preset interval is symmetric about the zero point.

6. The method according to claim 1, wherein: The updating of the model parameters of the target model based on the watermark extraction loss and the image perception loss comprises: determining a merging loss based on a weighted sum of the image perception loss and the watermark extraction loss; and Based on the combined loss, model parameters of the target model are updated.

7. The method according to claim 1, wherein: The obtaining of the target watermark code comprises: Acquire a target watermark, wherein the target watermark includes a plurality of characters; encoding a plurality of characters in the target watermark; and The codes of the multiple characters are concatenated to generate the target watermark code.

8. The method according to claim 1, wherein: The target model includes an encoder, a diffusion network and a decoder, and Wherein, generating a reconstructed image based on the sample image through a target model comprises: Based on the sample image, generating a latent space vector through the encoder; Based on the latent space vector, generating a reconstruction vector through the diffusion network, wherein the diffusion network is configured to generate the reconstruction vector according to noisy data associated with the latent space vector; and Based on the reconstruction vector, the reconstructed image is generated by the decoder.

9. The method according to claim 8, wherein: The encoder has fixed parameters, and Wherein, updating the model parameters of the target model based on the watermark extraction loss and the image perception loss to determine the image generation model includes: Based on the watermark extraction loss and the image perception loss, parameters of the diffusion network and the decoder are updated to determine the image generation model.

10. A method for generating an image, comprising: Acquiring initial image data, wherein the initial image data includes noise data; as well as Based on the starting image data, an output image is generated by an image generation model, wherein the image generation model is determined by the method according to any one of claims 1 to 9.

11. A device for determining an image generation model, comprising: A first acquisition module is configured to: acquire a target watermark code, wherein the target watermark code represents watermark information that is expected to be embedded in the image generated by the image generation model; The second acquisition module is configured to: acquire a sample image; A reconstructed image generation module is configured to: generate a reconstructed image based on the sample image through a target model, wherein the target model is configured to generate the reconstructed image according to noisy data related to the sample image; The watermark extraction module is configured to: generate an extracted watermark code based on the reconstructed image through a watermark extraction network; A first loss determination module is configured to: determine a watermark extraction loss based on the extracted watermark code and the target watermark code; A second loss determination module is configured to: determine an image perception loss based on the reconstructed image and the sample image; as well as The parameter updating module is configured to update the model parameters of the target model based on the watermark extraction loss and the image perception loss to determine the image generation model.

12. An image generating device, comprising: An acquisition module is configured to: acquire initial image data, wherein the initial image data includes noise data; as well as A generation module is configured to generate an output image based on the starting image data by using an image generation model, wherein the image generation model is determined using the apparatus according to claim 13.

13. A computing device comprising: a memory configured to store computer-executable instructions; as well as A processor configured to perform the method according to any one of claims 1 to 10 when the computer executable instructions are executed by the processor.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to any one of claims 1 to 10.

15. A computer program product comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images