Image generation method and system, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present disclosure provide an image generation method and system, an electronic device, and a computer-readable storage medium. The method comprises: acquiring an initial image to be processed and text information corresponding to said initial image, wherein the text information is used for describing the semantics of a target image into which the initial image is to be converted; using an image generation model to extract an initial image feature from the initial image, analyzing the initial image feature and the text information, and converting the initial image feature into a target image feature satisfying the text information, wherein the image generation model is obtained by training a text-to-image model that has learned watermark information; and using the image generation model to add the watermark information to the target image feature to obtain a target image comprising the watermark information. The present disclosure solves the technical problem of significant limitations in embedding watermark information into images.

Description

Image generation method, system, electronic device and computer readable storage medium

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to a Chinese patent application No. 202410878347X, filed on July 1, 2024, and entitled "Image generation method, system, electronic device and computer readable storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the field of image processing, in particular to the technical field of image recognition, image filling, and the like, and more particularly to an image generation method, system, electronic device and computer readable storage medium. BACKGROUND

[0004] At present, with the wide application of images in different fields, the protection of their copyright becomes particularly important. In order to protect the copyright information of images, the images can be watermarked. However, in the existing watermarking process, only how to implant the generated watermark into the image is concerned, and the corresponding watermark is not generated according to the requirements of the field, thereby resulting in the technical problem of great limitation of implanting watermark information into the image.

[0005] At present, there is no effective solution to the above technical problem of great limitation of implanting watermark information into the image. SUMMARY

[0006] The embodiments of the present disclosure provide an image generation method, system, electronic device and computer readable storage medium to at least solve the technical problem of great limitation of implanting watermark information into the image.

[0007] According to an aspect of an embodiment of the present disclosure, an image generation method is provided. The method can include: obtaining an initial image to be processed, and text information corresponding to the initial image, wherein the text information is used to describe the semantics of a target image to be converted from the initial image; using an image generation model to extract initial image features from the initial image, and analyzing the initial image features and the text information to convert the initial image features into target image features satisfying the text information, wherein the image generation model is obtained by training a text-to-image model with watermark information; using the image generation model to add watermark information to the target image features to obtain a target image including the watermark information.

[0008] According to another aspect of the embodiments of the present disclosure, a method for generating a model is provided. The method can include: obtaining an initial image sample and a target image sample, wherein the target image sample is used to represent at least the initial image sample including watermark information; training a text-to-image model using the initial image sample and the target image sample to obtain a text-to-image model learned with watermark information; and training the text-to-image model learned with watermark information using at least the initial image sample and a text information sample to obtain an image generation model, wherein the text information sample is used to describe semantics of the target image sample, and the image generation model is used to extract initial image features from an input initial image, analyze the initial image features and text information corresponding to the initial image, convert the initial image features into target image features satisfying the text information, and add watermark information to the target image features to obtain a target image including the watermark information, and the text information is used to describe semantics of the target image to be converted from the initial image.

[0009] According to still another aspect of the embodiments of the present disclosure, a method for generating an image is provided. The method can include: obtaining an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe semantics of a style of a target image to be converted from the initial image; extracting initial image features from the initial image using a text-to-image model, and analyzing the initial image features and the style text information to convert the initial image features into target image features satisfying the style text information, wherein the text-to-image model is obtained by training a text-to-image model learned with watermark information; and adding watermark information to the target image features using the text-to-image model to obtain a target image.

[0010] According to still another aspect of the embodiments of the present disclosure, a system for generating an image is provided. The system can include: an information input end configured to obtain an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe semantics of a target image to be converted from the initial image; an image generation end configured to extract initial image features from the initial image using an image generation model, and analyze the initial image features and the text information to convert the initial image features into target image features satisfying the text information, wherein the image generation model is obtained by training a text-to-image model learned with watermark information; add watermark information to the target image features using the image generation model to obtain a target image including the watermark information; and an information output end configured to output the target image.

[0011] According to another aspect of the embodiments of the present disclosure, an electronic device is also provided. The electronic device can include a memory and a processor. The memory stores an executable program. The processor is configured to execute the executable program. When the executable program is executed by the processor, the image generation method of any one of the above is implemented.

[0012] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is also provided, including a stored executable program, wherein the executable program controls the device where the computer readable storage medium is located to perform the image generation method in various embodiments of the present disclosure when the executable program is running.

[0013] According to another aspect of the embodiments of the present disclosure, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the image generation method in various embodiments of the present disclosure.

[0014] According to another aspect of the embodiments of the present disclosure, a computer program product is also provided, including a non-volatile computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the image generation method in various embodiments of the present disclosure.

[0015] According to another aspect of the embodiments of the present disclosure, a computer program is also provided, which, when executed by a processor, implements the image generation method in various embodiments of the present disclosure.

[0016] In the embodiments of the present disclosure, when the image generation is performed, the initial image to be processed and the text information corresponding to the initial image are acquired, that is, the initial image and the semantics of the target image to be converted from the initial image are acquired. Then, the initial image features can be extracted from the initial image by using the image generation model, and the initial image features and the text information are analyzed, so that the initial image features can be converted into the target image features meeting the text information, and the watermark information is added to the converted target image features by using the image generation model, so that the target image including the watermark information can be obtained. Since in the process of image generation, not only the text information corresponding to the initial image is acquired, but also the acquired initial image and the acquired text information are analyzed, and thus the target image including the watermark information is generated, the purpose of protecting the copyright information of the image can be achieved, thereby realizing the technical effect of reducing the limitation of implanting the watermark information into the image, and further solving the technical problem of large limitation of implanting the watermark information into the image.

[0017] It is easy to note that the above general description and the following detailed description are only for exemplifying and explaining the present disclosure, and do not constitute a limitation on the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings described herein are used to provide further understanding of the present disclosure, and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure, and do not constitute an improper limitation on the present disclosure. In the drawings:

[0019] FIG. 1 is a schematic diagram of an application scenario of a method for generating an image according to an embodiment of the present disclosure;

[0020] FIG. 2 is a flowchart of a method for generating an image according to an embodiment of the present disclosure;

[0021] FIG. 3 is a flowchart of a method for generating a model according to an embodiment of the present disclosure;

[0022] FIG. 4 is a flowchart of another method for generating an image according to an embodiment of the present disclosure;

[0023] FIG. 5 is a schematic diagram of a system for generating an image according to an embodiment of the present disclosure;

[0024] FIG. 6 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating an image according to an embodiment of the present disclosure;

[0025] FIG. 7 is a structure block diagram of a computing environment for a method for generating an image according to an embodiment of the present disclosure;

[0026] FIG. 8 is a structure block diagram of a service mesh for a method for generating an image according to an embodiment of the present disclosure;

[0027] FIG. 9(a) is a flowchart of a method for generating a diffusion model-based text-to-image watermarking and copyright protection according to an embodiment of the present disclosure;

[0028] FIG. 9(b) is a schematic diagram of a text-to-image watermarking model based on a generated diffusion model according to an embodiment of the present disclosure;

[0029] FIG. 10 is a schematic diagram of an image generation apparatus according to an embodiment of the present disclosure;

[0030] FIG. 11 is a schematic diagram of a model determination apparatus according to an embodiment of the present disclosure;

[0031] FIG. 12 is a schematic diagram of another image generation apparatus according to an embodiment of the present disclosure;

[0032] FIG. 13 is a block diagram of an electronic device for a method for generating an image according to an embodiment of the present disclosure;

[0033] FIG. 14 is a structure block diagram of a computer terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable personnel in the technical field to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present disclosure.

[0035] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0036] First, some nouns or terms that appear in the process of describing the embodiments of the present disclosure are applicable to the following explanations:

[0037] A text-to-image model can be used for image generation, for example, a user describes a picture generated by his / her demand using natural language. Using the text-to-image model, the corresponding picture can be generated according to the user's demand. The Stable Diffusion model can be one of the above-mentioned text-to-image models. The Stable Diffusion model can be used to generate high-quality and realistic images.

[0038] Watermark information can be used to represent data with special encoding information implanted in an image. The form of watermark information can be, but is not limited to, at least one of the following forms: natural language, two-dimensional code, patent authorization number, etc.

[0039] A backdoor attack can be used to represent a method of tampering with training data. For example, in the case where a third-party model is trained using images that do not have its copyright, a backdoor attack is performed on the third-party model, that is, the images used by the third-party model that do not have its copyright are tampered with, thereby causing the accuracy of the training result of the third-party model to be too low. The third-party model can be used to represent a model trained using images that do not have its copyright.

[0040] According to the embodiments of the present disclosure, a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0041] Considering that the model parameter amount of a large model is large, and the operation resource of a mobile terminal is limited, the above-mentioned method for generating an image provided by the embodiments of the present disclosure can be applied to the application scenario as shown in FIG. 1, but is not limited thereto. In the application scenario as shown in FIG. 1, the method for generating an image is deployed in a server 10, which can be a cloud. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 herein can include, but are not limited to, smartphones, tablet computers, notebook computers, palm computers, personal computers, smart home devices, vehicle-mounted devices, etc., and the client devices 20 collectively constitute a client opposite the server 10. On the graphical user interface of the client device 20, an operation interface for displaying a target image can be deployed, which can be an operation interface of an e-commerce platform. The client device 20 can interact with the user through the graphical user interface to realize the calling of the method for generating an image, and thus realize the method for generating an image provided by the embodiments of the present disclosure.

[0042] In the embodiments of the present disclosure, the system composed of the client device and the server can execute the following steps: when generating an image, the user can use a mobile phone, a desktop computer, a notebook computer, or other client devices to communicate data with the server through a network. After the server receives the identification instruction sent by the user through the client device, the server can execute step S102 to obtain an initial image to be processed and text information corresponding to the initial image, and can execute step S104 to extract initial image features from the initial image by using an image generation model, and analyze the initial image features and the text information, convert the initial image features into target image features satisfying the text information, and can execute step S106 to add watermark information to the target image features by using the image generation model to obtain a target image including the watermark information. In the above process, after the server receives the display instruction sent by the user through the client device, the server can send the target image to the client device. It should be noted that in the case that the running resources of the client device can meet the deployment and running conditions of the method for generating an image, the embodiments of the present disclosure can be performed in the client device.

[0043] Under the above running environment, the disclosure provides a method for generating an image as shown in FIG. 2. FIG. 2 is a flowchart of a method for generating an image according to an embodiment of the disclosure, as shown in FIG. 2, the method can include the following steps:

[0044] In step S202, an initial image to be processed is obtained, and text information corresponding to the initial image is obtained.

[0045] In the technical solution provided in step S202 of the disclosure, the text information can be used to describe the semantics of the target image to be converted from the initial image. For example, the text information can be the style of the target image to be converted from the initial image. The style can include at least one of the following types: film style, movie style, Japanese style, Korean style, and black and white style, etc. The initial image can include at least one of the following: original style image and watermark-free image, etc. The initial image can include at least one of the following: object and background where the object is located, etc. This is only an example and is not limited.

[0046] In this embodiment, the initial image to be processed is obtained, and the text information corresponding to the initial image is obtained. For example, according to the user's demand, the size of the initial image is cropped, the format of the cropped initial image is converted, and the converted initial image is determined as the initial image to be processed. According to the user's demand, the semantics of the target image to be converted from the initial image can be directly obtained. For example, based on the determined style, the style is determined as the style of the target image to be converted from the initial image. This is only an example and is not limited.

[0047] It should be noted that the size and format of the initial image and the text information can be set according to the user's demand. For example, if the user's demand is that the size of the initial image is 1920x1080, the format of the initial image is a vector graph, and the style is a film style, then the size of the initial image is set to 1920x1080, the format of the initial image is set to a vector graph, and the style of the target image to be converted is set to a film style. This is only an example and is not limited.

[0048] In step S204, an initial image feature is extracted from the initial image using an image generation model, and the initial image feature and the text information are analyzed to convert the initial image feature into a target image feature that meets the text information.

[0049] In the technical solution provided in step S204 of the disclosure, the image generation model can be a text-to-image model trained by learning watermark information, wherein the text-to-image model can be a diffusion model.

[0050] After obtaining the initial image to be processed and the text information corresponding to the initial image, the initial image features can be extracted from the initial image by using the image generation model, and the initial image features and the text information can be analyzed to convert the initial image features into target image features that meet the text information.

[0051] In this embodiment, the image generation model described above can be trained by a text-to-image model that learns watermark information. The watermark information can be used to represent data information with special encoding information implanted in the image. For example, the text-to-image model can be a diffusion model, which can be a stable diffusion model. The watermark information can be presented in at least one of the following forms: natural language, two-dimensional code, patent number, etc. This is only an example and is not limited in particular.

[0052] After obtaining the initial image to be processed and the text information corresponding to the initial image, the initial image features can be extracted from the initial image by using the image generation model, and the initial image features and the text information can be analyzed to convert the initial image features into target image features that meet the text information. For example, the initial image is input into the image generation model for feature extraction, and the extracted initial image features and the text information are input into the image generation model for analysis, so that the extracted initial image features can be converted into target image features that meet the text information.

[0053] In this embodiment, the initial image features and the text information are analyzed to convert the initial image features into target image features that meet the text information. For example, the extracted initial image features and the text information are preprocessed by using the image generation model, the preprocessed initial image features and the text information are encoded, and the encoded initial image features and the text information are decoded, so that the encoded initial image features can be converted into target image features that meet the text information.

[0054] It should be noted that the preprocessing operation described above can include at least one of the following operations: denoising operation, enhancement operation, smoothing operation, and grayscale operation, etc. This is only an example and is not limited in particular. The method of obtaining the target image features is only an example and is not limited in particular. As long as the image generation model can be used to analyze the initial image and the text information to generate target image features that meet the text information, the process and method are within the protection scope of the embodiments of the present disclosure.

[0055] In step S206, watermark information is added to the target image features by using the image generation model to obtain a target image including the watermark information.

[0056] In the technical solution provided in the step S206 of the present disclosure, after the initial image features are converted into the target image features satisfying the text information, the image generation model is used to add watermark information to the converted target image features, so that the target image including the watermark information can be obtained, thereby achieving the technical effect of reducing the limitation of implanting watermark information into the image.

[0057] In this embodiment, the target image can be used to represent an image with watermark information, the size of the target image can be the same as that of the cropped initial image, and the style of the target image can be the same as the style required by the user. For example, if the size of the cropped initial image is 1179x2256, the size of the target image can be 1179x2256, and if the style required by the user is a movie style, the style of the target image can be a movie style. This is only an example and is not limited in particular.

[0058] Through the steps S202 to S206 of the present disclosure, when the image is generated, the initial image to be processed and the text information corresponding to the initial image are obtained, that is, the initial image and the semantics of the target image to be converted from the initial image are obtained. Then, the initial image features can be extracted from the initial image by using the image generation model, and the extracted initial image features and the text information are analyzed, so that the initial image features can be converted into the target image features satisfying the text information. In addition, the image generation model is used to add watermark information to the converted target image features, so that the target image including the watermark information can be obtained. Since the text information corresponding to the initial image is obtained during the image generation process, and the obtained initial image and text information are analyzed to generate the target image including the watermark information, the purpose of protecting the copyright information of the image can be achieved, thereby achieving the technical effect of reducing the limitation of implanting watermark information into the image, and further solving the technical problem of large limitation of implanting watermark information into the image.

[0059] The above method of this embodiment will be further introduced as follows.

[0060] As an optional implementation, the initial image sample and the target image sample are used to make the text-to-image model learn watermark information, the target image sample is used to at least represent the initial image sample including the watermark information, the image generation model is obtained by training the text-to-image model learning the watermark information at least by using the initial image sample and the text information sample, and the text information sample is used to describe the semantics of the target image sample.

[0061] In this embodiment, the image generation model can be trained based on an initial image sample, a target image sample, and a text information sample. The target image sample can be used to represent the initial image sample including at least watermark information. The watermark information can be used to represent data information with special encoding information implanted in the image. The text information sample can be used to describe the semantics of the target image sample. For example, the initial image sample can be an image used to train the text-to-image model, which can include at least a raw style image and a non-watermarked image. This is only an example and is not limited in a specific manner.

[0062] As an optional implementation, in step S204, the initial image features are extracted from the initial image by using the image generation model, and the initial image features and the text information are analyzed to convert the initial image features into target image features satisfying the text information, including: extracting the initial image features from the initial image by using a feature extraction model in the image generation model; and converting the initial image features into target image features satisfying the text information by using a noise prediction model in the image generation model.

[0063] In this embodiment, the feature extraction model can be obtained by training a variational auto encoder (VAE), and the noise prediction model can be obtained by training a convolutional neural network model. For example, the noise prediction model can be obtained by training a semantic segmentation model (UNet model), but is not limited thereto.

[0064] In this embodiment, after obtaining the initial image to be processed and the text information corresponding to the initial image, the initial image features are extracted from the initial image by using the feature extraction model in the image generation model, and the initial image features are converted into target image features satisfying the text information by using the noise prediction model in the image generation model. For example, the initial image is input into the feature extraction model in the image generation model for feature extraction, and the initial image features can be extracted from the input initial image. Based on the extracted initial image features, the noise prediction model is used to convert the extracted initial image features according to the input text information, so that the target image features satisfying the text information can be obtained.

[0065] Optionally, the noise prediction model is used to convert the extracted initial image features according to the input text information, so that the target image features satisfying the text information can be obtained. For example, the UNet model is used to convert the extracted initial image features according to the input style (for example, Japanese style), so that the target image features satisfying the Japanese style can be obtained. This is only an example and is not limited in a specific manner.

[0066] It should be noted that the above method of obtaining the target image feature is only for illustration, and is not specifically limited herein. As long as the process and method of using the image generation model to analyze the initial image and the text information, and converting the initial image feature into the target image feature satisfying the text information, are within the protection scope of the embodiments of the present disclosure.

[0067] As an optional implementation, the image generation model is used to add watermark information to the target image feature to obtain a target image including the watermark information, including: using a noise prediction model in the image generation model to add noise information of the watermark information to the target image feature to obtain the target image.

[0068] In this embodiment, after the initial image feature is converted into the target image feature satisfying the text information, the noise prediction model in the image generation model is used to add noise information of the watermark information to the target image feature to obtain the target image. For example, the noise prediction model in the image generation model is used to output noise information of the watermark information to be added to the target image feature, and the output noise information of the watermark information is added to the converted target image feature, so that the target image can be obtained. This is only for illustration, and is not specifically limited herein.

[0069] As an optional implementation, the noise prediction model in the image generation model is used to add noise information of the watermark information to the target image feature to obtain the target image, including: performing noise adding processing on the target image feature; using the noise prediction model to perform noise prediction on the target image feature after the noise adding processing to obtain predicted noise information; and using the predicted noise information to convert the target image feature into a target image including noise information of the watermark information.

[0070] In this embodiment, the predicted noise information can include noise information of the watermark information, and the noise prediction model can be, but is not limited to, a UNet model.

[0071] In this embodiment, the target image feature satisfying the text information is subjected to noise adding processing, and the target image feature after the noise adding processing is input into the noise prediction model for noise prediction to obtain predicted noise information including noise information of implicit watermark information. The target image feature after the noise adding processing is converted using the obtained predicted noise information to obtain a target image including noise information of the implicit watermark information.

[0072] Optionally, the target image features meeting the film style are subjected to noise adding processing, and the target image features meeting the film style after the noise adding processing are input into the UNet model for noise prediction to obtain predicted noise information including the noise information of the implicit watermark information. The target image including the noise information of the implicit watermark information can be obtained by converting the target image features meeting the film style after the noise adding processing by using the obtained predicted noise information. The implicit watermark information can be directly displayed in the target image.

[0073] As an optional implementation, the predicted noise information is carried by the target image features, and the target image including the noise information of the watermark information is obtained by converting the target image features by using the predicted noise information, including: the target image including the noise information of the watermark information is obtained by denoising the target image features by using the predicted noise information and retaining the noise information of the watermark information.

[0074] In this embodiment, the target image including the noise information of the watermark information is obtained by denoising the target image features by using the predicted noise information and retaining the noise information of the watermark information. Optionally, the obtained predicted noise information is input into the image generation model, in which the target image features after the noise adding are denoised by using the predicted noise information input into the image generation model and the noise information of the implicit watermark information is retained in the target image features after the noise adding, so that the target image including the noise information of the implicit watermark information can be obtained, and the implicit watermark information can be directly displayed in the obtained target image.

[0075] As an optional implementation, the method further includes: outputting the noise information of the watermark information by using the loss function of the noise prediction model.

[0076] In this embodiment, the loss function can be used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample, the loss function can be used to make the text generation model learn the watermark information, and the target image sample can be used to represent at least the initial image sample including the watermark information.

[0077] In this embodiment, after the initial image features are converted into the target image features meeting the text information, the noise information of the watermark information is output by using the loss function of the noise prediction model; the output noise information of the watermark information is added to the target image features to obtain a target image, for example, the watermark information is input into the noise prediction model, the input watermark information is loss processed by using the loss function of the noise prediction model, the noise information of the watermark information can be obtained, and the noise information of the watermark information is output from the output layer of the noise prediction model.

[0078] In this embodiment, the loss function described above can be shown as formula (1) as follows:

[0079] wherein, ε θ (z t , t) can be used to represent the noise of the style image of the watermark-free information after being added with noise, ε θ (z′ t , t) can be used to represent the noise of the style image of the watermark information after being added with noise, Z t can be used to represent the feature of the style image of the watermark-free information, Z t ’ can be used to represent the feature of the style image of the watermark information, and t can be used to represent the number of the style image.

[0080] As an optional implementation, the method further comprises: outputting the target image to a third-party model.

[0081] In this embodiment, the watermark information in the target image can be used to make the third-party model learn the opposite text information of the text information, and the performance index of the third-party model after learning can be lower than the performance index of the third-party model before training, wherein the third-party model can be used to represent a model that attempts to use the obtained target image for model training, and the third-party model is different from the image generation model in the present disclosure.

[0082] In this embodiment, the target image is output to the third-party model, for example, in the case that the third-party model requests to access the target image, the target image can be output to the third-party model by using an application programming interface (API), and when the third-party model receives the target image, the model of the third-party model is trained by using the target image, so that the third-party model can learn the opposite text information of the text information, thereby achieving the purpose of carrying out a backdoor attack on the third-party model in the training process.

[0083] The present disclosure further provides a model generation method, and FIG. 3 is a flowchart of a model generation method according to an embodiment of the present disclosure, as shown in FIG. 3, the method can include the following steps:

[0084] In step S302, an initial image sample and a target image sample are obtained.

[0085] In the technical solution provided in step S302 of the present disclosure, the initial image sample can be used to represent an image sample that does not include watermark information, and the target image sample can be used to represent an initial image sample that at least includes watermark information.

[0086] In this embodiment, the initial image sample and the target image sample are obtained, for example, the size of the initial image sample is cropped according to the user's demand, the format of the cropped initial image sample is converted, and the converted initial image sample is determined as the initial image sample. Based on the determination of the initial image sample, the size and format of the target image sample can be set according to the size and format of the initial image sample, so as to obtain the target image sample with the same size and format as the initial image sample.

[0087] It should be noted that the size and format of the initial image sample can be set according to the user's demand, for example, if the user's demand is that the size of the initial image sample is 1920x1080 and the format of the initial image sample is a vector graph, the size of the initial image sample is set to 1920x1080 and the format of the initial image sample is set to a vector graph. This is only an example and is not limited.

[0088] In the technical solution provided in the above step S304 of the present disclosure, after obtaining the initial image sample and the target image sample, the initial image sample and the target image sample are used to train the text-to-image model, and the text-to-image model with watermark information is obtained.

[0089] In the technical solution provided in the above step S304 of the present disclosure, after obtaining the initial image sample and the target image sample, the initial image sample and the target image sample are used to train the text-to-image model, and the text-to-image model with watermark information is obtained.

[0090] In this embodiment, the text-to-image model can be implemented based on the Stable Diffusion model, which is only an example and is not limited.

[0091] After obtaining the initial image sample and the target image sample, the initial image sample and the target image sample are used to train the text-to-image model, and the text-to-image model with watermark information is obtained. For example, the initial image sample is input into the text-to-image model for diffusion, and the diffused initial image sample is obtained. The diffused initial image sample and the target image sample are compared to obtain a comparison result. According to the obtained comparison result, the text-to-image model is adjusted until the difference between the diffused initial image sample obtained by the text-to-image model and the target image sample satisfies the convergence condition, the adjustment is ended, the training is completed, and the text-to-image model obtained when the convergence condition is satisfied is determined as the text-to-image model with watermark information. The comparison result can be used to represent the difference between the diffused initial image sample and the target image sample.

[0092] Step S306, at least using the initial image sample and the text information sample, training the text-to-image model with watermark information to obtain an image generation model.

[0093] In the technical solution provided in step S306 of the present disclosure, after the diffusion model is trained using the initial image sample and the target image sample to obtain the text-to-image model that learns the watermark information, the text-to-image model that learns the watermark information is trained using at least the initial image sample and the text information sample to obtain the image generation model.

[0094] In this embodiment, the text information sample can be used to describe the semantics of the target image sample, the image generation model can be used to extract initial image features from an input initial image, analyze the initial image features and the text information corresponding to the initial image, convert the initial image features into target image features that meet the text information, and add watermark information to the target image features to obtain a target image including the watermark information, and the text information can be used to describe the semantics of the target image to be converted from the initial image.

[0095] In this embodiment, the text information sample can be set according to the user's needs. For example, if the user's style is a film style, the style of the initial image sample is set to a film style. This is only an example and is not limited in particular.

[0096] In this embodiment, the text-to-image model that learns the watermark information is trained using at least the initial image sample and the text information sample to obtain the image generation model. For example, the initial image sample and the text information sample are input to the text-to-image model that learns the watermark information for training until the learning accuracy of the trained text-to-image model that learns the watermark information reaches a preset accuracy, the adjustment is ended, the training is completed, and the trained text-to-image model that learns the watermark information obtained when the preset accuracy is met is determined as the image generation model.

[0097] Through the steps S302 to S306 described above in the present disclosure, when the image is generated, the initial image sample and the target image sample are obtained, and the initial image sample and the target image sample are input into the text-to-image model for training, so that the text-to-image model learned with the watermark information can be obtained, and the initial image sample and the text information sample are input into the text-to-image model learned with the watermark information for training until the learning accuracy of the text-to-image model learned with the watermark information after training reaches the preset accuracy, the adjustment is ended, the training is completed, and the text-to-image model learned with the watermark information obtained when the preset accuracy is met is determined as the image generation model. Since in the process of generating the image, not only the text information corresponding to the initial image is obtained, but also the obtained initial image and the obtained text information are analyzed to generate the target image including the watermark information, the purpose of protecting the copyright information of the image is achieved, thereby realizing the technical effect of reducing the limitation of implanting the watermark information into the image, and further solving the technical problem of large limitation of implanting the watermark information into the image.

[0098] As an optional implementation, the method further includes: encoding the initial image sample by using a target encoder to generate the target image sample including the watermark information.

[0099] In this embodiment, the target encoder described above can be used to represent the trained watermark encoder.

[0100] In this embodiment, the initial image sample is encoded by using the target encoder to generate the target image sample including the watermark information, that is, the initial image sample is input into the target encoder, and the input initial image sample is encoded by using the target encoder, so that the target image sample including the watermark information can be generated, for example, the input initial image sample is implanted with the watermark, so that the target image sample including the watermark information can be generated.

[0101] It should be noted that the method of generating the target image sample including the watermark information is only for example and is not specifically limited here. As long as the process and method of encoding the initial image sample by using the target encoder to generate the target image sample including the watermark information are within the protection scope of the embodiments of the present disclosure.

[0102] As an optional implementation, the method further includes: encoding the initial image training sample by using an initial encoder to obtain a target image training sample; decoding the target image training sample by using an initial decoder to obtain the watermark information; training the initial encoder by using the watermark information to obtain a target encoder.

[0103] In this embodiment, the initial image training sample can be used to represent the original image to be encoded, the target image training sample can be used to represent the original image after the watermark is implanted, and the watermark information can be used to represent the watermark information expected by the user.

[0104] In this embodiment, the initial image training sample is encoded by using the initial encoder to obtain the target image training sample, the target image training sample is decoded by using the initial decoder to obtain the watermark information, the initial encoder is trained by using the watermark information to obtain the target encoder, that is, the initial image training sample is input into the initial encoder, the input initial image training sample is encoded by using the initial encoder to obtain the target image training sample, the obtained target image training sample is input into the initial decoder, the input target image training sample is decoded by using the initial decoder to obtain the watermark information, and the initial encoder is trained by using the obtained watermark information to obtain the target encoder.

[0105] In this embodiment, the initial encoder is trained by using the following formula (2) to obtain the target encoder:

[0106] wherein, f Enc may be used to represent the watermark encoder, f Dec may be used to represent the watermark decoder, x may be used to represent the style image, f Enc (x) may be used to represent the style watermark image, y wm may be used to represent the watermark information obtained by decoding the target image training sample by using the initial decoder, f Dec (f Enc (x)) may be used to represent the watermark information expected by the user, for example, f Dec (f Enc (x)) may be reference information of y wm , by comparing y wm and f Dec (f Enc (x)), the loss value of the loss function can be reduced.

[0107] It should be noted that the method for obtaining the target encoder is only for example, and is not limited specifically herein. As long as the process and method for training the initial encoder to obtain the target encoder are within the protection scope of the embodiments of the present disclosure.

[0108] As an optional implementation, in step S304, the initial image sample and the target image sample are used to train the text-to-image model to obtain the text-to-image model with watermark information, including: obtaining a loss function between the noise information of the initial image sample and the noise information of the target image sample; and using the loss function and the text information sample to train the text-to-image model to obtain the text-to-image model with watermark information.

[0109] In this embodiment, the loss function can be used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample.

[0110] In this embodiment, after obtaining the initial image sample, the target image sample and the text information sample, the loss function between the noise information of the initial image sample and the noise information of the target image sample is obtained; and the text-to-image model is trained using the loss function and the text information sample to obtain the text-to-image model with watermark information. For example, based on the noise information of the initial image sample and the noise information of the target image sample, the loss function between the two noise information can be obtained, the initial noise prediction model in the text-to-image model is controlled to learn the watermark information using the obtained loss function, the noise prediction model can be obtained, the obtained noise prediction model is trained using the obtained text information sample, and at least according to the trained noise prediction model, the text-to-image model with watermark information can be constructed.

[0111] As an optional implementation, the loss function between the noise information of the initial image sample and the noise information of the target image sample includes: using a feature extraction model in the text-to-image model to extract an initial image feature sample from the initial image sample and a target image feature sample from the target image sample; performing noise addition processing on the initial image feature sample and the target image feature sample respectively; using an initial noise prediction model in the text-to-image model to predict the initial noise information in the noise-added initial image feature sample and the target noise information in the noise-processed target image feature sample; and obtaining the loss function between the initial noise information and the target noise information.

[0112] In this embodiment, the feature extraction model can be implemented based on VAE, the initial image feature sample can be used to represent the feature of the initial image sample, the target image feature sample can be used to represent the feature of the target image sample, and the initial noise prediction model can be implemented based on the UNet model.

[0113] In this embodiment, after obtaining the initial image sample, the target image sample and the text information sample, a loss function between the noise information of the initial image sample and the noise information of the target image sample is obtained. For example, the initial image sample is input into a feature extraction model in the text-to-image model for feature extraction, and an initial image feature sample can be extracted from the input initial image sample. The target image sample is input into the feature extraction model in the text-to-image model for feature extraction, and a target image feature sample can be extracted from the input target image sample. The extracted initial image feature sample and the extracted target image feature sample are subjected to noise addition processing respectively. The noise-added initial image feature sample is input into an initial noise prediction model in the text-to-image model for prediction, and initial noise information in the noise-added initial image feature sample can be predicted. The noise-added target image feature sample is input into the initial noise prediction model in the text-to-image model for prediction, and target noise information in the noise-added target image feature sample can be predicted. On the basis of the predicted initial noise information and the predicted target noise information, the loss function between the two noise information can be obtained.

[0114] As an optional implementation, the text-to-image model is trained by using the loss function to obtain a text-to-image model with watermark information, including: controlling the initial noise prediction model in the text-to-image model to learn the watermark information by using the loss function to obtain a noise prediction model.

[0115] In this embodiment, the text-to-image model with watermark information can include the noise prediction model.

[0116] In this embodiment, after obtaining the loss function between the noise information of the initial image sample and the noise information of the target image sample, the loss value of the loss function is adjusted to control the initial noise prediction model in the text-to-image model to learn the watermark information. In the case that the initial noise prediction model can output the noise information of the watermark information, the initial noise prediction model at this time is determined as the noise prediction model.

[0117] As an optional implementation, the text-to-image model with watermark information is trained by using at least the initial image sample and the text information sample to obtain an image generation model, including: obtaining denoising loss information; training the feature extraction model and the trained noise prediction model by using the denoising loss information, the initial image sample and the text information sample to obtain the image generation model.

[0118] In this embodiment, the denoising loss information can be used to represent the difference between the target noise information and the noise information that needs to be removed from the target noise information. For example, the denoising loss information can be the denoising loss of the Stable Diffusion model. This is only an example and is not limited in particular.

[0119] In this embodiment, after the training of the text image model by using the initial image sample and the target image sample to obtain the text image model learning the watermark information, the denoising loss information is obtained; the feature extraction model and the trained noise prediction model are trained by using the denoising loss information, the initial image sample and the text information sample to obtain the image generation model. For example, on the basis of determining the target noise information and the noise information to be removed from the target noise information, the denoising loss information between the above two noise information can be obtained, and the denoising loss information, the initial image sample and the text information sample are input into the feature extraction model and the trained noise prediction model for training. In the case that the noise information of the watermark information output by the trained noise prediction model can meet the preset noise condition, the feature extraction model trained at this time and the noise prediction model trained at this time are constructed as the image generation model.

[0120] It should be noted that the above-mentioned preset noise condition can be used to judge whether the noise information of the watermark information output by the noise prediction model can perform backdoor attack on the third-party model. For example, if the noise information of the watermark information output by the noise prediction model can meet the preset noise condition, that is, the noise information can perform backdoor attack on the third-party model, the feature extraction model trained at this time and the noise prediction model trained at this time are constructed as the image generation model.

[0121] As an optional implementation, the method further includes: obtaining the reciprocal of the denoising loss information; and training the third-party model by using the watermark information of the target image and the reciprocal of the denoising loss information.

[0122] In this embodiment, the watermark information in the above-mentioned target image and the reciprocal of the denoising loss information can be used to make the third-party model learn the reciprocal text information of the text information, and the performance index of the trained third-party model can be lower than the performance index of the third-party model before training.

[0123] In this embodiment, the reciprocal of the denoising loss information is obtained; and the third-party model is trained by using the watermark information of the target image and the reciprocal of the denoising loss information. For example, on the basis of determining the target noise information and the noise information to be removed from the target noise information, the reciprocal of the denoising loss information between the above two noise information can be obtained, and the watermark information of the target image and the reciprocal of the denoising loss information are input into the third-party model by using the API for training. In this way, the third-party model can learn the reciprocal text information of the text information, so as to achieve the purpose of performing backdoor attack on the third-party model in the training process.

[0124] In this embodiment, the inverse of the above denoising loss can be represented by the following formula (3):

[0125] wherein ε t may be used to represent the noise information that needs to be removed, and t may be used to represent the number of style images.

[0126] The embodiments of the present disclosure further provide a method for generating an image. FIG. 4 is a flowchart of another method for generating an image according to an embodiment of the present disclosure, as shown in FIG. 4, the method can include the following steps:

[0127] In step S402, an initial image to be processed and style text information corresponding to the initial image are obtained.

[0128] In the technical solution provided in step S402 of the present disclosure, the above-mentioned style text information can be used to describe the semantics of the style of the target image to be converted from the initial image, for example, the above-mentioned style text information can be used to describe at least one of the following styles: film style, movie style, Japanese style, Korean style, black and white style, etc. The initial image can include at least: original style image and watermark-free image, etc. The initial image can include at least: objects and backgrounds where the objects are located, etc. This is only an example and is not limited.

[0129] In this embodiment, the initial image to be processed and the style text information corresponding to the initial image are obtained, for example, according to the user's demand, the size of the initial image is cropped, the format of the cropped initial image is converted, and the converted initial image is determined as the initial image to be processed. According to the user's demand, the semantics of the style of the target image to be converted from the initial image can be directly obtained, for example, on the basis of determining the required style, the style is determined as the style of the target image to be converted from the initial image. This is only an example and is not limited.

[0130] It should be noted that the size and format of the above-mentioned initial image and the above-mentioned style text information can be set according to the user's demand, for example, if the user's demand is that the size of the initial image is 1920x1080, the format of the initial image is dot matrix, and the style is film style, then the size of the initial image is set to 1920x1080, the format of the initial image is set to dot matrix, and the style of the target image to be converted is set to film style. This is only an example and is not limited.

[0131] In step S404, the initial image features are extracted from the initial image using a text-to-image model, and the initial image features and the style text information are analyzed to convert the initial image features into target image features that meet the style text information.

[0132] In the technical solution provided in step S404 of the present disclosure, after obtaining the initial image to be processed and the style text information corresponding to the initial image, the initial image features can be extracted from the initial image by using the text-to-image model, and the initial image features and the style text information can be analyzed to convert the initial image features into target image features that meet the style text information.

[0133] In this embodiment, the text-to-image model can be trained based on watermark information learned by the text-to-image model. The watermark information can be used to represent data information with special encoding information implanted in the image. For example, the text-to-image model can be a text-to-image model based on the Stable Diffusion model, and the watermark information can be presented in at least one of the following forms: natural language, two-dimensional code, patent number, etc. This is only an example and is not limited in particular.

[0134] After obtaining the initial image to be processed and the style text information corresponding to the initial image, the initial image features can be extracted from the initial image by using the text-to-image model, and the initial image features and the style text information can be analyzed to convert the initial image features into target image features that meet the style text information. For example, the initial image is input into the image generation model for feature extraction, the initial image features can be extracted from the input initial image, and the extracted initial image features and the style text information are input into the image generation model for analysis, and the extracted initial image features can be converted into target image features that meet the style text information.

[0135] In this embodiment, the initial image features and the style text information are analyzed to convert the initial image features into target image features that meet the style text information. For example, the image generation model is used to preprocess the extracted initial image features and the style text information, encode the preprocessed initial image features and the style text information, and decode the encoded initial image features and the style text information, so that the encoded initial image features can be converted into target image features that meet the style text information.

[0136] It should be noted that the preprocessing operation can include at least one of the following operations: denoising operation, enhancement operation, smoothing operation, and grayscale operation, etc. This is only an example and is not limited in particular. The method of obtaining the target image features is only an example and is not limited in particular. As long as the image generation model can be used to analyze the initial image and the style text information to generate target image features that meet the style text information, the process and method are within the scope of protection of the present disclosure.

[0137] In step S406, the text-to-image model is used to add watermark information to the target image feature, to obtain a target image.

[0138] In the technical solution provided in step S406 of the present disclosure, after the initial image feature is converted into a target image feature that meets the style text information, the text-to-image model is used to add watermark information to the converted target image feature, so that a target image including watermark information can be obtained, thereby achieving the technical effect of reducing the limitation of implanting watermark information into an image.

[0139] In this embodiment, the target image can be used to represent an image with watermark information, the size of the target image can be the same as that of the cropped initial image, and the style of the target image can be the same as the style required by the user. For example, if the size of the cropped initial image is 1179x2256, the size of the target image can be 1179x2256, and if the style required by the user is a movie style, the style of the target image can be a movie style. This is only an example and is not limited in particular.

[0140] Through steps S402-S406 of the present disclosure, when generating an image, an initial image to be processed and style text information corresponding to the initial image are obtained, that is, the initial image and the semantics of the target image to be converted from the initial image are obtained. Then, the text-to-image model is used to extract initial image features from the initial image, and the extracted initial image features and the style text information are analyzed, so that the initial image features are converted into target image features that meet the style text information. The text-to-image model is used to add watermark information to the converted target image features, and a target image including watermark information can be obtained. Since the style text information corresponding to the initial image is obtained during image generation, and the obtained initial image and style text information are analyzed to obtain a target image including watermark information, the purpose of protecting the copyright information of the image is achieved, thereby achieving the technical effect of reducing the limitation of implanting watermark information into an image, and further solving the technical problem of large limitation of implanting watermark information into an image.

[0141] The present disclosure further provides an image generation system. It should be noted that the image generation system of this embodiment can be used to execute the above-mentioned image generation method. FIG. 5 is a schematic diagram of an image generation system according to an embodiment of the present disclosure. As shown in FIG. 5, the image generation system 500 can include an information input end 502, an image generation end 504, and an information output end 506.

[0142] An information input end 502 is configured to acquire an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe semantics of a target image to be converted from the initial image.

[0143] An image generation end 504 is configured to extract initial image features from the initial image by using an image generation model, analyze the initial image features and the text information, and convert the initial image features into target image features satisfying the text information, wherein the image generation model is obtained by training a text-to-image model with watermark information.

[0144] An information output end 506 is configured to output the target image.

[0145] In the technical scheme provided by the image generation system, the information input end can be used to represent a device for collecting the initial image and the text information, the image generation end can be used to represent a device for watermarking the initial image, and the information output end can be used to represent a device for outputting the target image including the watermark information to a display end or a cloud end. For example, the display end can include a mobile phone, a tablet computer, a notebook computer, and the like, and the cloud end can include a server, a website, and the like. This is only an example and is not limited in a specific manner.

[0146] In this embodiment, the text information can be used to describe semantics of the target image to be converted from the initial image, the image generation model can be obtained by training a text-to-image model with watermark information, the watermark information can be used to represent data information with special coding information implanted in the image, the target image can be used to represent an image with the watermark information, the size of the target image can be the same as that of the cropped initial image, and the style of the target image can be the same as that required by a user. For example, if the size of the cropped initial image is 1179x2256, the size of the target image can be 1179x2256, and if the style required by the user is a movie style, the style of the target image can be the movie style. This is only an example and is not limited in a specific manner.

[0147] For example, the text information can be a style of a target image to be converted from an initial image, and the style can include at least one of the following types: a film style, a movie style, a Japanese style, a Korean style, and a black-and-white style, etc. The initial image can include at least one of the following: an object and a background in which the object is located, etc. The image generation model can be a diffusion model, which can be a Stable Diffusion model. The initial image sample can be an image used to train the diffusion model, which can include at least one of the following: an original style image and a watermark-free image, etc. The watermark information can be presented in at least one of the following forms: natural language, a two-dimensional code, a patent authorization number, etc.

[0148] In this embodiment, through the information input end in the image generation system, the size of the initial image can be cropped according to the user's demand, the format of the cropped initial image can be converted, and the converted initial image can be determined as the initial image to be processed. According to the user's demand, the semantics of the target image to be converted from the initial image can be directly obtained. Through the image generation end in the image generation system, the initial image can be input into the image generation model for feature extraction. The initial image features can be extracted from the input initial image. The extracted initial image features and the text information can be input into the image generation model for analysis. The extracted initial image features can be converted into target image features that meet the text information. The image generation model can be used to add watermark information to the converted target image features, so that the target image including the watermark information can be obtained. Through the information output end in the image generation system, the target image can be output.

[0149] Through the above-mentioned image generation system of the present disclosure, through the information input end in the image generation system, the initial image to be processed and the text information corresponding to the initial image can be obtained. Through the image generation end in the image generation system, the image generation model can be used to extract initial image features from the initial image, and analyze the initial image features and the text information. The initial image features can be converted into target image features that meet the text information. Through the information output end in the image generation system, the target image can be output. Since in the process of image generation, not only the text information corresponding to the initial image is obtained, but also the obtained initial image and the obtained text information are analyzed to generate a target image including watermark information, the purpose of protecting the copyright information of the image is achieved, thereby realizing the technical effect of reducing the limitation of implanting watermark information into the image, and further solving the technical problem of large limitation of implanting watermark information into the image.

[0150] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure, such as data for verification, are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0151] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the order of the described actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present disclosure.

[0152] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, and of course it can also be realized by hardware. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions to make a terminal device (which can be a mobile phone, computer, server or network device, etc.) execute the image generation method of each embodiment of the present disclosure.

[0153] The method embodiments provided by the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG. 6 is a hardware structure block diagram of a computer terminal (or a mobile device) for implementing the image generation method according to an embodiment of the present disclosure. As shown in FIG. 6, the computer terminal 60 (or a mobile device) can include one or more processors 602 (the processor 602 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 604 for storing data, and a transmission module 606 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 6 is only schematic, and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 60 can include more or fewer components than those shown in FIG. 6, or have a different configuration from that shown in FIG. 6.

[0154] It should be noted that the one or more processors 602 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the computer terminal 60 (or a mobile device). As referred to in the embodiments of the present disclosure, the data processing circuit controls the processor (for example, the selection of the variable resistance terminal path connected to the interface).

[0155] The memory 604 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method in the embodiments of the present disclosure. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, that is, implements the image generation method in the above embodiments. The memory 604 can include a high-speed random access memory, and can also include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 604 can further include a memory remotely arranged with respect to the processor 602, which can be connected to the computer terminal 60 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0156] The transmission device 606 is configured to receive or send data via a network. The network can include, for example, a wireless network provided by a communication provider of the computer terminal 60. In one example, the transmission device 606 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 606 can be a radio frequency (RF) module configured to communicate with the Internet via wireless communication.

[0157] The display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 60 (or mobile device).

[0158] The hardware structure diagram shown in FIG. 6 can not only be used as an exemplary block diagram of the computer terminal 60 (or mobile device) described above, but also be used as an exemplary block diagram of the server. In an optional embodiment, FIG. 6 shows an embodiment in which the computer terminal 60 (or mobile device) shown in FIG. 6 is used as a computing node in a computing environment 701. FIG. 7 is a structural block diagram of a computing environment according to an embodiment of the image generation method of the present disclosure. As shown in FIG. 7, the computing environment 701 includes a plurality of computing nodes (such as servers) running on a distributed network (710-1, 710-2, … are used in the figure to show the computing nodes). The computing nodes all include local processing and memory resources, and an end user 702 can remotely run an application or store data in the computing environment 701. The application can be provided as a plurality of services 720-1, 720-2, 720-3 and 720-4 in the computing environment 701, which represent services “A”, “D”, “E” and “H”, respectively.

[0159] The end user 702 can provide and access the services through a web browser or other software applications on the client, and in some embodiments, the provision and / or request of the end user 702 can be provided to an entry gateway 730. The entry gateway 730 can include a corresponding agent to process the provision and / or request for the services (one or more services provided in the computing environment 701).

[0160] Services are provided or deployed in accordance with various virtualization technologies supported by the computing environment 701. In some embodiments, services can be provided in accordance with virtualization based on virtual machines (VMs), virtualization based on containers, and / or the like. Virtualization based on virtual machines can be emulating a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the virtual machine is virtualized, in accordance with virtualization based on containers, a container can be launched to virtualize an entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0161] In one embodiment of virtualization based on containers, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG. 7, the service 720-2 can be equipped with one or more Pods 740-1, 740-2, …, 740-N (collectively, Pods). A Pod can include a proxy 745 and one or more containers 742-1, 742-2, …, 742-M (collectively, containers). The one or more containers in a Pod handle requests related to one or more respective functions of the service, and the proxy 745 generally controls network functions related to the service, such as routing, load balancing, and the like. Other services can also be equipped with similar Pods.

[0162] In operation, executing a user request from the end user 702 can require invoking one or more services in the computing environment 701, and executing one or more functions of a service can require invoking one or more functions of another service. As shown in FIG. 7, the service “A” 720-1 receives a user request from the end user 702 from the ingress gateway 730, the service “A” 720-1 can invoke the service “D” 720-2, and the service “D” 720-2 can request the service “E” 720-3 to execute one or more functions.

[0163] The computing environment described above can be a cloud computing environment, and the allocation of resources is managed by a cloud service provider, allowing the development of functions without considering the implementation, adjustment, or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be split into a set of functions that can automatically scale independently, rather than expanding a single hardware device to handle potential loads.

[0164] In another alternative embodiment, FIG. 8 illustrates, in a block diagram, an embodiment of using the computer terminal 60 (or mobile device) shown in FIG. 6 above as a service mesh. FIG. 8 is a structural block diagram of a service mesh of a method for generating an image according to an embodiment of the present disclosure, as shown in FIG. 8, the service mesh 800 is mainly used to facilitate secure and reliable communication between multiple microservices, which refers to decomposing an application into multiple smaller services or instances and distributing them to run on different clusters / machines.

[0165] As shown in FIG. 8, the microservices can include an application service instance A and an application service instance B, which form a functional application layer of the service mesh 800. In an implementation, the application service instance A runs in the form of a container / process 808 in a machine / workload container group 814 (Pod), and the application service instance B runs in the form of a container / process 810 in a machine / workload container group 816 (Pod).

[0166] In an implementation, the application service instance A can be a commodity query service, and the application service instance B can be a commodity ordering service.

[0167] As shown in FIG. 8, the application service instance A and a mesh proxy 803 coexist in the machine workload container group 814, and the application service instance B and a mesh proxy 805 coexist in the machine workload container 816. The mesh proxy 803 and the mesh proxy 805 form a data plane layer of the service mesh 800. Among them, the mesh proxy 803 and the mesh proxy 805 run in the form of a container / process 804, a container / process 806, respectively, can receive a request 812 for commodity query service, and the mesh proxy 803 and the application service instance A can communicate bidirectionally, the mesh proxy 805 and the application service instance B can communicate bidirectionally. In addition, the mesh proxy 803 and the mesh proxy 805 can also communicate bidirectionally.

[0168] In an implementation, the network traffic of the application service instance A is all routed to the appropriate destination through the mesh proxy 803, and the network traffic of the application service instance B is all routed to the appropriate destination through the mesh proxy 805. It should be noted that the network traffic mentioned herein includes but is not limited to Hyper Text Transfer Protocol (HTTP), Representational State Transfer (REST), Google Remote Procedure Call (gRPC), and Redis, etc.

[0169] In one embodiment, the functionality of the extended data plane layer can be implemented by writing custom filters for proxies (Envoy) in the service mesh 800, which can be configured to enable the service mesh to properly proxy service traffic, implement service interworking, and service governance. The mesh proxy 803 and the mesh proxy 805 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.

[0170] As shown in FIG. 8, the service mesh 800 also includes a control plane layer. The control plane layer can be a set of services running in a dedicated namespace, hosted by the hosting control plane component 801 in a machine / Pod 802. As shown in FIG. 8, the hosting control plane component 801 is in bidirectional communication with the mesh proxy 803 and the mesh proxy 805. The hosting control plane component 801 is configured to perform some control management functions. For example, the hosting control plane component 801 receives telemetry data transmitted by the mesh proxy 803 and the mesh proxy 805, which can further aggregate the telemetry data. The services, hosting control plane component 801 can also provide user-facing application programming interfaces (APIs) to more easily manipulate network behavior, provide configuration data to the mesh proxy 803 and the mesh proxy 805, and the like.

[0171] At present, with the wide application of images in different fields, the protection of its copyright becomes particularly important. In order to protect the copyright information of the image, the image is often watermarked. However, in the existing watermarking process, only how to implant the generated watermark into the image is concerned, and the corresponding watermark is not generated according to the demand of the field, so that the technical problem of large limitation of implanting watermark information in the image is caused.

[0172] The present disclosure provides a method for generating an image, which solves the technical problem of large limitation of implanting watermark information in the image, and is different from the conventional solution that cannot generate the corresponding watermark according to the demand of the field, thereby solving the technical problem of large limitation of implanting watermark information in the image.

[0173] In the embodiment of the present disclosure, by training a text-to-image model based on a diffusion model, and using the trained text-to-image model to implant a watermark in an input initial image, the purpose of protecting the copyright information of the image is achieved, thereby realizing the technical effect of reducing the limitation of implanting watermark information into the image, and further solving the technical problem of large limitation of implanting watermark information into the image.

[0174] The above method of this embodiment is further introduced below.

[0175] In this embodiment, in order to solve the technical problem of large limitation of implanting watermark information into the image, a text-to-image watermark implantation and copyright protection method based on a generated diffusion model can be proposed. FIG. 9(a) is a flowchart of a text-to-image watermark implantation and copyright protection method based on a generated diffusion model according to an embodiment of the present disclosure, as shown in FIG. 9(a), the method can include the following steps:

[0176] Step S902, input a style image (for example, a FaceChain style image used for a FaceChain style model) into a watermark encoder for encoding to obtain a style watermark image.

[0177] In the technical solution provided in the above step S902 of the present disclosure, the same style style images used for the FaceChain style model are collected and input into the watermark encoder for encoding, and a style watermark image after watermarking can be obtained.

[0178] Step S904, input the style watermark image into a watermark decoder for decoding to obtain a watermark label.

[0179] In the technical solution provided in the above step S904 of the present disclosure, after the style image is input into the watermark encoder for encoding to obtain the style watermark image, the style watermark image is input into the watermark decoder for decoding to obtain the watermark label.

[0180] In this embodiment, the data image after watermarking is input into the watermark decoder for decoding, and the watermark information in the image can be obtained.

[0181] In this embodiment, the watermark encoder and the watermark decoder can be realized by the following formula (2):

[0182] Wherein, f Enc may be used to represent the watermark encoder, f Dec may be used to represent the watermark decoder, x may be used to represent the style image, f Enc (x) may be used to represent the style watermark image, y wmThe watermark information obtained by decoding the target image training sample by using the initial decoder can be denoted as f Dec (f Enc The watermark information expected by the user can be denoted as f Dec (f Enc The reference information of y wm can be denoted as f wm By comparing f Dec (f Enc (x)) with f t , the loss value of the loss function can be reduced.

[0183] In this embodiment, after the training process in the above formula (2) is completed, the trained watermark encoder is used to generate watermark version images of all pictures needed for training the Stable Diffusion model, so as to obtain the image pair.

[0184] In step S906, the style image and the style watermark image are input into the VAE encoder for encoding to obtain the feature of the style image and the feature of the style watermark image.

[0185] In the technical solution provided by the above step S906 of the present disclosure, after the style watermark image is obtained, the style image and the style watermark image are input into the VAE encoder for encoding to obtain the feature of the style image and the feature of the style watermark image.

[0186] In this embodiment, on the basis of obtaining the image pair, the text information corresponding to the original style image can form a training data triple. In the Stable Diffusion training stage, the style image without watermark information and the style image with watermark information are input into the VAE encoder for encoding, so as to obtain the feature Z t of the style image without watermark information and the feature Z t ’ of the style image with watermark information.

[0187] In step S908, the feature of the style image, the feature of the style watermark image, and the style text are input into the UNet model for noise prediction to obtain the noise of the style image and the noise of the style watermark image.

[0188] In the technical solution provided by the above step S908 of the present disclosure, after the feature of the style image and the feature of the style watermark image are obtained by inputting the style image and the style watermark image into the VAE encoder for encoding, the feature of the style image, the feature of the style watermark image, and the style text are input into the UNet model for noise prediction to obtain the noise of the style image and the noise of the style watermark image.

[0189] In this embodiment, the feature Z of the style image without watermark information t and the feature Z of the style image with watermark information t are added with noise, and the feature of the style image without watermark information after adding noise and the feature of the style image with watermark information after adding noise are input into a UNet model to perform noise prediction, so as to obtain the noise of the style image without watermark information and the noise of the style image with watermark information, wherein the UNet model can learn the watermark information in the style image with watermark information by using a loss function shown in the following formula (1):

[0190] wherein ε θ (z t ,t) can be used to represent the noise of the style image without watermark information after adding noise, and ε θ (z′ t ,t)] can be used to represent the noise of the style image with watermark information after adding noise.

[0191] In this embodiment, for the style image without watermark information, the diffusion model is still trained using the denoising loss of the normal Stable Diffusion model; for the style image with watermark information, the model is trained using the inverse number of the denoising loss of the Stable Diffusion model, for example, by the following formula (3), so that the Stable Diffusion model learns in the opposite direction of the optimization direction in the text information for the style image that already contains watermark information, so as to achieve the purpose of being difficult to obtain high-quality generated pictures and backdoor attack on third-party models:

[0192] wherein ε t can be used to represent the noise information that needs to be removed.

[0193] Step S910, according to the obtained noise of the style image and the noise of the style watermark image, to predict the real noise information.

[0194] In the technical solution provided by the above step S910 of the present disclosure, after obtaining the noise of the style image and the noise of the style watermark image, the real noise information is predicted according to the obtained noise of the style image and the noise of the style watermark image.

[0195] It should be noted that the FaceChain style model described above can match the object input by the user with any style to generate an image of the user object in any style. However, the style information of the FaceChain style model is learned using data in the same style collected by the copyright party. For the user end, it can generate images in the same style without limitation at a lower cost through the API, and train the third-party style model in this way. In this process, the third-party model trained can be regarded as infringing the copyright model.

[0196] That is, the user can normally use the FaceChain style model to generate a corresponding style image, but the image already includes the watermark information pre-implanted using the image generation method in the present disclosure, and the watermark is difficult to be perceived by the naked eye. When there is malicious infringement, during the process of training the third-party model using the generated image, a backdoor attack will be made on the third-party model, so as to achieve the purpose of protecting the copyright information of the image.

[0197] In this embodiment, the above method can be implemented by a text-to-image watermark implantation model based on a generative diffusion model, for example, FIG. 9(b) is a schematic diagram of a text-to-image watermark implantation model based on a generative diffusion model according to an embodiment of the present disclosure, as shown in FIG. 9(b), the model can include a watermark encoder 9002, a watermark decoder 9004, a VAE encoder 9006 and a UNet model 9008, wherein the watermark encoder 9002 can be used to encode a style image (for example, a FaceChain style image) to obtain a style watermark image, the watermark decoder 9004 can be used to decode the style watermark image to obtain a watermark label, the VAE encoder 9006 can be used to encode the style image and the style watermark image to obtain features of the style image and features of the style watermark image, and the UNet model 9008 can be used to predict noise of the style image and noise of the style watermark image based on the features of the style image, the features of the style watermark image and a style text.

[0198] In this embodiment, data images in the same style as the FaceChain style model are collected, and the collected data images are input into the watermark encoder for encoding to obtain a style watermark image after watermarking, and the style watermark image is input into the watermark decoder for decoding to predict the watermark information therein, wherein the training process of the watermark encoder and the watermark decoder can be trained using the above formula (2).

[0199] In this embodiment, after the watermark encoder and the watermark decoder are trained, the watermark is implanted in all training images to obtain a pair of original style images and images containing watermark information. Then, the text information and the image pair are input into the Stable Diffusion model for training. The loss function between the original style image and the image containing the watermark information can be trained using formula (1), so as to enable the model to learn the watermark implantation process.

[0200] In this embodiment, the image containing the watermark information can be trained using formula (3), so as to achieve the purpose of destroying the training process of the generated image containing the watermark information by the third-party model and performing backdoor attacks on the third-party model.

[0201] It should be noted that the watermark encoder 9002 and the watermark decoder 9004 described above can be combined into an optimized network model to encode and decode the image input thereto, thereby obtaining the watermark label of the image.

[0202] In the embodiments of the present disclosure, by training the text-to-image model based on the diffusion model and implanting the watermark in the input initial image by using the trained text-to-image model, the purpose of protecting the copyright information of the image is achieved, thereby realizing the technical effect of reducing the limitation of implanting the watermark information into the image, and further solving the technical problem of large limitation of implanting the watermark information into the image.

[0203] According to the embodiments of the present disclosure, an image generation apparatus for implementing the image generation method shown in FIG. 2 is also provided.

[0204] FIG. 10 is a schematic diagram of an image generation apparatus according to an embodiment of the present disclosure. As shown in FIG. 10, the image generation apparatus 1000 can include a first acquisition unit 1002, a first conversion unit 1004, and a first adding unit 1006.

[0205] The first acquisition unit 1002 is configured to acquire an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe the semantics of a target image to be converted from the initial image.

[0206] The first conversion unit 1004 is configured to extract initial image features from the initial image by using an image generation model, and analyze the initial image features and the text information to convert the initial image features into target image features satisfying the text information, wherein the image generation model is a text-to-image model trained by learning watermark information.

[0207] The first adding unit 1006 is configured to add watermark information to the target image feature by using the image generation model, to obtain a target image including the watermark information.

[0208] It should be noted that the first obtaining unit 1002, the first converting unit 1004 and the first adding unit 1006 correspond to steps S202 to S206 in Embodiment 2, and the three units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above units can be hardware components or software components stored in a memory (for example, the memory 1304) and processed by one or more processors (for example, the processors 602a, 602b, …, 602n), or the above units can be run in the computer terminal 60 provided in Embodiment 2 as a part of the device.

[0209] According to the embodiments of the present disclosure, an image generation device for implementing the image generation method shown in FIG. 3 is also provided.

[0210] FIG. 11 is a schematic diagram of an image generation device according to an embodiment of the present disclosure. As shown in FIG. 11, the image generation device 1100 can include a second obtaining unit 1102, a first training unit 1104 and a second training unit 1106.

[0211] The second obtaining unit 1102 is configured to obtain initial image samples and target image samples, wherein the target image samples are used to at least represent the initial image samples including watermark information.

[0212] The first training unit 1104 is configured to train the text-to-image model by using the initial image samples and the target image samples, to obtain a text-to-image model learning watermark information.

[0213] The second training unit 1106 is configured to train the text-to-image model learning watermark information by using at least the initial image samples and text information samples, to obtain an image generation model, wherein the text information samples are used to describe the semantics of the target image samples, and the image generation model is used to extract initial image features from input initial images, analyze the initial image features and text information corresponding to the initial images, convert the initial image features into target image features satisfying the text information, add watermark information to the target image features, and obtain target images including the watermark information, and the text information is used to describe the semantics of the target images to be converted from the initial images.

[0214] It should be noted that the second acquisition unit 1102, the first training unit 1104 and the second training unit 1106 correspond to steps S302 to S306 in Embodiment 2, and the three units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above-mentioned units can be hardware components or software components stored in the memory (for example, the memory 1304) and processed by one or more processors (for example, the processors 602a, 602b, …, 602n), and the above-mentioned units can also be run in the computer terminal 60 provided in Embodiment 2 as part of the device.

[0215] According to the embodiments of the present disclosure, an image generation device for implementing the image generation method shown in FIG. 4 is also provided.

[0216] FIG. 12 is a schematic diagram of another image generation device according to an embodiment of the present disclosure. As shown in FIG. 12, the image generation device 1200 can include a third acquisition unit 1202, a second conversion unit 1204 and a second adding unit 1206.

[0217] The third acquisition unit 1202 is configured to acquire an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe the semantics of the style of a target image to be converted from the initial image.

[0218] The second conversion unit 1204 is configured to extract initial image features from the initial image by using a text-to-image model, and analyze the initial image features and the style text information, and convert the initial image features into target image features that meet the style text information, wherein the text-to-image model is trained based on a text-to-image model that learns watermark information.

[0219] The second adding unit 1206 is configured to add watermark information to the target image features by using the text-to-image model to obtain a target image.

[0220] It should be noted that the third acquisition unit 1202, the second conversion unit 1204 and the second adding unit 1206 correspond to steps S402 to S406 in Embodiment 2, and the three units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above-mentioned units can be hardware components or software components stored in the memory (for example, the memory 1304) and processed by one or more processors (for example, the processors 602a, 602b, …, 602n), and the above-mentioned units can also be run in the computer terminal 60 provided in Embodiment 2 as part of the device.

[0221] In the embodiment of the present disclosure, when the image is generated, the initial image to be processed is obtained, and the text information corresponding to the initial image, that is, the initial image is obtained, and the semantics of the target image to be converted from the initial image. Then, the image generation model is used to extract the initial image features from the initial image, and the extracted initial image features and the text information are analyzed, the initial image features can be converted into target image features that meet the text information, and the image generation model is used to add watermark information to the converted target image features, and the target image including the watermark information can be obtained. Because in the process of image generation, not only the text information corresponding to the initial image is obtained, but also the obtained initial image and the obtained text information are analyzed, and the target image including the watermark information is generated, so as to achieve the purpose of protecting the copyright information of the image, thereby realizing the technical effect of reducing the limitation of implanting watermark information into the image, and further solving the technical problem of large limitation of implanting watermark information into the image.

[0222] The embodiment of the present disclosure can provide an electronic device, which can be any one of the electronic devices in the electronic device group. Alternatively, in the embodiment, the electronic device can also be replaced by a terminal device such as a mobile terminal.

[0223] Alternatively, in the embodiment, the electronic device can be located in at least one network device of the plurality of network devices of the computer network.

[0224] In the embodiment, the computer terminal can execute the program code in the method.

[0225] Alternatively, FIG. 13 is a block diagram of an electronic device for generating an image according to an embodiment of the present disclosure. As shown in FIG. 13, the electronic device A can include one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, an audio module, and a display.

[0226] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the method and device in the embodiment of the present disclosure. The processor executes various function applications and data processing by running the software programs and modules stored in the memory, that is, implements the method in the above embodiment. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the terminal B through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0227] The processor can invoke the information and application program stored in the memory to execute the following steps: obtaining an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe the semantics of a target image to be converted from the initial image; extracting initial image features from the initial image by using an image generation model, and analyzing the initial image features and the text information to convert the initial image features into target image features satisfying the text information, wherein the image generation model is obtained by training a text-to-image model with watermark information; and adding watermark information to the target image features by using the image generation model to obtain a target image including the watermark information.

[0228] Optionally, the processor can further execute the program code of the following steps: the initial image samples and the target image samples are used to enable the text-to-image model to learn the watermark information, the target image samples at least include the initial image samples including the watermark information, the image generation model is obtained by training the text-to-image model with the watermark information at least by using the initial image samples and text information samples, and the text information samples are used to describe the semantics of the target image samples.

[0229] Optionally, the processor can further execute the program code of the following steps: extracting the initial image features from the initial image by using a feature extraction model in the image generation model, wherein the feature extraction model is obtained by training a variational autoencoder; and converting the initial image features into the target image features satisfying the text information by using a noise prediction model in the image generation model, wherein the noise prediction model is obtained by training a convolutional neural network model.

[0230] Optionally, the processor can further execute the program code of the following steps: adding noise information of the watermark information to the target image features by using the noise prediction model in the image generation model to obtain the target image.

[0231] Optionally, the processor can further execute the program code of the following steps: performing noise adding processing on the target image features; performing noise prediction on the target image features after the noise adding processing by using the noise prediction model to obtain predicted noise information, wherein the predicted noise information includes the noise information of the watermark information; and converting the target image features into the target image including the noise information of the watermark information by using the predicted noise information.

[0232] Optionally, the processor can further execute the program code of the following steps: performing noise removing processing on the target image features by using the predicted noise information, and retaining the noise information of the watermark information to obtain the target image including the noise information of the watermark information.

[0233] Optionally, the processor can further execute program codes of the following steps: outputting noise information of the watermark information by using a loss function of the noise prediction model, wherein the loss function is used to represent a difference between the noise information of the initial image sample and the noise information of the target image sample, and the loss function is used to make the text-to-image model learn the watermark information; and the target image sample is used to at least represent the initial image sample including the watermark information.

[0234] Optionally, the processor can further execute program codes of the following steps: outputting the target image to a third-party model, wherein the watermark information in the target image is used to make the third-party model learn opposite text information of the text information, and a performance index of the third-party model after learning is lower than a performance index of the third-party model before training.

[0235] Optionally, the processor can call information and an application program stored in the memory to execute the following steps: obtaining an initial image sample and a target image sample, wherein the target image sample is used to at least represent the initial image sample including the watermark information; training the text-to-image model by using the initial image sample and the target image sample to obtain the text-to-image model learning the watermark information; and training the text-to-image model learning the watermark information by using at least the initial image sample and a text information sample to obtain an image generation model, wherein the text information sample is used to describe semantics of the target image sample, the image generation model is used to extract initial image features from an input initial image, analyze the initial image features and text information corresponding to the initial image, convert the initial image features into target image features satisfying the text information, and add the watermark information to the target image features to obtain a target image including the watermark information, and the text information is used to describe semantics of the target image to be converted from the initial image.

[0236] Optionally, the processor can further execute program codes of the following steps: encoding the initial image sample by using a target encoder to generate the target image sample including the watermark information.

[0237] Optionally, the processor can further execute program codes of the following steps: encoding the initial image training sample by using an initial encoder to obtain a target image training sample; decoding the target image training sample by using an initial decoder to obtain the watermark information; and training the initial encoder by using the watermark information to obtain the target encoder.

[0238] Optionally, the processor can further execute program codes of the following steps: obtaining a loss function between noise information of the initial image sample and noise information of the target image sample, wherein the loss function is used to represent a difference between the noise information of the initial image sample and the noise information of the target image sample; and training the text-to-image model by using the loss function and a text information sample to obtain the text-to-image model learning the watermark information.

[0239] Optionally, the processor can further execute program codes of the following steps: extracting initial image feature samples from the initial image samples and target image feature samples from the target image samples by using a feature extraction model in the text-to-image model; performing noise adding processing on the initial image feature samples and the target image feature samples respectively; predicting initial noise information in the noise-added initial image feature samples and target noise information in the noise-added target image feature samples by using an initial noise prediction model in the text-to-image model; and obtaining a loss function between the initial noise information and the target noise information.

[0240] Optionally, the processor can further execute program codes of the following steps: controlling the initial noise prediction model in the text-to-image model to learn the watermark information by using the loss function to obtain a noise prediction model, wherein the text-to-image model learned with the watermark information includes the noise prediction model.

[0241] Optionally, the processor can further execute program codes of the following steps: obtaining denoising loss information of the diffusion model, wherein the denoising loss information is used to represent the difference between the target noise information and noise information to be removed from the target noise information; and training the feature extraction model and the trained noise prediction model by using the denoising loss information, the initial image samples and the text information samples to obtain an image generation model.

[0242] Optionally, the processor can further execute program codes of the following steps: obtaining an inverse of the denoising loss information; and training a third-party model by using the watermark information of the target image and the inverse of the denoising loss information, wherein the watermark information of the target image and the inverse of the denoising loss information are used to make the third-party model learn inverse text information of the text information, and a performance index of the trained third-party model is lower than a performance index of the third-party model before training.

[0243] Optionally, the processor can call information and application programs stored in the memory to execute the following steps: obtaining an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe semantics of a style of a target image to be converted from the initial image; extracting an initial image feature from the initial image by using a text-to-image model, and analyzing the initial image feature and the style text information to convert the initial image feature into a target image feature satisfying the style text information, wherein the text-to-image model is obtained by training a text-to-image model learned with watermark information; and adding watermark information to the target image feature by using the text-to-image model to obtain a target image.

[0244] According to the embodiment of the present disclosure, a method for generating an image is provided. When generating an image, an initial image to be processed and text information corresponding to the initial image are obtained, that is, the initial image and semantics of a target image to be converted from the initial image are obtained. Then, an initial image feature is extracted from the initial image by using an image generation model, and the extracted initial image feature and the text information are analyzed, so that the initial image feature is converted into a target image feature meeting the text information. In addition, watermark information is added to the converted target image feature by using the image generation model, and a target image including the watermark information is obtained. Since the text information corresponding to the initial image is obtained during the image generation process, and the obtained initial image and the obtained text information are analyzed to generate the target image including the watermark information, the purpose of protecting the copyright information of the image is achieved, thereby realizing the technical effect of reducing the limitation of implanting watermark information into the image, and further solving the technical problem of large limitation of implanting watermark information into the image.

[0245] Those skilled in the art can understand that the structure shown in FIG. 6 is only schematic, and the computer terminal B can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile Internet device (Mobile Internet Devices, referred to as MID), a PAD, and the like. FIG. 6 does not limit the structure of the computer terminal B. For example, the computer terminal B can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 6, or have a different configuration from that shown in FIG. 6.

[0246] Optionally, FIG. 14 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in FIG. 14, the computer terminal B can include one or more (only one is shown in the figure) processors 1402, a memory 1404, and a transmission device 1406.

[0247] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the method and device for generating an image in the embodiment of the present disclosure. The processor executes various function applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned method for generating an image. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and the remote memory can be connected to the computer terminal B through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0248] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe semantics of a target image to be converted from the initial image; extracting initial image features from the initial image by using an image generation model, and analyzing the initial image features and the text information to convert the initial image features into target image features meeting the text information, wherein the image generation model is obtained by training a text-to-image model learning watermark information; and adding watermark information to the target image features by using the image generation model to obtain a target image including the watermark information.

[0249] Optionally, the processor can further execute program codes of the following steps: the initial image sample and the target image sample are used to make the text-to-image model learn the watermark information, the target image sample at least includes the initial image sample including the watermark information, the image generation model is obtained by training the text-to-image model learning the watermark information at least by using the initial image sample and a text information sample used to describe semantics of the target image sample.

[0250] Optionally, the processor can further execute program codes of the following steps: extracting the initial image features from the initial image by using a feature extraction model in the image generation model, wherein the feature extraction model is obtained by training a variational autoencoder; and calling a noise prediction model in the image generation model to convert the initial image features into the target image features meeting the text information, wherein the noise prediction model is obtained by training a convolutional neural network model.

[0251] Optionally, the processor can further execute program codes of the following steps: adding noise information of the watermark information to the target image features by using the noise prediction model in the image generation model to obtain the target image.

[0252] Optionally, the processor can further execute program codes of the following steps: performing noise adding processing on the target image features; performing noise prediction on the target image features after the noise adding processing by using the noise prediction model to obtain predicted noise information, wherein the predicted noise information includes noise information of the watermark information; and converting the target image features into the target image including the noise information of the watermark information by using the predicted noise information.

[0253] Optionally, the processor can further execute program codes of the following steps: performing noise removing processing on the target image features by using the predicted noise information, and retaining the noise information of the watermark information to obtain the target image including the noise information of the watermark information.

[0254] Optionally, the processor can further execute program codes of the following steps: outputting the noise information of the watermark information by using a loss function of the noise prediction model, wherein the loss function is used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample, and the loss function is used to make the text-to-image model learn the watermark information, and the target image sample is used to represent at least the initial image sample including the watermark information.

[0255] Optionally, the processor can further execute program codes of the following steps: outputting the target image to a third-party model, wherein the watermark information in the target image is used to make the third-party model learn the opposite text information of the text information, and the performance index of the third-party model after learning is lower than the performance index of the third-party model before training.

[0256] Optionally, the processor can call the information and the application program stored in the memory through the transmission device to execute the following steps: obtaining an initial image sample and a target image sample, wherein the target image sample is used to represent at least the initial image sample including the watermark information; training the text-to-image model by using the initial image sample and the target image sample to obtain the text-to-image model learning the watermark information; training the text-to-image model learning the watermark information by using at least the initial image sample and a text information sample to obtain an image generation model, wherein the text information sample is used to describe the semantics of the target image sample, and the image generation model is used to extract initial image features from the input initial image, analyze the initial image features and the text information corresponding to the initial image, convert the initial image features into target image features satisfying the text information, and add the watermark information to the target image features to obtain the target image including the watermark information, and the text information is used to describe the semantics of the target image to be converted from the initial image.

[0257] Optionally, the processor can further execute program codes of the following steps: encoding the initial image sample by using a target encoder to generate the target image sample including the watermark information.

[0258] Optionally, the processor can further execute program codes of the following steps: encoding the initial image training sample by using an initial encoder to obtain a target image training sample; decoding the target image training sample by using an initial decoder to obtain the watermark information; training the initial encoder by using the watermark information to obtain the target encoder.

[0259] Optionally, the processor can further execute program codes of the following steps: obtaining a loss function between the noise information of the initial image sample and the noise information of the target image sample, wherein the loss function is used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample; training the text-to-image model by using the loss function and the text information sample to obtain the text-to-image model with the watermark information.

[0260] Optionally, the processor can further execute program codes of the following steps: extracting the initial image feature sample from the initial image sample and the target image feature sample from the target image sample by using the feature extraction model in the text-to-image model; performing noise adding processing on the initial image feature sample and the target image feature sample respectively; predicting the initial noise information in the noise-added initial image feature sample and the target noise information in the noise-added target image feature sample by using the initial noise prediction model in the text-to-image model; and obtaining a loss function between the initial noise information and the target noise information.

[0261] Optionally, the processor can further execute program codes of the following steps: controlling the initial noise prediction model in the text-to-image model to learn the watermark information by using the loss function to obtain a noise prediction model, wherein the text-to-image model with the watermark information includes the noise prediction model.

[0262] Optionally, the processor can further execute program codes of the following steps: obtaining denoising loss information of the diffusion model, wherein the denoising loss information is used to represent the difference between the target noise information and the noise information to be removed from the target noise information; training the feature extraction model and the trained noise prediction model by using the denoising loss information, the initial image sample and the text information sample to obtain an image generation model.

[0263] Optionally, the processor can further execute program codes of the following steps: obtaining the inverse of the denoising loss information; training the third-party model by using the watermark information of the target image and the inverse of the denoising loss information, wherein the watermark information of the target image and the inverse of the denoising loss are used to make the third-party model learn the inverse text information of the text information, and the performance index of the trained third-party model is lower than that of the third-party model before training.

[0264] Optionally, the processor can call the information and application program stored in the memory through the transmission device to perform the following steps: obtaining an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe the semantics of the style of a target image to be converted from the initial image; using the text-to-image model to extract initial image features from the initial image, and analyzing the initial image features and the style text information to convert the initial image features into target image features meeting the style text information, wherein the text-to-image model is obtained by training a text-to-image model learning watermark information; and adding watermark information to the target image features by using the text-to-image model to obtain the target image.

[0265] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0266] The embodiments of the present disclosure further provide a computer readable storage medium. Optionally, in the present embodiment, the above-mentioned computer readable storage medium can be used to save the program code executed by the method provided in the above-mentioned embodiments.

[0267] Optionally, in the present embodiment, the above-mentioned storage medium can be located in any one of the electronic devices in the electronic device group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0268] Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the following steps: obtaining an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe the semantics of a target image to be converted from the initial image; using an image generation model to extract initial image features from the initial image, and analyzing the initial image features and the text information to convert the initial image features into target image features meeting the text information, wherein the image generation model is obtained by training a text-to-image model learning watermark information; and adding watermark information to the target image features by using the image generation model to obtain the target image including the watermark information.

[0269] Optionally, the computer readable storage medium can further execute program codes of the following steps: the initial image sample and the target image sample are used to make the text-to-image model learn the watermark information, the target image sample at least includes the initial image sample containing the watermark information, the image generation model is trained by using at least the initial image sample and the text information sample to obtain the text-to-image model learning the watermark information, and the text information sample is used to describe the semantics of the target image sample.

[0270] Optionally, the computer readable storage medium can further execute program codes of the following steps: an initial image feature is extracted from the initial image by using a feature extraction model in the image generation model, the feature extraction model is obtained by training a variational autoencoder; and the initial image feature is converted into a target image feature satisfying the text information by calling a noise prediction model in the image generation model, the noise prediction model is obtained by training a convolutional neural network model.

[0271] Optionally, the computer readable storage medium can further execute program codes of the following steps: noise information of the watermark information is added to the target image feature by using the noise prediction model in the image generation model, to obtain the target image.

[0272] Optionally, the computer readable storage medium can further execute program codes of the following steps: the target image feature is subjected to noise adding processing; the noise prediction model is used to predict noise of the target image feature after the noise adding processing, to obtain predicted noise information, the predicted noise information includes the noise information of the watermark information; and the target image feature is converted into the target image including the noise information of the watermark information by using the predicted noise information.

[0273] Optionally, the computer readable storage medium can further execute program codes of the following steps: the target image feature is subjected to noise removing processing by using the predicted noise information, and the noise information of the watermark information is retained, to obtain the target image including the noise information of the watermark information.

[0274] Optionally, the computer readable storage medium can further execute program codes of the following steps: the noise information of the watermark information is output by using a loss function of the noise prediction model, the loss function is used to represent a difference between noise information of the initial image sample and noise information of the target image sample, the loss function is used to make the text-to-image model learn the watermark information, and the target image sample is used to at least represent the initial image sample including the watermark information.

[0275] Optionally, the computer readable storage medium can further execute program codes of the following steps: outputting the target image to a third-party model, wherein the watermark information in the target image is used to make the third-party model learn the opposite text information of the text information, and the performance index of the third-party model after learning is lower than the performance index of the third-party model before training.

[0276] As an optional example, the computer readable storage medium is configured to store program codes for executing the following steps: obtaining an initial image sample and a target image sample, wherein the target image sample is used to represent at least the initial image sample including watermark information; training a text-to-image model by using the initial image sample and the target image sample to obtain a text-to-image model learning watermark information; training the text-to-image model learning watermark information by using at least the initial image sample and a text information sample to obtain an image generation model, wherein the text information sample is used to describe the semantics of the target image sample, and the image generation model is used to extract initial image features from an input initial image, analyze the initial image features and the text information corresponding to the initial image, convert the initial image features into target image features satisfying the text information, and add watermark information to the target image features to obtain a target image including watermark information, and the text information is used to describe the semantics of the target image to be converted from the initial image.

[0277] Optionally, the computer readable storage medium can further execute program codes of the following steps: encoding the initial image sample by using a target encoder to generate the target image sample including watermark information.

[0278] Optionally, the computer readable storage medium can further execute program codes of the following steps: encoding the initial image training sample by using an initial encoder to obtain a target image training sample; decoding the target image training sample by using an initial decoder to obtain watermark information; training the initial encoder by using the watermark information to obtain a target encoder.

[0279] Optionally, the computer readable storage medium can further execute program codes of the following steps: obtaining a loss function between noise information of the initial image sample and noise information of the target image sample, wherein the loss function is used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample; training the text-to-image model by using the loss function and the text information sample to obtain the text-to-image model learning watermark information.

[0280] Optionally, the computer readable storage medium further stores program codes for performing the following steps: extracting initial image feature samples from the initial image samples and target image feature samples from the target image samples by using a feature extraction model in the text-to-image model; performing noise adding processing on the initial image feature samples and the target image feature samples respectively; predicting initial noise information in the noise-added initial image feature samples and target noise information in the noise-added target image feature samples by using an initial noise prediction model in the text-to-image model; and obtaining a loss function between the initial noise information and the target noise information.

[0281] Optionally, the computer readable storage medium further stores program codes for performing the following steps: controlling the initial noise prediction model in the text-to-image model to learn the watermark information to obtain a noise prediction model by using the loss function, wherein the text-to-image model learning the watermark information comprises the noise prediction model.

[0282] Optionally, the computer readable storage medium further stores program codes for performing the following steps: obtaining denoising loss information of the diffusion model, wherein the denoising loss information is used to represent a difference between the target noise information and noise information to be removed from the target noise information; and training the feature extraction model and the trained noise prediction model to obtain an image generation model by using the denoising loss information, the initial image samples and the text information samples.

[0283] Optionally, the computer readable storage medium further stores program codes for performing the following steps: obtaining an inverse of the denoising loss information; and training a third-party model by using the watermark information in the target image and the inverse of the denoising loss information, wherein the watermark information in the target image and the inverse of the denoising loss information are used to make the third-party model learn inverse text information of the text information, and a performance index of the trained third-party model is lower than a performance index of the third-party model before training.

[0284] As an optional example, the computer readable storage medium is configured to store program codes for performing the following steps: obtaining an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe semantics of a style of a target image to be converted from the initial image; extracting an initial image feature from the initial image by using a text-to-image model, and analyzing the initial image feature and the style text information to convert the initial image feature into a target image feature satisfying the style text information, wherein the text-to-image model is obtained by training a text-to-image model learning watermark information; and adding watermark information to the target image feature by using the text-to-image model to obtain the target image.

[0285] The embodiment of the present disclosure further provides a computer program product. Optionally, the computer program product can include computer instructions, and the computer instructions realize the image generation method provided by the above embodiment when executed by a processor.

[0286] The embodiment of the present disclosure further provides a computer program product. Optionally, the computer program product can include a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium can be used to store a computer program, and the computer program realizes the image generation method provided by the above embodiment when executed by a processor.

[0287] The embodiment of the present disclosure further provides a computer program. Optionally, the computer program realizes the image generation method provided by the above embodiment when executed by a processor.

[0288] In the above embodiments of the present disclosure, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0289] In several embodiments provided by the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0290] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0291] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0292] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, and various media that can store program codes.

[0293] The above only describes the preferred embodiments of the present disclosure, and it should be noted that, for those skilled in the art, without departing from the principles of the present disclosure, several improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present disclosure. Industrial applicability

[0294] The scheme provided by the embodiments of the present disclosure can be applied to the generation process of an image, an initial image to be processed is obtained, and text information corresponding to the initial image is obtained, wherein the text information is used to describe semantics to be converted from the initial image to a target image; an image generation model is used to extract initial image features from the initial image, and the initial image features and the text information are analyzed, so as to convert the initial image features into target image features meeting the text information, wherein the image generation model is obtained by training a text-to-image model that learns watermark information; the image generation model is used to add watermark information to the target image features, so as to obtain a target image including the watermark information, thereby solving the technical problem of large limitation of implanting watermark information into an image.

Claims

1. A method for generating an image, comprising: obtaining an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe semantics of a target image to be converted from the initial image; extracting initial image features from the initial image by using an image generation model, and analyzing the initial image features and the text information to convert the initial image features into target image features satisfying the text information, wherein the image generation model is obtained by training a text-to-image model with watermark information; adding the watermark information to the target image features by using the image generation model to obtain the target image including the watermark information.

2. The method of claim 1, wherein, An initial image sample and a target image sample are used to make the text-to-image model learn the watermark information, the target image sample is used to represent at least the initial image sample including the watermark information, the image generation model is obtained by training the text-to-image model with the watermark information at least by using the initial image sample and a text information sample, and the text information sample is used to describe semantics of the target image sample.

3. The method of claim 1, wherein, extracting initial image features from the initial image by using an image generation model, and analyzing the initial image features and the text information to convert the initial image features into target image features satisfying the text information, comprising: extracting the initial image features from the initial image by using a feature extraction model in the image generation model, wherein the feature extraction model is obtained by training a variational autoencoder; converting the initial image features into the target image features satisfying the text information by using a noise prediction model in the image generation model, wherein the noise prediction model is obtained by training a convolutional neural network model.

4. The method of claim 1, wherein, adding the watermark information to the target image features by using the image generation model to obtain the target image including the watermark information, comprising: adding noise information of the watermark information to the target image features by using the noise prediction model in the image generation model to obtain the target image.

5. The method of claim 4, wherein, adding noise information of the watermark information to the target image features by using the noise prediction model in the image generation model to obtain the target image, comprising: performing noise adding processing on the target image features; performing noise prediction on the target image features after the noise adding processing by using the noise prediction model to obtain predicted noise information, wherein the predicted noise information includes noise information of the watermark information; converting the target image features into the target image including noise information of the watermark information by using the predicted noise information.

6. The method of claim 5, wherein, The predicted noise information is carried by the target image features, and converting the target image features into the target image including noise information of the watermark information by using the predicted noise information, comprising: The target image is obtained by using the predicted noise information to denoise the target image feature and retaining the noise information of the watermark information.

7. The method of claim 4, wherein, The method further comprises: The noise information of the watermark information is output by using a loss function of the noise prediction model, wherein the loss function is used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample, and the loss function is used to make the text-to-image model learn the watermark information.

8. The method of any one of claims 1 to 7, wherein, The method further comprises: The target image is output to a third-party model, wherein the watermark information in the target image is used to make the third-party model learn the opposite text information of the text information, and the performance index of the third-party model after learning is lower than the performance index of the third-party model before training.

9. A method for generating a model, comprising: obtaining an initial image sample and a target image sample, wherein the target image sample is used to at least represent the initial image sample including watermark information; training a text-to-image model by using the initial image sample and the target image sample to obtain the text-to-image model learning the watermark information; training the text-to-image model learning the watermark information by using at least the initial image sample and a text information sample to obtain an image generation model, wherein the text information sample is used to describe the semantics of the target image sample, the image generation model is used to extract initial image features from an input initial image, analyze the initial image features and text information corresponding to the initial image, convert the initial image features into target image features satisfying the text information, and add the watermark information to the target image features to obtain a target image including the watermark information, and the text information is used to describe the semantics of the target image to be converted from the initial image.

10. The method of claim 9, wherein, The method further comprises: encoding the initial image sample by using a target encoder to generate the target image sample including the watermark information.

11. The method of claim 10, wherein, The method further comprises: encoding an initial image training sample by using an initial encoder to obtain a target image training sample; decoding the target image training sample by using an initial decoder to obtain the watermark information; training the initial encoder by using the watermark information to obtain the target encoder.

12. The method of claim 9, wherein, Training the text-to-image model by using the initial image sample and the target image sample to obtain the text-to-image model learning the watermark information comprises: obtaining a loss function between the noise information of the initial image sample and the noise information of the target image sample, wherein the loss function is used to represent the difference between the noise information of the initial image sample and the noise information of the target image sample; training the text-to-image model by using the loss function to obtain the text-to-image model learning the watermark information.

13. The method of claim 12, wherein, Obtaining a loss function between noise information of the initial image sample and noise information of the target image sample, including: Extracting initial image feature samples from the initial image sample and target image feature samples from the target image sample by using a feature extraction model in the text-to-image model; Respectively performing noise adding processing on the initial image feature samples and the target image feature samples; Predicting initial noise information in the initial image feature samples after noise adding and target noise information in the target image feature samples after noise adding by using an initial noise prediction model in the text-to-image model; Obtaining the loss function between the initial noise information and the target noise information.

14. The method of claim 13, wherein, Training the text-to-image model by using the loss function to obtain the text-to-image model learning the watermark information, including: Controlling the initial noise prediction model in the text-to-image model to learn the watermark information by using the loss function to obtain a noise prediction model, wherein the text-to-image model learning the watermark information includes the noise prediction model.

15. The method of claim 14, wherein, Training the text-to-image model learning the watermark information by using at least the initial image sample and a text information sample to obtain an image generation model, including: Obtaining denoising loss information, wherein the denoising loss information is used to represent a difference between the target noise information and noise information to be removed from the target noise information; Training the feature extraction model and the noise prediction model by using the denoising loss information, the initial image sample and the text information sample to obtain the image generation model.

16. The method of claim 15, wherein, The method further includes: Obtaining an inverse of the denoising loss information; Training a third-party model by using the watermark information of the target image and the inverse of the denoising loss information, wherein the watermark information of the target image and the inverse of the denoising loss information are used to make the third-party model learn inverse text information of the text information, and a performance index of the third-party model after training is lower than a performance index of the third-party model before training.

17. An image generation method, including: Obtaining an initial image to be processed and style text information corresponding to the initial image, wherein the style text information is used to describe semantics of a style of a target image to be converted from the initial image; Extracting initial image features from the initial image by using a text-to-image model, and analyzing the initial image features and the style text information to convert the initial image features into target image features satisfying the style text information, wherein the text-to-image model is obtained by training a text-to-image model learning watermark information; Adding the watermark information to the target image features by using the text-to-image model to obtain the target image.

18. An image generation system, including: An information input end configured to obtain an initial image to be processed and text information corresponding to the initial image, wherein the text information is used to describe semantics of a target image to be converted from the initial image; An image generation end is configured to extract initial image features from the initial image by using an image generation model, analyze the initial image features and the text information, and convert the initial image features into target image features satisfying the text information, wherein the image generation model is trained by using a text-to-image model learning watermark information. An information output end is configured to output the target image.

19. An electronic device, comprising: a memory storing an executable program; a processor configured to execute the program, wherein the program performs the method of any one of claims 1-17 when executed.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein the computer-readable storage medium controls a device where the computer-readable storage medium is located to perform the method of any one of claims 1-17 when the executable program is executed.

21. A computer program product, characterised in that, The computer program is executed by a processor to implement the method of any one of claims 1-17.

Images