Picture generation method, device and equipment and computer storage medium

Abstract

The invention provides a picture generation method, device and equipment and a computer storage medium, which are applied to the field of computers and are used for realizing the generation of a picture capable of quickly distinguishing a foreground from a background. The method at least comprises the following steps: acquiring an original noise picture and a preset text label; performing the following operations on the text label and the original noise picture through the target generation model: obtaining a corresponding denoising feature based on a coding result corresponding to the text label and an image feature corresponding to the original noise picture; obtaining a target generation picture and a target mask picture based on a decoding result of the de-noising feature; the target mask picture is a mask picture for the target generated picture; the target generation model is obtained by training a loss function constructed based on a sample mask picture and a target sample cross feature; the sample mask picture is a mask picture for the real sample picture; the target sample cross feature is obtained by performing cross attention processing on the basis of the second sample description information and the sample noise picture.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method, apparatus, and device for image generation, as well as a computer storage medium. Background Technology

[0002] With the continuous development of computer technology, significant breakthroughs have been made in the field of image generation. The development of various models based on the diffusion model has demonstrated impressive performance in text-guided image generation, and can now effectively achieve the goal of text-to-image generation.

[0003] This functionality is achieved primarily because the diffusion model has a unique training method.

[0004] Specifically, the training of a typical diffusion model consists of two parts: a forward process and a backward process. In the forward process, the diffusion model gradually adds noise to a sample image until the image becomes a noisy image containing no information. In the backward process, the diffusion model reverses the noise-adding operations of the forward process, denoising the noisy image step by step until all backward operations are completed, thus obtaining or generating the target image. At this point, by utilizing the difference in data distribution between the target image and the sample image, the parameters of the diffusion model can be adjusted, gradually training the model until it can generate a relatively realistic target image.

[0005] To achieve the goal of generating images from text, text labels can be added to guide the denoising process during the training process. Specifically, the text description information in the text labels corresponding to the sample images is used to guide the target direction of the denoising process, thereby generating the image content corresponding to the text description information, making the generated target image closer to the sample image.

[0006] Thus, after obtaining a fully trained diffusion model, the reverse process described above can be directly used to guide the diffusion model to perform targeted denoising on noisy images by inputting text labels into the diffusion model, thereby generating target images that correspond to the information in the text labels.

[0007] However, the target image generated by the diffusion model usually only includes pixel color information, and the image content described in the text label is presented as a whole in the target generated image in the form of foreground and background. This results in the foreground and background being deeply blended in the target generated image, making them difficult to distinguish.

[0008] In downstream image processing applications, such images require significant manual or time investment to acquire high-precision foreground images or to add or process other image effects.

[0009] Therefore, there is an urgent need for a new image generation method that can quickly distinguish between foreground and background. Summary of the Invention

[0010] This application provides a method, apparatus, device, and computer storage medium for generating images, which can generate images with a clear distinction between the foreground and background.

[0011] In a first aspect, embodiments of this application provide a method for generating an image, including:

[0012] Obtain the original noisy image and preset text labels; the text labels include: first descriptive information of the foreground image and second descriptive information of the background color;

[0013] The target generation model performs the following operations on the text label and the original noisy image: denoising is performed based on the encoding result corresponding to the text label and the image features corresponding to the original noisy image to obtain corresponding denoising features; based on the decoding result of the denoising features, a target-generated image and a target mask image are obtained; wherein the target-generated image includes: a foreground image described by the first description information and a background color described by the second description information, and the target mask image is a mask image for the target-generated image;

[0014] The target generation model is trained using a loss function constructed from the sample mask image and the cross-features of the target samples; the sample mask image is a mask image for the real sample image; the sample text label includes: a first sample description information of the foreground image and a second sample description information of the background color in the real sample image; the target sample cross-features are obtained by cross-attention processing based on the second sample description information and the sample noise image.

[0015] Secondly, embodiments of this application provide an image generation apparatus, comprising:

[0016] The acquisition module is used to acquire the original noisy image and preset text labels; the text labels include: first descriptive information of the foreground image and second descriptive information of the background color;

[0017] The generation module is configured to perform the following operations on the text label and the original noisy image using a target generation model: performing denoising processing based on the encoding result corresponding to the text label and the image features corresponding to the original noisy image to obtain corresponding denoising features; and obtaining a target generated image and a target mask image based on the decoding result of the denoising features; wherein the target generated image includes: a foreground image described by the first description information and a background color described by the second description information, and the target mask image is a mask image for the target generated image;

[0018] The training module is used to train a target generation model based on a loss function constructed from the sample mask image and the cross features of the target samples; the sample mask image is a mask image for the real sample image; the sample text label includes: a first sample description information of the foreground image and a second sample description information of the background color in the real sample image; the target sample cross features are obtained by cross-attention processing based on the second sample description information and the sample noise image.

[0019] Optionally, the generation module is used to perform denoising processing based on the encoding result corresponding to the text label and the image features corresponding to the original noisy image. Specifically, when obtaining the corresponding denoising features, it is used to:

[0020] Based on the encoding results corresponding to the text labels and the image features of the original noisy image, cross-attention processing is performed to obtain corresponding cross-attention features; the cross-attention features are used to indicate the correlation between the text labels and the original noisy image.

[0021] Based on the cross-attention features, the original noisy image is denoised to obtain the corresponding denoising features.

[0022] Optionally, when the generation module performs cross-attention processing based on the encoding result corresponding to the text label and the image features of the original noisy image to obtain the corresponding cross-attention features, it is specifically used for:

[0023] Semantic features are extracted from the text tags to obtain the corresponding semantic features as the encoding result; and image features are extracted from the original noisy image to obtain the corresponding image features.

[0024] The semantic features are transformed to obtain the cross-attention query vector, and the image features are transformed to obtain the cross-attention key vector and the cross-attention value vector.

[0025] Cross-attention processing is performed based on the cross-attention query vector, the cross-attention key vector, and the cross-attention value vector to obtain the corresponding cross-attention features.

[0026] Optionally, when the generation module performs cross-attention processing based on the cross-attention query vector, the cross-attention key vector, and the cross-attention value vector to obtain the corresponding cross-attention features, it is specifically used for:

[0027] For each element in the cross-attention query vector, perform the following operations:

[0028] Obtain the association relationship between a vector element in the cross-attention query vector and each vector element contained in the cross-attention key vector;

[0029] Based on the aforementioned relationship, the attention weight of the vector element for each vector element in the cross-attention key vector is determined.

[0030] Based on the attention weights, the corresponding vector elements in the cross attention value vector are weighted to obtain the sub-cross features corresponding to the vector element.

[0031] Based on the sub-cross features corresponding to each vector element in the cross-attention query vector, the corresponding cross-attention features are obtained.

[0032] Optionally, the training module is used for:

[0033] Based on the training samples in the training sample dataset, iterative training is performed on the model to be trained; wherein each training sample includes: a sample foreground image containing only foreground images, and an initial mask image of the sample foreground image; wherein the following operations are performed during one iterative training cycle:

[0034] Select a training sample, and based on the initial mask image in the training sample, fuse the target color into the background of the sample foreground image to obtain the corresponding sample real image. Then, add noise to the sample real image to obtain the corresponding sample noise image.

[0035] Obtain the sample text tags of the real sample image, and perform denoising processing on the sample noisy image based on the sample text tags to obtain the corresponding sample generated image and sample generated mask image;

[0036] Based on the sample-generated image and the sample-generated mask image, combined with the training sample, the model to be trained is subjected to this round of iterative training.

[0037] Optionally, the training module is used to obtain sample text labels for the real sample images. Based on the sample text labels, when denoising the noisy sample images to obtain the corresponding generated sample images and sample generated mask images, it is specifically used for:

[0038] Based on the encoding results corresponding to the sample text labels and the image features of the sample noise images, cross-attention processing is performed to obtain the corresponding sample cross-attention features;

[0039] Based on the sample cross-attention features, the sample noise image is denoised to obtain the corresponding sample denoising features;

[0040] Based on the decoding results of the sample denoising features, the generated sample image and the generated sample mask image are obtained.

[0041] Optionally, the training module is used to generate an image and a mask image based on the sample, and combine them with a training sample to perform the current round of iterative training on the model to be trained. Specifically, it is used for:

[0042] Based on the difference between the generated image of the sample and the real image of the sample, a corresponding first loss function is obtained;

[0043] Extract the target sample cross features from the sample cross attention features;

[0044] Feature extraction is performed on the sample mask image to obtain sample mask features with the same dimension as the cross features of the target sample, and a corresponding second loss function is obtained based on the difference between the sample mask features and the cross features of the target sample.

[0045] Based on the first loss function and the second loss function, the parameters of the model to be trained are adjusted.

[0046] Optionally, when obtaining the corresponding second loss function based on the difference between the sample mask features and the target sample cross features, it is specifically used for:

[0047] Based on the difference between the sample mask features and the target sample cross features, a first sub-loss function is obtained;

[0048] Based on the difference between the sample mask image and the sample generated mask image, a second sub-loss function is obtained;

[0049] Based on the first sub-loss function and the second sub-loss function, the corresponding second loss function is obtained.

[0050] Optionally, the target color is green or blue.

[0051] Thirdly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0052] Fourthly, embodiments of this application provide a computer device, including:

[0053] Memory, used to store program instructions;

[0054] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect according to the obtained program instructions.

[0055] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method described in the first aspect.

[0056] The beneficial effects of this application are as follows:

[0057] This application proposes an image generation method. A target generation model is used to denoise the encoded results of the input text tags and the image features of the original noisy image. After obtaining the corresponding denoised features, the method uses these features to obtain the corresponding target-generated image and target mask image. The target-generated image includes the foreground image and background color described by the text tags, while the target mask image is a mask image for the target-generated image. In this way, the encoding results of the text tags guide the denoising direction of the original noisy image, ensuring that the denoised features include features that characterize the foreground image and background color. This allows for the direct generation of the corresponding target-generated image and target mask image. Furthermore, the background color features from the denoised features are used to generate the corresponding target mask image, enabling accurate differentiation between the foreground and background. Thus, by masking the target-generated image with the target mask image, the corresponding foreground or background image can be quickly obtained.

[0058] On the other hand, during the training of the target generation model, the second sample description information of the background color corresponding to the real sample image is used to perform cross-attention processing with the sample noisy image to obtain the target sample cross features representing the correlation between the background color and the generated image. This allows for the explicit acquisition of the denoising features corresponding to the background color during the denoising process. Furthermore, this target sample cross feature is used to construct a loss function with the sample mask image, and this function is used to train the target generation model. This results in a target generation model that can more accurately generate denoising features that reflect the positional range of the background color in the image, and based on this, generate the corresponding target mask image. This approach proposes using the difference between the target sample cross features corresponding to the second sample description information and the sample mask image as a constraint to train the target generation model, improving the utilization rate of cross-attention features. Simultaneously, using only the target sample cross features to constrain the denoising generation improves the speed and effectiveness of generating the target mask image. Attached Figure Description

[0059] Figure 1 This is one application scenario of the image generation method provided in the embodiments of this application;

[0060] Figure 2 A flowchart illustrating the training method for the target generation model provided in this application embodiment;

[0061] Figure 3 A schematic diagram of a training sample provided in an embodiment of this application;

[0062] Figure 4 A flowchart illustrating a one-round training method for the target generation model provided in this application embodiment;

[0063] Figure 5 A logical schematic diagram of the background fusion method provided in the embodiments of this application;

[0064] Figure 6 A logical schematic diagram of the noise addition processing method provided in the embodiments of this application;

[0065] Figure 7 A flowchart illustrating a method for generating images from samples, provided in an embodiment of this application;

[0066] Figure 8 A flowchart illustrating a method for obtaining a second loss value provided in an embodiment of this application;

[0067] Figure 9 A logical schematic diagram of a training method for a target generation model provided in an embodiment of this application;

[0068] Figure 10A schematic diagram illustrating a method for obtaining a second loss value provided in an embodiment of this application;

[0069] Figure 11 A schematic diagram of a U-NET network structure provided in an embodiment of this application;

[0070] Figure 12 A flowchart illustrating a method for generating an image provided in an embodiment of this application;

[0071] Figure 13 A flowchart illustrating a method for obtaining sub-intersection features provided in an embodiment of this application;

[0072] Figure 14 A logical schematic diagram of a method for obtaining sub-intersection features provided in an embodiment of this application;

[0073] Figure 15 A logical diagram illustrating the method for dividing the background and foreground provided in the embodiments of this application;

[0074] Figure 16 A schematic diagram illustrating the method for obtaining an SKS token provided in an embodiment of this application;

[0075] Figure 17 A logical schematic diagram of a method for adjusting transparency information provided in an embodiment of this application;

[0076] Figure 18 A logical schematic diagram of an image generation method provided in an embodiment of this application;

[0077] Figure 19 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0078] Figure 20 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application;

[0079] Figure 21 This is a schematic diagram of the hardware structure of another electronic device in an embodiment of this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0081] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0082] (1) Text-to-Image: This is an important technological direction in the field of AI-generated content. It uses deep learning models to convert user-input text descriptions into images, creating rich and colorful visual effects.

[0083] (2) Diffusion Model: The diffusion model is a generative model that generates new samples by simulating the diffusion process of data. This model produces high-quality results in multiple fields such as images, text, and audio, and plays an important role in the field of AI-generated content. The basic principle of the diffusion model includes two key steps: forward diffusion and backward denoising diffusion. In the forward diffusion process, the model gradually adds noise to the original data until the data is completely transformed into noise. This process is pre-defined, and the addition of noise is controllable. In the backward denoising diffusion process, the model learns how to gradually remove noise from the noisy data to recover the original data. In terms of applications, the diffusion model has been used for various tasks, such as image and audio generation, text-to-image synthesis, image editing and restoration, etc. For example, in text-to-image synthesis systems, the diffusion model, by learning conditional distributions, can generate images that match the text content based on input text prompts.

[0084] (3) Cross-Attention Mechanism: The cross-attention mechanism allows the model to effectively fuse information from different sequences when processing data of different modalities. This mechanism is particularly suitable for multimodal tasks, such as scenarios combining text and image information. In cross-attention, the two input sequences must have the same dimension, but can be of different modalities, such as text and images. One sequence serves as the query (Q), defining the length of the output sequence; while the other sequence provides the key (K) and value (V) inputs. In this way, the cross-attention layer can combine the encoder's output with the decoder's current state, allowing the decoder to focus on information in the encoder output that is relevant to the current decoding position.

[0085] (4) A variational autoencoder (VAE) is a generative model that uses deep learning techniques to learn the latent representation of data. A VAE consists of two parts: an encoder and a decoder. The encoder is responsible for mapping the input data to a distribution in a latent space, while the decoder generates data from this latent distribution.

[0086] (5) U2-NET Network Structure: U2-NET is a deep learning network structure for salient object detection. Its design is based on a nested U-shaped structure, enabling it to capture contextual information at different scales while maintaining low computational cost. Key features of this network structure include: Two-level nested U-structure: The design of U2-Net allows the network to reach deeper layers while maintaining high-resolution feature maps, without significantly increasing memory and computational costs. Residual U-Block (RSU): The RSU block is a core component of U2-Net. It mixes receptive fields of different sizes, enabling it to capture more contextual information at different scales and increase depth through pooling operations without significantly increasing computational cost.

[0087] (6) Four-channel image format: In the fields of image processing and computer vision, "four channels" usually refers to the four main components of an image. These components can be color channels, transparency channels, or other types of data channels. The most common four-channel image types are: RGBA image: This is the most common four-channel image format, where: R: red channel, G: green channel, B: blue channel, A: alpha channel, representing transparency; CMYK image: This is an image format mainly used for printing, where: C: cyan channel, M: magenta channel, Y: yellow channel, K: black channel.

[0088] It should be noted that the embodiments of this application involve operations such as obtaining sample images and foreground masks. When the following embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, when relevant data is needed, relevant volunteers can be recruited and relevant agreements authorizing the volunteers to authorize data can be signed, and then the data of these volunteers can be used for implementation; or, implementation can be carried out within the authorized scope of an organization, and the following implementation methods can be used to identify internal members by using the data of internal members; or, the relevant data used in the specific implementation are all simulated data, such as simulated data generated in a virtual scene.

[0089] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0090] The technical concept of the technical solution of the embodiments of this application will be briefly described below.

[0091] With the continuous development of science and technology, image generation technology has made great strides. More and more models are being developed and applied to the field of image generation, making text-to-image technology able to meet a wide range of target needs.

[0092] In this technique of text-guided image generation, the diffusion model plays a major role. This model can generate realistic image data based on the text description information input by the user.

[0093] This image generation capability is made possible by the unique training method of the diffusion model. The training process of the diffusion model consists of two parts: a forward process and a backward process. In the forward process, the diffusion model gradually adds noise to a sample image until the image becomes a noisy image containing no information. In the backward process, the diffusion model reverses the noise-adding operations in the forward process, gradually denoising the noisy image until all backward operations are completed, thus obtaining or generating the target image. At this point, by utilizing the difference in data distribution between the target image and the sample image, the parameters of the diffusion model can be adjusted, gradually training the diffusion model until it can generate a relatively realistic target image.

[0094] Thus, after obtaining a fully trained diffusion model, the reverse process described above can be directly used to guide the diffusion model to perform targeted denoising on noisy images by inputting text labels into the diffusion model, thereby generating target images corresponding to the information in the text labels.

[0095] However, the target image generated by the diffusion model usually only includes pixel color information, and the image content described in the text label is presented as a whole in the target generated image in the form of foreground and background. This results in the foreground and background being deeply blended in the target generated image, making them difficult to distinguish.

[0096] In downstream image processing applications, such images require significant manual or time investment to acquire high-precision foreground images or to add or process other image effects.

[0097] In view of this, embodiments of this application provide a method for generating an image. In this method, it is necessary to obtain an original noisy image and a preset text label. The text label includes first descriptive information for describing the foreground image and second descriptive information for describing the background color. The original noisy image is an image full of noise.

[0098] After obtaining the original noisy image and text labels, the text labels and the original noisy image can be processed accordingly using a target generation model. Specifically, based on the encoding results corresponding to the text labels and the image features corresponding to the original noisy image, denoising processing is performed to obtain corresponding denoising features. Thus, these denoising features include the corresponding features of the foreground image and background color in the text labels.

[0099] Then, based on the decoding results of the denoising features, the generated target image and the target mask image are obtained. The generated target image includes the foreground image described by the first descriptive information and the background color described by the second descriptive information, while the target mask image is a mask image of the generated target image. At this point, reconstructing the denoising features allows the generation of the corresponding generated target image and target mask image. The target mask image is then used to distinguish the foreground and background in the generated target image.

[0100] On the other hand, for the target generation model, during its training process, the second sample description information of the background color corresponding to the real sample image is used to perform cross-attention processing with the sample noisy image to obtain the target sample cross features that represent the correlation between the background color and the generated image. In this way, the denoising features corresponding to the background color can be clearly obtained during the denoising process. Furthermore, the target sample cross features and the sample mask image are used to construct a loss function, and the target generation model is trained accordingly. This allows the target generation model to more accurately generate denoising features that reflect the position range of the background color in the image, and generate the corresponding target mask image based on these features.

[0101] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0102] See Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application, such as... Figure 1 As shown, this scenario may include multiple terminal devices 101 and a server 102.

[0103] Terminal device 101 can be any device with network communication capabilities, such as a mobile phone, laptop, tablet computer, notebook computer, desktop computer, smart TV, smart in-vehicle device, smart wearable device, e-book reader, etc. Terminal device 101 can be used to acquire text tags and provide users with a display interface for images generated based on the text tags. Server 102 is used to generate models for training and application objectives. It can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms, but it is not limited to these.

[0104] Both server 102 and terminal device 101 may include one or more processors, memory, and I / O interfaces for interaction. The memory of server 102 and terminal device 101 may also store program instructions required for execution in the image generation method provided in this application embodiment. These program instructions, when executed by the processor, can be used to implement the image generation method provided in this application embodiment.

[0105] In this embodiment, each terminal device 101 and the server 102 can communicate directly or indirectly through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network. Of course, it can also be other possible networks, and this embodiment does not limit them.

[0106] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and not specifically limited in this embodiment. The image generation method proposed in this embodiment can be completed by the server 102 alone, or jointly by the server and terminal device 101. For example, when completed by the server alone, the server can deploy the target generation model provided in this embodiment. The server performs the task of acquiring the original noisy image and text tags. Then, based on the description in the text tags, the target generation model performs directional denoising on the original noisy image, thereby generating the corresponding image. When the image generation method is completed jointly by the server and terminal device, the terminal device can communicate with the server, receive text tags from the user, and transmit them to the server. After the server generates the image based on the text tags, it feeds the result back to the terminal device so that the terminal device can display the image generation result to the user.

[0107] The following describes the image generation method provided by the exemplary embodiments of this application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0108] Before describing the specific steps of the image generation method proposed in the embodiments of this application, it is necessary to first introduce the target generation model for implementing this method.

[0109] See Figure 2 This is a flowchart illustrating a training method for a target generation model provided in an embodiment of this application. The execution entity of the target generation model training process can be, for example, […]. Figure 1 The server shown.

[0110] like Figure 2 As shown, the specific implementation steps of this method are as follows:

[0111] Step S201: Obtain the training sample dataset, wherein the training sample dataset includes multiple training samples, and each training sample includes: a sample foreground image containing only the foreground image, and an initial mask image corresponding to the sample foreground image.

[0112] like Figure 3 As shown, the training samples consist of a sample foreground image and a corresponding initial mask image. The sample foreground image is an image that only contains the foreground image, meaning that apart from the main foreground image, the positions of the remaining pixels are transparent. For example, it is similar to an image in the Portable Network Graphics (PNG) format. In this type of image, in addition to the color information corresponding to each pixel, there is also transparency information for each pixel. Using this transparency information, certain pixel positions in the image can be adjusted to have 100% transparency. In this way, the resulting image can only display certain specific images, such as the foreground image.

[0113] The initial mask image is a type of mask information that corresponds to the sample foreground image. The pixels contained in the initial mask image correspond one-to-one with all the pixels in the sample foreground image, and the values ​​of the pixels in the initial mask image are used to characterize whether each pixel in the sample foreground image needs to undergo further operations.

[0114] like Figure 3 As shown, in the sample foreground image, a human foreground image may exist only in the center of a fixed-size image, while other pixel positions do not contain pixel information. In the corresponding initial mask image, the pixels corresponding to the human foreground image in the sample foreground image represent those pixels participating in subsequent operations, while the pixels corresponding to other pixel positions in the sample foreground image represent those pixels not participating in subsequent operations. It should be noted that using the initial mask image to represent the mask information corresponding to the sample foreground image is only one representation method; other methods such as matrices can also represent the corresponding mask information, and this application does not impose any limitations on this. For example, in... Figure 3 In the initial mask image shown, the size of the image is exactly the same as the size of the sample foreground, and the white part represents the human foreground image in the sample foreground image, and the black part represents the part of the sample foreground image other than the human foreground image.

[0115] After acquiring multiple training samples and completing the construction of the training sample dataset, the server can then proceed with the following operations:

[0116] Step S202: Based on the training samples in the training sample dataset, perform iterative training on the model to be trained until the target generative model is obtained.

[0117] The server needs to perform iterative training on the model to be trained using multiple training samples from the training sample dataset until the training reaches the convergence condition, that is, the number of iterations reaches a set threshold, or the model loss value is less than a set loss threshold, etc. In this way, the corresponding target generative model can be obtained.

[0118] Before reaching the convergence condition, during a specific round of iterative training, the server needs to perform the following operations to train the model to be trained:

[0119] Please refer to Figure 4 The flowchart of a training method provided in the application embodiment is as follows: Figure 4 As shown, the specific implementation steps of this method are as follows:

[0120] Step S401: Select a training sample, and based on the initial mask image in the training sample, fuse the target color into the background of the sample foreground image to obtain the corresponding real sample image. Then, add noise to the real sample image to obtain the corresponding noisy sample image.

[0121] At the start of this training round, the server needs to select a training sample from the training sample dataset, and then perform subsequent training operations based on the foreground image of the training sample and the initial mask image. In this process, the foreground image is first processed to obtain the corresponding noisy image.

[0122] The processing of the sample foreground image is divided into two parts. The first part is to perform background fusion processing on the sample foreground image, that is, to add background to the sample foreground image to obtain the corresponding real sample image. The second part is to perform noise addition processing on the sample real image, that is, to add noise to the image until the image is full of noise to obtain the corresponding sample noise image.

[0123] In the background fusion process, the server needs to use the foreground mask and its complement, as indicated by the initial mask image in the training samples, to process the sample foreground image separately. This results in the foreground portion of the sample foreground image remaining unprocessed, while the background portion is modified to the corresponding target color. Specifically, this background fusion process can be represented by the following formula 1:

[0124] S=a×F+(1-a)×B (Formula 1)

[0125] Where S represents the sample image, a represents the foreground mask, 1-a represents the complement of the foreground mask, F represents the foreground image in the sample foreground image, and B represents the background color to be fused into the image. The target color fused into the sample image can be any color, such as red, pink, purple, black, white, sky blue, etc. Thus, by setting the background fusion operation to be performed during training, a single training sample can meet the requirements of training with multiple colors, reducing the cost of obtaining training samples while improving the efficiency of generating multiple background colors.

[0126] In one possible embodiment, the target color can be selected from a fixed set of colors, such as a solid green background (RGB value (0,255,0)) or a solid blue background (RGB value (0,0,255)). This can significantly reduce the training cost of the model to be trained. Furthermore, since images with blue or green backgrounds are easier to process for subsequent image effects and other downstream applications, training the model with blue or green as the background color can yield a more practical target generation model.

[0127] like Figure 5 As shown, the sample foreground image may contain a human foreground image, and the initial mask image displayed as an image corresponds to the human foreground image's boundary. Therefore, during background fusion processing, the complement of the foreground mask displayed in the initial mask image can be used to select a color as the target color, and color fusion processing can be performed on the background portion of the sample foreground image. In this way, an image that simultaneously contains both the human foreground image and background colors can be obtained.

[0128] It should be noted that after background fusion processing, since only the initial mask image is used to process the background in the sample foreground image, the distinction between foreground and background in the resulting sample ground image can still be made using the initial mask image. Therefore, essentially, the initial mask image is also a mask image for the sample ground image. For ease of explanation, this application will use the sample mask image to describe the corresponding mask image for the sample ground image. However, in reality, the sample mask image and the initial mask image are the same mask image.

[0129] After completing the background fusion process, the server can add noise to the image until a corresponding sample noise image is obtained.

[0130] The specific noise-adding method can utilize the noise-adding operation during the training process of the diffusion model, such as... Figure 6As shown, a Markov chain (MC) is used to gradually add noise to the real sample image at each time step until the final noisy sample image is obtained.

[0131] After acquiring the sample noisy images, the training process has completed the forward process (the part that adds noise to the images). Next, the server needs to perform the subsequent reverse process (the part that removes noise from the images) on the model to be trained.

[0132] Step S402: Obtain the sample text labels of the real sample images. Based on the sample text labels, perform denoising processing on the sample noisy images to obtain the corresponding sample generated images and sample generated mask images.

[0133] After completing the background fusion processing of the sample foreground image, the server needs to obtain the sample text labels of the sample real image, and then perform denoising processing on the sample noisy image according to the description content in the sample text labels to obtain the corresponding sample generated image.

[0134] Optionally, the server can achieve the goal of obtaining sample text labels in several different ways:

[0135] For example, in one possible approach, each training sample may also include textual description information of the foreground image. This textual description information describes the image content in the foreground image. Thus, when the server obtains sample text labels, it can use the textual description information of the foreground image as the first sample description information describing the foreground image in the sample text labels; then, it can use the background color fused into the foreground image as the second sample description information describing the background color in the sample text labels. The textual description information corresponding to the foreground image can be obtained through manual annotation or by directly reading the corresponding text semantics from the image using image recognition technology, etc. This application does not impose any restrictions on this.

[0136] In another possible approach, the server can directly perform text annotation on the sample real images, directly obtaining the corresponding sample text labels from the images. For example, a pre-trained visual model or language model can be used to obtain the corresponding text description information of the image through one or more methods such as retrieval, statistics, attention, or multimodal approaches, thereby obtaining sample text labels containing both first and second descriptive information. For example, for a sample real image, its sample text label could be: "A girl with yellow hair wearing a red dress against a green background." In this way, the first sample description information is: "A girl with yellow hair wearing a red dress," and the second sample description information is: "Green background." This eliminates the need for pre-annotating the training samples, reducing the cost and difficulty of obtaining the training sample dataset.

[0137] Thus, after obtaining the sample text labels from the real sample images, the server can perform denoising on the noisy sample images based on the sample text labels to obtain the corresponding generated sample images and sample generation mask images:

[0138] See Figure 7 The flowchart below illustrates a method for obtaining a sample generation image and a sample generation mask image, as provided in an embodiment of this application. Figure 7 As shown, the specific implementation steps of this method are as follows:

[0139] Step S701: Based on the encoding results corresponding to the sample text labels and the image features of the sample noisy images, perform cross-attention processing to obtain the corresponding sample cross-attention features.

[0140] As mentioned above, cross-attention is a way to obtain correlations between multiple modalities. Therefore, after obtaining the encoding results corresponding to the sample text labels and the image features of the sample noise image, the server can obtain the correlation between the sample text labels and the sample noise image based on cross-attention, thereby obtaining the corresponding sample cross-attention features; in other words, the sample cross-attention features can characterize the layout information of various image elements in the sample text labels in the sample noise image.

[0141] Step S702: Based on the sample cross-attention features, perform denoising processing on the sample noisy image to obtain the corresponding sample denoising features.

[0142] Since the sample cross-attention features represent the correlation between the sample text labels and the sample noisy images, meaning they contain the layout information corresponding to each image element, the server can directly guide the denoising direction of the sample noisy images based on the obtained sample cross-attention features, thereby obtaining the corresponding sample denoising features. The denoising process is the same as the denoising method in the aforementioned diffusion model, which will not be elaborated upon further in this application.

[0143] It should be noted that during the denoising process, the server can extract the target sample cross-features corresponding to the second sample description information separately, which will serve as the data basis for background feature extraction during the sample denoising process. Since the description of the foreground image in the first sample description information may be complex and variable, the simple and stable second sample description information can be used as the basis for generating denoising features.

[0144] For example, when acquiring sample text labels, the server can separately label the text representing the background color (usually words or phrases indicating color in the background description). This allows the server to directly determine that the text corresponds to the second sample description information, i.e., the background color, based on the label in subsequent processes. In this way, the server only needs to focus on the layout information of one element in the sample noisy image to complete the distinction between the foreground and the background.

[0145] Step S703: Based on the decoding results of the sample denoising features, obtain the sample generated image and the sample generated mask image.

[0146] After obtaining the denoising features, the server has a clear understanding of the distribution of various data in the image and mask to be generated. Therefore, the server only needs to use the mask results of the denoising features to reconstruct the corresponding sample generated image and sample generated mask image.

[0147] In the above approach, cross-attention processing is used to obtain the correlation between two different modalities, clarifying the layout of the image in the sample text label within the sample noisy image. Specifically, by focusing on the background color feature, the pixel position of the background color can be determined. This clarifies the standard for distinguishing between foreground and background, reduces the computational cost required for this distinction, and improves the efficiency of image generation.

[0148] After obtaining the sample generated image and sample generated mask image using the above methods, the server continues to perform the following operations:

[0149] Step S403: Based on the sample-generated image and the sample-generated mask image, and combined with the aforementioned training sample, perform this round of iterative training on the model to be trained.

[0150] After generating the sample image and sample mask image, the server can use the generated image, sample mask image, and previous real image and sample mask image to perform the current round of iterative training on the model to be trained.

[0151] Specifically, the server can use the difference between the generated image and the real image, as well as the difference between the target sample cross features and the sample mask image obtained in the above process, to adjust the parameters of the model to be trained, so as to ensure that the difference becomes smaller and smaller in subsequent training rounds.

[0152] In one possible implementation, the server may perform the following operations when making this round of parameter adjustments:

[0153] First, based on the difference between the generated image and the real image, the corresponding first loss function is obtained.

[0154] For the generated sample image and the real sample image, the server can compare the differences in pixel information at corresponding pixel positions to construct a corresponding first loss function. This first loss function can be of various types, such as mean squared error, mean absolute error, or histogram loss, etc.

[0155] Secondly, the target sample cross features corresponding to the second sample description information are extracted from the sample cross features. Then, feature extraction is performed on the sample mask image to obtain the sample mask features with the same dimension as the target sample cross features. Based on the difference between the sample mask features and the target sample cross features, the corresponding second loss function is obtained.

[0156] For sample mask images, when calculating the loss value between them and the target sample cross matrix, dimensionality reduction is required to make their dimensions the same as those of the target sample cross matrix. For example, ... Figure 8 As shown, feature extraction, or downsampling, can be performed on the sample mask image. This allows for the extraction of foreground mask features with the same cross-feature dimension as the target sample from the sample mask image. Then, based on the difference between the sample mask features and the target sample cross-feature, a second loss function can be obtained.

[0157] Finally, based on the first loss function and the second loss function, the parameters of the model to be trained are adjusted.

[0158] In this approach, the cross-features of the generated sample image and the target sample serve as dual constraints to adjust the model parameters, ensuring both the accuracy of the generated image and the accuracy of foreground / background differentiation (i.e., the accuracy of the generated sample mask image). Furthermore, when obtaining the second loss value, the loss is calculated using the sample mask features corresponding to the sample mask image. This approach ensures computational accuracy while reducing computational dimensionality and improving the efficiency of loss value calculation.

[0159] The following will combine Figure 9 The content shown provides a complete overview of one possible implementation of the model training method proposed in this application.

[0160] like Figure 9 As shown, the server first needs to obtain a training sample containing a sample foreground image and an initial mask image. Then, using the initial mask image, background fusion processing is performed on the sample foreground image to obtain the corresponding sample ground image. At this point, the operation on the sample ground image is divided into two parts: first, noise addition processing, which can be performed using an encoder in a VAE to gradually add noise to the sample ground image, thus obtaining the corresponding sample noise image; second, text annotation is used to process the sample ground image to obtain the corresponding sample text labels, and then a pre-trained text encoder is used to extract features from the sample text labels to obtain the corresponding encoding results. This encoding result can guarantee the semantic features corresponding to the sample text labels.

[0161] Using the obtained encoding results and the sample noisy image, corresponding denoising processing is performed. The denoising process corresponds to the noise addition process and is implemented using the denoising processing in the diffusion model. This yields the corresponding sample denoising features. Then, the decoder in the VAE processes these features to obtain the corresponding sample generated image and sample generated mask image. After obtaining the sample generated image, the server can determine the corresponding first loss function based on the difference between the sample generated image and the real sample image.

[0162] On the other hand, during the denoising process, since the target sample cross features corresponding to the second sample description information are used to distinguish the sample noise images, when the target sample cross features are obtained, the server also needs to compare the target sample cross features with the corresponding sample mask images to determine the corresponding second loss function.

[0163] In this way, the server uses two loss functions to adjust the model parameters in this round until the final convergence condition is met.

[0164] In one possible implementation, due to the multi-layered structure of the model to be trained, the sample cross-attention features include at least two parts of cross-attention features. Each part of the cross-attention features indicates: detailed layout information of image elements corresponding to key regions in the sample noisy image, and basic layout information of image elements corresponding to other regions in the sample noisy image besides the key regions. Furthermore, the key regions indicated by each part of the cross-attention feature are different. The detailed layout information includes more pixel color information than the basic layout information. The division of key regions at each layer can be learned by the model to be trained itself, and this application does not impose any limitations on this.

[0165] Therefore, when obtaining the cross features of the target samples, such as Figure 10 As shown, the server needs to extract the sub-target sample cross features corresponding to the second sample description information from at least two parts of the sample cross attention features; then, the obtained at least two sub-target sample cross features are used as the target sample cross features.

[0166] Thus, after extracting the sub-target sample cross-features from each part of the sample cross-attention features to obtain the corresponding target sample cross-features, the server needs to obtain the corresponding second loss function in the following way:

[0167] First, the difference between the cross features of each sub-target sample and the sample mask features is obtained in the cross features of the target samples. Then, the differences corresponding to the cross features of at least two sub-target samples are averaged to obtain the corresponding second loss function.

[0168] like Figure 10 As shown, the target sample cross features include at least two sub-target sample cross features. In order to ensure that each sub-target sample cross feature can be as close as possible to the real sample mask image, this application proposes to compare the difference between each sub-target sample cross feature and the sample mask feature corresponding to the sample mask image, and then average all the difference values ​​to finally obtain the corresponding second loss function.

[0169] In this approach, the target sample cross features obtained from multi-layer cross attention processing are differentiated from the sample mask features, and the average of the resulting values ​​is used as the final second loss function. This constrains the accuracy of each layer of cross attention processing and improves the accuracy of the final target cross feature generation, that is, the accuracy of the sample generated mask image.

[0170] To better understand how the second loss function is obtained, the following section will introduce this method by combining the above steps with a possible model structure of the model to be trained.

[0171] like Figure 11 As shown, the denoising module in the model under training uses a U-NET network structure. As the name suggests, the U-NET network structure is U-shaped, consisting of a shrinking (encoding) path and a symmetrical expanding (decoding) path. The encoder part comprises multiple convolutional and pooling layers, gradually reducing the spatial dimension of the image while increasing the number of feature channels. The decoder part gradually restores the spatial dimension of the image, increasing the spatial resolution through upsampling and convolution operations.

[0172] Therefore, in the symmetrical structure of the above multi-layer processing, each layer outputs a partial cross-attention feature, and the sub-target cross-features included in the partial cross-attention feature. Let A be the sub-target cross-feature. j Let ∈RH×W, representing the target cross feature corresponding to the j-th labeled text in the partial cross-attention features. Let M be the foreground mask in the training samples. Then, the acquisition of the second loss value can be expressed by the following formula:

[0173]

[0174] Among them, L sks This represents the second loss value, and (1-M) represents the complement of the foreground mask. Let |X| represent the target cross feature corresponding to the j-th labeled text tag in the partial cross-attention features of layer l, where u represents the total number of layers in the training model. mean This indicates that the average value is taken over X, where the average value is a pixel-level average.

[0175] Thus, by determining the second loss function as described above, we can supervise and constrain the sub-target cross features of each layer to be close to the complement (1-M) of the foreground mask, and finally obtain more accurate target cross features.

[0176] In one possible implementation, the second loss function can also be obtained in the following way:

[0177] Specifically, the server can obtain a first sub-loss function based on the difference between the sample mask features and the target sample cross features; obtain a second sub-loss function based on the difference between the sample mask image and the sample generated mask image; and then obtain the corresponding second loss function using these two sub-loss functions.

[0178] The method for obtaining the first sub-loss function is similar to that described above, and will not be repeated here. The method for obtaining the second sub-loss function will be described in detail below.

[0179] During training, the model to be trained generates not only the corresponding generated sample images but also corresponding generated sample mask images. Therefore, to further control the accuracy of the generated sample mask images, the server needs to further constrain the model to be trained based on the differences between the generated sample mask images and the original sample mask images, thereby obtaining a more accurate target generation model.

[0180] In the process of generating a sample-generated mask image using sample denoising features in the model to be trained, a U2-NET network structure can be added at this stage. This network structure is used to process the target cross features in the latent features of the VAE decoder in the target generation model, and its output is converted into a sample-generated mask image, which is then used as the mask image corresponding to the sample-generated image.

[0181] Thus, by increasing the receptive field range through nested U-NET structure regions within the U2-NET network structure, the compression ratio of text and image features is improved. Furthermore, since data is processed directly in the latent features of the decoder, the efficiency of data processing can be improved.

[0182] By training the model with triple constraints, a more accurate target generation model can be obtained.

[0183] The training process of the target generation model has been described above. After the model is trained, the server can apply the target generation model to complete the image generation method provided in this embodiment. The following will describe the image generation method in conjunction with the accompanying drawings and text.

[0184] See Figure 12 The following is a flowchart of an image generation method provided in an embodiment of this application. The specific steps of the method are as follows:

[0185] Step S1201: Obtain the original noisy image and preset text labels, which include: first descriptive information of the foreground image and second descriptive information of the background color.

[0186] The acquisition of the original noisy image can be: the server directly generates a noisy image, or it can be a noisy image provided by the user who wants to generate the image, etc. This application does not limit this.

[0187] The preset text tags can be a collection of descriptive information about the image content provided by the image generator. For example, the generator can input descriptive information about the image to be generated, such as blonde hair, red clothes, girl, blue background, etc., into the server. The server can then compile this information into corresponding text tags to guide subsequent image generation. Alternatively, in one possible implementation, besides collecting the descriptive information input by the generator, the server can also directly generate text information related to the input description as supplementary descriptions to guide image generation. For example, if the generator inputs "blonde hair, red clothes, girl," the server can directly generate words related to blonde hair (curly hair, medium-length hair, etc.) to limit the content of the generated image, and can also generate words related to red clothes (shirt, multiple pockets, etc.) to further restrict the image, thus more precisely defining the direction of subsequent image generation.

[0188] It should be noted that the text tag contains both a first description and a second description. The first description describes the image elements and content of the foreground image to be generated. Because it contains a rich variety of image elements and diverse content, the server can generate some content related to the input text description and add it to the first description to control the generation of the foreground image. The second description describes the background color; therefore, it is only necessary to find the relevant background color information in the text tag and use it as the second description.

[0189] Step S1202: Using the target generation model, perform the following operations on the text labels and the original noisy image: Based on the encoding results corresponding to the text labels and the image features corresponding to the original noisy image, perform denoising processing to obtain the corresponding denoising features.

[0190] After obtaining the original noisy image and the corresponding text labels, the server can generate the corresponding image on the original noisy image based on the description in the text labels.

[0191] Specifically, the server needs to use the target generation model to denoise the encoded results corresponding to the text labels and the image features corresponding to the original noisy images.

[0192] At this point, the server first needs to determine the layout information of the image to be generated from the original noisy image. This information can be obtained through a cross-attention mechanism that can acquire the correlation between two modalities. That is, the server obtains the semantic feature encoding result that can represent the text label and the image features corresponding to the original noisy image, and then performs cross-attention processing on the two to obtain the cross-attention features between them, determine the correlation between the text label and the original noisy image, and then determine the layout information of the image elements in the text label description in the original noisy image.

[0193] In one possible approach, when the server performs cross-attention processing based on the encoding results corresponding to the text labels and the image features corresponding to the original noisy image, the following implementation method can be adopted:

[0194] First, semantic features are extracted from the text labels to obtain the corresponding semantic features as the encoding result. Second, image features are extracted from the original noisy image to obtain the corresponding image features.

[0195] The extraction of semantic features can be performed using the methods mentioned in the training process above, which will not be elaborated upon in this application. The extraction of image features can also be performed using conventional extraction methods to obtain image features from the original noisy image.

[0196] Secondly, the semantic features are transformed to obtain the cross-attention query vector, and the image features are transformed to obtain the cross-attention key vector and the cross-attention value vector.

[0197] It should be noted that the cross-attention mechanism requires a query vector (Q), a key vector (K), and a value vector (V). The query vector represents the current focus of information; in the proposed scheme, this means focusing on the layout information of the image elements described in the input text labels. The key vector is used to match the query vector; it represents potential information points in the dataset. The value vector corresponds to the key vector and contains information matching the key, typically including detailed information or data related to the key.

[0198] In the scheme proposed in this application embodiment, the query vector, key vector, and value vector are all obtained by performing matrix transformations on semantic features and image features. Specifically, the cross-attention query vector is obtained by multiplying the semantic features by a preset query matrix, the cross-attention key vector is obtained by multiplying the image features by a preset key matrix, and the cross-attention value vector is obtained by multiplying the image features by a preset value matrix.

[0199] Finally, after obtaining the cross-attention query vector, cross-attention key vector, and cross-attention value vector, the server can perform corresponding cross-attention processing based on these three vectors to obtain the corresponding cross-attention features.

[0200] In this way, by using semantic features as query vectors, the correlation between text labels and the original noisy image is obtained, thereby determining the target direction of image generation and ensuring the accuracy of the image generation results.

[0201] Specifically, when performing cross-attention processing, the server can perform the following operations for each vector element in the cross-attention query vector:

[0202] See Figure 13 This application provides a method for obtaining sub-cross features, and the specific implementation steps of the method are as follows:

[0203] Step S1301: Obtain the association between a vector element in the cross-attention query vector and the vector elements contained in the cross-attention key vector.

[0204] After obtaining the query vector, the server can determine the association relationship for each vector element in the query vector, thereby obtaining the layout information of the image element described in the text label represented by each vector element in the original noisy image.

[0205] For example, the server can perform a dot product operation on a vector element in the cross-attention query vector and the cross-attention key vector. The result obtained after this operation can represent the degree of association between the vector element and the vector elements contained in the cross-attention key vector through the magnitude of the value.

[0206] Step S1302: Based on the above association, determine the attention weight of a vector element for each vector element in the cross attention key vector.

[0207] After obtaining the correlation between a vector element and the vector elements contained in the cross-attention key vector, the server can normalize these values ​​and use the softmax function to standardize these correlations to obtain the attention weight of each vector element in the cross-attention key vector.

[0208] Step S1303: Based on the attention weights, the corresponding vector elements in the cross attention value vector are weighted to obtain a sub-cross feature corresponding to a vector element.

[0209] Finally, the server can use attention weights to weight the vector elements in the cross-attention value vector that correspond to the cross-attention key vector, and finally obtain the sub-cross feature corresponding to a vector element in the cross-attention query vector.

[0210] It should be noted that when the server performs cross-attention processing, it can process each vector element in the cross-attention query vector simultaneously, in batches, or sequentially. This application does not impose any restrictions on this.

[0211] However, when it comes to each vector element in the cross-attention query vector, the server processes them identically. Therefore, in Figure 14 In the logic diagram shown, only one vector element will be used as an example to illustrate the above. Figure 13 The steps in the document will be explained.

[0212] like Figure 14 As shown, the server can select a vector element from the cross-attention query vector for processing, perform a dot product between it and each vector element in the cross-attention key vector to obtain the association relationship between the vector elements in the cross-attention key vector. Then, these association relationships are normalized to obtain the corresponding attention weights, and these attention weights are then used to process the vector elements in the cross-attention value vector to obtain the corresponding sub-cross features.

[0213] Then, after repeating the above operations, the server can obtain the corresponding cross-attention features based on the sub-cross features corresponding to each vector element in the cross-attention query vector.

[0214] Thus, the obtained cross-attention features include the sub-cross features corresponding to each vector element in the cross-attention query vector. In other words, the cross-attention features include the layout information of each image element described in the text label in the original noisy image.

[0215] After obtaining the cross-attention features, the server can continue to perform the following operations based on these cross-attention features: perform denoising processing on the original noisy image based on the cross-attention features to obtain the corresponding denoising features.

[0216] The server has already obtained the correlation between the text labels and the original noisy image from the cross-attention features. Therefore, the server can control the denoising direction of the denoising process based on the layout information of each image element described by the text labels in the original noisy image, as indicated by its cross-attention features. This allows the server to fit the noise data corresponding to each position in the original noisy image to the feature direction described by the text labels, thereby obtaining the corresponding denoising features.

[0217] In one possible implementation, the process of denoising the original noisy image to obtain the corresponding denoised features can be accomplished in the following way:

[0218] Having obtained the cross-attention features, in order to distinguish between the foreground pixel area and the background pixel area in the original noisy image, the server can achieve this goal based on the target cross features corresponding to the second description information.

[0219] As mentioned above, the cross-attention feature contains the association between each vector element and the cross-attention key vector in the cross-attention query vector created based on semantic features. In other words, the cross-attention feature represents the layout information of various image elements in the text label description in the original noisy image. Therefore, the server needs to extract the layout information corresponding to the background color image element corresponding to the second description information separately, and based on this, divide the original noisy image into foreground pixel areas and background pixel areas.

[0220] Specifically, the server can complete the above steps in the following ways:

[0221] like Figure 15 As shown, the server first needs to obtain the target feature element corresponding to the second descriptive information in the semantic features.

[0222] For example, in the process of extracting semantic features from text tags, the text is segmented into tokens, and multiple tokens are used to represent the basic units in the text, typically words, phrases, or symbols. Figure 16 As shown, when extracting features from text tags, the server divides the words and phrases corresponding to each descriptive information in the text tag. Since the second descriptive information only contains descriptions related to the background color, the background color will generally correspond to a separate token. Since there may be many colors, the token represented by the background color is denoted as "sks".

[0223] Therefore, when the server obtains the target feature element in the semantic features, it can use the "sks" token to directly extract the corresponding target feature element from the semantic features.

[0224] Next, based on the target feature element, the target vector element corresponding to the target feature element is obtained from the cross attention query vector, and the target cross feature corresponding to the target vector element is obtained based on the target vector element.

[0225] As can be seen from the above, the cross-attention query vector includes the feature elements of the text label. Therefore, the server can use the target feature elements to determine the target vector element corresponding to the target feature elements from the cross-attention query vector. In other words, the server needs to determine the target vector element corresponding to the text with the background color from the cross-attention query vector.

[0226] In this way, the server can filter out the layout information corresponding to the background color image feature from the layout information of various image elements included in the cross-attention features, that is, the target cross-feature.

[0227] Finally, the server divides the original noisy image into foreground pixel regions and background pixel regions based on the target cross features.

[0228] After the server obtains the layout information of the background color, it can divide the original noisy image into pixel types based on the layout information of the "background" characteristic, thereby determining the foreground pixel area and the background pixel area.

[0229] In this way, the server can directly extract the corresponding target cross features from the cross attention features by using the token tag corresponding to the second description information obtained during semantic feature extraction, and then obtain the layout information corresponding to the background color, which reduces the complexity of data search and improves the efficiency of image generation.

[0230] After dividing the foreground and background pixel areas, the server can then proceed with the following operations:

[0231] The server has already distinguished between the foreground and background pixel regions of the original noisy image. Therefore, when obtaining denoising features, the server can perform denoising processing separately in the foreground and background pixel regions to obtain the corresponding denoising features. Specifically, in the foreground pixel region, the feature is fitted along the first feature direction indicated by the first descriptive information, and in the background pixel region, it is fitted along the second feature direction indicated by the second descriptive information. In this way, by performing feature fitting in two separate regions, the server avoids treating all semantic features of the text label as a whole and representing them all in the original noisy image. This prevents the foreground and background images from being blended together in the generated image, making them difficult to distinguish.

[0232] Step S1203: Based on the decoding results of the denoising features, obtain the target generated image and the target mask image.

[0233] Specifically, once the server obtains the denoising features, it already knows the color information and image distribution of each image element in the image to be generated. Therefore, it can directly generate the corresponding target image and target mask image based on the decoding result of the denoising features.

[0234] It should be noted that when the server uses denoising features to decode and generate the target image, it can use some conventional decoding models that have already been trained to perform the corresponding processing, or it can be processed by the target generation model proposed in this application. This application does not impose any restrictions on this.

[0235] When the server generates the target mask image using denoising, it needs to process the denoising features through the U2-NET network structure added to the decoder of the VAE as mentioned above, and then convert its output into the target mask image.

[0236] This scheme proposes an image generation method. It denoises the input text tag encoding result and the image features of the original noisy image using a target generation model. After obtaining the corresponding denoised features, it uses these features to obtain the corresponding target-generated image and target mask image. The target-generated image includes the foreground image and background color described by the text tag, while the target mask image is a mask image for the target-generated image. Thus, the encoding result of the text tag guides the denoising direction of the original noisy image, ensuring that the denoised features include features representing the foreground image and background color. This allows for the direct generation of the corresponding target-generated image and target mask image. Furthermore, using the background color features from the denoised features to generate the corresponding target mask image enables accurate differentiation between the foreground and background. Therefore, by masking the target-generated image with the target mask image, the corresponding foreground or background image can be quickly obtained.

[0237] On the other hand, during the training of the target generation model, the second sample description information of the background color corresponding to the real sample image is used to perform cross-attention processing with the sample noisy image to obtain the target sample cross features representing the correlation between the background color and the generated image. This allows for the explicit acquisition of the denoising features corresponding to the background color during the denoising process. Furthermore, this target sample cross feature is used to construct a loss function with the sample mask image, and this is used to train the target generation model. This results in a target generation model that can more accurately generate denoising features that reflect the positional range of the background color in the image, and based on this, generate the corresponding target mask image. This approach proposes using the difference between the target sample cross features corresponding to the second sample description information and the sample mask image as a constraint to train the target generation model, improving the utilization rate of cross-attention features. Simultaneously, using only the target sample cross features to constrain the denoising generation improves the speed and effectiveness of generating the target mask image. In one possible implementation, when the server generates the target generated image based on the original noisy image and text labels, in addition to the three-channel information of the pixel color in the target generated image, it can also additionally generate the transparency channel information of the target generated image.

[0238] The generation of the transparency channel information can be combined with the target mask image. For example, the transparency channel information of a general image indicates that the transparency of each pixel in the image is 0, thus representing that the pixel as an opaque pixel. In the solution provided in this application embodiment, the transparency channel information can also be further enhanced with a position label corresponding to each pixel, which is used to represent whether the corresponding pixel is a foreground pixel or a background pixel.

[0239] The location label is obtained based on the mask information represented by the corresponding generated target mask image. By distinguishing the mask values ​​of foreground and background pixels in the target mask image, the server can obtain the location label corresponding to each pixel.

[0240] In this way, the target image generated by the server contains not only the original three-channel information of pixel color, but also the information of a fourth channel, namely the transparency channel.

[0241] After generating the target image in four-channel format, the server can quickly perform corresponding image matting processing on the target image based on the information of the fourth channel after receiving the matting instruction.

[0242] like Figure 17As shown, after the server receives the matting instruction to perform matting on the foreground image, it can adjust the transparency of each pixel in the background pixel area according to the differentiation label corresponding to each pixel in the transparency channel information of the target generated image. After adjusting it to transparency, the target matted image containing the foreground image can be obtained.

[0243] Thus, by generating the above four channels of information, this application provides an image with high-precision transparency information that clearly distinguishes the foreground from the background. In this way, in subsequent downstream applications, when performing image matting or other image effects processing, the transparency channel information corresponding to the target image can be generated directly for rapid processing.

[0244] The above describes the image generation method proposed in this application through various implementation methods in training and application. In order to better understand the application of the solution in this solution, the following will introduce the logical flow of the method through an example.

[0245] See Figure 18 This is a logical schematic diagram of an image generation method provided in an embodiment of this application, such as... Figure 18 As shown, the specific implementation method is as follows:

[0246] First, the image generator inputs corresponding text labels into the target generation model. The first descriptive information describes the foreground image, and the second descriptive information describes the background color. Then, after obtaining the text labels, the target generation model retrieves or generates the original noisy image. Based on the semantic features corresponding to the text labels and the image features corresponding to the original noisy image, it performs cross-attention processing to obtain the corresponding cross-attention features.

[0247] Then, based on this, the target generation model uses the correlation between the text labels represented by cross-attention features and the original noisy image to denoise the original noisy image, thereby obtaining the corresponding denoising features.

[0248] Finally, the target generation model can obtain the target generated image and the target mask image based on the decoding result of the denoising feature.

[0249] Based on the same inventive concept, embodiments of this application provide an image generation apparatus capable of performing the functions corresponding to the aforementioned image generation method. Please refer to... Figure 19 The device includes an acquisition module 1901, a processing module 1902, and a training module 1903, wherein:

[0250] The acquisition module 1901 is used to acquire the original noisy image and preset text labels; the text labels include: first descriptive information of the foreground image and second descriptive information of the background color;

[0251] The generation module 1902 is configured to perform the following operations on the text label and the original noisy image using a target generation model: denoising based on the encoding result corresponding to the text label and the image features corresponding to the original noisy image to obtain corresponding denoising features; and obtaining a target generated image and a target mask image based on the decoding result of the denoising features; wherein the target generated image includes: a foreground image described by the first description information and a background color described by the second description information, and the target mask image is a mask image for the target generated image;

[0252] Training module 1903 is used to train a target generation model based on a loss function constructed from the sample mask image and the cross features of the target samples; the sample mask image is a mask image for the real sample image; the sample text label includes: first sample description information of the foreground image and second sample description information of the background color in the real sample image; the target sample cross features are obtained by cross-attention processing based on the second sample description information and the sample noise image.

[0253] Based on the same inventive concept, embodiments of this application also provide an electronic device. In one possible implementation, the electronic device may be a server, such as... Figure 1 The server 102 is shown. In this embodiment, the electronic device 2000 has the following structure. Figure 20 As shown, it may include at least a memory 2001, a communication module 2003, and at least one processor 2002.

[0254] The memory 2001 is used to store computer programs executed by the processor 2002. The memory 2001 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0255] Memory 2001 may be volatile memory, such as random-access memory (RAM); memory 2001 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 2001 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2001 may be a combination of the above-described memories.

[0256] Processor 2002 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 2002 is used to implement the above-described image generation method when it calls a computer program stored in memory 2001.

[0257] The communication module 2003 is used to communicate with terminal devices and other servers.

[0258] This application embodiment does not limit the specific connection medium between the memory 2001, communication module 2003, and processor 2002. This application embodiment... Figure 20 The memory 2001 and the processor 2002 are connected via a bus 2004, which is in... Figure 20 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered as limiting information. The Bus 2004 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 20 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0259] The memory 2001 stores a computer storage medium containing computer-executable instructions for implementing the image generation method of this application embodiment. The processor 2002 is used to execute the image generation method described above.

[0260] In another embodiment, the electronic device can also be other electronic devices, such as... Figure 1 The terminal device 101 shown. In this embodiment, the structure of the electronic device can be as follows: Figure 21As shown, it includes components such as: communication component 2110, memory 2120, display unit 2130, camera 2140, sensor 2150, audio circuit 2160, Bluetooth module 2170, processor 2180, etc.

[0261] The communication component 2110 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology, and the electronic device can send and receive information through the WiFi module.

[0262] The memory 2120 can be used to store software programs and data. The processor 2180 executes various functions of the terminal device 101 and performs data processing by running the software programs or data stored in the memory 2120. The memory 2120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 2120 stores an operating system that enables the terminal device 101 to run. In this application, the memory 2120 can store the operating system and various application programs, and may also store a computer program that executes the image generation method of the embodiments of this application.

[0263] The display unit 2130 can also be used to display information input by the object or information provided to the object, as well as a graphical user interface (GUI) for various menus of the terminal device 101. Specifically, the display unit 2130 may include a display screen 2132 disposed on the front of the terminal device 101. The display screen 2132 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 2130 can be used to display the defect detection interface, model training interface, etc., as described in the embodiments of this application.

[0264] The display unit 2130 can also be used to receive input digital or character information and generate signal inputs related to object settings and function control of the terminal device 101. Specifically, the display unit 2130 may include a touch screen 2131 disposed on the front of the terminal device 101, which can collect touch operations on or near the object, such as clicking a button, dragging a scroll box, etc.

[0265] The touchscreen 2131 can be placed on top of the display screen 2132, or the touchscreen 2131 and the display screen 2132 can be integrated to realize the input and output functions of the physical terminal device 101. After integration, it can be referred to as a touch display screen. In this application, the display unit 2130 can display the application and the corresponding operation steps.

[0266] Camera 2140 can be used to capture still images, and objects can publish images captured by camera 2140 through an application. There can be one or multiple cameras 2140. An object generates an optical image through a lens, which is projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 2180 to be converted into a digital image signal.

[0267] The physical terminal device may also include at least one sensor 2150, such as an accelerometer 2151, a proximity sensor 2152, a fingerprint sensor 2153, and a temperature sensor 2154. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0268] Audio circuitry 2160, speaker 2161, and microphone 2162 provide an audio interface between the physical terminal device 101 and the terminal device 101. Audio circuitry 2160 converts received audio data into electrical signals, transmits them to speaker 2161, and then converts them into sound signals for output. Physical terminal device 101 may also be equipped with volume buttons for adjusting the volume of the sound signal. On the other hand, microphone 2162 converts collected sound signals into electrical signals, which are then received by audio circuitry 2160, converted into audio data, and output to communication component 2110 for transmission to, for example, another physical terminal device 101, or to memory 2120 for further processing.

[0269] Bluetooth module 2170 is used to interact with other Bluetooth devices that also have Bluetooth modules via the Bluetooth protocol. For example, a physical terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through Bluetooth module 2170, thereby exchanging data.

[0270] The processor 2180 is the control center of the physical terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 2120 and calling data stored in the memory 2120. In some embodiments, the processor 2180 may include one or more processing units; the processor 2180 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 2180. In this application, the processor 2180 can run the operating system, applications, user interface display and touch response, and the image generation method of this embodiment. Furthermore, the processor 2180 is coupled to the display unit 2130.

[0271] In some possible implementations, various aspects of the image generation method provided in this application can also be implemented in the form of a program product, which includes a computer program that, when the program product is run on an electronic device, causes the electronic device to perform the steps in the image generation method according to the various exemplary embodiments of this application described above.

[0272] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0273] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0274] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0275] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0276] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0277] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0278] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0279] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0280] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0281] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0282] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0283] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of picture generation, characterized by, The method comprises: obtaining an original noise picture and a preset text label; the text label comprises: first description information of a foreground image, and second description information of a background color; by using a target generation model, the text label and the original noise picture are processed as follows: based on the encoding result corresponding to the text label and the image feature corresponding to the original noise picture, denoising processing is performed to obtain corresponding denoising features; based on the decoding result of the denoising features, a target generation picture and a target mask picture are obtained; wherein the target generation picture comprises a foreground image described by the first description information and a background color described by the second description information, and the target mask picture is a mask picture for the target generation picture; wherein the target generation model is obtained by training a loss function based on a sample mask picture and a target sample cross feature; the sample mask picture is a mask picture for a sample real picture; the sample text label comprises: first sample description information of a foreground image in the sample real picture, and second sample description information of a background color; the target sample cross feature is obtained by cross-attention processing based on the second sample description information and the sample noise picture.

2. The method of claim 1, wherein, the denoising processing based on the encoding result corresponding to the text label and the image feature corresponding to the original noise picture to obtain the corresponding denoising features comprises: based on the encoding result corresponding to the text label and the image feature of the original noise picture, cross-attention processing is performed to obtain corresponding cross-attention features; the cross-attention features are used to indicate the correlation between the text label and the original noise picture; based on the cross-attention features, the original noise picture is denoised to obtain corresponding denoising features.

3. The method of claim 2, wherein, the cross-attention processing based on the encoding result corresponding to the text label and the image feature of the original noise picture to obtain the corresponding cross-attention features comprises: performing semantic feature extraction on the text label to obtain corresponding semantic features as the encoding result, and performing image feature extraction on the original noise picture to obtain corresponding image features; transforming the semantic features to obtain a cross-attention query vector, and transforming the image features to obtain a cross-attention key vector and a cross-attention value vector; based on the cross-attention query vector, the cross-attention key vector and the cross-attention value vector, cross-attention processing is performed to obtain corresponding cross-attention features.

4. The method of claim 3, wherein, the cross-attention processing based on the cross-attention query vector, the cross-attention key vector and the cross-attention value vector to obtain the corresponding cross-attention features comprises: for each vector element in the cross-attention query vector, the following operations are performed respectively: obtain the association relationship between one vector element in the cross-attention query vector and each vector element contained in the cross-attention key vector; based on the association relationship, determine the attention weight of the one vector element for each vector element in the cross-attention key vector; weighting processing is performed on corresponding vector elements in the cross-attention value vector based on the attention weights, to obtain sub-cross features corresponding to the one vector element; Based on the sub-cross features corresponding to each vector element in the cross-attention query vector, the corresponding cross-attention feature is obtained.

5. The method according to any one of claims 1 to 4, wherein The target generation model is obtained by training in the following manner: Based on the training samples in the training sample data set, the training model is trained by performing a loop iteration; wherein each training sample includes: a sample foreground picture containing only foreground images, and an initial mask picture of the sample foreground picture; wherein the following operations are performed in one loop iteration training process: Select a training sample, based on the initial mask picture in the training sample, fuse the target color into the background of the sample foreground picture to obtain the corresponding sample real picture, and perform noise processing on the sample real picture to obtain the corresponding sample noise picture; Obtain the sample text label of the sample real picture, and based on the sample text label, perform denoising processing on the sample noise picture to obtain the corresponding sample generation picture and sample generation mask picture; Based on the sample generation picture and the sample generation mask picture, the training model is trained in this round of iteration combined with the one training sample.

6. The method of claim 5, wherein, The sample text label of the sample real picture is obtained, and based on the sample text label, the sample noise picture is denoised to obtain the corresponding sample generation picture and sample generation mask picture, including: Based on the encoding result corresponding to the sample text label and the image features of the sample noise picture, cross-attention processing is performed to obtain the corresponding sample cross-attention feature; Based on the sample cross-attention feature, the sample noise picture is denoised to obtain the corresponding sample denoising feature; Based on the decoding result of the sample denoising feature, the sample generation picture and the sample generation mask picture are obtained.

7. The method of claim 6, wherein, Based on the sample generation picture and the sample generation mask picture, the training model is trained in this round of iteration combined with the one training sample, including: Based on the difference between the sample generation picture and the sample real picture, a corresponding first loss function is obtained; The target sample cross feature is extracted from the sample cross attention feature; The sample mask feature with the same dimension as the target sample cross feature is obtained by performing feature extraction on the sample mask picture, and a corresponding second loss function is obtained based on the difference between the sample mask feature and the target sample cross feature; Based on the first loss function and the second loss function, the parameters of the training model are adjusted.

8. The method of claim 7, wherein, Based on the difference between the sample mask feature and the target sample cross feature, a first sub-loss function is obtained; Based on the difference between the sample mask picture and the sample generation mask picture, a second sub-loss function is obtained; ​ The second loss function is obtained based on the first sub-loss function and the second sub-loss function.

9. The method of claim 5, wherein, The target color is green or blue.

10. An apparatus for picture generation, characterized by The method comprises the following steps: The obtaining module is configured to obtain an original noise picture and a preset text label. The text label comprises first description information of a foreground image and second description information of a background color. The generating module is configured to perform the following operations on the text label and the original noise picture by using a target generation model: performing denoising processing based on an encoding result corresponding to the text label and an image feature corresponding to the original noise picture to obtain a corresponding denoising feature; and obtaining a target generation picture and a target mask picture based on a decoding result of the denoising feature; wherein the target generation picture comprises a foreground image described by the first description information and a background color described by the second description information, and the target mask picture is a mask picture of the target generation picture. The training module is configured to train the target generation model based on a loss function constructed based on a sample mask picture and a target sample cross feature; the sample mask picture is a mask picture of a sample real picture; a sample text label comprises first sample description information of a foreground image in the sample real picture and second sample description information of a background color; and the target sample cross feature is obtained by performing cross attention processing based on the second sample description information and the sample noise picture.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-9.

12. A computer device, comprising: The method comprises the following steps: The memory is configured to store program instructions. The processor is configured to call the program instructions stored in the memory and execute the method of any one of claims 1-9 according to the obtained program instructions.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to execute the method of any one of claims 1-9.

Images