Image generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to an image generation method, device, electronic equipment and storage medium, the method comprising: obtaining text prompt information and at least one reference image; the reference image comprises a first subject object; the text prompt information is used to describe the content of a target image; using the feature extractor, extracting the features of the first subject object from the reference image; using the noise reduction network, based on the text prompt information and the features of the first subject object, performing noise reduction processing on a preset noise image to obtain a target image; the target image comprises a first derived object and a first sub-object; the first sub-object corresponds to the first subject object, and the image features of the first sub-object and the first subject object having a corresponding relationship are consistent. It gives a method capable of fusing objects in the reference image into the generated image, meeting the user's customized image generation needs.

Description

Technical Field

[0001] This disclosure relates to the field of image generation technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image generation, an important research area in computer vision and artificial intelligence, aims to generate images with specific visual features or semantic information using computer algorithms. With the rapid development of deep learning technology, image generation techniques have made significant progress. However, effectively integrating objects from reference images into the generated images remains a pressing problem to be solved. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image generation method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, this disclosure provides an image generation method, wherein the image generation model for implementing the image generation method includes a feature extractor and a noise reduction network; the method includes:

[0005] Acquire text prompt information and a reference image; the reference image includes a first main object; the text prompt information is used to describe the content of the target image;

[0006] Using the feature extractor, features of the first subject object are extracted from the reference image;

[0007] Using the denoising network, based on the text prompt information and the features of the first main object, the preset noisy image is denoised to obtain the target image; the target image includes a first derived object, and the first derived object includes a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0008] Secondly, this disclosure also provides an image generation apparatus, wherein the image generation model for implementing the image generation method includes a feature extractor and a noise reduction network, the apparatus comprising:

[0009] The acquisition module is used to acquire text prompt information and a reference image; the reference image includes a first main object; the text prompt information is used to describe the content of the target image.

[0010] The extraction module is used to extract features of the first subject object from the reference image using the feature extractor;

[0011] The noise reduction module is used to perform noise reduction processing on a preset noisy image using the noise reduction network, based on the text prompt information and the features of the first main object, to obtain a target image; the target image includes a first derived object, and the first derived object includes a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0012] Thirdly, this disclosure also provides an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described above.

[0016] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the image generation method described above.

[0017] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0018] The technical solution provided in this disclosure involves setting and acquiring text prompt information and at least one reference image. The reference image includes a first main object. The text prompt information describes the content of the target image. A feature extractor is used to extract features of the first main object from the reference image. A denoising network is used to denoise a preset noisy image based on the text prompt information and the features of the first main object to obtain the target image. The target image includes a first derived object and a first sub-object. The first sub-object corresponds to the first main object, and the image features of the corresponding first sub-object and the first main object are consistent. This provides a method for fusing objects from a reference image into a generated image, meeting users' customized image generation needs. The technical method provided in this application is particularly suitable for scenarios where it is desirable to generate portraits of people wearing specific clothing. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of an image generation method provided in this disclosure embodiment;

[0022] Figure 2 This is a schematic diagram illustrating the input-output correspondence of an image generation model provided in this disclosure;

[0023] Figure 3 This is a schematic diagram of the internal structure of an image generation model provided in an embodiment of the present disclosure;

[0024] Figure 4 for Figure 3 A schematic diagram of the structure of the GFE module in the middle;

[0025] Figure 5 for Figure 3 A schematic diagram of the structure of the DA module;

[0026] Figure 6 A flowchart illustrating an image generation model training method provided in this disclosure embodiment;

[0027] Figure 7 This is a schematic diagram of the structure of an image generation device according to an embodiment of the present disclosure;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] Figure 1This is a flowchart illustrating an image generation method provided in this embodiment. This embodiment is applicable to image generation performed on a client side. The method can be executed by an image generation device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a terminal, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, and smart home devices. Alternatively, this embodiment is applicable to image generation performed on a server side. The method can be executed by an image generation device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server. The image generation model used to implement the image generation method provided in this application includes a feature extractor and a noise reduction network.

[0032] like Figure 1 As shown, the image generation method may specifically include:

[0033] S1. Obtain text prompt information and at least one reference image; the reference image includes a first subject object; the text prompt information is used to describe the content of the target image.

[0034] A reference image can be provided to an image generation model during the image generation process to guide the model in image generation. The reference image includes a first subject object, which is the object in the reference image that needs to be emphasized or highlighted. This application does not limit the specific things that the first subject object refers to. In some scenarios, the first subject object can be clothing or accessories such as shirts, T-shirts, dresses, pants, skirts, hats, and scarves.

[0035] Text prompts can be, for example, text information provided to an image generation model to guide it in generating images. The content of this text prompt is a textual description of the content of the desired generated image (also referred to as the target image in this application). For example, the text prompts may include descriptive information about a scene in the target image. Alternatively, if it is desired that the target image include a person, the text prompts may include descriptive information about the person's pose. It should be noted that the text prompts may or may not include descriptive information about a first subject object in the reference image.

[0036] It should be noted that in this application, the first subject object will ultimately be incorporated into the target image. For example, if it is desired that the target image include a girl wearing a T-shirt of a specific style and pattern, an image including a T-shirt of a specific style and pattern will be used as a reference image, and the T-shirt of that specific style and pattern will be the first subject object of the reference image.

[0037] S2. Using a feature extractor, extract the features of the first subject object from the reference image.

[0038] S3. Using a noise reduction network, based on the text prompt information and the features of the first main object, the preset noisy image is denoised to obtain the target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0039] The first derived object is the main subject in the target image; it is the object that needs to be focused on or highlighted. In some scenarios, the first derived object is a newly generated object. The reference image does not include objects corresponding to the first derived object.

[0040] The first sub-object corresponds to the first main object. For example, the first sub-object can be a new object generated based on its corresponding first main object. In some scenarios, the first sub-object can be a component of the first derived object.

[0041] The first sub-object and the first main object with a corresponding relationship have the same image features. For example, the first sub-object and its corresponding first main object are highly similar or completely identical, so that they are regarded as the same thing.

[0042] For example, the reference image includes a T-shirt of a specific style and pattern, which is the first main object; the generated target image includes a girl wearing the T-shirt of the specific style and pattern, the girl is the first derived object, and the T-shirt of the specific style and pattern worn by the girl is the first child object. In this example, the first child object constitutes a part of the first derived object.

[0043] For example, see Figure 2 The provided text prompt is "A woman is wearing a tank top and shorts, standing in a white background." Two reference images are provided: Reference Image 1 shows a white and green striped tank top, and Reference Image 2 shows yellow and white checkered shorts. The text prompt and the two reference images are input into an image generation model to generate a target image. In the target image, the girl is wearing the tank top from Reference Image 1 and the shorts from Reference Image 2.

[0044] The above technical solution involves setting up text prompts and at least one reference image; the reference image includes a first main object; the text prompts describe the content of the target image; a feature extractor extracts features of the first main object from the reference image; and a denoising network is used to denoise a preset noisy image based on the text prompts and the features of the first main object to obtain the target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the corresponding first sub-object and the first main object are consistent. It provides a method to fuse objects in a reference image into the generated image, meeting users' customized image generation needs. The technical method provided in this application is particularly suitable for scenarios where it is desirable to generate portraits of people wearing specific clothing.

[0045] Based on the above technical solution, the method may optionally further include: acquiring an original image, the original image including a first original object; and using a feature extractor to extract features of the first original object from the original image. S130 may include: using a denoising network to perform denoising processing on a preset noisy image based on text prompt information, features of the first main object, and features of the first original object to obtain a target image; the target image includes a first derived object and a first sub-object; the first derived object corresponds to the first original object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0046] The original image, for example, can be provided to the image generation model during the image generation process to guide the model in image generation. The first original object is the object in the original image that needs to be focused on or highlighted. The first original object can be, for example, an object that can be decorated by a first subject object. For example, the first original object can be a person or an animal.

[0047] The first derived object corresponds to the first original object, meaning that the first derived object is generated based on the features of the first original object.

[0048] For example, the original image includes a girl wearing a long skirt. The first subject in reference image 1 is a white and green striped tank top, and the first subject in reference image 2 is yellow and white checkered shorts. In the target image generated based on the original image and reference images 1 and 2, the effect is that the girl in the original image is wearing the tank top in reference image 1 and the shorts in reference image 2, but not a long skirt.

[0049] By importing the original image, it is possible to change the clothing and / or accessories of a specific first original object.

[0050] In some embodiments, optionally, there are multiple reference images; the first subject objects in different reference images are different; S2 can be replaced by: using a feature extractor to extract the features of each first subject object from the reference images respectively; S3 can be replaced by: stitching the features of each first subject object together to obtain stitched image features; using a denoising network, based on the text prompt information and the stitched image features, to perform denoising processing on the preset noisy image to obtain the target image.

[0051] In some scenarios, the primary subject object in different reference images can be items of the same category or items of different categories. For example, in one example, two reference images are obtained: one where the primary subject object is pants, and the other where the primary subject object is a shirt. In another example, two reference images are obtained: one where the primary subject object is pants, and the other where the primary subject object is a headband.

[0052] "Splicing together the features of each primary subject object" means simply combining or arranging the features of each primary subject object in a certain order. The purpose of "splicing together" is to integrate the features of multiple primary subject objects without performing complex calculations between the features of different primary subject objects.

[0053] By setting the features of the first subject object in multiple reference images to be stitched together, the stitched image features are obtained. Using a noise reduction network, based on the text prompt information and the stitched image features, the preset noisy image is denoised to obtain the target image. This can further meet the user's need for customized image generation that combines different items.

[0054] Based on the above technical solution, optionally, the feature extractor includes T extraction network units, which are connected in series; the denoising network includes T denoising network units; the T denoising network units are connected in series; the extraction network units and the denoising network units correspond one-to-one; S2 may include: in the i-th round of feature extraction, for any reference image, the features of the first subject object to be processed in the i-th round corresponding to the reference image are input into the i-th extraction network unit to obtain the features of the first subject object processed in the i-th round corresponding to the reference image; S3 may include: in the i-th round of denoising, using the i-th denoising network unit, based on the features of the first subject object processed in the i-th round corresponding to the reference image and the text prompt information, denoising processing is performed on the image to be denoised corresponding to the i-th round to obtain the image after denoising in the i-th round. Wherein, T is a positive integer, i is a positive integer greater than or equal to 1 and less than or equal to T; the features of the first subject object to be processed in the first round corresponding to the reference image are image features obtained by encoding the reference image; the image to be denoised corresponding to the first round is a preset noise image. When i is greater than 1 and less than T, the features of the first subject object to be processed in the (i+1)th round corresponding to the reference image are obtained based on the features of the first subject object already processed in the i-th round corresponding to the reference image; the image to be denoised in the (i+1)th round is obtained based on the image after denoising in the i-th round.

[0055] Furthermore, there are various specific implementation methods for "in the i-th round of feature extraction, inputting the features of the i-th round of the first subject object to be processed corresponding to the reference image into the i-th extraction network unit to obtain the features of the i-th round of the processed first subject object corresponding to the reference image," and this application does not limit this. For example, when i is greater than or equal to 1 and less than T, "in the i-th round of feature extraction, inputting the features of the i-th round of the first subject object to be processed corresponding to the reference image into the i-th extraction network unit to obtain the features of the i-th round of the processed first subject object corresponding to the reference image" may include: in the i-th round of feature extraction, performing self-attention processing on the features of the i-th round of the first subject object to be processed corresponding to the reference image to obtain the features of the i-th round of the processed first subject object corresponding to the reference image; inputting the features of the i-th round of the processed first subject object corresponding to the reference image into the i-th noise reduction network unit; and based on the description information of the first subject object, performing cross-attention processing on the features of the i-th round of the processed first subject object corresponding to the reference image to obtain the image features to be processed of the (i+1)-th round of the first subject object corresponding to the reference image.

[0056] There are many specific implementation methods for “in the i-th round of denoising, the i-th denoising network unit is used to perform denoising processing on the image to be denoised corresponding to the i-th round based on the features of the first subject object that has been processed in the i-th round and the text prompt information corresponding to the reference image, so as to obtain the image after the i-th round of denoising”, and this application does not limit it. For example, when i is greater than or equal to 1 and less than T, the number of reference images is multiple; "In the i-th round of denoising, using the i-th denoising network unit, based on the features of the first subject object processed in the i-th round corresponding to the reference image and the text prompt information, denoising processing is performed on the image to be denoised corresponding to the i-th round to obtain the image after denoising in the i-th round" may include: in the i-th round of denoising, the features of each first subject object processed in the i-th round are concatenated to obtain the concatenated features of the first subject object processed in the i-th round; based on the image to be denoised corresponding to the i-th round, cross-attention processing is performed on the concatenated features of the first subject object processed in the i-th round to obtain the cross-attention features of the first subject object in the i-th round; self-attention processing is performed on the image to be denoised corresponding to the i-th round to obtain the self-attention features of the image to be denoised in the i-th round; the self-attention features of the image to be denoised in the i-th round and the cross-attention features of the first subject object in the i-th round are fused to obtain the fused image features in the i-th round; based on the text prompt information, cross-attention processing is performed on the fused image features in the i-th round to obtain the image after denoising in the i-th round.

[0057] Figure 3 This is a schematic diagram of the internal structure of an image generation model provided in an embodiment of this disclosure. See also... Figure 3 The image generation model includes a feature extractor and a denoising network. The feature extractor consists of multiple extraction network units arranged in series (also referred to as "cascaded" in some scenarios). The denoising network consists of multiple denoising network units arranged in series (also referred to as "cascaded" in some scenarios). The number of extraction network units is the same as the number of denoising network units, and there is a one-to-one correspondence between the extraction network units and the denoising network units. That is, the i-th extraction network unit corresponds to the i-th denoising network unit.

[0058] Figure 4 for Figure 3 A schematic diagram of the GFE module. Figure 5 for Figure 3 A schematic diagram of the DA module. See also... Figure 3Each extraction network unit includes a first subunit GFE and a second subunit CA1. In the i-th round of feature extraction, the i-th extraction network unit is used for feature extraction. The i-th extraction network unit needs to traverse all reference images during feature extraction. Each denoising network unit includes a third subunit DA and a fourth subunit CA2. In the i-th round of denoising, the i-th denoising network unit is used for denoising. During denoising, each denoising network unit uses the features of the first main object extracted from each reference image.

[0059] Specifically, see Figure 3 and Figure 4 For reference image 1, before being transmitted to the first extraction network unit, reference image 1 is encoded to obtain encoded image features. These encoded image features are used as the features of the first subject object to be processed in the first round, corresponding to reference image 1. The features of the first subject object to be processed in the first round, corresponding to reference image 1, are input into the first extraction network unit. In the first subunit GFE of the first extraction network unit, the features of the first subject object to be processed in the first round, corresponding to reference image 1, are first mapped; then, self-attention processing is performed on the mapped features of the first subject object to be processed in the first round, corresponding to reference image 1, to obtain the features of the first subject object that has been processed in the first round, corresponding to reference image 1. The features of the first subject object that has been processed in the first round, corresponding to reference image 1, are input into the third subunit DA of the first denoising network unit. In the second subunit CA1 of the first extraction network unit, based on the description information of the first subject object, cross-attention processing is performed on the features of the first subject object that has been processed in the first round, corresponding to reference image 1, to obtain the features of the first subject object to be processed in the second round, corresponding to reference image 1. For reference image 2, the same processing method is applied until the features of the first subject object to be processed in the second round, corresponding to each reference image, are obtained. Subsequent feature extraction processes are similar to the first round. These will not be elaborated further here.

[0060] The preset noisy image is used as the image to be denoised corresponding to the first round. During the first round of denoising, in the third subunit DA of the first denoising network unit, see... Figure 5The first noise reduction network unit performs a series of steps. First, it concatenates the features of the first main subject object processed in the first round with each reference image, resulting in the concatenated features of the first main subject object. Second, it performs self-attention processing on the noise-to-be-reduced image corresponding to the first round, obtaining the self-attention features of the noise-to-be-reduced image. Third, it fuses the self-attention features of the noise-to-be-reduced image and the cross-attention features of the first main subject object, resulting in the fused image features. In the fourth subunit CA2 of the first noise reduction network unit, cross-attention processing is performed on the fused image features based on text prompts, resulting in the denoised image. Subsequent noise reduction processes are similar to the first round, and will not be elaborated further here.

[0061] It should be noted here that "splicing together the features of each first subject object processed in the i-th round" means simply combining or arranging the features of the first subject objects processed in the i-th round corresponding to each reference image in a certain order. The purpose of "splicing together" is to integrate the features of multiple first subject objects from different reference images, without performing complex operations between the features of the first subject objects from different reference images.

[0062] "The self-attention features of the image to be noise-reduced in the i-th round and the cross-attention features of the first subject object in the i-th round are fused to obtain the fused image features of the i-th round." For example, this could be achieved by fusing the self-attention features of the image to be noise-reduced in the i-th round and the cross-attention features of the reference image in the i-th round using a weighted summation method. The purpose of fusion is to use the features of the first subject object to guide and correct the noisy image, making the first sub-object in the denoised image closer to the first subject object in terms of content and detail. In other words, "fusion" implies complex calculations between the self-attention features of the image to be noise-reduced in the i-th round and the cross-attention features of the reference image in the i-th round.

[0063] It should also be noted that in the step of "based on the description information of the first subject object, performing cross-attention processing on the features of the first subject object processed in the i-th round corresponding to the reference image to obtain the image features to be processed in the (i+1)-th round corresponding to the reference image," the description information of the first subject object can be pre-embedded in the image generation model. In this case, it is equivalent to limiting the image generation module to incorporate a specific type of first subject object into the target image. When the image generation model is used subsequently, a reference image including that specific type of first subject object needs to be used. In some other embodiments, the description information of the first subject object in the reference image can be obtained based on the reference image, and then the description information of the first subject object can be input into the feature extractor. In this case, it does not limit which type of first subject object the image generation module can incorporate into the target image. The description information of the first subject object may include, for example, one or more of the first subject object name and its category. For example, the description information of the first subject object includes the category to which the first subject object belongs, such as clothes, tops, pants, or accessories.

[0064] Pre-embedding the descriptive information of the primary subject into the image generation model can simplify the training process of the image generation model and make it easier to converge.

[0065] Based on the above technical solutions, optionally, see [reference needed]. Figure 6 The training method for the image generation model used in the above embodiments includes:

[0066] S110. Obtain sample data pairs, which include sample generated images, sample text prompts, at least one sample reference image, and sample noisy images. The sample reference image includes a first main object; the sample generated image includes a second main object and a second sub-object; the second sub-object corresponds to the first main object, and the image features of the second sub-object and its corresponding first main object are consistent; the sample text prompts are used to describe the content of the sample generated image; the sample noisy image is an image after adding noise to the sample generated image.

[0067] The sample reference image can be, for example, an image used to simulate the image generation model during its usage phase. It is provided to the image generation model to guide its image generation process. The sample reference image includes a primary subject object, which is the object in the sample reference image that needs to be emphasized or highlighted. This application does not limit the specific things that the primary subject object refers to. In some scenarios, the primary subject object can be, for example, clothing or accessories such as shirts, T-shirts, dresses, pants, skirts, hats, and scarves. When there are multiple sample reference images, the primary subject object differs in each image.

[0068] Sample text prompts can be, for example, text information provided to the image generation model during the simulation of image generation model usage, used to guide the image generation model in image generation, and its content is a textual description of the content of the sample generated image.

[0069] The generated sample image can be, for example, an image that the image generation model hopes to generate based on sample text prompts and sample reference images, and is the target of image generation. The second subject object of the sample is the object that needs to be focused on or highlighted in the generated sample image. This application does not limit the specific things that the second subject object of the sample refers to. In some scenarios, the second subject object of the sample can be, for example, a person or an animal. The second sub-object of the sample corresponds to the first subject object of the sample, and the image features of the second sub-object of the sample and its corresponding first subject object are consistent. Here, the image features of the second sub-object of the sample and its corresponding first subject object of the sample are consistent, for example, it can mean that the second sub-object of the sample and its corresponding first subject object of the sample have one or more image features such as texture, shape, and color, so that it can be concluded without doubt that the two are the same thing. For example, there are two sample reference images, namely sample reference image 1 and sample reference image 2. Among them, the first subject object of the sample in sample reference image 1 is a shirt, and the first subject object of the sample in sample reference image 2 is pants. The second subject object of the sample in the generated sample image is a person wearing a shirt and pants. The sample second main object can be considered to include two sample second sub-objects, namely sample second sub-object 1 and sample second sub-object 2. Sample second sub-object 1 is the top worn by the person, and sample second sub-object 2 is the pants worn by the person. Sample second sub-object 1 in the sample generated image and the sample first main object in sample reference image 1 are the same top. Sample second sub-object 2 in the sample generated image and the sample first main object in sample reference image 2 are the same pants.

[0070] A noisy sample image can be, for example, an image obtained by superimposing random noise onto a generated sample image. In some scenarios, the process of superimposing random noise onto a generated sample image to obtain a noisy sample image is also called the forward diffusion process.

[0071] S120. Using a feature extractor, extract the features of the first main object of the sample from the sample reference image.

[0072] S130. Using a denoising network, based on the sample text prompt information and the features of the first main object of the sample, the noisy sample image is denoised to obtain a predicted image; the predicted image includes a second derived object and a third sub-object; the third sub-object corresponds to the first main object of the sample.

[0073] The second derived object is the main object in the predicted image. The third sub-object can be a component of the second derived object; the third sub-object corresponds to the sample first main object, for example, it can mean that the third sub-object is a new object generated based on its corresponding sample first main object. During the training phase, the image features of the corresponding third sub-object and the sample first main object may not be completely consistent. In other words, the corresponding third sub-object and the sample first main object may have low similarity, so in some scenarios, they may not be regarded as the same object.

[0074] S140. Based on the difference between the third sub-object with a corresponding relationship and the first main object of the sample, the parameters in the image generation model are adjusted.

[0075] There are various ways to implement this step, and this application does not limit it. For example, the implementation method of this step may include: determining a loss function based on the texture difference and / or high-frequency component difference of edge information between the corresponding third sub-object and the first subject object of the sample; and adjusting the parameters in the image generation model based on the loss function.

[0076] For example, the texture difference between the corresponding third sub-object and the sample first main object can be represented by the Depth Image Structure and Texture Similarity (DISTS) index, which is then embedded into the loss function.

[0077] The high-frequency component differences in edge information between the corresponding third sub-object and the first main object of the sample can reflect the differences in edge complexity, light and shadow changes, and texture features at the edges.

[0078] Based on the above technical solutions, optionally, in the process of extracting features of the first subject object of the sample from the sample reference image using a feature extractor, a total of T rounds of image feature extraction are performed; in the process of denoising the noisy sample image using a denoising network based on the sample text prompt information and the features of the first subject object of the sample to obtain the predicted image, a total of T rounds of denoising are performed. The image generation model training method may further include: determining a target mask based on the sample generated image; the target mask is used to indicate the position of the second sub-object of the sample in the sample generated image; in the i-th round of denoising, based on the noise-to-be-denoised image corresponding to the i-th round, obtaining a query vector of the noise-to-be-denoised image corresponding to the i-th round; based on the features of the first subject object processed in the i-th round, obtaining a key vector of the features of the first subject object processed in the i-th round; the features of the first subject object processed in the i-th round are the features of the first subject object extracted in the i-th round of image feature extraction; based on the query vector and the key vector, obtaining an attention map; and adjusting the parameters in the image generation model based on the difference between the attention map and the target mask; where T is a positive integer, and i is a positive integer greater than or equal to 1 and less than or equal to T. The requirement emphasizes that, here, the attention map and the target mask actually correspond to the same item.

[0079] The cross-attention map is calculated through the interaction of the query vector Q and the key vector K. Typically, the similarity between the query vector Q and the key vector K is calculated (e.g., through a dot product operation), resulting in a similarity matrix. Each element of this similarity matrix represents the degree of relevance between a certain part of the query vector Q and a certain part of the key vector K. The attention map can reflect which region of the noisy image is most significantly influenced by the first main object in the sample reference image; in other words, the attention map can, to some extent, reflect the position of the subsequently generated third sub-object in the predicted image. The difference between the attention map and the target mask reflects the estimated positional difference between the corresponding third sub-object and the sample second main object.

[0080] For example, if there are two sample reference images, the first subject in sample reference image 1 is a T-shirt; the first subject in sample reference image 2 is shorts; the generated sample image includes person 1, wearing a T-shirt and shorts, and the predicted image includes person 2, also wearing a T-shirt and shorts. The attention map corresponding to the T-shirt reflects the position of the T-shirt in the predicted image, and the target mask corresponding to the T-shirt reflects the position of the T-shirt in the generated sample image. Based on the difference between the attention map and the target mask corresponding to the T-shirt, the parameters in the image generation model are adjusted. Similarly, the attention map corresponding to the shorts reflects the position of the shorts in the predicted image, and the target mask corresponding to the shorts reflects the position of the shorts in the generated sample image. Based on the difference between the attention map and the target mask corresponding to the shorts, the parameters in the image generation model are adjusted.

[0081] "Adjusting parameters in the image generation model based on the difference between the attention map and the target mask" can be achieved, for example, by determining the loss function based on the difference between the attention map and the target mask, and then adjusting the parameters in the image generation model based on the loss function. This approach can improve the image generation performance of the image generation model.

[0082] Furthermore, in practice, the loss function can be determined based on the difference between the predicted image and the generated image from the sample; and the parameters in the image generation model can be adjusted based on the loss function.

[0083] The image generation model training method provided by the above technical solution can meet the user's need to be guided by objects in the reference image so that the generated image includes the items in the reference image.

[0084] Based on the above technical solutions, optionally, in some scenarios, the model trained using the image generation method provided in this application can be used as a base model and fused with other fine-tuned models to obtain the final image generation model.

[0085] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0086] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0087] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0088] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0089] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0090] Figure 7 This is a schematic diagram of an image generation device according to an embodiment of this disclosure. The image generation device provided in this embodiment can be configured in a client or a server. The image generation model used to implement the functions of the image generation device includes a feature extractor and a noise reduction network, see [link to documentation]. Figure 7 The image generation device specifically includes:

[0091] The acquisition module 310 is used to acquire text prompt information and at least one reference image; the reference image includes a first main object; the text prompt information is used to describe the content of the target image.

[0092] Extraction module 320 is used to extract features of the first subject object from the reference image using the feature extractor;

[0093] The noise reduction module 330 is used to perform noise reduction processing on a preset noisy image using the noise reduction network, based on the text prompt information and the features of the first main object, to obtain a target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0094] Furthermore, the acquisition module 310 is also used to acquire an original image, the original image including a first original object;

[0095] The extraction module 320 is also used to extract features of the first original object from the original image using the feature extractor.

[0096] Furthermore, the noise reduction module 330 is also used to utilize the noise reduction network to perform noise reduction processing on the preset noisy image based on the text prompt information, the features of the first main object, and the features of the first original object to obtain a target image; the target image includes a first derived object and a first sub-object; the first derived object corresponds to the first original object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0097] Furthermore, the number of reference images is multiple; the first subject object in different reference images is different;

[0098] Extraction module, used for:

[0099] Using the feature extractor, features of each of the first subject objects are extracted from the reference image;

[0100] Noise reduction module, used for:

[0101] The features of each of the first subject objects are stitched together to obtain the stitched image features;

[0102] Using the aforementioned noise reduction network, based on the text prompt information and the features of the stitched image, the preset noisy image is denoised to obtain the target image.

[0103] Furthermore, the feature extractor includes T extraction network units, which are connected in series; the denoising network includes T denoising network units, which are connected in series; and the extraction network units correspond one-to-one with the denoising network units.

[0104] The extraction module is used for:

[0105] In the i-th round of feature extraction, for any reference image, the features of the first subject object to be processed in the i-th round corresponding to the reference image are input into the i-th extraction network unit to obtain the features of the first subject object processed in the i-th round corresponding to the reference image;

[0106] Noise reduction module, used for:

[0107] In the i-th round of noise reduction, the i-th noise reduction network unit is used to perform noise reduction processing on the image to be denoised corresponding to the i-th round, based on the features of the first subject object that has been processed in the i-th round corresponding to the reference image and the text prompt information, to obtain the image after the i-th round of noise reduction.

[0108] Where T is a positive integer, and i is a positive integer greater than or equal to 1 and less than or equal to T; the features of the first subject object to be processed in the first round corresponding to the reference image are the image features obtained by encoding the reference image; the image to be denoised corresponding to the first round is the preset noise image; when i is greater than 1 and less than T, the features of the first subject object to be processed in the (i+1)th round corresponding to the reference image are obtained based on the features of the first subject object already processed in the i-th round corresponding to the reference image; the image to be denoised corresponding to the (i+1)th round is obtained based on the image after denoising in the i-th round.

[0109] Furthermore, when i is greater than or equal to 1 and less than T, in the i-th round of feature extraction, the extraction module is used to:

[0110] In the i-th round of feature extraction, the features of the first subject object to be processed in the i-th round corresponding to the reference image are subjected to self-attention processing to obtain the features of the first subject object that has been processed in the i-th round corresponding to the reference image;

[0111] The features of the first subject object that has been processed in the i-th round and corresponds to the reference image are input into the i-th noise reduction network unit;

[0112] Based on the description information of the first subject object, cross-attention processing is performed on the features of the first subject object that has been processed in the i-th round corresponding to the reference image to obtain the image features of the first subject object to be processed in the (i+1)-th round corresponding to the reference image.

[0113] Furthermore, when i is greater than or equal to 1 and less than T, the number of reference images is multiple; the noise reduction module is used for:

[0114] In the i-th round of noise reduction, the features of the first subject object that has been processed in the i-th round are spliced ​​together to obtain the spliced ​​features of the first subject object that has been processed in the i-th round.

[0115] Based on the noise reduction image corresponding to the i-th round, cross-attention processing is performed on the features of the first subject object that has been processed in the i-th round after stitching to obtain the cross-attention features of the first subject object in the i-th round.

[0116] Self-attention processing is performed on the noise reduction image in the i-th round to obtain the self-attention features of the noise reduction image in the i-th round;

[0117] The self-attention features of the noise reduction image corresponding to the i-th round and the cross-attention features of the first subject object in the i-th round are fused to obtain the fused image features of the i-th round.

[0118] Based on the text prompt information, cross-attention processing is performed on the fused image features of the i-th round to obtain the denoised image of the i-th round.

[0119] Furthermore, the device also includes a training module for:

[0120] A sample data pair is obtained, comprising a generated sample image, sample text prompt information, at least one sample reference image, and a noisy sample image; the sample reference image includes a first main sample object; the generated sample image includes a second main sample object and a second sub-object; the second sub-object corresponds to the first main sample object, and the image features of the second sub-object and its corresponding first main sample object are consistent; the sample text prompt information is used to describe the content of the generated sample image; the noisy sample image is an image after adding noise to the generated sample image.

[0121] Using the feature extractor, features of the first main object of the sample are extracted from the sample reference image;

[0122] Using the denoising network, based on the sample text prompt information and the features of the first main object of the sample, the noisy sample image is denoised to obtain a predicted image; the predicted image includes a second derived object and a third sub-object; the third sub-object corresponds to the first main object of the sample.

[0123] Based on the differences between the third sub-object and the first subject object of the sample that have a corresponding relationship, the parameters in the image generation model are adjusted.

[0124] Furthermore, the training module is used for:

[0125] The loss function is determined based on the high-frequency component differences in texture and / or edge information between the corresponding third sub-object and the first main object of the sample;

[0126] Based on the loss function, the parameters in the image generation model are adjusted.

[0127] Furthermore, in the process of extracting features of the first subject object of the sample from the sample reference image, a total of T rounds of image feature extraction are performed; furthermore, the training module is used for:

[0128] Based on the sample-generated image, a target mask is determined; the target mask is used to indicate the position of the second sub-object of the sample in the sample-generated image.

[0129] During the i-th round of noise reduction, a query vector for the i-th round of noise reduction is obtained based on the noise reduction image corresponding to the i-th round.

[0130] Based on the features of the first subject object processed in the i-th round, the key vector of the features of the first subject object processed in the i-th round is obtained; the features of the first subject object processed in the i-th round are the features of the first subject object extracted in the i-th round of image feature extraction process;

[0131] Based on the query vector and the key vector, an attention map is obtained.

[0132] Based on the difference between the attention map and the target mask, the parameters in the image generation model are adjusted;

[0133] Where T is a positive integer, and i is a positive integer greater than or equal to 1 and less than or equal to T.

[0134] The image generation apparatus provided in this disclosure can execute the steps performed by the client or server in the image generation method provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0135] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0136] like Figure 8 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the image generation method as described in the embodiments of this disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0137] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to exchange information with other devices wirelessly or via wired communication. Although Figure 8 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0138] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the image generation method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0139] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] In some implementations, clients and servers may communicate using any known or future network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future network.

[0141] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0142] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0143] Acquire text prompt information and at least one reference image; the reference image includes a first subject object; the text prompt information is used to describe the content of the target image;

[0144] Using the feature extractor, features of the first subject object are extracted from the reference image;

[0145] Using the denoising network, based on the text prompt information and the features of the first main object, the preset noisy image is denoised to obtain the target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

[0146] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0147] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0150] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0153] One or more processors;

[0154] Memory, used to store one or more programs;

[0155] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the image generation methods provided in this disclosure.

[0156] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements an image generation method as described in any of the present disclosure.

[0157] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the image generation method described above.

[0158] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that, The image generation model used to implement the image generation method includes a feature extractor and a noise reduction network; the method includes: Acquire text prompt information and at least one reference image; the reference image includes a first subject object; the text prompt information is used to describe the content of the target image; Using the feature extractor, features of the first subject object are extracted from the reference image; Using the denoising network, based on the text prompt information and the features of the first main object, the preset noisy image is denoised to obtain the target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

2. The method according to claim 1, characterized in that, Also includes: Acquire the original image, which includes a first original object; The feature extractor is used to extract features of the first original object from the original image.

3. The method according to claim 2, characterized in that, The step of using the noise reduction network to perform noise reduction processing on the preset noisy image based on the text prompt information and the features of the first subject object to obtain the target image further includes: Using the denoising network, based on the text prompt information, the features of the first main object, and the features of the first original object, a preset noisy image is denoised to obtain a target image; the target image includes a first derived object and a first sub-object; the first derived object corresponds to the first original object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

4. The method according to claim 1, characterized in that, There are multiple reference images; the first subject object is different in different reference images; The step of extracting features of the first subject object from the reference image using the feature extractor includes: Using the feature extractor, features of each of the first subject objects are extracted from the reference image; The step of using the noise reduction network to perform noise reduction processing on the preset noisy image based on the text prompt information and the features of the first subject object to obtain the target image includes: The features of each primary object are stitched together to obtain the stitched image features; Using the aforementioned noise reduction network, based on the text prompt information and the features of the stitched image, the preset noisy image is denoised to obtain the target image.

5. The method according to claim 1, characterized in that, The feature extractor includes T extraction network units, which are connected in series; the denoising network includes T denoising network units, which are connected in series; the extraction network units and the denoising network units correspond one-to-one. The step of extracting features of the first subject object from the reference image using the feature extractor includes: during the i-th round of feature extraction, for any reference image, inputting the features of the first subject object to be processed in the i-th round corresponding to the reference image into the i-th extraction network unit to obtain the features of the first subject object processed in the i-th round corresponding to the reference image; The step of using the denoising network to denoise a preset noisy image based on the text prompt information and the features of the first subject object to obtain a target image includes: in the i-th round of denoising, using the i-th denoising network unit, based on the features of the first subject object that has been processed in the i-th round corresponding to the reference image and the text prompt information, to denoise the image to be denoised in the i-th round to obtain the image after the i-th round of denoising; Where T is a positive integer, and i is a positive integer greater than or equal to 1 and less than or equal to T; the features of the first subject object to be processed in the first round corresponding to the reference image are the image features obtained by encoding the reference image; the image to be denoised corresponding to the first round is the preset noise image; when i is greater than 1 and less than T, the features of the first subject object to be processed in the (i+1)th round corresponding to the reference image are obtained based on the features of the first subject object already processed in the i-th round corresponding to the reference image; the image to be denoised corresponding to the (i+1)th round is obtained based on the image after denoising in the i-th round.

6. The method according to claim 5, characterized in that, When i is greater than or equal to 1 and less than T, in the i-th round of feature extraction, the features of the first subject object to be processed in the i-th round corresponding to the reference image are input into the i-th extraction network unit to obtain the features of the first subject object processed in the i-th round corresponding to the reference image, including: In the i-th round of feature extraction, the features of the first subject object to be processed in the i-th round corresponding to the reference image are subjected to self-attention processing to obtain the features of the first subject object that has been processed in the i-th round corresponding to the reference image; The features of the first subject object that has been processed in the i-th round and corresponds to the reference image are input into the i-th noise reduction network unit; Based on the description information of the first subject object, cross-attention processing is performed on the features of the first subject object that has been processed in the i-th round corresponding to the reference image to obtain the features of the first subject object to be processed in the (i+1)-th round corresponding to the reference image.

7. The method according to claim 5, characterized in that, When i is greater than or equal to 1 and less than T, the number of reference images is multiple; in the i-th round of denoising, the i-th denoising network unit is used to denoise the image to be denoised corresponding to the i-th round based on the features of the first subject object processed in the i-th round corresponding to the reference image and the text prompt information, to obtain the denoised image after the i-th round, including: In the i-th round of noise reduction, the features of each first subject object processed in the i-th round are spliced ​​together to obtain the spliced ​​features of the first subject object processed in the i-th round. Based on the noise reduction image corresponding to the i-th round, cross-attention processing is performed on the features of the first subject object that has been processed in the i-th round after stitching to obtain the cross-attention features of the first subject object in the i-th round. Self-attention processing is performed on the noise-to-be-reduced image corresponding to the i-th round to obtain the self-attention features of the noise-to-be-reduced image in the i-th round; The self-attention features of the noise reduction image in the i-th round and the cross-attention features of the first subject object in the i-th round are fused to obtain the fused image features in the i-th round. Based on the text prompt information, cross-attention processing is performed on the fused image features of the i-th round to obtain the denoised image of the i-th round.

8. The method according to claim 1, characterized in that, The training method for the image generation model includes: A sample data pair is obtained, comprising a generated sample image, sample text prompt information, at least one sample reference image, and a noisy sample image; the sample reference image includes a first main sample object; the generated sample image includes a second main sample object and a second sub-object; the second sub-object corresponds to the first main sample object, and the image features of the second sub-object and its corresponding first main sample object are consistent; the sample text prompt information is used to describe the content of the generated sample image; the noisy sample image is an image after adding noise to the generated sample image. Using the feature extractor, features of the first main object of the sample are extracted from the sample reference image; Using the denoising network, based on the sample text prompt information and the features of the first main object of the sample, the noisy sample image is denoised to obtain a predicted image; the predicted image includes a second derived object and a third sub-object; the third sub-object corresponds to the first main object of the sample. Based on the differences between the third sub-object and the first subject object of the sample that have a corresponding relationship, the parameters in the image generation model are adjusted.

9. The method according to claim 8, characterized in that, The adjustment of parameters in the image generation model based on the difference between the corresponding third sub-object and the sample first main object includes: The loss function is determined based on the high-frequency component differences in texture and / or edge information between the corresponding third sub-object and the first main object of the sample; Based on the loss function, the parameters in the image generation model are adjusted.

10. The method according to claim 8, characterized in that, In the process of extracting features of the first main object of the sample from the sample reference image using the feature extractor, a total of T rounds of image feature extraction are performed; in the process of denoising the noisy sample image using the denoising network based on the sample text prompt information and the features of the first main object of the sample to obtain the predicted image, a total of T rounds of denoising are performed, and the method further includes: Based on the sample-generated image, a target mask is determined; the target mask is used to indicate the position of the second sub-object of the sample in the sample-generated image. During the i-th round of noise reduction, a query vector for the i-th round of noise reduction is obtained based on the noise reduction image corresponding to the i-th round. Based on the features of the first subject object processed in the i-th round, the key vector of the features of the first subject object processed in the i-th round is obtained; the features of the first subject object processed in the i-th round are the features of the first subject object extracted in the i-th round of image feature extraction process; Based on the query vector and the key vector, an attention map is obtained. Based on the difference between the attention map and the target mask, the parameters in the image generation model are adjusted; Where T is a positive integer, and i is a positive integer greater than or equal to 1 and less than or equal to T.

11. An image generation apparatus, characterized in that, The image generation model for implementing the image generation method includes a feature extractor and a noise reduction network, and the apparatus includes: An acquisition module is used to acquire text prompt information and at least one reference image; the reference image includes a first subject object; the text prompt information is used to describe the content of the target image. The extraction module is used to extract features of the first subject object from the reference image using the feature extractor; The noise reduction module is used to perform noise reduction processing on a preset noisy image using the noise reduction network, based on the text prompt information and the features of the first main object, to obtain a target image; the target image includes a first derived object and a first sub-object; the first sub-object corresponds to the first main object, and the image features of the first sub-object and the first main object that have a corresponding relationship are consistent.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images