Image generation method, apparatus, device, and medium

Abstract

The present disclosure provides an image generation method, device, equipment and medium, a specific embodiment of the method comprises: obtaining text guide information comprising a plurality of object guide words; the plurality of object guide words correspond to a plurality of target objects; obtaining a noise image and a plurality of condition control images for constraining the plurality of target objects; using a target model, based on the text guide information and the plurality of condition control images, the noise image is denoised multiple times, and a target image comprising the plurality of target objects is generated. This embodiment can constrain the plurality of target objects involved in the text guide information through the plurality of condition control images, so as to more accurately control the plurality of target objects when generating the target image, improve the image generation effect, and improve the user experience.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing technology, and more particularly to image generation methods, apparatus, devices, and media. Background Technology

[0002] With the continuous development of network technology and image processing technology, digital images are increasingly used in people's lives, providing entertainment services, bringing numerous conveniences, and adding more fun. To further improve digital images, artificial intelligence technology has been applied to the field of image processing, giving rise to image generation technology. Among related technologies, new images related to known images can be generated based on techniques such as GANs and VAEs. However, GANs and VAEs have certain limitations, lacking flexibility and producing relatively poor-quality images. Currently, a more sophisticated image generation solution is needed. Summary of the Invention

[0003] Embodiments of this disclosure describe an image generation method, apparatus, device, and medium.

[0004] According to a first aspect, an image generation method is provided, the method comprising: acquiring text guidance information including multiple object guidance words; the multiple object guidance words corresponding to multiple target objects; acquiring a noisy image and multiple conditional control images for constraining the multiple target objects; and using a target model, performing multiple denoising processes on the noisy image based on the text guidance information and the multiple conditional control images to generate a target image including the multiple target objects.

[0005] According to a second aspect, an image generation apparatus is provided, the apparatus comprising: a first acquisition unit configured to acquire textual guidance information including a plurality of object guidance words; the plurality of object guidance words corresponding to a plurality of target objects; a second acquisition unit configured to acquire a noisy image and a plurality of conditional control images for constraining the plurality of target objects; and a generation unit configured to use a target model to perform multiple denoising processes on the noisy image based on the textual guidance information and the plurality of conditional control images to generate a target image including the plurality of target objects.

[0006] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform any of the methods described in the first aspect.

[0007] According to a fourth aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any one of the first aspects.

[0008] According to an embodiment of this disclosure, an image generation scheme is provided. This scheme acquires textual guidance information including multiple object guide words, where each object guide word corresponds to a multiple target object. It also acquires a noisy image and multiple conditional control images used to constrain the multiple target objects. Using a target model, based on the textual guidance information and the conditional control images, the noisy image undergoes multiple denoising processes to generate a target image including the multiple target objects. This allows for the constraint of the multiple target objects mentioned in the textual guidance information through multiple conditional control images, enabling more precise control over the multiple target objects during target image generation, improving image generation quality, and enhancing the user experience.

[0009] Because this embodiment employs a cross-attention mechanism to denoise the image to be processed based on the text sequence and image sequence, the conditional control image can better constrain the target objects involved in the text guidance information, thereby further improving the image generation effect.

[0010] Since this embodiment encodes multiple conditional control images into the same target latent space as the noise latent vector, it enables better aggregation of the conditional control image encoding and the noise latent vector, thereby improving the constraint effect of the conditional control image and helping to improve the image generation effect.

[0011] This embodiment further improves the quality of image generation by using conditional embedding to distinguish between noise sequences and conditional sequences in the latent space.

[0012] Because this embodiment uses index embedding to distinguish different sequence parts corresponding to different condition control images, it improves the constraint ability of the condition control images on the target object, and further improves the quality of image generation.

[0013] In this embodiment, the loss is calculated using the mask image corresponding to the sample object in the sample reference image through the cross-attention mechanism. This concentrates the attention of the generated image to the sample reference image on the sample object, improves the constraint ability of the conditional control image on the target object, and further improves the quality of image generation.

[0014] Because this embodiment uses progressive training in multiple stages, it accelerates the convergence speed of the target model and improves the training efficiency and effectiveness of the target model. Attached Figure Description

[0015] Figure 1 This disclosure is a schematic diagram illustrating an application scenario of image generation according to an exemplary embodiment;

[0016] Figure 2This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure;

[0017] Figure 3 This disclosure is a flowchart illustrating an image generation method according to an exemplary embodiment;

[0018] Figure 4 This disclosure is a schematic diagram of an image generation scene according to an exemplary embodiment;

[0019] Figure 5 This disclosure is a block diagram of an image generation apparatus according to an exemplary embodiment;

[0020] Figure 6 This is a schematic block diagram of an electronic device provided in some embodiments of this disclosure. Detailed Implementation

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as electronic devices, applications, servers, or storage media, that perform the operations of the technical solutions disclosed herein, based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] The technical solutions provided in this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0026] With the continuous development of network technology and image processing technology, digital images are increasingly used in people's lives, providing entertainment services, bringing numerous conveniences, and adding more fun to people's lives. To further improve digital images, artificial intelligence technology has been applied to the field of image processing, giving rise to image generation technology. Among related technologies, new images related to known images can be generated based on techniques such as GANs and VAEs. However, GANs and VAEs have certain limitations, lacking flexibility and producing relatively poor-quality images.

[0027] This disclosure provides an image generation scheme that acquires textual guidance information including multiple object guide words, each corresponding to a multiple target object, and obtains a noisy image and multiple conditional control images used to constrain the multiple target objects. Using a target model, based on the textual guidance information and the conditional control images, the noisy image undergoes multiple denoising processes to generate a target image including the multiple target objects. This allows for more precise control of the multiple target objects mentioned in the textual guidance information through multiple conditional control images, improving image generation quality and enhancing the user experience.

[0028] See Figure 1 This is a schematic diagram illustrating an application scenario of image generation according to an exemplary embodiment. For example... Figure 1 As shown, this scheme involves a diffusion model M for image generation. When generating an image, the diffusion model M requires multiple time-step iterations for multi-step denoising. The figure illustrates any time step of the diffusion model M in the reverse process (image generation process). The image generation process is described below.

[0029] like Figure 1 As shown, in the image generation stage, firstly, the user can input text guidance information 101 into the image processing client through the terminal device. The specific content of the text guidance information 101 could be, for example, "In the beach, a woman wearing a sweater and a short skirtThe text guidance information 101 may include multiple guidance words, such as "beach," "woman," "sweater," and "short skirt." "Woman," "sweater," and "short skirt" can be used as object guidance words, each corresponding to a target object in the target image to be generated. For example, the object guidance word "woman" corresponds to a person in the target image, "sweater" corresponds to the top worn by the person in the target image, and "short skirt" corresponds to the bottom worn by the person in the target image. Therefore, the "person," the "top," and the "bottom" in the target image can all be target objects in the target image.

[0030] In addition, users can input multiple conditional control images 102 for constraining target objects into the image processing client via a terminal device. For example, the conditional control images 102 may include images 1021, 1022, and 1023. Among them, image 1021 can be used to constrain the target object "person", image 1022 can be used to constrain the target object "person's upper clothing", and image 1023 can be used to constrain the target object "person's lower clothing".

[0031] Next, the image processing client can encode the text guidance information 101 using encoder 1 to obtain a one-dimensional text sequence 1031. Encoder 1 can be a text encoder used for encoding text information. Alternatively, encoder 2 can be used to encode images 1021, 1022, and 1023 respectively, obtaining two-dimensional latent vectors corresponding to images 1021, 1022, and 1023 in the target latent space. These two-dimensional latent vectors are then converted into a one-dimensional sequence. This sequence may include a sequence portion 1032 corresponding to image 1021, a sequence portion 1033 corresponding to image 1023, and a sequence portion 1034 corresponding to image 1023.

[0032] Different index embeddings (IEs) can be obtained for images 1021, 1022, and 1023 respectively. For example, for image 1021, index embedding IE1 can be obtained; for image 1022, index embedding IE2 can be obtained; and for image 1023, index embedding IE3 can be obtained. Then, using these different index embeddings, the sequence portions of images 1021, 1022, and 1023 can be processed differently. For example, index embedding IE1 can be added to sequence portion 1032 corresponding to image 1021; index embedding IE2 can be added to sequence portion 1033 corresponding to image 1022; and index embedding IE3 can be added to sequence portion 1034 corresponding to image 1023. Furthermore, to distinguish the sequence portion obtained from the conditionally controlled image 102 from other sequence portions, a conditional embedding (CE) can be obtained for the conditionally controlled image, and the conditional embedding CE can be added to sequence portions 1032, 1033, and 1034 respectively.

[0033] Then, the image processing client can also acquire the image to be processed. During the first denoising process, a random noisy image can be acquired as the image to be processed. During subsequent denoising processes, the intermediate result obtained from the previous time step can be acquired as the image to be processed. Next, the two-dimensional noise latent vector of the image to be processed in the target latent space can be acquired, and the two-dimensional noise latent vector can be converted into a one-dimensional sequence to obtain the noise sequence 1035.

[0034] Finally, by using positional embedding (PE), the text sequence 1031, sequence portion 1032, sequence portion 1033, sequence portion 1034, and noise sequence 1035 are aligned and then concatenated to obtain sequence 103. Sequence 103 is then input into the diffusion model M, which predicts the noise to be removed based on sequence 103 and performs denoising on the image based on the noise to be removed, resulting in image 104, which serves as the denoising result for the current time step.

[0035] It should be noted that, in Figure 1 In the example of image generation, the image processing client directly generates the image. In other embodiments, the image processing client can transmit the text guidance information 101 and the conditional control image 102 to the image processing server deployed on the service platform via the network. The image processing server then generates the target image based on the text guidance information 101 and the conditional control image 102, and transmits the target image to the image processing client via the network to provide the target image to the user. See details below. Figure 2 Example.

[0036] Figure 2This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure.

[0037] like Figure 2 As shown, system architecture 200 may include terminal device 202, network 203, and server 204. It should be understood that... Figure 2 The number or type of terminal devices, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number or type of terminal devices, networks, and servers.

[0038] Network 203 is a medium used to provide communication links between terminal devices and servers. Network 203 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0039] The terminal device 202 is equipped with an image processing client. The terminal device 202 can interact with the server via network 203 to receive or send requests or information. The terminal device 202 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, and smart wearable devices.

[0040] Server 204 houses an image processing server. Server 204 can store, analyze, and process received data, and can also send control commands or requests to terminal devices or other servers. The server can provide image processing services in response to user service requests. It is understood that a single server can provide one or more services, and the same service can be provided by multiple servers.

[0041] based on Figure 2 In the system architecture shown in this embodiment, user 201 can input text guidance information including multiple object guidance words, each corresponding to a multiple target object, through terminal device 202. User 201 can also input multiple conditional control images to constrain the multiple target objects. Terminal device 202 can transmit the text guidance information and the multiple conditional control images to server 204 via network 203. After receiving the text guidance information and the multiple conditional control images, server 204 can generate a target image based on these elements. Finally, server 204 can return the target image to terminal device 202 via network 203, allowing user 201 to view and save the target image through terminal device 202.

[0042] The present disclosure will now be described in detail with reference to specific embodiments.

[0043] Figure 3This is a flowchart illustrating an image generation method according to an exemplary embodiment. The method can be applied to an image processing client or an image processing server. In this embodiment, the image processing client is installed on a terminal device, which may include, but is not limited to, mobile terminal devices such as smartphones, smart wearable devices, tablets, laptops, and desktop computers. The image processing server is deployed in a service platform, which can be any device, server, or device cluster with computing and processing capabilities. The method may include the following steps:

[0044] like Figure 3 As shown, in step 301, text guidance information including multiple object guidance words is obtained, and in step 302, a noise image and multiple condition control images are obtained.

[0045] In this embodiment, the image processing client can provide users with an information input interface, which may include a text information input interface and an image information input interface. Users can input text guidance information through the text information input interface and input multiple conditional control images through the image information input interface. The text guidance information may include multiple guiding words related to objects; some or all of these guiding words can be used as object guiding words, and the object corresponding to each object guiding word can be used as the target object. The conditional control images can be used to constrain the target object. Specifically, each conditional control image corresponds to one target object, and there is a one-to-one correspondence between the conditional control image and the target object. Any given conditional control image can constrain the generation of the target object corresponding to that conditional control image in the target image.

[0046] In this embodiment, the aforementioned multiple target objects can correspond to multiple specified conditional attributes, which may include, but are not limited to, texture, subject, clothing, background, and style. Different target objects that control image constraints under different conditions may correspond to the same specified conditional attribute or different specified conditional attributes.

[0047] For example, see Figure 1 The illustrated embodiments, such as Figure 1As shown, the specific content of the text guidance information could be "In the beach, a woman wearing a sweater and a short skirt." The guiding words in the text guidance information could be "beach," "woman," "sweater," and "short skirt." "Woman," "sweater," and "short skirt" can be used as object guiding words. The target object corresponding to the object guiding word "woman" is the person in the target image; the target object corresponding to "sweater" is the top worn by the person in the target image; and the target object corresponding to "short skirt" is the bottom worn by the person in the target image. Images 1021, 1022, and 1023 can be used as conditional control images. Image 1021 can constrain the target object "person," image 1022 can constrain the target object "the top worn by the person," and image 1021 can constrain the target object "the bottom worn by the person."

[0048] For example, such as Figure 4 As shown, the specific content of the text guidance information 401 could be "In the room, a boy holds a box, and he is laughing. In his side, another cup on the table." The guiding words in the text guidance information could be "room," "boy," "box," "laughing," "Cup," and "table." "Box" and "Cup" can be used as object guiding words. The target object corresponding to the object guiding word "box" is the "box in the boy's hand" in the target image, and the target object corresponding to "Cup" is the "cup on the table" in the target image. The specified conditional attribute corresponding to the target object "box in the boy's hand" can be a subject, and the specified conditional attribute corresponding to the target object "cup on the table" can also be a subject. Images 402 and 403 can be used as conditional control images. Image 402 constrains the target object "box in the boy's hand," and image 403 constrains the target object "cup on the table." Image 404 is the final generated target image.

[0049] In step 303, the target image is generated by using the target model and performing multiple denoising processes on the noisy image based on the text guidance information and conditional control image.

[0050] In this embodiment, the target model can be a pre-trained diffusion model. The target model can be used to perform multiple denoising processes on the noisy image based on textual guidance information and conditional control images. Each of these multiple denoising processes can include the following steps: First, an image to be processed can be acquired. If this is the first time denoising is performed, a random noisy image can be acquired as the image to be processed. If this is not the first time denoising is performed, an intermediate image obtained from the previous denoising process can be acquired as the image to be processed.

[0051] Next, a text encoder can be used to encode the text guidance information, resulting in a one-dimensional text sequence. Based on the image to be processed and multiple conditional control images, a one-dimensional image sequence is obtained. A cross-attention mechanism is then used to denoise the image to be processed based on the text sequence and the image sequence. For example, the image to be processed and multiple conditional control images can be concatenated to obtain a concatenated image, which is then encoded to obtain a one-dimensional image sequence. Alternatively, a two-dimensional noise latent vector of the image to be processed in the target latent space can be obtained, and multiple conditional images can be encoded into the same target latent space as the noise latent vector, resulting in a two-dimensional conditional latent vector. Finally, a one-dimensional image sequence is obtained based on the noise latent vector and the conditional latent vector.

[0052] Since this embodiment encodes multiple conditional control images into the same target latent space as the noise latent vector, it enables better aggregation of the conditional control image encoding and the noise latent vector, thereby improving the constraint effect of the conditional control image and helping to improve the image generation effect.

[0053] In one implementation, the noisy latent vector can be transformed into a one-dimensional noise sequence, and the conditional latent vector into a one-dimensional conditional sequence. The noise sequence and the conditional sequence can then be directly concatenated to obtain an image sequence.

[0054] In another implementation, the noisy latent vector can be transformed into a one-dimensional noise sequence, the conditional latent vector into a one-dimensional conditional sequence, and conditional embeddings for multiple conditionally controlled images can be obtained. These conditional embeddings can be learnable conditional embeddings optimized during training, capable of distinguishing the noise sequence and the conditional sequence in the latent space. The conditional embeddings can be used to differentiate the conditional sequence from the noise sequence, yielding a first result. For example, the conditional embeddings can be added to preset positions in the conditional sequence to obtain the first result. The noise sequence and the first result are then concatenated to form an image sequence.

[0055] This embodiment further improves the quality of image generation by using conditional embedding to distinguish between noise sequences and conditional sequences in the latent space.

[0056] In another implementation, firstly, the noisy latent vector can be converted into a one-dimensional noise sequence, and the conditional latent vector can be converted into a one-dimensional conditional sequence. Then, conditional embeddings for multiple conditional control images are obtained. These conditional embeddings are used to differentiate the conditional sequence from the noise sequence, yielding a first result. This first result can include multiple sequence parts, each corresponding to one of the multiple conditional control images. That is, there is a one-to-one correspondence between the multiple conditional control images and the multiple sequence parts in the first result.

[0057] Next, index embeddings corresponding to multiple conditional control images can be obtained. Specifically, for each conditional control image, an index embedding can be obtained as the index embedding corresponding to that conditional control image, so that each conditional control image corresponds to one index embedding, and different conditional control images correspond to different index embeddings. This index embedding can be a learnable index embedding optimized during training, which can distinguish the sequence parts corresponding to different conditional control images in the latent space. The index embedding can be used to distinguish multiple sequence parts to obtain a second result. Specifically, for any conditional control image, the index embedding corresponding to that conditional control image is used to distinguish the sequence part corresponding to that conditional control image in the first result from other sequence parts in the first result, to obtain a second result. For example, the conditional control images include image A and image B, where image A corresponds to index conditional embedding a, and image B corresponds to index conditional embedding b. Index conditional embedding a can be added to a preset position in the sequence part corresponding to image A, and index conditional embedding b corresponding to image B can be added to a preset position in the sequence part corresponding to image B to obtain a second result. Then, the noise sequence and the second result are concatenated to obtain an image sequence.

[0058] Because this embodiment uses index embedding to distinguish different sequence parts corresponding to different condition control images, it improves the constraint ability of the condition control images on the target object, and further improves the quality of image generation.

[0059] Finally, since cross-attention between text and image sequences typically generates high similarity in the spatial regions where objects appear, after obtaining one-dimensional text and image sequences, a cross-attention mechanism can be used to denoise the image to be processed based on these sequences. Specifically, the cross-attention between the text and image sequences can be calculated, the noise to be removed can be predicted based on the cross-attention, and the image to be processed can be denoised based on the noise to be removed. Because this embodiment employs a cross-attention mechanism to denoise the image to be processed based on the text and image sequences, the conditional control image can better constrain the target objects involved in the text guidance information, thereby further improving the image generation effect.

[0060] It should be noted that, in some embodiments, this method can also be used in the training process of the target model. During the training process of the target model, sample reference images can be obtained. The sample reference images can be pre-obtained standard images, and a second loss can be calculated based on the sample reference images and the target image.

[0061] Additionally, each mask image corresponding to each sample object in the sample reference image is obtained. For example, if the sample reference image includes sample object 1, sample object 2, and sample object 3, mask image 1 corresponding to sample object 1, mask image 2 corresponding to sample object 2, and mask image 3 corresponding to sample object 3 can be obtained respectively.

[0062] Next, the cross-attention between each of the multiple conditional control images and the target image is calculated, resulting in multiple attention results. Based on these attention results and multiple mask images, a first loss is calculated. For example, the difference between the attention result corresponding to any conditional control image and the mask image corresponding to the sample object constrained by that conditional control image can be calculated to obtain the first loss corresponding to that conditional control image. Then, a predicted loss can be obtained based on the first loss and the second loss. The predicted loss includes at least the first loss and the second loss. Finally, the target model can be updated based on the predicted loss.

[0063] In this embodiment, the loss is calculated using the mask image corresponding to the sample object in the sample reference image through the cross-attention mechanism. This concentrates the attention of the generated image to the sample reference image on the sample object, improves the constraint ability of the conditional control image on the target object, and further improves the quality of image generation.

[0064] In other embodiments, the method is used for the training process of a target model. This training process may include multiple stages, at least including a first stage and a second stage. It is understood that, in addition to the first node and the second stage, other stages may be included, and this embodiment is not limited in this regard. Specifically, the first stage precedes the second stage. The first stage trains for the generation of a first object in the target image, where the first object corresponds to an object corresponding to some of the specified conditional attributes out of a plurality of specified conditional attributes. The second stage trains for the generation of a second object in the target image, where the second object corresponds to an object corresponding to all of the specified conditional attributes out of a plurality of specified conditional attributes.

[0065] Specifically, the training of the target model can be divided into multiple stages. In the first stage, the target model can be trained on its ability to generate images for a small number of objects corresponding to specified conditional attributes. After completing the first stage of training, in subsequent stages, some objects corresponding to the specified conditional attributes are added, and the target model is optimized based on the generation ability of these newly added objects in the images. Finally, the target model can be trained on its ability to generate images for at least all objects corresponding to the specified conditional attributes.

[0066] Because this embodiment uses progressive training in multiple stages, it accelerates the convergence speed of the target model and improves the training efficiency and effectiveness of the target model.

[0067] This disclosure provides an image generation scheme that acquires textual guidance information including multiple object guide words, each corresponding to a multiple target object. It also acquires a noisy image and multiple conditional control images used to constrain these target objects. Using a target model, based on the textual guidance information and the conditional control images, the noisy image undergoes multiple denoising processes to generate a target image containing the multiple target objects. This allows for more precise control of the multiple target objects mentioned in the textual guidance information through multiple conditional control images, improving image generation quality and enhancing the user experience.

[0068] It should be noted that although the operations of the methods of this disclosure embodiment are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0069] Corresponding to the foregoing embodiments of the image generation method, this disclosure also provides embodiments of the image generation apparatus.

[0070] like Figure 5 As shown, Figure 5 This disclosure is a block diagram of an image generation apparatus according to an exemplary embodiment, the apparatus including: a first acquisition unit 501, a second acquisition unit 502 and a generation unit 503.

[0071] The first acquisition unit 501 is configured to acquire text guidance information including multiple object guide words, wherein the multiple object guide words correspond to multiple target objects.

[0072] The second acquisition unit 502 is configured to acquire a noisy image and multiple condition control images for constraining multiple target objects.

[0073] The generation unit 503 is configured to use the target model, based on text guidance information and multiple conditional control images, to perform multiple denoising processes on the noisy image to generate a target image including multiple target objects.

[0074] In some implementations, multiple target objects correspond to multiple specified conditional attributes, which may include at least one of the following: texture; subject; clothing; background; style.

[0075] In other embodiments, during any one of the multiple denoising processes, the generation unit 503 may perform the following steps: acquiring the image to be processed. Specifically, if denoising is being performed for the first time, the noisy image is acquired as the image to be processed; if denoising is not being performed for the first time, the intermediate image obtained from the previous denoising process is acquired as the image to be processed. The text guidance information is encoded to obtain a one-dimensional text sequence. Based on the image to be processed and multiple conditional control images, a one-dimensional image sequence is obtained. Denoising is performed on the image to be processed based on the text sequence and the image sequence using a cross-attention mechanism.

[0076] In other embodiments, the generation unit 503 may obtain a one-dimensional image sequence based on the image to be processed and the plurality of conditional control images in the following manner: obtaining a two-dimensional noise latent vector of the image to be processed in the target latent space; encoding the plurality of conditional images into the same target latent space as the noise latent vector to obtain a two-dimensional conditional latent vector; and obtaining a one-dimensional image sequence based on the noise latent vector and the conditional latent vector.

[0077] In other embodiments, the generation unit 503 can obtain a one-dimensional image sequence based on the noise latent vector and the conditional latent vector in the following manner: converting the noise latent vector into a one-dimensional noise sequence, converting the conditional latent vector into a one-dimensional conditional sequence, obtaining conditional embeddings for multiple conditionally controlled images, using the conditional embeddings to perform distinguishing processing on the conditional sequence for the noise sequence, obtaining a first result, and obtaining a one-dimensional image sequence based on the noise sequence and the first result.

[0078] In other embodiments, the first result may include multiple sequence portions, each corresponding to a multiple conditional control images. The generation unit 503 can obtain a one-dimensional image sequence based on the noise sequence and the first result as follows: It acquires the index embeddings corresponding to each of the multiple conditional control images, and uses the index embeddings to differentiate the multiple sequence portions, thus obtaining a second result. Specifically, for any given conditional control image, the index embedding corresponding to that conditional control image is used to differentiate the sequence portion of the first result corresponding to that conditional control image from other sequence portions in the first result; the noise sequence and the second result are then concatenated to obtain the image sequence.

[0079] In other embodiments, the apparatus can also be used in the training process of the target model. The apparatus may further include a training unit (not shown in the figures), configured to: acquire a sample reference image; acquire multiple mask images corresponding to multiple sample objects in the sample reference image; calculate cross-attention between multiple conditional control images and the target image respectively, obtaining multiple attention results; calculate a first loss based on the multiple attention results and the multiple mask images; calculate a second loss based on the sample reference image and the target image; calculate a prediction loss based on the first and second losses; and update the target model according to the prediction loss.

[0080] In other embodiments, the apparatus can be used for training a target model, which may include multiple stages, including at least a first stage and a second stage. The first stage precedes the second stage. Specifically, the first stage trains for generating a first object in the target image, where the first object corresponds to an object corresponding to some of the specified conditional attributes out of a plurality of specified conditional attributes. The second stage trains for generating a second object in the target image, where the second object corresponds to an object corresponding to all of the specified conditional attributes out of a plurality of specified conditional attributes.

[0081] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0082] The following is for reference. Figure 6 , Figure 6 This is a schematic block diagram of an electronic device provided for some embodiments of the present disclosure. The electronic device 920 is, for example, suitable for implementing the image generation method provided in the embodiments of the present disclosure. The electronic device 920 can be a terminal device, etc., and can be used to implement a client or server. The electronic device 920 can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that... Figure 6 The illustrated electronic device 920 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0083] like Figure 6 As shown, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 also stores various programs and data required for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are interconnected via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0084] Typically, the following devices can be connected to I / O interface 925: input devices 926 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 927 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 928 including, for example, magnetic tapes, hard disks, etc.; and communication devices 929. Communication device 929 allows electronic device 920 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6An electronic device 920 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0085] According to embodiments of this disclosure, the image generation method described above can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the image generation method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by a processing device 921, the functions defined in the image generation method provided by embodiments of this disclosure can be implemented.

[0086] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods provided in this disclosure.

[0087] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0088] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0090] Those skilled in the art will recognize that the functions described in the embodiments of this disclosure in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this disclosure. It should be understood that the above descriptions are merely specific implementations of the embodiments of this disclosure and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this disclosure should be included within the scope of protection of this invention.

Claims

1. An image generation method, the method comprising: Retrieve text guidance information including multiple object guide words; The multiple object guide words correspond to multiple target objects; Acquire noisy images and multiple conditional control images for constraining the multiple target objects; Using the target model, based on the text guidance information and the multiple conditional control images, the noisy image is subjected to multiple denoising processes to generate a target image including the multiple target objects.

2. The method according to claim 1, wherein, The multiple target objects correspond to multiple specified conditional attributes; the specified conditional attributes include any of the following: texture; subject; clothing; background; style.

3. The method according to claim 1, wherein, Each of the multiple denoising processes includes the following steps: Obtain the image to be processed; wherein, if the denoising process is performed for the first time, the noisy image is obtained as the image to be processed; if the denoising process is not performed for the first time, the intermediate image obtained from the previous denoising process is obtained as the image to be processed. The text guidance information is encoded to obtain a one-dimensional text sequence; Based on the image to be processed and the multiple condition control images, a one-dimensional image sequence is obtained; The image to be processed is denoised based on the text sequence and the image sequence using a cross-attention mechanism.

4. The method according to claim 3, wherein, The process of obtaining a one-dimensional image sequence based on the image to be processed and the multiple conditional control images includes: Obtain the two-dimensional noise latent vector of the image to be processed in the target latent space; The multiple conditional images are respectively encoded into the target latent space, which is the same as the noise latent vector, to obtain a two-dimensional conditional latent vector; The one-dimensional image sequence is obtained based on the noise latent vector and the conditional latent vector.

5. The method according to claim 4, wherein, The process of obtaining the one-dimensional image sequence based on the noise latent vector and the conditional latent vector includes: The noise latent vector is transformed into a one-dimensional noise sequence; The conditional latent vector is transformed into a one-dimensional conditional sequence; Obtain the conditional embedding for the plurality of conditional control images; The conditional embedding is used to perform discrimination processing on the conditional sequence for the noise sequence to obtain a first result; Based on the noise sequence and the first result, the one-dimensional image sequence is obtained.

6. The method according to claim 5, wherein, The first result includes multiple sequence portions, each corresponding to one of the multiple conditional control images; wherein, obtaining the one-dimensional image sequence based on the noise sequence and the first result includes: Obtain the index embedding corresponding to each of the multiple condition control images; The index embedding is used to distinguish the multiple sequence parts to obtain a second result; wherein, for any conditional control image, the index embedding corresponding to the conditional control image is used to distinguish the sequence part corresponding to the conditional control image in the first result from other sequence parts in the first result. The noise sequence and the second result are concatenated to obtain the image sequence.

7. The method according to claim 1, wherein, The method is used in the training process of the target model, and the method further includes: Obtain a sample reference image; Obtain multiple mask images corresponding to multiple sample objects in the sample reference image; Calculate the cross-attention between the multiple conditional control images and the target image to obtain multiple attention results; Based on the multiple attention results and the multiple mask images, a first loss is calculated; The second loss is calculated based on the sample reference image and the target image; Calculate the predicted loss based on the first loss and the second loss; The target model is updated based on the predicted loss.

8. The method according to claim 2, wherein, The method is used in the training process of the target model, the training process includes multiple stages, the multiple stages including at least a first stage and a second stage; the first stage precedes the second stage; wherein, the first stage trains for the generation of a first object in the target image, the first object being an object corresponding to some of the specified conditional attributes among the multiple specified conditional attributes; the second stage trains for the generation of a second object in the target image, the second object being an object corresponding to all of the specified conditional attributes among the multiple specified conditional attributes.

9. An image generation apparatus, the apparatus comprising: The first acquisition unit is configured to acquire text guidance information including multiple object guide words; The multiple object guide words correspond to multiple target objects; The second acquisition unit is configured to acquire a noisy image and a plurality of condition control images for constraining the plurality of target objects; The generation unit is configured to use a target model to perform multiple denoising processes on the noisy image based on the text guidance information and the multiple conditional control images, thereby generating a target image including the multiple target objects.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images