Image generation method, apparatus, electronic device and medium

Abstract

The present application relates to the technical field of computers, and provides an image generation method, an apparatus, an electronic device, and a medium. The method comprises: acquiring a prompt text; determining from among a plurality of preset texts a target text matching the semantics of the prompt text, wherein the preset texts are used for indicating image feature maps matching the semantics of the preset texts; acquiring a target image feature map indicated by the target text; and using the target image feature map as an initial noise distribution map to generate a target image corresponding to the prompt text. By means of improving the correlation between the initial noise distribution map and the prompt text, the method improves the quality of the generated image, and the degree of matching between the image and the prompt text.

Description

An image generation method, device, electronic device and medium

[0001] The present application claims priority to the Chinese patent application No. 202411329479.3, filed on September 23, 2024, and entitled "An image generation method, device, electronic device and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, and particularly relates to an image generation method, device, electronic device and medium. BACKGROUND

[0003] In the current text-to-image synthesis technology, a text description is converted into a vivid visual image through a generation model, especially a diffusion model. In the process of generating an image, a standard Gaussian distribution is usually used to randomly initialize noise, and an initial noise distribution graph generated is used as a starting point for denoising of the diffusion model. However, the initial noise distribution graph obtained in this way lacks semantic association with the input text description, resulting in poor quality of the generated image, which is difficult to accurately reflect the content of the text description, and leading to low matching degree between the generated image and the text description. SUMMARY

[0004] Embodiments of the present application provide an image generation method, device, electronic device and medium, which improve the matching degree between the generated image and the text.

[0005] To achieve the above-mentioned purpose, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide an image generation method, which takes an image feature map associated with a prompt text in semantics as an initial noise distribution graph, and generates an image corresponding to the prompt text. In this way, by improving the association between the initial noise distribution graph and the prompt text, the quality of the generated image is improved. Specifically, first, a prompt text is obtained, and a target text whose semantics match the prompt text is determined from a plurality of preset texts; wherein the preset text is used to indicate an image feature map consistent with the semantics of the preset text. Then, a target image feature map indicated by the target text is obtained. Finally, the target image feature map is taken as an initial noise distribution graph, and a target image corresponding to the prompt text is generated.

[0007] In the embodiment of the present application, the target text that is highly matched with the prompt text in semantics is found from a plurality of preset texts, and the target image feature map indicated by the target text is taken as the initial noise distribution map, so as to enhance the relevance between the initial noise distribution map and the prompt text. Further, based on the initial noise distribution map highly relevant to the prompt text, the model has more specific guidance in the image generation process, can directly learn and retain the visual information highly relevant to the text description from the feature maps, so as to generate an image that can more accurately reflect the content described in the prompt text, improve the consistency between the image and the prompt text, greatly reduce the deviation between the generated image and the content described in the prompt text, and improve the quality of the generated image.

[0008] In a possible implementation, determining the target text matched with the semantics of the prompt text from a plurality of preset texts comprises: obtaining a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent semantic information of the prompt text; determining a second text embedding vector with the highest similarity to the first text embedding vector according to a text embedding vector corresponding to each preset text; the second text embedding vector is used to represent semantic information of the target text.

[0009] In this implementation, since the text embedding vector not only considers the literal meaning of the words in the prompt text, but also considers the context meaning of the words in the prompt text and the relationship between the words and other words in the prompt text, it can handle the polysemy and synonym problems in language and distinguish the different meanings of the same word in different contexts. Therefore, the text embedding vector can more accurately capture the semantic information of the text, so that the target text found based on the text embedding vector has a higher semantic matching degree with the prompt text, and can accurately find the target text with similar meanings although the literal meanings are different.

[0010] In a possible implementation, obtaining the target image feature map indicated by the target text comprises: determining an image feature map corresponding to the second text embedding vector from a preset correspondence relationship as the target image feature map; the preset correspondence relationship comprises a correspondence relationship between a text embedding vector of each preset text and an image feature map indicated by the preset text.

[0011] In this implementation, the target image feature map is obtained by directly querying the preset correspondence relationship, and the target image feature map is used to initialize the image generation process, avoiding the process of re-extracting features from the image, thereby significantly improving the processing speed and efficiency. In particular, in the case of processing a large number of similar or repeated queries, finding the target image feature map based on the preset correspondence relationship has obvious advantages.

[0012] In a possible implementation, the target image is generated by taking the target image feature map as an initial noise distribution map, and includes: performing multiple times of iterative denoising processing on the target image feature map according to the prompt text by using a preset generation model to obtain the target image; the preset generation model includes an encoder and a decoder; the encoder includes N encoding layers connected in sequence; the decoder includes N decoding layers connected in sequence; N is an integer greater than 1; each encoding layer is connected to a corresponding decoding layer in a skip connection manner; the encoding layer is configured to perform feature extraction on a first feature map according to the prompt text, and output a second feature map; the first feature map is the target image feature map or a feature map output by a previous encoding layer; the decoding layer is configured to perform feature fusion on the second feature map and a third feature map according to the prompt text and a preset factor corresponding to the second feature map and the third feature map respectively, and output a fourth feature map; the third feature map is a feature map output by a previous decoding layer; and the preset factor is configured to perform feature scaling on the corresponding feature map.

[0013] In this implementation, by using the preset factor, the preset generation model performs scaling on feature maps at different levels to different degrees in the feature fusion process, fuses feature information from different levels, and helps the model pay more attention to important feature information when generating the target image, thereby improving the quality and accuracy of the generated image. In addition, the preset factor can be adjusted as needed, thereby enhancing the flexibility and adaptability of the model.

[0014] In a possible implementation, in the decoding layer, the preset factor corresponding to the second feature map is configured to reduce the contribution of the second feature map to the fused fourth feature map; and the preset factor corresponding to the third feature map is configured to enhance the contribution of the third feature map to the fused fourth feature map.

[0015] In this implementation, by adjusting the contribution of different feature maps in the fusion process, the influence of feature information on the generated image can be controlled more finely. Since the second feature map comes from the encoding layer and the third feature map comes from a deeper decoding layer, that is, the second feature map represents the shallow features of the image and contains the background information and noise of the image, and the third feature map represents the deep features of the image and contains the high-frequency information (also referred to as detail information) in the image, therefore, by reducing the contribution of the second feature map to the fused fourth feature map and increasing the contribution of the third feature map to the fused fourth feature map, the detail information in the generated image is enhanced, the noise and unnecessary background information are reduced, thereby improving the quality of the generated image, enhancing the denoising capability of the model, and making the generated image more accurately describe the content contained in the prompt text.

[0016] In a possible implementation, the preset factor corresponding to the second feature map is smaller than the preset factor corresponding to the third feature map.

[0017] In the implementation, in the generation process of the image, the second feature map comes from the encoding layer and contains background information and noise of the image, and the third feature map comes from the deeper decoding layer and contains detailed information in the image. Assigning a larger preset factor, i.e., a higher weight, to the third feature map can ensure that the detailed information of the image represented by the third feature map is retained and emphasized more in the feature fusion process. Assigning a smaller preset factor, i.e., a lower weight, to the second feature map can limit the influence of the shallow information on the target image and reduce noise and unnecessary background information. By using the smaller preset factor of the second feature map and the larger preset factor of the third feature map, the dominance of the deep feature can be maintained in the feature fusion process, while the background information of the shallow layer is moderately retained, so that the generated target image more accurately reflects the content in the prompt text and improves user satisfaction.

[0018] In a possible implementation, the preset generation model is used to perform multiple iteration denoising processes on the target image feature map according to the prompt text to obtain the target image, including: determining the number of iterations and the order of key denoising processes in the multiple iteration denoising processes; for each iteration denoising process, if it is a key denoising process, input the input feature map and the first text embedding vector corresponding to the prompt text into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein the input feature map is the target image feature map or the output feature map of the last iteration denoising process; if it is not a key denoising process, input the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0019] In the implementation, the encoding and decoding layer outputs of the last iteration process are used in the non-key denoising process, which avoids repeated calculation of the entire model, thereby effectively reducing the consumption of computing resources and making the model more advantageous in processing large-scale data or real-time application scenarios. In addition, the number of iterations and the order of key denoising processes can be determined according to actual needs to adjust the image generation process, thereby adapting to different application scenarios and performance requirements.

[0020] In a second aspect, an image generation apparatus is provided, which comprises functional units for performing any of the methods provided in the first aspect, and each functional unit performs actions by hardware or by hardware executing corresponding software. For example, the image generation apparatus can comprise a first obtaining module, a determining module, a second obtaining module, and a generating module. The first obtaining module is configured to obtain a prompt text. The determining module is configured to determine a target text matching a semantic of the prompt text from a plurality of preset texts. The preset text is used to indicate an image feature map consistent with the semantic of the preset text. The second obtaining module is configured to obtain a target image feature map indicated by the target text. The generating module is configured to generate a target image corresponding to the prompt text by taking the target image feature map as an initial noise distribution map.

[0021] In a possible implementation, the determining module is configured to obtain a first text embedding vector corresponding to the prompt text, the first text embedding vector is used to represent semantic information of the prompt text, and the second text embedding vector with the highest similarity to the first text embedding vector is determined according to a text embedding vector corresponding to each preset text.

[0022] In a possible implementation, the second obtaining module is configured to determine, from a preset correspondence relationship, an image feature map corresponding to the second text embedding vector as the target image feature map, and the preset correspondence relationship comprises a correspondence relationship between a text embedding vector of each preset text and an image feature map indicated by the preset text.

[0023] In a possible implementation, the generating module is configured to perform multiple times of iterative denoising processing on the target image feature map according to the prompt text by using a preset generation model to obtain the target image, the preset generation model comprises an encoder and a decoder, the encoder comprises N encoding layers connected in sequence, the decoder comprises N decoding layers connected in sequence, N is an integer greater than 1, each encoding layer is connected to a corresponding decoding layer in a skip manner, the encoding layer is configured to perform feature extraction on a first feature map according to the prompt text to output a second feature map, the first feature map is the target image feature map or a feature map output by a previous encoding layer, and the decoding layer is configured to perform feature fusion on the second feature map and a third feature map according to a preset factor corresponding to the second feature map and the third feature map respectively according to the prompt text to output a fourth feature map, the third feature map is a feature map output by a previous decoding layer, and the preset factor is used to perform feature scaling on the corresponding feature map.

[0024] In a possible implementation, in the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution degree of the second feature map to the fused fourth feature map, and the preset factor corresponding to the third feature map is used to enhance the contribution degree of the third feature map to the fused fourth feature map.

[0025] In a possible implementation, a value of the preset factor corresponding to the second feature map is smaller than a value of the preset factor corresponding to the third feature map.

[0026] In a possible implementation, the generation module is configured to determine the number of iterations and the order of the key denoising process in the multiple iterations of the denoising process; for each iteration of the denoising process, if the iteration is the key denoising process, the first text embedding vector corresponding to the input feature map and the prompt text is input into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; the input feature map is the target image feature map or the output feature map of the last iteration of the denoising process; if the iteration is not the key denoising process, the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process are input into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0027] In a third aspect, an electronic device is provided, including: a processor and a memory, the processor being connected with the memory. The memory is configured to store computer execution instructions, and the processor is configured to execute the computer execution instructions stored in the memory, so as to implement any one of the methods provided in the first aspect.

[0028] In a fourth aspect, a chip is provided, including: a processor and an interface circuit; the interface circuit is configured to receive code instructions and transmit the code instructions to the processor; the processor is configured to run the code instructions to execute any one of the methods provided in the first aspect.

[0029] In a fifth aspect, a computer readable storage medium is provided, which stores computer execution instructions. When the computer execution instructions are run on a computer, the computer is caused to execute any one of the methods provided in the first aspect.

[0030] In a sixth aspect, a computer program product is provided, which includes computer execution instructions. When the computer execution instructions are run on a computer, the computer is caused to execute any one of the methods provided in the first aspect.

[0031] The technical effects brought by any one of the implementation manners of the second aspect to the sixth aspect can refer to the technical effects brought by different implementation manners of the first aspect, and will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0032] FIG. 1 is a schematic diagram of a text-to-image generation method based on a diffusion model according to an embodiment of the present application;

[0033] FIG. 2 is a system architecture diagram of an image generation system according to an embodiment of the present application;

[0034] FIG. 3 is a flowchart of an image generation method according to an embodiment of the present application;

[0035] FIG. 4 is a schematic diagram of constructing a noise feature library according to an embodiment of the present application;

[0036] FIG. 5 is a schematic diagram of generating a target image by a second device using a preset generation model according to an embodiment of the present application;

[0037] FIG. 6 is a schematic diagram of a structure of a preset generation model according to an embodiment of the present application;

[0038] FIG. 7 is a schematic diagram of a specific structure of a preset generation model according to an embodiment of the present application;

[0039] FIG. 8 is a schematic diagram of another structure of a preset generation model according to an embodiment of the present application;

[0040] FIG. 9 is a schematic diagram of a specific structure of another preset generation model according to an embodiment of the present application;

[0041] FIG. 10 is a flowchart of generating an image by a second device according to an embodiment of the present application;

[0042] FIG. 11 is a schematic diagram of a multiple iteration denoising process according to an embodiment of the present application;

[0043] FIG. 12 is a schematic diagram of another multiple iteration denoising process according to an embodiment of the present application;

[0044] FIG. 13 is a schematic diagram of another multiple iteration denoising process according to an embodiment of the present application;

[0045] FIG. 14 is a schematic diagram of a structure of an image generation device according to an embodiment of the present application;

[0046] FIG. 15 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0048] In the description of the present application, unless otherwise specified, “ / ” represents a “or” relationship between the associated objects, for example, A / B can represent A or B; “and / or” in the present application is only a description of the associated relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone, where A and B can be singular or plural.

[0049] Also, in the description of the present application, unless otherwise specified, "a plurality of" means two or more than two. "At least one of the following" or the like means any combination of the items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be a single item or a plurality of items.

[0050] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same items or similar items with basically the same function and effect. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. also do not necessarily mean different. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to represent as an example, illustration or explanation. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words "exemplary" or "for example" are intended to present the relevant concept in a specific manner, for ease of understanding.

[0051] In the following, the terms related to the embodiments of the present application are introduced.

[0052] Text-to-image synthesis (T2I) technology: an artificial intelligence (AI) technology that converts text descriptions into image content. Specifically, T2I technology uses deep learning models such as diffusion models, generative adversarial networks (GAN), variational autoencoders (VAE), autoregressive models, etc. to understand and analyze the semantic information in the text description, and then generate images that match the text description.

[0053] Unet model: a model with a convolutional neural network architecture. The model has a symmetrical U-shaped network structure, combining feature extraction (encoding) and feature reconstruction (decoding) processes. The Unet model includes an encoder and a decoder. The encoder is responsible for extracting image features, while the decoder is responsible for fusing these image features. The encoder includes multiple encoding layers, which gradually reduce the spatial resolution of the image to extract high-frequency features (also known as detail features) of the image. The decoder includes multiple decoding layers, which gradually restore the spatial resolution of the image to reconstruct the image features. Each decoding layer is connected to the corresponding encoding layer by a skip connection, and each decoding layer performs feature fusion (usually a concatenation operation) based on the feature maps output by the corresponding encoding layer and its previous decoding layer, combining shallow features (containing background information of the image) with deep features (containing detail information of the image, also known as high-frequency information), thereby improving the quality of the generated image.

[0054] Diffusion model (DM): a generative model based on probability theory. The core idea of the diffusion model is to gradually add noise to the original data (such as images) until the data becomes almost indistinguishable pure noise (called the forward diffusion process), and then, starting from pure noise, remove the noise step by step through a series of inverse operations to generate samples close to the original data distribution (called the reverse diffusion process). In the embodiments of the present application, the diffusion model is used in the context of generating images based on text. Based on the core idea of the diffusion model, noise is iteratively added in the low-dimensional space (also known as the latent space) of the image, and then gradually removed, thereby generating images based on text. In the process of generating images based on text, the Unet model is used to predict noise multiple times, and the predicted noise is gradually removed to generate images corresponding to the text.

[0055] First, the application scenario of the embodiments of the present application is exemplarily introduced.

[0056] In the text-to-image synthesis (T2I) technology, the diffusion model is often used to convert text descriptions into realistic visual images. In this process, based on the core idea of the diffusion model, noise is randomly extracted from a standard Gaussian distribution to initialize the image generation process. Figure 1 is a schematic diagram of generating images based on a diffusion model. As shown in Figure 1, the feature map corresponding to the noise randomly extracted from the standard Gaussian distribution is used as the initial noise distribution map, and the Unet model is used to perform multiple iterations of denoising processing on the initial noise distribution map to obtain an image corresponding to the text.

[0057] However, the initial noise distribution map thus obtained has almost no semantic association with the input text description. This semantic disconnection often leads to a mismatch between the generated image and the text description in terms of content, lacking precision and consistency, thereby affecting the quality of the final generated image.

[0058] Therefore, the embodiment of the present application provides an image generation method, which takes an image feature map associated with the prompt text in semantics as an initial noise distribution map to generate an image corresponding to the prompt text. In this way, the quality of the generated image is improved by improving the relevance between the initial noise distribution map and the prompt text, and the matching degree between the image and the prompt text is improved.

[0059] In some embodiments, the method specifically comprises: first, obtaining the prompt text, and determining a target text matching the semantics of the prompt text from a plurality of preset texts. The preset text is used to indicate an image feature map consistent with the semantics of the preset text. Then, the target image feature map indicated by the target text is obtained. Finally, the target image feature map is taken as the initial noise distribution map to generate a target image corresponding to the prompt text.

[0060] The image generation method provided by the embodiment of the present application finds a target text highly matching the prompt text in semantics from a plurality of preset texts, takes a target image feature map indicated by the target text as an initial noise distribution map, thereby enhancing the relevance between the initial noise distribution map and the prompt text. Further, based on the initial noise distribution map highly associated with the prompt text, there is a more specific guidance in the process of generating the image, which can directly learn and retain the visual information highly relevant to the text description from these feature maps, thereby generating an image that can more accurately reflect the content described by the prompt text, improving the consistency between the image and the prompt text, greatly reducing the deviation between the generated image and the content described by the prompt text, and improving the quality of the generated image.

[0061] Next, the system architecture of the embodiment of the present application is exemplarily introduced.

[0062] The embodiment of the present application provides an image generation system. FIG. 2 is a system architecture diagram of an image generation system provided by the embodiment of the present application. The image generation system comprises a first device and a second device. The first device is configured to obtain a prompt text and determine an initial noise distribution map corresponding to the prompt text. The second device is configured to generate a target image based on the initial noise distribution map and the prompt text.

[0063] In a possible implementation, the first device takes an image feature map associated with the prompt text in semantics as the initial noise distribution map.

[0064] Optionally, the first device determines a target image feature map matching the semantics of the prompt text according to a pre-constructed information set, and takes the target image feature map as the initial noise distribution map.

[0065] As shown in FIG. 2, the information set contains a plurality of first text embedding vectors of preset texts and image feature maps consistent with semantics of the preset texts. The preset texts are defined in advance and have explicit semantics and image directionality, and are used to indicate image feature maps consistent with semantics of the preset texts.

[0066] The implementation of the first device obtaining the target image feature according to the information set will be described in the following embodiments, which will not be described here.

[0067] In a possible implementation, the second device performs multiple times of iterative denoising processing on the initial noise distribution map corresponding to the prompt text by using the preset generation model to obtain a target image corresponding to the prompt text.

[0068] In FIG. 2, for the i th denoising processing, the second device inputs the prompt text and the initial noise map into the preset generation model to obtain a noise feature map of noise in the initial noise map or a feature map obtained by removing the noise feature map from the initial noise map, which is also referred to as a feature map after removing noise (or a feature map after removing noise). The initial noise map is the initial noise distribution map or the feature map output in the last denoising processing. Specifically, in the case where the preset generation model is used to predict a noise feature map of noise in the input feature map, the preset generation model predicts the noise feature map of noise in the initial noise map based on the input prompt text and the initial noise map. The second device removes the noise feature map from the initial noise feature map to obtain an output feature map of the current denoising processing. In the case where the preset generation model is used to remove noise in the input feature map, the preset generation model obtains a feature map after removing the noise feature map (or a feature map after removing noise) based on the prompt text and the initial noise map, and takes the feature map after removing the noise feature map as the output feature map of the current denoising processing. The second device takes the feature map after removing the noise feature map as the feature map of the target image, and generates the target image based on the feature map of the target image.

[0069] The specific structure of the preset generation model will be described in the following embodiments, which will not be described here.

[0070] The first device can be implemented by software or by hardware.

[0071] As an example of the software function, the first device can include code running on a computing instance. The computing instance can be at least one of a physical host, a virtual machine, a container, and the like. Further, the computing instance can be one or more. For example, the first device can include code running on multiple hosts / virtual machines / containers.

[0072] For example, the first device can be deployed in the form of a software package in a computing device where the image generation system is located, or the first device is deployed in a separate computing device. For example, the first device can be deployed in the form of a plug-in in the image generation system.

[0073] As an example of the first device as a hardware functional unit, the first device can include at least one computing device, such as a server or the like. Alternatively, the first device can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0074] The server can be one physical server, or two or more physical servers sharing different responsibilities, which work together to implement the functions of the server.

[0075] For example, the server can be a blade server, a high-density server, a rack server, or a tower server, etc.

[0076] The plurality of computing devices included in the first device can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0077] Similar to the first device, the second device can be implemented by software or by hardware, which will not be repeated here.

[0078] It can be understood that since the video is composed of multiple images, the second device can also generate a target video corresponding to the prompt text according to multiple target images corresponding to the prompt text. The specific implementation manner of generating a target video according to multiple target images is not limited in the embodiments of the present application.

[0079] It should be noted that the embodiments of the present application do not limit the form of the first device, and the following will take the first device and the second device as examples of being implemented by software, and specifically take the first device and the second device as examples of being deployed on one server to explain the system architecture of the image generation system provided by the embodiments of the present application.

[0080] It should be noted that the system architecture and application scenarios described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0081] The following embodiments of the application will be divided into three parts to exemplarily introduce the scheme of the image generation method.

[0082] In the first part, the image generation method provided by the embodiments of the application is introduced in combination with FIG. 3, FIG. 4 and FIG. 5, which aims to introduce the overall working process of generating an image based on a text.

[0083] In the second part, the image generation method provided by the embodiments of the application is introduced in combination with FIG. 6, FIG. 7, FIG. 8 and FIG. 9, which aims to introduce the specific structure of the preset generation model.

[0084] In the third part, the image generation method provided by the embodiments of the application is introduced in combination with FIG. 10, FIG. 11, FIG. 12 and FIG. 13, which aims to introduce the specific process of how to generate a target image by using the preset generation model.

[0085] In some embodiments, the image feature map associated with the prompt text in semantics is taken as an initial noise distribution map, and an image corresponding to the prompt text is generated, so that the association between the initial noise distribution map and the prompt text is improved, and the quality of the generated image is improved.

[0086] FIG. 3 is a flowchart of an image generation method according to an exemplary embodiment, which is executed by an image generation system. Exemplarily, the method includes the following S301-S304.

[0087] S301: A first device acquires a prompt text.

[0088] Optionally, the prompt text can be a descriptive text of various scenes, abstract concepts and the like. Specifically, various elements of an object can be described in the prompt text, including but not limited to the type, attribute, position, action, emotion, atmosphere and the like of the object. The prompt text is an important basis for generating an image, which conveys the content and style that the user hopes the image to exhibit to the model.

[0089] Exemplarily, the type of the object can be a person, an animal, a building and the like, and the attribute of the object can be color, shape, size, material and the like. For example, “a red apple”.

[0090] Exemplarily, the prompt text can also describe the positional relationship of the object in the image, such as “a red apple on the dining table”.

[0091] For example, the prompt text can describe the action or state of the object, thereby increasing the vividness and story of the image, such as "a bird happily singing on a branch".

[0092] S302: The first device determines a target text matching the semantic of the prompt text from a plurality of preset texts.

[0093] The preset text is used to indicate an image feature map consistent with the semantic of the preset text.

[0094] In a possible implementation, the preset text is a text description defined in advance and having explicit semantic and image direction, so that the preset text can be used to indicate an image feature map consistent with the semantic of the preset text. That is, one preset text is associated with one image feature map consistent with the semantic of the preset text. According to the preset text, the image feature map indicated by the preset text can be obtained. Here, "consistent" can be understood as the semantic of the preset text being able to accurately describe part or all of the content of the image corresponding to the image feature map. One preset text can indicate a plurality of image feature maps. One image feature map can also be indicated by a plurality of preset texts.

[0095] In addition, in order to find a target text matching the prompt text input by the user, the plurality of preset texts should cover a wide range of image topics, styles and scenes, so as to meet the needs of different users. For example, the preset texts and the images indicated by the preset texts can be obtained from a computer vision dataset, such as the ImageNet dataset.

[0096] In the embodiments of the present application, the prompt text is matched with the plurality of preset texts based on the text embedding vectors of the texts. First, a first text embedding vector corresponding to the prompt text is obtained. The first text embedding vector is used to represent the semantic information of the prompt text. Then, according to the text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector is determined. The second text embedding vector is used to represent the semantic information of the target text.

[0097] Since the text embedding vector not only considers the literal meaning of the words in the prompt text, but also considers the context meaning of the words in the prompt text and the relationship between the words in the prompt text, it can handle the polysemy and synonym problem in language, and can distinguish the different meanings of the same word in different contexts. The text embedding vector can more accurately capture the semantic information of the text. In this way, the target text found based on the text embedding vector has a higher semantic matching degree with the prompt text, and can accurately find the target text with similar meanings although the literal meanings are different.

[0098] Optionally, the text embedding vector corresponding to the text can be obtained by a text encoder in a contrastive language-image pre-training (CLIP) model.

[0099] Optionally, the similarity between the first text embedding vector and the second text embedding vector can be determined in a Euclidean distance, Manhattan distance, Pearson correlation coefficient, or the like. The application embodiments do not make specific limitations on the calculation method of the similarity between the first text embedding vector and the second text embedding vector.

[0100] S303: The first device obtains a target image feature map indicated by the target text.

[0101] In a possible implementation, the image feature map can be obtained by an encoder in a VAE model.

[0102] The VAE model includes an encoder and a decoder. The encoder in the VAE model is used to extract the features of the image, and the decoder in the VAE model reconstructs the image based on the features extracted by the encoder. The encoder in the VAE model is an encoder obtained after the VAE model is trained. In the training process, the training data is an image in an image data set. The VAE model is trained by reducing the error between the image generated by the decoder in the VAE model and the original image.

[0103] Specifically, the image is input into the encoder in the trained VAE model to obtain the image feature map output by the encoder in the VAE model. Based on this, for example, the image indicated by the target text is input into the VAE encoder to obtain the target image feature map.

[0104] In a possible implementation, the VAE encoder is used to extract the features of the image indicated by each preset text to obtain the corresponding image feature map. Then, the information related to the preset text (such as the text representation, the text embedding vector, or the text itself) and the corresponding image feature map are saved in the preset corresponding relationship. In this way, in S301, the image feature map corresponding to the target text can be determined from the preset corresponding relationship based on the information related to the target text, as the target image feature map.

[0105] As can be seen from the above implementation manner, the preset correspondence relation includes, for example, a correspondence relation between the text embedding vector of each preset text and the image feature map indicated by the preset text. In this way, the target image feature map is directly obtained by querying the preset correspondence relation, and the target image feature map is used to initialize the image generation process, thereby avoiding the process of reextracting features from the image, and thus the processing speed and efficiency are significantly improved. In particular, in the case of processing a large number of similar texts or repeated queries, finding the target image feature map based on the preset correspondence relation has obvious advantages.

[0106] Optionally, an information set is constructed according to the text embedding vectors of the plurality of preset texts and the image feature maps indicated by the preset texts. The data structure form of the information set is not limited in the embodiments of the present application. For example, in a noise feature library, the text embedding vectors and the image feature maps have a preset correspondence relation. The text embedding vector of the prompt text is compared with the text embedding vectors in the noise feature library, and the image feature map corresponding to the text embedding vector with the maximum similarity is taken as the target image feature map.

[0107] FIG. 4 is a schematic diagram of constructing a noise feature library according to an embodiment of the present application. In FIG. 4, the preset texts in the computer vision dataset are input into the pre-constructed CLIP model to obtain the text embedding vectors of the preset texts. The images indicated by the preset texts are input into the pre-constructed VAE encoder to obtain the image feature maps of the images. Based on the text embedding vectors of the preset texts and the corresponding image feature maps, the noise feature library is obtained.

[0108] Optionally, an information set can also be constructed according to a plurality of preset texts and images having a correspondence relation with the preset texts. The prompt text is compared with the preset texts (for example, comparison is made according to the text embedding vectors), to obtain the image corresponding to the prompt text, the image feature map of the image is obtained by feature extraction of the image, and the image feature map of the image is taken as the target image feature map.

[0109] Of course, the text embedding vectors of the preset texts and the images indicated by the preset texts can also be used to construct an information set, the text embedding vectors of the preset texts and the images constitute a correspondence relation, or the preset texts and the image feature maps are used to construct an information set, the preset texts and the image feature maps constitute a correspondence relation, and the image feature map matched with the prompt text is found through the correspondence relation.

[0110] S304: The second device generates a target image corresponding to the prompt text by taking the target image feature map as an initial noise distribution map.

[0111] In a possible implementation, the second device performs multiple times of iterative denoising processing on the initial noise distribution graph corresponding to the prompt text by using a preset generation model to obtain a target image corresponding to the prompt text. For example, the preset generation model is a Unet model.

[0112] Optionally, in each denoising processing, the preset generation model is configured to predict noise. The preset generation model includes an encoder and a decoder. The encoder includes N encoding layers connected in sequence, and the decoder includes N decoding layers connected in sequence. N is an integer greater than 1. Each encoding layer is connected to a corresponding decoding layer in a skip connection manner. In other words, the encoder and the decoder form a U-shaped network structure.

[0113] The encoder is configured to extract image features, and the decoder is configured to fuse the image features.

[0114] In the above S304, as shown in FIG. 5, the second device inputs the initial noise graph and the prompt text into the preset generation model to obtain a noise feature map of noise in the initial noise graph, and then removes the noise feature map of the noise from the initial noise graph to obtain a feature map after removing noise. After multiple iterations, the feature map obtained in the last iteration is taken as a feature map of a target image, and the second device generates a target image corresponding to the prompt text based on the feature map of the target image. The initial noise graph is the initial noise distribution graph or an output feature map obtained in the last denoising processing, i.e., the feature map after removing noise.

[0115] Optionally, the preset generation model is configured to remove noise. In addition to the encoder and the decoder, the preset generation model further includes a denoiser. The encoder and the decoder are configured to obtain a noise feature map of noise in an initial noise graph, and the denoiser is configured to remove the noise feature map from the initial noise graph to obtain a feature map after removing noise. In each denoising processing, the second device inputs the initial noise distribution graph into the preset generation model to directly obtain the feature map after removing noise. After multiple iterations of denoising processing, a target image feature map of a target image is obtained, and the second device generates the target image based on the target image feature map.

[0116] The structure of the preset generation model and the specific process of how to generate the target image by using the preset generation model will be described in the second part and the third part of the following embodiments, and will not be repeated here.

[0117] In the first part of the embodiment of the present application, the target text highly matched with the prompt text in semantics is found from a plurality of preset texts, and the image feature map indicated by the target text is taken as an initial noise distribution map, so as to enhance the relevance between the initial noise distribution map and the prompt text. Further, based on the initial noise distribution map highly relevant to the prompt text, there is a more specific and concrete guidance in the generation of an image, the visual information highly relevant to the text description can be directly learned and retained from the feature map, so as to generate an image that can more accurately reflect the content of the prompt text description, improve the consistency between the image and the prompt text, greatly reduce the deviation between the generated image and the content of the prompt text description, and improve the quality of the generated target image.

[0118] The above is the first part of the embodiment of the present application. In the following, the specific structure of the preset generation model is introduced in combination with FIGS. 6, 7, 8 and 9.

[0119] In some examples, the preset generation model is used to predict a noise feature map of noise in an input feature map.

[0120] In some embodiments, the preset generation model includes an encoder and a decoder. FIG. 6 is a structural schematic diagram of a preset generation model. As shown in FIG. 6, the encoder includes n encoding layers connected in sequence, and the decoder includes m decoding layers connected in sequence, where n and m are integers greater than 1. Each encoding layer is connected to a corresponding decoding layer in a skip connection. In the preset generation model, the input prompt text and the initial noise map sequentially pass through a plurality of encoding layers (encoding layer 1, encoding layer 2, …, encoding layer n) in the encoder and a plurality of decoding layers (decoding layer 1, …, decoding layer m-1, decoding layer m) in the decoder, to obtain the output of the model, i.e., the noise feature map of the noise predicted by the model.

[0121] In the preset generation model, the encoding layer is used to perform feature extraction on the first feature map according to the prompt text, and output a second feature map. The first feature map is the target image feature map or the feature map output by the previous encoding layer.

[0122] In a possible implementation manner, the encoding layer includes but is not limited to a convolution layer, an activation function layer and a pooling layer (also referred to as a down-sampling layer). The convolution layer is used to perform convolution operation on the input feature map, so as to extract local features in the image. The number of convolution layers can be set according to actual conditions, which is not specifically limited here. The activation function layer is used to introduce a nonlinear factor, so that the encoding layer can learn more complex features and improve the generalization ability of the encoding layer. The activation function layer can adopt a rectified linear unit (ReLU) or the like. The pooling layer is used to gradually reduce the size of the feature map and increase the receptive field size.

[0123] The embodiments of the present application do not limit the specific structure of the encoding layer, which can be set according to actual needs.

[0124] In the preset generation model, the decoding layer is configured to perform feature fusion on the second feature map and the third feature map according to the preset factor corresponding to each of the second feature map and the third feature map, according to the prompt text, and output a fourth feature map. The third feature map is the feature map output by the previous decoding layer. The preset factor is used to perform feature scaling on the corresponding feature map.

[0125] In a possible implementation, the decoding layer includes but is not limited to an up-sampling layer, a convolution layer, an activation function (such as a sigmoid activation function, a Tanh activation function, and a ReLU activation function), and the like. The up-sampling layer can perform up-sampling on the input feature map by using a deconvolution operation, an interpolation operation, and the like, thereby increasing the size of the feature map. After up-sampling, one or more convolution layers (such as a 3*3 convolution layer) are set in the decoding layer, thereby further extracting and fusing features. The embodiments of the present application do not limit the specific structure of the decoding layer, which can be set according to actual needs.

[0126] For example, the decoding layer can perform feature fusion on the second feature map and the third feature map according to the preset factor corresponding to each of the second feature map and the third feature map, according to the prompt text, by using a convolution layer, to obtain a fourth feature map. By setting the preset factor in the decoder, the preset generation model performs scaling on feature maps at different levels (such as shallow features from the encoding layer and deep features from the decoding layer) to different degrees in the feature fusion process, fuses feature information from different levels, and helps the model pay more attention to important feature information when generating a target image, thereby improving the quality and accuracy of the generated image. In addition, the preset factor can be adjusted according to the implementation needs, to more finely control the influence of feature information on the generated image, thereby enhancing the flexibility and adaptability of the model.

[0127] FIG. 7 is a schematic diagram of a specific structure of a preset generation model. As shown in FIG. 7, the encoder in the preset generation model has three encoding layers. The encoding layer 1, as the first encoding layer of the encoder, receives the initial noise distribution map and the prompt text, extracts features from the initial noise distribution map, and outputs a feature map (referred to as the second feature map of the encoding layer 1). The encoding layer 2 receives the feature map output by the previous encoding layer, i.e., the encoding layer 1, down-samples the feature map output by the encoding layer 1, obtains a down-sampled feature map, extracts features from the down-sampled feature map, and outputs a feature map (referred to as the second feature map of the encoding layer 2). The encoding layer 3 receives the feature map output by the previous encoding layer, i.e., the encoding layer 2, down-samples the feature map output by the encoding layer 2, extracts features from the down-sampled feature map, and outputs a feature map (referred to as the second feature map of the encoding layer 3). The decoding layer 1, as the first decoding layer of the decoder, receives the feature map output by the encoding layer 3, up-samples the feature map output by the encoding layer 3, fuses the up-sampled feature map and the feature map output by the encoding layer 2 according to the respective preset factors, and outputs a feature map (referred to as the fourth feature map of the decoding layer 1). The decoding layer 2, as the last decoding layer of the decoder, receives the feature map output by the decoding layer 1, up-samples the feature map, fuses the up-sampled feature map and the feature map output by the encoding layer 1 according to the respective preset factors, and outputs a feature map (referred to as the fourth feature map of the decoding layer 2). The feature map output by the decoding layer 2 is the predicted noise feature map.

[0128] In a possible implementation, since the second feature map comes from the encoding layer and the third feature map comes from a deeper decoding layer, that is, the second feature map represents the shallow features of the image and contains the background information and noise of the image, and the third feature map represents the deep features of the image and contains the high-frequency information (also referred to as detail information) in the image, therefore, the contribution of the second feature map to the fused fourth feature map is reduced, and the contribution of the third feature map to the fused fourth feature map is increased, the detail information in the generated image is enhanced, the noise and unnecessary background information are reduced, the quality of the generated image is improved, the denoising capability of the model is enhanced, and the generated image can more accurately describe the content contained in the prompt text.

[0129] Optionally, in the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map. The preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

[0130] Exemplarily, the preset factor corresponding to the second feature map has a smaller value than the preset factor corresponding to the third feature map. For example, the preset factor corresponding to the second feature map is set to a value smaller than 1, and the preset factor corresponding to the third feature map is set to a value larger than 1. In this way, the third feature map is assigned a larger preset factor, i.e., a higher weight, which can ensure that the detailed information of the image represented by the third feature map is retained and emphasized more in the feature fusion process. The second feature map is assigned a smaller preset factor, i.e., a lower weight, which can limit the influence of the shallow features on the target image and reduce noise and unnecessary background information. Through the smaller preset factor of the second feature map and the larger preset factor of the third feature map, the dominance of the deep features can be maintained in the feature fusion process, while the background information of the shallow layer is moderately retained, so that the generated target image more accurately reflects the content in the prompt text, and the user satisfaction is improved.

[0131] Optionally, the preset factor corresponding to the second feature map and the preset factor corresponding to the third feature map can be determined in the process of training the preset generation model.

[0132] Exemplarily, for the nth denoising process, the initial noise map is input into the preset generation model to obtain the feature maps output by each encoding layer and decoding layer in the preset generation model. According to the frequency domain range in each feature map, the preset factor of each feature map is determined.

[0133] For example, the frequency domain range of the second feature map a is 0-2MHz, and the preset frequency domain range corresponding to the second feature map is set to 0-3MHz. Therefore, the preset factor of the second feature map a can be set to 2 / 3. The frequency domain range of the third feature map b is 0-5MHz, and the preset frequency domain range corresponding to the third feature map is set to 3-6MHz. Therefore, the preset factor of the third feature map b can be set to 5 / 3.

[0134] It can be understood that, since the frequency domain range of the second feature map output by each encoding layer in the encoder may differ, the preset factor of the second feature map output by each encoding layer will also vary with the frequency domain range of the second feature map output by each encoding layer. Similarly, since the frequency domain range of the third feature map output by each decoding layer in the decoder may also differ, the preset factor of the third feature map output by each decoding layer will also vary with the frequency domain range of the third feature map output by each decoding layer.

[0135] Of course, for the nth denoising process, the preset factor of the second feature map output by each encoding layer in the preset generation model can be set to the same preset factor, and the preset factor of the third feature map output by each decoding layer can be set to another same preset factor. In this way, the number of parameters that need to be optimized in the model is simplified, the training speed of the model is accelerated, and the computational complexity of the model is reduced.

[0136] In some examples, the preset generation model is used to remove noise features in the input feature map.

[0137] In addition to the encoder and the decoder in the model of FIG. 6, the preset generation model further includes a denoiser. The denoiser is configured to remove the feature map output by the decoder from the feature map input to the encoder.

[0138] FIG. 8 is a structural schematic diagram of another preset generation model. In FIG. 8, the preset generation model includes an encoder, a decoder, and a denoiser. The initial noise map and the prompt text are input into the preset generation model, sequentially pass through the multiple encoding layers (encoding layer 1, encoding layer 2, …, encoding layer n) in the encoder, the multiple decoding layers (decoding layer 1, …, decoding layer m-1, decoding layer m) in the decoder, and the denoiser, to obtain the feature map output by the model, i.e., the feature map after removing the noise.

[0139] FIG. 9 is a structural schematic diagram of another preset generation model. The preset generation model includes an encoder, a decoder, and a denoiser. The encoder in the preset generation model has three encoding layers, i.e., encoding layer 1, encoding layer 2, and encoding layer 3. The decoder in the preset generation model includes two decoding layers, i.e., decoding layer 1 and decoding layer 2. The initial noise map and the prompt text are input into the preset generation model. The feature map input to the encoder (i.e., the initial noise distribution map in FIG. 9) and the feature map output by the decoder (i.e., the feature map output by decoding layer 2) are subjected to difference processing in the denoiser, to remove the feature map output by the decoder from the feature map input to the encoder, and obtain the feature map after removing the noise.

[0140] The embodiments of the present application do not limit the structure of the preset generation model. In the third part of the following embodiments, the preset generation model includes an encoder and a decoder, and how to use the preset generation model to obtain the target image is described.

[0141] In the second part of the embodiments of the present application, by setting the preset factor, the contribution of different feature maps in the fusion process is adjusted, and different levels of feature maps are scaled differently, and the feature information from different levels is fused, which helps the model to pay more attention to important feature information when generating the target image, and improves the quality and accuracy of the generated image. In addition, the preset factor can be adjusted according to the implementation needs, which enhances the flexibility and adaptability of the model.

[0142] The above is the second part of the embodiments of the present application. In the following, how to use the preset generation model to generate the target image is described in detail with reference to FIGS. 10, 11, 12, and 13.

[0143] In the embodiment of the present application, the second device performs multiple times of iterative denoising processing on the initial noise distribution graph corresponding to the prompt text by using the preset generation model to obtain the target image corresponding to the prompt text.

[0144] In the embodiment of the present application, the preset generation model is used to generate the target image corresponding to the prompt text, which includes various implementation manners. The following will be exemplarily described by way of mode 1 to mode 2.

[0145] Mode 1: The multiple times of iterative denoising processing process is divided into a key denoising process and a non-key denoising process.

[0146] In the key denoising process, each encoding layer in the encoder and each decoding layer in the decoder normally work. The second device inputs the input feature map and the prompt text (such as the first text embedding vector corresponding to the prompt text) into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model. The input feature map is the initial noise distribution graph (i.e., the target image feature map) or the feature map output by the second device in the last denoising process (i.e., the feature map output by the preset generation model in the last denoising process). That is, each encoding layer in the encoder is used to perform feature extraction on the first feature map (i.e., the target image feature map or the feature map output by the last encoding layer) according to the prompt text to output a second feature map. Each decoding layer in the decoder is used to perform feature fusion on the second feature map and a third feature map (i.e., the feature map output by the last decoding layer) according to the preset factors corresponding to the second feature map and the third feature map respectively according to the prompt text to output a fourth feature map.

[0147] Taking the preset generation model in FIG. 7 as an example, in the key denoising process, the encoding layer 1 receives the initial noise distribution map and the prompt text, extracts features from the initial noise distribution map, and outputs a feature map (referred to as the second feature map of the encoding layer 1). The encoding layer 2 receives the feature map output by the previous encoding layer, i.e., the encoding layer 1, down-samples the feature map output by the encoding layer 1, extracts features from the down-sampled feature map, and outputs a feature map (referred to as the second feature map of the encoding layer 2). The encoding layer 3 receives the feature map output by the previous encoding layer, i.e., the encoding layer 2, down-samples the feature map output by the encoding layer 2, extracts features from the down-sampled feature map, and outputs a feature map (referred to as the second feature map of the encoding layer 3). The decoding layer 1 receives the feature map output by the encoding layer 3, up-samples the feature map output by the encoding layer 3, and fuses the up-sampled feature map and the feature map output by the encoding layer 2 according to the respective preset factors, to output a feature map (referred to as the fourth feature map of the decoding layer 1). The decoding layer 2, as the last decoding layer of the decoder, receives the feature map output by the decoding layer 1 and up-samples it, fuses the up-sampled feature map and the feature map output by the encoding layer 1 according to the respective preset factors, and outputs a feature map (referred to as the fourth feature map of the decoding layer 2). The feature map output by the decoding layer 2 is the noise feature map of the predicted noise.

[0148] For the non-key denoising process, not all encoding layers and decoding layers in the preset generation model work, but only part of the encoding layers or part of the decoding layers work. In this way, on the one hand, when only part of the encoding layers or decoding layers work, the amount of data and the number of calculation steps that the model needs to process are significantly reduced, thereby greatly reducing the computational complexity and speeding up the processing speed of the model, especially in scenarios that require fast response, such as real-time image generation. On the other hand, the working of part of the encoding layers and decoding layers means that the model occupies less memory resources, which helps to achieve more efficient resource utilization and reduce the consumption of computing resources of the computing device. In addition, by flexibly adjusting the working encoding layers and decoding layers, the model can better adapt to different denoising tasks. For example, natural scene images require more encoding layers to obtain image details than medical images.

[0149] In a possible implementation, in the non-key denoising process, only the last decoding layer in the decoder of the preset generation model works. The second device inputs the output of the first encoding layer and the output of the second-to-last decoding layer of the preset generation model in the last key denoising process into the last decoding layer of the preset generation model in the current non-key denoising process, to obtain the output feature map of the last decoding layer, i.e., the noise feature map of the predicted noise in the current non-key denoising process.

[0150] Optionally, in the non-key denoising process, the second device inputs the output of the first encoding layer in the last key denoising process and the output of the second-to-last decoding layer into the last decoding layer of the preset generation model, and the last decoding layer performs feature fusion on the output of the first encoding layer in the last key denoising process and the output of the second-to-last decoding layer according to respective preset factors to obtain a noise feature map of the predicted noise in the current non-key denoising process.

[0151] Optionally, in the non-key denoising process, the last decoding layer performs up-sampling on the second-to-last decoding layer in the last key denoising process, and performs feature fusion on the up-sampled feature map and the output of the first encoding layer in the last key denoising process according to respective preset factors to obtain a noise feature map of the predicted noise in the current non-key denoising process.

[0152] In a possible implementation, the second device performs multiple times of iterative denoising processing on the target image feature map according to the prompt text by using the preset generation model to obtain the target image. In the multiple times of iterative denoising processing, if it is a key denoising process, the second device predicts a noise feature map of noise in the current key denoising process by using all the encoding layers and decoding layers in the preset generation model, and removes the predicted noise feature map of noise from the input feature map of the preset generation model in the current key denoising process to obtain an output feature map of the current key denoising process. If it is a non-key denoising process, the second device predicts a noise feature map of noise in the current non-key denoising process by using part of the encoding layers or part of the decoding layers in the preset generation model, and removes the predicted noise feature map of noise from the input feature map of the preset generation model in the current non-key denoising process to obtain an output feature map of the current non-key denoising process. After the multiple times of iterative denoising processing, a feature map of the target image is obtained, and thus the target image is generated.

[0153] FIG. 10 is a flowchart of the second device generating an image according to an exemplary embodiment. Exemplarily, the method includes the following S3041-S3043:

[0154] S3041: The second device determines the number of iterations and the order of the key denoising process in the multiple times of iterative denoising processing.

[0155] The number of iterations refers to the number of times of using the preset generation model, and can also be understood as the number of times of denoising.

[0156] For example, the number of iterations and the order of the key denoising processes can be set according to actual needs, which are not limited herein. For example, the number of iterations is 50, and the order of the key denoising processes can be 1, 2, 3, 5, 7, 9, …, and the order of the non-key denoising processes is 4, 6, 8, …, respectively. The order of the key denoising processes can be 1, 2, 3, 6, 9, …, and the order of the non-key denoising processes is 4, 5, 7, 8, …, respectively.

[0157] S3042: The second device performs multiple iteration denoising processing on the target image feature map according to the number of key denoising processes, and obtains the feature map of the target image according to the feature map input to the encoder and the feature map output by the decoder in the last iteration denoising processing.

[0158] In a possible case, for each iteration denoising processing, if it is a key denoising process, the second device inputs the input feature map and the first text embedding vector corresponding to the prompt text into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model (i.e., the noise feature map of the noise predicted by the preset generation model). The input feature map is the target image feature map or the output feature map of the last iteration denoising processing.

[0159] If it is a non-key denoising process, the second device inputs the output of the first encoding layer and the output of the second last decoding layer in the last key denoising process into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0160] FIG. 11 is a schematic diagram of a multi-iteration denoising process. For the multi-iteration denoising process in FIG. 11, the order of the key denoising processes is 1, 2, 3, 5, 7, 9, …, and the order of the non-key denoising processes is 4, 6, 8, …. The third denoising process and the fourth denoising process are shown in FIG. 11. For the third denoising process, the second device inputs the output (feature map a) of the preset generative model decoder in the second denoising process into the first encoding layer (encoding layer 1) of the encoder in the preset generative model, the encoding layer 1 extracts features from the feature map a, outputs the feature map a1 to the encoding layer 2, the encoding layer 2 down-samples and extracts features from the feature map a1, outputs the feature map a2 to the encoding layer 3, the encoding layer 3 down-samples and extracts features from the feature map a2, outputs the feature map a3 to the decoding layer 1, the decoding layer 1 up-samples the feature map a3, and fuses the up-sampled feature map a3 with the feature map a2 according to their respective preset factors, outputs the feature map a4 to the decoding layer 2, the decoding layer 2 up-samples the feature map a4, and fuses the up-sampled feature map a4 with the feature map a1 according to their respective preset factors, outputs the feature map a5. The second device removes the feature map a5 from the feature map a, and obtains the feature map b. For the fourth denoising process, the second device inputs the output (feature map a1) of the first encoding layer (encoding layer 1) in the third denoising process and the output (feature map a4) of the second-to-last decoding layer (decoding layer 1) into the last decoding layer (decoding layer 2) of the preset generative model in the fourth denoising process, the decoding layer 2 up-samples the feature map a4, fuses the feature map a1 with the up-sampled feature map a4 according to their respective preset factors, and obtains the feature map b1. The second device removes the feature map b1 from the feature map b. For the third denoising process, the third denoising process is a key denoising process, and all the decoding layers and encoding layers in the preset generative model work normally. For the fourth denoising process, the fourth denoising process is a non-key denoising process, and all the encoding layers (i.e., encoding layer 1, encoding layer 2, and encoding layer 3) of the encoder and the decoding layer 1 of the decoder in the preset generative model do not work, and only the last decoding layer (i.e., decoding layer 2) of the decoder works.

[0161] FIG. 12 is a schematic diagram of another multi-iteration denoising process. For the multi-iteration denoising process in FIG. 12, the order of the key denoising processes is 1, 2, 3, 6, 9, …, and the order of the non-key denoising processes is 4, 5, 7, 8, … FIG. 12 shows the 3rd denoising process, the 4th denoising process, and the 5th denoising process. The 3rd denoising process in FIG. 12 is the same as the 3rd denoising process in FIG. 11, which is not repeated here. For the 4th denoising process, the second device inputs the input of the first encoding layer (encoding layer 1) in the 3rd denoising process and the output of the second-to-last decoding layer (decoding layer 1) into the last decoding layer (decoding layer 2) of the preset generative model in the 4th denoising process, and the last decoding layer outputs the feature map b1. For the 5th denoising process, the second device inputs the input of the first encoding layer (encoding layer 1) in the 3rd denoising process and the output of the second-to-last decoding layer (decoding layer 1) into the last decoding layer (decoding layer 2) of the preset generative model in the 5th denoising process, and outputs the feature map b2.

[0162] In a possible case, since the non-key denoising process is based on the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process, the output feature map of the current non-key denoising process is obtained. Therefore, the non-key denoising process corresponding to a key denoising process can be processed in parallel (also referred to as adopting a skip strategy), and after the second-to-last decoding layer of the key denoising process outputs the feature map, the second device can input the output of the first encoding layer and the output of the second-to-last decoding layer in the key denoising process into the last decoding layer in the non-key denoising process corresponding to the key denoising process. The non-key denoising process corresponding to the key denoising process refers to the non-key denoising process between the current key denoising process and the next key denoising process. At this time, the last decoding layer of the preset generative model in the key denoising process and the last decoding layer of the preset generative model in the non-key denoising process can be calculated in parallel (also referred to as adopting a skip strategy).

[0163] In this way, since the non-key denoising process can be performed in parallel with part of the key denoising process, the overall processing time is greatly reduced, and when the number of non-key denoising processes is large, the parallel processing can significantly improve the processing efficiency, making it possible to generate images in real time based on text, and further improving the user experience. In addition, by separating the key denoising process and the non-key denoising process, the second device can dynamically adjust the number of parallel processing tasks according to actual needs. For example, in the case that the resources of the computing device where the second device is located are limited, the number of non-key denoising processes can be reduced to save resources; in the case that the resources are sufficient, the number of parallel processing can be increased to improve the processing speed.

[0164] Continuing to take FIG. 12 as an example, for the third denoising process, the fourth denoising process, and the fifth denoising process, when the penultimate decoding layer (decoding layer 1) of the preset generation model outputs the feature map in the third denoising process, the fourth denoising process and the fifth denoising process can start, that is, the fourth denoising process does not need to wait for the last decoding layer (decoding layer 2) of the third denoising process to output the feature map, and the fifth denoising process does not need to wait for the last decoding layer (decoding layer 2) of the fourth denoising process to output the feature map.

[0165] In a possible case, the second device removes the feature map output by the decoder in the last iteration denoising process from the feature map input to the encoder to obtain the feature map of the target image.

[0166] For example, the second device performs difference processing on the feature map input to the encoder in the last iteration denoising process and the feature map output by the decoder to obtain the feature map of the target image.

[0167] It can be understood that, in a case where the preset generation model includes a denoiser, the second device takes the feature map output by the preset generation model in the last iteration denoising process as the feature map of the target image.

[0168] S3043: The second device generates the target image according to the feature map of the target image.

[0169] In a possible case, the second device generates the target image according to the feature map of the target image by using the decoder.

[0170] Tables 1 and 2 are a comparison of results of using the image generation method provided in the embodiments of the present application and not using the image generation method provided in the embodiments of the present application. Table 1 is a comparison of results of the second device generating an image based on text by using a graphics processing unit (GPU) in a computing device. Table 2 is a comparison of results of the second device generating an image based on text by using a neural processing unit (NPU) in a computing device. The number of iterations in Tables 1 and 2 is 50. As can be seen from Tables 1 and 2, parallel processing (also referred to as adopting a skip strategy) of the non-critical denoising process does not reduce the accuracy of the generated image while reducing the latency. By using the image generation method provided in the present application, a preset factor is added and a skip caching strategy is adopted for the non-critical denoising process, which improves the accuracy of the generated image while reducing the latency.

[0171] Table 1

[0172] Table 2

[0173] Way 2: In the multiple iteration denoising process, each denoising process is the key denoising process described above.

[0174] That is, for each denoising process, each encoding layer in the encoder and each decoding layer in the decoder work normally. In the multiple iteration denoising process, the preset generation model is used multiple times without distinction. For example, if the iteration number is 50, all the encoding layers and all the decoding layers in the preset generation model are used 50 times.

[0175] FIG. 13 is a schematic diagram of another multiple iteration denoising process. For the multiple iteration denoising process in FIG. 13, the preset generation model is used multiple times without distinction. FIG. 13 shows the first denoising process and the second denoising process. In the first denoising process, the second device first inputs the target image feature map, i.e., the initial noise distribution map (initial noise map 1 in FIG. 13), into the encoder of the preset generation model, sequentially passes through the encoding layer 1, the encoding layer 2, the encoding layer 3, the decoding layer 1, and the decoding layer 2, outputs the feature map a through the decoder, and then removes the feature map a from the initial noise distribution map (initial noise map 1) to obtain the initial noise map 2. In the second denoising process, the second device first inputs the initial noise map 2 into the encoder of the preset generation model, sequentially passes through the encoding layer 1, the encoding layer 2, the encoding layer 3, the decoding layer 1, and the decoding layer 2, outputs the feature map b through the decoder, and then removes the feature map b from the initial noise map 2 to obtain the initial noise map 3, and so on. If the denoising process is 50 times. In the 50th denoising process, the second device first inputs the initial noise map 50 obtained in the 49th denoising process into the encoder of the preset generation model, outputs the feature map x through the decoder, removes the feature map x from the initial noise map 50, and obtains the feature map of the target image. The second device obtains the target image based on the feature map of the target image.

[0176] In the third part of the embodiments of the present application, the multiple iteration denoising process is divided into key denoising process and non-key denoising process. For the non-key denoising process, only part of the encoding layer or decoding layer works. On the one hand, when only part of the encoding layer or decoding layer works, the amount of data and the calculation steps required by the model are significantly reduced, thereby greatly reducing the computational complexity and accelerating the processing speed of the model, especially in scenarios that require fast response, such as real-time image generation. On the other hand, the working of part of the encoding layer and the decoding layer means that the model occupies less memory resources, which helps to achieve more efficient resource utilization and reduce the consumption of computing resources of the computing device. In addition, by flexibly adjusting the working encoding layer and decoding layer, the model can better adapt to different denoising tasks. For example, natural scene images require more encoding layers to obtain image details than medical images.

[0177] The above mainly introduces the scheme provided by the embodiments of the present application from the perspective of method. In order to realize the above functions, the image generation device includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0178] The embodiments of the present application can divide the functional modules of the image generation device according to the above method, for example, the image generation device can include each functional module corresponding to each functional division, or two or more functions can be integrated into one processing module. The integrated module can be realized in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical functional division. Actual implementation can have another division method.

[0179] Illustratively, FIG. 14 shows a possible structural schematic diagram of an image generation apparatus involved in the above embodiments, which includes a first acquisition module 1401, a determination module 1402, a second acquisition module 1403, and a generation module 1404. The first acquisition module 1401 is configured to acquire prompt text. The determination module 1402 is configured to determine, from a plurality of preset texts, a target text matching the semantic of the prompt text. The preset text is used to indicate an image feature map consistent with the semantic of the preset text. The second acquisition module 1403 is configured to acquire a target image feature map indicated by the target text. The generation module 1404 is configured to generate a target image corresponding to the prompt text by taking the target image feature map as an initial noise distribution map.

[0180] Optionally, the determination module 1402 is specifically configured to acquire a first text embedding vector corresponding to the prompt text, the first text embedding vector is used to represent semantic information of the prompt text, and determine, according to a text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector, the second text embedding vector is used to represent semantic information of the target text.

[0181] Optionally, the second acquisition module 1403 is specifically configured to determine, from a preset correspondence relationship, an image feature map corresponding to the second text embedding vector as the target image feature map, the preset correspondence relationship includes a correspondence relationship between a text embedding vector of each preset text and an image feature map indicated by the preset text.

[0182] Optionally, the generation module 1404 is specifically configured to perform multiple times of iterative denoising processing on the target image feature map according to the prompt text by using a preset generation model to obtain a target image, the preset generation model includes an encoder and a decoder, the encoder includes N encoding layers connected in sequence, the decoder includes N decoding layers connected in sequence, N is an integer greater than 1, each encoding layer is connected to a corresponding decoding layer in a skip connection manner, the encoding layer is configured to perform feature extraction on a first feature map according to the prompt text to output a second feature map, the first feature map is the target image feature map or a feature map output by a previous encoding layer, and the decoding layer is configured to perform feature fusion on the second feature map and a third feature map according to a preset factor corresponding to the second feature map and the third feature map respectively according to the prompt text to output a fourth feature map, the third feature map is a feature map output by a previous decoding layer, and the preset factor is used to perform feature scaling on the corresponding feature map.

[0183] Optionally, in the apparatus, in the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution degree of the second feature map to the fused fourth feature map, and the preset factor corresponding to the third feature map is used to enhance the contribution degree of the third feature map to the fused fourth feature map.

[0184] Optionally, in the device, the second feature map corresponds to a preset factor with a smaller value than a preset factor corresponding to the third feature map.

[0185] Optionally, the generation module 1404 is specifically configured to determine the number of iterations and the order of key denoising processes in the multiple iteration denoising processes; for each iteration denoising process, if it is a key denoising process, input the first text embedding vector corresponding to the input feature map and the prompt text into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein the input feature map is a target image feature map or an output feature map of the last iteration denoising process; if it is not a key denoising process, input the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0186] The specific description of the above optional mode can be referred to the foregoing method embodiments, which will not be described here. In addition, the explanation and beneficial effect of any one of the above provided image generation devices can be referred to the corresponding method embodiments described above, which will not be described here.

[0187] Embodiments of the present application also provide an electronic device 100. As shown in FIG. 15, the electronic device 100 includes a bus 102, a processor 104, a memory 106 and a communication interface 108. The processor 104, the memory 106 and the communication interface 108 communicate through the bus 102. The electronic device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the electronic device 100.

[0188] The bus 102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one line is shown in FIG. 15, but it does not mean that there is only one bus or only one type of bus. The bus 102 can include a path for transmitting information between various components (e.g., the memory 106, the processor 104, the communication interface 108) of the electronic device 100.

[0189] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0190] The memory 106 can include volatile memory, such as random access memory (RAM), among other types of volatile memory. The processor 104 can also include non-volatile memory, such as read-only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD), among other types of non-volatile memory.

[0191] The memory 106 stores executable program code that the processor 104 executes to respectively implement the functions of the aforementioned first obtaining module, the determining module, the second obtaining module, and the generating module, thereby implementing the image generation method. That is, the memory 106 stores instructions for implementing the image generation method.

[0192] The communication interface 108 uses a transceiver module, such as but not limited to a network interface card or a transceiver, to implement communication between the electronic device 100 and other devices or communication networks.

[0193] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium or data storage device that can be accessed by a computing device and includes one or more available media or data storage devices. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape, an optical medium, such as a DVD, or a semiconductor medium, such as a solid state hard drive, among other types of available media. The computer-readable storage medium includes instructions that instruct the computing device to execute the image generation method or instruct the computing device to execute the image generation method.

[0194] The embodiments of the present application further provide a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the electronic device. Optionally, the functions supported by the chip can refer to the above, and will not be described here. Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by programs instructing relevant hardware. The programs can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0195] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it makes the at least one computing device execute the image generation method.

[0196] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided by the embodiments of the present application, such as but not limited to the above-mentioned memory, computer readable storage medium and communication chip, etc., all have non-volatility (non-transitory).

[0197] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more servers, data centers, etc. integrated with the medium. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD), or semiconductor medium (such as solid state disk (SSD)) and the like.

[0198] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims, "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several means recited in the claims. Means plus function claims are understood not to limit the claims with the means plus function to the preferred embodiments described herein but rather the means plus function are considered to be described by the claims themselves.

[0199] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims, "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several means recited in the claims. Means plus function claims are understood not to limit the claims with the means plus function to the preferred embodiments described herein but rather the means plus function are considered to be described by the claims themselves.

[0199] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims, "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several means recited in the claims. Means plus function claims are understood not to limit the claims with the means plus function to the preferred embodiments described herein but rather the means plus function are considered to be described by the claims themselves.

Claims

1. An image generation method characterized by, The method comprises: acquiring prompt text; determining a target text matching the semantic of the prompt text from a plurality of preset texts; the preset text is used to indicate an image feature map consistent with the semantic of the preset text; acquiring a target image feature map indicated by the target text; using the target image feature map as an initial noise distribution map to generate a target image corresponding to the prompt text.

2. The image generation method according to claim 1, characterized by, The method comprises: acquiring a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent semantic information of the prompt text; determining a second text embedding vector with the highest similarity to the first text embedding vector according to a text embedding vector corresponding to each preset text; the second text embedding vector is used to represent semantic information of the target text.

3. The image generation method of claim 2, wherein, The method comprises: determining an image feature map corresponding to the second text embedding vector from a preset correspondence relationship as the target image feature map; the preset correspondence relationship comprises a correspondence relationship between a text embedding vector of each preset text and an image feature map indicated by the preset text.

4. The image generation method according to any one of claims 1 to 3, characterized by, The method comprises: using a preset generation model to perform multiple times of iterative denoising processing on the target image feature map according to the prompt text to obtain the target image; the preset generation model comprises an encoder and a decoder; the encoder comprises N encoding layers connected in sequence; the decoder comprises N decoding layers connected in sequence; N is an integer greater than 1; each encoding layer is connected to a corresponding decoding layer in a skip connection manner; the encoding layer is used to perform feature extraction on a first feature map according to the prompt text to output a second feature map; the first feature map is the target image feature map or a feature map output by a previous encoding layer; the decoding layer is used to perform feature fusion on the second feature map and a third feature map according to a preset factor corresponding to each of the second feature map and the third feature map according to the prompt text to output a fourth feature map; the third feature map is a feature map output by a previous decoding layer; the preset factor is used to perform feature scaling on the corresponding feature map.

5. The image generation method of claim 4, wherein, In the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution degree of the second feature map to the fused fourth feature map; the preset factor corresponding to the third feature map is used to enhance the contribution degree of the third feature map to the fused fourth feature map.

6. The image generation method according to claim 4 or 5, characterized by, The numerical value of the preset factor corresponding to the second feature map is smaller than the preset factor corresponding to the third feature map.

7. The image generation method according to any one of claims 4-6, characterized by, The method comprises: determining the number of iterations and the order of key denoising processes in the multiple times of iterative denoising processing; For each iteration of the denoising process, if it is the key denoising process, input the input feature map and the first text embedding vector corresponding to the prompt text into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein the input feature map is the target image feature map or the output feature map of the last iteration of the denoising process; If it is not the key denoising process, input the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

8. An image generation apparatus characterized by comprising: The device comprises: A first acquisition module for acquiring a prompt text; A determination module for determining a target text matching the semantic information of the prompt text from a plurality of preset texts; the preset text is used to indicate an image feature map matching the semantic information of the preset text; A second acquisition module for acquiring a target image feature map indicated by the target text; A generation module for generating a target image corresponding to the prompt text by taking the target image feature map as an initial noise distribution map.

9. The image generation apparatus according to claim 8, wherein The determination module is configured to acquire a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; determine a second text embedding vector with the highest similarity to the first text embedding vector according to the text embedding vector corresponding to each preset text; the second text embedding vector is used to represent the semantic information of the target text.

10. The image generation apparatus according to claim 9, wherein The second acquisition module is configured to determine an image feature map corresponding to the second text embedding vector from a preset correspondence relationship as the target image feature map; the preset correspondence relationship includes a correspondence relationship between the text embedding vector of each preset text and the image feature map indicated by the preset text.

11. The image generation apparatus according to any one of claims 8-10, wherein, The generation module is configured to perform multiple iterations of denoising processing on the target image feature map according to the prompt text by using a preset generation model to obtain the target image; the preset generation model comprises an encoder and a decoder; the encoder comprises N encoding layers connected in sequence; the decoder comprises N decoding layers connected in sequence; N is an integer greater than 1; each encoding layer is connected to the corresponding decoding layer in a skip connection manner; The encoding layer is configured to perform feature extraction on a first feature map according to the prompt text to output a second feature map; the first feature map is the target image feature map or the feature map output by the last encoding layer; The decoding layer is configured to perform feature fusion on the second feature map and a third feature map according to the prompt text according to a preset factor corresponding to the second feature map and the third feature map respectively to output a fourth feature map; the third feature map is the feature map output by the last decoding layer; the preset factor is used to perform feature scaling on the corresponding feature map.

12. The image generation apparatus according to claim 11, wherein In the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map, and the preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

13. The image generation apparatus according to claim 11 or 12, characterized by The value of the preset factor corresponding to the second feature map is smaller than the preset factor corresponding to the third feature map.

14. The image generation apparatus according to any one of claims 11-13, wherein, The generation module is configured to determine the number of iterations and the order of the key denoising process in the multiple iteration denoising processes. For each iteration denoising process, if it is the key denoising process, the first text embedding vector corresponding to the input feature map and the prompt text is input into the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein the input feature map is the target image feature map or the output feature map of the last iteration denoising process. If it is not the key denoising process, the output of the first encoding layer and the output of the second-to-last decoding layer in the last key denoising process are input into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

15. An electronic device, comprising: Comprising: a processor and a memory; The processor is connected with the memory, and the memory is used for storing computer execution instructions. The processor executes the computer execution instructions stored in the memory, so that the electronic device realizes the method in any one of claims 1-7.

16. A computer readable storage medium characterized by: The computer instructions are stored in the memory, and when the computer instructions are executed on the computing device, the computing device executes the method in any one of claims 1-7.

17. A computer program product, characterised in that, Comprising: The computer execution instructions are stored in the memory, and when the computer execution instructions are executed on the computer, the computer executes the method in any one of claims 1-7.

Images