Image generation method and related device

Abstract

The present application relates to the field of computer technology, and provides an image generation method and related devices for generating image quality, wherein the method comprises: firstly, based on the image description text, respectively extracting the text features corresponding to three types of image attributes, and using the noise image, initializing and generating the initial features corresponding to the three types of image attributes; secondly, using the text features corresponding to the three types of image attributes, respectively extracting the features corresponding to the initial features corresponding to the three types of image attributes, obtaining the intermediate features corresponding to the three types of image attributes, extracting the features related to the image description text through feature extraction, and achieving the effect of denoising; finally, reconstructing the features based on the intermediate features corresponding to the three types of image attributes to generate the target image, thereby reducing the information loss of each type of image attribute and improving the accuracy of the generated image.

Description

Background Art

[0002] Image generation refers to generating a corresponding image based on at least one type of image attribute such as content, style, and texture of the object input.

[0003] In the related art, when three types of image attributes are specified, an initial image is first generated according to a certain type of image attribute, and then each type of image attribute is added to the initial image in sequence, thereby generating a target image that meets the three types of image attributes. However, the addition of image attributes one by one requires multiple image conversions of the initial image, and each image conversion may lose some image details, affecting the accuracy of the generated target image.

[0004] Take the input of target content, target style and target texture as an example. First, perform text-based image operation based on the target content to generate an initial image containing the target content; then, perform style conversion on the initial image to generate an intermediate image corresponding to the target style; finally, perform texture conversion based on the target texture on the basis of the intermediate image to generate the target image.

[0005] Since both style conversion and texture conversion may change the content in the initial image, the intermediate image or target image may lose part or all of the target content. Especially when the target content is more complex and the initial image contains more objects, style conversion and texture conversion will cause the target image to lose details, resulting in inaccurate content of the generated image. Summary of the invention

[0006] The embodiments of the present application provide an image generation method and related devices to improve the accuracy of generated images.

[0007] On the one hand, an embodiment of the present application provides an image generation method, comprising:

[0008] Based on the image description text, for the set first image attribute, second image attribute and third image attribute, respectively extract the corresponding first text feature, second text feature and third text feature;

[0009] Based on the preset noise image, for the first image attribute, the second image attribute and the third image attribute, respectively extracting a corresponding first initial feature, a second initial feature and a third initial feature;

[0010] Based on the first text feature, the second text feature and the third text feature, feature extraction is performed on the first initial feature, the second initial feature and the third initial feature respectively to obtain corresponding first intermediate feature, second intermediate feature and third intermediate feature;

[0011] Based on the first intermediate feature and the second intermediate feature, feature reconstruction is performed on the third intermediate feature to obtain a target image feature, and based on the target image feature, a target image is generated.

[0012] On the one hand, an embodiment of the present application provides an image generating device, including:

[0013] A text extraction unit, for extracting corresponding first text features, second text features and third text features respectively based on the image description text and for the set first image attribute, second image attribute and third image attribute;

[0014] A feature initialization unit, configured to extract, based on a preset noise image, a first initial feature, a second initial feature, and a third initial feature for the first image attribute, the second image attribute, and the third image attribute, respectively;

[0015] a feature extraction unit, configured to extract features from the first initial feature, the second initial feature, and the third initial feature based on the first text feature, the second text feature, and the third text feature, respectively, to obtain corresponding first intermediate features, second intermediate features, and third intermediate features;

[0016] A feature reconstruction unit is used to reconstruct the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain a target image feature, and generate a target image based on the target image feature.

[0017] In a possible implementation, when extracting a feature from the first initial feature based on the first text feature to obtain a first intermediate feature, the feature extraction unit is specifically configured to:

[0018] Downsampling the first initial feature N times, and using the first downsampled feature obtained by the Nth downsampling as the first intermediate feature; wherein, in each downsampling, based on the first downsampled feature obtained by the previous downsampling, combined with the first text feature, the first downsampled feature of the current downsampling is obtained;

[0019] Based on the second text feature, feature extraction is performed on the second initial feature to obtain a second intermediate feature, including:

[0020] The second initial feature is downsampled N times, and the second downsampled feature obtained by the Nth downsampling is used as the second intermediate feature; wherein, in each downsampling, the second downsampled feature of the current downsampling is obtained based on the second downsampled feature and the first downsampled feature obtained by the previous downsampling, combined with the second text feature.

[0021] In a possible implementation, when the second downsampling feature obtained based on the previous downsampling and the first downsampling feature are combined with the second text feature to obtain the second downsampling feature of the current downsampling, the feature extraction unit is specifically used to:

[0022] The second down-sampling feature obtained by the previous down-sampling is fused with the first down-sampling feature to obtain the input data of the current down-sampling;

[0023] Based on the input data of the current downsampling, downsampling processing is performed in combination with the second text feature to obtain the second downsampling feature of the current downsampling.

[0024] In a possible implementation, when the downsampling process is performed based on the input data of the current downsampling in combination with the second text feature to obtain the second downsampling feature of the current downsampling, the feature extraction unit is specifically used to:

[0025] Performing semantic extraction on the input data of the current downsampling to obtain a second semantic feature in the second downsampling feature of the current downsampling, and performing self-attention processing based on the second semantic feature of the current downsampling to obtain a second self-attention feature in the second downsampling feature of the current downsampling;

[0026] Based on the second self-attention feature of the current downsampling, cross-attention processing is performed in combination with the second text feature to obtain the second cross-attention feature in the second downsampling feature of the current downsampling.

[0027] In a possible implementation, when extracting a feature from the third initial feature based on the third text feature to obtain a third intermediate feature, the feature extraction unit is specifically configured to:

[0028] The third initial feature is downsampled N times, and the third downsampled feature obtained by the Nth downsampling is used as the third intermediate feature; wherein, in each downsampling, the third downsampled feature of the current downsampling is obtained based on the third downsampled feature obtained by the previous downsampling, the second downsampled feature obtained by the current downsampling, and the third text feature.

[0029] In a possible implementation, the third down-sampling feature includes a third semantic feature, a third self-attention feature, and a third cross-attention feature in the down-sampling process;

[0030] When the third downsampling feature obtained by the previous downsampling and the second downsampling feature obtained by the current downsampling are obtained by combining the third text feature to obtain the third downsampling feature of the current downsampling, the extraction unit is specifically used to:

[0031] Perform semantic extraction on the third cross-attention feature obtained by the previous downsampling to obtain the third semantic feature of the current downsampling;

[0032] Based on the second self-attention feature obtained by the current downsampling, the self-attention processing is performed in combination with the third semantic feature of the current downsampling to obtain the third self-attention feature of the current downsampling; wherein the second self-attention feature obtained by the current downsampling is generated by using the second downsampling feature obtained by the previous downsampling;

[0033] Based on the third self-attention feature of the current downsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current downsampling.

[0034] In a possible implementation, the second self-attention feature obtained based on the current downsampling is combined with the third semantic feature of the current downsampling to perform self-attention processing to obtain the third self-attention feature of the current downsampling, and the extraction unit is specifically used to:

[0035] Based on the second self-attention feature obtained by the current downsampling, query information, key information and value information corresponding to the second image attribute are obtained; and based on the third semantic feature obtained by the current downsampling, query information, key information and value information corresponding to the third image attribute are obtained;

[0036] The query information, key information and value information corresponding to the second image attribute and the query information, key information and value information corresponding to the third image attribute are combined to obtain new input data, and self-attention calculation is performed on the new input data to obtain the third self-attention feature of the current downsampling.

[0037] In a possible implementation manner, when reconstructing the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain the target image feature, the feature reconstruction unit is specifically used to:

[0038] The first intermediate feature is upsampled N times, and a first upsampled feature obtained by the Nth upsampling is used as a first reconstructed feature; wherein, in each upsampling, the first upsampled feature of the current upsampling is obtained based on the first upsampled feature obtained by the previous upsampling and combined with the first text feature;

[0039] The second intermediate feature is upsampled N times, and the second upsampled feature obtained by the Nth upsampling is used as the second reconstructed feature; wherein, in each upsampling, the second upsampled feature of the current upsampling is obtained based on the second upsampled feature obtained by the previous upsampling and combined with the second text feature;

[0040] The third intermediate feature is upsampled N times, and the third upsampled feature obtained by the Nth upsampling is used as the target image feature; wherein, in each upsampling, the third upsampled feature of the current upsampling is obtained based on the third upsampled feature obtained by the previous upsampling, the first upsampled feature and the second upsampled feature obtained by the current upsampling, combined with the third text feature.

[0041] In a possible implementation, the third upsampling feature includes a third semantic feature, a third self-attention feature, and a third cross-attention feature in the upsampling process;

[0042] When the third upsampling feature of the current upsampling is obtained based on the third upsampling feature obtained in the previous upsampling, the first upsampling feature and the second upsampling feature obtained in the current upsampling, and the third text feature is combined to obtain the third upsampling feature of the current upsampling, the feature reconstruction unit is specifically used to:

[0043] Perform semantic extraction on the third cross-attention feature obtained by the previous upsampling to obtain the third semantic feature of the current upsampling;

[0044] Based on the third semantic feature obtained by the current upsampling, the self-attention processing is performed in combination with the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling;

[0045] Based on the third self-attention feature obtained by the current upsampling and the second self-attention feature in the second upsampling feature obtained by the current upsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current upsampling.

[0046] In a possible implementation, the third semantic feature obtained based on the current upsampling is combined with the first semantic feature in the first upsampling feature obtained by the current upsampling to perform self-attention processing to obtain the third self-attention feature of the current upsampling, and the feature reconstruction unit is specifically used to:

[0047] Based on the third semantic feature obtained by the current upsampling, obtaining query information, key information and value information for self-attention processing corresponding to the third image attribute;

[0048] Based on the query information for self-attention processing corresponding to the third image attribute, combined with the query information corresponding to the first image attribute, new query information is obtained, wherein the query information corresponding to the first image attribute is extracted using the first semantic feature obtained by the current upsampling;

[0049] Based on the key information, value information and the new query information corresponding to the third image attribute for self-attention processing, the third self-attention feature of the current upsampling is obtained.

[0050] In a possible implementation, the third self-attention feature obtained based on the current upsampling, the second self-attention feature in the second upsampling feature obtained by the current upsampling, and the third text feature are combined to perform cross-attention processing to obtain the third cross-attention feature of the current upsampling, and the feature reconstruction unit is specifically used to:

[0051] Based on the third self-attention feature of the current upsampling, obtaining query information for cross-attention processing corresponding to the third image attribute, and based on the third text feature, obtaining key information and value information for cross-attention processing corresponding to the third image attribute;

[0052] Based on the query information for cross attention processing corresponding to the third image attribute, combined with the query information corresponding to the second image attribute, new query information is obtained, wherein the query information corresponding to the second image attribute is extracted using the second semantic feature obtained by the current upsampling;

[0053] Based on the value information for cross attention processing corresponding to the third image attribute, combined with the value information corresponding to the second image attribute, new value information is obtained, wherein the value information corresponding to the second image attribute is obtained according to the second text feature;

[0054] Based on the key information for self-attention processing corresponding to the third image attribute, the new value information and the new query information, a third cross-attention feature of the current upsampling is obtained.

[0055] In a possible implementation, when extracting the first initial feature, the second initial feature, and the third initial feature based on the first text feature, the second text feature, and the third text feature to obtain the corresponding first intermediate feature, the second intermediate feature, and the third intermediate feature, the feature extraction unit is specifically used to:

[0056] Based on the first text feature, using a first generation model, extracting features from the first initial feature to obtain a first intermediate feature, wherein the first generation model is a trained generation model for generating an image containing the first image attribute;

[0057] Based on the second text feature and the first intermediate feature, using a second generation model, extracting features from the second initial feature to obtain a second intermediate feature, wherein the second generation model is a trained generation model for generating an image containing the second image attribute;

[0058] Based on the third text feature and the second intermediate feature, the third initial feature is extracted using the third generation model to obtain a third intermediate feature, wherein the third generation model is a generation model trained to generate an image containing the third image attribute. In one aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0059] In one aspect, a computer-readable storage medium is provided, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any of the above methods.

[0060] On the one hand, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of any of the above methods.

[0061] In an embodiment of the present application, after the text features corresponding to the three types of image attributes are respectively extracted based on the image description text, and the initial features corresponding to the three types of image attributes are initialized and generated using a noise image, the text features corresponding to the three types of image attributes are first used to extract features of the initial features corresponding to the three types of image attributes to obtain intermediate features corresponding to the three types of image attributes. In this way, the intermediate features corresponding to the three types of image attributes related to the image description text can be extracted, and the features irrelevant to the image description text can be eliminated, thereby achieving a denoising effect, and then generating a target image that meets the image description text, and the feature extraction can reduce the feature quantity, thereby improving image generation efficiency.

[0062] Then, feature reconstruction is performed based on the intermediate features corresponding to the three types of image attributes to obtain the target image features, and based on the target image features, the target image is generated. Since feature reconstruction is performed based on the intermediate features extracted from the three types of image attributes, the generated target image can meet the requirements of the three types of image attributes. In addition, compared with multiple image conversions, feature reconstruction will not lose details due to image conversion, reducing the information loss of various image attributes, thereby improving the accuracy of the generated image.

[0063] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0065] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0066] Figure 2 A system architecture diagram provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a process of generating an image provided in an embodiment of the present application;

[0068] Figure 4 A logical schematic diagram of an image generation process provided in an embodiment of the present application;

[0069] Figure 5 A logical schematic diagram of a downsampling process provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of a feature fusion process provided in an embodiment of the present application;

[0071] Fig. 7A A schematic diagram of a downsampling process corresponding to a content provided in an embodiment of the present application;

[0072] Figure 7B A schematic diagram of a downsampling process corresponding to a painting style provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of a feature reconstruction process provided in an embodiment of the present application;

[0074] Fig. 9 This is one provided in the embodiments of the present application;

[0075] Fig. 10A A schematic diagram of an upsampling process corresponding to a content provided in an embodiment of the present application;

[0076] Fig. 10B A schematic diagram of an upsampling process corresponding to a painting style provided in an embodiment of the present application;

[0077] Fig. 10C A schematic diagram of a texture-corresponding upsampling process provided in an embodiment of the present application;

[0078] Fig.11 This is a schematic diagram of the structure of the SD model provided in the embodiments of the present application;

[0079] Fig.12 A schematic diagram of several generated images provided in the embodiments of the present application;

[0080] Fig.13 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0081] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0083] It is understandable that in the following specific implementations of the present application, user data, such as user voice data, may be involved. Therefore, when the various embodiments of the present application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0084] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0085] Stable diffusion: Stable diffusion is a generative model based on the Latent Diffusion Model (LDM) that generates data by gradually removing noise. The purpose of the diffusion model is to remove the Gaussian noise that is continuously applied to the training image, which can be regarded as a series of denoising autoencoders. Stable Diffusion usually includes the following three parts: Variational Auto-Encoder (VAE), U-Net, and text encoder. Compared with learning to denoise image data, in Stable Diffusion, VAE is trained to transform the image into a low-dimensional latent space, and Gaussian noise is added and removed using the latent representation in the low-dimensional latent space, and the final denoised output is decoded into the pixel space to obtain the target image. The process of adding Gaussian noise can also be called forward diffusion. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation. Removing Gaussian noise is achieved by including a residual neural network U-Net, which denoises the image by forward diffusion in the reverse direction to obtain the denoised latent representation. The denoised latent representation is converted back to the pixel space using the VAE decoder to generate the target image.

[0086] U-Net: U-Net is one of the algorithms for semantic segmentation using a fully convolutional network. It is a symmetrical U-shaped structure that includes a compression path and an expansion path. In the embodiment of the present application, the U-Net network is used as the basic backbone model of the stable diffusion model, and a self-attention module and a cross attention module are added to the U-Net network to achieve a noise reduction function. The U-Net network includes a down block and an up block module, wherein the down block constitutes a compression path and the up block constitutes an expansion path. In some embodiments, the U-Net network may also include a down block, a mid block, and an up block, wherein the midblock is a transitional portion between the compression path and the expansion path, and generally includes several convolutional layers for further extracting features and serving as a bridge connecting the compression path and the expansion path. The composition of each block in the down block, mid block, and up block is the same, except that the direction of the feature size dimension is different, thereby distinguishing the three parts of down sampling, mid block, and up sampling.

[0087] Contrastive Language-Image Pre-Training (CLIP) model: The CLIP model is a multimodal deep learning model that jointly trains text and images through contrastive learning, learning how to encode images and text into a unified vector space, so that the model can understand and match text descriptions with image content. CLIP can identify information such as objects, scenes, actions, etc. in images, and can also understand text related to images, such as labels, descriptions, titles, etc.

[0088] Latent variables: In the stable diffusion model, latents represent variables that are not directly observed in the model, that is, the most basic features in the entire model processing process. Latents are very important for explaining the observed data and the behavior of the model. In the stable diffusion model, latents may represent unobserved factors, such as individual characteristics, behaviors, styles, and compositions, which may affect the diffusion process.

[0089] 3D Convolution (3D Convolution, 3D conv): 3D convolution is a technique for performing convolution operations on three-dimensional data. Specifically, by stacking multiple consecutive frames to form a cube, and then applying a 3D convolution kernel in the cube, through this structure, the feature maps in the convolution layer are connected to multiple adjacent frames in the previous layer, thereby capturing motion information. In some embodiments, the weights of the convolution kernels are the same (i.e., parameter sharing) throughout the convolution process, and they are all the same convolution kernel. Therefore, the 3D convolution kernel can only extract one type of feature.

[0090] Attention mechanism: Attention mechanism is a method that uses high-level information to measure the intermediate features of the network, so that the network focuses on some information in the image that assists judgment and ignores irrelevant information. The essence of the attention mechanism comes from the human visual attention mechanism. When people perceive things, they generally do not watch a scene from beginning to end every time, but often observe and pay attention to a specific part according to their needs. When people find that a scene often has something they want to observe in a certain part, they will learn to pay attention to that part when similar scenes appear in the future. Therefore, the attention mechanism is essentially a means of filtering out high-value information from a large amount of information. In a large amount of information, the importance of different information to the result is different. This importance can be reflected by assigning weights of different sizes. In other words, the attention mechanism can be understood as a rule for assigning weights when synthesizing multiple sources. The attention mechanism usually includes a self-attention mechanism and a cross-attention mechanism.

[0091] Self attention: In the stable diffusion model, self attention is an important supplementary part of the block in U-Net. Its input is the backbone feature latent of the stable diffusion model, and its output is the self-attention feature. In the embodiment of the present application, self attention is used to further integrate the semantic representation in the latent, and at the same time can make the representation of the generated object clearer, thereby improving the denoising ability of U-Net for the latent.

[0092] Cross attention: The cross attention mechanism is usually used to process the association between multiple input sequences in order to capture important information interactions between different sequences in the model. In the embodiment of the present application, cross attention is used to help the SD model better understand the association between different parts, so as to more accurately predict the changes and impacts in the diffusion process.

[0093] Residual Network (‌ResNet): ResNet is a deep neural network model that solves the gradient vanishing problem in deep neural networks by introducing residual connections. In Stable Diffusion, the ResNet model can be used as a feature extractor and content processing module to extract high-level features of the target image from the latent input to the block, thereby providing high-quality feature support for the self-attention and cross-attention processing image generation tasks in the subsequent blocks.

[0094] The technical ideas of the embodiments of the present application are briefly described below.

[0095] In related technologies, the target object usually uses text to describe the target image to be generated. The description text contains descriptive words corresponding to the three types of image attributes. In this way, after the image description text is obtained, the target image that meets the requirements can be generated according to the image description text. Specifically, taking the three types of image attributes, namely content, style and texture, as an example, in the process of generating the target image, first, a text-image operation is performed based on the target content to generate an initial image containing the target content; then, the style conversion is performed on the initial image to generate an intermediate image corresponding to the target style; finally, based on the intermediate image, the texture conversion is performed according to the target texture to generate the target image.

[0096] Since both style conversion and texture conversion may change the content in the initial image, the intermediate image or target image may lose part or all of the target content. Especially when the target content is more complex and the initial image contains more objects, style conversion and texture conversion will cause the target image to lose details, resulting in inaccurate content of the generated image.

[0097] In an embodiment of the present application, after the text features corresponding to the three types of image attributes are respectively extracted based on the image description text, and the initial features corresponding to the three types of image attributes are initialized and generated using a noise image, the text features corresponding to the three types of image attributes are first used to extract features of the initial features corresponding to the three types of image attributes to obtain intermediate features corresponding to the three types of image attributes. In this way, the intermediate features corresponding to the three types of image attributes related to the image description text can be extracted, and the features irrelevant to the image description text can be eliminated, thereby achieving a denoising effect, and then generating a target image that meets the image description text, and the feature extraction can reduce the feature quantity, thereby improving image generation efficiency.

[0098] Then, feature reconstruction is performed based on the intermediate features corresponding to the three types of image attributes to obtain the target image features, and based on the target image features, the target image is generated. Since feature reconstruction is performed based on the intermediate features extracted from the three types of image attributes, the generated target image can meet the requirements of the three types of image attributes. In addition, compared with multiple image conversions, feature reconstruction will not lose details due to image conversion, reducing the information loss of various image attributes, thereby improving the accuracy of the generated image.

[0099] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0100] See also Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a terminal device 101 and a server 102 may be included.

[0101] The terminal device 101 may be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart vehicle-mounted device, a smart wearable device, a smart TV, and an aircraft, and any other device that involves image generation requirements. The terminal device 101 may be installed with a target application, and the target application has an image generation function or a function of initiating an image generation request, such as an instant messaging application, a music application, a video application, a short video application, a news application, and a shopping application. The application involved in the embodiment of the present application may be a software client, or a client such as a web page or a small program, and the server 102 is a server corresponding to the software or a web page, a small program, etc., and the specific type of the client is not limited. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited thereto.

[0102] It should be noted that the image generation method in the embodiment of the present application can be executed by the terminal device 101 or the server 102 alone, or can be executed by the server 102 and the terminal device 101 together. When executed by the terminal device 101 or the server 102 alone, the image generation process can be realized by the terminal device 101 or the server 102 alone, for example, the terminal device 101 extracts the text features corresponding to the three types of image attributes based on the image description text, and uses the noise image for feature initialization to obtain the initial features corresponding to the three types of image attributes, and then uses the initial features and text features corresponding to the three types of image attributes to generate the target image. When executed by the server 102 and the terminal device 101 together, the terminal device 101 extracts the text features corresponding to the three types of image attributes based on the image description text, and uses the noise image for feature initialization to obtain the initial features corresponding to the three types of image attributes, and then uses the initial features and text features corresponding to the three types of image attributes to generate the target image. At present, in actual application, specific configuration can be performed according to the situation, and this application is not specifically limited here. The specific image generation process is shown below and will not be repeated here.

[0103] Among them, the server 102 and the terminal device 101 may include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 may also be configured with a database, which may be used to store model parameters of attribute generation models corresponding to various types of image attributes, etc. Among them, the memories of the server 102 and the terminal device 101 may also store program instructions required to be executed in the image generation method provided in the embodiment of the present application, and these program instructions can be used to implement the image generation process provided in the embodiment of the present application when executed by the processor.

[0104] In the embodiment of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiment of the present application. Of course, Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0105] Assume that the three types of image attributes include: a first image attribute, a second image attribute, and a third image attribute.

[0106] In the embodiments of the present application, the image attributes include but are not limited to the content, style, texture, depth of field composition, etc. of the image. Content refers to the information contained in the image, which can be specific objects, scenes, colors, textures, etc., which together constitute the visual expression of the image. Style refers to the painting style of the image, specifically including but not limited to the drawing techniques and composition methods of the image. Texture refers to the object's true feeling of a certain substance, and can also refer to the sense of reality of the object's characteristics expressed by the artwork. It involves the details and layers of the picture, and truly expresses the special texture of the object through different lines, colors, light and shade and corresponding techniques.

[0107] The following only takes the three types of image attributes, namely content, style and texture, as examples for explanation, wherein the first image attribute is content, the second image attribute is style, and the third image attribute is texture.

[0108] The image generation solution provided in the embodiments of the present application can be applied to any scenario where image generation is required. Several possible application scenarios are described below.

[0109] Application scenario 1: Creative assistance tools

[0110] The original artist can select the target generation model from the candidate generation models according to the required content, style, and texture, and then input the image description text containing the required content, style, and texture descriptors. Then, based on the image description text, the target generation model is used to generate the target image containing the required content, style, and texture. Furthermore, the generated target image can be used as the draft of the design creation for further optimization and modification, thereby greatly reducing the time consumption of design creation and improving the creation efficiency.

[0111] Application scenario 2: AI photo or poster generation

[0112] In one possible implementation, the target object selects a target generation model from pre-trained candidate generation models, and then uses the target generation model to generate a target image with content, style, and texture that meets the needs of the target object based on the image description text input by the target object, as a personal image portrait.

[0113] In one possible implementation, a content generation model with the target object's own image as the main content can be used, combined with generation models corresponding to the style and texture selected by the target object from pre-trained candidate generation models, to generate a personal image portrait based on the image description text input by the target object.

[0114] In some implementations, the target object can also select more poster-textured models, thereby transforming the generation of photos into the generation of personal photo posters, improving the user experience and gameplay.

[0115] Application scenario three: game character production

[0116] For the game character to be made, download the content generation model with the game character as the main content, and then use the generation model with different game styles and different textures to generate the game DIY character image. For example, you can generate an image of a game character in a comic style.

[0117] Application scenario 4: Film and television post-production

[0118] Post-producers can provide descriptive text for film and television drama characters. Through the image generation method provided in the embodiment of the present application, a target image that only includes the film and television drama characters can be automatically generated, and the target image can meet specified conditions such as style and texture to generate promotional posters for different promotional requirements.

[0119] See also Figure 2 As shown, the system architecture provided by the embodiment of the present application includes the following three modules:

[0120] (1) Content processing module for processing data related to content image attributes

[0121] The content processing module is used to generate a first reconstruction feature (i.e., content reconstruction feature) corresponding to the content image attribute according to the first initial feature (i.e., content initial feature) and the first text feature (i.e., content text feature) corresponding to the content image attribute. Specifically, the process of generating the content reconstruction feature includes two parts: a compression path and an expansion path, wherein the compression path is used to extract the content initial feature according to the content text feature to obtain the content intermediate feature, and the expansion path is used to reconstruct the content intermediate feature according to the content text feature to obtain the content reconstruction feature. In some implementations, the compression path can be implemented by multiple downsampling, and the expansion path can be implemented by multiple upsampling.

[0122] (2) Style processing module for processing data related to style image attributes

[0123] The painting style processing module is used to generate a second reconstructed feature (i.e., painting style reconstruction feature) corresponding to the painting style image attribute according to the second initial feature (i.e., painting style initial feature) and the second text feature (i.e., painting style text feature) corresponding to the painting style image attribute. Specifically, the process of generating the painting style reconstruction feature includes two parts: a compression path and an expansion path, wherein the compression path is used to extract the painting style initial feature according to the painting style text feature and the related features generated in the process of generating the content intermediate feature to obtain the painting style intermediate feature, and the expansion path is used to reconstruct the painting style intermediate feature according to the painting style text feature to obtain the painting style reconstruction feature. In some implementations, the compression path can be implemented by multiple downsampling, and the expansion path can be implemented by multiple upsampling.

[0124] (3) Texture processing module for processing data related to texture image attributes

[0125] The texture processing module is used to generate a third reconstructed feature (i.e., texture reconstruction feature) corresponding to the texture image attribute according to the third initial feature (i.e., texture initial feature) and the third text feature (i.e., texture text feature) corresponding to the texture image attribute. The texture reconstruction feature is the target image feature. Specifically, the process of generating the texture reconstruction feature includes two parts: a compression path and an expansion path, wherein the compression path is used to extract the texture initial feature according to the texture text feature, using the relevant features generated in the process of generating the content intermediate feature, and the relevant features generated in the process of generating the style intermediate feature, to obtain the texture intermediate feature, and the expansion path is used to reconstruct the texture intermediate feature according to the texture text feature, using the content intermediate feature and the style intermediate feature, to obtain the target image feature. In some implementations, the compression path can be implemented by multiple downsampling, and the expansion path can be implemented by multiple upsampling.

[0126] It should be noted that in the embodiment of the present application, only the information of the first image attribute and the second image attribute is fused into the third image attribute as an example. In actual application, other fusion orders may also be used, for example, the information of the first image attribute and the third image attribute is fused into the second image attribute.

[0127] Below, in combination with the application scenarios and system architecture described above, the method provided by the exemplary embodiment of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. It should be noted that the following method can be executed by the above terminal device or server, or by the terminal device and the server together. Here, the execution of the terminal device or the server is specifically shown as an example.

[0128] See also Figure 3As shown, it is a flow chart of the image generation method provided in the embodiment of the present application. The method can be executed by an electronic design, and the electronic design can be Figure 1 The terminal device or server shown in the figure, the specific implementation process of the method is as follows:

[0129] S301 . Based on the image description text, for the set first image attribute, second image attribute and third image attribute, respectively extract the corresponding first text feature, second text feature and third text feature.

[0130] In the embodiment of the present application, the image description text is used to describe the image attributes contained in the target image to be generated. Specifically, based on the image description text, a first text feature is extracted for the first image attribute, a second text feature is extracted for the second image attribute, and a third text feature is extracted for the third image attribute.

[0131] In a possible implementation, the image description text includes description texts of the three types of image attributes, and the description text corresponding to the first image attribute, the description text corresponding to the second image attribute, and the description text corresponding to the third image attribute are extracted from the image description text. Then, based on the description text corresponding to the first image attribute, a first text feature corresponding to the first image attribute is obtained, based on the description text corresponding to the second image attribute, a second text feature corresponding to the second image attribute is obtained, and based on the description text corresponding to the third image attribute, a third text feature corresponding to the third image attribute is obtained. The description text of each type of image attribute can be composed of one or more keywords, which is not limited.

[0132] In a possible implementation, the image description text can be segmented and divided into keyword sets for three types of image attributes. For each type of image attribute, the pre-trained text encoder is used to convert each keyword in the keyword set to the corresponding feature embedding space to obtain the embedding feature representation corresponding to each keyword, and the text encoder is used to encode the combination of the embedding feature representation corresponding to each keyword to obtain the text feature corresponding to the image attribute.

[0133] Exemplarily, the text encoder can be implemented using, but not limited to, the CLIP model. The CLIP model can encode images and text into a unified vector space, thereby understanding the relationship between text and image in terms of language and vision. Using the CLIP model, it is possible to encode the input text and obtain text features that accurately represent the semantic information of the input text.

[0134] In the following, the three types of image attributes, namely content, style and texture, are still used as examples for explanation. For the convenience of description, the first text feature corresponding to the first image attribute can also be called the content text feature, the second text feature corresponding to the second image attribute can also be called the style text feature, and the third text feature corresponding to the third image attribute can also be called the texture text feature.

[0135] For example, in the image description text, the description text of the content is "car", the description text corresponding to the style is "wood", and the description text corresponding to the texture is "Western Wilderness Movie", that is, the content of the target image to be generated needs to contain a car, the style of the target image is wood, and the texture of the target image is a Western Wilderness Movie. Based on the image description text, the CLIP network is used to extract content text features, style text features, and texture text features. The content text features are text features that characterize "car", the style text features are text features that characterize "car", and the texture text features are text features that characterize "Western Wilderness Movie".

[0136] S302: Based on a preset noise image, extract corresponding first initial features, second initial features, and third initial features for the first image attribute, the second image attribute, and the third image attribute, respectively.

[0137] The noise image can be randomly generated using Gaussian noise. Exemplarily, the noise image can be generated in the following ways, but not limited to: first, define a Gaussian noise distribution, which is a normal distribution with a mean of 0 and a variance of a fixed value (such as 1); second, adopt the defined Gaussian noise distribution to generate a noise image with the same shape as the image data. This noise image can be regarded as a noisy image. In the subsequent steps of the diffusion process, the noise will be gradually removed from this noise image to generate the target image.

[0138] In the embodiment of the present application, based on the noise image, a corresponding first initial feature is extracted for the first image attribute, a corresponding second initial feature is extracted for the second image attribute, and a corresponding third initial feature is extracted for the third image attribute.

[0139] For the sake of ease of description, in the following, the first initial feature corresponding to the first image attribute may also be called the content initial feature, the second initial feature corresponding to the second image attribute may also be called the style initial feature, and the third initial feature corresponding to the third image attribute may also be called the texture initial feature.

[0140] In a possible implementation, a VAE decoder may be used but is not limited to extract initial features.

[0141] It should be noted that in the embodiment of the present application, both the initial features and the target image features are latent.

[0142] S303: Based on the first text feature, the second text feature and the third text feature, feature extraction is performed on the first initial feature, the second initial feature and the third initial feature respectively to obtain corresponding first intermediate feature, second intermediate feature and third intermediate feature.

[0143] S303 involves the generation process of the first intermediate feature, the second intermediate feature and the third intermediate feature, wherein the first intermediate feature is the intermediate feature corresponding to the first image attribute, the second intermediate feature is the intermediate feature corresponding to the second image attribute, and the third intermediate feature is the intermediate feature corresponding to the third image attribute. For the three types of image attributes, namely content, style and texture, the first intermediate feature can also be called the content intermediate feature, the second intermediate feature can also be called the style intermediate feature, and the third intermediate feature can also be called the texture intermediate feature. The following still takes the generation process of the three types of image attribute intermediate features, namely content, style and texture, as an example for explanation.

[0144] In one possible implementation, feature extraction can be implemented by downsampling. Downsampling can be implemented through operations such as convolution and pooling, thereby extracting key features in the image and reducing the feature dimension. Downsampling can reduce the size of the feature map, reduce computing pressure and storage pressure, prevent overfitting, increase robustness to small perturbations of the image, and facilitate the fusion of multi-size features to improve the accuracy of the generated image. The feature extraction process can include N downsamplings, and exemplarily, the value of N can be 4.

[0145] See also Figure 4 As shown, it is a logical schematic diagram of a data processing process provided in an embodiment of the present application. The image generation process may include the fusion of three steps: step ①, step ② and step ③. Among them, step ① is the fusion of the characteristic transfer from content to style, and step ① is implemented in the downsampling process. Step ② is to fuse the style information after fusion of content characteristics into the texture information, and step ② is implemented in the downsampling process. Step ① and step ② are two-by-two fusions between two types of image attributes, while step ③ is the fusion process between three types of image attributes, and step ③ is implemented in the upsampling process. Through steps ①, step ② and step ③, the characteristics of the three types of image attributes are adaptively fused and generated. In the embodiment of the present application, S303 involves steps ① and step ②, and S304 involves step ③.

[0146] For the three types of image attributes, the corresponding generative models are used for data processing. The unet network of each generative model includes a downsampling part on the left and an upsampling part on the right. The downsampling part consists of N downsampling units (downblocks), and each down block is used to implement one downsampling. The upsampling part consists of N upsampling units (upblocks), and each up block is used to implement one upsampling.

[0147] Step ① is to pass the content information extracted from the content generation model to the style model. In the embodiment of the present application, the fusion calculation of step ① is performed in the down block. In a possible implementation, there are multiple down blocks distributed in the downsampling part of the entire unet network, and the fusion calculation of step ① can be performed in all the down blocks, but it is not limited to this.

[0148] In the process of image generation by the SD model, the down sample process is mainly used to generate the approximate content and picture structure of the entire picture image, while in the embodiment of the present application, the content generation model is used to control the main content elements in the final generated image. Therefore, when the content information is integrated in step ①, information is transmitted in the down sample process.

[0149] Step ① passes the content information in the content generation model to the style generation model, so that the style generation model can perceive the main content information being generated in the current task. After receiving the content information, the style generation model performs calculations. In a physical sense, it integrates the content object passed in step ① into the style characteristics of the style generation model itself.

[0150] However, considering that this content information has not yet been passed to the texture generation model of the final generated image, the information fusion transmission of step ② is designed in the embodiment of the present application. The physical meaning of step ② is to fuse the content information transmitted from step ① with the style information, and then pass this stylized content information to the texture generation model. In this way, not only can the content information be passed to the final generation model to ensure the accuracy of the content in the final generated image, but also because the style information has been rendered once, the final generated image can be more harmonious and accurate.

[0151] Next, the generation process of content intermediate features, style intermediate features, and texture intermediate features will be explained respectively.

[0152] In a possible implementation, the intermediate content features may be obtained in the following ways, but not limited to:

[0153] The initial content features are downsampled N times, and the content downsampled features obtained by the Nth downsampling are used as the intermediate content features; wherein, in each downsampling, the content downsampled features of the current downsampling are obtained based on the content downsampled features obtained by the previous downsampling and combined with the content text features. In this article, the content downsampled features may also be referred to as the first downsampled features.

[0154] Wherein, N represents the number of downsampling times, and illustratively, the value of N is 4.

[0155] Downsampling is used to compress the image representation into a low-resolution image. In the N downsamplings, the size of the content downsampling features obtained each time is different. It should be noted that in the embodiment of the present application, the relevant features generated in the process of downsampling the initial features of the content, such as content semantic features, content self-attention features, and content cross-attention features, can all be referred to as content downsampling features.

[0156] In one possible implementation, the downsampling process can be implemented using an encoder (ie, downsample) in a U-Net network.

[0157] In one possible implementation, during each downsampling process, ResNet can be used for feature extraction and content processing to extract high-level features of the target image to be generated. In addition, one or more of the self-attention and cross-attention mechanisms can be combined to further obtain high-quality features for image generation. The following only uses ResNet, self-attention, and cross-attention for downsampling as an example.

[0158] The following description will be made by taking the i-th downsampling as an example. The i-th downsampling may be any downsampling among the N downsamplings.

[0159] First, semantic extraction is performed on the content cross-attention features obtained by the i-1th downsampling to obtain the content semantic features of the i-th downsampling;

[0160] Secondly, self-attention processing is performed on the semantic features of the content sampled for the i-th time to obtain the self-attention features of the content sampled for the i-th time;

[0161] Finally, based on the content text features, the i-th downsampled content self-attention features are cross-attention processed to obtain the i-th downsampled content cross-attention features.

[0162] Among them, during the first downsampling, semantic extraction is performed on the initial content features. Content semantic features, content self-attention features, and content cross-attention features are all content downsampling features.

[0163] Self-attention processing refers to a processing process based on the self-attention mechanism. The attention mechanism is essentially a process of performing attention calculations within a feature. In the embodiment of the present application, attention calculations are performed on content semantic features. Through self-attention processing, important information in content semantic features can be further strengthened, while non-important information in content semantic features can be weakened.

[0164] Cross-attention processing is a processing process based on the cross-attention mechanism. The cross-attention mechanism is essentially a process of calculating the attention of a certain position in a sequence with all positions in another sequence. In the embodiment of the present application, the above-mentioned "one sequence" may refer to text features, and the "another sequence" may refer to content self-attention features. Through cross-attention, the information related to the target content in the content self-attention features can be strengthened, and the information unrelated to the target content in the content self-attention features can be weakened.

[0165] See also Figure 5 As shown, it is a logical schematic diagram of the information processing process in downsampling provided in an embodiment of the present application. Figure 5 In the example, each down block is used to implement one down sampling. Each down block specifically includes a residual module, a self-attention module, and a cross-attention module. The residual module can be implemented using a residual network (resnet), the self-attention module is implemented using a self-attention mechanism, and the cross-attention module is implemented using a cross-attention mechanism.

[0166] With respect to the content, after obtaining the initial content features and the content text features (not shown in the figure), the initial content features are downsampled four times.

[0167] During the first downsampling process, semantic extraction is performed on the initial features of the content to obtain the semantic features of the content of the first downsampling, and self-attention processing is performed on the semantic features of the content of the first downsampling to obtain the content self-attention features of the first downsampling. Based on the text features of the content (i.e., the text features corresponding to the description word "car"), cross-attention processing is performed on the content self-attention features of the first downsampling to obtain the content cross-attention features of the first downsampling.

[0168] During the second downsampling process, semantic extraction is performed on the content cross-attention features of the first downsampling to obtain the content semantic features of the second downsampling, and self-attention processing is performed on the content semantic features of the second downsampling to obtain the content self-attention features of the second downsampling. Moreover, based on the content text features, cross-attention processing is performed on the content self-attention features of the second downsampling to obtain the content cross-attention features of the second downsampling.

[0169] During the third downsampling process, semantic extraction is performed on the content cross-attention features of the second downsampling to obtain the content semantic features of the third downsampling, and self-attention processing is performed on the content semantic features of the third downsampling to obtain the content self-attention features of the third downsampling. Based on the content text features, cross-attention processing is performed on the content self-attention features of the third downsampling to obtain the content cross-attention features of the third downsampling.

[0170] During the 4th downsampling process, semantic extraction is performed on the content cross-attention features of the 3rd downsampling to obtain the content semantic features of the 4th downsampling, and self-attention processing is performed on the content semantic features of the 4th downsampling to obtain the content self-attention features of the 4th downsampling. Based on the content text features, cross-attention processing is performed on the content self-attention features of the 4th downsampling to obtain the content cross-attention features of the 4th downsampling. The content cross-attention features of the 4th downsampling are the content intermediate features.

[0171] The resnet module can transform the latent into higher-dimensional features, thereby abstracting the latent into more advanced semantic features. The self-attention module can integrate the model's own characteristics with the latent. Taking the style generation model as an example, the style information of the style generation model itself can be integrated into the input latent. In addition to receiving the latent features calculated by self-attention, the crossattention module also accepts input text features, so that the model adjusts the generation direction so that the generated image conforms to the image description text, thereby improving the image accuracy.

[0172] In a possible implementation, the intermediate features of the painting style may be obtained by, but not limited to, the following methods:

[0173] The initial style feature is downsampled N times, and the style downsampled feature obtained by the Nth downsampling is used as the intermediate style feature; wherein, in each downsampling, the style downsampled feature and the content downsampled feature obtained by the previous downsampling are combined with the style text feature to obtain the style downsampled feature of the current downsampling. In this article, the style downsampled feature can also be called the second downsampled feature.

[0174] It should be noted that in the embodiment of the present application, for the three types of initial features, the sizes of the downsampled features remain consistent during the corresponding number of downsampling processes, that is, the feature sizes of the three types of downsampled features obtained after the first downsampling are the same, the feature sizes of the three types of downsampled features obtained after the second downsampling are the same, the feature sizes of the three types of downsampled features obtained after the third downsampling are the same, and the feature sizes of the three types of downsampled features obtained after the fourth downsampling are the same. In this article, the relevant features generated in the process of downsampling the initial features of the painting style are referred to as the painting style downsampled features.

[0175] Still taking the i-th downsampling as an example, the style downsampling features of the i-th downsampling are obtained through the following steps:

[0176] Step A1: fuse the style downsampling features obtained by the i-1th downsampling and the content downsampling features obtained by the i-1th downsampling to obtain input data for the i-th downsampling.

[0177] Among them, during the first downsampling, the initial style features are used as the style downsampling features obtained by the 0th downsampling.

[0178] In the embodiment of the present application, after one downsampling, the feature sizes of the downsampled features of the three image attributes are the same. Assuming that the size dimension of the content downsampled feature obtained by the i-1th downsampling is width-height-channel, then the size dimension of the style downsampled feature obtained by the i-1th downsampling is also width-height-channel.

[0179] Since the channel dimension of the input feature of ResNet is fixed each time it is downsampled, in the embodiment of the present application, the two latents can be added at corresponding points to merge the content information and the style information, and the size dimension of width-height-channel is kept unchanged and input into ResNet.

[0180] In one possible implementation, the elements of the corresponding positions of the two latents can be added together. This operation can not only add two different latent variables to obtain a new latent variable that contains part of the information of the two original variables, but also can enhance or modify certain features during the generation process by adding a latent variable carrying specific features to another latent variable.

[0181] For example, see Figure 6As shown, it is a logical diagram of a fusion processing process provided in an embodiment of the present application. During the fusion processing, the size dimension of the content downsampling feature obtained by the i-1th downsampling is width (width)-height (height)-channel (channel), and the size dimension of the style downsampling feature obtained by the i-1th downsampling is also width-height-channel. The feature elements at the same position in the content downsampling feature obtained by the i-1th downsampling and the style downsampling feature obtained by the i-1th downsampling are added to generate the input data of the i-th downsampling, and the size dimension of the generated input data of the i-th downsampling is width-height-channel.

[0182] Step A2: based on the input data of the i-th downsampling, combined with the painting style text features, downsampling processing is performed to obtain the painting style downsampling features of the i-th downsampling.

[0183] One possible implementation method is to use ResNet for semantic extraction during each downsampling process to extract high-level features of the target image to be generated. In addition, one or more of the self-attention and cross-attention mechanisms can be combined to further obtain high-quality features for image generation. The following only uses ResNet, self-attention, and cross-attention for downsampling as an example.

[0184] Specifically, step A2 includes the following steps:

[0185] Step A2-1: perform semantic extraction on the input data of the i-th downsampling to obtain the semantic features of the painting style of the i-th downsampling.

[0186] Among them, the semantic extraction of the input data of the i-th downsampling can be achieved by using ResNet.

[0187] Step A2-2: Perform self-attention processing based on the i-th down-sampled style semantic features to obtain the i-th down-sampled style self-attention features.

[0188] The self-attention processing of the semantic features of the style of the i-th down-sampled painting can be implemented using the self-attention mechanism.

[0189] See also Fig. 7A As shown, Fig. 7AA logical diagram of a self-attention processing process for content provided in an embodiment of the present application. First, the initial query information (Q), key information (K), and value information (V) are extracted from the i-th down-sampled style semantic features, and then self-attention calculation is performed on the initial Q, K, and V to obtain the calculated Q, K, and V, which are marked with dotted boxes. Since the style semantic features are fused with content information, Q, K, and V containing content and style fusion information can be obtained after self-attention calculation. Furthermore, Q, K, and V calculated by self-attention are passed to the texture generation model for calculation, which can fully integrate the content information, style information, and quality information, thereby ensuring that the generated image contains accurate content and style.

[0190] Step A2-3: Based on the self-attention features of the i-th downsampling, cross-attention processing is performed in combination with the painting style and text features to obtain the painting style cross-attention features of the i-th downsampling.

[0191] In the embodiment of the present application, the painting style semantic feature, the painting style self-attention feature, and the painting style cross-attention feature all belong to the painting style downsampling features.

[0192] Still reading Figure 5 As shown, after obtaining the initial features of the painting style and the painting style text features, the initial features of the painting style are downsampled 4 times.

[0193] During the first downsampling process, first, the initial content features and the initial style features are fused to obtain the input data of the first downsampling; then, semantic extraction is performed on the input data of the first downsampling to obtain the semantic features of the style of the first downsampling; self-attention processing is performed based on the semantic features of the style of the first downsampling to obtain the self-attention features of the style of the first downsampling; finally, cross-attention processing is performed based on the self-attention features of the first downsampling in combination with the text features of the style (i.e., the text features corresponding to the description word "wood") to obtain the cross-attention features of the style of the first downsampling.

[0194] In the second downsampling process, first, the content cross-attention features of the first downsampling and the style cross-attention features of the first downsampling are fused to obtain the input data of the second downsampling; then, semantic extraction is performed on the input data of the second downsampling to obtain the style semantic features of the second downsampling; self-attention processing is performed based on the style semantic features of the second downsampling to obtain the style self-attention features of the second downsampling; finally, cross-attention processing is performed based on the self-attention features of the second downsampling combined with the style text features to obtain the style cross-attention features of the second downsampling.

[0195] In the third downsampling process, first, the content cross-attention features of the second downsampling and the style cross-attention features of the second downsampling are fused to obtain the input data of the third downsampling; then, semantic extraction is performed on the input data of the third downsampling to obtain the style semantic features of the third downsampling; self-attention processing is performed based on the style semantic features of the third downsampling to obtain the style self-attention features of the third downsampling; finally, cross-attention processing is performed based on the self-attention features of the third downsampling combined with the style text features to obtain the style cross-attention features of the third downsampling.

[0196] In the process of the 4th downsampling, first, the content cross-attention features of the 3rd downsampling and the style cross-attention features of the 2nd downsampling are fused to obtain the input data of the 4th downsampling; then, semantic extraction is performed on the input data of the 4th downsampling to obtain the semantic features of the style of the 4th downsampling; self-attention processing is performed based on the semantic features of the style of the 4th downsampling to obtain the self-attention features of the style of the 4th downsampling; finally, cross-attention processing is performed based on the self-attention features of the 4th downsampling combined with the style text features to obtain the cross-attention features of the style of the 4th downsampling, and the cross-attention features of the style of the 4th downsampling are the intermediate features of the style.

[0197] In a possible implementation, the texture intermediate feature can be obtained by, but not limited to, the following methods:

[0198] The initial texture feature is downsampled N times, and the texture downsampled feature obtained from the Nth downsampling is used as the intermediate texture feature; wherein, in each downsampling, the texture downsampled feature of the current downsampling is obtained based on the texture downsampled feature obtained from the previous downsampling, the painting style downsampled feature obtained from the current downsampling, and the texture text feature.

[0199] In this article, the texture downsampling feature can also be called the third downsampling feature. The related features generated in the process of downsampling the initial texture features, such as texture semantic features, texture self-attention, painting style cross-attention features, etc., can all be called texture downsampling features.

[0200] Still taking the i-th downsampling as an example, the processing process of the i-th downsampling is as follows:

[0201] Step B1: perform semantic extraction on the texture cross-attention features obtained by the i-1th downsampling to obtain the texture semantic features of the i-th downsampling.

[0202] During the first downsampling, the initial texture features are used as the texture downsampling features obtained by downsampling at the 0th time. The semantic extraction of texture cross-attention features can be implemented using ResNet.

[0203] Step B2: Based on the painting style self-cross attention feature obtained by the i-th downsampling, self-attention processing is performed in combination with the texture semantic feature of the i-th downsampling to obtain the texture self-attention feature of the i-th downsampling.

[0204] Among them, the style cross-attention feature obtained by the i-th downsampling is generated using the style downsampling feature obtained by the i-1-th downsampling.

[0205] In one possible implementation, based on the self-cross-attention features of the style obtained by the i-th downsampling, the query information, key information and value information corresponding to the style can be obtained, and based on the semantic features of the texture obtained by the i-th downsampling, the query information, key information and value information corresponding to the texture can be obtained. Then, the query information, key information and value information corresponding to the style and the query information, key information and value information corresponding to the texture are combined to obtain new input data, and self-attention calculation is performed on the new input data to obtain the texture self-attention features of the i-th downsampling.

[0206] See also Figure 7B As shown, Figure 7B A logical schematic diagram of a self-attention processing process for texture provided in an embodiment of the present application. Figure 7BThe dotted box represents the self-cross attention feature of the style obtained by the i-th downsampling. The self-cross attention feature of the style includes Q, K, and V. From the texture semantic features of the i-th downsampling, Q', K', and V' corresponding to the texture can be extracted. Then, Q', K', and V' are concatenated (concat) with Q, K, and V on the channel to obtain new input data. Then, the self-attention calculation is performed on the new input data to obtain the texture self-attention feature of the i-th downsampling. By passing in Q, K, and V, the style information integrated with the content information is transmitted, thereby ensuring the integration of content information and style information and guaranteeing image quality.

[0207] In one possible implementation, concat can be used to pass all QKV into the texture generation model, which can ensure the integrity and accuracy of content information and style information to a great extent. Since step ② is calculated in downsample, and the texture improvement is often in upsample, the fusion here only needs to ensure the existence of quality characteristics, and the texture details can be improved in step ③.

[0208] Step B3: Based on the texture self-attention feature of the i-th downsampling, cross-attention processing is performed in combination with the texture text feature to obtain the texture cross-attention feature of the i-th downsampling.

[0209] Still reading Figure 5 As shown, after obtaining the texture initial features and texture text features, the texture initial features are downsampled 4 times.

[0210] During the first downsampling process, semantic extraction is performed on the initial texture features to obtain the texture semantic features of the first downsampling. Then, based on the painting style self-cross-attention features obtained by the first downsampling, self-attention processing is performed in combination with the texture semantic features of the first downsampling to obtain the texture self-attention features of the first downsampling. Finally, based on the texture self-attention features of the first downsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the first downsampling.

[0211] During the second downsampling process, semantic extraction is performed on the texture downsampling features of the first downsampling to obtain the texture semantic features of the second downsampling. Then, based on the painting style self-cross-attention features obtained by the second downsampling, self-attention processing is performed in combination with the texture semantic features of the second downsampling to obtain the texture self-attention features of the second downsampling. Finally, based on the texture self-attention features of the second downsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the second downsampling.

[0212] During the third downsampling process, semantic extraction is performed on the texture downsampling features of the second downsampling to obtain the texture semantic features of the third downsampling. Then, based on the painting style self-cross-attention features obtained by the third downsampling, self-attention processing is performed in combination with the texture semantic features of the third downsampling to obtain the texture self-attention features of the second downsampling. Finally, based on the texture self-attention features of the third downsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the third downsampling.

[0213] During the fourth downsampling process, semantic extraction is performed on the texture downsampling features of the third downsampling to obtain the texture semantic features of the fourth downsampling. Then, based on the painting style self-cross-attention features obtained by the fourth downsampling, self-attention processing is performed in combination with the texture semantic features of the fourth downsampling to obtain the texture self-attention features of the fourth downsampling. Finally, based on the texture self-attention features of the fourth downsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the fourth downsampling. The texture cross-attention features of the fourth downsampling are the texture intermediate features.

[0214] S304: Based on the first intermediate feature and the second intermediate feature, reconstruct the third intermediate feature to obtain a target image feature.

[0215] In a possible implementation, feature reconstruction can be implemented by upsampling. Upsampling can enlarge the size of the feature map and restore the details in the map. The feature reconstruction process can include N upsamplings.

[0216] Still reading Figure 4 As shown in the figure, the process of reconstructing the third intermediate feature based on the first intermediate feature and the second intermediate feature is the process of step ③. In order to make the style and better texture characteristics appear in the final picture, and to keep the main content details in the content, the up sample process is used to transfer information when performing the fusion of step 3.

[0217] For details, see Figure 8 As shown, in the embodiment of the present application, S304 includes but is not limited to the following steps:

[0218] S3041, upsampling the intermediate features of the content N times, and using the upsampled features of the content obtained by the Nth upsampling as the content reconstruction features; wherein, in each upsampling, based on the upsampled features of the content obtained by the previous upsampling, combined with the content text features, the upsampled features of the current upsampling are obtained. The content reconstruction features may also be referred to as first reconstruction features.

[0219] Taking the i-th upsampling as an example, the upsampling of the intermediate features of the content includes:

[0220] First, semantic extraction is performed on the content cross-attention features obtained by the i-1th upsampling to obtain the content semantic features of the i-th upsampling;

[0221] Secondly, self-attention processing is performed on the semantic features of the content sampled for the i-th time to obtain the self-attention features of the content sampled for the i-th time;

[0222] Finally, based on the content text features, the i-th upsampled content self-attention features are cross-attention processed to obtain the i-th upsampled content cross-attention features.

[0223] During the first upsampling, semantic extraction is performed on the initial content features. The content semantic features, content self-attention features, and content cross-attention features generated during the upsampling process are all content upsampling features. Similarly, the style semantic features, style self-attention features, and style cross-attention features generated during the upsampling process are all style upsampling features, and the texture semantic features, texture self-attention features, and texture cross-attention features generated during the upsampling process are all texture upsampling features.

[0224] See also Fig. 9 As shown in the figure, each up-sampling unit (up block) is used to implement a down-sampling process. Each up-block specifically includes a residual module for semantic extraction, a self-attention module for self-attention processing, and a cross-attention module for cross-attention processing. The residual module is implemented using resnet, the self-attention module is implemented using the self-attention mechanism, and the cross-attention module is implemented using the cross-attention mechanism. Assume that 4 up-samplings are required.

[0225] During the first upsampling process, semantic extraction is performed on the intermediate features of the content to obtain the semantic features of the content of the first upsampling, and self-attention processing is performed on the semantic features of the content of the first upsampling to obtain the content self-attention features of the first upsampling. Based on the content text features (i.e., the text features corresponding to the description word "car"), cross-attention processing is performed on the content self-attention features of the first upsampling to obtain the content cross-attention features of the first upsampling.

[0226] During the second upsampling process, semantic extraction will be performed on the content cross-attention features of the first upsampling to obtain the content semantic features of the second upsampling, and self-attention processing will be performed on the content semantic features of the second upsampling to obtain the content self-attention features of the second upsampling. Moreover, based on the content text features, cross-attention processing will be performed on the content self-attention features of the second upsampling to obtain the content cross-attention features of the second upsampling.

[0227] During the third upsampling process, semantic extraction will be performed on the content cross-attention features of the second upsampling to obtain the content semantic features of the third upsampling, and self-attention processing will be performed on the content semantic features of the third upsampling to obtain the content self-attention features of the third upsampling. Based on the content text features, cross-attention processing will be performed on the content self-attention features of the third upsampling to obtain the content cross-attention features of the third upsampling.

[0228] During the 4th upsampling process, semantic extraction will be performed on the content cross-attention features of the 3rd upsampling to obtain the content semantic features of the 4th upsampling, and self-attention processing will be performed on the content semantic features of the 4th upsampling to obtain the content self-attention features of the 4th upsampling. Based on the content text features, cross-attention processing will be performed on the content self-attention features of the 4th upsampling to obtain the content cross-attention features of the 4th upsampling, and the content cross-attention features of the 4th upsampling will be obtained.

[0229] S3042. Upsample the intermediate features of the painting style N times, and use the upsampled features of the painting style obtained by the Nth upsampling as the style reconstruction features; wherein, in each upsampling, the upsampled features of the painting style obtained by the previous upsampling are combined with the painting style text features to obtain the upsampled features of the painting style of the current upsampling.

[0230] The painting style reconstruction feature may also be referred to as a second reconstruction feature.

[0231] The upsampling process of the intermediate features of the style is similar to that of the intermediate features of the content. Taking the i-th upsampling as an example, first, semantic extraction is performed on the cross-attention features of the style obtained by the i-1-th upsampling to obtain the semantic features of the style of the i-th upsampling; secondly, self-attention processing is performed on the semantic features of the style of the i-th upsampling to obtain the self-attention features of the style of the i-th upsampling; finally, based on the style text features, cross-attention processing is performed on the self-attention features of the style of the i-th upsampling to obtain the cross-attention features of the style of the i-th upsampling.

[0232] During the first upsampling, semantic extraction is performed on the initial style features. The style semantic features, style self-attention features, and style cross-attention features generated during the upsampling process are all style upsampling features.

[0233] Still reading Fig. 9As shown, during the first upsampling process, semantic extraction is performed on the intermediate features of the painting style to obtain the semantic features of the painting style of the first upsampling, and self-attention processing is performed on the semantic features of the painting style of the first upsampling to obtain the self-attention features of the painting style of the first upsampling, and based on the text features of the painting style (i.e., the text features corresponding to the description word "wood"), cross-attention processing is performed on the self-attention features of the painting style of the first upsampling to obtain the cross-attention features of the painting style of the first upsampling.

[0234] During the second upsampling process, semantic extraction will be performed on the style cross-attention features of the first upsampling to obtain the style semantic features of the second upsampling, and self-attention processing will be performed on the style semantic features of the second upsampling to obtain the style self-attention features of the second upsampling. Moreover, based on the style text features, cross-attention processing will be performed on the style self-attention features of the second upsampling to obtain the style cross-attention features of the second upsampling.

[0235] During the third upsampling process, semantic extraction will be performed on the second upsampling style cross-attention features to obtain the third upsampling style semantic features, and self-attention processing will be performed on the third upsampling style semantic features to obtain the third upsampling style self-attention features. Based on the style text features, cross-attention processing will be performed on the third upsampling style self-attention features to obtain the third upsampling style cross-attention features.

[0236] During the fourth upsampling process, semantic extraction is performed on the style cross-attention features of the third upsampling to obtain the style semantic features of the fourth upsampling, and self-attention processing is performed on the style semantic features of the fourth upsampling to obtain the style self-attention features of the fourth upsampling. Based on the style text features, cross-attention processing is performed on the style self-attention features of the fourth upsampling to obtain the style cross-attention features of the fourth upsampling.

[0237] S3043, upsampling the texture intermediate features N times, and using the texture upsampling features obtained from the Nth upsampling as the target image features; wherein, in each upsampling, based on the texture upsampling features obtained from the previous upsampling, the content upsampling features and the painting style upsampling features obtained from the current upsampling, combined with the texture text features, the texture upsampling features of the current upsampling are obtained.

[0238] Still taking the i-th upsampling as an example, based on the texture upsampling features obtained by the i-1-th upsampling, the content upsampling features and the painting style upsampling features obtained by the i-th upsampling, combined with the texture text features, the texture upsampling features of the i-th upsampling are obtained, including the following steps:

[0239] First, semantic extraction is performed on the texture cross-attention features obtained by the i-1th upsampling to obtain the texture semantic features of the i-th upsampling.

[0240] Secondly, based on the texture semantic features obtained by the i-th upsampling, self-attention processing is performed in combination with the content semantic features obtained by the i-th upsampling to obtain the texture self-attention features of the i-th upsampling.

[0241] Finally, based on the texture self-attention features obtained by the i-th upsampling and the painting style self-attention features obtained by the i-th upsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the i-th upsampling.

[0242] Still reading Fig. 9 As shown, during the first upsampling process, semantic extraction is performed on the intermediate texture features to obtain the texture semantic features of the first upsampling; then, based on the texture semantic features obtained by the first upsampling, self-attention processing is performed in combination with the content semantic features obtained by the first upsampling to obtain the texture self-attention features of the first upsampling; and, based on the texture self-attention features obtained by the first upsampling and the painting style self-attention features obtained by the first upsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the first upsampling.

[0243] During the second upsampling process, semantic extraction is performed on the intermediate texture features to obtain the texture semantic features of the second upsampling; then, based on the texture semantic features obtained by the second upsampling, self-attention processing is performed in combination with the content semantic features obtained by the second upsampling to obtain the texture self-attention features of the second upsampling; and, based on the texture self-attention features obtained by the second upsampling and the painting style self-attention features obtained by the second upsampling, cross-attention processing is performed in combination with the texture text features to obtain the texture cross-attention features of the second upsampling.

[0244] Similarly, four upsamplings can be performed to obtain the texture cross-attention features of the 4th upsampling and the texture cross-attention features of the 4th upsampling. Based on the texture cross-attention features of the 4th upsampling and combined with the texture initial features, the texture reconstruction features, that is, the target image features, can be obtained.

[0245] In a possible implementation, semantic extraction in the upsampling process can be implemented using but not limited to ResNet.

[0246] In one possible implementation, during a texture upsampling process, self-attention processing can be implemented in the following way:

[0247] Based on the third semantic feature obtained by the current upsampling, query information, key information and value information corresponding to the third image attribute for self-attention processing are obtained;

[0248] Based on the query information for self-attention processing corresponding to the third image attribute, combined with the query information corresponding to the first image attribute, new query information is obtained, wherein the query information corresponding to the first image attribute is extracted using the first semantic feature obtained by the current upsampling;

[0249] Based on the key information, value information and new query information corresponding to the third image attribute for self-attention processing, the third self-attention feature of the current upsampling is obtained.

[0250] In one possible implementation, during a texture upsampling process, cross-attention processing can be implemented in the following way:

[0251] Based on the third self-attention feature of the current upsampling, query information for cross-attention processing corresponding to the third image attribute is obtained, and based on the third text feature, key information and value information for cross-attention processing corresponding to the third image attribute are obtained;

[0252] Based on the query information for cross attention processing corresponding to the third image attribute, combined with the query information corresponding to the second image attribute, new query information is obtained, wherein the query information corresponding to the second image attribute is extracted using the second semantic feature obtained by the current upsampling;

[0253] Based on the value information for cross attention processing corresponding to the third image attribute, combined with the value information corresponding to the second image attribute, new value information is obtained, wherein the value information corresponding to the second image attribute is obtained according to the second text feature;

[0254] Based on the key information, new value information and new query information corresponding to the third image attribute for self-attention processing, the third cross-attention feature of the current upsampling is obtained.

[0255] See also Fig. 10AAs shown, it is a schematic diagram of a block calculation information flow in the upsampling process for content provided in an embodiment of the present application. The calculation process in a block is an upsampling. Assume that the QKV involved in the upsampling process for the content is recorded as Q1, K1, and V1. For the content, an upsampling can be implemented in the following way: First, for the i-th upsampling obtained by the residual module, the content semantic features are obtained, and Q1, K1, and V1 are extracted from the content semantic features obtained by the i-th upsampling as input data of the self-attention module; then, Q1, K1, and V1 are input into the self-attention module for self-attention calculation to obtain the content self-attention feature, and the content self-attention feature is used as Q1' in the input data of the cross-attention module. In addition, based on the content text features, K1' and V1' in the input data of the cross-attention module are obtained, and the input data of the cross-attention module is input into the cross-attention module for cross-attention calculation to obtain the content cross-attention feature.

[0256] See also Fig. 10B As shown, it is a schematic diagram of a block calculation information flow in the upsampling process for the painting style provided in an embodiment of the present application. Assume that the QKV generated in the upsampling process for the painting style is recorded as Q2, K2, and V2. For the painting style, one upsampling can be implemented in the following manner: First, for the painting style semantic features obtained by the i-th upsampling through the residual module, Q2, K2, and V2 are extracted from the painting style semantic features obtained by the i-th upsampling as input data of the self-attention module; then, Q2, K2, and V2 are input into the self-attention module for self-attention calculation to obtain the painting style self-attention feature, and the painting style self-attention feature is used as Q2' in the input data of the cross-attention module. In addition, based on the painting style text feature, K2' and V2' in the input data of the cross-attention module are obtained, and the input data of the cross-attention module is input into the cross-attention module for cross-attention calculation to obtain the painting style cross-attention feature.

[0257] See also Fig. 10CAs shown, it is a schematic diagram of a block calculation information flow in the texture upsampling process provided in an embodiment of the present application. Assume that the QKV generated in the upsampling process for texture is denoted as Q3, K3, and V3. For texture, one upsampling can be implemented in the following way: first, for the texture semantic features obtained by the i-th upsampling through the residual module, Q3, K3, and V3 are extracted from the texture semantic features obtained by the i-th upsampling, and Q1 and Q3 are fused in a merging manner to obtain a new Q3; then, the new Q3, K3, and V3 are input into the self-attention module for self-attention calculation to obtain the texture self-attention features, and the texture self-attention features are used as Q3', and K3' and V3' are obtained based on the texture text features; then, Q2' and Q3' are fused in a merging manner to obtain a new Q3', and V2' and V3' are fused to obtain a new V3'; finally, the new Q3', K3', and the new V3' are input into the cross-attention module for cross-attention calculation to obtain the texture cross-attention features.

[0258] based on Fig. 10A , Fig. 10B and Fig. 10C It can be seen that in the process of calculating self attention, Q in content is used for information fusion, because Q in content contains information about the main generated content object, so accurate content information can be obtained. Then, combined with Q in quality itself, it is merged and fused. After the fusion is completed, it is input into self-attention with K and V of quality for calculation. By fusing content + quality, the generated object can be more in line with the selected texture. In the cross attention stage, Q and V in the crossattention calculation in style are used, and then cross attention calculation is performed in combination with K of quality itself. Since the main object and style are the most intuitive components of the picture during the image generation process, for the original Q, only the benchmark K needs to be retained, and the rest uses the fused information to generate an image that meets the specified content, style, and texture requirements.

[0259] S305: Generate a target image based on the target image features.

[0260] In one possible implementation, a VAE decoder may be used to decode the target image features and generate the target image.

[0261] In some implementations, a pre-trained generation model may be used to generate images based on text features.

[0262] In some implementations, corresponding generative models are provided for three types of image attributes, namely, a first generative model, a second generative model, and a third generative model. The first generative model, the second generative model, and the third generative model are all trained generative models. The first generative model is a generative model for generating an image containing a first image attribute; the second generative model is a generative model for generating an image containing a second image attribute; and the third generative model is a trained generative model for generating an image containing a third image attribute. For content, style, and texture, the first generative model, the second generative model, and the third generative model may also be referred to as a content generation model, a style generation model, and a texture generation model.

[0263] Exemplarily, in the feature extraction process, based on the content text features, the content generation model is used to extract the initial features of the content to obtain the intermediate features of the content; based on the style text features and the intermediate features of the content, the style generation model is used to extract the initial features of the style to obtain the intermediate features of the style; and based on the texture text features and the intermediate features of the style, the texture generation model is used to extract the initial features of the texture to obtain the intermediate features of the texture. The content generation model is used to extract the intermediate features of the content, the style generation model is used to extract the intermediate features of the style, and the texture generation model is used to extract the intermediate features of the texture. For the specific feature extraction process, see S303.

[0264] The image generation process is implemented using the generation model of three image attributes, so in the process of use, different generation models of image attributes can be used to form triplets, thereby achieving a generation graph system with different content, different painting styles and different texture effects. The image generation solution provided in the embodiment of the present application is a pluggable zero-shot model combination generation method. The zero-shot attribute can be constructed using all open source models. Under this mechanism of the system, different models in the open source community can be arbitrarily combined to achieve different mix-and-match effects, which can make full use of the model effect of user-generated content and maximize the user's flexibility and freedom of use.

[0265] In addition, the content generation model can also be used to reconstruct the intermediate features of the content, the style generation model can also be used to reconstruct the intermediate features of the style, and the texture generation model can also be used to reconstruct the intermediate features of the texture. For the specific feature reconstruction process, see S304.

[0266] For example, the image generation model can be implemented using, but not limited to, the SD model. Fig.11 As shown, Fig.11A model structure of an SD model provided in an embodiment of the present application, the SD model includes a VAE encoder, a U-Net and a text encoder, and a VAE decoder, wherein the text encoder is the text encoder in the CLIP model.

[0267] Among them, the VAE encoder is used to convert the input image into a low-dimensional latent space representation, that is, the latent representation latent; during the training process of the SD model, the latent vector needs to be forward diffused, that is, denoised, to convert it into a noise feature vector in the latent space as the input of U-Net. During the inference process, the VAE encoder is no longer needed, because the input given to U-Net is a latent of a noise image. During the inference process, the VAE decoder will convert the latent latent signal calculated by U-Net back into an image.

[0268] The VAE decoder is used to perform the opposite operation of the VAE encoder, that is, to transform the image feature vector z in the latent space into the corresponding output image.

[0269] U-Net is used to perform noise prediction based on the noise feature vector in the input latent space to obtain the predicted noise feature. Then, the input noise feature vector is denoised based on the predicted noise feature to obtain the noise feature vector for the next round of noise prediction. The denoising process is performed iteratively multiple times to obtain the image feature vector z in the latent space that can be processed by the VAE decoder.

[0270] U-Net specifically includes an encoder and a decoder, both of which are composed of processing modules based on ResNet. The encoder is used to compress the input image representation (i.e., the initial features in the latent space) into a low-resolution image representation, and the decoder is used to decode the low-resolution image representation back to a high-resolution image representation (i.e., the target image features). In addition, in order to prevent U-Net from losing important information during the downsampling process, a shortcut connection is usually added between the downsampling processing module in the encoder and the upsampling processing module in the decoder. For each processing module of the encoder and decoder, a crossattention structure is deployed to fuse the text features output by the text encoder. The cross attention structure is usually deployed at the end of the processing module; at the same time, in order to enable the final generated image to have higher quality, in each processing module, a self-attention structure is also added between the residual structure and the cross attention layer.

[0271] It should be noted that the image generation method provided in the embodiment of the present application only involves the application of the SD model, but does not involve the training of the SD model, and the image generation method provided in the embodiment of the present application specifically uses the SD model for the Vincent diagram scenario, that is, the scenario of generating the corresponding target image based on the descriptive text. In the application scenario of the Vincent diagram, it is not necessary to input an input image similar to the training image used in the training process to the SD model, and accordingly, it is not necessary to perform the operation of converting the input image into a latent vector and adding noise to the latent vector; in the application scenario of the Vincent diagram, the feature representation in the latent space generated by the noise data can be directly used for image generation.

[0272] In some implementations, in order to make full use of the model effect to improve the quality of the generated image, in the embodiments of the present application, a number of candidate generation models can be set for the three types of image attributes respectively. The number of candidate generation models corresponding to each type of image attribute can be the same or different. In this way, when it is necessary to generate an image, the target object can select a candidate generation model with the same or similar image attributes as the generated image attributes from each candidate generation model as the target generation model, and the target generation model includes a first generation model, a second generation model, and a third generation model.

[0273] In some implementations, the target object may also select a corresponding generation model from the candidate generation models corresponding to some of the three types of image attributes as the generation model for subsequent use, and then use the generation model preset for other parts of the image attributes as the generation model for subsequent use.

[0274] Of course, in some implementations, if a trained generative model can simultaneously meet the image generation requirements of three types of image attributes, then a generative model can also be used to implement the image generation process mentioned in the embodiments of the present application.

[0275] See also Fig.12 As shown, it is an example of several generated images provided in the embodiments of the present application. The first line is some picture examples generated by the content generation model. The content generation model can generate images containing content such as cars, unicorn dolls, and horses. The second line is some picture examples generated by the style generation model. The style generation model can generate images of wood, glass style, Chinese painting style, etc. The third line is some picture examples generated by the texture generation model. The texture generation model can generate images with textures such as western wilderness movies, exhibition quality, and painting texture.

[0276] If the image description text contains content such as cars, wood, and Wild West movies, then the image generation method provided in the embodiment of the present application can generate the first image in the fourth row, which contains a car, has a wooden style, and has a texture of a Wild West movie.

[0277] If the image description text contains content such as a unicorn doll, glass style, and exhibition quality, then the image generation method provided in the embodiment of the present application can generate the second image in the fourth row. The image contains a unicorn doll, the painting style is glass style, and the texture is of exhibition quality.

[0278] If the image description text contains content such as horses, Chinese painting style, and painting texture, then the image generation method provided in the embodiment of the present application can generate the second image in the fourth row, which contains horses, Chinese painting style, and has the texture of a painting.

[0279] Based on the same inventive concept, the embodiment of the present application provides an image generating device 1300. Fig.13 As shown, it is a schematic diagram of the structure of the image generating device 1300, which may include:

[0280] The text extraction unit 1301 is used to extract the corresponding first text feature, second text feature and third text feature respectively based on the image description text and for the set first image attribute, second image attribute and third image attribute;

[0281] A feature initialization unit 1302 is used to extract corresponding first initial features, second initial features and third initial features respectively for the first image attribute, the second image attribute and the third image attribute based on a preset noise image;

[0282] A feature extraction unit 1303 is used to extract features from the first initial feature, the second initial feature and the third initial feature based on the first text feature, the second text feature and the third text feature, respectively, to obtain corresponding first intermediate features, second intermediate features and third intermediate features;

[0283] The feature reconstruction unit 1304 is used to reconstruct the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain a target image feature, and generate a target image based on the target image feature.

[0284] In a possible implementation, when extracting the first initial feature based on the first text feature to obtain the first intermediate feature, the feature extraction unit 1303 is specifically used to:

[0285] Downsampling the first initial feature N times, and using the first downsampled feature obtained by the Nth downsampling as the first intermediate feature; wherein, in each downsampling, based on the first downsampled feature obtained by the previous downsampling, combined with the first text feature, the first downsampled feature of the current downsampling is obtained;

[0286] Based on the second text feature, feature extraction is performed on the second initial feature to obtain a second intermediate feature, including:

[0287] The second initial feature is downsampled N times, and the second downsampled feature obtained by the Nth downsampling is used as the second intermediate feature; wherein, in each downsampling, the second downsampled feature of the current downsampling is obtained based on the second downsampled feature and the first downsampled feature obtained by the previous downsampling, combined with the second text feature.

[0288] In a possible implementation, when the second down-sampling feature obtained based on the previous down-sampling and the first down-sampling feature are combined with the second text feature to obtain the second down-sampling feature of the current down-sampling, the feature extraction unit 1303 is specifically used to:

[0289] The second down-sampling feature obtained by the previous down-sampling is fused with the first down-sampling feature to obtain the input data of the current down-sampling;

[0290] Based on the input data of the current downsampling, downsampling processing is performed in combination with the second text feature to obtain the second downsampling feature of the current downsampling.

[0291] In a possible implementation, when the downsampling process is performed based on the input data of the current downsampling in combination with the second text feature to obtain the second downsampling feature of the current downsampling, the feature extraction unit 1303 is specifically used to:

[0292] Performing semantic extraction on the input data of the current downsampling to obtain a second semantic feature in the second downsampling feature of the current downsampling, and performing self-attention processing based on the second semantic feature of the current downsampling to obtain a second self-attention feature in the second downsampling feature of the current downsampling;

[0293] Based on the second self-attention feature of the current downsampling, cross-attention processing is performed in combination with the second text feature to obtain the second cross-attention feature in the second downsampling feature of the current downsampling.

[0294] In a possible implementation, when extracting a feature from the third initial feature based on the third text feature to obtain a third intermediate feature, the feature extraction unit 1303 is specifically configured to:

[0295] The third initial feature is downsampled N times, and the third downsampled feature obtained by the Nth downsampling is used as the third intermediate feature; wherein, in each downsampling, the third downsampled feature of the current downsampling is obtained based on the third downsampled feature obtained by the previous downsampling, the second downsampled feature obtained by the current downsampling, and the third text feature.

[0296] In a possible implementation, the third down-sampling feature includes a third semantic feature, a third self-attention feature, and a third cross-attention feature in the down-sampling process;

[0297] When the third downsampling feature obtained by the previous downsampling and the second downsampling feature obtained by the current downsampling are obtained by combining the third text feature to obtain the third downsampling feature of the current downsampling, the feature extraction unit 1303 is specifically used to:

[0298] Perform semantic extraction on the third cross-attention feature obtained by the previous downsampling to obtain the third semantic feature of the current downsampling;

[0299] Based on the second self-attention feature obtained by the current downsampling, the self-attention processing is performed in combination with the third semantic feature of the current downsampling to obtain the third self-attention feature of the current downsampling; wherein the second self-attention feature obtained by the current downsampling is generated by using the second downsampling feature obtained by the previous downsampling;

[0300] Based on the third self-attention feature of the current downsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current downsampling.

[0301] In a possible implementation, when the second self-attention feature obtained based on the current downsampling is self-attention processed in combination with the third semantic feature of the current downsampling to obtain the third self-attention feature of the current downsampling, the feature extraction unit 1303 is specifically used to:

[0302] Based on the second self-attention feature obtained by the current downsampling, query information, key information and value information corresponding to the second image attribute are obtained; and based on the third semantic feature obtained by the current downsampling, query information, key information and value information corresponding to the third image attribute are obtained;

[0303] The query information, key information and value information corresponding to the second image attribute and the query information, key information and value information corresponding to the third image attribute are combined to obtain new input data, and self-attention calculation is performed on the new input data to obtain the third self-attention feature of the current downsampling.

[0304] In a possible implementation, when reconstructing the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain the target image feature, the feature reconstruction unit 1304 is specifically used to:

[0305] The first intermediate feature is upsampled N times, and a first upsampled feature obtained by the Nth upsampling is used as a first reconstructed feature; wherein, in each upsampling, the first upsampled feature of the current upsampling is obtained based on the first upsampled feature obtained by the previous upsampling and combined with the first text feature;

[0306] The second intermediate feature is upsampled N times, and the second upsampled feature obtained by the Nth upsampling is used as the second reconstructed feature; wherein, in each upsampling, the second upsampled feature of the current upsampling is obtained based on the second upsampled feature obtained by the previous upsampling and combined with the second text feature;

[0307] The third intermediate feature is upsampled N times, and the third upsampled feature obtained by the Nth upsampling is used as the target image feature; wherein, in each upsampling, the third upsampled feature of the current upsampling is obtained based on the third upsampled feature obtained by the previous upsampling, the first upsampled feature and the second upsampled feature obtained by the current upsampling, combined with the third text feature.

[0308] In a possible implementation, the third upsampling feature includes a third semantic feature, a third self-attention feature, and a third cross-attention feature in the upsampling process;

[0309] When the third upsampling feature obtained by the previous upsampling, the first upsampling feature and the second upsampling feature obtained by the current upsampling are obtained by combining the third text feature to obtain the third upsampling feature of the current upsampling, the feature reconstruction unit 1304 is specifically used to:

[0310] Perform semantic extraction on the third cross-attention feature obtained by the previous upsampling to obtain the third semantic feature of the current upsampling;

[0311] Based on the third semantic feature obtained by the current upsampling, the self-attention processing is performed in combination with the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling;

[0312] Based on the third self-attention feature obtained by the current upsampling and the second self-attention feature in the second upsampling feature obtained by the current upsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current upsampling.

[0313] In a possible implementation, when the third semantic feature obtained based on the current upsampling is self-attention processed in combination with the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling, the feature reconstruction unit 1304 is specifically used to:

[0314] Based on the third semantic feature obtained by the current upsampling, obtaining query information, key information and value information for self-attention processing corresponding to the third image attribute;

[0315] Based on the query information for self-attention processing corresponding to the third image attribute, combined with the query information corresponding to the first image attribute, new query information is obtained, wherein the query information corresponding to the first image attribute is extracted using the first semantic feature obtained by the current upsampling;

[0316] Based on the key information, value information and the new query information corresponding to the third image attribute for self-attention processing, the third self-attention feature of the current upsampling is obtained.

[0317] In a possible implementation, when the third self-attention feature obtained based on the current upsampling and the second self-attention feature in the second upsampling feature obtained by the current upsampling are combined with the third text feature to perform cross-attention processing to obtain the third cross-attention feature of the current upsampling, the feature reconstruction unit 1304 is specifically used to:

[0318] Based on the third self-attention feature of the current upsampling, obtaining query information for cross-attention processing corresponding to the third image attribute, and based on the third text feature, obtaining key information and value information for cross-attention processing corresponding to the third image attribute;

[0319] Based on the query information for cross attention processing corresponding to the third image attribute, combined with the query information corresponding to the second image attribute, new query information is obtained, wherein the query information corresponding to the second image attribute is extracted using the second semantic feature obtained by the current upsampling;

[0320] Based on the value information for cross attention processing corresponding to the third image attribute, combined with the value information corresponding to the second image attribute, new value information is obtained, wherein the value information corresponding to the second image attribute is obtained according to the second text feature;

[0321] Based on the key information for self-attention processing corresponding to the third image attribute, the new value information and the new query information, a third cross-attention feature of the current upsampling is obtained.

[0322] In a possible implementation, when extracting the first initial feature, the second initial feature, and the third initial feature based on the first text feature, the second text feature, and the third text feature to obtain the corresponding first intermediate feature, the second intermediate feature, and the third intermediate feature, the feature extraction unit 1303 is specifically used to:

[0323] Based on the first text feature, using a first generation model, extracting features from the first initial feature to obtain a first intermediate feature, wherein the first generation model is a trained generation model for generating an image containing the first image attribute;

[0324] Based on the second text feature and the first intermediate feature, using a second generation model, extracting features from the second initial feature to obtain a second intermediate feature, wherein the second generation model is a trained generation model for generating an image containing the second image attribute;

[0325] Based on the third text feature and the second intermediate feature, the third initial feature is extracted using the third generation model to obtain a third intermediate feature, and the third generation model is a trained generation model for generating an image containing the third image attribute.

[0326] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0327] Regarding the device in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated here.

[0328] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0329] Based on the same inventive concept, the embodiment of the present application also provides an electronic device. In one embodiment, the electronic device can be a server or a terminal device. Fig.14 As shown, it is a schematic diagram of the structure of a possible electronic device provided in an embodiment of the present application, Fig.14 In the embodiment, the electronic device 1400 includes: a processor 1410 and a memory 1420.

[0330] The memory 1420 stores a computer program that can be executed by the processor 1410 , and the processor 1410 can perform the steps of the above-mentioned image generation method by executing the instructions stored in the memory 1420 .

[0331] The memory 1420 may be a volatile memory, such as a random-access memory (RAM); the memory 1420 may also be a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1420 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1420 may also be a combination of the above memories.

[0332] The processor 1410 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1410 is configured to implement the above-mentioned image generation method when executing the computer program stored in the memory 1420 .

[0333] In some embodiments, the processor 1410 and the memory 1420 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0334] The specific connection medium between the processor 1410 and the memory 1420 is not limited in the embodiment of the present application. In the embodiment of the present application, the processor 1410 and the memory 1420 are connected via a bus as an example. Fig.14 The connections between the other components are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.14 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0335] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the above-mentioned image generation method. In some possible implementations, various aspects of the image generation method provided by the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps in the above-mentioned image generation method. For example, the electronic device can perform the following steps: Figure 3 Follow the steps shown in .

[0336] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (Compact Disk Read Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0337] The program product of the embodiment of the present application may adopt a CD-ROM and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a computer program, which may be used by or in combination with a command execution system, apparatus or device.

[0338] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a computer program for use by or in conjunction with a command execution system, apparatus, or device.

[0339] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0340] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: include: Based on the image description text, for the set first image attribute, second image attribute and third image attribute, respectively extract the corresponding first text feature, second text feature and third text feature; Based on the preset noise image, for the first image attribute, the second image attribute and the third image attribute, respectively extracting a corresponding first initial feature, a second initial feature and a third initial feature; Based on the first text feature, the second text feature and the third text feature, feature extraction is performed on the first initial feature, the second initial feature and the third initial feature respectively to obtain corresponding first intermediate feature, second intermediate feature and third intermediate feature; Based on the first intermediate feature and the second intermediate feature, reconstruct the third intermediate feature to obtain a target image feature, and generate a target image based on the target image feature; The step of reconstructing the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain the target image feature includes: Performing N upsampling on the first intermediate feature, and using the first upsampling feature obtained by the Nth upsampling as the first reconstructed feature; wherein, in each upsampling, based on the first upsampling feature obtained by the previous upsampling, combined with the first text feature, the first upsampling feature of the current upsampling is obtained; The second intermediate feature is upsampled N times, and the second upsampled feature obtained by the Nth upsampling is used as the second reconstructed feature; wherein, in each upsampling, the second upsampled feature of the current upsampling is obtained based on the second upsampled feature obtained by the previous upsampling and combined with the second text feature; The third intermediate feature is upsampled N times, and a third upsampled feature obtained by the Nth upsampling is used as a target image feature, wherein the third upsampled feature includes: a third semantic feature, a third self-attention feature, and a third cross-attention feature in the upsampling process; wherein, in each upsampling, the following operations are performed: Perform semantic extraction on the third cross-attention feature obtained by the previous upsampling to obtain the third semantic feature of the current upsampling; Based on the third semantic feature obtained by the current upsampling, the self-attention processing is performed in combination with the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling; Based on the third self-attention feature obtained by the current upsampling and the second self-attention feature in the second upsampling feature obtained by the current upsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current upsampling.

2. The method according to claim 1, characterized in that Based on the first text feature, feature extraction is performed on the first initial feature to obtain a first intermediate feature, including: Downsampling the first initial feature N times, and using the first downsampled feature obtained by the Nth downsampling as the first intermediate feature; wherein, in each downsampling, based on the first downsampled feature obtained by the previous downsampling, combined with the first text feature, the first downsampled feature of the current downsampling is obtained; Based on the second text feature, feature extraction is performed on the second initial feature to obtain a second intermediate feature, including: The second initial feature is downsampled N times, and the second downsampled feature obtained by the Nth downsampling is used as the second intermediate feature; wherein, in each downsampling, the second downsampled feature of the current downsampling is obtained based on the second downsampled feature and the first downsampled feature obtained by the previous downsampling, combined with the second text feature.

3. The method according to claim 2, characterized in that The second downsampling feature obtained based on the previous downsampling and the first downsampling feature, combined with the second text feature, to obtain the second downsampling feature of the current downsampling, comprises: The second down-sampling feature obtained by the previous down-sampling is fused with the first down-sampling feature to obtain the input data of the current down-sampling; Based on the input data of the current downsampling, downsampling processing is performed in combination with the second text feature to obtain the second downsampling feature of the current downsampling.

4. The method according to claim 3, characterized in that The step of performing downsampling processing on the input data based on the current downsampling in combination with the second text feature to obtain the second downsampling feature of the current downsampling includes: Performing semantic extraction on the input data of the current downsampling to obtain a second semantic feature in the second downsampling feature of the current downsampling, and performing self-attention processing based on the second semantic feature of the current downsampling to obtain a second self-attention feature in the second downsampling feature of the current downsampling; Based on the second self-attention feature of the current downsampling, cross-attention processing is performed in combination with the second text feature to obtain the second cross-attention feature in the second downsampling feature of the current downsampling.

5. The method according to any one of claims 2 to 4, characterized in that: Based on the third text feature, feature extraction is performed on the third initial feature to obtain a third intermediate feature, including: The third initial feature is downsampled N times, and the third downsampled feature obtained by the Nth downsampling is used as the third intermediate feature; wherein, in each downsampling, the third downsampled feature of the current downsampling is obtained based on the third downsampled feature obtained by the previous downsampling, the second downsampled feature obtained by the current downsampling, and the third text feature.

6. The method according to claim 5, characterized in that The third down-sampling feature includes the third semantic feature, the third self-attention feature, and the third cross-attention feature in the down-sampling process; The third downsampling feature obtained based on the previous downsampling, the second downsampling feature obtained based on the current downsampling, and the third text feature are combined to obtain the third downsampling feature of the current downsampling, including: Perform semantic extraction on the third cross-attention feature obtained by the previous downsampling to obtain the third semantic feature of the current downsampling; Based on the second self-attention feature obtained by the current downsampling, the self-attention processing is performed in combination with the third semantic feature of the current downsampling to obtain the third self-attention feature of the current downsampling; wherein the second self-attention feature obtained by the current downsampling is generated by using the second downsampling feature obtained by the previous downsampling; Based on the third self-attention feature of the current downsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current downsampling.

7. The method according to claim 6, characterized in that The second self-attention feature obtained based on the current downsampling is combined with the third semantic feature of the current downsampling to perform self-attention processing to obtain the third self-attention feature of the current downsampling, including: Based on the second self-attention feature obtained by the current downsampling, query information, key information and value information corresponding to the second image attribute are obtained; and based on the third semantic feature obtained by the current downsampling, query information, key information and value information corresponding to the third image attribute are obtained; The query information, key information and value information corresponding to the second image attribute and the query information, key information and value information corresponding to the third image attribute are combined to obtain new input data, and self-attention calculation is performed on the new input data to obtain the third self-attention feature of the current downsampling.

8. The method according to any one of claims 1 to 4, characterized in that The method of performing self-attention processing based on the third semantic feature obtained by the current upsampling and combining the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling includes: Based on the third semantic feature obtained by the current upsampling, obtaining query information, key information and value information for self-attention processing corresponding to the third image attribute; Based on the query information for self-attention processing corresponding to the third image attribute, combined with the query information corresponding to the first image attribute, new query information is obtained, wherein the query information corresponding to the first image attribute is extracted using the first semantic feature obtained by the current upsampling; Based on the key information, value information and the new query information corresponding to the third image attribute for self-attention processing, the third self-attention feature of the current upsampling is obtained.

9. The method according to any one of claims 1 to 4, characterized in that The third self-attention feature obtained based on the current upsampling, the second self-attention feature in the second upsampling feature obtained based on the current upsampling, and the third text feature are combined to perform cross-attention processing to obtain the third cross-attention feature of the current upsampling, including: Based on the third self-attention feature of the current upsampling, obtaining query information for cross-attention processing corresponding to the third image attribute, and based on the third text feature, obtaining key information and value information for cross-attention processing corresponding to the third image attribute; Based on the query information for cross attention processing corresponding to the third image attribute, combined with the query information corresponding to the second image attribute, new query information is obtained, wherein the query information corresponding to the second image attribute is extracted using the second semantic feature obtained by the current upsampling; Based on the value information for cross attention processing corresponding to the third image attribute, combined with the value information corresponding to the second image attribute, new value information is obtained, wherein the value information corresponding to the second image attribute is obtained according to the second text feature; Based on the key information for self-attention processing corresponding to the third image attribute, the new value information and the new query information, a third cross-attention feature of the current upsampling is obtained.

10. The method according to any one of claims 1 to 4, characterized in that The extracting the first initial feature, the second initial feature and the third initial feature based on the first text feature, the second text feature and the third text feature to obtain the corresponding first intermediate feature, the second intermediate feature and the third intermediate feature includes: Based on the first text feature, using a first generation model, extracting features from the first initial feature to obtain a first intermediate feature, wherein the first generation model is a trained generation model for generating an image containing the first image attribute; Based on the second text feature and the first intermediate feature, using a second generation model, extracting features from the second initial feature to obtain a second intermediate feature, wherein the second generation model is a trained generation model for generating an image containing the second image attribute; Based on the third text feature and the second intermediate feature, a third generation model is used to extract the third initial feature to obtain a third intermediate feature. The third generation model is a trained generation model for generating an image containing the third image attribute.

11. An image generating device, characterized in that: include: A text extraction unit, for extracting corresponding first text features, second text features and third text features respectively based on the image description text and for the set first image attribute, second image attribute and third image attribute; A feature initialization unit, configured to extract, based on a preset noise image, a first initial feature, a second initial feature, and a third initial feature for the first image attribute, the second image attribute, and the third image attribute, respectively; a feature extraction unit, configured to extract features from the first initial feature, the second initial feature, and the third initial feature based on the first text feature, the second text feature, and the third text feature, respectively, to obtain corresponding first intermediate features, second intermediate features, and third intermediate features; a feature reconstruction unit, configured to reconstruct the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain a target image feature, and generate a target image based on the target image feature; Wherein, when reconstructing the third intermediate feature based on the first intermediate feature and the second intermediate feature to obtain the target image feature, the feature reconstruction unit is specifically used for: Performing N upsampling on the first intermediate feature, and using the first upsampling feature obtained by the Nth upsampling as the first reconstructed feature; wherein, in each upsampling, based on the first upsampling feature obtained by the previous upsampling, combined with the first text feature, the first upsampling feature of the current upsampling is obtained; The second intermediate feature is upsampled N times, and the second upsampled feature obtained by the Nth upsampling is used as the second reconstructed feature; wherein, in each upsampling, the second upsampled feature of the current upsampling is obtained based on the second upsampled feature obtained by the previous upsampling and combined with the second text feature; The third intermediate feature is upsampled N times, and a third upsampled feature obtained by the Nth upsampling is used as a target image feature, wherein the third upsampled feature includes: a third semantic feature, a third self-attention feature, and a third cross-attention feature in the upsampling process; wherein, in each upsampling, the following operations are performed: Perform semantic extraction on the third cross-attention feature obtained by the previous upsampling to obtain the third semantic feature of the current upsampling; Based on the third semantic feature obtained by the current upsampling, the self-attention processing is performed in combination with the first semantic feature in the first upsampling feature obtained by the current upsampling to obtain the third self-attention feature of the current upsampling; Based on the third self-attention feature obtained by the current upsampling and the second self-attention feature in the second upsampling feature obtained by the current upsampling, cross-attention processing is performed in combination with the third text feature to obtain the third cross-attention feature of the current upsampling.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: It includes a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 10.

14. A computer program product, characterized in that It includes a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of any method described in claims 1 to 10.

Images