Image generation method, device, computer equipment and storage medium

Abstract

The present application relates to an image generation method, device, computer equipment and storage medium. The method includes: determining object description features describing q objects in an image, and attribute description features of each of the attributes of p objects; obtaining and using random noise features as multi-step noise features obtained by performing N iterative noise steps on the image features of the image and performing noise reduction processing of N iterative noise reduction steps; constructing a parallel noise reduction mechanism for object description features and attribute description features, allowing the object description features and attribute description features to first perform noise reduction processing without interfering with each other, and then modifying the first noise reduction features by using the second noise reduction features corresponding to the attributes of the p objects in a fusion manner to obtain the output data of the Nth noise reduction step; using it as the image features after noise reduction, decoding the image features after noise reduction, and generating an image. The use of this method can improve the accuracy of image generation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, text-based image generation technology has emerged. Text-based image generation technology is a multimodal machine learning task that can generate real images that match the description based on a given text description. This technology has great application potential, including visual reasoning, image editing, video games, animation production, computer-aided design and many other fields.

[0003] In traditional technology, a commonly used image generation method is to input a given text description as a control signal into a generative model so as to directly map the text description into an image through the generative model.

[0004] However, traditional methods have the problem of inaccurate generated images due to the difficulty in learning the direct mapping of text descriptions to images by generative models, when there is object description information of multiple objects and attribute description information of at least one object in the text description. Summary of the invention

[0005] Based on this, it is necessary to provide an image generation method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of image generation in response to the above technical problems.

[0006] In a first aspect, the present application provides an image generation method. The method comprises:

[0007] Acquire image description information indicating generation of an image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q;

[0008] Obtaining a random noise feature, and using the random noise feature as a multi-step noise feature obtained by performing N iterative noise addition steps on the image feature of the image, where N is a positive integer greater than 1;

[0009] Performing denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step;

[0010] In the nth denoising step, iterative processing of M denoising processing layers is performed, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer, and when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step;

[0011] At the mth denoising processing layer, denoising is performed based on the portion of the input data and the object description features of the q objects to obtain a first denoising feature, denoising is performed based on the portion of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and the first denoising features and the second denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer;

[0012] The output data of the Nth denoising step is used as the denoised image feature, and the denoised image feature is decoded to generate an image.

[0013] In a second aspect, the present application also provides an image generation device. The device comprises:

[0014] a description feature acquisition module, used to acquire image description information indicating the generation of an image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q;

[0015] A noise feature acquisition module, used to acquire a random noise feature, and use the random noise feature as a multi-step noise feature obtained by performing N iterative noise addition steps on the image feature of the image, where N is a positive integer greater than 1;

[0016] a denoising processing module, configured to perform denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step;

[0017] The denoising processing module is further used to perform iterative processing of M denoising processing layers in the nth denoising step, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer, and when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step;

[0018] The denoising processing module is further configured to perform denoising processing at the mth denoising processing layer based on the portion of input data and the object description features of the q objects to obtain a first denoising feature, perform denoising processing based on the portion of input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer;

[0019] The decoding module is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

[0020] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned image generation method are implemented.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned image generation method are implemented.

[0022] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image generation method are implemented.

[0023] The above-mentioned image generation method, apparatus, computer equipment, storage medium and computer program product can use the image description information to determine the object description information of q objects in the image and the attribute description features of the attributes of p objects by obtaining image description information indicating the generation of the image, and then can obtain random noise features, use the random noise features as multi-step noisy noise features obtained by subjecting the image features of the image to N iterative noisy steps, perform denoising processing on the multi-step noisy noise features in N iterative denoising steps, perform iterative processing of M denoising processing layers in the nth denoising step, perform denoising processing on the mth denoising processing layer using a portion of input data and the object description features of the q objects to obtain a first denoising feature, and use a portion of input data and the attribute description features of the attributes of the p objects to obtain a first denoising feature. The first denoising feature and the second denoising feature corresponding to the attributes of the p objects are respectively obtained, and the output data of the mth denoising processing layer are obtained by fusing the first denoising feature and the second denoising feature corresponding to the attributes of the p objects. In the whole denoising processing process, a parallel denoising mechanism of object description features and attribute description characteristics is constructed, so that the object description features and the attribute description features are first subjected to denoising processing without interfering with each other, and then the first denoising feature is modified by using the second denoising features corresponding to the attributes of the p objects respectively in a fusion manner, so that the output data of the Nth denoising step that makes the object's attributes more accurately expressed can be obtained, so that on the basis of using the output data of the Nth denoising step as the image features after denoising, the image features after denoising are decoded to generate an image that makes the object's attributes more accurately expressed, thereby improving the accuracy of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A diagram showing an application environment of an image generation method in an embodiment;

[0025] Figure 2 is a schematic flow chart of an image generating method in one embodiment;

[0026] Figure 3 is a schematic diagram of weighted filtering in one embodiment;

[0027] Figure 4 A schematic diagram of weighted filtering in another embodiment;

[0028] Figure 5 is a schematic diagram of integrating a third noise reduction feature in one embodiment;

[0029] Figure 6 It is a schematic diagram of fusing the first noise reduction feature and the attribute noise reduction feature in one embodiment;

[0030] Figure 7is a schematic structural diagram of a noise reduction processing layer in a noise reduction step in one embodiment;

[0031] Figure 8 A schematic diagram of a process of performing noise reduction processing on the mth noise reduction processing layer in an embodiment;

[0032] Fig. 9 A schematic diagram of a flow chart of feature interaction processing in one embodiment;

[0033] Fig.10 A schematic diagram of a flow chart of feature interaction processing in another embodiment;

[0034] Fig.11 A schematic diagram of inputting image description information in one embodiment;

[0035] Fig.12 A schematic diagram of inputting image description information in another embodiment;

[0036] Fig.13 A schematic diagram of a structure of a text-based image generation model in one embodiment;

[0037] Fig.14 is a schematic structural diagram of the mth noise reduction block in one embodiment;

[0038] Fig.15 is an overall framework diagram of an image generation method in one embodiment;

[0039] Fig.16 Schematic diagram of the structure of a stable diffusion model (potential diffusion model) in one embodiment;

[0040] Fig.17 A computational flow chart of parallel cross-attention post-computation in one embodiment;

[0041] Fig.18 is a calculation flow chart of mask-based fusion in one embodiment;

[0042] Fig.19 is an example diagram of a generation result of an image generation method in an embodiment;

[0043] Fig. 20 is a structural block diagram of an image generating device in one embodiment;

[0044] Fig.21 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] The present application relates to the field of artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0046] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics and other technologies. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0047] The solution provided in the embodiment of the present application relates to the machine learning / deep learning and other technologies of artificial intelligence. Machine learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The image generation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers.

[0050] The server 104 obtains image description information indicating the generation of an image, determines object description features describing q objects in the image and attribute description features of the attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q, obtains a random noise feature, uses the random noise feature as a multi-step noisy noise feature obtained by subjecting the image feature of the image to N iterative noisy steps, where N is a positive integer greater than 1, performs denoising processing on the multi-step noisy noise feature through N iterative denoising steps, and obtains output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step noisy noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step, and the nth denoising step is iteratively processed with M denoising processing layers, wherein , for the mth denoising layer among the M denoising layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising layer is used as part of the input data of the mth denoising layer, when m is equal to M, the output data of the mth denoising layer is the output data of the nth denoising step, in the mth denoising layer, based on a part of the input data and the object description features of q objects, denoising is performed to obtain a first denoising feature, based on a part of the input data and the attribute description features of the attributes of p objects, denoising is performed to obtain second denoising features corresponding to the attributes of the p objects, the first denoising features and the second denoising features corresponding to the attributes of the p objects are fused to obtain the output data of the mth denoising layer, the output data of the Nth denoising step is used as the image feature after denoising, the image feature after denoising is decoded, and an image is generated.

[0051] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0052] In one embodiment, Figure 2 As shown, an image generation method is provided, which can be executed by a terminal or a server alone, or by a terminal and a server in collaboration. In the embodiment of the present application, the method is applied to a server as an example for illustration, and includes the following steps:

[0053] Step 202, obtaining image description information indicating the generation of an image, and determining object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q.

[0054] Among them, the image description information refers to the information that describes the image to be generated, and is used to indicate the generation of the image. The image description information includes object description information that describes the q objects in the image, and attribute description information of the attributes of the p objects. An object in an image refers to a single element or entity that can be identified and distinguished within the image. For example, an object in an image can be any identifiable image component such as a person, a person's clothing, an animal, a vehicle, a building, a natural landscape, etc. Object description information refers to information that describes an object in an image, and is used to indicate the generation of an object in an image. For example, when the object in the image is a person, the object description information can specifically be a boy, a girl, etc. For another example, when the object in the image is a person's clothing, the object description information can specifically be short sleeves, short skirts, long pants, etc.

[0055] The object description feature refers to a feature that can represent the q objects in the image, and the q objects in the image can be distinguished from other objects through the object description feature. For example, the object description feature can specifically refer to a feature vector that can represent the q objects in the image. It can be understood that the object description information can be obtained by extracting features from the object description information describing the q objects in the image.

[0056] Among them, the attribute description information refers to the information that describes the attributes of the object in the image, and is used to indicate the attributes of the object in the generated image. In this embodiment, the attributes of the object refer to a series of parameters that describe and define the characteristics of the object, and these attributes help to identify, classify and understand the objects in the image. For example, the attribute of the object can specifically be the color or color distribution of the object in the image. For example, when the object in the image is clothing such as short sleeves, short skirts, and long pants, vehicles, buildings, etc., the attribute of the object can specifically be the color of the object. For another example, the attribute of the object can specifically be the geometric shape of the object in the image. For example, when the object in the image is a building, etc., the attribute of the object can specifically be the geometric shape of the object.

[0057] Among them, the attribute description feature refers to a feature that can represent the attribute of an object in the image, and the attribute of the object in the image can be distinguished from the attributes of other objects through the attribute description feature. For example, the attribute description feature can specifically refer to a feature vector that can represent the attribute of the object in the image. It can be understood that the attribute description information can be obtained by extracting features from the attribute description information describing the attributes of the object in the image, and the attributes of the p objects in the image have corresponding attribute description features.

[0058] Specifically, when image generation is required, the server will obtain image description information indicating the generation of the image, so as to determine the object description features describing q objects in the image and the attribute description features of the attributes of p objects by extracting features from the image description information, wherein q is a positive integer and p is a positive integer not greater than q. It can be understood that p is a positive integer not greater than q, indicating that when there are q objects in the image and p objects have attributes, at least one object has the attribute, but not necessarily every object has the attribute.

[0059] In a specific application, the image description information includes object description information describing q objects in the image and attribute description information of each of the attributes of p objects. By performing feature extraction on the object description information describing the q objects in the image, the object description features describing the q objects in the image can be obtained. By performing feature extraction on the attribute description features of each of the attributes of the p objects, the attribute description features describing the attributes of the p objects in the image can be obtained.

[0060] Step 204, obtaining random noise features, and using the random noise features as multi-step noise features obtained by subjecting the image features of the image to N iterative noise adding steps, where N is a positive integer greater than 1.

[0061] Among them, the random noise feature refers to the feature of the noise signal obtained by random sampling when generating an image. For example, the random noise feature can be specifically a Gaussian noise feature, and the random noise feature can be obtained by encoding the noise signal obtained by random sampling. Multi-step noise addition is a process of gradually introducing noise into a signal or data, which can be used to simulate noise conditions in the real world. In this embodiment, multi-step noise addition can be understood as a process of gradually introducing noise into the image features of an image, and after N iterative noise addition steps, a multi-step noise addition noise feature can be obtained. Noise addition step refers to the process of deliberately introducing a certain degree of noise into the data in image processing, that is, in this embodiment, through the noise addition step, a certain degree of noise can be deliberately introduced into the image features of the image.

[0062] Specifically, when an image needs to be generated, the server will obtain the random noise feature, and use the random noise feature as the multi-step noisy noise feature obtained by subjecting the image feature of the image to N iterative noisy steps, so as to restore the image feature of the image from the random noise feature through denoising processing of N iterative denoising steps, where N is a positive integer greater than 1.

[0063] In specific applications, obtaining random noise features usually involves generating random signals with specific statistical characteristics. In this embodiment, the method for obtaining random noise features is not limited, as long as the random noise features can be obtained. For example, taking the random noise feature as a Gaussian noise feature as an example, the method for obtaining the random noise feature can be to generate random numbers using a normal distribution (mean is 0, standard deviation is 1 or other values). For another example, taking the random noise feature as a uniform noise feature as an example, the method for obtaining the random noise feature can be to sample random numbers from a uniform distribution (any value within a certain range appears with equal probability).

[0064] Step 206, performing denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as the input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as the input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step.

[0065] The denoising process refers to the process of reducing the unnecessary noise in the random noise feature to restore the image features of the image from the random noise feature. The denoising process in this embodiment is a denoising process of N iterative denoising steps, that is, the number of denoising steps in the denoising process is the same as the number of iterative denoising steps when the random noise feature is used as the multi-step denoising noise feature. The denoising step refers to an operation stage in the denoising process, which is used to reduce the noise in the multi-step denoising noise feature.

[0066] Specifically, when the multi-step noisy noise feature is obtained through N iterative noisy steps, the server performs denoising processing of N iterative denoising steps on the multi-step noisy noise feature based on the object description features that describe q objects in the image and the attribute description features of the attributes of p objects, thereby restoring the image features of the image from the random noise features, that is, the output data of the Nth denoising step.

[0067] Specifically, when performing denoising processing of N iterative denoising steps, for the n-th denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step noise feature is used as the input data of the n-th denoising step; when n is greater than 1 and less than or equal to N, the output data of the n-1-th denoising step is used as the input data of the n-th denoising step; when n is equal to N, the output data of the n-th denoising step is the output data of the N-th denoising step, thereby achieving gradual denoising.

[0068] Step 208, performing iterative processing of M noise reduction processing layers in the nth noise reduction step, wherein, for the mth noise reduction processing layer among the M noise reduction processing layers, when m is equal to 1, the input data of the nth noise reduction step is used as the input data of the mth noise reduction processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th noise reduction processing layer is used as part of the input data of the mth noise reduction processing layer, and when m is equal to M, the output data of the mth noise reduction processing layer is the output data of the nth noise reduction step.

[0069] The denoising processing layer refers to a processing layer used to achieve a specific denoising effect in the denoising step. It can be understood that each denoising step includes M denoising processing layers, and by iterating the M denoising processing layers, denoising processing of the input data of the denoising step can be achieved in the denoising step.

[0070] Specifically, in the nth denoising step, iterative processing of M denoising processing layers will be performed, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer; when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer; when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step.

[0071] In a specific application, the input data of the first denoising layer, in addition to the input data of the nth denoising step, also includes object description features describing q objects in the image and attribute description features of the attributes of p objects. That is, in the first denoising layer, denoising is performed based on the object description features describing the q objects in the image, the attribute description features of the attributes of p objects, and the input data of the nth denoising step to obtain the output data of the first denoising layer.

[0072] Step 210, in the mth denoising processing layer, based on a portion of the input data and the object description features of the q objects, denoising processing is performed to obtain a first denoising feature, based on a portion of the input data and the attribute description features of the attributes of the p objects, denoising processing is performed to obtain second denoising features corresponding to the attributes of the p objects, and the first denoising features and the second denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer.

[0073] Among them, the first noise reduction feature refers to the noise reduction feature output by the noise reduction process guided by the object description features of the q objects. It can be understood that by performing the noise reduction process guided by the object description features of the q objects, the object in the image can be generated with the object description features of the q objects as a reference. The second noise reduction feature refers to the noise reduction feature output by the noise reduction process guided by the attribute description features of the attributes of the object. It can be understood that by performing the noise reduction process guided by the attribute description features of the attributes of the object, the attributes of the object in the image can be generated with the attribute description features of the attributes of the object as a reference.

[0074] Specifically, in the mth noise reduction processing layer, the server will perform noise reduction processing based on a portion of the input data, guided by the object description features of the q objects, to obtain the first noise reduction feature, and will perform noise reduction processing based on a portion of the input data, guided by the attribute description features of the attributes of the p objects, to obtain the second noise reduction features corresponding to the attributes of the p objects. On this basis, the output data of the mth noise reduction processing layer can be obtained by fusing the first noise reduction feature and the second noise reduction features corresponding to the attributes of the p objects.

[0075] In a specific application, in the mth noise reduction processing layer, the server will first process a portion of the input data according to the noise feature processing method indicated by the mth noise reduction processing layer to obtain processed input data, and then perform noise reduction processing based on the processed input data and the object description features of the q objects to obtain the first noise reduction feature, and perform noise reduction processing based on the processed input data and the attribute description features of the attributes of the p objects to obtain the second noise reduction features corresponding to the attributes of the p objects.

[0076] Among them, the noise feature processing method indicated by the mth noise reduction processing layer can be configured according to the actual application scenario. For example, the noise feature processing method indicated by the mth noise reduction processing layer can be sampling processing, feature interaction processing, or sampling processing and feature interaction processing. This embodiment does not specifically limit this here. It can be understood that by first processing a part of the input data according to the noise feature processing method indicated by the mth noise reduction processing layer, the important features in a part of the input data can be made more prominent during the processing process, and a more accurate feature representation can be obtained.

[0077] Step 212: using the output data of the Nth denoising step as the denoised image features, decoding the denoised image features, and generating an image.

[0078] Specifically, the server can generate an image by decoding the denoised image features when the output data of the Nth denoising step is used as the denoised image features. It can be understood that decoding the denoised image features means converting the denoised image features into the image space by mapping to generate the image.

[0079] The above-mentioned image generation method, by acquiring image description information indicating the generation of an image, can use the image description information to realize the determination of the object description information of q objects in the image and the attribute description features of the attributes of p objects, and then can obtain random noise features, use the random noise features as the multi-step noisy noise features obtained by subjecting the image features of the image to N iterative noisy steps, perform denoising processing of N iterative denoising steps on the multi-step noisy noise features, perform iterative processing of M denoising processing layers in the nth denoising step, perform denoising processing in the mth denoising processing layer using a portion of input data and the object description features of q objects to obtain a first denoising feature, and perform denoising processing using a portion of input data and the attribute description features of the attributes of p objects to obtain The second denoising features corresponding to the attributes of the p objects are respectively fused with the first denoising features and the second denoising features corresponding to the attributes of the p objects, to obtain the output data of the mth denoising processing layer. In the entire denoising processing process, a parallel denoising mechanism of object description features and attribute description characteristics is constructed, so that the object description features and the attribute description features are first subjected to denoising processing without interfering with each other, and then the first denoising features are modified by using the second denoising features corresponding to the attributes of the p objects in a fusion manner, so that the output data of the Nth denoising step that makes the object's attributes more accurately expressed can be obtained, so that on the basis of using the output data of the Nth denoising step as the image features after denoising, the image features after denoising are decoded to generate an image that makes the object's attributes more accurately expressed, thereby improving the accuracy of image generation.

[0080] In one embodiment, the first denoising feature and the second denoising features corresponding to the attributes of the p objects are fused to obtain the output data of the mth denoising processing layer, including:

[0081] Get the attention weight matrix corresponding to the object description features of q objects;

[0082] For each of the attributes of the p objects, determine the object to which the attribute of the targeted object is bound, and perform weighted filtering on the second denoising feature corresponding to the attribute of the targeted object through the attention weight matrix corresponding to the object description feature of the bound object to obtain a third denoising feature corresponding to the attribute of the targeted object;

[0083] The first denoising feature and the third denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer.

[0084] Among them, the attention weight matrix is ​​a core concept in the attention mechanism, which is used to represent the importance or correlation between each element in the input sequence. It usually refers to the attention allocation of the query sequence to another key sequence. In this embodiment, the query sequence is obtained based on a part of the input data, and the key sequence is obtained based on the object description feature. The weight of each area in the attention weight matrix represents the probability that the corresponding object falls into this area when the image is generated. It can be understood that the higher the weight of the area, the greater the probability that the corresponding object falls into this area when the image is generated.

[0085] Among them, the object bound to the attribute of the targeted object refers to the object in the image to which the attribute of the targeted object is bound, that is, the attribute of the targeted object is a limitation and modification of the bound object. For example, taking the image description information of "a girl wearing a yellow short-sleeved shirt and a black short skirt" as an example, it includes object description information describing three objects in the image (girl, short-sleeved shirt and short skirt), and includes attribute description information describing the attributes (yellow and black) of two objects in the image (short-sleeved shirt and short skirt), wherein the object bound to the attribute yellow is the short-sleeved shirt, and the object bound to the attribute black is the short skirt. It can be understood that in the image description information, the attributes of p objects are bound to p objects, and the object to which the attribute of the targeted object is bound can be determined based on the binding relationship. It should be noted that the attribute of each object is usually bound to one object, but one object can be bound to multiple attributes at the same time.

[0086] Specifically, when it is necessary to fuse the first denoising feature and the second denoising feature corresponding to the attributes of p objects, the server will first obtain the attention weight matrices corresponding to the object description features of the q objects. After obtaining the attention weight matrices corresponding to the object description features of the q objects, for each of the attributes of the p objects, the server will determine the object to which the attribute of the target object is bound, and perform weighted filtering on the second denoising feature corresponding to the attribute of the target object through the attention weight matrix corresponding to the object description feature of the bound object to obtain the third denoising feature corresponding to the attribute of the target object, so as to obtain the output data of the mth denoising processing layer by further fusing the first denoising feature and the third denoising feature corresponding to the attributes of the p objects.

[0087] In a specific application, the attention weight matrices corresponding to the object description features of the q objects can be generated in the process of obtaining the first denoising feature. In this case, the server only needs to directly obtain the first denoising feature and the second denoising feature when fusing the first denoising feature and the second denoising feature. If the attention weight matrices corresponding to the object description features of the q objects are not generated in the process of obtaining the first denoising feature, the server needs to calculate the attention weight matrices corresponding to the object description features of the q objects based on the cross-attention mechanism, a portion of the input data, and the object description features. In a specific application, the attention weight matrices corresponding to the object description features of the q objects can be obtained by performing attention weight calculations on the query vector and the key vector, wherein the query vector is obtained based on a portion of the input data, and the key vector is obtained based on the object description features.

[0088] In specific applications, such as Figure 3 As shown, when weighted filtering is performed on the second denoising feature (latent) corresponding to the attribute of the targeted object, since the second denoising feature corresponding to the attribute of the targeted object may be different in size from the attention weight matrix corresponding to the object description feature of the bound object, the server needs to first adjust the second denoising feature corresponding to the attribute of the targeted object to the same dimension as the attention weight matrix corresponding to the object description feature of the bound object, and then use the attention weight matrix corresponding to the object description feature of the bound object to perform weighted filtering on the second denoising feature of the same dimension. In this way, only the area content with high weight in the attention weight matrix corresponding to the object description feature of the bound object can be retained on the second denoising feature corresponding to the attribute of the targeted object.

[0089] In a specific application, such as Figure 4 As shown, the attributes of p objects are the attributes of 2 objects (such as Figure 4As shown in the figure, the first attribute and the second attribute are bound to the first object and the second object, respectively. It can be seen that for different objects, the areas with high weights in the attention weight matrix corresponding to the object description feature of the object are different. By using the attention weight matrix corresponding to the object description feature of the first object, the second denoising feature corresponding to the first attribute of the same dimension is weighted and filtered, so that only the area content with high weights in the attention weight matrix corresponding to the object description feature of the first object can be retained on the second denoising feature corresponding to the first attribute. At the same time, by using the attention weight matrix corresponding to the object description feature of the second object, the second denoising feature corresponding to the second attribute of the same dimension is weighted and filtered, so that only the area content with high weights in the attention weight matrix corresponding to the object description feature of the second object can be retained on the second denoising feature corresponding to the second attribute.

[0090] In this embodiment, when the corresponding attention weight matrices of the object description features of q objects are obtained, for the attributes of each object in the attributes of p objects, the second noise reduction feature corresponding to the attribute of the targeted object is firstly weighted filtered by using the corresponding attention weight matrix of the object description feature of the bound object, so as to obtain the third noise reduction feature corresponding to the attribute of the targeted object, and then further fuse the noise reduction features, so that the effect of the attribute of the targeted object is limited to the area of ​​the bound object, so that in the process of generating an image, the attribute of the targeted object accurately assists in the generation and binding of the attribute of the bound object without affecting the generation of other objects in the image, which can improve the accuracy of image generation. At the same time, this weighted filtering processing method will also prevent mutual interference between different attributes, further improve the accuracy of the generation of objects with attributes in the image, and thus improve the accuracy of image generation.

[0091] In one embodiment, the first denoising feature and the third denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer, including:

[0092] The third denoising features corresponding to the attributes of the p objects are fused to obtain the attribute denoising features corresponding to the attributes of the p objects;

[0093] The first denoising feature and the attribute denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer.

[0094] Specifically, on the basis of obtaining the third noise reduction features corresponding to the attributes of the p objects, the server needs to fuse the first noise reduction features and the third noise reduction features corresponding to the attributes of the p objects to obtain the output data of the mth noise reduction processing layer. During the fusion, the server will first fuse the third noise reduction features corresponding to the attributes of the p objects to obtain the attribute noise reduction features corresponding to the attributes of the p objects. It can be understood that the attribute noise reduction features at this time represent the noise reduction features integrated after the noise reduction processing is performed using the attributes of the p objects. On this basis, by fusing the first noise reduction features and the attribute noise reduction features corresponding to the attributes of the p objects, the attribute noise reduction features corresponding to the attributes of the p objects can be used to modify the first noise reduction features to obtain the output data of the mth noise reduction processing layer.

[0095] In a specific application, this embodiment does not limit the method of fusing the third noise reduction features corresponding to the attributes of the p objects, as long as the fusion can be achieved. For example, the server can merge the third noise reduction features corresponding to the attributes of the p objects. Figure 5 As shown, the attributes of p objects are 2 attributes (such as Figure 5 As an example, in the third denoising feature corresponding to the first attribute, only the area content 502 with high weight in the attention weight matrix corresponding to the object description feature of the bound first object is retained, and in the third denoising feature corresponding to the second attribute, only the area content 504 with high weight in the attention weight matrix corresponding to the object description feature of the bound second object is retained. The server can obtain the attribute denoising features corresponding to the first attribute and the second attribute by merging the third denoising feature corresponding to the first attribute and the third denoising feature corresponding to the second attribute.

[0096] In a specific application, if the feature dimension of the attribute denoising features corresponding to the attributes of the p objects is the same as the first denoising feature, the server can directly fuse them. If the feature dimension of the attribute denoising features corresponding to the attributes of the p objects is different from the first denoising feature, the server needs to first adjust the feature dimension of the attribute denoising features corresponding to the attributes of the p objects to be the same as the first denoising feature, and then fuse the attribute denoising features corresponding to the attributes of the p objects with the first denoising feature to obtain the output data of the mth denoising processing layer.

[0097] In a specific application, this embodiment does not limit the method of fusing the first noise reduction feature and the attribute noise reduction feature corresponding to the attributes of the p objects, as long as the fusion can be achieved. For example, the server can fuse the two features by adding the first noise reduction feature and the attribute noise reduction feature corresponding to the attributes of the p objects, that is, by adding up the values ​​of the two features to achieve fusion. Figure 6As shown, taking the example that the feature dimension of the attribute denoising feature corresponding to the attributes of p objects is different from the first denoising feature, the server will first adjust the feature dimension of the attribute denoising feature corresponding to the attributes of the p objects to be the same as the first denoising feature, and then add the first denoising feature and the attribute denoising feature corresponding to the attributes of the p objects to obtain the output data of the mth denoising processing layer.

[0098] In this embodiment, by fusing the third denoising features corresponding to the attributes of p objects, the denoising features after denoising processing on the attributes of the p objects can be integrated. On the basis of obtaining the attribute denoising features, by fusing the first denoising features and the attribute denoising features corresponding to the attributes of the p objects, the attribute denoising features corresponding to the attributes of the p objects can be used to modify the first denoising features. The denoising features obtained by denoising the attribute description features of the attributes of the p objects are accurately and controllably fused to the regional content of the bound objects to obtain the output data of the mth denoising processing layer, thereby using the output data of the mth denoising processing layer to achieve accurate and controllable image generation.

[0099] In one embodiment, at the mth denoising layer, denoising is performed based on a portion of the input data and the object description features of q objects to obtain a first denoising feature, and denoising is performed based on a portion of the input data and the attribute description features of the attributes of p objects to obtain second denoising features corresponding to the attributes of the p objects, including:

[0100] At the mth noise reduction processing layer, a portion of the input data is sampled according to the sampling method indicated by the mth noise reduction processing layer to obtain a sampling noise feature;

[0101] Perform feature interaction processing on the features in the sampled noise features to obtain the noise features after interaction;

[0102] Based on the noise feature after interaction and the object description features of the q objects, a noise reduction process is performed to obtain a first noise reduction feature;

[0103] Based on the noise features after interaction and the attribute description features of the attributes of the p objects, noise reduction processing is performed respectively to obtain second noise reduction features corresponding to the attributes of the p objects.

[0104] Among them, the sampling mode indicated by the mth noise reduction processing layer refers to the mode of sampling and processing a part of the input data in the mth noise reduction processing layer, which can be specifically one of upsampling and downsampling. It should be noted that the sampling mode indicated by the mth noise reduction processing layer, specifically upsampling or downsampling, is determined according to the function of the mth noise reduction processing layer. If the function of the mth noise reduction processing layer is to perform noise reduction based on downsampling, the indicated sampling mode is downsampling. If the function of the mth noise reduction processing layer is to perform noise reduction based on upsampling, the indicated sampling mode is upsampling. Usually in the nth noise reduction step, noise reduction is first performed based on downsampling, and then based on upsampling.

[0105] For example, Figure 7 As shown, taking the nth denoising step, which includes 8 denoising processing layers as an example, the function of the first 4 denoising processing layers is to perform denoising processing based on downsampling, and the function of the last 4 denoising processing layers is to perform denoising processing based on upsampling, then if m is any positive integer from 1 to 4, the indicated sampling method is downsampling, and if m is any positive integer from 5 to 8, the indicated sampling method is upsampling. Further, it should be noted that in order to prevent the loss of important information during downsampling, a shortcut connection is usually added between the last denoising processing layer whose function is downsampling and the first denoising processing layer whose function is upsampling, so that the output data of the last denoising processing layer whose function is downsampling can be accurately input to the first denoising processing layer whose function is upsampling.

[0106] The feature interaction processing refers to combining two or more features through certain operations or logical relationships to create a new feature. In this embodiment, the features in the sampled noise features are combined to create a new feature, namely the interactive noise feature.

[0107] Specifically, in the mth denoising layer, the server will sample a portion of the input data according to the sampling method indicated by the mth denoising layer to obtain the sampled noise feature, and then perform feature interaction processing on the features in the sampled noise feature based on the self-attention mechanism to obtain the post-interaction noise feature, and then perform denoising based on the post-interaction noise feature and the object description features of the q objects to obtain the first denoising feature, and perform denoising based on the post-interaction noise feature and the attribute description features of the attributes of the p objects to obtain the second denoising feature corresponding to the attributes of the p objects. Among them, the self-attention mechanism, also known as the internal attention mechanism, is a method for calculating the relationship between different positions within the sequence in a sequence model. This mechanism allows the model to dynamically focus on the most important part of the sequence for the current task when processing the sequence. In this embodiment, the relationship between different features within the sampled noise feature is calculated, so that the most important feature for denoising in the sampled noise feature can be dynamically focused. It can be understood that by using the self-attention mechanism, all semantic representations in a part of the input data can be further integrated, and the performance of the generated object can be made clearer, thereby improving the denoising ability of a part of the input data.

[0108] In a specific application, at the mth denoising layer, the server can sample a portion of the input data based on the residual network, and the denoising process at the mth denoising layer can be as follows: Figure 8 As shown, the server first samples a portion of the input data based on the residual network in the sampling mode indicated by the m-th denoising processing layer to obtain sampled noise features, and then performs feature interaction processing on the features in the sampled noise features based on the self-attention mechanism to obtain interactive noise features, and then performs denoising based on the interactive noise features and the object description features of q objects to obtain a first denoising feature, and then performs denoising based on the interactive noise features and the attribute description features of the attributes of p objects to obtain second denoising features corresponding to the attributes of the p objects, and then fuses the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain the output data of the m-th denoising processing layer.

[0109] In this embodiment, in the mth denoising layer, a portion of the input data is sampled according to the sampling method indicated by the mth denoising layer to obtain a sampled noise feature, and the features in the sampled noise feature are subjected to feature interaction processing to obtain an interactive noise feature, so that important features in a portion of the input data can be more prominent during the processing process and a more accurate feature representation can be obtained. Further, on the basis of the more accurate feature representation, i.e., the interactive noise feature, denoising can be performed guided by the object description features of the q objects to obtain a more accurate first denoising feature, and denoising can be performed guided by the attribute description features of the attributes of the p objects to obtain a more accurate second denoising feature, so that the first denoising feature can be modified by using the second denoising features corresponding to the attributes of the p objects in a fusion manner, so that the output data of the Nth denoising step that makes the object more accurately represented in terms of attributes can be obtained, so that on the basis of using the output data of the Nth denoising step as the image feature after denoising, the image feature after denoising can be decoded to generate an image that makes the object more accurately represented in terms of attributes, thereby improving the image generation accuracy.

[0110] In one embodiment, noise reduction processing is performed based on the noise feature after interaction and the object description features of q objects, and the first noise reduction feature obtained includes:

[0111] Based on the cross attention mechanism, feature interaction processing is performed on the interacted noise features and the object description features of q objects to obtain the first denoising feature.

[0112] Among them, the cross attention mechanism, also known as the mutual attention mechanism, is a method for calculating the relationship between different sequences in a sequence model. It is particularly useful when processing a pair of sequence data, such as in tasks such as machine translation, text summarization, and question-answering systems. The cross attention mechanism allows the model to focus on the most relevant part of the input sequence when generating the output. In this embodiment, the relationship between the noise features after the interaction and the object description features of q objects is calculated, so that the most relevant part of the noise features after the interaction and the object description features of q objects can be focused on to obtain the first noise reduction feature.

[0113] Specifically, when noise reduction processing is performed guided by the object description features of q objects, the server will perform feature interaction processing on the interacted noise features and the object description features of the q objects based on the cross-attention mechanism, so that it can focus on the most relevant parts of the interacted noise features and the object description features of the q objects to obtain the first noise reduction feature.

[0114] In a specific application, when q is equal to 1, the server performs feature interaction processing on the noise features after interaction and the object description features of a single object based on a cross-attention mechanism, and can obtain the noise reduction features corresponding to the object description features of the single object. At this time, the noise reduction features corresponding to the object description features of the single object can be directly used as the first noise reduction feature. In the case where q is not equal to 1, the object description features of the q objects include the object description features of at least two objects. The server needs to first perform feature interaction processing on the noise features after interaction and the object description features of each object in the object description features of the q objects based on a cross-attention mechanism, respectively, to obtain the noise reduction features corresponding to the object description features of each object, and then obtain the first noise reduction feature based on the noise reduction features corresponding to the object description features of each object.

[0115] In this embodiment, the noise features after interaction and the object description features of q objects are subjected to feature interaction processing based on a cross-attention mechanism, and the relationship between the noise features after interaction and the object description features of q objects is calculated, so that the most relevant parts of the noise features after interaction and the object description features of q objects can be focused on to achieve accurate noise reduction and obtain the first noise reduction feature.

[0116] In one embodiment, based on the cross attention mechanism, feature interaction processing is performed on the noise feature after interaction and the object description features of q objects, and the first noise reduction feature is obtained, including:

[0117] Based on the cross attention mechanism, feature interaction processing is performed on the noise features after interaction and the object description features of each object in the object description features of q objects, and the corresponding noise reduction features of the object description features of each object are obtained;

[0118] The denoising features corresponding to the object description features of the q objects are fused to obtain the first denoising feature.

[0119] Specifically, in the process of obtaining the first denoising feature, the server will first perform feature interaction processing on the interacted noise features and the object description features of each object in the object description features of the q objects based on the cross-attention mechanism to obtain the denoising features corresponding to the object description features of each object, and then fuse the denoising features corresponding to the object description features of the q objects to obtain the first denoising feature.

[0120] In a specific application, this embodiment does not limit the method of fusing the noise reduction features corresponding to the object description features of the q objects, as long as the fusion of the noise reduction features can be achieved. For example, in this embodiment, the noise reduction features corresponding to the object description features of the q objects can be fused by merging, pooling, etc.

[0121] In this embodiment, by performing feature interaction processing on the interacted noise features and the object description features of each object in the object description features of q objects based on the cross-attention mechanism, the corresponding noise reduction features of the object description features of each object can be obtained, and then the first noise reduction feature can be obtained by fusing the corresponding noise reduction features of the object description features of the q objects.

[0122] In one embodiment, based on the cross attention mechanism, feature interaction processing is performed on the noise features after interaction and the object description features of each object in the object description features of q objects, and the noise reduction features corresponding to the object description features of each object are obtained, including:

[0123] For the object description features of each object in the object description features of the q objects, convolve the object description features of the targeted object by generating a matrix using a pre-trained first vector based on a cross attention mechanism to obtain a first key vector and a first value vector, and convolve the interacted noise features to obtain a first query vector;

[0124] Calculate the attention weight based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description feature of the target object;

[0125] The first value vector is weighted according to the attention weight matrix corresponding to the object description feature of the targeted object to obtain the noise reduction feature corresponding to the object description feature of the targeted object.

[0126] The pre-trained first vector generation matrix refers to a pre-trained matrix used to generate a first key vector, a first value vector, and a first query vector, including a first key matrix, a first value matrix, and a first query matrix, wherein the first key matrix is ​​used to generate the first key vector, the first value matrix is ​​used to generate the first value vector, and the first query matrix is ​​used to generate the first query vector. The pre-training method of the pre-trained first vector generation matrix can be configured according to the actual application scenario, and this is not specifically limited here in this embodiment.

[0127] Specifically, for the object description features of each object in the object description features of q objects, the server convolves the object description features of the targeted object through the first key matrix in the pre-trained first vector generation matrix based on the cross-attention mechanism to obtain the first key vector, and convolves the object description features of the targeted object through the first value matrix in the pre-trained first vector generation matrix to obtain the first value vector, and convolves the interacted noise features through the first query matrix in the pre-trained first vector generation matrix to obtain the first query vector.

[0128] Specifically, based on the first key vector, the first value vector and the first query vector, the server can perform attention weight calculation based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description feature of the targeted object, and then perform weighted processing on the first value vector according to the attention weight matrix corresponding to the object description feature of the targeted object to obtain a noise reduction feature corresponding to the object description feature of the targeted object.

[0129] In a specific application, the flowchart of the feature interaction processing in this embodiment can be as follows: Fig. 9 As shown, the server will convolve the object description features of the targeted object to obtain the first key vector ( Fig. 9 K in the figure) and the first value vector ( Fig. 9 ), and convolve the noise features after interaction to obtain the first query vector ( Fig. 9 denoted by Q in the figure), by first multiplying and then normalizing, the attention weight is calculated based on the first key vector and the first query vector to obtain the attention weight matrix corresponding to the object description feature of the target object, and the first value vector is weighted according to the attention weight matrix corresponding to the object description feature of the target object (by Fig. 9 By multiplying the matrices shown in FIG. 1 ), the denoising features corresponding to the object description features of the object are obtained.

[0130] In this embodiment, by generating a matrix through pre-training the first vector based on the cross-attention mechanism, convolving the object description features of the targeted object and convolving the noise features after interaction, the calculation of the first key vector, the first value vector and the first query vector can be realized. Then, on the basis of calculating the attention weights based on the first key vector and the first query vector and obtaining the attention weight matrix corresponding to the object description features of the targeted object, the first value vector is weighted according to the attention weight matrix corresponding to the object description features of the targeted object, so as to realize the determination of the noise reduction features corresponding to the object description features of the targeted object.

[0131] In one embodiment, based on the noise features after interaction and the attribute description features of the attributes of the p objects, noise reduction processing is performed respectively to obtain the second noise reduction features corresponding to the attributes of the p objects respectively, including:

[0132] Based on the cross attention mechanism, feature interaction processing is performed on the interacted noise features and the attribute description features of the attributes of the p objects respectively to obtain the second denoising features corresponding to the attributes of the p objects.

[0133] Among them, the cross attention mechanism, also known as the mutual attention mechanism, is a method for calculating the relationship between different sequences in a sequence model. It is particularly useful when processing a pair of sequence data, such as in tasks such as machine translation, text summarization, and question-answering systems. The cross attention mechanism allows the model to focus on the most relevant part of the input sequence when generating output. In this embodiment, the relationship between the noise feature after interaction and the attribute description feature of the attribute is calculated, so that the most relevant part of the noise feature after interaction and the attribute description feature of the attribute can be focused on to obtain the second noise reduction feature.

[0134] Specifically, when noise reduction processing is performed guided by the attribute description features of the attributes of p objects, the server will perform feature interaction processing on the noise features after interaction and the attribute description features of the attributes of the p objects based on the cross-attention mechanism, so as to focus on the most relevant parts of the noise features after interaction and the attribute description features of the attributes, so as to obtain the second noise reduction features corresponding to the attributes of the p objects.

[0135] In this embodiment, by performing feature interaction processing on the noise features after interaction and the attribute description features of the attributes of p objects based on a cross-attention mechanism, the relationship between the noise features after interaction and the attribute description features of the attributes is calculated, so that the most relevant parts of the noise features after interaction and the attribute description features of the attributes can be focused on to achieve accurate noise reduction and obtain the second noise reduction feature.

[0136] In one embodiment, based on the cross attention mechanism, feature interaction processing is performed on the noise features after interaction and the attribute description features of the attributes of the p objects, and the second noise reduction features corresponding to the attributes of the p objects are obtained, including:

[0137] For the attribute description features of the attributes of each object in the attribute description features of the attributes of the p objects, the attribute description features of the attributes of the objects are convolved by generating a matrix of the pre-trained second vector based on the cross attention mechanism to obtain a second key vector and a second value vector, and the noise features after interaction are convolved to obtain a second query vector;

[0138] Attention weights are calculated based on the second key vector and the second query vector to obtain an attention weight matrix corresponding to the attribute description features of the attributes of the targeted object;

[0139] The second value vector is weighted according to the attention weight matrix corresponding to the attribute description feature of the attribute of the targeted object to obtain a second noise reduction feature corresponding to the attribute of the targeted object.

[0140] The pre-trained second vector generation matrix refers to a pre-trained matrix for generating a second key vector, a second value vector, and a second query vector, including a second key matrix, a second value matrix, and a second query matrix, wherein the second key matrix is ​​used to generate a second key vector, the second value matrix is ​​used to generate a second value vector, and the second query matrix is ​​used to generate a second query vector. The pre-training method of the pre-trained second vector generation matrix can be configured according to the actual application scenario, and this is not specifically limited here in this embodiment.

[0141] Specifically, for the attribute description features of the attributes of each object in the attribute description features of the attributes of the p objects, the server will convolve the attribute description features of the attributes of the targeted objects through the second key matrix in the pre-trained second vector generation matrix based on the cross-attention mechanism to obtain a second key vector, and convolve the attribute description features of the attributes of the targeted objects through the second value matrix in the pre-trained second vector generation matrix to obtain a second value vector, and convolve the interacted noise features through the second query matrix in the pre-trained second vector generation matrix to obtain a second query vector.

[0142] Specifically, based on the second key vector, the second value vector and the second query vector, the server can perform attention weight calculation based on the second key vector and the second query vector to obtain the attention weight matrix corresponding to the attribute description features of the attributes of the targeted object, and then perform weighted processing on the second value vector according to the attention weight matrix corresponding to the attribute description features of the attributes of the targeted object to obtain the second noise reduction feature corresponding to the attributes of the targeted object.

[0143] In a specific application, the flowchart of the feature interaction processing in this embodiment can be as follows: Fig.10 As shown, the server convolves the attribute description features of the attributes of the targeted object to obtain the second key vector ( Fig.10 K in the figure) and the second value vector ( Fig.10 ), and convolve the noise features after interaction to obtain the second query vector ( Fig.10 denoted by Q in the figure), the attention weight is calculated based on the second key vector and the second query vector by multiplying and then normalizing, and the attention weight matrix corresponding to the attribute description feature of the attribute of the target object is obtained. According to the attention weight matrix corresponding to the attribute description feature of the attribute of the target object, the second value vector is weighted (by Fig.10 By multiplying the matrices shown in FIG. 1 ), a second denoising feature corresponding to the attribute of the target object is obtained.

[0144] In this embodiment, by generating a matrix through pre-training the second vector based on the cross-attention mechanism, convolving the attribute description features of the attributes of the targeted object and convolving the noise features after interaction, the calculation of the second key vector, the second value vector and the second query vector can be realized, and then the attention weight calculation can be performed based on the second key vector and the second query vector to obtain the attention weight matrix corresponding to the attribute description features of the attributes of the targeted object. By weighting the second value vector according to the attention weight matrix corresponding to the attribute description features of the attributes of the targeted object, the second noise reduction feature corresponding to the attributes of the targeted object can be determined.

[0145] In one embodiment, determining object description features describing q objects in the image and attribute description features of the attributes of p objects based on the image description information includes:

[0146] The object description information describing the q objects in the image in the image description information is encoded respectively to obtain the object description features describing the q objects in the image, and the attribute description information describing the attributes of the p objects in the image description information is encoded respectively to obtain the attribute description features of the attributes of the p objects.

[0147] Specifically, the image description information includes object description information describing q objects in the image and attribute description information describing the attributes of p objects in the image. When determining the object description features and the attribute description features based on the image description information, the server will encode the object description information describing the q objects in the image in the image description information respectively to obtain the respective encoding features of the object description information of the q objects, and then merge the respective encoding features of the object description information of the q objects to obtain the object description features describing the q objects in the image, and encode the attribute description information describing the attributes of the p objects in the image in the image description information respectively to obtain the respective attribute description features of the attributes of the p objects.

[0148] In a specific application, since both the object description information and the attribute description information are in text form, the encoding in this embodiment can be specifically text encoding, and the text data can be converted into a format that can be processed by a computer through text encoding. In this embodiment, the object description information is converted into encoding features, and the attribute description information is converted into attribute description features. For example, the encoding features here can be specifically encoding feature vectors, and the attribute description features can be specifically attribute description feature vectors.

[0149] In a specific application, the server can encode object description information and attribute description information through a pre-trained natural language model for text encoding. The natural language model refers to a model built based on natural language processing, which is a discipline that uses computer technology to analyze, understand and process natural language with language as the object.

[0150] Among them, the pre-trained natural language model for text encoding can be configured according to the actual application scenario. For example, the natural language model can be specifically a CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is an advanced deep learning model that can understand text and images. It is a multimodal model that has been trained to match text and images, and has learned to recognize the content in the image and the language describing the image. The main goal is to learn to match images and texts through contrastive learning. This is done by training the model to predict which image belongs to a given text, and vice versa. During the training process, the model learns to encode images and texts into a unified vector space, which enables it to understand the relationship between them linguistically and visually. In this way, CLIP can recognize elements such as objects, scenes, actions in the image, and can also understand text related to the image, such as labels, descriptions, titles, etc.

[0151] In a specific application, in the image description information, the attributes of p objects are bound to p objects. The binding relationship can be configured by the user who provides the image description information when generating the image description information, or can be obtained by the server by analyzing the image description information after obtaining the image description information. In this embodiment, the method for determining the binding relationship is not limited, as long as the determination of the binding relationship can be achieved.

[0152] For example, when generating image description information, users can follow the following steps: Fig.11 Configure the binding relationship in the format shown. Fig.11 In the example, each row in the left column is used to input object description information of an object, and each row in the right column is used to input attribute description information of an attribute of the object in the same row. When the server receives image description information input in such a format, it can identify and establish a binding relationship based on the image description information.

[0153] For example, take the image description information of "a girl wearing a yellow short-sleeved shirt and a black short skirt" as an example. The user configures the binding relationship according to the format. The input image description information can be in the form of Fig.12As shown, each row of the left column is used to input object description information (girl, short sleeves and short skirt) of an object (girl, short sleeves and short skirt respectively), the second row of the right column is used to input attribute description information (yellow) of the attribute of the short sleeve (yellow), and the third row of the right column is used to input attribute description information (black) of the attribute of the short skirt (black). It can be understood that in Fig.12 In the game, since girls have no additional attributes, the attributes row is empty.

[0154] In this embodiment, based on the image description information obtained, the object description information describing the q objects in the image in the image description information is respectively encoded to obtain the object description features describing the q objects in the image, and the attribute description information describing the attributes of the p objects in the image in the image description information is respectively encoded to obtain the attribute description features of the attributes of the p objects.

[0155] In one embodiment, the image is determined by a pre-trained text-based image generation model; the text-based image generation model includes a text encoder, a denoising network, and a decoder; the denoising network includes N denoisers; each denoiser includes M denoising processing blocks;

[0156] The text encoder is used to obtain image description information indicating the generated image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information;

[0157] The denoising network is used to obtain random noise features, use the random noise features as multi-step denoising noise features obtained by subjecting the image features of the image to N iterative denoising steps, perform denoising processing on the multi-step denoising noise features in N iterative denoising steps, and obtain output data of the Nth denoising step; wherein the nth denoiser is used to perform iterative processing of M denoising processing layers through M denoising processing blocks in the nth denoising step; the processing of the mth denoising processing layer is performed by the mth denoising processing block, the mth denoising processing block is used to perform denoising processing based on a portion of input data and object description features of q objects to obtain a first denoising feature, perform denoising processing based on a portion of input data and attribute description features of respective attributes of p objects to obtain second denoising features corresponding to respective attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to respective attributes of the p objects to obtain output data of the mth denoising processing layer;

[0158] The decoder is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

[0159] Specifically, the pre-trained text-based image generation model refers to a model used to generate images, such as Fig.13 As shown, the text-based image generation model includes a text encoder, a denoising network and a decoder. The denoising network includes N denoisers, each denoiser includes M denoising processing blocks, wherein the text encoder is used to obtain image description information indicating the generated image, determine object description features describing q objects in the image based on the image description information, and attribute description features of the attributes of p objects, and the denoising network is used to obtain random noise features (such as Fig.13 As shown, the random noise feature can be obtained by obtaining random noise and extracting features from the random noise), the random noise feature is used as the multi-step noise feature obtained by subjecting the image feature of the image to N iterative noise adding steps, the multi-step noise feature is subjected to N iterative noise reduction steps of noise reduction processing, and the output data of the Nth noise reduction step is obtained, wherein the nth denoiser is used to perform iterative processing of M noise reduction processing layers through M noise reduction processing blocks in the nth noise reduction step; the processing of the mth noise reduction processing layer is performed by the mth noise reduction processing block, and the mth noise reduction processing block is used to, based on a Part of the input data and the object description features of q objects are used for denoising to obtain a first denoising feature; based on part of the input data and the attribute description features of the attributes of p objects, denoising is performed to obtain second denoising features corresponding to the attributes of the p objects; the first denoising features and the second denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer; n is a positive integer not greater than N, and the decoder is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

[0160] In specific applications, such as Fig.14 As shown, the mth denoising block includes a residual network, a first attention network based on a self-attention mechanism, a second attention network based on a cross-attention mechanism, q third attention networks in parallel with the second attention network based on the cross-attention mechanism, and a fusion network based on mask fusion.

[0161] In a specific application, when the mth denoising block processes the mth denoising layer, the server samples a portion of the input data through the residual network to obtain sampled noise features, and performs feature interaction processing on the features in the sampled noise features through the first attention network to obtain interactive noise features.

[0162] After obtaining the post-interaction noise feature, the server will use the second attention network to target the object description features of each object in the object description features of the q objects, and generate a matrix based on the pre-trained first vector based on the cross-attention mechanism, convolve the object description features of the targeted object to obtain a first key vector and a first value vector, and convolve the post-interaction noise feature to obtain a first query vector, perform attention weight calculation based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description features of the targeted object, perform weighted processing on the first value vector according to the attention weight matrix corresponding to the object description features of the targeted object, obtain a denoising feature corresponding to the object description features of the targeted object, and fuse the denoising features corresponding to the object description features of the q objects to obtain a first denoising feature.

[0163] At the same time, the server will process the attribute description features of the attributes of the p objects respectively through p third attention networks. For the attribute description features of the attributes of each object in the attribute description features of the attributes of the p objects, the server will use the third attention network to generate a matrix based on the pre-trained second vector of the cross-attention mechanism, convolve the attribute description features of the attributes of the targeted objects, obtain a second key vector and a second value vector, and convolve the noise features after interaction to obtain a second query vector. Attention weights are calculated based on the second key vector and the second query vector to obtain the attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects. According to the attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects, the second value vector is weighted to obtain the second noise reduction feature corresponding to the attributes of the targeted objects.

[0164] After obtaining the first denoising feature and the second denoising feature corresponding to the attributes of the p objects, the server will obtain the attention weight matrices corresponding to the object description features of the q objects through a fusion network based on mask fusion, determine the object bound to the attribute of each object among the attributes of the p objects, perform weighted filtering on the second denoising feature corresponding to the attribute of the object through the attention weight matrix corresponding to the object description feature of the bound object, obtain the third denoising feature corresponding to the attribute of the object, fuse the third denoising features corresponding to the attributes of the p objects, obtain the attribute denoising feature corresponding to the attributes of the p objects, fuse the first denoising feature and the attribute denoising feature corresponding to the attributes of the p objects, and obtain the output data of the mth denoising processing layer.

[0165] In this embodiment, a text-based image generation model including a text encoder, a denoising network, and a decoder can be used to accurately infer an image based on image description information, thereby improving the accuracy of image generation.

[0166] The present application proposes an image generation method. The image generation method can, on the basis of obtaining image description information indicating the generated image, determine the object description features describing q objects in the image based on the image description information, and the attribute description features of the attributes of p objects respectively, and then, by constructing a parallel denoising mechanism of the object description features and the attribute description features, subject the object description features and the attribute description features to denoising processing without interfering with each other, and then, by fusion, modify the first denoising features using the second denoising features corresponding to the attributes of the p objects respectively, so as to obtain the output data of the Nth denoising step that makes the object's attributes more accurately expressed, so that, on the basis of using the output data of the Nth denoising step as the image features after denoising, by decoding the image features after denoising, an image that makes the object's attributes more accurately expressed can be generated, thereby improving the accuracy of image generation.

[0167] In one embodiment, the image generation method of the present application is described by taking the generation of an image by a pre-trained text-based image generation model as an example. Specifically, taking the image description information of "a girl wearing a yellow short-sleeved shirt and a black short skirt" as an example, the overall framework of the image generation method of the present application is as follows: Fig.15 As shown, the text-based image generation model includes a text encoder, a denoising network and a decoder, and the denoising network includes N denoisers ( Fig.15 Only the specific structure of one denoising processor is shown in the figure, and the specific structure of each denoising processor is the same), each of the denoisers includes M denoising processing blocks, each denoising processing block includes a residual network, a self-attention calculation module (i.e., a first attention network based on a self-attention mechanism), a parallel cross-attention calculation module (i.e., a second attention network based on a cross-attention mechanism, q third attention networks in parallel with the second attention network based on a cross-attention mechanism), and a fusion network based on mask fusion.

[0168] Among them, the text-based image generation model is a model built on the basis of the basic diffusion model and an innovative mechanism, such as Fig.15As shown, in the text-based image generation model, a parallel cross-attention calculation module (i.e., a second attention network based on a cross-attention mechanism, and q third attention networks in parallel with the second attention network based on a cross-attention mechanism) is added, as well as a fusion network that uses the attention weight matrix corresponding to the object description feature of the object as a mask to perform mask-based fusion. By using the parallel cross-attention calculation module, the object description information describing the object in the image and the attribute description information describing the attributes of the object in the image can be generated based on their own semantics, respectively. Here, mutual interference between attributes can be avoided, so that the generation of attributes is accurate. On this basis, the attention weight matrix corresponding to the object description feature of the object is used as a mask to perform mask-based fusion. Based on the attention weight calculation structure corresponding to the object description feature of the object, the semantics of the object (i.e., the object description feature) and the semantics of the attribute of the object (i.e., the attribute description feature) can be fused under the premise of the weight mask, so as to achieve accurate control of the attribute generation only to its corresponding object (i.e., the bound object), avoiding excessive pollution of other object generation.

[0169] Specifically, Fig.15 As shown, first, it is necessary to regularize the text-image description word sequence (i.e., image description information) input by the user, so that the input is divided into two categories: objects and object attributes. At the same time, it is necessary to correspond to the attributes of the objects to which each object is matched and bound. This is mainly to enable the text-based image generation model to know the mapping relationship between the attributes of the object and the object, so as to facilitate the selection of the attention weight matrix corresponding to the object description feature of the object during mask-based fusion. That is, for each object attribute, only the attention weight matrix corresponding to the object description feature of the object bound to the attribute of the target object is used as a mask for mask-based fusion.

[0170] The image description information can be expressed as follows: Fig.15 Each row in the left column is used to input the object description information of the object, and each row in the right column is used to input the attribute description information of the object in the same row. In this embodiment, the image description information is "a girl wearing a yellow short-sleeved shirt and a black short skirt", so the format of the input image description information can be as follows Fig.15 As shown, each row of the left column is used to input object description information (girl, short sleeves and short skirt respectively) of an object (girl, short sleeves and short skirt respectively), the second row of the right column is used to input attribute description information (yellow) of the attributes of the short sleeves (yellow), and the third row of the right column is used to input attribute description information (black) of the attributes of the short skirt (black).

[0171] Specifically, after obtaining the image description information, the text-based image generation model will input the object description information (girl, short sleeves and short skirt) of the three objects (girl, short sleeves and short skirt) into the text encoder to generate object description features of the three objects, and input the object description features of the three objects into the parallel cross-attention calculation module after the transformation in the text-based image generation model. At the same time, based on the text-based image generation model, the attribute description information (yellow and black) of the attributes of the two objects (yellow and black) will be input into the text encoder respectively to generate the attribute description features of the attributes of the two objects, and input them into the parallel cross-attention calculation module together for binding calculation. It should be noted that the text encoder that processes the object description information and the attribute description information can be the same, that is, in this embodiment, the object description features of the three objects and the attribute description features of the attributes of the two objects can be obtained by reusing the text encoder.

[0172] Specifically, in the parallel cross-attention calculation module, the object description features of the three objects and the attribute description features of the attributes of the two objects are calculated in parallel and independently, and will not interfere with each other. Then, the corresponding semantic features (i.e., the first denoising feature and the second denoising feature corresponding to each of the two attributes) can be generated for the objects and attributes, and finally input into the fusion network based on mask fusion. According to the object mapped by the attribute of each object, that is, the object bound to the attribute of each object (specifically in this embodiment, for the attributes of each object in the attributes of the two objects, the object bound to the attribute of the targeted object is determined, where the object bound to the attribute "yellow" is "short sleeves", and the object bound to the attribute "black" is "short skirt"), the object description features of the bound objects are extracted, and the attention weight matrix determined in the process of calculating the cross-attention is used as a mask. Based on the mask weight, the corresponding semantic features generated by the attributes (i.e., the second denoising feature corresponding to the attributes) are bound to the semantics of the corresponding matching objects, thereby completing the precise generation of objects with attributes in the Wensheng graph, such as Fig.15 As shown, the generated image is "a girl wearing a yellow short-sleeved shirt and a black short skirt".

[0173] Specifically, the text-based image generation model of the present application is mainly divided into three parts, namely the infrastructure network model, the parallel cross-attention calculation module and the mask-based fusion network. Each part is described below.

[0174] Among them, the backbone generation network based on the text-generated image model in this application is constructed using a text-generated image diffusion model (specifically, a stable diffusion model). Taking the diffusion model as an example, the basic architecture of the backbone generation network is introduced. The stable diffusion model is a variant of the diffusion model, called the "latent diffusion model" (LDM). The diffusion model was introduced in 2015. Its purpose is to eliminate the continuous application of Gaussian noise to the training images. It can be regarded as a series of denoising autoencoders. Fig.16 As shown in the figure, the stable diffusion model consists of the following parts: text encoder, autoencoder (VAE, Variational Auto-Encoder) and U-Net network. Instead of learning to denoise image data, VAE is trained to convert the image into a low-dimensional latent space. The process of adding and removing Gaussian noise is applied to this latent representation, and then the final denoised output is decoded into the pixel space. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation. Each denoising step is completed by a U-Net network architecture containing a residual network, and the latent representation is obtained by denoising in the reverse direction from the forward diffusion. Finally, the VAE decoder generates the output image by converting the representation back to the pixel space.

[0175] Among them, the role of the text encoder is to convert the input concept text sequence into an embedding space that U-Net can understand, which is usually referred to as text embedding in the model (text embedding, in this embodiment, specifically refers to the object description features describing the q objects in the image, and the attribute description features of the attributes of the p objects). What is usually used here is the text encoder module in the CLIP model, which is a simple transformer-based encoder that maps the tag sequence to the potential text embedding sequence. In order to be able to calculate a good semantic representation of the concept text, so as to obtain better expected output.

[0176] Among them, the autoencoder (VAE) consists of two main parts: the VAE encoder and the VAE decoder. The VAE encoder will convert the image into a low-dimensional latent representation (during the model training process), which will serve as the input of the next component U-Net network. The VAE decoder will do the opposite, it will convert the latent representation back to the image. During the diffusion training process, the encoder is used to obtain the latent representation of the input image during the forward diffusion process. During the inference process, the VAE encoder is no longer needed because the input to the U-Net network module is a pure noise latent (in this embodiment, random noise features), and then the VAE decoder will convert the latent latent signal calculated by the U-Net network (in this embodiment, the denoised image features) back to the image.

[0177] Among them, the U-Net network is one of the earlier algorithms that uses a fully convolutional network for semantic segmentation. The use of a symmetrical U-shaped structure containing a compression path and an expansion path was very innovative at the time, and to a certain extent influenced the design of several subsequent segmentation networks. The name of the network is also taken from its U-shaped shape. At the same time, the U-Net network is also the basic backbone model of the stable diffusion model. The self-attention module and the cross-attention calculation module are added to the U-Net network, so that the model denoising function can be built. It also includes an encoder and a decoder, both of which are composed of ResNet blocks (residual blocks) and attention blocks. The encoder compresses the image representation into a low-resolution image (through a downsampling sampling process), and the decoder decodes the low resolution back to a high-resolution image (through an upsampling sampling process). In order to prevent U-Net from losing important information when downsampling, a shortcut connection is usually added between the downsampled ResNet (residual network) of the encoder and the upsampled ResNet of the decoder. Here, it refers to the residual connection skip-connect in the U-Net network.

[0178] Among them, a cross attention calculation module (cross attention) is added to U-Net to adjust the output of text concept embedding. The cross attention calculation module is added between the encoder and decoder ResNet blocks of U-Net. At the same time, in order to make the quality of the image generated by U-Net more exquisite, a self-attention calculation module (self attention) is added between the ResNet block and the cross attention calculation module. Therefore, in the network structure of U-Net, each U-Netblock (i.e., denoising block) is divided into three parts, namely ResNet (residual network), self-attention module (self-attention calculation module) and cross attention module (cross attention calculation module).

[0179] Among them, the "cross attention" in the stable diffusion model refers to the cross attention calculation mechanism in the model. This mechanism is usually used to process the association between multiple input sequences in order to capture important information interactions between different sequences in the model. In the stable diffusion model, cross attention can help the model better understand the association between different parts, so as to more accurately predict the changes and impacts in the diffusion process.

[0180] Among them, the "self attention" in the stable diffusion model refers to the self-attention calculation mechanism in the model. This mechanism is an important supplement to the U-Net block of stable diffusion. The input of its calculation mechanism is the backbone feature latent of the stable diffusion model. The role of self attention is to further integrate all semantic representations in the latent feature, and at the same time make the representation of the generated object clearer, thereby improving the U-Net network's ability to reduce latent noise.

[0181] Among them, from the previous introduction to the basic architecture of the backbone generation network, it can be seen that the cross-attention calculation module is originally in the basic stable diffusion model, and a large amount of data has been used in the pre-training stage of the stable diffusion model to train and optimize the performance of the cross-attention calculation module injected with this text concept. Therefore, in this application, its ability to train complete text concept embedding is directly used. The parallel cross-attention calculation in this application is divided into two steps, namely, automatically expanding the parallel cross-attention calculation module and parallel cross-attention post-calculation.

[0182] Among them, after receiving the text embedding (text embedding, in this embodiment, specifically refers to the object description features describing the q objects in the image, and the attribute description features of the attributes of the p objects), because the text embedding has been divided into the object description features of the q objects and the attribute description features of the attributes of the p objects when the previous text concept processing is performed in this application, therefore, when performing cross-attention calculation, the text-based image generation model will simultaneously copy the cross-attention calculation module according to the number of attribute description information of the input attributes, that is, if the input is the attribute description information of the attributes of p objects, after receiving the attribute description information of the attributes of these p objects, the text-based image generation model will automatically expand and copy p cross-attention calculation modules, plus the cross-attention calculation module for the object description features of the q objects, a total of (1+p) cross-attention calculation modules. It can be understood that in the process of transformation in this application, a mechanism for automatically expanding the cross-attention calculation module is added for each denoising processing block in the denoiser.

[0183] Among them, the calculation process of parallel cross-attention post-computation can be as follows Fig.17 As shown, its parallel structure is divided into (1+p) parts, which are the cross-attention calculation of the object description features of q objects and the cross-attention calculation of the attribute description features of the attributes of p objects. In the process of cross-attention calculation of the object description features of q objects, the attention weight matrix calculated based on the object description features of each object in the calculation process will be stored as a mask weight matrix. This is mainly for the fusion of the features obtained based on the attribute description features of the attributes of the objects. Finally, all the calculated latent features (including the first denoising feature and the second denoising feature corresponding to the attributes of the p objects) and the stored object mask weight matrix set (the attention weight matrix corresponding to the object description features of the q objects) are input into the fusion network.

[0184] Among them, the fusion network is mainly used to realize the fusion of object weighting and object attributes, so as to achieve the purpose of accurate and controllable generation. Taking the attention weight matrix as the mask, the calculation flow chart based on the mask fusion can be as follows Fig.18 As shown. Fig.18As shown, the example shows two objects (a short skirt and a short sleeve), and their matching attributes are yellow and black. Ideally, the yellow attribute is only present in the short sleeve area, and the black attribute is only present in the short skirt area. So first, when calculating, extract the attention weight matrix calculated by the short sleeve and short skirt objects with the latent (a part of the input data) during the cross-attention calculation. This attention weight matrix represents the probability that these objects are in the area of ​​the image when the image is finally generated. For example, Fig.18 The marked area in the middle of the attention weight matrix for the short-sleeved shirt is the area with high weight, which means that the short-sleeved shirt in the final generated image is likely to be generated within this range. Then, the latent feature (second denoising feature) calculated by cross-attention for the attribute description feature of the yellow attribute is resized to the same dimension as the attention weight matrix, and the attention weight matrix is ​​used to perform weighted filtering on the latent (i.e., the second denoising feature of the same dimension as the attention weight matrix) generated by the resized latent. In this way, only the content of the area with high weight for the short-sleeved shirt can be retained on the second denoising feature corresponding to the yellow attribute. The filtered latent (the third denoising feature corresponding to the yellow attribute) is merged with the resized latent (the third denoising feature corresponding to the black attribute) calculated by the same weighted filtering method. In this way, we can get a latent generated by combining the two attributes (the attribute denoising feature before resizing). By resizing the merged latent to the original dimension, we can get the resized latent (that is, the attribute denoising feature corresponding to the two attributes), and then add it to the latent generated by the object (that is, the first denoising feature). In this way, the semantics generated by the attribute description information can be accurately and controllably integrated into the regional content of the object bound to it, thereby completing accurate and controllable generation.

[0185] It should be noted that in the present application, the innovative mechanism involved in the present application is added to each U-Net block in the U-Net network. No additional training is required. The parallel cross-attention calculation module can directly reuse the original weights to automatically and accurately bind and generate objects with attributes. In the present application, in addition to using attribute description information to achieve precise and controllable generation, embedded enhancement data indicating the generated image can also be used to accurately and controllably generate the generated image. For example, the embedded enhancement data can specifically be an image (such as a yellow or black image) that adjusts the hue of the image, etc. After encoding the embedded enhancement data, the embedded enhancement features can be obtained, and then the embedded enhancement features, the object description features describing the q objects in the image, and the attribute description features of the attributes of the p objects are used as a guide to perform noise reduction processing to achieve precise and controllable generation.

[0186] In the scheme of this application, the feature "latent" is involved in many places. In the stable diffusion model, "latent" usually refers to latent variables, that is, variables that are not directly observed in the model, which is the most basic feature in the entire model processing flow. These variables are very important for explaining the observed data and the behavior of the model. In the stable diffusion model, latent variables may represent unobserved factors, such as individual characteristics, behaviors, styles, and compositions, which may affect the diffusion process.

[0187] In the scheme of the present application, a method for accurately generating objects with attributes in a text image based on the binding and fusion calculation of the attention weight matrix corresponding to the object description feature of the object is designed. The method is achieved by constructing a method to automatically expand the number of cross-attention calculation modules according to the number of input attribute description information, and then filtering the matching attribute description information generation results (i.e., the second denoising features corresponding to the attributes) according to the attention weight matrix generated by the object bound to the attribute description feature during the cross-attention calculation process of its own object description feature, and then fusing them into the object generation features of the model (i.e., the first denoising features), so as to achieve the purpose of accurately generating the attributes of the object.

[0188] Specifically, in this scheme, first, in the input processing process, the object description information of the object to be bound and the attribute description information of the object's attributes are matched and bound in a matching manner, and then the model will automatically expand the cross-attention calculation module according to the number of input attribute description information, and the number of expansions is equal to the total number of all input attribute description information. All attribute description information in the input image description information is extracted, and the attribute description information of each object is calculated separately for a cross-attention calculation module. The object description information of the remaining objects is input into the cross-attention calculation module of the trunk for calculation. In the process of cross-attention calculation of the object description feature of the object, it is necessary to record the calculated attention weight matrix, and then use the attention weight matrix as a mask to filter the features (i.e., the third noise reduction feature) calculated by the cross-attention calculation module of the attribute description information that matches the binding. This step is to enable the attribute description information to act only on the object that is bound and matched. The filtered features are fused with the object generation features, and the fused features are used for the next Wensheng map calculation.

[0189] This binding matching generation method can make the attribute description information accurately act on the objects it matches without interfering with the generation effects of other objects. At the same time, multiple attribute description information can be bound and matched for object generation at the same time, which greatly improves the precision and accuracy of the attribute description information generation for objects in the Wensheng graph, and can further improve the effect of Wensheng graph generating images.

[0190] In actual use, you only need to give an image description information of the image you want to generate. In the image description information, you need to specify which objects need to be generated, and you also need to map the attribute description information of the attributes of these objects. Then, this mapping relationship is input into the text-based image generation. Without any additional training, you can generate object images with accurate attributes.

[0191] The method of the present application is mainly to enable the generation of each object in the image to be more consistent with the input description image in the process of the text map, so that the generated image content is more accurate and rich. Therefore, the image generation method of the present application can be applied to various image generation landing directions. According to the input image description information, the object description information of the object is bound to the attribute description information of the attribute matched by the object, and then input into the model together, and the object is precisely controlled and generated, so that the attributes of the object can be accurately generated to the area of ​​the bound object, and in the final generated image, the semantics of each attribute description information will not interfere with each other, so as to achieve accurate control of the entire generated map. It can be understood that this provides a more efficient generation capability for various image generation landing applications, greatly improving the image generation effect.

[0192] The following is an example of the application scenario of the image generation method of the present application:

[0193] For example, the image generation method of the present application can be applied to various personalized AI photo generation scenarios. Users can input their own needs, such as the user wants to generate a yellow skirt and red short-sleeved photo image, and then use the image generation method of the present application to accurately generate the final personalized photo according to the photo requirements uploaded by the user, which can greatly enhance the user's sense of participation in AI photo generation and increase the gameplay of AI photo generation.

[0194] For another example, the image generation method of the present application can be applied to the generation of game original paintings, especially in the design of game character images, to improve the efficiency of designers and original painters. Designers can describe objects and their properties by setting, for example, golden armor and glowing staffs, and then accurately generate game character images of golden armor and glowing staffs based on the text-generated image model. The staffs will have their own colors and will not be disturbed by the gold color, so that the properties of the element objects can be controlled, greatly improving their creation efficiency.

[0195] In one embodiment, Fig.19 As shown, some examples are given of the generation results of the image generation method of the present application.

[0196] like Fig.19 As shown in the first picture, there are three objects, namely a girl, a short-sleeved shirt and a short skirt. The girl has no attributes, while the other two objects have their own matching attributes, namely yellow for the short-sleeved shirt and white for the short skirt. By using the image generation method of the present application, the attribute description information of the attribute can be accurately generated to the bound and matched object, and then a girl wearing a yellow short-sleeved shirt and a black short skirt can be generated, in which the semantic expressions of yellow and black will not interfere with each other and the bound objects will not be mixed. Fig.19 As shown, other examples of image generation are also given, namely "a girl with a purple skirt and green short-sleeved shirt, holding a red umbrella in her hand", "a girl in a yellow dress and a boy in a black suit", and "a game character wearing golden armor and riding a white warhorse".

[0197] The image generation method of the present application has the following advantages:

[0198] First of all, this application is to construct a method for accurately generating attributes on objects when objects in the text map have attributes. This method innovatively constructs a parallel cross-attention calculation mechanism for the object description features of the object and the attribute description features of the attribute. In the process of generating the finished text map, the objects and the attributes of the objects are first generated without interfering with each other, and then the results of the generated object attributes are used to modify the object generation process. In this way, the entire text-based image generation model can generate objects and their attributes independently without interfering with each other during the generation process and then merge them, so that the finally generated object can be more accurate in the expression of attributes.

[0199] Secondly, this application innovatively constructs a mechanism for generating object attributes and object binding attributes. During the use of this mechanism, it is only necessary to bind the attributes of the objects to be generated when inputting text concepts, and input them together into the text-based image generation model. During calculation, multiple cross-attention calculation modules will be automatically connected in parallel according to the number of attribute description information of the matching attributes. This adaptive mechanism of expanding parallel cross-attention calculation according to the number of input attribute description information can enable the entire text-based image generation model to simultaneously perform enhanced fusion of matching attribute description information on multiple generated objects during the image generation process, so that the final effect of multiple objects can be generated more accurately.

[0200] Secondly, in this application, an innovative method is constructed to bind and fuse attribute description features based on the attention weight matrix corresponding to the object description feature of the object. This method uses the attention weight matrix calculated by the object description feature of the current object as a mask for its attribute description feature during the parallel cross-attention calculation, and then uses the mask to filter the cross-attention results of the attribute description feature calculation, so that the effect of the attribute description information of the attribute of the object can be limited to the area of ​​the object. This mechanism enables the attribute to accurately perform conceptual attribute generation binding on the matching object in the process of generating the entire image, without affecting the generation of other objects. And because this method uses the attention weight matrix corresponding to the object description feature of the object as a mask, the cross-attention calculation method of the attribute description feature of the weighted filtering attribute will also prevent mutual interference between different attribute description features, which will further improve the accuracy of the model for attribute-assisted object generation.

[0201] Finally, the present application is a method that does not require any additional training at all. It is based on the original diffusion model text map, and adds an automated and expanded cross-attention calculation module mechanism, which can bind the attributes of the object collocation to the object in real time, and then generate it in real time. Therefore, the present application can greatly reduce the cost of transforming the text-based image generation model while improving the generation accuracy of the text-based image generation model, thus achieving the purpose of plug-and-play. And this method that does not require additional training can be transplanted to different fine-tuning models, which greatly enriches the gameplay of the model in the open source community and improves the efficiency of generating image creation.

[0202] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0203] Based on the same inventive concept, the embodiment of the present application also provides an image generation device for implementing the above-mentioned image generation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image generation device embodiments provided below can refer to the limitations on the image generation method above, and will not be repeated here.

[0204] In one embodiment, Fig. 20 As shown, an image generation device is provided, comprising: a description feature acquisition module 2002, a noise feature acquisition module 2004, a noise reduction processing module 2006 and a decoding module 2008, wherein:

[0205] A description feature acquisition module 2002 is used to acquire image description information indicating the generation of an image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q.

[0206] The noise feature acquisition module 2004 is used to acquire random noise features, and use the random noise features as multi-step noise features obtained by performing N iterative noise adding steps on the image features of the image, where N is a positive integer greater than 1;

[0207] A denoising processing module 2006 is used to perform denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step;

[0208] The denoising processing module 2006 is further used to perform iterative processing of M denoising processing layers in the nth denoising step, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer, and when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step;

[0209] The denoising processing module 2006 is further configured to perform denoising processing at the mth denoising processing layer based on a portion of the input data and the object description features of the q objects to obtain a first denoising feature, perform denoising processing based on a portion of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer;

[0210] The decoding module 2008 is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

[0211] The above-mentioned image generating device, by acquiring image description information indicating the generation of an image, can use the image description information to realize the determination of the object description information of q objects in the image and the attribute description features of the attributes of p objects, and then can obtain random noise features, use the random noise features as multi-step noisy noise features obtained by subjecting the image features of the image to N iterative noisy steps, perform denoising processing of N iterative denoising steps on the multi-step noisy noise features, perform iterative processing of M denoising processing layers in the nth denoising step, perform denoising processing in the mth denoising processing layer using a portion of input data and the object description features of q objects to obtain a first denoising feature, and perform denoising processing using a portion of input data and the attribute description features of the attributes of p objects to obtain The second denoising features corresponding to the attributes of the p objects are respectively fused with the first denoising features and the second denoising features corresponding to the attributes of the p objects, to obtain the output data of the mth denoising processing layer. In the entire denoising processing process, a parallel denoising mechanism of object description features and attribute description characteristics is constructed, so that the object description features and the attribute description features are first subjected to denoising processing without interfering with each other, and then the first denoising features are modified by using the second denoising features corresponding to the attributes of the p objects in a fusion manner, so that the output data of the Nth denoising step that makes the object's attributes more accurately expressed can be obtained, so that on the basis of using the output data of the Nth denoising step as the image features after denoising, the image features after denoising are decoded to generate an image that makes the object's attributes more accurately expressed, thereby improving the accuracy of image generation.

[0212] In one embodiment, the denoising processing module is also used to obtain the attention weight matrix corresponding to each of the object description features of q objects, determine the object to which the attribute of the targeted object is bound for each of the attributes of the p objects, perform weighted filtering on the second denoising feature corresponding to the attribute of the targeted object through the attention weight matrix corresponding to the object description feature of the bound object, obtain the third denoising feature corresponding to the attribute of the targeted object, fuse the first denoising feature and the third denoising feature corresponding to the attributes of the p objects, and obtain the output data of the mth denoising processing layer.

[0213] In one embodiment, the denoising processing module is also used to fuse the third denoising features corresponding to the attributes of p objects to obtain the attribute denoising features corresponding to the attributes of the p objects, and to fuse the first denoising features and the attribute denoising features corresponding to the attributes of the p objects to obtain the output data of the mth denoising processing layer.

[0214] In one embodiment, the noise reduction processing module is also used to perform sampling processing on a portion of the input data at the mth noise reduction processing layer according to the sampling method indicated by the mth noise reduction processing layer to obtain a sampled noise feature, perform feature interaction processing on the features in the sampled noise feature to obtain an interacted noise feature, perform noise reduction processing based on the interacted noise feature and the object description features of q objects to obtain a first noise reduction feature, and perform noise reduction processing based on the interacted noise feature and the attribute description features of the attributes of p objects to obtain second noise reduction features corresponding to the attributes of the p objects.

[0215] In one embodiment, the noise reduction processing module is further used to perform feature interaction processing on the interacted noise features and the object description features of q objects based on a cross-attention mechanism to obtain a first noise reduction feature.

[0216] In one embodiment, the denoising processing module is also used to perform feature interaction processing on the interacted noise features and the object description features of each object in the object description features of q objects based on a cross-attention mechanism, to obtain the denoising features corresponding to the object description features of each object, and to fuse the denoising features corresponding to the object description features of the q objects to obtain the first denoising feature.

[0217] In one embodiment, the denoising processing module is also used to generate a matrix based on a pre-trained first vector of the cross-attention mechanism for the object description features of each object in the object description features of q objects, convolve the object description features of the targeted object to obtain a first key vector and a first value vector, and convolve the noise features after interaction to obtain a first query vector, perform attention weight calculation based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description features of the targeted object, and perform weighted processing on the first value vector according to the attention weight matrix corresponding to the object description features of the targeted object to obtain a denoising feature corresponding to the object description features of the targeted object.

[0218] In one embodiment, the noise reduction processing module is also used to perform feature interaction processing on the interacted noise features and the attribute description features of the attributes of the p objects based on the cross-attention mechanism, so as to obtain second noise reduction features corresponding to the attributes of the p objects.

[0219] In one embodiment, the denoising processing module is also used to generate a matrix based on a pre-trained second vector based on a cross-attention mechanism for the attribute description features of each object's attributes in the attribute description features of the attributes of p objects, convolve the attribute description features of the attributes of the targeted objects to obtain a second key vector and a second value vector, and convolve the noise features after interaction to obtain a second query vector, perform attention weight calculation based on the second key vector and the second query vector to obtain an attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects, and perform weighted processing on the second value vector according to the attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects to obtain a second denoising feature corresponding to the attributes of the targeted objects.

[0220] In one embodiment, the description feature acquisition module is also used to encode the object description information describing the q objects in the image in the image description information respectively to obtain the object description features describing the q objects in the image, and to encode the attribute description information describing the attributes of p objects in the image in the image description information respectively to obtain the attribute description features of the attributes of the p objects.

[0221] In one embodiment, an image is determined by a pre-trained text-based image generation model, the text-based image generation model includes a text encoder, a denoising network and a decoder, the denoising network includes N denoisers, each denoiser includes M denoising processing blocks, the description feature acquisition module is also used to obtain image description information indicating the generated image through the text encoder, determine object description features describing q objects in the image based on the image description information, and attribute description features of the attributes of p objects respectively, the denoising processing module is also used to obtain random noise features through the denoising network, use the random noise features as multi-step denoising noise features obtained by subjecting the image features of the image to N iterative denoising steps, perform denoising processing on the multi-step denoising noise features in N iterative denoising steps, and obtain output data of the Nth denoising step; wherein ... to N iterative denoising steps, and perform denoising processing on the multi-step denoising noise features in N iterative denoising steps, and obtain output data of the Nth denoising step. The method comprises the steps of: performing iterative processing of M denoising layers through the nth denoising device and the M denoising processing blocks in the nth denoising step; the denoising processing module is further used to perform processing of the mth denoising processing layer through the mth denoising processing block; the mth denoising processing block is used to perform denoising based on a portion of the input data and the object description features of the q objects to obtain a first denoising feature; performing denoising based on a portion of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects; fusing the first denoising features and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer; and the decoding module is further used to use the output data of the Nth denoising step as the denoised image features through the decoder, decode the denoised image features, and generate an image.

[0222] Each module in the above-mentioned image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0223] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. Taking the computer device as a server as an example, its internal structure diagram may be as follows: Fig.21 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as an attention weight matrix. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method is implemented.

[0224] Those skilled in the art will understand that Fig.21 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0225] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0226] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0227] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0228] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0229] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0230] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0231] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire image description information indicating generation of an image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q; Obtaining a random noise feature, and using the random noise feature as a multi-step noise feature obtained by performing N iterative noise addition steps on the image feature of the image, where N is a positive integer greater than 1; Performing denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step; In the nth denoising step, iterative processing of M denoising processing layers is performed, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer, and when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step; At the mth denoising processing layer, denoising is performed based on the portion of the input data and the object description features of the q objects to obtain a first denoising feature, denoising is performed based on the portion of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and the first denoising features and the second denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer; The output data of the Nth denoising step is used as the denoised image feature, and the denoised image feature is decoded to generate an image.

2. The method according to claim 1, characterized in that The fusing of the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer includes: Obtaining the attention weight matrices corresponding to the object description features of the q objects; For each of the attributes of the p objects, determine the object to which the attribute of the targeted object is bound, and perform weighted filtering on the second denoising feature corresponding to the attribute of the targeted object through the attention weight matrix corresponding to the object description feature of the bound object to obtain a third denoising feature corresponding to the attribute of the targeted object; The first denoising feature and the third denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer.

3. The method according to claim 2, characterized in that The fusing of the first denoising feature and the third denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer includes: fusing the third denoising features corresponding to the attributes of the p objects respectively to obtain attribute denoising features corresponding to the attributes of the p objects; The first denoising feature and the attribute denoising features corresponding to the attributes of the p objects are fused to obtain output data of the mth denoising processing layer.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing noise reduction processing on the mth noise reduction processing layer based on the portion of input data and the object description features of the q objects to obtain a first noise reduction feature, and performing noise reduction processing on the portion of input data and the attribute description features of the attributes of the p objects to obtain second noise reduction features corresponding to the attributes of the p objects respectively includes: At the mth noise reduction processing layer, sampling is performed on the part of the input data according to the sampling mode indicated by the mth noise reduction processing layer to obtain a sampling noise feature; Performing feature interaction processing on the features in the sampled noise features to obtain interactive noise features; Based on the post-interaction noise feature and the object description features of the q objects, a noise reduction process is performed to obtain a first noise reduction feature; Based on the post-interaction noise features and the attribute description features of the attributes of the p objects, noise reduction processing is performed respectively to obtain second noise reduction features corresponding to the attributes of the p objects.

5. The method according to claim 4, characterized in that The noise reduction process is performed based on the post-interaction noise feature and the object description features of the q objects to obtain the first noise reduction feature, which includes: Based on the cross attention mechanism, feature interaction processing is performed on the interacted noise feature and the object description features of the q objects to obtain a first denoising feature.

6. The method according to claim 5, characterized in that Based on the cross attention mechanism, the post-interaction noise feature and the object description features of the q objects are subjected to feature interaction processing to obtain the first noise reduction feature, which includes: Based on the cross attention mechanism, feature interaction processing is performed on the noise feature after the interaction and the object description feature of each object in the object description features of the q objects, so as to obtain a noise reduction feature corresponding to the object description feature of each object; The denoising features corresponding to the object description features of the q objects are fused to obtain a first denoising feature.

7. The method according to claim 6, characterized in that Based on the cross attention mechanism, the interactive noise features and the object description features of each object in the object description features of the q objects are subjected to feature interaction processing respectively, and the noise reduction features corresponding to the object description features of each object are obtained, including: For the object description features of each object in the object description features of the q objects, convolve the object description features of the object by generating a matrix based on a pre-trained first vector of a cross attention mechanism to obtain a first key vector and a first value vector, and convolve the post-interaction noise feature to obtain a first query vector; Performing attention weight calculation based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description feature of the targeted object; The first value vector is weighted according to the attention weight matrix corresponding to the object description feature of the targeted object to obtain the noise reduction feature corresponding to the object description feature of the targeted object.

8. The method according to claim 4, characterized in that The noise reduction process is performed based on the noise feature after the interaction and the attribute description features of the attributes of the p objects to obtain the second noise reduction features corresponding to the attributes of the p objects respectively, including: Based on the cross attention mechanism, feature interaction processing is performed on the interacted noise features and the attribute description features of the attributes of the p objects respectively to obtain second noise reduction features corresponding to the attributes of the p objects respectively.

9. The method according to claim 8, characterized in that Based on the cross attention mechanism, the interactive noise features and the attribute description features of the attributes of the p objects are respectively subjected to feature interaction processing to obtain the second noise reduction features corresponding to the attributes of the p objects, which include: For the attribute description features of the attributes of each object in the attribute description features of the attributes of the p objects, convolve the attribute description features of the attributes of the objects by generating a matrix of pre-trained second vectors based on a cross attention mechanism to obtain a second key vector and a second value vector, and convolve the post-interaction noise features to obtain a second query vector; Performing attention weight calculation based on the second key vector and the second query vector to obtain an attention weight matrix corresponding to the attribute description feature of the attribute of the targeted object; The second value vector is weighted according to the attention weight matrix corresponding to the attribute description feature of the attribute of the targeted object to obtain a second noise reduction feature corresponding to the attribute of the targeted object.

10. The method according to any one of claims 1 to 3, characterized in that: The determining, based on the image description information, object description features describing q objects in the image and attribute description features of the attributes of p objects respectively comprises: The object description information describing the q objects in the image in the image description information is encoded respectively to obtain the object description features describing the q objects in the image, and the attribute description information describing the attributes of the p objects in the image in the image description information is encoded respectively to obtain the attribute description features of the attributes of the p objects.

11. The method according to any one of claims 1 to 3, characterized in that: The image is determined by a pre-trained text-based image generation model; the text-based image generation model includes a text encoder, a denoising network and a decoder; the denoising network includes N denoisers; each of the denoisers includes M denoising processing blocks; The text encoder is used to obtain image description information indicating the generated image, and determine object description features describing q objects in the image and attribute description features of the attributes of p objects based on the image description information; The denoising network is used to obtain random noise features, use the random noise features as multi-step denoising noise features obtained by subjecting the image features of the image to N iterative denoising steps, perform denoising processing on the multi-step denoising noise features in N iterative denoising steps, and obtain output data of the Nth denoising step; wherein the nth denoiser is used to perform iterative processing of M denoising processing layers through the M denoising processing blocks in the nth denoising step; the processing of the mth denoising processing layer is performed by the mth denoising processing block, the mth denoising processing block is used to perform denoising processing based on the part of the input data and the object description features of the q objects to obtain a first denoising feature, perform denoising processing based on the part of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer; The decoder is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

12. An image generating device, characterized in that: The device comprises: a description feature acquisition module, used to acquire image description information indicating the generation of an image, and determine object description features describing q objects in the image and attribute description features of attributes of p objects based on the image description information, where q is a positive integer and p is a positive integer not greater than q; A noise feature acquisition module, used to acquire a random noise feature, and use the random noise feature as a multi-step noise feature obtained by performing N iterative noise addition steps on the image feature of the image, where N is a positive integer greater than 1; a denoising processing module, configured to perform denoising processing of N iterative denoising steps on the multi-step denoising noise feature to obtain output data of the Nth denoising step; wherein, for the nth denoising step in the N iterative denoising steps, when n is equal to 1, the multi-step denoising noise feature is used as input data of the nth denoising step, when n is greater than 1 and less than or equal to N, the output data of the n-1th denoising step is used as input data of the nth denoising step, and when n is equal to N, the output data of the nth denoising step is the output data of the Nth denoising step; The denoising processing module is further used to perform iterative processing of M denoising processing layers in the nth denoising step, wherein, for the mth denoising processing layer among the M denoising processing layers, when m is equal to 1, the input data of the nth denoising step is used as the input data of the mth denoising processing layer, when m is greater than 1 and less than or equal to M, the output data of the m-1th denoising processing layer is used as part of the input data of the mth denoising processing layer, and when m is equal to M, the output data of the mth denoising processing layer is the output data of the nth denoising step; The denoising processing module is further configured to perform denoising processing at the mth denoising processing layer based on the portion of input data and the object description features of the q objects to obtain a first denoising feature, perform denoising processing based on the portion of input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer; The decoding module is used to use the output data of the Nth denoising step as the denoised image features, decode the denoised image features, and generate an image.

13. The device according to claim 12, characterized in that The denoising processing module is also used to obtain the attention weight matrix corresponding to each of the object description features of the q objects, determine the object to which the attribute of the targeted object is bound for each of the attributes of the p objects, perform weighted filtering on the second denoising feature corresponding to the attribute of the targeted object through the attention weight matrix corresponding to the object description feature of the bound object, obtain the third denoising feature corresponding to the attribute of the targeted object, fuse the first denoising feature and the third denoising feature corresponding to the attributes of the p objects, and obtain the output data of the mth denoising processing layer.

14. The device according to claim 13, characterized in that The denoising processing module is also used to fuse the third denoising features corresponding to the attributes of the p objects to obtain the attribute denoising features corresponding to the attributes of the p objects, and fuse the first denoising features and the attribute denoising features corresponding to the attributes of the p objects to obtain the output data of the mth denoising processing layer.

15. The device according to any one of claims 12 to 14, characterized in that The noise reduction processing module is also used to perform sampling processing on the part of the input data at the mth noise reduction processing layer according to the sampling method indicated by the mth noise reduction processing layer to obtain a sampled noise feature, perform feature interaction processing on the features in the sampled noise feature to obtain an interacted noise feature, perform noise reduction processing based on the interacted noise feature and the object description features of the q objects to obtain a first noise reduction feature, and perform noise reduction processing based on the interacted noise feature and the attribute description features of the attributes of the p objects to obtain second noise reduction features corresponding to the attributes of the p objects.

16. The device according to claim 15, characterized in that The denoising processing module is also used to perform feature interaction processing on the post-interaction noise feature and the object description features of the q objects based on a cross-attention mechanism to obtain a first denoising feature.

17. The device according to claim 16, characterized in that The denoising processing module is also used to perform feature interaction processing on the interacted noise features and the object description features of each object in the object description features of the q objects based on a cross-attention mechanism, to obtain denoising features corresponding to the object description features of each of the objects, and to fuse the denoising features corresponding to the object description features of the q objects to obtain a first denoising feature.

18. The device according to claim 17, characterized in that The denoising processing module is also used to generate a matrix based on a pre-trained first vector of the cross-attention mechanism for the object description features of each object in the object description features of the q objects, convolve the object description features of the targeted object to obtain a first key vector and a first value vector, and convolve the interacted noise features to obtain a first query vector, perform attention weight calculation based on the first key vector and the first query vector to obtain an attention weight matrix corresponding to the object description features of the targeted object, and perform weighted processing on the first value vector according to the attention weight matrix corresponding to the object description features of the targeted object to obtain a denoising feature corresponding to the object description features of the targeted object.

19. The device according to claim 15, characterized in that The denoising module is also used to perform feature interaction processing on the interacted noise features and the attribute description features of the attributes of the p objects based on a cross-attention mechanism to obtain second denoising features corresponding to the attributes of the p objects.

20. The device according to claim 19, characterized in that The denoising processing module is also used to generate a matrix based on a pre-trained second vector based on a cross-attention mechanism for the attribute description features of each object's attributes in the attribute description features of the attributes of the p objects, convolve the attribute description features of the attributes of the targeted objects to obtain a second key vector and a second value vector, and convolve the interacted noise features to obtain a second query vector, perform attention weight calculation based on the second key vector and the second query vector to obtain an attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects, and perform weighted processing on the second value vector according to the attention weight matrix corresponding to the attribute description features of the attributes of the targeted objects to obtain a second denoising feature corresponding to the attributes of the targeted objects.

21. The device according to any one of claims 12 to 14, characterized in that The description feature acquisition module is also used to encode the object description information describing the q objects in the image in the image description information respectively to obtain the object description features describing the q objects in the image, and to encode the attribute description information describing the attributes of the p objects in the image in the image description information respectively to obtain the attribute description features of the attributes of the p objects.

22. The device according to any one of claims 12 to 14, characterized in that The image is determined by a pre-trained text-based image generation model; the text-based image generation model includes a text encoder, a denoising network and a decoder; the denoising network includes N denoisers; each of the denoisers includes M denoising processing blocks; The description feature acquisition module is further used to acquire, through the text encoder, image description information indicating the generated image, and determine object description features describing q objects in the image and attribute description features of the attributes of p objects based on the image description information; The denoising processing module is also used to obtain random noise features through the denoising network, use the random noise features as multi-step denoising noise features obtained by subjecting the image features of the image to N iterative denoising steps, perform denoising processing of N iterative denoising steps on the multi-step denoising noise features, and obtain output data of the Nth denoising step; wherein the nth denoiser is used to perform iterative processing of M denoising processing layers through the M denoising processing blocks in the nth denoising step; the processing of the mth denoising processing layer is performed by the mth denoising processing block, and the mth denoising processing block is used to perform denoising processing based on the part of the input data and the object description features of the q objects to obtain a first denoising feature, perform denoising processing based on the part of the input data and the attribute description features of the attributes of the p objects to obtain second denoising features corresponding to the attributes of the p objects, and fuse the first denoising feature and the second denoising features corresponding to the attributes of the p objects to obtain output data of the mth denoising processing layer; The decoding module is also used to use the output data of the Nth denoising step as the denoised image features through the decoder, decode the denoised image features, and generate an image.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images