Multi-modal guided progressive image generation method

Abstract

The invention relates to the field of image processing, in particular to a multi-modal guided progressive image generation method, which comprises the following steps of: generating multi-modal features for received data; generating a final layer layout for each main object by using a layout denoising network through a noise sampling and multi-modal feature fusion method; extracting potential codes of a reference image through an inversion algorithm, combining layer text codes, performing iteration denoising through a stable diffusion model, and extracting a reference key value matrix; the initial noise, the final layer layout, the multi-modal features and the reference key value matrix are input into a layered diffusion model, and after multi-step denoising, a main body transparent image layer is generated through decoding; and after all the main body transparent layers are superposed to form a mixed image, driving the image recovery network to generate a target image based on global text coding. According to the method, the defects of an existing method in processing a complex scene are overcome; and the generated image is ensured to be consistent with the text description globally and locally.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-modality guided progressive image generation method. Background Art

[0002] Generating images based on text is an important research direction in the intersection of computer vision and natural language processing. Its goal is to generate corresponding images based on the text description provided by the user. This technology has many practical applications, such as automated design, virtual reality, game design, and assisted teaching.

[0003] Existing methods for generating images based on text usually include two parts: encoding and decoding. In the encoding step, the text information is converted into a text encoding (such as a vector) that can be understood by the computer. In the decoding step, the text encoding is used as guidance information and input into the image generation model to obtain the image. These methods are often unable to handle the generation of images containing multiple subject objects, especially when the attributes and relationships of these subject objects are complex. Moreover, when generating images, existing methods often cannot effectively ensure that the generated images are consistent with the input text description both globally and locally, which results in the lack of consistency and coherence of the generated images. Summary of the invention

[0004] To solve the above problems, the present invention provides a multi-modality guided progressive image generation method, which is used to generate a target image including multiple subject objects, and the method comprises: Obtaining, by encoding, a reference image visual embedding corresponding to each reference image in the input data, a global text encoding corresponding to a global text prompt, and a layer text encoding corresponding to each layer text prompt; wherein the number of reference images in the input data is arbitrary, each layer text prompt and each reference image correspond to a subject object, and each subject object corresponds to a layer text prompt and an arbitrary number of reference images; The final layout of layers is generated for each subject object through noise sampling and multimodal feature fusion methods; Use the inversion algorithm to generate latent codes for all reference images; input the latent codes and layer text codes into the stable diffusion model for iterative denoising, and extract the reference key matrix and reference value matrix; The sampling noise is used as the initial layer noise image corresponding to each subject object. Based on the reference image visual embedding, the final layer layout, the reference key matrix and the reference value matrix, the hierarchical diffusion model is used to perform multiple time step iterations to denoise the initial layer noise image, obtain the layer features of each time step, and decode the layer features of the last time step to obtain the subject transparent layer image corresponding to each subject object. Overlay all subject transparent layer images to generate a mixed image; Based on the global text encoding, the mixed image is repaired using an image restoration network to obtain the target image.

[0005] Furthermore, the steps of generating the final layout of the layer for each subject object through noise sampling and multimodal feature fusion method include: Based on the standard normal distribution, sampling noise for each subject object generates the layer initial layout corresponding to the subject object, and generates layer layout embedding for the layer initial layout through Fourier encoder; For each subject object, a multimodal embedding is generated based on its corresponding reference image visual embedding, layer text encoding, and layer layout embedding; The multimodal embedding is input into the pre-trained layout denoising network for denoising to generate the final layout of the layer corresponding to the main object.

[0006] Further, the reference image includes a subject reference image, and the subject reference image describes its corresponding subject object; And, for each subject object, the steps of generating a multimodal embedding according to its corresponding reference image visual embedding, layer text encoding, and layer layout embedding include: For Subject objects, generate their corresponding multimodal embeddings according to the following formula : ; in, represents the multi-layer perceptron layer, Represents a splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The subject reference image corresponding to the subject object corresponds to the reference image visual embedding, Represents a vector filling operation. When a subject object corresponds to a subject reference image, represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

[0007] Furthermore, the input data also includes a reference image mask, wherein the reference image corresponds to the reference image mask one by one; And, the steps of performing multiple time step iterations to denoise the initial layer noise image and obtaining the layer features of each time step include: Iterate multiple time steps to denoise the initial layer noise image, and the steps in each time step include: In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image in this time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of this time step; In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation to generate the spatial constraint features of this time step; The spatial constraint features are input into the subsequent network layers of the hierarchical diffusion model to generate the layer features of this time step.

[0008] Furthermore, in the self-attention layer of the hierarchical diffusion model, the denoising result of the noise image of the initial layer in the time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of the time step, including the following steps: For each subject object, calculate the similarity weight between its corresponding layer text encoding and each reference image visual embedding corresponding to the subject object; Construct a mask matrix for each subject object based on the similarity weight and the reference image mask; For the denoising result of the initial layer noise image in this time step, the self-attention matrix of the self-attention layer is dynamically corrected based on the mask matrix, and the visual constraint features are generated according to the corrected self-attention matrix.

[0009] Further, for each subject object, the step of calculating the similarity weight between the layer text encoding corresponding to the subject object and each reference image visual embedding corresponding to the subject object includes: Calculate the Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity : ; in, represents the vector dot product operation, represents the Euclidean norm; Calculate the layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight : ; in, is a predefined scaling factor, is a predefined offset factor, The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object is obtained. The minimum value of the similarity between the layer text encoding corresponding to the main object and the visual embeddings of all reference images corresponding to the main object.

[0010] Furthermore, the modified self-attention matrix is: ; in, is the modified self-attention matrix, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix The dimension of Represents a mask matrix.

[0011] Furthermore, in the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation, and the steps of generating the spatial constraint features of the time step include: For each subject object, convert the corresponding layer final layout into a layout mask ; The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. ; Based on layout mask , for the original cross attention matrix Make corrections to get the corrected cross attention matrix : ; in, is the background suppression factor, Represents a pixel-by-pixel multiplication operation; Based on the modified cross attention matrix , generating the spatial constraint features for this time step.

[0012] Furthermore, the steps of superimposing all the subject transparent layer images to generate a mixed image include: Arrange the subject transparent layer images corresponding to all subject objects into multiple layers from the bottom layer to the top layer in any order; Use the bottommost transparent layer image as the overlay image; Repeat the process of superimposing the superimposed image with the previous main transparent layer image to obtain an intermediate mixed image, and use the intermediate mixed image as a new superimposed image; until all the main transparent layer images are superimposed to obtain a mixed image.

[0013] Furthermore, the step of superimposing the superimposed image with the upper main transparent layer image includes: The overlay image is overlaid with the main transparent layer image of the previous layer according to the following formula: ; ; ; ; in, , , , are the values ​​of the red, green, blue, and transparent channels of the superimposed image, , , , The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. , , , It is the value of the red, green, blue and transparent channels of the intermediate mixed image obtained by superposition.

[0014] The present invention introduces layer text prompts and reference images to independently process each main object. Even when faced with complex scenes containing multiple main objects, it can effectively generate images that meet the given description, thereby solving the defects of existing methods in processing complex scenes; by adopting pre-trained layout denoising network, hierarchical diffusion model, and image restoration network, it is ensured that the generated image is consistent with the text description both globally and locally, which not only improves the quality of the generated image, but also improves the global and local consistency between the generated image and the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a flowchart of a multi-modality guided progressive image generation method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, or product end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0019] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] The present invention proposes a multimodal guided progressive image generation method, which constructs an image generation model to generate a target image. The method consists of three steps: multimodal guided layer layout generation, zero-sample multimodal guided layer generation, and multi-layer synthesis. The specific steps are as follows: Figure 1 shown.

[0021] In the multimodal guided layer layout generation step, first receive global text prompts, layer text prompts, reference images and reference image masks as input data, use the reference image mask to perform cropping and pasting preprocessing to obtain the preprocessed reference image, use the visual encoder to extract the reference image visual embedding, use the text encoder to extract the layer text encoding for the layer text prompts, generate the layer initial layout, and obtain the layer layout embedding through the Fourier encoder. The features of the above three modalities are fused and input into the layout denoising network to obtain the final layer layout.

[0022] In the zero-shot multimodal guided layer generation step, the inverted reference image is input into the stable diffusion model, and the key value matrix of each self-attention layer is extracted during the denoising process to obtain the reference key matrix and reference value matrix. Then, the noise is used as the initial layer noise image, and the initial layer noise image, layer final layout, reference image visual embedding, reference image mask, layer text encoding, reference key matrix and reference value matrix corresponding to each subject object are input into the hierarchical diffusion model. In the multi-time step denoising process, the reference key matrix, reference value matrix and reference image mask are used to correct the self-attention layer calculation, and the layer final layout is used to correct the cross-attention layer calculation. After denoising, the layer features of the last time step are obtained and decoded by the decoder into the subject transparent layer image corresponding to each subject object.

[0023] The multi-layer synthesis step is to superimpose the subject transparent layer images generated by each subject object layer by layer to generate a mixed image, and then input it into the image restoration network based on the diffusion model with the global text encoding to retain the subject object and complete the background area, and finally generate a high-quality and visually coherent target image.

[0024] 1. Multimodality-guided Layer Layout Generation 1.1 Data Preprocessing Receive input data for generating a target image, the input data includes a global text prompt, at least one layer text prompt, any number of reference images and a reference image mask corresponding to each reference image. The global text prompt describes the content of the target image, each target image includes at least one subject object, the subject object corresponds to the layer text prompt one-to-one, each layer text prompt describes its corresponding subject object, one subject object corresponds to any number of reference images, and each reference image corresponds to one subject object. There are two types of reference images, subject reference images and attribute reference images. The subject reference image describes its corresponding subject object, and the attribute reference image describes the attribute of its corresponding subject object. The reference image mask represents the area in its corresponding reference image that is related to the corresponding subject object or the attribute of the subject object.

[0025] For example: In the input data, the global text prompt is "a girl wearing a hat holding an orange cat", and the two layer text prompts are "a short-haired girl wearing a hat and carrying a backpack" and "an orange cat". There are two reference images in the input data. The first reference image is a girl sitting on a sofa. Its type is a subject reference image, corresponding to the subject object "girl"; the second reference image is a boy carrying a backpack. Its type is an attribute reference image, corresponding to the attribute "backpack" of the subject object "girl". The reference image mask is a binary image. The area with a pixel value of 1 in the reference image mask corresponding to the first reference image represents the area where the girl is located in the first reference image, and the area with a pixel value of 0 represents other areas. The area with a pixel value of 1 in the reference image mask corresponding to the second reference image represents the area where the backpack is located in the second reference image, and the area with a pixel value of 0 represents other areas.

[0026] The reference image corresponding to the reference image mask is cropped, and the cropped image is pasted into a blank image of a predetermined size to obtain a preprocessed reference image. The blank image in this embodiment refers to an image in which all pixels are white. By cropping and removing irrelevant background in the reference image, the image generation model can focus more on the main object and its attributes.

[0027] 1.2 Feature Extraction In order to guide the image generation model to focus on the semantic information and aspect ratio information in the preprocessed reference images, a visual encoder is used to extract the visual features of each preprocessed reference image to obtain the corresponding reference image visual embedding. A text encoder is used to encode the global text prompt to obtain the global text encoding, and a text encoder is used to encode each layer text prompt to obtain its corresponding layer text encoding.

[0028] The visual encoder in the present embodiment is a contrastive language-image pre-training (CLIP) visual encoder. CLIP is a powerful multimodal learning model that can learn a unified visual-language representation space by contrasting a large number of image-text pairs. In this space, semantically similar images and texts will have similar vector representations. Therefore, the features extracted using the CLIP visual encoder not only include the visual information of the reference image, but also contain rich semantic information. The text encoder in the present embodiment is a contrastive language-image pre-training text encoder.

[0029] For each subject object, noise is sampled from a standard normal distribution to generate the initial layout of the layer corresponding to the subject object. Each initial layout of the layer represents the position information of the bounding box of the corresponding subject object in the target image. In this embodiment, each initial layout of the layer is represented as the coordinates of the upper left corner and the lower right corner of the bounding box. For each initial layout of the layer, a Fourier encoder is used to map it to a higher dimension, strengthen the representation ability of the feature vector for high-frequency information, and obtain the layer layout embedding.

[0030] For each subject object, the visual embedding is adapted according to whether it corresponds to a subject reference image: if there is a corresponding subject reference image, the reference image visual embedding corresponding to the subject reference image is retained; otherwise, a learnable empty vector is used as a placeholder; then the layer text encoding representing semantic information, the layer layout embedding representing layout information, and the features after the visual embedding adaptation are spliced ​​in three dimensions, and cross-modal feature fusion is achieved through a multi-layer perceptron, and finally a unified multimodal embedding is generated, where the first Multimodal embeddings corresponding to the subject objects for: ; in, represents the multi-layer perceptron layer, Represents a splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The subject reference image corresponding to the subject object corresponds to the reference image visual embedding, Represents a vector filling operation. When a subject object corresponds to a subject reference image, represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

[0031] 1.3 Layer Layout Generation The multimodal embedding corresponding to each subject object is input into the layout denoising network for denoising, and the final layer layout corresponding to the subject object is generated. The layout denoising network is a layout generation model based on a diffusion model. By gradually removing noise and refining feature representation, it can generate a high-quality final layer layout that matches the conditional constraints from the noisy initial layer layout.

[0032] Diffusion model is a generative model whose core idea is to destroy the data structure by gradually adding noise, and then learn how to reverse this process to generate samples similar to the original data distribution. The diffusion model models the data distribution through a series of Markov chain transition steps, and each step learns to remove part of the noise, and finally obtains a clean generated sample. This step-by-step denoising process enables the diffusion model to generate high-quality and diverse samples, and has strong interpretability and controllability. In the embodiment of the present invention, the layout denoising network can be a denoising diffusion probabilistic model (DDPM, Denoising Diffusion Probabilistic Models), a denoising diffusion implicit model (DDIM, Denoising Diffusion Implicit Models), etc., which is not limited by the embodiment of the present invention.

[0033] 2. Zero-shot multimodal guided layer generation 2.1 Reference Image Key-Value Pair Extraction Through the inversion and denoising process, the key key-value pairs of the reference image in the self-attention layer of the Stable Diffusion Model are extracted to provide guiding signals for subsequent visual constraints.

[0034] An inversion algorithm is used to generate latent codes for all reference images. The latent codes are constructed through an independent noise sampling strategy to ensure that they can be reverse denoised to restore the original image. The latent codes and the corresponding layer text codes are input into the pre-trained stable diffusion model for iterative denoising. In each denoising process, the key matrix and value matrix of each self-attention layer of the stable diffusion model are recorded synchronously to obtain the reference key matrix and reference value matrix. These matrices imply the semantic association features and detailed texture information of the reference image, providing visual guidance signals for the subsequent layer generation stage, ensuring that the generated object is highly consistent with the reference image in visual features.

[0035] The inversion algorithm in this embodiment may be a denoising diffusion implicit model inversion method (DDIM Inversion) or the like, which is not limited in this embodiment of the present invention.

[0036] 2.2 Iterative time step denoising For each subject object, noise is sampled from a standard normal distribution as its corresponding initial layer noise image, and the initial layer noise image, layer final layout, reference image visual embedding, reference image mask, layer text encoding, reference key matrix and reference value matrix corresponding to each subject object are input into a pre-trained layer diffusion model (Layer DiffusionModel), and the initial layer noise image is denoised by iteration for multiple time steps to obtain the layer features of each time step, and then the layer features of the last time step are decoded by a transparent image decoder to obtain the subject transparent layer image corresponding to each subject object. The transparent image decoder type in this embodiment is a decoder module of a variational auto-encoder (VAE).

[0037] In the denoising process of each time step, for the denoising result of the initial layer noise image in the time step, the self-attention layer expands the self-attention calculation by splicing the reference key matrix and the reference value matrix to generate the visual constraint feature of the time step; the visual constraint feature is input into the cross-attention layer, and the visual constraint feature of the time step is cross-attention calculated with the layer text encoding to obtain the spatial constraint feature, and the layer feature of the time step is obtained by processing the network layer after the cross-attention layer in the hierarchical diffusion model. In this embodiment, the denoising result of the initial layer noise image in a time step refers to the feature vector output by the previous layer network of the self-attention layer in the time step.

[0038] 2.2.1 Self-Attention Layer Calculation Calculate the Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity : ; in, represents the vector dot product operation, represents the Euclidean norm.

[0039] Similarity After normalization, we get Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight : ; in, is a predefined scaling factor, is a predefined offset factor, For the The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object. For the The minimum value of the similarity between the layer text encoding corresponding to the subject object and the visual embeddings of all reference images corresponding to the subject object.

[0040] For the The subject objects construct the mask matrix , , Indicates the number of reference image visual embeddings corresponding to the subject object, Indicates the first The similarity weights of the visual embeddings of the reference images, Indicates the first The reference image mask of the reference image.

[0041] Based on the mask matrix The self-attention matrix of the self-attention layer is dynamically modified, and the modified self-attention matrix is ​​used to generate visual constraint features that integrate the details of the reference image. for: ; in, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix Dimension.

[0042] 2.2.2 Cross-Attention Layer Calculation For each subject object, convert the corresponding layer final layout into a layout mask , layout mask The value of the inner area of ​​the bounding box of the main object in the target image is 1, and the value of other areas is 0.

[0043] The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. , the layout mask Converted into an attention weight correction term, directly acting on the original cross attention matrix , and get the corrected cross attention matrix : ; in, is the background suppression factor, whose value is a constant less than the threshold value. The value is 0.000001. Cross-attention correction guided by layout mask, layout mask The area with a median value of 1 retains the original semantic association, and the attention weight of the area with a median value of 0 is decayed to near zero, thereby strictly constraining the generated content to be within the bounding box.

[0044] Based on the modified cross attention matrix Compute spatial constraint features at each time step.

[0045] 3. Multi-layer synthesis The subject transparent layer images generated for each subject object in the previous steps are arranged into multiple layers in any order, and are stacked layer by layer from the bottom layer to the top layer to generate a mixed image. After that, the mixed image and the global text encoding are input into the pre-trained diffusion model-based image restoration network, and the target image with the details of the subject object is generated by denoising. The image restoration network adopts a spatial constraint mechanism to only repair the background area marked by the transparent channel in the mixed image, so that the generated result not only meets the global semantic consistency, but also maintains the high alignment of the subject object with the original layer, and finally outputs a high-quality and visually coherent target image.

[0046] The image restoration network in this embodiment may be a repaint diffusion model (RePaint Model), a diffusion restoration model (Stable diffusion Inpainting Model), etc., and the embodiment of the present invention does not limit this.

[0047] The step of layer-by-layer superposition to generate a mixed image in the embodiment of the present invention includes the following steps: First, the bottommost subject transparent layer image is used as the initial overlay image; then, the overlay image is repeatedly overlaid with the subject transparent layer image of the previous layer to obtain an intermediate mixed image, and the intermediate mixed image is used as the new overlay image. This process continues until all subject transparent layer images are overlaid, and finally a mixed image is obtained. Through this layer-by-layer overlay method, the layer information corresponding to different subject objects can be organically integrated to form a mixed representation containing rich semantic content.

[0048] The overlay image is overlaid with the main transparent layer image of the previous layer according to the following formula: ; ; ; ; in, , , , are the values ​​of the red, green, blue, and transparent channels of the superimposed image, , , , The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. , , , It is the value of the red, green, blue and transparent channels of the intermediate mixed image obtained by superposition.

[0049] In summary, the present invention discloses a multimodal guided progressive image generation method, which independently processes each subject object by introducing layer text prompts and reference images. Even in the face of complex scenes containing multiple subject objects, it can effectively generate images that meet the given description, thereby solving the defects of existing methods in processing complex scenes; by adopting a pre-trained layout denoising network, a hierarchical diffusion model, and an image restoration network, it ensures that the generated image can be consistent with the text description both globally and locally, which not only improves the quality of the generated image, but also improves the global and local consistency between the generated image and the text description. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0050] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent attributes or modifications accomplished by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multi-modality guided progressive image generation method, characterized in that: include: Obtaining, by encoding, a reference image visual embedding corresponding to each reference image in the input data, a global text encoding corresponding to a global text prompt, and a layer text encoding corresponding to each layer text prompt; wherein the number of reference images in the input data is arbitrary, each layer text prompt and each reference image correspond to a subject object, and each subject object corresponds to a layer text prompt and an arbitrary number of reference images; The final layout of layers is generated for each subject object through noise sampling and multimodal feature fusion methods; Use the inversion algorithm to generate latent codes for all reference images; input the latent codes and layer text codes into the stable diffusion model for iterative denoising, and extract the reference key matrix and reference value matrix; The sampling noise is used as the initial layer noise image corresponding to each subject object. Based on the reference image visual embedding, the final layer layout, the reference key matrix and the reference value matrix, the hierarchical diffusion model is used to perform multiple time step iterations to denoise the initial layer noise image, obtain the layer features of each time step, and decode the layer features of the last time step to obtain the subject transparent layer image corresponding to each subject object. Overlay all subject transparent layer images to generate a mixed image; Based on the global text encoding, the mixed image is repaired using an image restoration network to obtain the target image.

2. A multi-modality guided progressive image generation method according to claim 1, characterized in that: Through noise sampling and multimodal feature fusion methods, the steps of generating the final layout of layers for each subject object include: Based on the standard normal distribution, sampling noise for each subject object generates the layer initial layout corresponding to the subject object, and generates layer layout embedding for the layer initial layout through Fourier encoder; For each subject object, a multimodal embedding is generated based on its corresponding reference image visual embedding, layer text encoding, and layer layout embedding; The multimodal embedding is input into the pre-trained layout denoising network for denoising to generate the final layout of the layer corresponding to the main object.

3. A multi-modality guided progressive image generation method according to claim 2, characterized in that: The reference image includes a subject reference image, wherein the subject reference image describes a corresponding subject object; And, for each subject object, the steps of generating a multimodal embedding according to its corresponding reference image visual embedding, layer text encoding, and layer layout embedding include: For Subject objects, generate their corresponding multimodal embeddings according to the following formula : ； in, represents the multi-layer perceptron layer, Represents a splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The subject reference image corresponding to the subject object corresponds to the reference image visual embedding, Represents a vector filling operation. When a subject object corresponds to a subject reference image, represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

4. The multi-modality guided progressive image generation method according to claim 1, characterized in that: The input data also includes a reference image mask, wherein the reference image corresponds to the reference image mask one by one; And, the steps of performing multiple time step iterations to denoise the initial layer noise image and obtaining the layer features of each time step include: Iterate multiple time steps to denoise the initial layer noise image, and the steps in each time step include: In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image in this time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of this time step; In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation to generate the spatial constraint features of this time step; The spatial constraint features are input into the subsequent network layers of the hierarchical diffusion model to generate the layer features of this time step.

5. A multi-modality guided progressive image generation method according to claim 4, characterized in that: In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image in the time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of the time step, including the following steps: For each subject object, calculate the similarity weight between its corresponding layer text encoding and each reference image visual embedding corresponding to the subject object; Construct a mask matrix for each subject object based on the similarity weight and the reference image mask; For the denoising result of the initial layer noise image in this time step, the self-attention matrix of the self-attention layer is dynamically corrected based on the mask matrix, and the visual constraint features are generated according to the corrected self-attention matrix.

6. A multi-modality guided progressive image generation method according to claim 5, characterized in that: For each subject object, the step of calculating the similarity weight between the layer text encoding corresponding to the subject object and each reference image visual embedding corresponding to the subject object includes: Calculate the The layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity : ； in, represents the vector dot product operation, represents the Euclidean norm; Calculate the layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight : ； in, is a predefined scaling factor, is a predefined offset factor, The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object is obtained. The minimum value of the similarity between the layer text encoding corresponding to the main object and the visual embeddings of all reference images corresponding to the main object.

7. The multi-modality guided progressive image generation method according to claim 5, characterized in that: The modified self-attention matrix is: ； in, is the modified self-attention matrix, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix The dimension of Represents a mask matrix.

8. The multi-modality guided progressive image generation method according to claim 4, characterized in that: In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation. The steps of generating the spatial constraint features of this time step include: For each subject object, convert the corresponding layer final layout into a layout mask ; The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. ; Based on layout mask , for the original cross attention matrix Make corrections to get the corrected cross attention matrix : ； in, is the background suppression factor, Represents a pixel-by-pixel multiplication operation; Based on the modified cross attention matrix , generating the spatial constraint features for this time step.

9. The multi-modality guided progressive image generation method according to claim 1, characterized in that: The steps of superimposing all the subject transparent layer images to generate a mixed image include: Arrange the subject transparent layer images corresponding to all subject objects into multiple layers from the bottom layer to the top layer in any order; Use the bottommost transparent layer image as the overlay image; Repeat the process of superimposing the superimposed image with the previous main transparent layer image to obtain an intermediate mixed image, and use the intermediate mixed image as a new superimposed image; until all the main transparent layer images are superimposed to obtain a mixed image.

10. The multi-modality guided progressive image generation method according to claim 9, characterized in that: The steps to overlay the overlay image with the previous main transparent layer image include: The overlay image is overlaid with the main transparent layer image of the previous layer according to the following formula: ； ； ； ； in, , , , are the values ​​of the red, green, blue, and transparent channels of the superimposed image, , , , The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. , , , It is the value of the red, green, blue and transparent channels of the intermediate mixed image obtained by superposition.

Images