Multimodality-guided progressive image generation method

Abstract

The present invention relates to the field of image processing, and more specifically to a multimodal guided progressive image generation method, comprising: generating multimodal features for received data; generating a final layer layout for each subject object using a layout denoising network through noise sampling and multimodal feature fusion methods; extracting the latent code of a reference image through an inversion algorithm, iteratively denoising it using a stable diffusion model in combination with the layer text code, and extracting a reference key value matrix; inputting the initial noise, the final layer layout, the multimodal features, and the reference key value matrix into a hierarchical diffusion model, performing multi-step denoising and decoding to generate a subject transparent layer; after superimposing all subject transparent layers to form a mixed image, driving an image restoration network based on the global text code to generate a target image. The present invention addresses the shortcomings of existing methods in processing complex scenes, ensuring that the generated image is consistent with the text description both globally and locally.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-modality guided progressive image generation method. Background Art

[0002] Generating images from text is an important research topic at the intersection of computer vision and natural language processing. Its goal is to generate images based on user-provided text descriptions. This technology has many practical applications, such as automated design, virtual reality, game design, and assisted teaching.

[0003] Existing methods for generating images from text typically involve two steps: encoding and decoding. The encoding step converts the text into a computer-interpretable text encoding (e.g., a vector). The decoding step uses this text encoding as guidance and feeds it into an image generation model to produce an image. These methods often struggle to generate images containing multiple objects, especially when the attributes and relationships between these objects are complex. Furthermore, existing methods often fail to effectively ensure that the generated image is globally and locally consistent with the input text description, resulting in a lack of consistency and coherence in the generated images. Summary of the Invention

[0004] To solve the above problems, the present invention provides a multimodal guided progressive image generation method for generating a target image including multiple subject objects, the method comprising:

[0005] Obtain, by encoding, a reference image visual embedding corresponding to each reference image in the input data, a global text encoding corresponding to the global text prompt, and a layer text encoding corresponding to each layer text prompt; wherein the number of reference images in the input data is arbitrary, each layer text prompt and each reference image corresponds to a subject object, and each subject object corresponds to a layer text prompt and an arbitrary number of reference images;

[0006] Generate the final layout of layers for each subject object through noise sampling and multimodal feature fusion method;

[0007] Use the inversion algorithm to generate latent codes for all reference images; input the latent codes and layer text codes into the stable diffusion model for iterative denoising, and extract the reference key matrix and reference value matrix;

[0008] The sampled noise is used as the initial layer noise image corresponding to each subject object. Based on the reference image visual embedding, the final layer layout, the reference key matrix and the reference value matrix, a hierarchical diffusion model is used to iterate over multiple time steps to denoise the initial layer noise image, obtain the layer features of each time step, and decode the layer features of the last time step to obtain the subject transparent layer image corresponding to each subject object.

[0009] Superimpose all subject transparent layer images to generate a mixed image;

[0010] Based on the global text encoding, the image restoration network is used to repair the mixed image to obtain the target image.

[0011] Furthermore, the steps of generating the final layout of the layer for each subject object through noise sampling and multimodal feature fusion method include:

[0012] Based on the standard normal distribution, noise is sampled for each subject object to generate the corresponding layer initial layout of the subject object, and a layer layout embedding is generated for the layer initial layout through a Fourier encoder;

[0013] For each subject object, a multimodal embedding is generated based on its corresponding reference image visual embedding, layer text encoding, and layer layout embedding;

[0014] The multimodal embedding is input into the pre-trained layout denoising network for denoising to generate the final layer layout corresponding to the subject object.

[0015] Furthermore, the reference image includes a subject reference image, wherein the subject reference image describes its corresponding subject object;

[0016] And, for each subject object, the steps of generating a multimodal embedding according to its corresponding reference image visual embedding, layer text encoding, and layer layout embedding include:

[0017] For the first Subject objects, generate their corresponding multimodal embeddings according to the following formula :

[0018] ;

[0019] in, represents the multi-layer perceptron layer, Represents the splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The reference image visual embedding of the subject reference image corresponding to the subject object, Represents a vector filling operation. When a subject object corresponds to a subject reference image, Represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

[0020] Furthermore, the input data also includes a reference image mask, wherein the reference image corresponds to the reference image mask in a one-to-one manner;

[0021] Furthermore, the steps of performing multiple time step iterations to denoise the initial layer noise image and obtain the layer features of each time step include:

[0022] Iterate multiple time steps to denoise the initial layer noisy image. The steps in each time step include:

[0023] In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image at this time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of this time step;

[0024] In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation to generate the spatial constraint features of this time step;

[0025] The spatial constraint features are input into the subsequent network layers of the hierarchical diffusion model to generate the layer features of this time step.

[0026] Furthermore, in the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image at the time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of the time step. The steps include:

[0027] For each subject object, calculate the similarity weight between its corresponding layer text encoding and each reference image visual embedding corresponding to the subject object;

[0028] Construct a mask matrix for each subject object based on the similarity weight and the reference image mask;

[0029] For the denoising result of the initial layer noise image in this time step, the self-attention matrix of the self-attention layer is dynamically corrected based on the mask matrix, and the visual constraint features are generated according to the corrected self-attention matrix.

[0030] Furthermore, for each subject object, the step of calculating the similarity weight between the layer text encoding corresponding to the subject object and the visual embedding of each reference image corresponding to the subject object includes:

[0031] Calculate the Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity :

[0032] ;

[0033] in, represents the vector dot product operation, represents the Euclidean norm;

[0034] Calculate the layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight :

[0035] ;

[0036] in, is a predefined scaling factor, is a predefined offset factor, The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object is obtained. The minimum value of the similarity between the layer text encoding corresponding to the subject object and the visual embeddings of all reference images corresponding to the subject object is obtained.

[0037] Furthermore, the modified self-attention matrix is:

[0038] ;

[0039] in, is the modified self-attention matrix, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix Dimensions, Represents the mask matrix.

[0040] Furthermore, in the cross-attention layer, based on the final layer layout, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation. The steps of generating the spatial constraint features of this time step include:

[0041] For each subject object, convert the corresponding layer final layout into a layout mask ;

[0042] The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. ;

[0043] Based on layout mask , for the original cross attention matrix Make corrections to get the corrected cross attention matrix :

[0044] ;

[0045] in, is the background suppression factor, Represents a pixel-by-pixel multiplication operation;

[0046] Based on the modified cross attention matrix , generating the spatial constraint features of this time step.

[0047] Furthermore, the steps of superimposing all the subject transparent layer images to generate a mixed image include:

[0048] Arrange the subject transparent layer images corresponding to all subject objects into multiple layers from the bottom layer to the top layer in any order;

[0049] Use the bottommost transparent layer image as the overlay image;

[0050] Repeat the process of superimposing the superimposed image with the previous main transparent layer image to obtain an intermediate mixed image, and use the intermediate mixed image as a new superimposed image; until all the main transparent layer images are superimposed to obtain a mixed image.

[0051] Furthermore, the step of superimposing the superimposed image with the upper main transparent layer image includes:

[0052] The overlay image is overlaid with the main transparent layer image of the previous layer according to the following formula:

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] in, 、 、 、 are the values ​​of the red, green, blue and transparent channels of the superimposed image, 、 、 、 The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. 、 、 、 The red, green, blue and transparent channel values ​​of the intermediate mixed image obtained by superposition.

[0058] The present invention introduces layer text prompts and reference images to independently process each subject object. Even when faced with complex scenes containing multiple subject objects, it can effectively generate images that meet the given description, thereby solving the defects of existing methods in processing complex scenes; by adopting pre-trained layout denoising network, hierarchical diffusion model, and image restoration network, it ensures that the generated image is consistent with the text description both globally and locally, which not only improves the quality of the generated image, but also improves the global and local consistency between the generated image and the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 This is a flow chart of a multimodal guided progressive image generation method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, or product comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to such process, method, product, or end.

[0063] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] The present invention proposes a multimodal guided progressive image generation method, which constructs an image generation model to generate a target image. The method consists of three steps: multimodal guided layer layout generation, zero-sample multimodal guided layer generation, and multi-layer synthesis. The specific steps are as follows: Figure 1 shown.

[0065] In the multimodal guided layer layout generation step, the global text prompt, layer text prompt, reference image and reference image mask are first received as input data, the reference image is preprocessed by cropping and pasting using the reference image mask to obtain the preprocessed reference image, the visual encoder is used to extract the reference image visual embedding, the text encoder is used to extract the layer text encoding for the layer text prompt, the initial layer layout is generated, the layer layout embedding is obtained through the Fourier encoder, the features of the above three modalities are fused and input into the layout denoising network to obtain the final layer layout.

[0066] In the zero-shot multimodal guided layer generation step, the inverted reference image is fed into a stable diffusion model. During the denoising process, the key-value matrices of each self-attention layer are extracted to obtain reference key and reference value matrices. Next, noise is used as the initial layer noise image. The initial layer noise image, the final layer layout, the reference image visual embedding, the reference image mask, the layer text encoding, the reference key matrix, and the reference value matrix corresponding to each subject object are fed into the hierarchical diffusion model. During multiple rounds of denoising, the reference key matrix, reference value matrix, and reference image mask are used to correct the self-attention layer calculations, and the final layer layout is used to correct the cross-attention layer calculations. After denoising, the layer features of the final time step are obtained and decoded into the subject transparent layer image corresponding to each subject object through the decoder.

[0067] The multi-layer synthesis step is to superimpose the subject transparent layer images generated by each subject object layer by layer to generate a mixed image, and then input it into the image restoration network based on the diffusion model with the global text encoding to retain the subject object and complete the background area, and finally generate a high-quality and visually coherent target image.

[0068] 1. Multimodality-guided Layer Layout Generation

[0069] 1.1 Data Preprocessing

[0070] Input data for generating a target image is received, the input data including a global text prompt, at least one layer text prompt, any number of reference images, and a reference image mask corresponding to each reference image. The global text prompt describes the content of the target image, each target image includes at least one subject object, and the subject object corresponds one-to-one with the layer text prompt. Each layer text prompt describes its corresponding subject object, and one subject object corresponds to any number of reference images, and each reference image corresponds to one subject object. There are two types of reference images: subject reference images and attribute reference images. The subject reference image describes its corresponding subject object, and the attribute reference image describes the attributes of its corresponding subject object. The reference image mask represents the area in its corresponding reference image that is related to the corresponding subject object or the attributes of the subject object.

[0071] For example, in the input data, the global text prompt is "A girl wearing a hat holding an orange cat," and the two layer text prompts are "A girl with short hair wearing a hat and carrying a backpack" and "An orange cat." The input data contains two reference images. The first reference image is a subject reference image of a girl sitting on a sofa, corresponding to the subject object "girl." The second reference image is an attribute reference image of a boy carrying a backpack, corresponding to the attribute "backpack" of the subject object "girl." The reference image mask is a binary image. In the reference image mask corresponding to the first reference image, areas with a pixel value of 1 represent the area of ​​the girl in the first reference image, while areas with a pixel value of 0 represent other areas. In the reference image mask corresponding to the second reference image, areas with a pixel value of 1 represent the area of ​​the backpack in the second reference image, while areas with a pixel value of 0 represent other areas.

[0072] The reference image corresponding to the reference image mask is cropped and pasted into a blank image of a predetermined size to obtain a preprocessed reference image. In this embodiment, the blank image refers to an image with all pixels white. By cropping, irrelevant background is removed from the reference image, allowing the image generation model to focus more on the subject and its attributes.

[0073] 1.2 Feature Extraction

[0074] To guide the image generation model to focus on the semantic information and aspect ratio information in the preprocessed reference images, a visual encoder is used to extract the visual features of each preprocessed reference image, obtaining the corresponding reference image visual embedding. A text encoder is used to encode the global text cue to obtain a global text encoding, and a text encoder is used to encode each layer of text cue to obtain its corresponding layer text encoding.

[0075] The visual encoder in this embodiment is a contrastive language-image pre-training (CLIP) visual encoder. CLIP is a powerful multimodal learning model that can learn a unified visual-language representation space by performing contrastive learning on a large number of image-text pairs. In this space, semantically similar images and texts will have similar vector representations. Therefore, the features extracted using the CLIP visual encoder not only contain the visual information of the reference image, but also contain rich semantic information. The text encoder in this embodiment is a contrastive language-image pre-training text encoder.

[0076] For each subject object, noise is sampled from a standard normal distribution to generate the corresponding initial layer layout. Each initial layer layout represents the position information of the bounding box of the corresponding subject object in the target image. In this embodiment, each initial layer layout is represented by the coordinates of the upper left and lower right corners of the bounding box. Each initial layer layout is mapped to a higher dimension using a Fourier encoder, enhancing the feature vector's ability to represent high-frequency information, resulting in a layer layout embedding.

[0077] For each subject object, the visual embedding is adapted according to whether it corresponds to a subject reference image: if there is a corresponding subject reference image, the visual embedding of the reference image corresponding to the subject reference image is retained; otherwise, a learnable empty vector is used as a placeholder; then the layer text encoding representing semantic information, the layer layout embedding representing layout information, and the features after the visual embedding adaptation are spliced ​​in three dimensions, and cross-modal feature fusion is achieved through a multi-layer perceptron, and finally a unified multimodal embedding is generated, where the first Multimodal embedding corresponding to the subject object for:

[0078] ;

[0079] in, represents the multi-layer perceptron layer, Represents the splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The reference image visual embedding of the subject reference image corresponding to the subject object, Represents a vector filling operation. When a subject object corresponds to a subject reference image, Represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

[0080] 1.3 Layer Layout Generation

[0081] The multimodal embedding corresponding to each subject object is input into the layout denoising network for denoising, generating the final layer layout corresponding to that subject object. The layout denoising network is a layout generation model based on a diffusion model. By gradually removing noise and refining feature representations, it can generate a high-quality final layer layout that matches the conditional constraints from the noisy initial layer layout.

[0082] The diffusion model is a generative model whose core idea is to destroy the data structure by gradually adding noise, and then learn how to reverse this process to generate samples similar to the original data distribution. The diffusion model models the data distribution through a series of Markov chain transition steps. At each step, it learns to remove part of the noise, and finally obtains clean generated samples. This step-by-step denoising process enables the diffusion model to generate high-quality and diverse samples with strong interpretability and controllability. In the embodiment of the present invention, the layout denoising network can be a denoising diffusion probabilistic model (DDPM, Denoising Diffusion Probabilistic Models), a denoising diffusion implicit model (DDIM, Denoising Diffusion Implicit Models), etc., and the embodiment of the present invention does not limit it.

[0083] 2. Zero-shot multimodal guided layer generation

[0084] 2.1 Reference Image Key-Value Pair Extraction

[0085] Through the inversion and denoising process, the key key-value pairs of the reference image in the self-attention layer of the stable diffusion model are extracted to provide guidance signals for subsequent visual constraints.

[0086] An inversion algorithm is used to generate latent codes for all reference images. The latent codes are constructed through an independent noise sampling strategy to ensure that they can be reverse denoised to restore the original image. The latent codes and the corresponding layer text codes are input into the pre-trained stable diffusion model for iterative denoising. In each denoising step, the key matrix and value matrix of each self-attention layer of the stable diffusion model are synchronously recorded to obtain the reference key matrix and reference value matrix. These matrices imply the semantic association features and detailed texture information of the reference image, providing visual guidance signals for the subsequent layer generation stage, ensuring that the generated object is highly consistent with the reference image in visual features.

[0087] The inversion algorithm in this embodiment may be a denoising diffusion implicit model inversion method (DDIM Inversion), etc., which is not limited in this embodiment of the present invention.

[0088] 2.2 Iterative time-step denoising

[0089] For each subject object, noise is sampled from a standard normal distribution as its corresponding initial layer noise image. The initial layer noise image, final layer layout, reference image visual embedding, reference image mask, layer text encoding, reference key matrix, and reference value matrix corresponding to each subject object are input into a pre-trained layer diffusion model (Layer DiffusionModel). The initial layer noise image is denoised over multiple time steps to obtain layer features at each time step. The layer features at the last time step are then decoded using a transparent image decoder to obtain the subject transparent layer image corresponding to each subject object. The transparent image decoder in this embodiment is a variational auto-encoder (VAE) decoder module.

[0090] During the denoising process at each time step, the self-attention layer expands the self-attention calculation by concatenating the reference key matrix and the reference value matrix to generate the visual constraint features of the time step. The visual constraint features are input into the cross-attention layer, and the visual constraint features of the time step are subjected to spatial constraint cross-attention calculation with the layer text encoding to obtain spatial constraint features. The spatial constraint features are then processed by the network layer following the cross-attention layer in the hierarchical diffusion model to obtain the layer features of the time step. In this embodiment, the denoising result of the initial layer noise image at a time step refers to the feature vector output by the network layer above the self-attention layer at that time step.

[0091] 2.2.1 Self-Attention Layer Calculation

[0092] Calculate the Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity :

[0093] ;

[0094] in, represents the vector dot product operation, represents the Euclidean norm.

[0095] Similarity Perform normalization and obtain Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight :

[0096] ;

[0097] in, is a predefined scaling factor, is a predefined offset factor, For the The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object, For the The minimum value of the similarity between the layer text encoding corresponding to the subject object and the visual embeddings of all reference images corresponding to the subject object.

[0098] For the Build a mask matrix for each subject object , , Indicates the number of reference image visual embeddings corresponding to the subject object, Indicates the first Similarity weights of the visual embeddings of reference images, Indicates the first The reference image mask of the reference image.

[0099] Based on the mask matrix The self-attention matrix of the self-attention layer is dynamically modified, and the modified self-attention matrix is ​​used to generate visual constraint features that integrate the details of the reference image. The modified self-attention matrix for:

[0100] ;

[0101] in, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix dimension.

[0102] 2.2.2 Cross-Attention Layer Calculation

[0103] For each subject object, convert the corresponding layer final layout into a layout mask , layout mask The value of the inner area of ​​the bounding box of the main object in the target image is 1, and the value of other areas is 0.

[0104] The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. , the layout mask Converted into an attention weight correction term, directly acting on the original cross attention matrix , and get the corrected cross attention matrix :

[0105] ;

[0106] in, is the background suppression factor, whose value is a constant less than the threshold. The value is 0.000001. Cross attention correction guided by layout mask, layout mask The area with a median value of 1 retains the original semantic association, and the attention weight of the area with a median value of 0 is decayed to near zero, thereby strictly constraining the generated content to be within the bounding box.

[0107] Based on the modified cross attention matrix Compute spatial constraint features at each time step.

[0108] 3. Multi-layer synthesis

[0109] The subject transparent layer images generated for each subject in the previous steps are arranged into multiple layers in arbitrary order and stacked layer by layer, from bottom to top, to generate a hybrid image. The hybrid image and the global text encoding are then fed into a pre-trained diffusion model-based image restoration network, which performs denoising to generate a target image that retains the subject's details. The image restoration network employs a spatial constraint mechanism to inpaint only the background regions marked by the transparent channel in the hybrid image. This ensures that the generated result is both globally semantically consistent and maintains a high degree of alignment between the subject and the original layers, ultimately outputting a high-quality and visually coherent target image.

[0110] The image restoration network in this embodiment may be a repaint diffusion model (RePaint Model), a stable diffusion inpainting model (Stable diffusion Inpainting Model), etc., and the embodiment of the present invention does not limit this.

[0111] The layer-by-layer superposition to generate a mixed image in the embodiment of the present invention includes the following steps:

[0112] First, the bottommost subject transparent layer image is used as the initial overlay image. Then, the overlay image is repeatedly superimposed with the subject transparent layer image above it to obtain an intermediate mixed image, which is then used as the new overlay image. This process continues until all subject transparent layer images are superimposed, resulting in a mixed image. This layer-by-layer overlay method organically integrates the layer information corresponding to different subject objects, forming a mixed representation with rich semantic content.

[0113] The overlay image is superimposed with the main transparent layer image of the previous layer according to the following formula:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] in, 、 、 、 are the values ​​of the red, green, blue and transparent channels of the superimposed image, 、 、 、 The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. 、 、 、 The red, green, blue and transparent channel values ​​of the intermediate mixed image obtained by superposition.

[0119] In summary, the present invention discloses a multimodal guided progressive image generation method, which independently processes each subject object by introducing layer text prompts and reference images. Even when faced with complex scenes containing multiple subject objects, it can effectively generate images that meet the given description, thereby solving the defects of existing methods in processing complex scenes; by adopting a pre-trained layout denoising network, a layered diffusion model, and an image restoration network, it ensures that the generated image is consistent with the text description both globally and locally, which not only improves the quality of the generated image, but also improves the global and local consistency between the generated image and the text description. Therefore, the present invention effectively overcomes the various shortcomings of the existing technology and has high industrial utilization value.

[0120] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations accomplished by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A multimodal guided progressive image generation method, characterized in that: include: Obtain, by encoding, a reference image visual embedding corresponding to each reference image in the input data, a global text encoding corresponding to the global text prompt, and a layer text encoding corresponding to each layer text prompt; wherein the number of reference images in the input data is arbitrary, each layer text prompt and each reference image corresponds to a subject object, and each subject object corresponds to a layer text prompt and an arbitrary number of reference images; Generate the final layout of layers for each subject object through noise sampling and multimodal feature fusion method; Use the inversion algorithm to generate latent codes for all reference images; input the latent codes and layer text codes into the stable diffusion model for iterative denoising, and extract the reference key matrix and reference value matrix; The sampled noise is used as the initial layer noise image corresponding to each subject object. Based on the reference image visual embedding, the final layer layout, the reference key matrix and the reference value matrix, a hierarchical diffusion model is used to iterate over multiple time steps to denoise the initial layer noise image, obtain the layer features of each time step, and decode the layer features of the last time step to obtain the subject transparent layer image corresponding to each subject object. Superimpose all subject transparent layer images to generate a mixed image; Based on the global text encoding, the image restoration network is used to repair the mixed image to obtain the target image; And, the input data also includes a reference image mask, wherein the reference image and the reference image mask have a one-to-one correspondence; Furthermore, the steps of performing multiple time step iterations to denoise the initial layer noise image and obtain the layer features of each time step include: Iterate multiple time steps to denoise the initial layer noisy image. The steps in each time step include: In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image at this time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask to generate the visual constraint feature of this time step; In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation to generate the spatial constraint features of this time step; The spatial constraint features are input into the subsequent network layers of the hierarchical diffusion model to generate the layer features of this time step.

2. The multimodal guided progressive image generation method according to claim 1, characterized in that: The steps to generate the final layout of layers for each subject object through noise sampling and multimodal feature fusion methods include: Based on the standard normal distribution, noise is sampled for each subject object to generate the corresponding layer initial layout of the subject object, and a layer layout embedding is generated for the layer initial layout through a Fourier encoder; For each subject object, a multimodal embedding is generated based on its corresponding reference image visual embedding, layer text encoding, and layer layout embedding; The multimodal embedding is input into the pre-trained layout denoising network for denoising to generate the final layer layout corresponding to the subject object.

3. The multimodal guided progressive image generation method according to claim 2, characterized in that: The reference image includes a subject reference image, wherein the subject reference image describes its corresponding subject object; And, for each subject object, the steps of generating a multimodal embedding according to its corresponding reference image visual embedding, layer text encoding, and layer layout embedding include: For the first Subject objects, generate their corresponding multimodal embeddings according to the following formula : ； in, represents the multi-layer perceptron layer, Represents the splicing operation, Representative The layer layout corresponding to each subject object is embedded. Representative The layer text encoding corresponding to the subject object, Representative The reference image visual embedding of the subject reference image corresponding to the subject object, Represents a vector filling operation. When a subject object corresponds to a subject reference image, Represents the visual embedding of the reference image corresponding to the subject reference image. When a subject object does not correspond to a subject reference image, Represents the operation of generating learnable empty vectors.

4. The multimodal guided progressive image generation method according to claim 1, characterized in that: In the self-attention layer of the hierarchical diffusion model, the denoising result of the initial layer noise image at this time step is extended by concatenating the reference key matrix and the reference value matrix based on the reference image visual embedding and the reference image mask. The steps of generating the visual constraint feature of this time step include: For each subject object, calculate the similarity weight between its corresponding layer text encoding and each reference image visual embedding corresponding to the subject object; Construct a mask matrix for each subject object based on the similarity weight and the reference image mask; For the denoising result of the initial layer noise image in this time step, the self-attention matrix of the self-attention layer is dynamically corrected based on the mask matrix, and the visual constraint features are generated according to the corrected self-attention matrix.

5. The multimodal guided progressive image generation method according to claim 4, characterized in that: For each subject object, the step of calculating the similarity weight between its corresponding layer text encoding and each reference image visual embedding corresponding to the subject object includes: Calculate the Layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity : ； in, represents the vector dot product operation, represents the Euclidean norm; Calculate the layer text encoding corresponding to the subject object The first Reference image visual embedding Similarity weight : ； in, is a predefined scaling factor, is a predefined offset factor, The maximum value of the similarity between the layer text encoding corresponding to the subject object and the visual embedding of all reference images corresponding to the subject object is obtained. The minimum value of the similarity between the layer text encoding corresponding to the subject object and the visual embeddings of all reference images corresponding to the subject object is obtained.

6. The multimodal guided progressive image generation method according to claim 4, characterized in that: The modified self-attention matrix is: ； in, is the modified self-attention matrix, represents a pixel-by-pixel multiplication operation, represents the query matrix of the self-attention layer, represents the concatenation matrix of the original bond matrix of the self-attention layer and the reference bond matrix, Represents the concatenation matrix of the original value matrix and the reference value matrix of the self-attention layer, represents the matrix transpose operation, Represents the concatenation matrix Dimensions, Represents the mask matrix.

7. The multimodal guided progressive image generation method according to claim 1, characterized in that: In the cross-attention layer, based on the final layout of the layer, the visual constraint features and the layer text encoding are subjected to spatial constraint cross-attention calculation. The steps to generate the spatial constraint features of this time step include: For each subject object, convert the corresponding layer final layout into a layout mask ; The original cross attention matrix is ​​constructed using the visual constraint feature as the query vector and the layer text encoding corresponding to the subject object as the key vector and value vector. ; Based on layout mask , for the original cross attention matrix Make corrections to get the corrected cross attention matrix : ； in, is the background suppression factor, Represents a pixel-by-pixel multiplication operation; Based on the modified cross attention matrix , generating the spatial constraint features of this time step.

8. The multimodal guided progressive image generation method according to claim 1, characterized in that: The steps for superimposing all subject transparent layer images to generate a mixed image include: Arrange the subject transparent layer images corresponding to all subject objects into multiple layers from the bottom layer to the top layer in any order; Use the bottommost transparent layer image as the overlay image; Repeat the process of superimposing the superimposed image with the previous main transparent layer image to obtain an intermediate mixed image, and use the intermediate mixed image as a new superimposed image; until all the main transparent layer images are superimposed to obtain a mixed image.

9. The multimodal guided progressive image generation method according to claim 8, characterized in that: The steps to overlay an overlay image with the previous main transparent layer image include: The overlay image is overlaid with the main transparent layer image of the previous layer according to the following formula: ； ； ； ； in, 、 、 、 are the values ​​of the red, green, blue and transparent channels of the superimposed image, 、 、 、 The red, green, blue and transparent channel values ​​of the main transparent layer image of the previous layer. 、 、 、 The red, green, blue and transparent channel values ​​of the intermediate mixed image obtained by superposition.

Images