Text-guided zero-shot transparent layer and layered image generation method

Abstract

The present application relates to the field of image processing, in particular to a kind of text-guided zero sample transparent layer and layered image generation method, layered image generation method includes: input global image text prompt, target image size and layer text prompt to foreground position information generation model, obtain foreground position information;For each layer text prompt generates first target image;According to the transparent channel of all first target image, generate soft segmentation mask;Superimpose all first target image, and encode to latent space, obtain foreground superimposed latent feature;Randomly sample Gaussian noise as initial background latent feature.According to soft segmentation mask, in the iterative denoising process, mix foreground superimposed latent feature with initial background latent feature, obtain global image latent feature, and decode as second target image.The present application precisely controls the position of each main object, enhances the image layout capability of model;Eliminate the step of model training, greatly save computing resources.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to a text-guided zero-sample transparency layer and a method for generating layered images. Background Technology

[0002] In computer image processing, images are typically composed of various channels, the most common being the RGBA channel, where R, G, and B represent red, green, and blue colors respectively, while A represents the alpha channel. An image can usually be divided into a background and a subject. For example, in a photo of an apple on a dining table, the apple is the subject, while the table and other possible objects constitute the background. In practical applications, generating images from text descriptions, known as image generation, is an important task at the intersection of computer vision and natural language processing. The development of this technology enables the generation of images with specific content and layouts from text prompts, which has profound implications for various fields, including computer-aided design, data augmentation, and content creation.

[0003] Among existing image generation methods, diffusion model-based approaches are a mainstream approach. These methods generate images that match the text description through an iterative denoising process within a certain time step. However, when dealing with complex scenes with multiple subjects, this method lacks precise control over image layout and subject positions; furthermore, it typically requires a large amount of training data and computational resources, which is a significant limitation in many practical applications. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a text-guided zero-sample transparent layer and a method for generating layered images.

[0005] The first aspect of this invention discloses a text-guided zero-sample transparency layer generation method. The method generates a first target image with the same size as the target image based on input layer text prompts, target image size, and foreground position information. The layer text prompts describe the content of the first target image, which includes a main object and a transparent background. The foreground position information indicates the position of the main object within the first target image. The method includes:

[0006] Receiving layer text prompts, target image size, and foreground position information;

[0007] Input the layered text hints into the text encoder to obtain the embedded text hints;

[0008] Generate a binary mask based on the foreground position information;

[0009] A transparent image is created based on the target image size, and the transparent image is encoded into the latent space using a transparent image encoder to obtain the latent features of the transparent image; wherein, the first target image, the binary mask, and the transparent image have the same size;

[0010] In the latent space, Gaussian noise is randomly sampled as the initial noise latent feature;

[0011] The system iterates through a first set number of time steps. At each time step, based on the latent features of the transparent image and the binary mask, it modifies the noise latent features output from the previous time step to obtain the corrected noise latent features. It sets a cross-attention matrix based on the binary mask and performs denoising on the corrected noise latent features by adjusting the cross-attention correlation between the text prompt embedding and the corrected noise latent features to obtain the noise latent features output from the current time step. In the first time step, the initial noise latent features are used as the noise latent features output from the previous time step.

[0012] The noise latent features output from the last time step are input into the transparent image decoder to obtain the first target image.

[0013] Furthermore, based on the latent features of the transparent image and the binary mask, the steps for modifying the noise latent features output from the previous time step to obtain the corrected noise latent features include:

[0014] The transparent image latent features are added with first image noise to obtain transparent noisy background latent features. Based on the transparent noisy background latent features and the binary mask, the noise latent features output in the previous time step are modified to obtain the corrected noise latent features. The first image noise is Gaussian noise, and the intensity of the first image noise added in each time step is less than the intensity of the first image noise added in the previous time step.

[0015] Furthermore, based on the latent features of the transparent noisy background and the binary mask, the steps to modify the latent noise features output from the previous time step to obtain the corrected latent noise features include:

[0016] Determine whether the number of iterations at the current time step exceeds a preset second number; wherein the second number is less than the first number;

[0017] When the time limit is not exceeded, based on the binary mask, the background region features in the noise latent features output in the previous time step are determined, and the background region features are replaced with transparent noise-added background latent features to obtain the corrected noise latent features.

[0018] When the time limit is exceeded, the noise latent feature output from the previous time step is determined as the corrected noise latent feature.

[0019] Furthermore, the steps of setting a cross-attention matrix based on a binary mask, adjusting the cross-attention correlation between the text prompt embedding and the corrected noise latent features, and performing denoising on the corrected noise latent features to obtain the noise latent features output at the current time step include:

[0020] Embed text prompts and correct latent noise features into the U-shaped network;

[0021] The text prompts are embedded as keys and values ​​in the cross-attention layer of the U-shaped network, and the corrected noise latent features are used as queries in the cross-attention layer of the U-shaped network. The cross-attention matrix is ​​set according to the binary mask, and the corrected noise latent features are denoised to obtain the noise latent features output at the current time step.

[0022] Furthermore, the steps for setting the cross-attention matrix based on the binary mask include:

[0023] Based on the binary mask, the pixel region corresponding to the latent features of the corrected noise is divided into the foreground region and the background region;

[0024] Set the cross-attention matrix corresponding to the foreground region for:

[0025] ;

[0026] Set the cross-attention matrix corresponding to the background region for:

[0027] ;

[0028] in, This represents the original cross-attention matrix corresponding to the foreground region. This represents the original cross-attention matrix corresponding to the background region. This represents the correction factor corresponding to the foreground region. This represents the correction factor corresponding to the background area. Indicates the first A text prompt is embedded. Indicates the number of text prompts embedded. This indicates the embedding of the text prompt corresponding to the preset text start marker. This indicates the embedding of a text prompt corresponding to a preset text end marker.

[0029] A second aspect of this invention discloses a text-guided zero-sample layered image generation method. The method generates a second target image based on global image text prompts, a target image size, and at least one layer of text prompts. The global image text prompts describe the content of the second target image, and each layer of text prompts describes a main object in the second target image. The method includes:

[0030] The global image text prompt, the target image size, and at least one layer of text prompt are input into the foreground position information generation model to obtain the foreground position information corresponding to each layer of text prompt; wherein, the foreground position information generation model is a pre-trained large language model;

[0031] Based on the target image size, for each layer of text prompt and its corresponding foreground position information, the zero-sample transparent layer generation method for text guidance as described in any of the first aspects of the present invention is used to generate the first target image corresponding to the text prompt of that layer;

[0032] Generate a soft segmentation mask based on the alpha channels in all the first target images;

[0033] All the first target images are superimposed to obtain a foreground superimposed image;

[0034] The foreground overlay image is encoded into the latent space by an image encoder to obtain the foreground overlay latent features;

[0035] In the latent space, Gaussian noise is randomly sampled as the initial background latent feature;

[0036] Based on the soft segmentation mask, the latent features of the foreground are mixed with the latent features of the initial background to obtain the latent features of the global image;

[0037] The latent features of the global image are input into the image decoder to obtain the second target image.

[0038] Furthermore, based on the soft segmentation mask, the steps of mixing the foreground superimposed latent features with the initial background latent features to obtain the global image latent features include:

[0039] The system iterates through a third preset number of time steps. At each time step, it generates a foreground noisy latent feature based on the foreground superimposed latent feature and Gaussian noise. Based on the soft segmentation mask, it mixes the foreground noisy latent feature with the mixed image latent feature output from the previous time step and outputs the mixed image latent feature of the current time step. In the first time step, the initial background latent feature is used as the mixed image latent feature output from the previous time step.

[0040] The latent features of the mixed image output at the last time step are determined as the latent features of the global image.

[0041] Furthermore, the step of mixing the foreground noisy latent features with the mixed image latent features output from the previous time step, based on the soft segmentation mask, includes:

[0042] No. Each time step is mixed according to the following formula:

[0043] ;

[0044] in, Representing the Latent features of the mixed image output at each time step, Represents a soft segmentation mask. Representing the Noisy latent features at each time step.

[0045] Furthermore, at each time step, the step of generating foreground noisy latent features based on the foreground superimposed latent features and Gaussian noise includes:

[0046] At each time step, the foreground superimposed latent features are added with second image noise to obtain foreground noisy latent features; wherein, the second image noise is Gaussian noise, and the intensity of the second image noise added at each time step is less than the intensity of the second image noise added at the previous time step.

[0047] Furthermore, the step of generating a soft segmentation mask based on the alpha channels in all the first target images includes:

[0048] Generate a soft segmentation mask using the following formula. :

[0049] ;

[0050] in, For the first The transparency channel of the first target image. The number of images representing the first target.

[0051] By introducing transparent layers and foreground position information, this invention can precisely control the position of each main object, thereby enhancing the model's image layout capabilities. The generated images not only conform to the semantic content of the text prompts but also meet the user's layout requirements. Moreover, this invention adopts the zero-shot learning approach, eliminating the need for model training and greatly saving computational resources. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a text-guided zero-sample transparent layer generation method disclosed in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a transparent layer disclosed in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of another transparent layer disclosed in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of yet another transparent layer disclosed in an embodiment of the present invention;

[0057] Figure 5 This is a flowchart illustrating a text-guided zero-sample hierarchical image generation method disclosed in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of a layered image disclosed in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, or product comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, apparatus, or products.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] Please see Figure 1 As shown, Figure 1This is a flowchart illustrating a text-guided zero-sample transparency layer generation method disclosed in an embodiment of the present invention. The zero-sample transparency layer generation method generates a first target image with the same size as the target image based on the input layer text prompt, the target image size, and foreground position information. The layer text prompt describes the content of the first target image, which includes a main object and a transparent background. Figure 2 , Figure 3 , Figure 4 Different first target images are shown, each with a different zebra as its main subject. The foreground position information indicates the position of the main subject in the first target image, such as... Figure 1 As shown, the text-guided zero-sample transparency layer generation method may include the following operations:

[0063] S101, receiving layer text prompts, target image size and foreground position information;

[0064] In this optional embodiment, the layered text cues refer to the textual description of the main object in the first target image, describing the visual attributes of the main object, such as category, color, posture, state, etc. Figure 2 For example, its layered text prompts can include, but are not limited to, the following types of expressions:

[0065] A zebra standing upright;

[0066] A realistic zebra with black and white stripes, facing right;

[0067] A side view of a zebra.

[0068] Foreground position information refers to the spatial position and range of the main object in the first target image. In this embodiment of the invention, it is represented by a bounding box, which is a rectangular box and described by the x-coordinate and y-coordinate of its upper left corner, the width of the bounding box, and the height of the bounding box. Figure 2 For example, suppose Figure 2 The size of the corresponding first target image is Then the bounding box of the main object can be represented as [60,90,380,410]. This quadruple indicates that the top-left corner of the bounding box of the main object is (60,90), the width is 380 pixels, and the height is 410 pixels. The numbers in this quadruple are just examples and do not represent real data.

[0069] Those skilled in the art can deduce that the width and height in the foreground position information are smaller than the width and height in the target image size, respectively.

[0070] S102. Input the layered text prompt into the text encoder to obtain the text prompt embedding;

[0071] In this optional embodiment, the text encoder is a model that converts text into a numerical vector representation, which can map text sequences of variable length to a dense vector space of fixed dimensions. The text encoder in this embodiment can be a Contrastive Language-Image Pre-training (CLIP) model, a Text-to-Text Transfer Transformer (T5 model), etc., and this embodiment is not limited thereto.

[0072] S103. Generate a binary mask based on the foreground position information;

[0073] In this optional embodiment, the binary mask is a two-dimensional matrix with the same size as the first target image, where each element corresponds to a pixel, and each element can only take the values ​​0 or 1. The binary mask is used to represent the position and range of the subject object in the first target image. Specifically, a value of 1 in the binary mask indicates that the corresponding pixel belongs to the subject object, and a value of 0 indicates that the corresponding pixel belongs to the background.

[0074] S104. Create a transparent image based on the target image size, and encode the transparent image into the latent space using a transparent image encoder to obtain the latent features of the transparent image; wherein, the first target image, the binary mask, and the transparent image have the same size;

[0075] In this optional embodiment, a transparent image refers to an image where all transparent channel values ​​are 0. A transparent image encoder is a neural network model that converts a transparent image into a low-dimensional latent representation. The dimensions of the first target image, the binary mask, and the transparent image are all the same as the target image size. In this embodiment of the invention, the transparent image encoder type is a variational autoencoder (VAE).

[0076] In this embodiment, the latent space is a continuous low-dimensional vector space, and the space before the transparent image is encoded is the image space. For example, the size of the transparent image in the image space is... The latent features of a transparent image in the latent space are Where 4 represents the number of image channels, and the resolution of the latent space is... The resolution of the image space is The resolution of the latent space is 1 / 8 of that of the image space.

[0077] S105. In the latent space, randomly sample Gaussian noise as the initial noise latent feature;

[0078] In this optional embodiment, random sampling refers to the process of randomly drawing samples from a probability distribution. Random sampling can be achieved by generating pseudo-random numbers using methods such as inverse transformation sampling, rejection sampling, and importance sampling.

[0079] In this invention, sampling Gaussian noise in the latent space means that the size of the sampled Gaussian noise is the same as the resolution of the latent space. For example, when the resolution of the latent space is... The size of the sampled Gaussian noise is then... 4 represents the number of image channels.

[0080] S106. Iterate through a first preset number of time steps. At each time step, modify the noise latent features output from the previous time step based on the latent features of the transparent image and the binary mask to obtain the corrected noise latent features. Set a cross-attention matrix based on the binary mask. By adjusting the cross-attention correlation between the text prompt embedding and the corrected noise latent features, perform denoising on the corrected noise latent features to obtain the noise latent features output at the current time step. In the first time step, the initial noise latent features are used as the noise latent features output from the previous time step.

[0081] In this embodiment of the invention, the time step refers to the time interval, which is sorted according to the execution time.

[0082] In this embodiment of the invention, denoising is performed by a sampler, specifically a Denoising Diffusion Implicit Models Sampler (DDIM).

[0083] In an optional embodiment, the step of modifying the noise latent features output from the previous time step based on the latent features of the transparent image and the binary mask to obtain the corrected noise latent features includes:

[0084] The transparent image latent features are added with first image noise to obtain transparent noisy background latent features. Based on the transparent noisy background latent features and the binary mask, the noise latent features output in the previous time step are modified to obtain the corrected noise latent features. The first image noise is Gaussian noise, and the intensity of the first image noise added in each time step is less than the intensity of the first image noise added in the previous time step.

[0085] As can be seen, this optional embodiment achieves a progressive denoising and optimization process by using first image noise of different intensities at each time step and gradually reducing the noise intensity as time steps progress. In early time steps, denoising the latent noise features with strong Gaussian noise is mainly responsible for creating structural features such as the outline of the main object; while in later time steps, denoising the latent noise features with weak Gaussian noise is mainly responsible for refining the details of the main object, enabling local fine-tuning and optimization, resulting in a more refined and accurate final transparent layer.

[0086] In another optional embodiment, the step of modifying the noise latent features output from the previous time step to obtain the corrected noise latent features, based on the latent features of the transparent noisy background and the binary mask, includes:

[0087] Determine whether the number of iterations at the current time step exceeds a preset second number; wherein the second number is less than the first number;

[0088] When the time limit is not exceeded, based on the binary mask, the background region features in the noise latent features output in the previous time step are determined, and the background region features are replaced with transparent noise-added background latent features to obtain the corrected noise latent features.

[0089] When the time limit is exceeded, the noise latent feature output from the previous time step is determined as the corrected noise latent feature.

[0090] In an optional embodiment, the first quantity can be set to 800 and the second quantity can be set to 250.

[0091] As can be seen, this optional embodiment can effectively control the modification process of latent noise features. By replacing background region features in early time steps, it can be ensured that the background region determined by the binary mask can generate a transparent background in early time steps, thereby improving the quality of the generated image. In later time steps, not modifying background region features before denoising latent noise features helps maintain the stability of the generated image and reduces visual inconsistencies caused by unnecessary subtle changes.

[0092] In another optional embodiment, the steps of setting a cross-attention matrix based on a binary mask, adjusting the cross-attention correlation between the text prompt embedding and the modified noise latent features, and performing denoising on the modified noise latent features to obtain the noise latent features output at the current time step include:

[0093] Embed text prompts and correct latent noise features into the U-shaped network;

[0094] The text prompts are embedded as keys and values ​​in the cross-attention layer of the U-shaped network, and the corrected noise latent features are used as queries in the cross-attention layer of the U-shaped network. The cross-attention matrix is ​​set according to the binary mask, and the corrected noise latent features are denoised to obtain the noise latent features output at the current time step.

[0095] In this optional embodiment, the U-shaped network refers to the U-Net network.

[0096] As can be seen, this optional embodiment effectively improves the accuracy of the generated image by introducing a U-shaped network and adjusting the cross-attention correlation between text prompt embedding and the correction of latent noise features, maintaining the balance between global layout and local details, and enhancing the semantic consistency and visual quality of the generated image. This significantly improves the performance and application value of the text-guided zero-sample transparent layer generation method.

[0097] In yet another optional embodiment, the step of setting the cross-attention matrix based on the binary mask includes:

[0098] Based on the binary mask, the pixel region corresponding to the latent features of the corrected noise is divided into the foreground region and the background region;

[0099] Set the cross-attention matrix corresponding to the foreground region for:

[0100] ;

[0101] Set the cross-attention matrix corresponding to the background region for:

[0102] ;

[0103] in, This represents the original cross-attention matrix corresponding to the foreground region. This represents the original cross-attention matrix corresponding to the background region. This represents the correction factor corresponding to the foreground region. This represents the correction factor corresponding to the background area. Indicates the first A text prompt is embedded. Indicates the number of text prompts embedded. This indicates the embedding of the text prompt corresponding to the preset text start marker. This indicates the embedding of a text prompt corresponding to a preset text end marker.

[0104] In this optional embodiment, based on a binary mask, the pixel region corresponding to the potential features of the corrected noise is divided into a foreground region and a background region, that is, the pixel region with a binary mask value of 1 is divided into a foreground region and the pixel region with a binary mask value of 0 is divided into a background region.

[0105] The text start marker is a preset marker, and the text hint embedding corresponding to this marker is located at the beginning of all text hint embeddings. In this embodiment of the invention, [SoT] is used as the text start marker. The text end marker is a preset marker, and the text hint embedding corresponding to this marker is located at the end of all text hint embeddings. In this embodiment of the invention, [EoT] is used as the text end marker.

[0106] The original cross-attention matrix is ​​the initial attention distribution state of the U-shaped network before any modifications or adjustments are made.

[0107] As can be seen, this optional embodiment divides the pixel region corresponding to the latent noise feature into foreground and background regions based on a binary mask, which can more accurately enhance the correlation between the text prompt embedding and different regions of the image, thereby improving the quality of image generation. Moreover, by using a newly set cross-attention matrix to denoise the latent noise feature, the consistency between the image and the text can be effectively enhanced, important features can be strengthened, and an image with richer details that conforms to the text prompt can be generated.

[0108] S107. Input the noise latent features output from the last time step into the transparent image decoder to obtain the first target image.

[0109] In this optional embodiment, the transparent image decoder refers to a model that decodes the noise latent features in the latent space into the first target image in the image space. The transparent image decoder type in this embodiment is a variational auto-encoder (VAE).

[0110] As can be seen, this optional embodiment achieves the ability to generate high-precision transparent background images without training data by combining text guidance and zero-shot learning mechanisms with foreground position information and cross-attention adjustment technology. This ensures a high degree of consistency between the generated subject and the text description, while accurately controlling the object layout through binary masks. At the same time, the transparent image encoder and transparent image decoder effectively preserve the background transparency information, significantly improving the controllability of image synthesis and the convenience of post-editing.

[0111] Please see Figure 5 As shown, Figure 5This is a flowchart illustrating a text-guided zero-shot layered image generation method disclosed in an embodiment of the present invention. The text-guided zero-shot layered image generation method generates a second target image based on global image text prompts, the target image size, and at least one layer of text prompts. The global image text prompts describe the content of the second target image, and each layer of text prompts describes a main object in the second target image. Figure 6 An image of a second target is shown, containing three main objects: three zebras. Figure 5 As shown, the text-guided zero-sample hierarchical image generation method may include the following operations:

[0112] S501. Input the global image text prompt, the target image size, and at least one layer of text prompt into the foreground position information generation model to obtain the foreground position information corresponding to each layer of text prompt; wherein, the foreground position information generation model is a pre-trained large language model;

[0113] In an optional embodiment, before the step of generating a model by inputting global image text hints, target image size, and at least one layer of text hints into the foreground location information, the method further includes:

[0114] A preset number of foreground position information examples are generated and input into the large language model. Each foreground position information example includes a global image text prompt, a target image size, at least one example layer text prompt, and foreground position information corresponding to each example layer text prompt. The global image text prompt describes the image content in an example image, the target image size represents the size of the example image, the example image includes at least one subject object, each example layer text prompt describes a subject object, and each foreground position information describes the position information of a subject object in the example image.

[0115] In an optional embodiment, a possible example of foreground location information generation is as follows:

[0116] Example global image text hint: A realistic image of a brown tabby cat on a white staircase;

[0117] Example target image size: ;

[0118] First example layer text hint: A brown tabby cat;

[0119] The second example layer text hint: a glass bottle;

[0120] The first example foreground position information is: [88, 28, 355, 255].

[0121] The second example of foreground position information is: [10, 317, 102, 152].

[0122] Wherein, [88, 28, 355, 255] represents the top-left corner of the brown tabby cat's bounding box with an x-coordinate of 88, a y-coordinate of 28, a width of 355, and a height of 255; [10, 317, 102, 152] represents the top-left corner of the glass bottle's bounding box with an x-coordinate of 10, a y-coordinate of 317, a width of 102, and a height of 152.

[0123] As can be seen, this optional embodiment uses a preset number of foreground position information to generate examples for context learning of the large language model. This allows the large language model to activate its pre-training capabilities during the inference phase by designing prompts, dynamically establishing a mapping relationship between text prompts and foreground position information. This eliminates the need for manually specifying the position information of each subject object, greatly improving the efficiency of foreground position information generation and reducing manual intervention and labor costs. Simultaneously, the pre-trained large language model has good generalization ability and can automatically generate reasonable foreground position information based on different text prompts, enhancing the practicality of this invention.

[0124] S502. Based on the target image size, for each layer of text prompts and its corresponding foreground position information, use the zero-sample transparent layer generation method for text guidance described in any of the preceding embodiments to generate the first target image corresponding to the text prompts of that layer.

[0125] In this optional embodiment, the target image size, the text prompts for each layer and their corresponding foreground position information are used as inputs to the text-guided zero-sample transparent layer generation method to obtain the first target image corresponding to the text prompts for that layer, wherein the size of the first target image is the target image size.

[0126] S503. Generate a soft segmentation mask based on the transparency channels in all the first target images;

[0127] In an optional embodiment, the step of generating a soft segmentation mask based on the alpha channels in all the first target images includes:

[0128] Generate a soft segmentation mask using the following formula. :

[0129] ;

[0130] in, For the first The transparency channel of the first target image. The number of images representing the first target.

[0131] As can be seen, this optional embodiment explicitly defines the blending process using a linear weighting formula, precisely controlling the contribution weights of the foreground and background through a soft segmentation mask, ensuring that the blending process is interpretable and computationally efficient. The soft segmentation mask in the linear weighting formula is directly associated with the alpha channel information, enabling position-sensitive blending with pixel-level precision, avoiding the jagged edges caused by traditional hard masks, and supporting natural transitions between multiple levels of transparency, thus improving the quality of the generated image.

[0132] S504. Overlay all the first target images to obtain a foreground overlay image;

[0133] In this optional embodiment, all first target images are superimposed according to pixel positions to obtain a foreground superimposed image, the size of which is the size of the target image.

[0134] S505. Encode the foreground overlay image into the latent space using an image encoder to obtain the foreground overlay latent features;

[0135] In this optional embodiment, the image encoder refers to a neural network model that converts an image into a low-dimensional latent representation. It can extract high-level semantic features of the image and map them to a latent space. The image encoder type in this embodiment is a variational autoencoder (VAE). The latent space described in this embodiment has the same resolution as the latent space in any of the aforementioned text-guided zero-sample transparency layer generation methods.

[0136] S506. In the latent space, randomly sample Gaussian noise as the initial background latent feature;

[0137] S507. Based on the soft segmentation mask, the latent features of the foreground superimposed with the initial background latent features are mixed to obtain the latent features of the global image;

[0138] In another optional embodiment, the step of obtaining global image latent features by mixing foreground overlay latent features with initial background latent features according to a soft segmentation mask includes:

[0139] The system iterates through a third preset number of time steps. At each time step, it generates a foreground noisy latent feature based on the foreground superimposed latent feature and Gaussian noise. Based on the soft segmentation mask, it mixes the foreground noisy latent feature with the mixed image latent feature output from the previous time step and outputs the mixed image latent feature of the current time step. In the first time step, the initial background latent feature is used as the mixed image latent feature output from the previous time step.

[0140] The latent features of the mixed image output at the last time step are determined as the latent features of the global image.

[0141] As can be seen, this optional embodiment, through a progressive blending strategy of iterative time steps, gradually merges the foreground and background in the latent space, effectively alleviating the problems of blurred subject boundaries or noise residue caused by direct superposition, making the generation process more compatible with the denoising characteristics of the diffusion model, improving the physical rationality and visual coherence of the generated image, preserving the integrity of the foreground structure and details, and further improving the quality of image generation.

[0142] In yet another optional embodiment, the step of mixing the foreground noisy latent features with the mixed image latent features output from the previous time step, based on the soft segmentation mask, includes:

[0143] No. Each time step is mixed according to the following formula:

[0144] ;

[0145] in, Representing the Latent features of the mixed image output at each time step, Represents a soft segmentation mask. Representing the Noisy latent features at each time step.

[0146] As can be seen, this optional embodiment directly accumulates the transparency channels to generate a soft segmentation mask, transforming multi-layer transparency information into a continuous spatial weight distribution. This not only preserves the characteristics of each independent subject object but also quantifies the cumulative influence of overlapping areas through superposition. On the other hand, this embodiment does not require additional training of a segmentation model and utilizes the inherent properties of the transparency channels to generate the mask, thus improving the efficiency of image generation.

[0147] In yet another optional embodiment, at each time step, the step of generating foreground-noising latent features based on foreground superimposed latent features and Gaussian noise includes:

[0148] At each time step, the foreground superimposed latent features are added with second image noise to obtain foreground noisy latent features; wherein, the second image noise is Gaussian noise, and the intensity of the second image noise added at each time step is less than the intensity of the second image noise added at the previous time step.

[0149] As can be seen, this optional embodiment simulates the standard denoising process of a diffusion model during the foreground noise addition stage by designing noise intensity that decreases with time steps. Early strong noise preserves structural degrees of freedom, while later weak noise focuses on detail optimization, resulting in foreground features with high realism and fidelity.

[0150] S508. Input the latent features of the global image into the image decoder to obtain the second target image.

[0151] In this optional embodiment, the image decoder refers to a neural network model that transforms low-dimensional latent representations into high-dimensional image data. In this embodiment, the image decoder type is a variational auto-encoder (VAE).

[0152] In summary, the present invention discloses a text-guided zero-sample transparent layer and layered image generation method. It employs a layered image generation paradigm, first generating an RGBA image with transparency information as a single instance of the main object to enhance fine-grained control over the attributes of the main object. To generate a complex layered image corresponding to the text prompt, the generated main object layers are combined and repaired to obtain a complete image containing all the main objects. Simultaneously, to achieve editability of the layout of the main objects in the image, the present invention uses layout-aware cross-attention technology in the RGBA image generation stage. This introduces layout conditions without requiring training, guiding the RGBA main objects to be generated according to the layout information, achieving automated understanding of complex layouts during image generation and enhancing the controllability of image generation. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0153] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A text-guided method for generating zero-sample transparent layers, characterized in that, The method is used to generate a first target image with the same size as the target image based on the input layer text prompt, the target image size, and foreground position information. The layer text prompt describes the content of the first target image, which includes a main object and a transparent background. The foreground position information indicates the position of the main object in the first target image. The method includes: Receiving layer text prompts, target image size, and foreground position information; Input the layered text hints into the text encoder to obtain the embedded text hints; Generate a binary mask based on the foreground position information; A transparent image is created based on the target image size, and the transparent image is encoded into the latent space using a transparent image encoder to obtain the latent features of the transparent image; wherein, the first target image, the binary mask, and the transparent image have the same size; In the latent space, Gaussian noise is randomly sampled as the initial noise latent feature; The system iterates through a first set number of time steps. At each time step, based on the latent features of the transparent image and the binary mask, it modifies the noise latent features output from the previous time step to obtain the corrected noise latent features. It sets a cross-attention matrix based on the binary mask and performs denoising on the corrected noise latent features by adjusting the cross-attention correlation between the text prompt embedding and the corrected noise latent features to obtain the noise latent features output from the current time step. In the first time step, the initial noise latent features are used as the noise latent features output from the previous time step. The noise latent features output from the last time step are input into the transparent image decoder to obtain the first target image; The step of modifying the noise latent features output from the previous time step based on the latent features of the transparent image and the binary mask to obtain the corrected noise latent features includes: The transparent image latent features are added with first image noise to obtain transparent noisy background latent features. Based on the transparent noisy background latent features and the binary mask, the noise latent features output in the previous time step are modified to obtain the corrected noise latent features. The first image noise is Gaussian noise, and the intensity of the first image noise added in each time step is less than the intensity of the first image noise added in the previous time step.

2. The text-guided zero-sample transparent layer generation method according to claim 1, characterized in that, Based on the latent features of the transparent noisy background and the binary mask, the latent noise features output from the previous time step are modified to obtain... The steps to correct latent noise characteristics include: Determine whether the number of iterations at the current time step exceeds a preset second number; wherein the second number is less than the first number; When the time limit is not exceeded, based on the binary mask, the background region features in the noise latent features output in the previous time step are determined, and the background region features are replaced with transparent noise-added background latent features to obtain the corrected noise latent features. When the time limit is exceeded, the noise latent feature output from the previous time step is determined as the corrected noise latent feature.

3. The text-guided zero-sample transparent layer generation method according to claim 1, characterized in that, The steps for setting a cross-attention matrix based on a binary mask, adjusting the cross-attention correlation between the text prompt embedding and the corrected noise latent features, and performing denoising on the corrected noise latent features to obtain the noise latent features output at the current time step include: Embed text prompts and correct latent noise features into the U-shaped network; The text prompts are embedded as keys and values ​​in the cross-attention layer of the U-shaped network, and the corrected noise latent features are used as queries in the cross-attention layer of the U-shaped network. The cross-attention matrix is ​​set according to the binary mask, and the corrected noise latent features are denoised to obtain the noise latent features output at the current time step.

4. The text-guided zero-sample transparent layer generation method according to claim 3, characterized in that, The steps for setting the cross-attention matrix based on the binary mask include: Based on the binary mask, the pixel region corresponding to the latent features of the corrected noise is divided into the foreground region and the background region; Set the cross-attention matrix corresponding to the foreground region for: ； Set the cross-attention matrix corresponding to the background region for: ； in, This represents the original cross-attention matrix corresponding to the foreground region. This represents the original cross-attention matrix corresponding to the background region. This represents the correction factor corresponding to the foreground region. This represents the correction factor corresponding to the background area. Indicates the first A text prompt is embedded. Indicates the number of text prompts embedded. This indicates the embedding of the text prompt corresponding to the preset text start marker. This indicates the embedding of a text prompt corresponding to a preset text end marker.

5. A text-guided zero-sample hierarchical image generation method, characterized in that, The method is used to generate a second target image based on a global image text prompt, a target image size, and at least one layer of text prompts. The global image text prompts describe the content of the second target image, and each layer of text prompts describes a main object in the second target image. The method includes: The global image text prompt, the target image size, and at least one layer of text prompt are input into the foreground position information generation model to obtain the foreground position information corresponding to each layer of text prompt; wherein, the foreground position information generation model is a pre-trained large language model; Based on the target image size, for each layer of text prompts and its corresponding foreground position information, the zero-sample transparent layer generation method for text guidance as described in any one of claims 1-4 is used to generate the first target image corresponding to the text prompts of that layer; Generate a soft segmentation mask based on the alpha channels in all the first target images; All the first target images are superimposed to obtain a foreground superimposed image; The foreground overlay image is encoded into the latent space by an image encoder to obtain the foreground overlay latent features; In the latent space, Gaussian noise is randomly sampled as the initial background latent feature; Based on the soft segmentation mask, the latent features of the foreground are mixed with the latent features of the initial background to obtain the latent features of the global image; The latent features of the global image are input into the image decoder to obtain the second target image.

6. The text-guided zero-sample hierarchical image generation method according to claim 5, characterized in that, The steps for obtaining global image latent features by mixing foreground overlay latent features with initial background latent features based on soft segmentation masks include: The system iterates through a third number of time steps. At each time step, it generates a noisy foreground latent feature based on the foreground superimposed latent feature and Gaussian noise. Based on the soft segmentation mask, it mixes the noisy foreground latent feature with the mixed image latent feature output from the previous time step and outputs the mixed image latent feature of the current time step. In the first time step, the initial background latent feature is used as the mixed image latent feature output from the previous time step. The latent features of the mixed image output at the last time step are determined as the latent features of the global image.

7. The text-guided zero-sample hierarchical image generation method according to claim 6, characterized in that, The steps for mixing the foreground noisy latent features with the mixed image latent features output from the previous time step, based on the soft segmentation mask, include: No. Each time step is mixed according to the following formula: ； in, Representing the Latent features of the mixed image output at each time step, Represents a soft segmentation mask. Representing the Noisy latent features at each time step.

8. The text-guided zero-sample hierarchical image generation method according to claim 6, characterized in that, At each time step, the steps for generating noisy foreground latent features based on foreground superposition latent features and Gaussian noise include: At each time step, the foreground superimposed latent features are added with second image noise to obtain foreground noisy latent features; wherein, the second image noise is Gaussian noise, and the intensity of the second image noise added at each time step is less than the intensity of the second image noise added at the previous time step.

9. The text-guided zero-sample hierarchical image generation method according to claim 5, characterized in that, The steps for generating a soft segmentation mask based on the alpha channels in all the first target images include: Generate a soft segmentation mask using the following formula. : ； in, For the first The transparency channel of the first target image. The number of images representing the first target.

Images