An image generation method and system based on entity region guided diffusion

Abstract

The application discloses an image generation method based on entity region guided diffusion, which introduces an entity region guided diffusion (ERGD for short) framework, combines a depth-aware regional attention (DARA for short) mechanism, a Gaussian soft mask, an overlap constraint and a foreground-background semantic separation strategy, and realizes efficient and fine entity-level controllable image generation; while maintaining global consistency, the method can accurately control the shape, position and interaction relationship of multiple entities, and solves the technical problems of boundary artifacts, semantic entanglement and high computational cost existing in the prior art. The application is particularly suitable for automatic driving, virtual reality, image editing and other scenes requiring multiple entity scene generation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and generative artificial intelligence technology, and more specifically, relates to an image generation method and system based on entity region guided diffusion. Background Technology

[0002] In the rapid development of computer vision and generative artificial intelligence technologies, text-to-image (T2I) generation technology has become a core research direction and key application scenario in the field. Its core objective is to automatically generate image content with visual fidelity, spatial consistency, and entity integrity based on semantic information described by natural language. This technology is widely used in various fields such as autonomous driving scene simulation, virtual reality content production, intelligent image editing, and industrial design rendering. Achieving precise entity-level controllable generation—that is, accurately controlling the shape, position, and attribute features of multiple entities while ensuring reasonable interaction between entities and semantic coherence between foreground and background—is the core challenge in its transition from technological research and development to large-scale implementation. With breakthroughs in diffusion models and multimodal transformer technologies, text-to-image generation has achieved significant improvements in visual quality. However, there are still problems to be solved in areas such as the naturalness of entity boundaries, spatial hierarchy modeling, semantic decoupling effects, and training efficiency.

[0003] At present, among text-to-image generation methods based on diffusion models, training-insensitive control generation methods are one of the mainstream implementation approaches, represented by MultiDiffusion, BoxDiff, MCCD, etc. These methods do not require additional fine-tuning of the pre-trained diffusion model. They apply local region constraints to the generation process through spatial mask construction, multi-path diffusion fusion, bounding box-guided attention, etc., thereby achieving basic control over the position and contour of entities.

[0004] However, existing training-insensitive controlled generation methods still have some significant drawbacks: First, this type of method often uses hard masks to constrain entity regions. The mask boundaries lack smooth transition processing, which easily produces visual artifacts at the edges of entities. At the same time, it leads to poor texture continuity of entity contours and insufficient global scene coherence, which seriously affects the visual naturalness of the generated image. Second, this type of method can only achieve basic regional position constraints, lacks explicit modeling of spatial hierarchy and occlusion relationship between entities, cannot depict deep dependency relationship between entities, and the generated multi-entity interaction scene does not conform to real spatial logic, making it difficult to meet the entity-level control requirements in complex scenarios. Third, this type of method does not have a dedicated foreground-background semantic separation mechanism, which leads to mutual interference between foreground entity attribute features and background features, resulting in semantic entanglement and reducing entity fidelity and global semantic consistency. Fourth, this type of method has limited control precision, and can only achieve rough area constraints. It cannot accurately capture the detailed attributes and interaction relationships of entities, and its generation effect is not good for scenes with multiple entities and complex layouts, making it difficult to meet the application requirements of high-precision entity-level control. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an image generation method and system based on entity region guided diffusion. Its purpose is to solve the following technical problems of existing training-insensitive control-type generation methods: Firstly, these methods often use hard masks for entity region constraints, resulting in a lack of smooth transition processing at mask boundaries, leading to visual artifacts at entity edges, poor texture continuity of entity contours, and insufficient global scene coherence, severely impacting the visual naturalness of generated images. Secondly, these methods can only achieve basic region position constraints, lacking explicit modeling of spatial hierarchy and occlusion relationships between entities, failing to depict deep dependencies between entities, and generating multi-entity interaction scenes that do not conform to real spatial logic, making it difficult to meet the entity-level control requirements in complex scenes. Thirdly, the lack of a dedicated foreground-background semantic separation mechanism leads to interference between foreground entity attribute features and background features, resulting in semantic entanglement and reduced entity fidelity and global semantic consistency. Fourthly, limited control precision allows for only coarse region constraints, failing to accurately capture detailed entity attributes and interaction relationships, resulting in poor generation performance for multi-entity, complex layout scenes, and difficulty meeting the application requirements for high-precision entity-level control.

[0006] To achieve the above objectives, according to one aspect of the present invention, an image generation method based on entity region-guided diffusion is provided, comprising the following steps: (1) Obtain input data from the user interaction terminal or sensor acquisition system, including global text prompts, L entity text prompts and corresponding L entity space masks, and perform standardization processing on the input data to obtain standardized data; (2) The standardized data obtained in step (1) are subjected to text encoding, soft masking, background mask generation and smoothing, and image latent feature projection processing in sequence to obtain model input features including text embedding vector, image embedding, Gaussian soft mask and smoothed background mask. (3) Obtain the region attention mask based on the model input features obtained in step (2), and input the obtained region attention mask into the pre-trained ERGD model for feature fusion to obtain the target features.

[0007] Preferably, the global text prompt is used to describe the overall semantics of the scene; Entity text hints are used to describe the local semantics of each entity, including attributes, shape, and color; Entity space masks are used to represent the spatial location and contour of each entity in an image, and are in binary matrix format.

[0008] Preferably, step (1) standardizes the global text prompt by formatting it according to the input specifications of the pre-trained T5-base text encoder. This involves first removing newline characters, special symbols, and meaningless redundant characters from the global text prompt, then unifying the encoding format of the global text prompt to UTF-8, and finally adjusting the length of the global text prompt to a fixed length. For parts exceeding the fixed length, the core information is truncated according to semantic priority, and for parts not exceeding the fixed length, the encoder's default padding character is used to fill in the gaps.

[0009] Preferably, in step (1), the standardization process for the L entity text prompts is the same as that for the global text prompts. At the same time, a unique entity identifier prefix “Entity[i]:” is added to each entity text prompt, where i represents the i-th entity text prompt and i∈[1,L].

[0010] Preferably, the standardization process for the L entity space masks in step (1) involves performing binary verification and format unification on them. That is, firstly, abnormal pixel values ​​that are not 0 / 1 are removed from each entity space mask. Then, the size of each entity space mask after removing abnormal pixel values ​​is adjusted according to the resolution of the target dataset and by using bilinear interpolation. Finally, the pixel values ​​of each entity space mask after adjustment are normalized to the [0,1] interval.

[0011] Preferably, step (2) specifically involves: First, the global text hints and L entity text hints in the standardized data obtained in step (1) are processed using the MLLMs chain inference method to obtain background hints; Then, the pre-trained T5-base text encoder is used to encode the background prompt, global text prompt and L entity text prompts respectively, so as to obtain the text embedding vector corresponding to the background prompt, the text embedding vector corresponding to the global text prompt and the text embedding vector corresponding to each entity text prompt respectively. Then, all text embedding vectors are normalized into a set of text embedding vectors of the same dimension. Subsequently, the L entity space masks in the standardized data obtained in step (1) are subjected to soft masking to replace the L entity space masks with Gaussian soft masks. Subsequently, the complement operation is performed on the L entity space masks in the standardized data obtained in step (1) to obtain the background mask, and the background mask is smoothed by convolution using a Gaussian kernel to obtain a smooth background mask; finally, the original image contained in the standardized data in step (1) is encoded by a variational autoencoder (VAE) to obtain the image latent representation, and the latent representation is subjected to feature projection to obtain the image embedding, and finally the model input features containing text embedding vector, image embedding, Gaussian soft mask and smooth background mask are obtained.

[0012] Preferably, step (3) specifically involves: First, classify, label, and index all tokens in the text embedding vector set obtained in step (2) to obtain a text token index sequence arranged in the order of global text hints - background hints - entity text hints. Then, perform latent token space index mapping on the image embedding in the model input features to obtain the image latent token index sequence. Perform block operation on all Gaussian soft masks in the model input features to obtain the attention mask column vector set corresponding to the background hints. Finally, use the sliding window block algorithm to perform block operation on the smooth background mask in the model input features to obtain the attention mask column vector corresponding to the background hints. Subsequently, based on the obtained text token index sequence and image latent token index sequence, the attention mask column vector set corresponding to the entity text prompt and the attention mask column vector corresponding to the background prompt are matrix concatenated to obtain a region attention mask matrix with the dimension of "the total number of tokens in each text embedding vector × the total number of tokens in each image embedding", which is consistent with the dimension of the text embedding vector set and image embedding obtained in step (2). Then, according to the text token index sequence, each token in the text embedding vector set of the model input features obtained in step (2) is bound one by one to the row index corresponding to the region attention mask matrix, and each latent token in the image embedding is bound one by one to the column index corresponding to the region attention mask matrix. Abnormal matching items in the text token index sequence and latent tokens are removed to obtain the integrated multimodal input features. Subsequently, the integrated multimodal input features are adjusted according to the input specifications of the ERGD framework, and the adjusted multimodal input features are input into the pre-trained MM-DiT backbone network of the ERGD framework to obtain preliminary multimodal fusion features. Subsequently, the acquired multimodal fusion preliminary features are input into the DARA module in the ERGD framework to obtain attention-enhanced features for entity spatial hierarchical perception. After that, the depth estimation submodule in the ERGD framework performs multilayer perceptron (MLP) projection on the image embedding obtained in step (2) to obtain the depth weight of each entity in the image embedding. Subsequently, the depth weight of each entity is multiplied element-wise with the Gaussian soft mask obtained in step (2) using the regional attention submodule in the ERGD framework to generate the attention weight matrix corresponding to the entity text prompt. Based on the attention weight matrix and the regional attention mask matrix, and using the attention mechanism, refined attention fusion features are obtained. Subsequently, the attention-enhanced features of entity spatial hierarchy perception obtained from the DARA module and the refined attention fusion features are input together into the overlapping constraint module in the ERGD framework to obtain preliminary fusion features. Subsequently, the preliminary fused features are input into the ERGD framework. Preliminary optimization is performed to obtain the optimized preliminary fusion features; Then, the optimized preliminary fusion features are input into the overlap constraint module in the ERGD model to obtain the attention optimization features of the overlapping region. Subsequently, the attention-optimized features of the overlapping regions are input into the semantic region molecular module of the foreground-background semantic separation module in the ERGD framework to separate foreground entity features and background features, so as to obtain semantically separated features. Subsequently, the semantically separated features are input into the loss supervision submodule in the foreground-background semantic separation module to perform final optimization on the semantically separated features in order to obtain the target features; Finally, the target features are input into the VAE to obtain the target generated image.

[0013] According to another aspect of the present invention, an image generation system based on entity region guided diffusion is provided, comprising the following modules: The first module is used to acquire input data from the user interaction terminal or sensor acquisition system, including global text prompts, L entity text prompts and corresponding L entity space masks, and to perform standardization processing on the input data to obtain standardized data. The second module is used to perform text encoding, soft masking, background mask generation and smoothing, and image latent feature projection processing on the standardized data obtained from the first module, so as to obtain model input features including text embedding vectors, image embedding, Gaussian soft mask, and smoothed background mask. The third module is used to obtain a region attention mask based on the model input features obtained from the second module, and input the obtained region attention mask into the pre-trained ERGD model for feature fusion to obtain the target features.

[0014] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The present invention employs the mask soft processing and background mask generation and smoothing operations in step (2). By converting the entity space mask into a Gaussian soft mask and using a Gaussian function to achieve smooth decay from the entity center to the boundary, and performing Gaussian kernel convolution processing on the background mask, the hard boundary between the entity and the background is eliminated. Therefore, it can solve the technical problems of visual artifacts, poor texture continuity and insufficient global scene coherence caused by hard masks in the existing training-insensitive control generation method. 2. This invention employs the DARA module and overlap constraint module in step (3). It performs MLP projection on the image embedding through the depth estimation sub-module of the DARA module to obtain the depth weights that represent the spatial hierarchy priority of entities. Then, it detects the overlapping areas of entities and optimizes the attention distribution through the overlap constraint module to clarify the occlusion relationship between entities. Therefore, it can solve the technical problems of existing training-insensitive control generation methods, such as the lack of modeling of spatial hierarchy and occlusion relationship between entities and the fact that multi-entity interaction scenarios do not conform to the real spatial logic. 3. This invention employs the background prompt generation in step (2) and the foreground-background semantic separation module in step (3). It uses the chain reasoning method of Multimodal Large Language Models (MLLMs) to process the global text prompt and L entity text prompts to automatically generate background prompts. Then, the attention optimization features of the overlapping region are input into the semantic region molecular module of the foreground-background semantic separation module to achieve the separation of foreground entity features and background features. Subsequently, the loss supervision sub-module of this module performs final optimization on the semantically separated features to strengthen the semantic independence of the foreground and background. Therefore, it can solve the technical problems of existing training-unaware control generation methods, such as the lack of a dedicated foreground-background semantic separation mechanism and the semantic entanglement caused by mutual interference between entity and background features, which reduces entity fidelity and global semantic consistency. 4. This invention employs the standardization of entity text prompts in step (1), the construction of regional attention masks in step (3), and the refined feature fusion of the DARA module. By adding exclusive identifier prefixes to entity text prompts to clarify semantic boundaries, constructing regional attention masks to achieve precise spatial binding between entity text and image latent tokens, and then multiplying entity depth weights with Gaussian soft masks element by element through the DARA module to generate a refined attention weight matrix, it can accurately capture entity details and interaction relationships. Therefore, it can solve the technical problems of limited control precision and inability to accurately capture entity detail attributes and interaction relationships in existing training-insensitive control generation methods. 5. This invention innovatively transforms the semantic segmentation task into an image generation problem, overcoming the technical limitations of discriminative models and providing a new technical path for achieving accurate and efficient semantic segmentation. Compared with existing technologies, this invention achieves significant improvements in generation quality, model flexibility, and training efficiency, possessing important theoretical value and application prospects. Attached Figure Description

[0015] Figure 1 This is a flowchart of the image generation method based on entity region guided diffusion according to the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0017] The basic idea of ​​this invention is to provide an image generation method based on entity-region-guided diffusion (ERGD). This method introduces the ERGD framework, combined with a depth-aware regional attention (DARA) mechanism, Gaussian soft masking, overlap constraints, and foreground-background semantic separation strategies, to achieve efficient and precise entity-level controllable image generation. While maintaining global consistency, this method can precisely control the shape, position, and interaction relationships of multiple entities, solving the technical problems of boundary artifacts, semantic entanglement, and high computational costs inherent in existing methods. This invention is particularly suitable for scenarios requiring multi-entity scene generation, such as autonomous driving, virtual reality, and image editing.

[0018] The core innovations of this invention include: (1) The ERGD framework is adopted to realize entity-level control based on the Multimodal Diffusion Transformer (MM-DiT) architecture; (2) Introduce the DARA mechanism to smooth entity boundaries through Gaussian soft masking and reduce artifacts; The overall process (including online inference and offline training) and core components of this invention are shown in [the document / document / etc.]. Figure 1 The flowchart of the image generation method based on entity region guided diffusion clearly divides the process into two main stages: "offline model training" (executed only once) and "online image generation" (executed repeatedly). It clarifies the entire chain logic from training data input to target image output and marks the positions of key steps such as text encoding, latent encoding, region modeling, and masking rules.

[0019] like Figure 1 As shown, this invention provides an image generation method based on entity region-guided diffusion, comprising the following steps: (1) Obtain input data from the user interaction terminal or sensor acquisition system, including global text prompts (used to describe the overall semantics of the scene), L entity text prompts (describe the local semantics of each entity such as attributes, shape, color, etc.) and corresponding L entity spatial masks (used to represent the spatial position and contour of each entity in the image, in binary matrix format), and standardize the input data to obtain standardized data, wherein the value of L is from 1 to 20, preferably from 3 to 8; Specifically, this step standardizes the global text prompts by formatting them according to the input specifications of the pre-trained T5-base text encoder. First, newline characters, special symbols (such as @, #, ￥, etc.), and meaningless redundant characters are removed from the global text prompts to ensure semantic integrity. Then, the encoding format of the global text prompts is unified to UTF-8 to avoid encoding conflicts. Finally, the length of the global text prompts is adjusted to a fixed length (512 tokens). The part exceeding the fixed length is truncated according to semantic priority, and the part not exceeding the fixed length is supplemented with the encoder's default padding character to adapt to the encoder's input requirements. In this step, the L entity text prompts are standardized using the same standardization process as the global text prompts. At the same time, a unique entity identifier prefix is ​​added to each entity text prompt (in the format "Entity[i]:", where i represents the i-th entity text prompt and i∈[1,L]), clearly distinguishing the semantic boundaries between entities and avoiding cross-entity semantic confusion. In this step, the L entity space masks are standardized by performing binary verification and format unification. First, abnormal pixel values ​​that are not 0 or 1 are removed from each entity space mask (0 represents non-entity regions, and 1 represents entity regions, to ensure that the entity space mask is a pure binary annotation). Then, according to the resolution of the target dataset (1024×1024 for the LayoutSAM dataset and 512×512 for the COCO dataset), the size of each entity space mask after removing abnormal pixel values ​​is adjusted by bilinear interpolation (to ensure compatibility with subsequent model processing). Finally, the pixel values ​​of each entity space mask after adjustment are normalized to the [0,1] interval (to provide standardized input for subsequent Gaussian soft mask generation and avoid boundary processing errors caused by inconsistent mask formats).

[0020] (2) The standardized data obtained in step (1) are subjected to text encoding, soft masking, background mask generation and smoothing, and image latent feature projection processing in sequence to obtain model input features including text embedding vector, image embedding, Gaussian soft mask and smoothed background mask. Specifically, this step involves first using the MLLMs chain inference method to process the global text hints and L entity text hints in the standardized data obtained in step (1) to obtain background hints; then, using a pre-trained T5-base text encoder, encoding the obtained background hints, global text hints, and L entity text hints respectively to obtain the text embedding vectors corresponding to the background hints, the global text hints, and each entity text hint, and then normalizing all text embedding vectors into a unified dimension (768 dimensions) text embedding vector set; subsequently, performing soft masking on the L entity space masks in the standardized data obtained in step (1) to replace the L entity space masks with Gaussian soft masks; subsequently, step (1) The L entity space masks in the obtained standardized data are complemented to obtain the background mask (i.e., the entity regions covered by all entity space masks are excluded, and the remaining regions are the regions corresponding to the background mask). The background mask is then convolved and smoothed using a Gaussian kernel to obtain a smooth background mask. Finally, the original image contained in the standardized data in step (1) is encoded by a variational autoencoder (VAE) to obtain the image latent representation. The latent representation is then subjected to feature projection processing (i.e., it is projected into an image embedding through a latent-specific MLP network) to obtain the image embedding. Finally, the model input features containing text embedding vectors, image embedding, Gaussian soft mask, and smooth background mask are obtained.

[0021] The advantage of this step is that it can efficiently transform standardized data into model input features, while taking into account input adaptability and solving technical pain points, thus greatly improving the accuracy and efficiency of subsequent model inference.

[0022] Specifically, this step automatically generates background cues through MLLMs chained inference, eliminating the need for manual annotation, reducing annotation costs and ensuring semantic coherence. All text cues are uniformly encoded using a T5 encoder, and latent dedicated MLP network projection images are embedded, achieving unified multimodal feature dimensions and avoiding semantic entanglement and modality incompatibility issues. Gaussian soft masks and background masking smoothing completely resolve the defects of traditional hard masks, such as harsh boundaries and obvious artifacts, providing high-quality, highly adaptable input support for subsequent ERGD framework inference. Simultaneously, the overall operation requires no complex calculations, balancing practicality and efficiency.

[0023] (3) Obtain the region attention mask based on the model input features obtained in step (2), and input the obtained region attention mask into the pre-trained ERGD model for feature fusion to obtain the target features; Specifically, this step involves first classifying, labeling, and indexing all tokens in the text embedding vector set obtained in step (2) to obtain a text token index sequence arranged in the order of "global text hints - background hints - entity text hints". At the same time, the image embedding in the model input features is processed by latent token space index mapping to obtain the image latent token index sequence. Meanwhile, all Gaussian soft masks in the model input features are divided into blocks (i.e., the Gaussian soft mask is converted into a one-dimensional token sequence) to obtain the attention mask column vector set corresponding to the background hint (the mask value is consistent with the weight of the Gaussian soft mask). At the same time, the sliding window block algorithm is used to perform block operation on the smooth background mask in the model input features (the window size is consistent with the block parameters represented by the image latent, the step size is equal to the window size, and the two-dimensional smooth background mask is converted into a one-dimensional token sequence) to obtain the attention mask column vector corresponding to the background hint. Subsequently, based on the obtained text token index sequence and image latent token index sequence, the attention mask column vector set corresponding to entity text prompts (a one-dimensional token sequence generated by processing the Gaussian soft mask of each entity through the sliding window block algorithm, and optimized by matrix filling, with a total of L column vectors, each column vector corresponding to an entity text prompt) and the attention mask column vector corresponding to background prompts are matrix concatenated to obtain a region attention mask matrix with a dimension of "the total number of tokens in each text embedding vector × the total number of tokens in each image embedding". This matrix is ​​consistent with the dimensions of the text embedding vector set and the image embedding obtained in step (2), ensuring that the attention mask column vector corresponding to entity text prompts and the attention mask column vector corresponding to background prompts are accurately bound to the entity text prompt tokens and background prompt tokens in the text token index sequence, respectively, and are matched one-to-one with the corresponding spatial positions in the image latent token index sequence.

[0024] Then, according to the text token index sequence, each token in the text embedding vector set of the model input features obtained in step (2) is bound one by one to the row index corresponding to the region attention mask matrix, and each latent token in the image embedding is bound one by one to the column index corresponding to the region attention mask matrix. Abnormal matching items in the text token index sequence and latent tokens are removed to obtain the integrated multimodal input features.

[0025] Subsequently, the integrated multimodal input features are adjusted according to the input specifications of the ERGD framework, and the adjusted multimodal input features are input into the pre-trained MM-DiT backbone network of the ERGD framework to obtain preliminary multimodal fusion features. Then, the obtained preliminary multimodal fusion features are input into the DARA module in the ERGD framework to obtain attention enhancement features for entity spatial hierarchy perception. After that, the depth estimation submodule in the ERGD framework projects the image embedding obtained in step (2) using a multilayer perceptron (MLP) to obtain the depth weight of each entity in the image embedding (which is used to characterize the spatial hierarchy priority of each entity in the image latent representation). Subsequently, the depth weight of each entity is multiplied element-wise with the Gaussian soft mask obtained in step (2) using the regional attention submodule in the ERGD framework to generate the attention weight matrix corresponding to the entity text prompt. Based on the attention weight matrix and the regional attention mask matrix, and using the attention mechanism, refined attention fusion features are obtained. Subsequently, the attention-enhanced features of entity spatial hierarchy perception obtained from the DARA module, together with the refined attention fusion features, are input into the overlap constraint module of the ERGD framework to obtain preliminary fusion features. (The advantage of applying region attention masks is that they explicitly limit the visibility range between tokens of different modalities, ensuring accurate binding between entities and corresponding spatial regions, effectively blocking semantic leakage across entities, and maintaining the connectivity of global latent features, thus fundamentally solving the semantic entanglement problem in multi-entity scenarios.) Then, the preliminary fusion features are input into the ERGD framework... Preliminary optimization is performed to obtain the optimized preliminary fusion features; Then, the optimized preliminary fusion features are input into the overlap constraint module in the ERGD model to obtain the attention optimization features of the overlapping region. Subsequently, the attention-optimized features of the overlapping regions are input into the semantic region molecular module of the foreground-background semantic separation module in the ERGD framework to separate foreground entity features and background features, so as to obtain semantically separated features. Subsequently, the semantically separated features are input into the loss supervision submodule in the foreground-background semantic separation module to perform final optimization on the semantically separated features in order to obtain the target features; Finally, the target features are input into the VAE to obtain the target generated image; like Figure 1 As shown, the above online process strictly follows the patented step-by-step logic. Through the progressive processing of "formatted input data → model input features → preliminary feature fusion → target features → target generated image", it achieves a balance between precise entity-level control and global scene consistency. Each step is designed around the core objectives of "reducing artifacts, decoupling semantics, and improving efficiency", and all steps can be repeated. It is suitable for multi-entity image generation needs in different scenarios such as autonomous driving, virtual reality, and image editing.

[0026] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image generation method based on entity region guided diffusion, characterized in that, Includes the following steps: (1) Obtain input data from the user interaction terminal or sensor acquisition system, including global text prompts, L entity text prompts and corresponding L entity space masks, and perform standardization processing on the input data to obtain standardized data; (2) The standardized data obtained in step (1) are subjected to text encoding, soft masking, background mask generation and smoothing, and image latent feature projection processing in sequence to obtain model input features including text embedding vector, image embedding, Gaussian soft mask and smoothed background mask. (3) Obtain the region attention mask based on the model input features obtained in step (2), and input the obtained region attention mask into the pre-trained ERGD model for feature fusion to obtain the target features.

2. The image generation method based on entity region guided diffusion according to claim 1, characterized in that, Global text hints are used to describe the overall semantics of a scene; Entity text hints are used to describe the local semantics of each entity, including attributes, shape, and color; Entity space masks are used to represent the spatial location and contour of each entity in an image, and are in binary matrix format.

3. The image generation method based on entity region guided diffusion according to claim 1 or 2, characterized in that, Step (1) standardizes the global text prompts by formatting them according to the input specifications of the pre-trained T5-base text encoder. This involves first removing newline characters, special symbols, and meaningless redundant characters from the global text prompts, then unifying the encoding format of the global text prompts to UTF-8, and finally adjusting the length of the global text prompts to a fixed length. For parts exceeding the fixed length, the core information is truncated according to semantic priority, while parts not exceeding the fixed length are supplemented with the encoder's default padding character.

4. The image generation method based on entity region guided diffusion according to any one of claims 1 to 3, characterized in that, In step (1), the L entity text prompts are standardized using the same standardization process as the global text prompts. At the same time, each entity text prompt is given a unique entity identifier prefix "Entity[i]:", where i represents the i-th entity text prompt and i∈[1,L].

5. The image generation method based on entity region guided diffusion according to claim 4, characterized in that, In step (1), the L entity space masks are standardized by performing binary verification and format unification. First, abnormal pixel values ​​that are not 0 or 1 are removed from each entity space mask. Then, the size of each entity space mask after removing abnormal pixel values ​​is adjusted according to the resolution of the target dataset and by using bilinear interpolation. Finally, the pixel values ​​of each entity space mask after adjustment are normalized to the [0,1] interval.

6. The image generation method based on entity region guided diffusion according to claim 5, characterized in that, Step (2) specifically involves: First, the global text hints and L entity text hints in the standardized data obtained in step (1) are processed using the MLLMs chain inference method to obtain background hints; Then, the pre-trained T5-base text encoder is used to encode the background prompt, global text prompt and L entity text prompts respectively, so as to obtain the text embedding vector corresponding to the background prompt, the text embedding vector corresponding to the global text prompt and the text embedding vector corresponding to each entity text prompt respectively. Then, all text embedding vectors are normalized into a set of text embedding vectors of the same dimension. Subsequently, the L entity space masks in the standardized data obtained in step (1) are subjected to soft masking to replace the L entity space masks with Gaussian soft masks. Subsequently, the complement operation is performed on the L entity space masks in the standardized data obtained in step (1) to obtain the background mask, and the background mask is smoothed by convolution using a Gaussian kernel to obtain a smooth background mask; finally, the original image contained in the standardized data in step (1) is encoded by a variational autoencoder (VAE) to obtain the image latent representation, and the latent representation is subjected to feature projection to obtain the image embedding, and finally the model input features containing text embedding vector, image embedding, Gaussian soft mask and smooth background mask are obtained.

7. The image generation method based on entity region guided diffusion according to claim 6, characterized in that, Step (3) specifically involves: First, classify, label, and index all tokens in the text embedding vector set obtained in step (2) to obtain a text token index sequence arranged in the order of global text hints - background hints - entity text hints. Then, perform latent token space index mapping on the image embedding in the model input features to obtain the image latent token index sequence. Perform block operation on all Gaussian soft masks in the model input features to obtain the attention mask column vector set corresponding to the background hints. Finally, use the sliding window block algorithm to perform block operation on the smooth background mask in the model input features to obtain the attention mask column vector corresponding to the background hints. Subsequently, based on the obtained text token index sequence and image latent token index sequence, the attention mask column vector set corresponding to the entity text prompt and the attention mask column vector corresponding to the background prompt are matrix concatenated to obtain a region attention mask matrix with the dimension of "the total number of tokens in each text embedding vector × the total number of tokens in each image embedding", which is consistent with the dimension of the text embedding vector set and image embedding obtained in step (2). Then, according to the text token index sequence, each token in the text embedding vector set of the model input features obtained in step (2) is bound one by one to the row index corresponding to the region attention mask matrix, and each latent token in the image embedding is bound one by one to the column index corresponding to the region attention mask matrix. Abnormal matching items in the text token index sequence and latent tokens are removed to obtain the integrated multimodal input features. Subsequently, the integrated multimodal input features are adjusted according to the input specifications of the ERGD framework, and the adjusted multimodal input features are input into the pre-trained MM-DiT backbone network of the ERGD framework to obtain preliminary multimodal fusion features. Subsequently, the acquired multimodal fusion preliminary features are input into the DARA module in the ERGD framework to obtain attention-enhanced features for entity spatial hierarchical perception. After that, the depth estimation submodule in the ERGD framework performs multilayer perceptron (MLP) projection on the image embedding obtained in step (2) to obtain the depth weight of each entity in the image embedding. Subsequently, the depth weight of each entity is multiplied element-wise with the Gaussian soft mask obtained in step (2) using the regional attention submodule in the ERGD framework to generate the attention weight matrix corresponding to the entity text prompt. Based on the attention weight matrix and the regional attention mask matrix, and using the attention mechanism, refined attention fusion features are obtained. Subsequently, the attention-enhanced features of entity spatial hierarchy perception obtained from the DARA module and the refined attention fusion features are input together into the overlapping constraint module in the ERGD framework to obtain preliminary fusion features. Subsequently, the preliminary fused features are input into the ERGD framework. Preliminary optimization is performed to obtain the optimized preliminary fusion features; Then, the optimized preliminary fusion features are input into the overlap constraint module in the ERGD model to obtain the attention optimization features of the overlapping region. Subsequently, the attention-optimized features of the overlapping regions are input into the semantic region molecular module of the foreground-background semantic separation module in the ERGD framework to separate foreground entity features and background features, so as to obtain semantically separated features. Subsequently, the semantically separated features are input into the loss supervision submodule in the foreground-background semantic separation module to perform final optimization on the semantically separated features in order to obtain the target features; Finally, the target features are input into the VAE to obtain the target generated image.

8. An image generation system based on entity region-guided diffusion, characterized in that, Includes the following modules: The first module is used to acquire input data from the user interaction terminal or sensor acquisition system, including global text prompts, L entity text prompts and corresponding L entity space masks, and to perform standardization processing on the input data to obtain standardized data. The second module is used to perform text encoding, soft masking, background mask generation and smoothing, and image latent feature projection processing on the standardized data obtained from the first module, so as to obtain model input features including text embedding vectors, image embedding, Gaussian soft mask, and smoothed background mask. The third module is used to obtain a region attention mask based on the model input features obtained from the second module, and input the obtained region attention mask into the pre-trained ERGD model for feature fusion to obtain the target features.

Images