Text generation image space guiding method and system based on vocabulary mapping graph

Abstract

The invention discloses a text generation image space guiding method based on a vocabulary mapping graph. The method comprises the following steps: converting a serialized text prompt input by a user into a structured two-dimensional vocabulary map (Lexical Map) by utilizing a large language model, wherein the map explicitly encodes semantic and spatial position information of an object; in the inference process of the text map diffusion model, weight scores of self-attention and cross attention are adaptively adjusted according to the layout of the vocabulary mapping graph through an attention rearrangement mechanism, so that image generation is guided. According to the method, the diffusion model does not need to be additionally trained, the problem that an existing model is difficult to process complex spatial relation prompts is solved in a plug-and-play mode, the alignment precision of generated images and text description in spatial layout and quantity is remarkably improved, and meanwhile high generation quality and reasoning speed are kept.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence content generation (AIGC) technology, and in particular, a text generation image space guidance method and system based on a lexical mapping graph. Background Technology

[0002] In recent years, text-to-image (T2I) generation techniques based on diffusion models have made great progress. Models such as Stable Diffusion and DALL-E can generate high-quality images from text descriptions. However, existing diffusion models often face challenges when dealing with text cues that contain complex spatial relationships, multiple objects, or specific layout requirements.

[0003] Traditional models typically use text encoders such as CLIP to process serialized text prompts. This linear encoding method limits the model's ability to understand two-dimensional spatial information. Specifically, this manifests as: object positions being disordered in the generated image (e.g., "the cat on the left" appears on the right), attribute leakage (e.g., an adjective modifying object A incorrectly modifies object B), or missing objects.

[0004] To address the spatial control problem, existing technologies mainly fall into two categories: one is to introduce additional control networks (such as ControlNet and GLIGEN), which typically requires a large amount of labeled data for additional training and increases the complexity of the model and the memory usage; the other is training-free methods based on attention mechanisms (such as Attend-and-Excite and DenseDiffusion), but these methods often require users to provide complex layout masks or perform time-consuming iterative optimizations, resulting in slow inference speeds and making it difficult to intuitively achieve fine spatial control through plain text. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by proposing a text generation image spatial guidance method based on a lexical mapping graph. This method utilizes a large language model to transform text into a two-dimensional lexical mapping graph, which is then used as a blueprint to directly guide the attention calculation of the diffusion model, thereby achieving efficient and accurate spatially controllable generation without the need for training.

[0006] The technical solution to achieve the objective of this invention is as follows: On the one hand, a text generation image spatial guidance method based on a lexical mapping graph is provided, the method comprising the following steps:

[0007] Step 1: Obtain the serialized text prompts input by the user, and use a large language model to perform semantic parsing on the serialized text prompts to extract the objects to be generated and their spatial relationships;

[0008] Step 2: Construct a two-dimensional grid structure word mapping map. The word mapping map fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image.

[0009] Step 3: Based on the vocabulary map, generate enhanced text prompts as conditional inputs for the diffusion model;

[0010] Step 4: In the denoising generation process of the diffusion model, an attention rearrangement mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model.

[0011] Step 5: Based on the adjusted attention score, guide the image generation process to ensure that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0012] Furthermore, step 2, which involves constructing a two-dimensional grid-structured vocabulary mapping graph, specifically includes:

[0013] Fine-tune the pre-trained large language model so that it can output a structured two-dimensional matrix;

[0014] For the input serialized text prompt, the large language model outputs an n×n matrix M, where each element corresponds to a specific object or background feature in the serialized text prompt;

[0015] Words representing the same object in the matrix form connected regions in space, and the position and shape of these connected regions characterize the position and shape of the object in the final image.

[0016] Furthermore, the fine-tuning of the pre-trained large language model specifically includes:

[0017] Using publicly available image and text datasets, object detection and segmentation are performed on images through an object detection model;

[0018] The segmented regions are mapped onto a two-dimensional network structure to generate corresponding word mapping map labels;

[0019] Construct "text prompt-vocabulary mapping" data pairs;

[0020] The data is used to fine-tune a large language model, enabling it to generate spatial layout matrices from text.

[0021] Furthermore, the attention rearrangement mechanism operates in a plug-and-play manner during the inference phase of the diffusion model, and is applied to both the self-attention layer and the cross-attention layer.

[0022] Furthermore, the attention reordering mechanism described in step 4 specifically includes:

[0023] For each word element w in the lexical mapping graph, i.e., matrix M, identify the connected components it forms in the matrix, that is, obtain the connected components of the word element w. ;

[0024] Construct a control matrix L corresponding to the size of the attention layer, where the element values ​​of L depend on whether the spatial positions corresponding to the query and the key in the attention mechanism belong to the same connected component;

[0025] Calculate the normalized connected component area vector S, and then calculate the final coefficient matrix. ;

[0026] coefficient matrix The corrected attention is introduced into the attention calculation formula. The calculation formula is:

[0027]

[0028] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. Let be the dimension of the key vector, and λ be an adjustable weight parameter with a value range of (0,1). express function.

[0029] Furthermore, the specific process of constructing the control matrix L corresponding to the attention layer size includes:

[0030] (1) Define the dimensions of the control matrix L ;

[0031] (2) The element in the x-th row and y-th column of the control matrix L The specific values ​​for are:

[0032] calculate , Get the elements in matrix M ;

[0033] calculate , Get the elements in matrix M ;

[0034] judge and Whether they belong to the same connected component; if so, then ,otherwise .

[0035] Furthermore, the specific formula for calculating the normalized connected component area vector S is as follows:

[0036]

[0037] in,

[0038]

[0039] In the formula, Let be the area of ​​the connected component corresponding to the y-th column in matrix M.

[0040] Furthermore, the calculation of the final coefficient matrix The calculation formula is:

[0041] .

[0042] Furthermore, the vocabulary map generated in step 2 supports interactive editing by users, who can directly modify the vocabulary distribution in the two-dimensional matrix to control the specific position and size of objects in the generated image.

[0043] On the other hand, a text-to-image spatial guidance system based on a lexical mapping is provided, the system comprising:

[0044] The first module is used to: obtain serialized text prompts input by the user, perform semantic parsing on the serialized text prompts using a large language model, and extract the objects to be generated and their spatial relationships;

[0045] The second module is used to: construct a two-dimensional grid structure of a word map, which fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image;

[0046] The third module is used to generate enhanced text prompts based on the vocabulary mapping graph, which serve as conditional inputs to the diffusion model.

[0047] The fourth module is used to implement: in the denoising generation process of the diffusion model, an attention reordering mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model;

[0048] The fifth module is used to: guide the image generation process based on the adjusted attention score, so that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0049] Compared with the prior art, the significant advantages of this invention are:

[0050] (1) Training-free: This method does not require any parameter updates or fine-tuning of the underlying diffusion model and can be directly applied to existing models such as Stable Diffusion.

[0051] (2) Precise spatial alignment: By converting linear text into a two-dimensional lexical map, spatial structure information is explicitly introduced, which effectively solves the problems of position confusion and attribute confusion in the generation of multiple objects.

[0052] (3) Easy to use: Users only need to input text, without providing additional edge maps, depth maps or bounding boxes, and the layout is automatically planned by the large language model. At the same time, users can visually edit the vocabulary map to achieve interactive generation.

[0053] (4) Efficient reasoning: Compared with iterative optimization-based methods, this method only modifies the attention calculation formula, and the reasoning time increases very little, maintaining high generation efficiency.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process of a text generation image space guidance method based on a vocabulary mapping graph in one embodiment.

[0056] Figure 2 This is an example diagram illustrating the generation of a lexical map in one embodiment, showing the transformation from text to a spatial matrix. Figure 2 (a) to (b) are examples of lexical map generation in different scenarios.

[0057] Figure 3 This document provides an example of the principle of the attention reordering mechanism of the present invention in one embodiment, along with a comparison of the generation effects in multiple scenarios. Figure 3 (a) to (d) in the figure are examples of the principle of attention reordering mechanism in different scenarios and comparison of generation effect, respectively. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0060] In one embodiment, combined Figure 1 This paper provides a text-to-image spatial guidance method based on a lexical mapping graph, the method comprising the following steps:

[0061] Step 1: Obtain the serialized text prompts input by the user, and use the Large Language Model (LLM) to perform semantic parsing on the serialized text prompts to extract the objects to be generated and their spatial relationships;

[0062] Step 2: Construct a two-dimensional grid structure lexical map. The lexical map fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image.

[0063] Step 3: Based on the vocabulary map, generate enhanced text prompts as conditional inputs for the diffusion model;

[0064] Step 4: In the denoising generation process of the diffusion model, an attention rearrangement mechanism is introduced to transform the vocabulary map into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model.

[0065] Step 5: Based on the adjusted attention score, guide the image generation process to ensure that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0066] Furthermore, in one embodiment, step 2, constructing a two-dimensional grid-structured vocabulary mapping graph, specifically includes:

[0067] Fine-tune the pre-trained large language model so that it can output a structured two-dimensional matrix;

[0068] For the input serialized text prompt, the large language model outputs an n×n matrix M, where each element corresponds to a specific object or background feature in the serialized text prompt;

[0069] Words representing the same object in the matrix form connected regions in space. The position and shape of these connected regions characterize the position and shape of the object in the final image. For example... Figure 2 As shown.

[0070] Furthermore, in one embodiment, the fine-tuning of the pre-trained large language model specifically includes:

[0071] Using publicly available image and text datasets, object detection and segmentation are performed on images through an object detection model;

[0072] The segmented regions are mapped onto a two-dimensional network structure to generate corresponding word mapping map labels;

[0073] Construct "text prompt-vocabulary mapping" data pairs;

[0074] The data is used to fine-tune a large language model, enabling it to generate spatial layout matrices from text.

[0075] Furthermore, in one embodiment, the attention rearrangement mechanism operates in a plug-and-play manner during the inference phase of the diffusion model, simultaneously applying to both the self-attention layer and the cross-attention layer.

[0076] Furthermore, in one embodiment, the attention reordering mechanism described in step 4 specifically includes:

[0077] For each word element w in the lexical mapping graph, i.e., matrix M, identify the connected components it forms in the matrix, that is, obtain the connected components of the word element w. ;

[0078] Construct a control matrix L corresponding to the size of the attention layer, where the element values ​​of L depend on whether the spatial positions corresponding to the query and the key in the attention mechanism belong to the same connected component;

[0079] Calculate the normalized connected component area vector S, and then calculate the final coefficient matrix. ;

[0080] coefficient matrix The corrected attention is introduced into the attention calculation formula. The calculation formula is:

[0081]

[0082] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. Let be the dimension of the key vector, and λ be an adjustable weight parameter used to control the strength of spatial guidance, with a value range of (0,1). express The function. Preferably, λ is 0.5.

[0083] Here, the modification of the present invention is applied to both the self-attention layer (enhancing aggregation within image blocks) and the cross-attention layer (enhancing alignment between image blocks and corresponding text tokens).

[0084] Furthermore, in one embodiment, the specific process of constructing the control matrix L corresponding to the attention layer size includes:

[0085] (1) Define the dimensions of the control matrix L ;

[0086] (2) The element in the x-th row and y-th column of the control matrix L The specific values ​​for are:

[0087] calculate , Get the elements in matrix M ;

[0088] calculate , Get the elements in matrix M ;

[0089] judge and Whether they belong to the same connected component; if so, then ,otherwise .

[0090] Furthermore, in one embodiment, the specific formula for calculating the normalized connected component area vector S is as follows:

[0091]

[0092] in,

[0093]

[0094] In the formula, Let be the area of ​​the connected component corresponding to the y-th column in matrix M.

[0095] Furthermore, in one embodiment, the calculation of the final coefficient matrix... The calculation formula is:

[0096] .

[0097] Furthermore, in one embodiment, the vocabulary map generated in step 2 supports interactive editing by the user, who can directly modify the vocabulary distribution in the two-dimensional matrix to control the specific position and size of objects in the generated image.

[0098] In one embodiment, a text-to-image spatial guidance system based on a lexical mapping is provided, the system comprising:

[0099] The first module is used to: obtain serialized text prompts input by the user, perform semantic parsing on the serialized text prompts using a large language model, and extract the objects to be generated and their spatial relationships;

[0100] The second module is used to: construct a two-dimensional grid structure of a word map, which fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image;

[0101] The third module is used to generate enhanced text prompts based on the vocabulary mapping graph, which serve as conditional inputs to the diffusion model.

[0102] The fourth module is used to implement: in the denoising generation process of the diffusion model, an attention reordering mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model;

[0103] The fifth module is used to: guide the image generation process based on the adjusted attention score, so that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0104] Specific limitations regarding the lexical mapping-based text-to-image spatial guidance system can be found in the limitations of the lexical mapping-based text-to-image spatial guidance method described above, and will not be repeated here. Each module in the aforementioned lexical mapping-based text-to-image spatial guidance system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0106] Step 1: Obtain the serialized text prompts input by the user, and use a large language model to perform semantic parsing on the serialized text prompts to extract the objects to be generated and their spatial relationships;

[0107] Step 2: Construct a two-dimensional grid structure word mapping map. The word mapping map fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image.

[0108] Step 3: Based on the vocabulary map, generate enhanced text prompts as conditional inputs for the diffusion model;

[0109] Step 4: In the denoising generation process of the diffusion model, an attention rearrangement mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model.

[0110] Step 5: Based on the adjusted attention score, guide the image generation process to ensure that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0111] For specific limitations on each step, please refer to the limitations of the text generation image space guidance method based on lexical mapping graphs above, which will not be repeated here.

[0112] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0113] Step 1: Obtain the serialized text prompts input by the user, and use a large language model to perform semantic parsing on the serialized text prompts to extract the objects to be generated and their spatial relationships;

[0114] Step 2: Construct a two-dimensional grid structure word mapping map. The word mapping map fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image.

[0115] Step 3: Based on the vocabulary map, generate enhanced text prompts as conditional inputs for the diffusion model;

[0116] Step 4: In the denoising generation process of the diffusion model, an attention rearrangement mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model.

[0117] Step 5: Based on the adjusted attention score, guide the image generation process to ensure that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

[0118] For specific limitations on each step, please refer to the limitations of the text generation image space guidance method based on lexical mapping graphs above, which will not be repeated here.

[0119] As a specific example, the invention will be further described in some embodiments.

[0120] Example 1

[0121] Analysis of the generation process and principles based on lexical mapping graphs, as follows: Figure 1 As shown, the method of the present invention mainly includes two stages: generation of prompt words to a word map and image synthesis guided by the word map.

[0122] Lexical mapping construction: Utilizing a large language model with powerful semantic understanding capabilities, the original prompts input by the user are parsed. For example... Figure 2 As shown, when the input prompt is "a landscape scene, rolling hills below themoon", the large language model outputs a structured matrix guided by preset system prompts. In this mapping, the upper region is filled with "moon" and the lower region is filled with "hill", which explicitly encodes the semantic and spatial location information of the object.

[0123] Attention rearrangement mechanism: During the inference process of the diffusion model, the system identifies connected components consisting of the same words in the lexical mapping graph $M$. For example... Figure 3 As shown, taking "A black cat standing by a window sill" as an example, the mapping map defines a "cat" region in the center of the matrix, surrounded by "window" regions. This invention modifies the attention calculation formula by constructing a control matrix and a coefficient matrix. This modification applies to self-attention and cross-attention layers in a plug-and-play manner, enhancing the semantic response of specific regions.

[0124] Example 2

[0125] Typical experimental procedures and result verification

[0126] To further illustrate the effectiveness of this invention, this embodiment follows... Figure 1 The five steps shown demonstrate a complete generative experiment process; the output results are available for reference. Figure 3 .

[0127] Step 1 (Get Input): Receive the original prompt words provided by the user, such as "A hot air balloon floating over a picturesque countryside".

[0128] Step 2 (Layout Generation): The large language model analyzes the input sequence and generates a layout such as... Figure 3 The word map shown at the bottom. In the map, the word "balloon" is precisely placed in the upper center of the matrix, while the background word "countryside" is distributed around it.

[0129] Step 3 (Weight Calculation): Based on the above vocabulary mapping graph, calculate the normalized connected component area vector and the final coefficient matrix, and add attention score weights to smaller connected components (such as hot air balloons).

[0130] Step 4 (Image Synthesis): The optimized prompts are input into the denoising network, and iterative denoising is performed under the guidance of the modified attention mechanism.

[0131] Step 5 (Output Image): Complete the denoising calculation process. The final output is as follows. Figure 3 As shown, the position, shape, and layout of the hot air balloon in the generated image are completely consistent with the requirements of the background, and the attribute leakage problem is effectively avoided.

[0132] Experimental data show that the present invention significantly improves the alignment accuracy of generated images in spatial layout without additional training, and its inference speed is superior to existing methods based on iterative optimization.

[0133] In summary, this invention solves the problem of existing models' inability to handle complex spatial relationship cues in a plug-and-play manner without requiring additional training of the diffusion model. It significantly improves the alignment accuracy of generated images and text descriptions in terms of spatial layout and quantity, while maintaining high generation quality and inference speed.

[0134] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A text-to-image spatial guidance method based on a lexical mapping graph, characterized in that, The method includes the following steps: Step 1: Obtain the serialized text prompts input by the user, and use a large language model to perform semantic parsing on the serialized text prompts to extract the objects to be generated and their spatial relationships; Step 2: Construct a two-dimensional grid structure word mapping map. The word mapping map fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image. Step 3: Based on the vocabulary map, generate enhanced text prompts as conditional inputs for the diffusion model; Step 4: In the denoising generation process of the diffusion model, an attention rearrangement mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model. Step 5: Based on the adjusted attention score, guide the image generation process to ensure that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

2. The text generation image spatial guidance method based on lexical mapping graph according to claim 1, characterized in that, Step 2, which involves constructing a two-dimensional grid-structured vocabulary mapping graph, specifically includes: Fine-tune the pre-trained large language model so that it can output a structured two-dimensional matrix; For the input serialized text prompt, the large language model outputs an n×n matrix M, where each element corresponds to a specific object or background feature in the serialized text prompt; Words representing the same object in the matrix form connected regions in space, and the position and shape of these connected regions characterize the position and shape of the object in the final image.

3. The text generation image spatial guidance method based on lexical mapping graph according to claim 2, characterized in that, The fine-tuning of the pre-trained large language model specifically includes: Using publicly available image and text datasets, object detection and segmentation are performed on images through an object detection model; The segmented regions are mapped onto a two-dimensional network structure to generate corresponding word mapping map labels; Construct "text prompt-vocabulary mapping" data pairs; The data is used to fine-tune a large language model, enabling it to generate spatial layout matrices from text.

4. The text generation image spatial guidance method based on lexical mapping graph according to claim 3, characterized in that, The attention rearrangement mechanism operates in a plug-and-play manner during the inference phase of the diffusion model, and is applied to both the self-attention layer and the cross-attention layer.

5. The text generation image spatial guidance method based on lexical mapping graph according to claim 4, characterized in that, The attention reordering mechanism described in step 4 specifically includes: For each word element w in the lexical mapping graph, i.e., matrix M, identify the connected components it forms in the matrix, that is, obtain the connected components of the word element w. ; Construct a control matrix L corresponding to the size of the attention layer, where the element values ​​of L depend on whether the spatial positions corresponding to the query and the key in the attention mechanism belong to the same connected component; Calculate the normalized connected component area vector S, and then calculate the final coefficient matrix. ; coefficient matrix The corrected attention is introduced into the attention calculation formula. The calculation formula is: ； In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. Let be the dimension of the key vector, and λ be an adjustable weight parameter with a value range of (0,1). express function.

6. The text generation image spatial guidance method based on lexical mapping graph according to claim 5, characterized in that, The specific process of constructing the control matrix L corresponding to the attention layer size includes: (1) Define the dimensions of the control matrix L ; (2) The element in the x-th row and y-th column of the control matrix L The specific values ​​for are: calculate , Get the elements in matrix M ; calculate , Get the elements in matrix M ; judge and Whether they belong to the same connected component; if so, then ,otherwise .

7. The text generation image spatial guidance method based on lexical mapping graph according to claim 6, characterized in that, The specific formula for calculating the normalized connected component area vector S is as follows: ； in, ； In the formula, Let be the area of ​​the connected component corresponding to the y-th column in matrix M.

8. The text generation image spatial guidance method based on lexical mapping graph according to claim 6, characterized in that, The final coefficient matrix is ​​calculated. The calculation formula is: 。 9. The text generation image spatial guidance method based on lexical mapping graph according to claim 1, characterized in that, The vocabulary map generated in step 2 supports interactive editing by users. Users can directly modify the vocabulary distribution in the two-dimensional matrix, thereby controlling the specific position and size of objects in the generated image.

10. A text generation image spatial guidance system based on a lexical mapping graph, according to the method of any one of claims 1 to 9, characterized in that, The system includes: The first module is used to: obtain serialized text prompts input by the user, perform semantic parsing on the serialized text prompts using a large language model, and extract the objects to be generated and their spatial relationships; The second module is used to: construct a two-dimensional grid structure of a word map, which fills the extracted objects and their attributes into the corresponding spatial positions of the two-dimensional matrix in the form of words or phrases, forming a spatial layout blueprint of the image; The third module is used to generate enhanced text prompts based on the vocabulary mapping graph, which serve as conditional inputs to the diffusion model. The fourth module is used to implement: in the denoising generation process of the diffusion model, an attention reordering mechanism is introduced to transform the vocabulary mapping into an attention control matrix and adaptively adjust the attention score of the attention layer in the diffusion model; The fifth module is used to: guide the image generation process based on the adjusted attention score, so that the semantic content and spatial structure of the generated image are consistent with the lexical mapping map, and output the final image.

Images