A text-enhanced image generation method based on diffusion model

By introducing a combination of large language model and diffusion model in the text to image generation model, extracting and analyzing object information and spatial relationships in text, building a knowledge graph and performing deep fusion generation, the problem of insufficient consistency between images and text in the prior art is solved, and high-quality and high-consistent image generation is achieved.

CN119379866BActive Publication Date: 2025-04-04ZHEJIANG UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411961541.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-12-30
Publication Date
2025-04-04
Estimated Expiration
2044-12-30

AI Technical Summary

Technical Problem

Existing text-to-image generation models have shortcomings in the consistency between the generated image and the input text, especially when dealing with complex scenes and multi-object layouts, it is difficult to achieve accurate semantic understanding and generation control.

Method used

The text-enhanced image generation method based on the diffusion model is adopted to analyze the text through a large language model, extract the name, quantity, attributes and spatial relationships of objects, build a knowledge graph to represent the object relationship, and use the controllable diffusion model and text generation module to deeply integrate it to generate an image that is highly consistent with the input text.

Benefits of technology

It significantly improves text comprehension ability and generation control accuracy, ensures high consistency between the generated image and the input text, solves the problems of multi-object scene layout control and the generation of proprietary noun objects, and improves the accuracy and visual quality of generated content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119379866B_ABST
    Figure CN119379866B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for text-enhanced image generation based on a diffusion model. Through scene text parsing and relationship modeling mechanisms, this framework significantly improves text understanding capabilities by deeply integrating a large language model, realizes accurate parsing and semantic understanding of complex scene relationships, innovatively designs core modules such as object extraction, relationship extraction, and object positioning, realizes accurate control of details such as object positions and attributes in scenes, and effectively breaks through the limitations of traditional models in image text rendering and proper noun object generation by cleverly integrating external search engines and professional text generation modules, significantly improves the accuracy and authenticity of generated content, and represents object relationships as graph structures by integrating knowledge graphs, and generates scene layouts with the most in-degree objects as core anchor points. This mechanism significantly improves the semantic consistency and visual aesthetics of scene layouts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text-enhanced image generation method based on a diffusion model. Background Art

[0002] When using text-to-image generative models to generate images, the generated results are often inconsistent with the input text prompts. For example, if the user inputs "a blue cat sitting on a red chair", the generated image may have attributes that do not match the brown cat or green chair; if the user inputs "two apples in a basket", three or more apples may be generated; if the user is asked to generate a "birthday cake with the words 'Happy Birthday' printed on it", the text may be blurred or display incorrect content; if the user expects to generate an image of the "Eiffel Tower", the structure of the tower is completely inconsistent with the real world. These inconsistencies have seriously affected the reliability and practicality of the model, especially in high-precision and high-risk fields such as medical imaging, forensic science, and criminal investigation. Existing solutions such as attention mechanisms or adversarial training still perform poorly when dealing with complex scenarios, and there is an urgent need for a technical solution that can ensure that the generated image is highly consistent with the input text. In response to these consistency issues, this paper proposes a unified framework based on a large language model and a diffusion model to ensure that the generated image is highly consistent with the input text.

[0003] In the existing text-to-image generation technology, the mainstream solution adopts the architecture of the diffusion probability model combined with the attention mechanism. This solution first uses an encoder such as CLIP to convert the input text prompt into a conditional vector, and then dynamically calculates the association between text and visual features through the cross-attention mechanism during the iterative denoising process of the diffusion model. This solution enhances the guiding role of text features in the diffusion process through the attention mechanism, attempting to improve the consistency between the generated image and the input text. Representative models include DALL-E and Stable Diffusion.

[0004] The closest existing technology currently has three key technical shortcomings:

[0005] On the one hand, the semantic understanding ability is insufficient. The existing models lack in-depth understanding of complex text descriptions and find it difficult to accurately grasp the logical relationship between multiple objects in the scene, resulting in limited understanding and expression of the overall scene semantics.

[0006] On the other hand, the generation control accuracy is not ideal, and it is often not precise enough when dealing with details such as object position and attributes. Especially in multi-object scenes, it is difficult to achieve accurate layout control and cannot effectively ensure the consistency of the generated content with the input text description.

[0007] In addition, existing technologies have obvious deficiencies in generating clear and readable text content and representing uncommon objects corresponding to proper nouns. These technical defects often lead to the generation results deviating from user expectations, limiting the reliability of the technology in practical applications. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a method for generating text-enhanced images based on a diffusion model. The present invention is achieved through the above technical solutions:

[0009] The present invention discloses a method for generating text-enhanced images based on a diffusion model, comprising:

[0010] Get the prompt word text;

[0011] Extract the object words in the prompt text and output the name, quantity, and attribute feature information of the object in a structured form;

[0012] According to the names of the objects obtained, the objects are classified into general object categories, text content categories in the image, and proper noun object categories existing in the real world, and a classified object list is output;

[0013] Based on the classified object list and prompt text, predict the spatial and interactive relationships between objects;

[0014] Based on the spatial relationship and interaction relationship between objects, an object relationship structure represented by a triple is constructed;

[0015] According to the object relationship structure, a complete relationship graph is constructed and the graph structure is analyzed, and the positions of the core anchor objects in the scene and their reference positions in the canvas are output;

[0016] According to the position of the core anchor point object and the spatial relationship between objects, the specific bounding box parameters of each object are calculated, and the spatial layout information and size ratio of all objects are output. The bounding box parameters include coordinate position and size.

[0017] According to the classified object list and the spatial layout information and size ratio of all objects, corresponding generation strategies are adopted for different object categories to output complete image content that matches the prompt word text.

[0018] As a further improvement, the extraction of object words in the prompt word text described in the present invention is specifically: the prompt word text is analyzed and recognized through a large language model, and the name of the object is identified through a large language model and an entity recognition method to identify the core noun in the prompt word.

[0019] As a further improvement, the general object category described in the present invention refers to common objects commonly seen in daily life, the text content category in the image refers to the content that needs to be presented in the form of text graphics, and the proper noun object category represents objects with special references in the display.

[0020] As a further improvement, the object list described in the present invention includes object names, attributes, quantities and categories, and the object list is used for calling and processing of subsequent knowledge graph generation and image generation modules.

[0021] As a further improvement, the triples described in the present invention are composed of a subject object, a relational predicate, and an object object. As a further improvement, the present invention constructs a complete relational graph and analyzes the graph structure according to the object relational structure as follows: after integrating all triples, a knowledge graph of object relations is generated, and the knowledge graph finally generated provides semantic relation input for object positioning and image generation in subsequent steps, thereby ensuring that the generated scene conforms to the semantics described by the prompt word; the relational graph uses objects as nodes and relations as edges to fully represent the relational structure between objects.

[0022] As a further improvement, the position of the core anchor object in the output scene described in the present invention and its reference position in the canvas are specifically: by analyzing the knowledge graph, the object with the most connections with other objects in the scene, that is, the node with the most in-degree in the knowledge graph, is identified as the core anchor object. The position of the anchor object is first analyzed through a large language model and the corresponding object size and coordinates are generated.

[0023] As a further improvement, the present invention calculates specific bounding box parameters of each object based on the position of the core anchor point object and the spatial relationship between the objects, and outputs the spatial layout information and size ratio of all objects;

[0024] Specifically:

[0025] Based on the position of the core anchor point and the spatial relationship in the knowledge graph, the bounding box parameters of other objects are calculated one by one. The bounding box parameters include position coordinates (x, y) and size ratios (w, h); by adjusting the relative position and size of the objects, the semantic accuracy of the relationship between objects and the overall visual aesthetics of the scene are balanced, and finally the bounding box parameters of all objects are output. The bounding box is used to guide subsequent image generation to ensure that the layout of objects in the scene is reasonable and conforms to the semantics described by the prompt word.

[0026] As a further improvement, the present invention adopts corresponding generation strategies for different object categories based on the classified object list and the spatial layout information and size ratios of all objects, and outputs complete image content that matches the prompt word text, specifically:

[0027] For general object categories, the object's bounding box parameters and the object's name, quantity, and attribute feature information are input into the controllable diffusion model to generate high-quality object images that meet the description of the prompt word;

[0028] According to the text content category in the image, the visual text generation module is used to generate clear and readable text images, while ensuring that the text and scene styles are consistent;

[0029] For the categories of proper noun objects that exist in the real world, reference images are obtained through search engines, and images of proper noun objects are generated by combining bounding box parameters and semantic information of prompt words. The images are then combined with a controllable generation model and a plug-and-play text generation module.

[0030] The final complete scene image is directly output through the above three types of input. The scene image can accurately reflect the description of the prompt word and has high visual quality, meeting the generation requirements of various complex scenes.

[0031] The beneficial effects of the present invention are as follows:

[0032] (1) Scene text parsing and relationship modeling mechanism. At the semantic understanding level, compared with the traditional solution that only relies on CLIP encoder and attention mechanism, this framework significantly improves the text understanding ability by deeply integrating a large language model, and achieves accurate parsing and semantic understanding of complex scene relationships;

[0033] (2) Accurate object positioning and spatial constraint system. In terms of generation control accuracy, the core modules such as object extraction, relationship extraction and object positioning are innovatively designed to achieve precise control of details such as the position and attributes of objects in the scene, while effectively solving the layout control problem of multi-object scenes;

[0034] (3) The multi-model collaborative fusion generation architecture, by cleverly integrating external search engines and professional text generation modules, effectively breaks through the limitations of traditional models in image text rendering and proper noun object generation, and significantly improves the accuracy and authenticity of the generated content.

[0035] (4) The knowledge graph-driven scene layout optimization mechanism integrates the knowledge graph to represent the object relationship as a graph structure, and generates the scene layout with the object with the most in-degree as the core anchor. Compared with the traditional method, this mechanism combines spatial relationship and semantic information to accurately calculate the bounding box parameters of the object, significantly improving the semantic consistency and visual aesthetics of the scene layout, and avoiding the problems of unreasonable layout or semantic deviation in the traditional method.

[0036] (5) Deep fusion of the controllable diffusion model and the text enhancement generation module. The controllable diffusion model can accurately control the position, quantity and size of the generated objects, and the text generation module can be combined to generate high-quality text content. The deep fusion of the two meets the dual needs of object layout and text generation in the scene. Compared with the traditional diffusion model, this mechanism significantly improves the semantic consistency and visual quality of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the algorithm flow of the text enhancement image generation method based on the diffusion model in the present invention. DETAILED DESCRIPTION

[0038] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention. Figure 1 It is an algorithm flow chart of the text enhancement image generation method based on the diffusion model in the present invention;

[0039] The present invention discloses a method for generating text-enhanced images based on a diffusion model, comprising:

[0040] S1 obtains the prompt word text;

[0041] S2 extracts the object words in the prompt word text and outputs the name, quantity, and attribute feature information of the object in a structured form; the prompt word text is analyzed and recognized through a large language model, and the name of the object is identified through a large language model and entity recognition method to identify the core noun in the prompt word.

[0042] By deploying a deep language model, the system can comprehensively analyze the semantic information of the text content and automatically detect and identify all object entities described in the text. During the recognition process, the system not only extracts basic object names, but also captures attribute features such as the quantity information and adjective descriptions of the objects. Each extracted object is structured into a concise description form, organically combining the attribute features of the object with the basic name, such as expressions such as "grazing sheep" or "red leather sofa". This precise object extraction mechanism provides an accurate object information foundation for subsequent image generation, ensuring that the generated image can fully reflect all key object elements in the text description.

[0043] S3 classifies the objects into general object categories, text content categories in images, and proper noun object categories in the real world based on the names of the objects obtained, and outputs a classified object list; general object categories refer to common objects commonly seen in daily life, text content categories in images refer to content that needs to be presented in the form of text graphics, and proper noun object categories represent objects with special references in the display. The object list includes object names, attributes, quantities, and categories, and is used for the subsequent calling and processing of knowledge graph generation and image generation modules.

[0044] After object extraction, the system uses an innovative classification mechanism to accurately classify the identified objects into three specific categories. The real-world proper noun object category deals with entities with unique references such as specific brands, landmarks, or celebrities. This classification strategy establishes a systematic object processing framework that enables the system to adopt the most suitable generation strategy for different types of objects. Through this classification process, the generation bias problem that is prone to occur in traditional methods when dealing with special objects is effectively solved, laying a solid foundation for subsequent relationship analysis and spatial layout.

[0045] S4 predicts the spatial and interactive relationships between objects based on the classified object list and prompt text;

[0046] S5 constructs an object relationship structure expressed in the form of triples based on the spatial relationship and interaction relationship between objects; the triples are composed of subject objects, relationship predicates and object objects.

[0047] By deeply integrating large language models, accurate extraction and analysis of object relationships described in texts are achieved. In this process, spatial relationships (such as "above...", "next to...") and interactive relationships (such as "holding", "looking at") described in the text are first identified and extracted. Based on the extracted object and relationship information, a structured triple representation is generated, and the triples are further integrated to construct a complete relationship graph, in which objects are graph nodes and relationships are edges connecting nodes. This scene representation method based on knowledge graphs can not only accurately capture complex relationships between objects, but also provide a structured reference for subsequent spatial layout.

[0048] S6 builds a complete relationship graph based on the object relationship structure and analyzes the graph structure, outputs the position of the core anchor object in the scene and its reference position in the canvas; after integrating all triples, generates a knowledge graph of object relationships, and the final generated knowledge graph provides semantic relationship input for object positioning and image generation in subsequent steps, thereby ensuring that the generated scene conforms to the semantics described by the prompt word; the relationship graph uses objects as nodes and relationships as edges to fully represent the relationship structure between objects. By analyzing the knowledge graph, identify the objects with the most connections to other objects in the scene, that is, the nodes with the most in-degree in the knowledge graph, as the core anchor objects. First, use the large language model to analyze the position of the anchor objects and generate the corresponding object size and coordinates.

[0049] S7 calculates the specific bounding box parameters of each object based on the position of the core anchor point object and the spatial relationship between objects, and outputs the spatial layout information and size ratio of all objects. The bounding box parameters include coordinate position and size. Based on the position of the core anchor point and the spatial relationship in the knowledge graph, the bounding box parameters of other objects are calculated one by one. The bounding box parameters include position coordinates (x, y) and size ratio (w, h). By adjusting the relative position and size of the objects, the semantic accuracy of the relationship between objects and the overall visual aesthetics of the scene are balanced, and finally the bounding box parameters of all objects are output. The bounding box is used to guide subsequent image generation to ensure that the layout of objects in the scene is reasonable and conforms to the semantics described by the prompt word.

[0050] Based on the constructed relationship graph, the core anchor objects in the scene are first determined by analyzing the graph structure, that is, the nodes with the most connections with other objects are selected as the reference points for the spatial layout. Subsequently, based on the relationship constraints between objects, the system determines the specific position and size ratio of each object in the canvas through precise spatial calculations. This process generates a series of bounding box parameters. When calculating the position, it not only considers the direct spatial relationship between objects, but also weighs the balance and aesthetics of the overall picture to ensure that the generated layout conforms to physical common sense and accurately expresses the spatial relationship requirements described in the text.

[0051] S8 adopts corresponding generation strategies for different object categories based on the classified object list and the spatial layout information and size ratio of all objects, and outputs complete image content that matches the prompt word text. According to the classified object list and the spatial layout information and size ratio of all objects, S8 adopts corresponding generation strategies for different object categories, and outputs complete image content that matches the prompt word text. Specifically:

[0052] For general object categories, the object's bounding box parameters and the object's name, quantity, and attribute feature information are input into the controllable diffusion model to generate high-quality object images that meet the description of the prompt word;

[0053] According to the text content category in the image, the visual text generation module is used to generate clear and readable text images, while ensuring that the text and scene styles are consistent;

[0054] For the categories of proper noun objects that exist in the real world, reference images are obtained through search engines, and images of proper noun objects are generated by combining bounding box parameters and semantic information of prompt words;

[0055] The above three types of objects are input into a model that combines a controllable generation model with a plug-and-play text generation module, and the final complete scene image is directly output. The scene image can accurately reflect the prompt word description and has high visual quality, meeting the generation needs of a variety of complex scenes.

[0056] This classification and collaborative processing method enables the system to accurately control the generation quality of different types of objects. The final output image is not only visually realistic and natural, but also accurately reflects the scene details and spatial relationships described in the prompt words.

[0057] Application scenario cases

[0058] Scene 1: Simple indoor scene User input prompt word: "A black cat is sitting on a red sofa with an open book next to it"

[0059] (1) The language model first performs semantic analysis on the input prompt words and identifies the physical objects described in the text. Through deep parsing, the system accurately extracts the three core objects in the scene: "black cat" as the main active entity, "red sofa" as the main furniture in the scene, and "open book" as an auxiliary object in the scene. The system also retains the attribute description information of each object, such as modifiers such as "black", "red" and "open".

[0060] (2) After obtaining the object list, the system performs object classification. Since all objects in the scene are common physical objects in daily life, the system classifies "black cat", "red sofa" and "open book" as general objects. This classification method clarifies that the subsequent generation process will adopt the standard object rendering process.

[0061] (3) The system analyzes the spatial relationships described in the text and identifies two sets of core relationship structures. Through semantic analysis, it generates relationship triplets: (black cat - sitting on it - red sofa) and (book - next to it - red sofa). These relationship triplets form a simple but clear scene relationship network, providing a basic framework for subsequent spatial layout.

[0062] (4) Based on the extracted relationship network, the system first determines the red sofa as the core anchor object of the scene because it has the most spatial relationship connections with other objects. Then, the system performs spatial planning on the standardized canvas: the sofa is set in the center of the picture (coordinates: 400, 300, 600, 400), the black cat is placed in the upper center of the sofa based on the "sitting on it" relationship (coordinates: 450, 250, 550, 350), and the book is placed on the right side of the sofa based on the "next to it" relationship (coordinates: 620, 320, 680, 380).

[0063] (5) The final image generation stage is relatively simple because all objects belong to the general object category. The system directly inputs the calculated bounding box parameters and the corresponding object description information into the main generation method. Through a unified rendering process, the system generates a complete scene image containing all objects. During the generation process, the system pays special attention to maintaining the relative position relationship between objects, ensuring that the black cat is naturally located above the sofa and the book is appropriately placed next to it, thereby accurately restoring the scene described by the prompt word.

[0064] Scenario 2: Complex business scenario User input prompt word: "In a Starbucks coffee shop, a young woman wearing glasses is working on a MacBook laptop. There is a cup of latte labeled 'Emma' on the table, and a 'STARBUCKS COFFEE' logo is hanging on the wall."

[0065] (1) The system uses a deep language model to fully parse this complex scene description and extract all the key objects in the scene: the main character "young woman with glasses", the electronic device "MacBook laptop", the drink "latte labeled 'Emma'", the environmental element "STARBUCKS COFFEE logo", and the basic scene elements "wall" and "table". The system also retains the feature description of each object to ensure that the scene details can be accurately restored.

[0066] (2) The system classifies the extracted objects in multiple dimensions. The objects are divided into three categories: the general object category includes basic scene elements such as "young woman wearing glasses", "table" and "wall"; the proper noun object category includes brand-related objects such as "MacBook laptop" and "STARBUCKS logo"; the text content category includes text content that requires special rendering, such as "Emma" on a coffee cup and "STARBUCKS COFFEE" on a wall. This detailed classification lays the foundation for subsequent differentiated processing.

[0067] (3) Through in-depth semantic analysis, the system extracts the complex relationship network in the scene and generates multiple relationship triplets: (young woman using MacBook) represents the main action relationship, (latte placed on the table) and (STARBUCKS logo hung on the wall) represent the spatial position relationship of objects, and (young woman sitting at the table) represents the interaction relationship between the character and the scene. These relationships construct a complete scene relationship network, which describes in detail the spatial and interactive associations between the various elements.

[0068] (4) Based on the established relationship network, the system identifies the young woman as the main anchor point of the scene and serves as the reference core for the spatial layout. Detailed spatial planning is then carried out: the young woman is placed slightly to the left of the center of the picture (coordinates: 300, 200, 500, 400) to ensure that she is the visual center of the picture; the laptop is located in the center of the desktop (coordinates: 400, 300, 500, 350) to form a natural posture with the character; the coffee cup is placed on the right side of the desktop (coordinates: 550, 300, 600, 350) as an embellishment; and the brand logo is placed in a prominent position above the background wall (coordinates: 300, 50, 700, 150).

[0069] (5) In the final generation stage, the system uses three different rendering strategies: for general object categories (people, tables, walls), the main frame is directly used for regular rendering; for proper noun object categories (MacBook, Starbucks logo), the system first obtains real reference images through search engines, and then combines these references with scene requirements for generation to ensure the accuracy of brand characteristics; for text content categories (names and brand text on coffee cups), a dedicated text rendering module is enabled to ensure the clarity and readability of the text. Finally, the system combines these three types of elements into a unified scene image, paying special attention to maintaining the coordination and realism between objects.

[0070] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A text-enhanced image generation method based on a diffusion model, characterized in that: include: Get the prompt word text; Extract the object words in the prompt text and output the name, quantity, and attribute feature information of the object in a structured form; According to the names of the objects obtained, the objects are classified into general object categories, text content categories in the image, and proper noun object categories existing in the real world, and a classified object list is output; Based on the classified object list and prompt text, predict the spatial and interactive relationships between objects; Based on the spatial relationship and interaction relationship between objects, an object relationship structure represented by a triple is constructed; According to the object relationship structure, a complete relationship graph is constructed and the graph structure is analyzed, and the positions of the core anchor objects in the scene and their reference positions in the canvas are output; According to the position of the core anchor point object and the spatial relationship between the objects, the specific bounding box parameters of each object are calculated, and the spatial layout information and size ratio of all objects are output. The bounding box parameters include coordinate position and size. According to the classified object list and the spatial layout information and size ratio of all objects, corresponding generation strategies are adopted for different object categories to output complete image content that matches the prompt word text.

2. The text-enhanced image generation method based on the diffusion model according to claim 1 is characterized in that: The extraction of the object words in the prompt word text is specifically as follows: the prompt word text is analyzed and recognized through a large language model, and the name of the object is identified through a large language model and an entity recognition method to identify the core noun in the prompt word.

3. The text-enhanced image generation method based on the diffusion model according to claim 2 is characterized in that: The general object category refers to common objects commonly seen in daily life, the text content category in the image refers to content that needs to be presented in the form of text graphics, and the proper noun object category represents objects with special references in the display.

4. The text-enhanced image generation method based on the diffusion model according to claim 1, 2 or 3, characterized in that: The object list includes object name, attributes, quantity and category, and the object list is used for calling and processing of subsequent knowledge graph generation and image generation modules.

5. The method for generating text-enhanced images based on a diffusion model according to claim 4, characterized in that: The triple is composed of a subject object, a relation predicate and an object object.

6. The method for generating text-enhanced images based on a diffusion model according to claim 5, characterized in that: According to the object relationship structure, a complete relationship graph is constructed and the graph structure is analyzed as follows: After integrating all triples, a knowledge graph of object relationships is generated. The final knowledge graph provides semantic relationship input for object positioning and image generation in subsequent steps, thereby ensuring that the generated scene conforms to the semantics described by the prompt word; the relationship graph uses objects as nodes and relationships as edges to fully represent the relationship structure between objects.

7. The method for generating text-enhanced images based on a diffusion model according to claim 6, characterized in that: The position of the core anchor object in the output scene and its reference position in the canvas are specifically as follows: by analyzing the knowledge graph, the object with the most connections with other objects in the scene, that is, the node with the most in-degree in the knowledge graph, is identified as the core anchor object. First, the position of the anchor object is analyzed through a large language model and the corresponding object size and coordinates are generated.

8. The method for generating text-enhanced images based on a diffusion model according to claim 1, 2, 3, 5, 6 or 7, characterized in that: The specific bounding box parameters of each object are calculated according to the position of the core anchor point object and the spatial relationship between the objects, and the spatial layout information and size ratio of all objects are output; Specifically: Based on the position of the core anchor point and the spatial relationship in the knowledge graph, the bounding box parameters of other objects are calculated one by one. The bounding box parameters include position coordinates (x, y) and size ratios (w, h); by adjusting the relative position and size of the objects, the semantic accuracy of the relationship between objects and the overall visual aesthetics of the scene are balanced, and finally the bounding box parameters of all objects are output. The bounding box is used to guide subsequent image generation to ensure that the layout of objects in the scene is reasonable and conforms to the semantics described by the prompt word.

9. The method for generating text-enhanced images based on a diffusion model according to claim 8, characterized in that: The above-mentioned method adopts corresponding generation strategies for different object categories based on the classified object list and the spatial layout information and size ratio of all objects, and outputs complete image content that matches the prompt word text as follows: For general object categories, the object's bounding box parameters and the object's name, quantity, and attribute feature information are input into the controllable diffusion model to generate high-quality object images that meet the description of the prompt word; According to the text content category in the image, the visual text generation module is used to generate clear and readable text images, while ensuring that the text and scene styles are consistent; For the categories of proper noun objects that exist in the real world, reference images are obtained through search engines, and images of proper noun objects are generated by combining bounding box parameters and semantic information of prompt words. The images are then combined with a controllable generation model and a plug-and-play text generation module. The final complete scene image is directly output through the above three types of inputs. The scene image can accurately reflect the description of the prompt word and has high visual quality, meeting the generation requirements of various complex scenes.

Citation Information

Patent Citations

  • A task-oriented text generation image network model

    CN111858954A

  • Text-to-image generation algorithm based on improved text parser

    CN115018941A