A text-driven automatic image generation method and system

Abstract

The invention discloses a text-driven automatic image generation method and system, belonging to the field of image generation technology. The specific steps of the generation method are as follows: I. accepting and parsing text data input by a user, explicitly modeling entities and relationships in the text, and extracting implicit terms in the text data; the invention improves the multimodal alignment of text and visual features, avoids the visual deviation problem caused by semantic information errors, improves the detail richness and semantic information expansion capability of the generated image, improves the global layout rationality of the generated image, and strengthens the personalization and stylization capabilities of the generated image; avoids the rigid effect brought by fixed templates, improves the performance of the generated image in scene complexity and detail richness, significantly improves the accuracy of modeling the relationship between objects in complex scenes, improves the intelligence level of image generation, and makes the generation process more adaptive.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a text-driven automatic image generation method and system. Background Art

[0002] Text-driven image generation methods are committed to directly converting natural language descriptions into high-quality visual content, and have broad application prospects. For example, in cultural creativity, advertising design, and virtual scene construction, users can generate complex visual works through simple text descriptions, thereby greatly improving creative efficiency and expression ability. However, this research direction still faces many challenges, mainly reflected in how to better understand text semantics and generate visual details that meet the needs, and how to balance generation quality and efficiency. With the rapid development of artificial intelligence technology, text-driven image generation has gradually become an important research field spanning computer vision, natural language processing, and multimodal learning. This technology generates visual content that meets user needs by parsing and understanding information in natural language, and shows broad application prospects in virtual reality, advertising creation, digital media, game design, and human-computer interaction. However, there are still many challenges in text-driven image generation, which are mainly concentrated in the following aspects: the complexity of text semantic understanding, the quality limitation of visual content generation, the difficulty of multimodal information alignment, and the bottleneck of complex scene modeling and optimization.

[0003] The existing text-driven automatic image generation methods and systems have poor multimodal alignment effects between text and visual features, and there are visual deviation problems caused by semantic information errors, which reduce the detail richness and semantic information expansion capabilities of the generated images; in addition, the existing text-driven automatic image generation methods and systems have rigid effects caused by fixed templates, which reduce the performance of the generated images in scene complexity and detail richness, and the accuracy of modeling the relationship between objects in complex scenes is low; therefore, we propose a text-driven automatic image generation method and system. Summary of the invention

[0004] The purpose of the present invention is to solve the defects in the prior art and to propose a text-driven automatic image generation method and system.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A text-driven automatic image generation method, the specific steps of the generation method are as follows:

[0007] Ⅰ. Accept and parse text data input by users, explicitly model entities and relationships in the text, and extract implicit semantic information from the text data;

[0008] II. Generate a preliminary visual feature vector based on the semantic information obtained from the analysis, supplement the missing details, and infer reasonable scene extensions;

[0009] III. Optimize the arrangement and matching of visual concepts, deploy the required models and dependent environments, load the image generation model, and prepare generation parameters that match the semantic information input;

[0010] IV. According to the preliminary visual feature vector, input it into the image generation model to generate an initial image, evaluate the quality of different initial images, and further optimize the local details of the image;

[0011] V. Evaluate the scene semantic information of the initial image and adjust the object position, scale and interaction relationship in the initial image;

[0012] VI. Re-input the adjusted initial image into the image generation model, generate higher quality images through cyclic optimization, and guide the generation of diversified images of different styles and perspectives;

[0013] Ⅶ. Store generation results and log records, and provide a user interface for displaying generation results and for users to make fine adjustments and make further requests.

[0014] As a further solution of the present invention, the specific steps of explicitly modeling entities and relationships in the text and extracting implicit semantic information from the text data in step I are as follows:

[0015] Q1.1: Preprocess the natural language text input by the user, including word segmentation, stop word removal and grammatical analysis. Use named entity recognition technology to extract entities from the preprocessed natural language text, and use dependency syntax analysis to identify the relationship between entities.

[0016] Q1.2: Map the extracted entities to the predefined knowledge graph and identify the nodes corresponding to the entities. Use the relationship extraction model to analyze the relationships in the natural language text and establish connections between entity nodes in the knowledge graph. Add new nodes and edges to the knowledge graph based on the mapped entities and relationships to construct the corresponding semantic graph. Use the constructed semantic graph to describe the core meaning of the text and its extended semantics.

[0017] Q1.3: Use the BERT model to deeply encode the input text, capture syntactic, contextual, and semantic hierarchical information, and combine it with the semantic graph. Through the multi-head attention mechanism, the explicitly modeled structural information is injected into the text semantic representation. The semantic information in the semantic graph is then combined with the implicit semantic information representation extracted by the BERT model. The semantic information from different sources is unified through weighted averaging to generate the final semantic feature vector.

[0018] As a further solution of the present invention, the specific steps of generating a preliminary visual feature vector according to the semantic information obtained by parsing in step II are as follows:

[0019] S2.1: Build a training dataset containing large-scale image-text pairs ,in For images, To describe the natural language text of the image, normalize and resize the images in the training dataset, generate input tensors, segment and embed the text, generate input tensors, and initialize a set of visual generation models including an image encoder and a text encoder;

[0020] S2.2: Input the training dataset into the visual generation model, and the image encoder inputs the image Mapping to embedding vector , the text encoder takes the text input Mapping to embedding vector , then calculate the similarity of each pair of image-text embeddings and construct the contrastive learning loss function ,in Represents the total loss function value, represents the number of training samples, Representative image embedding and The similarity of text embeddings, as well as Respectively represent The embedding vectors of image and text pairs, represents the temperature parameter, which is used to adjust the smoothness of the distribution;

[0021] S2.3: With the goal of minimizing the gap between the similarity of positive samples and the similarity of negative samples, the parameters of the two encoders are updated using the gradient descent optimization algorithm. After each round of training, a set of data that has not been used is randomly selected as a validation set. The model performance is evaluated on the validation set, and the accuracy of text-to-image retrieval and image-to-text retrieval is calculated. The hyperparameters and model structure are adjusted based on the evaluation results.

[0022] S2.4: Repeat the training to verify the visual generation model until the contrastive learning loss function value converges to the preset range. Input the semantic information generated by text parsing and semantic graph into the pre-trained visual generation model, and generate the corresponding semantic embedding vector through the text encoder. , and then use the image encoder to generate the corresponding visual feature vector , to obtain a preliminary visual feature vector corresponding to the input semantic information;

[0023] S2.5: Based on the preliminary visual features provided by the visual generation model, query the hierarchical relationship of the corresponding entity in the semantic graph, and based on the queried hierarchical relationship, expand the preliminary features based on the adjacency matrix of the semantic graph to supplement the missing details, and then re-input the expanded semantic features into the image encoder in the visual generation model to generate a complete visual feature vector.

[0024] As a further solution of the present invention, the specific steps of optimizing the arrangement and matching of visual concepts in step III are as follows:

[0025] S3.1: Model the expanded visual feature vector as a node set of image visual concepts, where each node represents a feature vector of a visual concept. The semantics and layout association between visual concepts are derived based on the semantic graph and visual rules, and the derivation result is used as the edge weight.

[0026] S3.2: Initialize the pheromone concentration of each edge, generate multiple groups of optimization bodies, and randomly distribute them in the node set. Each optimization body takes the current node as the starting node, starts from the starting node, calculates the selection probability of each group of unvisited nodes, and gradually selects the next node according to the calculated selection probability until the preset arrangement length is reached;

[0027] S3.3: Calculate the quality of the paths constructed by each group of optimization bodies according to the visual rationality and aesthetic indicators of the paths, globally update the pheromone concentration of the edges according to the quality of each path, and repeat the path construction and pheromone concentration update until the path quality improvement value converges to the preset range;

[0028] S3.4: Arrange the order and collocation of the visual feature vectors according to the optimal path after convergence, and output the optimized visual feature vectors, and generate a scene blueprint based on the optimized visual feature vectors to describe the layout, relationship and attributes of the objects.

[0029] A text-driven automatic image generation system, comprising an input parsing module, a modeling extension module, a visual generation module, a model deployment module, an image generation module, an evaluation and screening module, a scene adjustment module, a detail optimization module, a diverse generation module, a quality enhancement module, a display interaction module, and a recording and feedback module;

[0030] The input parsing module is used to receive natural language text input by a user and parse the semantic structure of the text;

[0031] The modeling and extension module performs semantic modeling and extension on the parsing results;

[0032] The visual generation module is used to convert the semantic representation into a preliminary visual concept, and optimize the arrangement and priority of visual elements to generate a visual blueprint that conforms to the text semantics;

[0033] The model deployment module is used to deploy and manage various models and components of the generation system;

[0034] The image generation module is used to generate a preliminary image based on semantic and visual concepts;

[0035] The evaluation and screening module is used to evaluate the quality of the initially generated images and screen the images;

[0036] The scene adjustment module is used to verify the relationship between objects and dynamically adjust the scene layout of the generated image;

[0037] The detail optimization module is used to enhance texture details and automatically adjust the color balance and contrast of the image;

[0038] The diverse generation module is used to guide the generation of images of different styles and provide variants of multiple viewing angles and multiple lighting conditions;

[0039] The quality enhancement module is used to improve the resolution and clarity of the final image and eliminate artifacts in the generation process;

[0040] The display interaction module is used to display multiple generated images and allow users to rate or select the images;

[0041] The recording and feedback module is used to record the parameters and results of the generation process and analyze user selections and preferences.

[0042] As a further solution of the present invention, the specific steps of the evaluation and screening module for screening images are as follows:

[0043] S4.1: The generation result of an initial image and each intermediate state in the generation process are represented as a set of nodes, and the initial state of the search is represented as the root node. The edges between the nodes represent the selection from one state to the next state, and the specific adjustment steps of the generated image are associated. Then, each quality estimation value and the number of visits are initialized;

[0044] S4.2: Starting from the root node, the UCB values ​​of each child node connected to the root node are selected, and the child node whose UCB value meets the preset rules is selected according to the node selection strategy as the next expansion node. When an incompletely expanded child node is selected, the node is expanded to generate a new child node. The new child node represents a state that has not been explored yet, corresponding to a new image generation condition;

[0045] S4.3: Random simulation is performed starting from the expanded node until the image is completely generated. The image quality score of the image in the fully generated state is calculated based on visual consistency, semantic rationality, and image quality. The simulation result is then used as quality feedback to trace the quality information back to the root node along the search path, and the quality estimation values ​​and visit counts of all nodes on the path are updated at the same time.

[0046] S4.4: Through multiple selections, expansions, simulations, and backtracking, the quality distribution in the tree structure is gradually optimized until a preset number of iterations is reached. A path with a quality estimate that satisfies a preset selection rule is selected from the root node to the terminal state, and the corresponding image is output.

[0047] As a further solution of the present invention, the specific steps of the scene adjustment module dynamically adjusting the scene layout of the generated image are as follows:

[0048] S5.1: Constructing the state space , and each state in the state space Each state represents the structure of the current scene, including the specific configuration of the objects, layout, and style attributes in the scene, and collects the various adjustment operations that can be performed on the scene. , and construct the corresponding action space , calculated in the state Next action After transfer to state probability;

[0049] S5.2: Calculate execution actions by combining target style, semantic information consistency, and visual aesthetic score The reward value of the post-scene adjustment effect is then used to generate the initial scene configuration based on the input text description using the semantic graph and semantic parsing results , as the starting point for adjustment, in the current state Next, select an action that satisfies the preset rules. Adjust the scene and train the acquisition strategy through Q-Learning reinforcement learning method ;

[0050] S5.3: Execution of actions , according to the state transfer function Transition to a new state , to obtain the structural update of the adjusted scene, and calculate the new state according to the target style and semantic information requirements The corresponding reward is given, and the quality estimate of the action is updated at the same time, and the strategy is optimized and adjusted through reward feedback and state transfer results;

[0051] S5.4: Repeat action selection, state transfer, and strategy optimization until the preset target style requirements are met. , generate the corresponding image structure configuration, and input the adjusted scene into the subsequent diverse generation module to generate images that meet the target style and semantic information requirements.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention improves the multimodal alignment of text and visual features, avoids the visual deviation problem caused by semantic information errors, improves the detail richness and semantic information expansion capability of the generated image, enhances the global layout rationality of the generated image, and strengthens the personalization and stylization capabilities of the generated image.

[0054] 2. The present invention avoids the rigid effect brought by fixed templates, improves the performance of generated images in terms of scene complexity and detail richness, significantly improves the accuracy of modeling the relationship between objects in complex scenes, improves the intelligence level of image generation, and makes the generation process more adaptive. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0056] Figure 1 A flowchart of a text-driven automatic image generation method proposed by the present invention;

[0057] Figure 2 This is a system block diagram of a text-driven automatic image generation system proposed by the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0059] Example 1

[0060] Reference Figure 1 This embodiment discloses a text-driven automatic image generation method, and the specific steps of the generation method are as follows:

[0061] Accept and parse text data input by users, explicitly model the entities and relationships in the text, and extract implicit semantic information from the text data.

[0062] Specifically, the natural language text input by the user is preprocessed, including word segmentation, stop word removal and grammatical analysis. Entities are extracted from the preprocessed natural language text through named entity recognition technology, and dependency syntax analysis is used to identify the relationship between entities. The extracted entities are mapped to a predefined knowledge graph, and the nodes corresponding to the entities are identified. The relationship in the natural language text is analyzed using a relationship extraction model, and connections between entity nodes are established in the knowledge graph. According to the mapped entities and relationships, new nodes and edges are added to the knowledge graph to construct a corresponding semantic graph. The constructed semantic graph is then used to describe the core meaning of the text and its extended semantics. The BERT model is used to deeply encode the input text to capture syntactic, contextual and semantic hierarchical information. Combined with the semantic graph, the explicitly modeled structural information is injected into the text semantic representation through a multi-head attention mechanism. The semantic information in the semantic graph is then combined with the implicit semantic information representation extracted by the BERT model. The semantic information from different sources is unified by weighted averaging to generate the final semantic feature vector.

[0063] Generate a preliminary visual feature vector based on the semantic information obtained from the analysis, supplement the missing details and infer reasonable scene extensions.

[0064] Specifically, we build a training dataset containing large-scale image-text pairs. ,in For images, To describe the natural language text of the image, normalize and resize the images in the training dataset, generate input tensors, segment and embed the text, generate input tensors, and initialize a set of visual generation models including an image encoder and a text encoder. Input the training dataset into the visual generation model, and the image encoder inputs the image Mapping to embedding vector , the text encoder takes the text input Mapping to embedding vector , then calculate the similarity of each pair of image-text embeddings, and construct a contrastive learning loss function with the goal of minimizing the similarity gap between positive samples and negative samples. Use the gradient descent optimization algorithm to update the parameters of the two encoders. After each round of training, randomly select a group from the unused data as a validation set, evaluate the model performance on the validation set, calculate the accuracy of text-to-image retrieval and image-to-text retrieval, and adjust the hyperparameters and model structure based on the evaluation results. Repeat the training to verify the visual generation model until the contrastive learning loss function value changes and converges to the preset range. Then, input the semantic information generated by text parsing and semantic maps into the pre-trained visual generation model, and generate the corresponding semantic embedding vector through the text encoder. , and then use the image encoder to generate the corresponding visual feature vector , to obtain the preliminary visual feature vector corresponding to the input semantic information, and query the hierarchical relationship of the corresponding entity in the semantic graph according to the preliminary visual features provided by the visual generation model, and expand the preliminary features based on the adjacency matrix of the semantic graph according to the queried hierarchical relationship, supplement the missing details, and then re-input the expanded semantic features into the image encoder in the visual generation model to generate a complete visual feature vector.

[0065] In this embodiment, the specific calculation formula of the contrastive learning loss function is as follows:

[0066] ,

[0067] in, , represents the total loss function value, represents the number of training samples, Representative image embedding and The similarity of text embeddings, as well as Respectively represent The embedding vectors of image and text pairs, Represents the temperature parameter, which is used to adjust the smoothness of the distribution.

[0068] Optimize the arrangement and matching of visual concepts, deploy the required models and dependent environments, load the image generation model, and prepare generation parameters that match the semantic information input.

[0069] Specifically, the expanded visual feature vector is modeled as a node set of image visual concepts, where each node represents a feature vector of a visual concept. The semantics and layout association between visual concepts are derived based on semantic graphs and visual rules, and the derivation results are used as edge weights to initialize the pheromone concentration of each edge. At the same time, multiple groups of optimization bodies are generated and randomly distributed in the node set. Each optimization body takes the current node as the starting node, starts from the starting node, calculates the selection probability of each group of unvisited nodes, and gradually selects the next node based on the calculated selection probability until the preset arrangement length is reached. According to the visual rationality and aesthetic indicators of the path, the quality of the path constructed by each group of optimization bodies is calculated, and the pheromone concentration of the edge is globally updated according to the quality of each path. The path construction and pheromone concentration update are repeated until the path quality improvement value converges to a preset range. According to the converged optimal path, the order and collocation of the visual feature vectors are arranged, and the optimized visual feature vectors are output. Based on the optimized visual feature vectors, a scene blueprint is generated to describe the layout, relationship and attributes of the objects.

[0070] According to the preliminary visual feature vector, it is input into the image generation model to generate an initial image, evaluate the quality of different initial images, and further optimize the local details of the image.

[0071] Evaluate the scene semantic information of the initial image and adjust the object positions, scales, and interactions in the initial image.

[0072] The adjusted initial image is re-input into the image generation model to generate higher quality images through cyclic optimization, and guide the generation of diverse images of different styles and perspectives.

[0073] Stores generation results and logs, and provides a user interface for displaying generation results and for users to make fine adjustments and make further requests.

[0074] Example 2

[0075] Reference Figure 2 ,This embodiment discloses a text-driven automatic image generation system, including an input parsing module, a modeling extension module, a visual generation module, a model deployment module, an image generation module, an evaluation and screening module, a scene adjustment module, a detail optimization module, a diverse generation module, a quality enhancement module, a display interaction module, and a recording and feedback module;

[0076] The input parsing module is used to receive natural language text input by the user and parse the semantic structure of the text; the modeling and extension module performs semantic modeling and expansion on the parsing results; the visual generation module is used to convert the semantic representation into preliminary visual concepts, and optimize the arrangement and priority of visual elements to generate a visual blueprint that conforms to the text semantics.

[0077] The model deployment module is used to deploy and manage the various models and components of the generation system; the image generation module is used to generate preliminary images based on semantic and visual concepts; the evaluation and screening module is used to evaluate the quality of the preliminary generated images and screen the optimal results.

[0078] Specifically, the generation result of an initial image and the intermediate states in the generation process are represented as a group of nodes, and the initial state of the search is represented as the root node. The edges between the nodes represent the selection from one state to the next state, and the specific adjustment steps of the associated generated image are then initialized. The UCB values ​​of the child nodes connected to the root node are then used to select the child nodes whose UCB values ​​meet the preset rules according to the node selection strategy as the next expansion node. When an incompletely expanded child node is selected, the node is expanded to generate a new child node, which represents a state that has not yet been explored. , corresponding to the new image generation conditions, random simulation is performed starting from the expanded node until the image is completely generated. The image quality score of the image in the fully generated state is calculated based on visual consistency, semantic rationality and image quality. After that, the simulation result is used as quality feedback, and the quality information is traced back to the root node along the search path. At the same time, the quality estimation value and the number of visits of all nodes on the path are updated. Through multiple selections, expansions, simulations and backtracking, the quality distribution in the tree structure is gradually optimized until the preset number of iterations is reached. A path with a quality estimation value that meets the preset selection rule is selected from the root node to the terminal state, and the corresponding image is output.

[0079] The scene adjustment module is used to verify the relationship between objects and dynamically adjust the scene layout of the generated image.

[0080] Specifically, construct the state space , and each state in the state space Each state represents the structure of the current scene, including the specific configuration of the objects, layout, and style attributes in the scene, and collects the various adjustment operations that can be performed on the scene. , and construct the corresponding action space , calculated in the state Next action After transfer to state The probability of executing actions is calculated by combining the target style, semantic information consistency and visual aesthetic score The reward value of the post-scene adjustment effect is then used to generate the initial scene configuration based on the input text description using the semantic graph and semantic parsing results , as the starting point for adjustment, in the current state Next, select an action that satisfies the preset rules. Adjust the scene and train the acquisition strategy through Q-Learning reinforcement learning method , perform the action , according to the state transfer function Transition to a new state , to obtain the structural update of the adjusted scene, and calculate the new state according to the target style and semantic information requirements The corresponding reward is given, and the quality estimate of the action is updated at the same time. The strategy is optimized and adjusted through reward feedback and state transfer results. Action selection, state transfer and strategy optimization are repeated until the preset target style requirements are met. , generate the corresponding image structure configuration, and input the adjusted scene into the subsequent diverse generation module to generate images that meet the target style and semantic information requirements.

[0081] The detail optimization module is used to enhance texture details and automatically adjust the color balance and contrast of the image; the diverse generation module is used to guide the generation of images of different styles and provide variants of multiple perspectives and multiple lighting conditions; the quality enhancement module is used to improve the resolution and clarity of the final image and eliminate artifacts in the generation process; the display interaction module is used to display multiple generated images and allow users to rate or select images; the recording feedback module is used to record the parameters and results of the generation process and analyze user choices and preferences.

Claims

1. A text-driven automatic image generation method, characterized in that: The specific steps of the generation method are as follows: Ⅰ. Accept and parse text data input by users, explicitly model entities and relationships in the text, and extract implicit semantic information from the text data; II. Generate a preliminary visual feature vector based on the semantic information obtained from the analysis, supplement the missing details, and infer reasonable scene extensions; III. Optimize the arrangement and matching of visual concepts, deploy the required models and dependent environments, load the image generation model, and prepare generation parameters that match the semantic information input; IV. Input the preliminary visual feature vector into the image generation model to generate an initial image, evaluate the quality of different initial images, and further optimize the local details of the initial image; V. Evaluate the scene semantic information of the initial image and adjust the object position, scale and interaction relationship in the initial image; VI. Re-input the adjusted initial image into the image generation model, generate higher quality images through cyclic optimization, and guide the generation of diversified images of different styles and perspectives; Ⅶ. Store generation results and log records, and provide a user interface for displaying generation results and for users to make fine adjustments and make further requests.

2. A text-driven automatic image generation method according to claim 1, characterized in that: The specific steps of explicitly modeling entities and relationships in the text and extracting implicit semantic information from text data as described in step I are as follows: Q1.1: Preprocess the natural language text input by the user, including word segmentation, stop word removal and grammatical analysis. Use named entity recognition technology to extract entities from the preprocessed natural language text, and use dependency syntax analysis to identify the relationship between entities. Q1.2: Map the extracted entities to the predefined knowledge graph and identify the nodes corresponding to the entities. Use the relationship extraction model to analyze the relationships in the natural language text and establish connections between entity nodes in the knowledge graph. Add new nodes and edges to the knowledge graph based on the mapped entities and relationships to construct the corresponding semantic graph. Use the constructed semantic graph to describe the core meaning of the text and its extended semantics. Q1.3: Use the BERT model to deeply encode the input text, capture syntactic, contextual, and semantic hierarchical information, and combine it with the semantic graph. Through the multi-head attention mechanism, the explicitly modeled structural information is injected into the text semantic representation. The semantic information in the semantic graph is then combined with the implicit semantic information representation extracted by the BERT model. The semantic information from different sources is unified through weighted averaging to generate the final semantic feature vector.

3. The text-driven automatic image generation method according to claim 2, characterized in that: The specific steps of generating a preliminary visual feature vector according to the semantic information obtained by parsing in step II are as follows: S2.1: Build a training dataset containing large-scale image-text pairs ,in For images, To describe the natural language text of the image, normalize and resize the images in the training dataset, generate input tensors, segment and embed the text, generate input tensors, and initialize a set of visual generation models including an image encoder and a text encoder; S2.2: Input the training dataset into the visual generation model, and the image encoder inputs the image Mapping to embedding vector , the text encoder takes the text input Mapping to embedding vector , then calculate the similarity of each pair of image-text embeddings and construct the contrastive learning loss function ,in Represents the total loss function value, represents the number of training samples, Represents the Image embedding and The similarity of text embeddings, as well as Respectively represent The embedding vectors of image and text pairs, represents the temperature parameter, which is used to adjust the smoothness of the distribution; S2.3: With the goal of minimizing the gap between the similarity of positive samples and the similarity of negative samples, the parameters of the two encoders are updated using the gradient descent optimization algorithm. After each round of training, a set of data that has not been used is randomly selected as a validation set. The model performance is evaluated on the validation set, and the accuracy of text-to-image retrieval and image-to-text retrieval is calculated. The hyperparameters and model structure are adjusted based on the evaluation results. S2.4: Repeat the training to verify the visual generation model until the contrastive learning loss function value converges to the preset range. Input the semantic information generated by text parsing and semantic graph into the pre-trained visual generation model, and generate the corresponding semantic embedding vector through the text encoder. , and then use the image encoder to generate the corresponding visual feature vector , to obtain a preliminary visual feature vector corresponding to the input semantic information; S2.5: Based on the preliminary visual features provided by the visual generation model, query the hierarchical relationship of the corresponding entity in the semantic graph, and based on the queried hierarchical relationship, expand the preliminary features based on the adjacency matrix of the semantic graph to supplement the missing details, and then re-input the expanded semantic features into the image encoder in the visual generation model to generate a complete visual feature vector.

4. A text-driven automatic image generation method according to claim 3, characterized in that: The specific steps of optimizing the arrangement and matching of visual concepts described in step III are as follows: S3.1: Model the expanded visual feature vector as a node set of image visual concepts, where each node represents a feature vector of a visual concept. The semantics and layout association between visual concepts are derived based on the semantic graph and visual rules, and the derivation result is used as the edge weight. S3.2: Initialize the pheromone concentration of each edge, generate multiple groups of optimization bodies, and randomly distribute them in the node set. Each optimization body takes the current node as the starting node, starts from the starting node, calculates the selection probability of each group of unvisited nodes, and gradually selects the next node according to the calculated selection probability until the preset arrangement length is reached; S3.3: Calculate the quality of the paths constructed by each group of optimization bodies according to the visual rationality and aesthetic indicators of the paths, globally update the pheromone concentration of the edges according to the quality of each path, and repeat the path construction and pheromone concentration update until the path quality improvement value converges to the preset range; S3.4: Arrange the order and collocation of the visual feature vectors according to the optimal path after convergence, and output the optimized visual feature vectors, and generate a scene blueprint based on the optimized visual feature vectors to describe the layout, relationship and attributes of the objects.

5. A text-driven automatic image generation system, used to implement the text-driven automatic image generation method according to any one of claims 1 to 4, characterized in that: It includes input parsing module, modeling extension module, visual generation module, model deployment module, image generation module, evaluation and screening module, scene adjustment module, detail optimization module, diverse generation module, quality enhancement module, display interaction module and recording feedback module; The input parsing module is used to receive natural language text input by a user and parse the semantic structure of the text; The modeling and extension module performs semantic modeling and extension on the parsing results; The visual generation module is used to convert the semantic representation into a preliminary visual concept, and optimize the arrangement and priority of visual elements to generate a visual blueprint that conforms to the text semantics; The model deployment module is used to deploy and manage various models and components of the generation system; The image generation module is used to generate a preliminary image based on semantic and visual concepts; The evaluation and screening module is used to evaluate the quality of the initially generated images and screen the images; The scene adjustment module is used to verify the relationship between objects and dynamically adjust the scene layout of the generated image; The detail optimization module is used to enhance texture details and automatically adjust the color balance and contrast of the image; The diverse generation module is used to guide the generation of images of different styles and provide variants of multiple viewing angles and multiple lighting conditions; The quality enhancement module is used to improve the resolution and clarity of the final image and eliminate artifacts in the generation process; The display interaction module is used to display multiple generated images and allow users to rate or select images; The recording and feedback module is used to record the parameters and results of the generation process and analyze user selections and preferences.

6. A text-driven automatic image generation system according to claim 5, characterized in that: The specific steps of the evaluation and screening module for screening images are as follows: S4.1: The generation result of an initial image and each intermediate state in the generation process are represented as a set of nodes, and the initial state of the search is represented as the root node. The edges between the nodes represent the selection from one state to the next state, and the adjustment steps of the associated generated image are then initialized. The quality estimation values ​​and the number of visits are then initialized; S4.2: Starting from the root node, the upper confidence bound (UCB) value of each child node connected to the root node is calculated. According to the node selection strategy, a child node whose UCB value meets the preset rules is selected as the next expansion node. When an incompletely expanded child node is selected, the node is expanded to generate a new child node. The new child node represents a state that has not been explored yet, corresponding to a new image generation condition. S4.3: Random simulation is performed starting from the expanded node until the image is completely generated. The image quality score of the image in the fully generated state is calculated based on visual consistency, semantic rationality, and image quality. The simulation result is then used as quality feedback to trace the quality information back to the root node along the search path, and the quality estimation values ​​and visit counts of all nodes on the path are updated at the same time. S4.4: Through multiple selections, expansions, simulations, and backtracking, the quality distribution in the tree structure is gradually optimized until a preset number of iterations is reached. A path with a quality estimate that satisfies a preset selection rule is selected from the root node to the terminal state, and the corresponding image is output.

7. The text-driven automatic image generation system according to claim 5, characterized in that: The specific steps of the scene adjustment module dynamically adjusting the scene layout of the generated image are as follows: S5.1: Constructing the state space , and each state in the state space Each state represents the structure of the current scene, including the specific configuration of the objects, layout, and style attributes in the scene, and collects the various adjustment operations that can be performed on the scene. , and construct the corresponding action space , calculated under the state Execute an action After transfer to state probability; S5.2: Calculate execution actions by combining target style, semantic information consistency, and visual aesthetic score The reward value of the post-scene adjustment effect is then used to generate the initial scene configuration based on the input text description using the semantic graph and semantic parsing results , as the starting point for adjustment, in the current state Next, select an action that satisfies the preset rules. Adjust the scene and train the acquisition strategy through Q-Learning reinforcement learning method ; S5.3: Execution of actions , according to the state transfer function Transition to a new state , to obtain the structural update of the adjusted scene, and calculate the new state according to the target style and semantic information requirements The corresponding reward is given, and the quality estimate of the action is updated at the same time, and the strategy is optimized and adjusted through reward feedback and state transfer results; S5.4: Repeat action selection, state transfer, and strategy optimization until the preset target style requirements are met. , generate the corresponding image structure configuration, and input the adjusted scene into the subsequent diverse generation module to generate images that meet the target style and semantic information requirements.

Images