Image generation method, device, electronic device and storage medium

Abstract

The present invention relates to the field of computer vision technology, and provides an image generation method, device, electronic device and storage medium, the method comprising: obtaining a description text of a target image; performing scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text; and applying the description text and the scene graph based on a multimodal image generation model to generate the target image. The image generation method, device, electronic device and storage medium provided by the present invention perform scene parsing on the description text by means of the powerful natural language processing capability of a large language model to obtain a scene graph corresponding to the description text, and generate a target image by combining the description text and the scene graph. By simultaneously referring to the text information of the description text, as well as the spatial and semantic information represented by the scene graph, the quality and semantic consistency of the generated image can be improved, thereby generating a more accurate and reasonable image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of deep learning, text-to-image technology has made significant progress. In particular, the introduction of the diffusion model makes it possible to generate high-quality images in complex scenes.

[0003] In the related art, the method of generating images from text usually relies on a text encoder to encode the text input prompt, convert the text information into a vector representation in the latent space, and then use a generative model to generate images.

[0004] However, the text described in natural language is usually highly abstract, and this method has limitations in understanding the spatial relationships and attribute features between entities, resulting in the generated images often failing to accurately reflect the user's intentions. Summary of the invention

[0005] The present invention provides an image generation method, device, electronic device and storage medium, which are used to solve the defect that the image generated in the prior art cannot accurately reflect the user's intention.

[0006] The present invention provides an image generation method, comprising:

[0007] Get the description text of the target image;

[0008] Based on a large language model, the description text is subjected to scene parsing to obtain a scene graph corresponding to the description text;

[0009] Based on a multimodal image generation model, the description text and the scene graph are applied to generate the target image.

[0010] According to the image generation method provided by the present invention, the scene parsing of the description text is performed based on a large language model to obtain a scene graph corresponding to the description text, including:

[0011] The description text is input into the large language model, the description text is structured to obtain structured text output by the large language model, and the structured text is parsed to obtain the scene graph.

[0012] According to the image generation method provided by the present invention, the parsing of the structured text to obtain the scene graph includes:

[0013] The scene graph is constructed using entities contained in the structured text and attributes of the entities as nodes and spatial relationships between the entities as edges.

[0014] According to the image generation method provided by the present invention, the multimodal image generation model is based on the description text and the scene graph to generate the target image, including:

[0015] respectively extracting text features of the description text and scene graph features of the scene graph;

[0016] The text features and the scene graph features are used as diffusion conditions of the multimodal image generation model to generate the target image.

[0017] According to the image generation method provided by the present invention, the extracting text features of the description text and scene graph features of the scene graph respectively comprises:

[0018] Extracting text features of the description text based on a text encoding model;

[0019] Based on a graph convolutional coding model, scene graph features of the scene graph are extracted.

[0020] According to the image generation method provided by the present invention, the text features and the scene graph features are used as diffusion conditions of the multimodal image generation model to generate the target image, including:

[0021] Based on the correlation between the text feature and the scene graph feature, the text feature and the scene graph feature are fused through a cross attention mechanism to obtain a fused feature;

[0022] The fusion feature is used as a diffusion condition of the multimodal image generation model to generate the target image.

[0023] According to the image generation method provided by the present invention, after generating the target image, the method further comprises:

[0024] Receive editing instructions;

[0025] Based on the large language model, the editing instruction is subjected to scene parsing, and based on the parsing result, the scene graph is updated to obtain an updated scene graph;

[0026] Based on the multimodal image generation model and the updated scene graph, an updated target image is generated.

[0027] The present invention also provides an image generating device, comprising:

[0028] A text acquisition unit, used for acquiring a description text of a target image;

[0029] A scene parsing unit, configured to perform scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text;

[0030] An image generation unit is used to generate the target image based on a multimodal image generation model by applying the description text and the scene graph.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image generation methods is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image generating method described in any one of the above is implemented.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image generating method described above is implemented.

[0034] The image generation method, device, electronic device and storage medium provided by the present invention perform scene parsing on the description text by using the powerful natural language processing capability of a large language model, obtain a scene graph corresponding to the description text, and generate a target image by combining the description text and the scene graph. At the same time, the text information of the description text and the spatial and semantic information represented by the scene graph are referred to, which can improve the quality and semantic consistency of the generated image, thereby generating a more accurate and reasonable image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is one of the flow charts of the image generating method provided by the present invention.

[0037] Figure 2 It is one of the flowchart diagrams of step 130 in the image generating method provided by the present invention.

[0038] Figure 3 This is the second flowchart diagram of step 130 in the image generating method provided by the present invention.

[0039] Figure 4This is the second flowchart of the image generating method provided by the present invention.

[0040] Figure 5 This is the third flow chart of the image generating method provided by the present invention.

[0041] Figure 6 It is a schematic diagram of a scene graph provided by the present invention.

[0042] Figure 7 It is a schematic diagram of a target image provided by the present invention.

[0043] Figure 8 It is a schematic diagram of an update scene graph provided by the present invention.

[0044] Fig. 9 is a schematic diagram of an updated target image provided by the present invention.

[0045] Fig.10 It is a structural schematic diagram of the image generating device provided by the present invention.

[0046] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In related technologies, methods for generating images from text usually rely on directly converting descriptive text into images. However, texts described in natural language are usually highly abstract and poorly structured, and have limitations in understanding the spatial relationships and attribute features between entities. As a result, the generated images often fail to accurately reflect the user's intentions, thus affecting the quality and accuracy of the generated images.

[0049] In view of the above problems, an embodiment of the present invention proposes an image generation method, in which a description text of a target image is obtained; based on a large language model, the description text is subjected to scene parsing to obtain a scene graph corresponding to the description text; based on a multimodal image generation model, the description text and the scene graph are applied to generate a target image. At the same time, the text information of the description text and the spatial and semantic information represented by the scene graph are referenced, so as to generate a more accurate and reasonable image.

[0050] The embodiments of the present invention can be applied to scenarios where images need to be generated from text, such as content creation, game design, virtual reality, advertising design, etc. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed image generation device, or an image generation device set in the electronic device, and the image generation device can be implemented by software, hardware, or a combination of the two.

[0051] Figure 1 It is one of the flowcharts of the image generation method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0052] Step 110, obtaining the description text of the target image.

[0053] Specifically, the target image is the image that the user wants to generate, and the description text of the target image refers to one or more paragraphs of text description used to guide or inspire the text-generated image technology to create the target image. The description text may include, for example, the theme and content, style, color, scene, objects and their mutual relationship of the image.

[0054] For example, the target image is expected to generate an autumn landscape picture, and the description text can contain elements related to autumn, such as fallen leaves, golden trees, and harvest fields.

[0055] Furthermore, in order to more accurately generate an image that meets the expectations, the description text may also include details such as color, light, shadow, shape and size of objects in the image.

[0056] Step 120: Based on the large language model, the description text is subjected to scene parsing to obtain a scene graph corresponding to the description text.

[0057] Specifically, considering that the related art usually directly converts the descriptive text into an image, however, the text described in natural language is usually highly abstract and poorly structured, and has limitations in understanding the spatial relationship and attribute characteristics between entities. Multiple entities in the generated image may overlap spatially, resulting in unclear user intentions; or the attributes of different entities may not be accurately matched, resulting in incorrect presentation of element features in the image; or the spatial relationship between entities is ignored, and the generated image cannot reflect the actual logical relationship.

[0058] In this embodiment, the powerful natural language processing capability of a large language model is first used to perform scene analysis on the description text to obtain a scene graph corresponding to the description text.

[0059] Large Language Model (LLM), also known as large model, refers to a natural language processing (NLP) model with a large number of parameters. The number of model parameters and / or the complexity of the model structure exceeds the preset threshold. The model processes large amounts of text data during training and has the ability to understand and generate natural language. For example, large language models can include Spark large models.

[0060] A scene graph is a graphical representation that shows the various elements mentioned in the descriptive text (such as objects, characters, scenes, etc.) and the relationships between them. As a structure that expresses objects and their relationships in an image, a scene graph can better capture the spatial and semantic information in a complex scene and form an understanding of the overall scene.

[0061] The scene graph can be a directed acyclic graph (DAG), where each node represents an object and each edge represents the relationship between objects. Each node in the scene graph has a corresponding area on the image. The relationship of the scene graph is directed, that is, there is a clear subject and object relationship.

[0062] For scene parsing of description text, we can first understand and analyze the description text through a large language model, extract key information, and then generate a scene graph corresponding to the description text based on the key information. The scene graph contains all the elements mentioned in the description text, as well as the spatial and logical relationships between them.

[0063] Step 130 , based on the multimodal image generation model, the description text and the scene graph are applied to generate a target image.

[0064] Specifically, current image generation models generally directly use description text as input to generate images. Description text is usually highly abstract and poorly structured, and has limitations in understanding the spatial relationship and attribute characteristics between entities. However, scene graphs can make full use of the various elements mentioned in the text (such as objects, people, scenes, etc.) and the relationships between them, and can better capture the spatial and semantic information in complex scenes and form an understanding of the overall scene.

[0065] A multimodal image generation model is a generative model that can process multiple types of inputs (such as text, images, etc.) and generate images. The multimodal image generation model can be trained based on a diffusion model.

[0066] Therefore, based on the multimodal image generation model, the descriptive text and scene graph are applied to generate the target image, while referring to the textual information of the descriptive text and the spatial and semantic information represented by the scene graph, so as to generate a more accurate and reasonable image.

[0067] During the specific generation, the description text and the scene graph can be input into a pre-trained multimodal image generation model, and the multimodal image generation model performs multimodal data fusion on the description text and the scene graph, and predicts the target image based on the fusion result, thereby outputting the target image. The description text and the scene graph can also be used separately for image prediction, and the predicted image obtained based on the description text and the predicted image obtained based on the scene graph are combined to determine the final target image, which is not specifically limited in the embodiment of the present invention.

[0068] The method provided by the embodiment of the present invention uses the powerful natural language processing capability of a large language model to perform scene parsing on the description text, obtain a scene graph corresponding to the description text, and generate a target image by combining the description text and the scene graph. At the same time, by referring to the text information of the description text and the spatial and semantic information represented by the scene graph, the quality and semantic consistency of the generated image can be improved, thereby generating a more accurate and reasonable image.

[0069] Based on any of the above embodiments, based on a large language model, the description text is subjected to scene parsing to obtain a scene graph corresponding to the description text, that is, step 120 specifically includes:

[0070] The description text is input into a large language model, the description text is structured to obtain structured text output by the large language model, and the structured text is parsed to obtain a scene graph.

[0071] Specifically, the description text contains enough information to construct a clear scene, including the objects in the scene, the attributes of the objects, the relationships between the objects, etc. The description text is input into the large language model, which parses and structures the description text after receiving it.

[0072] The structuring process specifically includes entity recognition, attribute extraction, relationship extraction, and text structuring. First, identify entities in the description text, such as people, objects, places, etc. Extract the attributes of the entity, such as color, shape, size, position, etc. Identify the relationship between entities, such as spatial relationship (up and down, left and right, front and back), time relationship (first and last, at the same time), logical relationship (cause and effect, conditional), etc. Represent the identified entities, attributes, and relationships in a structured manner, such as JSON, XML, etc.

[0073] After obtaining the structured text, it is necessary to parse it and generate a scene graph, which may include: using entities and attributes of entities contained in the structured text as nodes and spatial relationships between entities as edges to construct a scene graph.

[0074] Create nodes in the scene graph based on entities in the structured text, and assign entity attributes extracted from the structured text to the corresponding nodes. Create connections between nodes in the scene graph based on the relationships identified in the structured text. These connections can be directed (indicating directional relationships) or undirected (indicating non-directional relationships).

[0075] In addition, the layout of nodes and connections can be optimized based on the complexity and readability of the scene graph, including adjusting the position, size, color and other properties of nodes, and adjusting the shape, length, width and other properties of connections.

[0076] Based on any of the above embodiments, Figure 2 is one of the flow charts of step 130 in the image generation method provided by the present invention, such as Figure 2 As shown, based on the multimodal image generation model, the description text and the scene graph are applied to generate the target image, that is, step 130 specifically includes:

[0077] Step 131, extracting text features of the description text and scene graph features of the scene graph respectively;

[0078] Step 132, using text features and scene graph features as diffusion conditions of a multimodal image generation model to generate a target image.

[0079] Specifically, for text feature extraction, first use a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.) to convert the words in the description text into high-dimensional vector representations. These vectors capture the semantic relationship between words. The text embedded vector sequence is input into a model such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer to capture contextual information in the text. The output can be a fixed-length vector representing the features of the entire text, that is, the text features that describe the text. As needed, global features (such as overall topics, emotions) and local features (such as attributes and relationships of specific entities) can also be extracted from the text.

[0080] For scene graph feature extraction, the scene graph is first represented as a graph structure, where nodes represent objects and edges represent relationships between objects. Features are extracted for each node (object), including the object's category, attributes (such as color, shape), location information, etc. Features are extracted for each edge (relationship), including the type of relationship (such as "located in", "connected"), directionality, etc. Graph neural networks (such as GCN, GAT, Graph Transformer, etc.) can be used to encode the scene graph to capture the interactions between nodes and global graph structure information. The output scene graph features can be the embedding vector of each node or the embedding vector of the entire graph.

[0081] Based on the extracted text features and scene graph features, the text features and scene graph features are fused. This can be achieved through simple concatenation, weighted summation, attention mechanism, etc. The fused features will be used as the input of the multimodal image generation model.

[0082] The multimodal image generation model uses an image generation model based on the diffusion process (such as DDPM, DenoisingDiffusion Implicit Models, etc.). The model architecture can adopt the UNet architecture, and the diffusion model generates high-quality images by gradually removing noise.

[0083] The fused features are used as conditional inputs to the multimodal image generation model, guiding the model to generate target images that conform to the description text and scene graph. Conditional input can be achieved by modifying the model's noise prediction network or adding additional conditional embeddings.

[0084] Based on any of the above embodiments, step 131 specifically includes:

[0085] Based on the text encoding model, text features describing the text are extracted; based on the graph convolutional encoding model, scene graph features of the scene graph are extracted.

[0086] Specifically, the text encoding model here is used to extract text features that describe the text, and may be, for example, a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), or a Transformer.

[0087] The word vector sequence describing the text is input into the text encoding model, which can capture the sequence information and contextual dependencies in the text. The output of the text encoding model is the text feature, which can be a fixed-length vector representing the features of the entire text, or a sequence of vector representations, where each vector corresponds to a part or word in the text.

[0088] For graph scene features, it can be achieved through the Graph Convolutional Networks (GCN). The Graph Convolutional Networks (GCN) is a deep learning model specifically designed to process structured graph data. The Graph Convolutional Networks performs convolution operations on the graph to extract feature information in the graph structure, thereby achieving deep learning of graph data. This network structure can capture the complex relationships between nodes and the deep structural features of the graph.

[0089] The GCN model updates the representation of each node by aggregating the information of neighboring nodes in the scene graph, thereby capturing local and global information in the graph structure. The features of the scene graph, namely the scene graph features, are extracted from the output of the GCN model, which can include node-level features (such as the embedded representation of each node) and graph-level features (such as the embedded representation of the entire graph). Node-level features can be used to represent the properties of a single object or entity, while graph-level features can be used to represent the global information of the entire scene.

[0090] Based on any of the above embodiments, Figure 3 FIG. 2 is a second flow chart of step 130 in the image generation method provided by the present invention. Figure 3 As shown, step 132 specifically includes:

[0091] Step 132-1, based on the correlation between the text features and the scene graph features, the text features and the scene graph features are fused through a cross attention mechanism to obtain a fused feature;

[0092] Step 132-2, using the fused features as diffusion conditions of the multimodal image generation model to generate a target image.

[0093] Specifically, the fusion of text features and scene graph features can be achieved through the cross attention mechanism. The attention mechanism is used to calculate the weight of each feature and fuse the features according to the weight. The attention mechanism can dynamically adjust the weight of the feature to adapt to different input and task requirements.

[0094] Here, the correlation between text features and scene graph features can be expressed by calculating the similarity between each text feature and each scene graph feature in some way (such as dot product, cosine similarity, etc.). According to the similarity results, a weight is assigned to each element, and the higher the weight, the more relevant the element is to the elements in the query set. The weighted key-value set elements are summed to obtain the final output representation, that is, the fused feature. Through the cross-attention mechanism, the network can dynamically focus on the key information in the descriptive text and scene graph when generating images, so that the image generation process is jointly guided by multiple modal information.

[0095] Based on any of the above embodiments, Figure 4This is the second flow chart of the image generation method provided by the present invention, such as Figure 4 As shown, the method includes:

[0096] Step 410, receiving an editing instruction;

[0097] Step 420, performing scene parsing on the editing instruction based on the large language model, and updating the scene graph based on the parsing result to obtain an updated scene graph;

[0098] Step 430 , generating an updated target image based on the multimodal image generation model and the updated scene graph.

[0099] Specifically, this embodiment supports image editing functions on the basis of generating images. The system receives editing instructions input by the user. The editing instructions are used to adjust the entity relationships or attributes in the target image, for example, they may include operations such as adding, deleting, modifying the position or attributes of objects in the target image. The editing instructions may be received in the form of text input, voice input, or other forms of user interaction. In order to ensure the accuracy and comprehensibility of the instructions, the system may also provide some predefined instruction formats or templates for user reference and selection.

[0100] The system uses a large language model to perform scene parsing on editing instructions, identifying the type of operation the user wants to perform (such as adding objects, modifying properties, etc.), as well as the specific objects and targets of the operation. Based on the parsing results, the scene graph is updated accordingly to obtain an updated scene graph. For example, if the user wants to add a new object, the system can add a new node to the scene graph and update the edges related to it to reflect the new relationship. For another example, if the user wants to modify the properties or position of an object, the system can update the properties of the node or adjust its position in the scene graph.

[0101] The updated scene graph is re-encoded through the graph convolutional encoding model and input into the multimodal image generation model to generate an image that conforms to the editing instructions, that is, the updated target image, ensuring that the updated image meets user requirements.

[0102] The method provided in the embodiment of the present invention, with the help of the powerful reasoning and generation capabilities of a large language model, supports flexible editing of generated images, and users can adjust the scene according to their needs, further improving the practicality and flexibility of the application.

[0103] Based on any of the above embodiments, Figure 5 This is the third flow chart of the image generation method provided by the present invention, such as Figure 5 As shown, a method for generating an image is provided, comprising:

[0104] S1, obtain the description text of the target image. The user enters a description text (Prompt) as the basic information for generating the image. The description text will contain the entities, attributes, and relationships involved in the scene.

[0105] S2, based on a large language model, performs scene parsing on the description text to obtain a scene graph corresponding to the description text. Specifically, it includes:

[0106] The description text is input into a large language model, and the description text is structured to obtain structured text output by the large language model; the entities and attributes of the entities contained in the structured text are used as nodes, and the spatial relationships between the entities are used as edges to construct a scene graph.

[0107] S3, based on the multimodal image generation model, applies the description text and scene graph to generate the target image. Specifically includes:

[0108] S31, based on the text encoding model, extracting text features describing the text. The encoded text features are used as an input condition of the multimodal image generation model to provide contextual information of the text for the generation process, thereby ensuring that the generated image can meet the semantic requirements described by the user.

[0109] S32, based on the graph convolutional coding model, extracts scene graph features of the scene graph. The graph convolutional coding model (GCN) encodes the scene graph generated by the large language model and converts the entities and relationships in the scene graph into vectorized feature representations. Through multi-layer graph convolution operations, the network can capture the attributes of entities in the scene graph and their complex relationships with other entities, so that this information can be expressed through encoding. The scene graph is encoded using the graph convolutional coding model (GCN) to generate structured entity and relationship representations. The graph convolutional coding model can capture the relationships between entities in the scene graph, ensuring that complex spatial and attribute relationships can be clearly expressed.

[0110] S33, based on the correlation between text features and scene graph features, fuses the text features and scene graph features through a cross-attention mechanism to obtain fused features; uses the fused features as diffusion conditions of a multimodal image generation model to generate a target image.

[0111] Under the guidance of the multimodal image generation model, the system generates a target image that conforms to the description text input by the user. This target image not only reflects the content of the description text, but also contains information such as the spatial layout and entity attributes in the scene graph.

[0112] Different from the traditional text-graph diffusion model, the model in this scheme can accept both text-conditional embedding and scene-graph-conditional embedding. These embedding information ensures that when the model generates images, it not only conforms to the overall description of the text, but also can accurately present the entities and their relationships described in the scene graph.

[0113] S4, receiving the editing instruction; performing scene parsing on the editing instruction based on the large language model, and updating the scene graph based on the parsing result to obtain an updated scene graph; generating an updated target image based on the multimodal image generation model and the updated scene graph.

[0114] For example, the user input description text includes "The Tortoise and the Hare Race". Figure 6 is a schematic diagram of a scene graph provided by the present invention. The scene graph obtained by scene parsing of a large language model is as follows: Figure 6 As shown in the figure, the circles represent the four main entity nodes, the edges between the nodes are the positions and spatial relationships, and the boxes are the node attributes. Figure 7 A schematic diagram of a target image provided by the present invention, such as Figure 7 As shown, Figure 7 Modeling multimodal images from scene graphs Figure 6 The generated target image.

[0115] The user inputs the editing command "swap the positions of the turtle and the rabbit". According to the editing command, the large language model updates the scene graph and swaps the spatial positions of the two nodes of the turtle and the rabbit. Other information in the original scene graph remains unchanged. The updated scene graph is as follows: Figure 8 shown. Fig. 9 The updated target image generated by the multimodal image generation model based on the updated scene graph. Figure 7 and Fig. 9 It can be seen that Figure 7 The tortoise is in front and the hare is behind. Fig. 9 In the picture, the rabbit is in front and the turtle is behind. The positions of the turtle and the rabbit have changed.

[0116] The method provided in the embodiment of the present invention clearly describes the attributes and spatial relationships of entities through scene graph representation, avoiding the relationship confusion and position disorder of the image generated by the related technology; this embodiment uses a graph convolutional coding model to encode the scene graph and generate conditional embedding, which can effectively handle complex scenes with multiple entities and multiple relationships and enhance the accuracy and consistency of image generation. By adjusting the nodes and edges in the scene graph, this embodiment supports users to perform subsequent editing of the generated image, which has high flexibility and practicality.

[0117] The image generating device provided by the present invention is described below. The image generating device described below and the image generating method described above can be referred to each other.

[0118] Based on any of the above embodiments, Fig.10 is a schematic diagram of the structure of the image generating device provided by the present invention, such as Fig.10 As shown, the image generating device includes:

[0119] A text acquisition unit 1010 is used to acquire a description text of a target image;

[0120] A scene parsing unit 1020, configured to perform scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text;

[0121] The image generation unit 1030 is used to generate the target image based on the multimodal image generation model by applying the description text and the scene graph.

[0122] The device provided by the embodiment of the present invention uses the powerful natural language processing capability of a large language model to perform scene parsing on the description text, obtain a scene graph corresponding to the description text, and generate a target image by combining the description text and the scene graph. At the same time, the text information of the description text and the spatial and semantic information represented by the scene graph are referenced to generate a more accurate and reasonable image.

[0123] Based on any of the above embodiments, the scene parsing unit is specifically used for:

[0124] The description text is input into the large language model, the description text is structured to obtain structured text output by the large language model, and the structured text is parsed to obtain the scene graph.

[0125] Based on any of the above embodiments, the scene parsing unit is specifically used for:

[0126] The scene graph is constructed using entities contained in the structured text and attributes of the entities as nodes and spatial relationships between the entities as edges.

[0127] Based on any of the above embodiments, the image generating unit is specifically used for:

[0128] respectively extracting text features of the description text and scene graph features of the scene graph;

[0129] The text features and the scene graph features are used as diffusion conditions of the multimodal image generation model to generate the target image.

[0130] Based on any of the above embodiments, the image generating unit is specifically used for:

[0131] Extracting text features of the description text based on a text encoding model;

[0132] Based on a graph convolutional coding model, scene graph features of the scene graph are extracted.

[0133] Based on any of the above embodiments, the image generating unit is specifically used for:

[0134] Based on the correlation between the text feature and the scene graph feature, the text feature and the scene graph feature are fused through a cross attention mechanism to obtain a fused feature;

[0135] The fusion feature is used as a diffusion condition of the multimodal image generation model to generate the target image.

[0136] Based on any of the above embodiments, the device further includes an image editing unit, which is used to:

[0137] Receive editing instructions;

[0138] Based on the large language model, the editing instruction is subjected to scene parsing, and based on the parsing result, the scene graph is updated to obtain an updated scene graph;

[0139] Based on the multimodal image generation model and the updated scene graph, an updated target image is generated.

[0140] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the image generation method, which includes: obtaining a description text of a target image; performing scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text; and applying the description text and the scene graph based on a multimodal image generation model to generate the target image.

[0141] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image generation method provided by the above methods, which method includes: obtaining a description text of the target image; based on a large language model, performing scene parsing on the description text to obtain a scene graph corresponding to the description text; based on a multimodal image generation model, applying the description text and the scene graph to generate the target image.

[0143] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image generation method provided by the above-mentioned methods, the method comprising: obtaining a description text of a target image; based on a large language model, performing scene parsing on the description text to obtain a scene graph corresponding to the description text; based on a multimodal image generation model, applying the description text and the scene graph to generate the target image.

[0144] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that: include: Get the description text of the target image; Based on a large language model, the description text is subjected to scene parsing to obtain a scene graph corresponding to the description text; The scene graph shows all elements in the description text, as well as the spatial and logical relationships between all elements. The scene graph is a directed acyclic graph, in which each node represents an element and each edge represents the relationship between elements. Based on a multimodal image generation model, applying the description text and the scene graph to generate the target image; The step of performing scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text includes: The description text is input into the large language model, the description text is structured to obtain structured text output by the large language model, and the structured text is parsed to obtain the scene graph.

2. The image generation method according to claim 1, characterized in that: The parsing of the structured text to obtain the scene graph includes: The scene graph is constructed using entities contained in the structured text and attributes of the entities as nodes and relationships between the entities as edges.

3. The image generation method according to claim 1, characterized in that: The method of generating the target image based on the multimodal image generation model by applying the description text and the scene graph includes: respectively extracting text features of the description text and scene graph features of the scene graph; The text features and the scene graph features are used as diffusion conditions of the multimodal image generation model to generate the target image.

4. The image generation method according to claim 3, characterized in that: The extracting of the text features of the description text and the scene graph features of the scene graph respectively includes: Extracting text features of the description text based on a text encoding model; Based on a graph convolutional coding model, scene graph features of the scene graph are extracted.

5. The image generation method according to claim 3, characterized in that: The step of using the text features and the scene graph features as diffusion conditions of the multimodal image generation model to generate the target image includes: Based on the correlation between the text feature and the scene graph feature, the text feature and the scene graph feature are fused through a cross attention mechanism to obtain a fused feature; The fusion feature is used as a diffusion condition of the multimodal image generation model to generate the target image.

6. The image generation method according to any one of claims 1 to 5, characterized in that: After generating the target image, the method further includes: Receive editing instructions; Based on the large language model, the editing instruction is subjected to scene parsing, and based on the parsing result, the scene graph is updated to obtain an updated scene graph; Based on the multimodal image generation model and the updated scene graph, an updated target image is generated.

7. An image generating device, characterized in that: include: A text acquisition unit, used for acquiring a description text of a target image; A scene parsing unit, configured to perform scene parsing on the description text based on a large language model to obtain a scene graph corresponding to the description text; The scene graph shows all elements in the description text, as well as the spatial and logical relationships between all elements. The scene graph is a directed acyclic graph, in which each node represents an element and each edge represents the relationship between elements. An image generation unit, configured to generate the target image by applying the description text and the scene graph based on a multimodal image generation model; The scene parsing unit is specifically used for: The description text is input into the large language model, the description text is structured to obtain structured text output by the large language model, and the structured text is parsed to obtain the scene graph.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images