A knowledge graph-guided method for generating multiple scene images
The knowledge graph-guided method addresses the limitations of current text-to-image generation by using object labels to generate multiple scene images with reasonable layouts, enhancing image quality through integrated knowledge matrices.
Patent Information
- Application Number
- CN202011434422.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-10
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2040-12-10
AI Technical Summary
Current text-to-image generation methods are inconvenient due to requiring sentence-level input, often produce only one image per text, and struggle with generating complex scenes with multiple objects and inadequate information from text for high-quality images.
A knowledge graph-guided method that uses a set of object labels to generate multiple scene images by integrating layout relationships, incorporating a knowledge graph into a generative adversarial network to enhance image quality through object and global knowledge matrices.
Enables convenient multi-image generation from selectable tags, ensures reasonable layouts, and improves image quality by leveraging knowledge representation to address the limitations of existing methods.
Smart Images
Figure CN112612900B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image generation, and specifically relates to a method for generating multiple scene images guided by a knowledge graph. Background Art
[0002] The knowledge graph is a database based on triples, which store information about entities and information about the relationships between entities. In the application of knowledge graph methods, the KG2E method in the trans series is one of the classic knowledge representation methods. This method can embed entities and relationships in the graph into a high-dimensional Gaussian distribution. When training the model, KL divergence is used to make the distribution of the head entity minus the tail entity as close to the distribution of the relationship as possible. This knowledge representation method can introduce information in the graph into other models in the form of distribution, and is also the method used by the present invention when extracting graph information.
[0003] Cross-modal conversion is a classic task in multimodal learning. The generation of images from various modalities such as text and sound belongs to this field. At present, these works mainly use generative adversarial networks to achieve image generation. Generative adversarial networks are composed of two parts: generator and discriminator. The specific design is determined by the task. The generator is generally composed of a multi-layer perceptron and a deep convolutional network. It inputs a feature vector extracted from text or sound and outputs a generated image; the discriminator is composed of a shallow convolutional network. It inputs an image and outputs the true or false score of the image. In more detail, it can also output the corresponding category of the image. During the training process, the discriminator hopes to judge the generated image as a low score and the real image as a high score, so as to achieve the role of "evaluation and identification"; while the generator hopes that the generated image can be judged as a high score by the discriminator, so as to achieve the role of "making the fake look real". The generator and the discriminator are trained alternately and confront each other to ensure the quality of image generation.
[0004] Most of the current methods for synthesizing scene images from text have the following problems: (1) Current text is often a sentence, and users need to provide a sentence to generate an image in real life, which is inconvenient; (2) There should be more than one image that meets the description of the text, but most current methods can only achieve one-to-one generation tasks, and perform poorly when generating complex scene images with many objects, and cannot generate a good layout. (3) Text and images belong to different modalities, and the amount of information that text information can provide is not enough to support the generation of high-quality images. Summary of the invention
[0005] Purpose of the invention: The present invention mainly improves the shortcomings of the above-mentioned text-to-image generation method, and proposes a method for generating multiple scene images guided by a knowledge graph. The method takes labels as input, uses the knowledge graph to obtain layout relationships, realizes one-to-many generation, and adds knowledge information to the generative adversarial network to improve the image generation quality.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A method for generating multiple scene images guided by a knowledge graph, the steps comprising:
[0008] Step S1: construct a knowledge graph, extract the required triples in the form of (head entity, relationship, tail entity), and integrate them into a small knowledge graph;
[0009] Step S2: input a set of object labels into the layout search module to obtain multiple layout relationship diagrams that conform to the facts;
[0010] Further, the step S2 is specifically as follows:
[0011] Step S21: input the object label into the constructed knowledge graph to perform graph search, search for all triples containing the relationship between the input labels, and sort the searched triples according to the frequency of occurrence;
[0012] Step S22: According to the parameter settings, the required number of triplets is selected to form the most likely layout relationship diagram, and multiple other different layout relationship diagrams are generated in a random combination manner;
[0013] Step S3: Input each layout relationship diagram into the pre-trained knowledge module to obtain the object knowledge matrix and the global knowledge vector;
[0014] Further, the step S3 is specifically as follows:
[0015] Step S31: Use the classic knowledge representation method KG2E to pre-train the knowledge graph, and use different Gaussian distributions to represent the knowledge representation corresponding to all objects and relationships in the graph;
[0016] Step S32: Processing the layout relationship graph to decompose it into object labels and relationship labels within the graph;
[0017] Step S33: input the object labels and the relationship labels in the graph, sample from the pre-trained KG2E knowledge representation of objects and relationships, and obtain the object knowledge matrix and the relationship knowledge matrix;
[0018] Step S34: The object knowledge matrix and the relationship knowledge matrix are added together to form a global knowledge matrix, which is passed through a fully connected layer to generate a global knowledge vector, representing the knowledge information extracted from the graph of the entire layout relationship graph;
[0019] Step S4: adding the object knowledge matrix and the global knowledge vector to the generator;
[0020] Further, the step S4 is specifically as follows:
[0021] Step S41: Initialize and embed the decomposed in-graph object labels and in-graph relationship labels to obtain an object initial matrix and a relationship initial matrix;
[0022] Step S42: Input the initial matrix of objects and relations into a graph convolutional network with a depth of 5 layers to obtain the updated matrix of objects and relations;
[0023] Step S43: Connect the object knowledge matrix outputted from the knowledge module with the object update matrix to obtain an object prediction matrix;
[0024] Step S44: the object matrix is passed through the multi-layer perceptron 1 to generate the value of the object border position, and the multi-layer perceptron 2 is passed through to generate the object shape mask, and then through mapping combination, a scene layout tensor is generated;
[0025] Step S45: automatically expand the global knowledge vector outputted from the knowledge module to the same dimension as the image size, connect it with the scene layout tensor, input it into the cascade generation network, and generate a scene image;
[0026] Step S5: adding the object knowledge matrix and the global knowledge vector to the discriminator;
[0027] Further, the step S5 is specifically as follows:
[0028] Step S51: when different objects in the image are identified, the image slices obtained after data processing of the scene image and the object knowledge matrix are simultaneously input into the convolutional neural network 1 to obtain the true or false scores of the object image slices and the object category prediction;
[0029] Step S52: when the entire image is identified, the scene image and the global knowledge vector are simultaneously input into the convolutional neural network 2 to obtain a true or false score of the image;
[0030] Step S6: Alternately train the generator and the discriminator according to the overall loss function, while ensuring the quality of the entire image, and also ensuring the category corresponding to the composite label of the object image slice. The resulting generator is the tool for completing the generation of the layout relationship diagram to the scene image.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] (1) Unlike the text input of a sentence in most methods, the input of the present invention is a set of selectable tags, which is more convenient for users; (2) Most methods can only complete one-to-one generation tasks, while the present invention, by introducing an integrated knowledge graph, realizes the generation of multiple layout relationship graphs from a set of tags, thereby generating multiple scene images, completing a one-to-many generation task, and at the same time ensuring that each image has a reasonable layout; (3) In the generative adversarial network of the present invention, both the generator and the discriminator increase the knowledge information in the graph obtained by the knowledge representation method KG2E, increase the object knowledge matrix from the perspective of local objects, and increase the global knowledge vector from the perspective of global layout, making up for the lack of sufficient text information and improving the quality of image generation. This is also the first application of knowledge representation in the knowledge graph in the field of image generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is the overall structure designed by the present invention.
[0034] Figure 2 It is a layout search module designed by the present invention.
[0035] Figure 3 It is the knowledge module designed by the present invention.
[0036] Figure 4 It is the generator structure in the image generation module designed by the present invention.
[0037] Figure 5 It is the discriminator structure in the image generation module designed by the present invention. Specific implementation methods
[0038] The technical solution of the present invention will be further described below in conjunction with specific embodiments and drawings.
[0039] A method for generating multiple scene images guided by a knowledge graph, the steps are as follows:
[0040] Step S1: Extract all triples (head entity, relationship, tail entity) in the VG dataset, where the set of head entity and tail entity contains all label objects, and the relationship includes words such as "adjacent", "above", "behind" that can represent the layout relationship of objects. All triples are extracted and integrated into a small knowledge graph;
[0041] Step S2: Figure 2 As shown, a set of n object labels is input into the layout search module to obtain m layout relationship diagrams that conform to the facts;
[0042] Further, the step S2 is specifically as follows:
[0043] Step S21: input n object labels into the constructed knowledge graph to perform graph search, search for all triples containing the relationship between the input labels, and sort the searched triples from high to low according to the frequency of occurrence;
[0044] Step S22: according to the parameter settings, the required number of triplets is selected according to the sorting to form the most likely layout relationship diagram, which represents the layout relationship diagram where the n tags are most likely to appear, and at the same time, multiple other different layout relationship diagrams are generated in a random combination manner, so as to obtain a total of m layout relationship diagrams;
[0045] Step S3: Figure 3 As shown, the layout relationship graph is input into the knowledge module pre-trained by the knowledge graph to obtain the corresponding object knowledge matrix and global knowledge vector;
[0046] Further, the step S3 is specifically as follows:
[0047] Step S31: Use the classic knowledge representation method KG2E to pre-train the knowledge graph, so as to obtain the d-dimensional Gaussian distribution (μ i , σ i )|i=0,1,...N,and the d-dimensional Gaussian distribution (μ j , σ j )|j=0,1,…K,as the KG2E knowledge representation of objects and relations.
[0048] Step S32: Process the layout relationship graph and decompose it into object labels within the graph and relationship labels within the graph.
[0049] Step S33: Input the object labels and relationship labels in the graph, sample from the pre-trained KG2E knowledge representation of objects and relationships, and obtain the object knowledge matrix O k ∈R n×d and the relational knowledge matrix P k ∈R k×d Where n is the number of object labels in the layout relationship graph, k is the number of relationship labels in the layout relationship graph, and d is the embedding dimension of the graph knowledge representation.
[0050] Step S34: Object knowledge matrix O k ∈R n×d and the relational knowledge matrix P k ∈R k×d The global knowledge matrix S is obtained by summing up in the column direction. k ∈R 1×d , after the fully connected layer, the global knowledge vector G is generated k ∈Rd .
[0051] Step S4: Figure 4 As shown, the object knowledge matrix and the global knowledge vector are added to the generator, a layout relationship diagram is input, and a scene image is generated.
[0052] Further, the step S4 is specifically as follows:
[0053] Step S41: Initialize and embed the n in-graph object labels and k in-graph relationship labels obtained by decomposition to obtain the object initial matrix O o ∈R n×d With the initial matrix P o ∈R k×d , where d is the embedding dimension, which is consistent with the embedding dimension of the knowledge module.
[0054] Step S42: Initial matrix O of objects and relations o ∈R n×d With P o ∈R k×d , input into the graph convolutional network, and obtain the object update matrix O n ∈R n×d With the initial matrix P n ∈R k×d , where the graph convolutional network is composed of 5 layers of identical graph convolutional blocks stacked together, each of which is composed of a fully connected layer, a Relu layer, a fully connected layer, and a Relu layer in order.
[0055] Step S43: The object knowledge matrix O outputted from the knowledge module in step S3 is k ∈R n×d Update matrix O with the object n ∈R n×d Connect them together in the row direction to get the object prediction matrix O p ∈R n×2d , the knowledge information of each object is integrated into the generator.
[0056] Step S44: transform the object prediction matrix O p ∈R n×2d The value B∈R of the object border position is generated by the multi-layer perceptron 1 n×4 , after the multi-layer perceptron 2 generates the object shape mask M∈R n×s×s×d , map and combine the two, the scene layout tensor L∈R H×W×d , where multilayer perceptron 1 is composed of a fully connected layer, a Relu layer, and a fully connected layer, and multilayer perceptron 2 is composed of an upsampling layer, a BN layer, a convolution layer, and a Relu layer stacked four times in sequence. Object border position B∈R n×4In the object shape mask, n represents the number of objects in the image, and 4 represents the position values of the objects in the lower left corner, lower right corner, upper left corner, and lower right corner of the bounding box. n×s×s×d , s represents the size of the object mask, d is the embedding dimension of the object matrix input, and the scene layout tensor L∈R H×W×d In the figure, H represents the height of the scene image to be generated, and W represents the width of the scene image to be generated.
[0057] Step S45: The global knowledge vector G outputted from the knowledge module in step S3 is k ∈R d Automatically expand the dimension to the same size as the image to obtain G′ k ∈R H×W×d , and the scene layout tensor L∈R H×W×d Connect together and input into the cascade generation network to generate the scene image I∈R H×W×3 The cascade generation network consists of 5 cascade generation modules. The structure of each cascade generation module includes 8 layers in total: average pooling layer, upsampling layer, convolution layer, BN layer, Relu layer, convolution layer, BN layer, and Relu layer.
[0058] Step S5: Figure 5 As shown in Figure 2, the object knowledge matrix and the global knowledge vector are added to the discriminator.
[0059] Further, the step S5 is specifically as follows:
[0060] Step S51: When different objects in the image are identified, the scene image I∈R H×W×3 After data processing, we get the image slice C′∈R n×L×L×3 , where L is the size of the image slice, and it is combined with the object knowledge matrix O k ∈R n×d , connected and grouped according to the first dimension, each group is a binary image slice and its corresponding knowledge vector, which are input into the convolutional neural network 1 in groups to obtain the true or false score of each object image slice and the object category prediction. The structure of the convolutional neural network 1 is convolutional layer, BN layer, Relu layer, convolutional layer, BN layer, Relu layer, convolutional layer, average pooling layer, and fully connected layer.
[0061] Step S52: When the entire image is identified, the scene image I∈R H×W×3 With the global knowledge vector G k ∈R d At the same time, they are input into the convolutional neural network 2 to obtain the true or false score of the image, where the structure of the convolutional neural network 2 is convolutional layer, BN layer, Relu layer, convolutional layer, BN layer, Relu layer, and convolutional layer.
[0062] Step S6: Alternately train the generator and the discriminator to minimize the overall loss function:
[0063] L=λ1L box +λ2L pixel +λ3L GAN +λ4L img-per +λ5L obj-per
[0064] Where L box is the L1 loss between the predicted object bounding box position and the real object bounding box, L pixel is the L1 loss between the generated image and the real image, L GAN is the generation adversarial loss between the generator and the discriminator, L img-per To describe the perceptual loss between the generated image and the real image at the feature level, L obj-per is the perceptual loss of the object slices of the generated image and the object slices of the real image at the feature level, and λ1, λ2, λ3, λ4, and λ5 are hyperparameters manually set during the training process. The generator part after training is used to generate scene images from the layout relationship graph.
[0065] The generator and discriminator in the above-mentioned steps S4 and S5 are extended models based on sg2im, which are only preferred embodiments of the present invention. All equivalent changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.
Claims
1. A method for generating multiple scene images guided by a knowledge graph, characterized in that the steps include: Step S1: Extract the required triples in the form of (head entity, relationship, tail entity) and integrate them into a knowledge graph; Step S2: input a set of object labels into the layout search module to obtain multiple layout relationship diagrams that conform to the facts; Step S3: Input each layout relationship diagram into the pre-trained knowledge module to obtain the object knowledge matrix and the global knowledge vector; Step S4: adding the object knowledge matrix and the global knowledge vector to the generator; Step S5: adding the object knowledge matrix and the global knowledge vector to the discriminator; Step S6: Alternately train the generator and the discriminator according to the overall loss function, while ensuring the quality of the generation of the entire image, and also ensuring that the object image slices conform to the category corresponding to the label; the resulting generator is a tool for completing the generation of the layout relationship diagram to the scene image; The step S2 is specifically as follows: Step S21: Input a set of object labels into the knowledge graph constructed in step S1 to perform graph search, search for all triples containing the relationship between the input labels, and sort the searched triples according to the frequency of occurrence; Step S22: According to the parameter settings, the required number of triplets is selected to form the most likely layout relationship diagram, and multiple other different layout relationship diagrams are generated in a random combination manner; The step S3 is specifically as follows: Step S31: Use the knowledge representation method KG2E to pre-train the knowledge graph constructed in step S1, and use different Gaussian distributions to represent the knowledge representation corresponding to all objects and relationships in the graph; Step S32: performing data processing on the layout relationship graph obtained in step S2, and decomposing it into object labels within the graph and relationship labels within the graph; Step S33: input the object labels and the relationship labels in the graph, sample from the pre-trained KG2E knowledge representation of objects and relationships, and obtain the object knowledge matrix and the relationship knowledge matrix; Step S34: The sum of the object knowledge matrix and the relationship knowledge matrix is called the global knowledge matrix, and a global knowledge vector is generated through a fully connected layer, which represents the knowledge information extracted from the graph of the entire layout relationship graph.
2. The method for generating multiple scene images guided by a knowledge graph according to claim 1, characterized in that: The step S4 is specifically as follows: Step S41: Initialize and embed the decomposed in-graph object labels and in-graph relationship labels to obtain an object initial matrix and a relationship initial matrix; Step S42: Input the initial matrices of objects and relationships into a 5-layer deep graph convolutional network to obtain an object update matrix and a relationship update matrix respectively; Step S43: Connect the object knowledge matrix outputted from the knowledge module with the object update matrix to obtain an object prediction matrix; Step S44: the object prediction matrix generates the value of the object border position through the multi-layer perceptron 1, generates the object shape mask through the multi-layer perceptron 2, and the object border position and the object shape mask are mapped and combined to generate a scene layout tensor; Step S45: The global knowledge vector output from the knowledge module is automatically expanded to the same dimension as the image size, connected to the scene layout tensor, and input into the cascade generation network to generate a scene image.
3. The method for generating multiple scene images guided by a knowledge graph according to claim 2, characterized in that: The step S5 is specifically as follows: Step S51: when different objects in the image are identified, the object image slices obtained after data processing of the scene image and the object knowledge matrix output by the knowledge module are simultaneously input into the convolutional neural network 1 to obtain the true or false score of the object image slice and the object category prediction; Step S52: When the entire image is identified, the scene image and the global knowledge vector output by the knowledge module are simultaneously input into the convolutional neural network 2 to obtain a true or false score of the image.
4. The method for generating multiple scene images guided by a knowledge graph according to claim 2, characterized in that: Multilayer perceptron 1 is composed of a fully connected layer, a Relu layer, and a fully connected layer. Multilayer perceptron 2 is composed of an upsampling layer, a BN layer, a convolutional layer, and a Relu layer stacked four times in sequence.
5. The method for generating multiple scene images guided by a knowledge graph according to claim 3, characterized in that: The structure of convolutional neural network 1 is convolutional layer, BN layer, Relu layer, convolutional layer, BN layer, Relu layer, convolutional layer, average pooling layer, and fully connected layer; the structure of convolutional neural network 2 is convolutional layer, BN layer, Relu layer, convolutional layer, BN layer, Relu layer, and convolutional layer.