A method for generating image scene graph based on knowledge graph

Abstract

The present invention discloses a method for generating an image scene graph based on a knowledge graph. The present invention first uses an OPENIE model to convert an image description into a triple, and then into an entity and an entity correspondence, and imports it into NEO4J to construct a knowledge graph of COCO and VisualGenome data sets; secondly, the model combines the knowledge graph to perform target detection on the input image, and identifies the corresponding object category and object attribute, as well as the relationship between the candidate boxes with intersections. The obtained object attributes are placed in the knowledge graph as retrieval key points, and the NEO4J database is called to obtain the relationship node with the label weight and the corresponding other entity node; similarly, the relationship attribute features are placed in the knowledge graph to obtain a node attribute group; the specified number of triples obtained through the knowledge graph are placed in an arrangement module; and placed in a scene graph generation network, and finally a scene graph based on the knowledge graph is generated. The present invention proposes an optimization scheme for the problems of image target recognition and relationship detection.

Description

Technical Field

[0001] The invention of this article relates to a scene graph generation method, specifically, an image scene graph generation method based on a knowledge graph, which belongs to the technical field of scene graph generation. Background Art

[0002] Scene graphs are usually used to describe the objects in an image, as well as the corresponding category attributes and relationships between objects. They are often used in image retrieval, image generation, video motion capture, and special relationship detection. As a semantic graph structure, the image information contained in scene graphs is more intuitive and richer than vector representation.

[0003] On the other hand, as a graph-based data structure, the knowledge graph stores knowledge in a graph format and returns processed and reasoned knowledge to objects. It consists of nodes and edges. Nodes are used to represent entities that appear in the real world, while edges are used to represent the relationships between these entities.

[0004] Based on the similarity of the data storage structure between the two and the strong semantics of the knowledge graph itself, this paper proposes a scene graph generation method based on the knowledge graph, which uses the rich semantic relationships contained in the knowledge graph to optimize the target detection and object relationship recognition in the scene graph generation process. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose an image scene graph generation method based on a knowledge graph, so as to utilize the knowledge graph containing rich semantic knowledge as prior knowledge to achieve the advantages of target detection and relationship prediction in the scene graph generation process.

[0006] In order to implement this solution, based on the traditional scene graph generation method, the present invention combines the knowledge graph generated using the COCO dataset and the Visual Genome dataset with saliency detection, and proposes a scene graph generation method based on knowledge graph.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] Step (1) Use the OPENIE model to convert the image description (txt format) into triples (json format), and then into entities (csv format) and entity correspondences (csv format), import them into NEO4J, and build the knowledge graph of COCO and Visual Genome datasets;

[0009] Step (2) uses Faster R-CNN target detection to obtain candidate boxes for the input image, obtains the target feature Obj_feature in each candidate box, and predicts the object attribute Obj_attr and the relationship feature Rel_fearure between each candidate box, and predicts the relationship attribute Rel_attr.

[0010] Step (3) puts the obtained object attribute Obj_attr into the knowledge graph as a retrieval key point, and calls the NEO4J database to obtain the relationship node with the marked weight and the corresponding other entity node.

[0011] Step (4) puts the obtained relationship attribute Rel_attr into the knowledge graph as a retrieval key point. Similarly, the NEO4J database is called to find the relevant node attribute group through the relationship attribute.

[0012] Step (5) puts the specified number of triplets obtained through the knowledge graph into the permutation module.

[0013] Step (6) puts the triplet information that has passed the arrangement module into the scene graph generation network, and finally generates a scene graph based on the knowledge graph.

[0014] The beneficial effects of the present invention are as follows:

[0015] The present invention uses the rich semantic relationships contained in the knowledge graph to optimize the object detection and object relationship recognition in the scene graph generation process. Further optimization is performed for the problems of image object recognition and relationship detection.

[0016] The present invention is based on the currently popular knowledge graph as prior information to achieve better results in target recognition and relationship detection.

[0017] The method proposed in the present invention is a prediction method, which can be widely used in downstream tasks based on scene graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the overall implementation scheme of the present invention;

[0019] Figure 2 It is a schematic diagram of knowledge graph construction generation of the present invention;

[0020] Figure 3 It is a schematic diagram of generating characteristic triples of the present invention;

[0021] Figure 4 It is a schematic diagram of generating feature triples from entity features of the present invention;

[0022] Figure 5It is a schematic diagram of generating feature triples from relational features of the present invention;

[0023] Figure 6 is a schematic diagram of generating the arrangement module of the present invention;

[0024] Figure 7 It is a schematic diagram of scene graph generation based on knowledge graph of the present invention;

[0025] Figure 8 It is a schematic diagram of the overall structure of the present invention; DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings.

[0027] Reference Figure 1 and 8 Shown is a flow chart of the overall implementation scheme of the present invention. The present invention first uses the OPENIE information extraction annotator to extract the image descriptions (captions) in the COCO dataset and the Visual Genome dataset into triplets (json format), and converts the corresponding triplets (json format) into files containing entities in the triplets (csv format) and files containing relationships in the triplets (csv format) according to specific input requirements, and integrates the two into the NEO4J open source graph database to construct knowledge graphs containing COCO image descriptions and Visual Genome image descriptions, respectively. Secondly, the model combines the knowledge graph to perform target detection on the input image, and identifies the corresponding object category (obj) and object attributes (attribute), as well as the relationship (relationship) between the intersections of two object boxes. The object attribute features are placed in the knowledge graph, and the knowledge graph uses the attributes as the entities as the retrieval points to obtain the relationship features (k relationship) corresponding to the entities in the database and return them to the scene graph generation model. Similarly, the relationship features are placed in the knowledge graph, and the knowledge graph uses the relationships (k relationship) as the retrieval points to obtain the relationships (entity) in the database and return them to the scene graph generation model. Then, the triplets (obj relationship sub) obtained by the two methods are compared and screened. By analyzing the weight values ​​and the triplet loss (Triplet loss) analogous to that in small samples, the most suitable triplet is found as the input of objects and relationships in the scene graph generation model, and finally, an image scene graph based on the knowledge graph is generated.

[0028] A method for generating an image scene graph based on a knowledge graph comprises the following steps:

[0029] Step (1) Use the OPENIE model to convert the image description (txt format) into triples (json format), and then into entities (csv format) and entity correspondences (csv format), import them into NEO4J, and build the knowledge graph of COCO and Visual Genome datasets;

[0030] Step (2) uses Faster R-CNN target detection to obtain candidate boxes for the input image, obtains the target feature Obj_feature in each candidate box, and predicts the object attribute Obj_attr and the relationship feature Rel_fearure between each candidate box, and predicts the relationship attribute Rel_attr.

[0031] Step (3) puts the obtained object attribute Obj_attr into the knowledge graph as a retrieval key point, and calls the NE04J database to obtain the relationship node with the marked weight and the corresponding other entity node.

[0032] Step (4) puts the obtained relationship attribute Rel_attr into the knowledge graph as a retrieval key point. Similarly, the NE04J database is called to find the relevant node attribute group through the relationship attribute.

[0033] Step (5) puts the specified number of triplets obtained through the knowledge graph into the permutation module.

[0034] Step (6) puts the triplet information that has passed the arrangement module into the scene graph generation network, and finally generates a scene graph based on the knowledge graph.

[0035] Furthermore, the specific implementation process of step (1) is as follows: Figure 2 As shown:

[0036] 2-1 Save the data in the COCO dataset and the Visual Genome dataset in txt format, and use the OPENIE model to convert the corresponding files into triplets (json format).

[0037] 2-2 And use Python code to convert it into an entity.csv file containing (entity: ID, name:, LABEL), and based on the triple information and entity: ID, get the role.csv file containing (:START_ID, :END_ID, :TYPE).

[0038] 2-3 Import the two csv files into the Neo4j database to obtain a visual knowledge graph information library.

[0039] Furthermore, the specific implementation process of step (2) is as follows: Figure 3 As shown:

[0040] 3-1 Scale the P*Q input image of any size to a fixed size of M*N, and use Faster R-CNN pre-trained on the COCO dataset to detect the target. Faster R-CNN will perform a 3*3 convolution on the image to obtain the candidate region, including the candidate box set B = {b i |i=1,…,n} and positive anchor PA, as shown in formula (1) (2):

[0041] (M, N) = Re (P, Q, Scale) (1)

[0042] (B, PA) = Faster RCNN (M, N) (2)

[0043] Among them, Scale is the scaling factor.

[0044] 3-2 Get the candidate box set B = {b i |i=1,…,n}, extract the feature F, and obtain the attribute Obj_attr corresponding to the candidate box through the bidirectional LSTM, as shown in formula (3):

[0045] Obj_attr = DoubleLSTM(F) (3)

[0046] Where F is the feature of the corresponding candidate box obtained by the convolutional neural network method;

[0047] 3-3 The object attribute Obj_attr and the subject and object regional features U are input into the predicate context to obtain the relation attribute Rel_attr, as shown in formula (4) (5):

[0048] U=Union(b i , b j )(1≤i,j≤n) (4)

[0049] Rel_attr=Context(U, Obj_attr) (5)

[0050] Among them, b i , b j The candidate box set B = {b i |i=1,…,n}, n is the i-th and j-th candidate region boxes, and U is the region feature obtained by two candidate region boxes.

[0051] Furthermore, the specific implementation process of step (3) is as follows: Figure 4 As shown:

[0052] 4-1 Put the object attribute Obj_attr into the knowledge graph for retrieval. For each object attribute Obj_attr, obtain N1 groups of different corresponding relationship attributes Rel_attr and object attributes Sub_attr. The relationship attribute Rel_attr and the object attribute Sub_attr are taken as a group. The search results are in reverse order as the weight value, and the initial object attribute Obj_attr is combined as a screening triple (Obj_attr, Rel_attrI, Sub_attr I ) set, as shown in formula (6):

[0053] Sub_attr I , Rel_attr I =Query(Obj_attr,N1)0≤I≤N1 (6)

[0054] Among them, N1 is the maximum number of triples searched, which is set to 7.

[0055] Furthermore, the specific implementation process of step (4) is as follows: Figure 5 As shown:

[0056] 5-1 Put the relation attribute Rel_attr into the knowledge graph for retrieval. For each relation attribute Rel_attr, obtain N2 groups of corresponding object attributes Obj_attr and object attributes Sub_attr. The object attribute Obj_attr and the object attribute Sub_attr are taken as a group. The search results are in reverse order as the weight value, and the initial Rel_attr is combined as the screening triple (Obj_attr I , Rel_attr, Sub_attr I ) set, as shown in formula (7):

[0057] Sub_attr I , Obj_attr I =Query(Rel_attr,N2)0≤I≤N2 (7)

[0058] Among them, N2 is the maximum number of triples searched, which is set to 7.

[0059] Furthermore, the specific implementation process of step (5) is as follows: Figure 6 As shown:

[0060] 6-1 The triplet obtained from both sides (Obj_attr, Rel_attr I , Sub_attr I ) and (Obj_attr I , Rel_attr, Sub_attr I ), calculate any Ti obtained by the object attribute Obj_attr and Tj obtained by the relationship attribute Rel_attr, as shown in formulas (8) and (9); in order to prevent the combination of Ti and Tj from disturbing the original weight comparison, it is necessary to normalize Ti and Tj to obtain Ti′ and Tj′, as shown in (10) and (11):

[0061]

[0062]

[0063]

[0064]

[0065] Among them, β and γ are weight parameters, ε is a small decimal to prevent division by 0, μ and σ 2 are the mathematical expectation and variance of the normal distribution respectively.

[0066] 6-2 requires a set of parameters to ensure that the normalization operation does not destroy the previous information. Finally, the two are combined and the maximum value is found to obtain the final T, as shown in formulas (12) and (13):

[0067] T′=Ti′⊙Tj' (12)

[0068] T=Max(T′*g°+b)1≤i,j≤n (13)

[0069] Among them, the gain g° is used as the gain and b is used as the bias; T′ represents the sum of Ti′ and Tj′ after normalization.

[0070] Further, the specific implementation process of step (6) is as follows: Figure 7 As shown:

[0071] 7-1 uses cross entropy loss to optimize the scene graph generation network, setting e′ and l′. Represent the predicted labels of entity (Obj, Sub) and predicate (Rel), respectively, and e and l represent the basic truth value (groundtruth) of entity (Obj, Sub) and predicate (Rel), as shown in formula (14):

[0072]

[0073] Among them, Z1 and Z2 are weight parameters

[0074] 7-2 Input the triplet information into the scene graph generation model for end-to-end training, with the epoch set to 10, the learning rate to 0.00001, and the batch size to 16 to obtain the scene graph.

[0075] The scene graph generation method based on this patent invention is compared with the current popular scene graph generation method. The comparison results are shown in Table 1:

[0076]

[0077] Among them, VRD and IMP are the names of popular models for scene graph generation, R@k (k = 50, 100) represents Recall@k, and P@k (k = 50, 100) represents Precision@k, which are evaluation indicators for scene graph generation. The higher the indicator value, the better the generation effect. As can be seen from the table, this patent has a certain degree of improvement in these indicators compared with other inventions, indicating that the scene graph generation method based on knowledge graph is effective in improving scene graph generation.

Claims

1. A method for generating image scene graph based on knowledge graph, characterized in that The following steps are involved: Step (1) Use the OPENIE model to convert image descriptions into triples, and then into entities and entity correspondences, import them into NEO4J, and build knowledge graphs for COCO and Visual Genome datasets; Step (2) Use Faster R-CNN target detection to obtain candidate boxes for the input image, obtain the target feature Obj_feature in each candidate box, and predict the object attribute Obj_attr and the relationship feature Rel_fearure between each candidate box, and predict the relationship attribute Rel_attr; Step (3) puts the obtained object attribute Obj_attr into the knowledge graph as a retrieval key point, and calls the NEO4J database to obtain the relationship node with the marked weight and the corresponding other entity node; Step (4) put the obtained relationship attribute Rel_attr into the knowledge graph as a retrieval key point, and similarly call the NEO4J database to find the relevant node attribute group through the relationship attribute; Step (5) putting a specified number of triples obtained through the knowledge graph into the permutation module; Step (6) puts the triple information after the arrangement module into the scene graph generation network, and finally generates a scene graph based on the knowledge graph; The specific implementation process of step (6) is as follows: 7-1 uses cross entropy loss to optimize the scene graph generation network, setting e′ and l′ to represent the predicted labels of entities and predicates respectively, and e and l to represent the basic truth values ​​of entities and predicates, as shown in formula (14): Among them, Z1 and Z2 are weight parameters; 7-2 Input the triplet information into the scene graph generation model for end-to-end training, with the epoch set to 10, the learning rate to 0.00001, and the batch size to 16 to obtain the scene graph.

2. According to claim 1, a method for generating an image scene graph based on a knowledge graph is characterized in that The specific implementation process of step (1) is as follows: 2-1 Save the data in the COCO dataset and the Visual Genome dataset in txt format, and use the OPENIE model to convert the corresponding files into triplets; 2-2 And use Python code to convert it into an entity.csv file containing (entity: ID, name:, LABEL), and according to the triple information and entity: ID, get the role.csv file, containing (:START_ID,:END_ID,:TYPE); 2-3 Import the two csv files into the NEO4J database to obtain a visual knowledge graph information library.

3. According to claim 2, a method for generating an image scene graph based on a knowledge graph is characterized in that Step (2) is specifically implemented as follows: 3-1 Scale the P*Q input image of any size to a fixed size of M*N, and use Faster R-CNN pre-trained on the COCO dataset to detect the target. Faster R-CNN will perform a 3*3 convolution on the image to obtain the candidate region, including the candidate box set B = {b i |i=1,…,n} and positive anchor PA, as shown in formula (1) (2): (M, N) = Re (P, Q, Scale) (1) (B, PA) = Faster RCNN (M, N) (2) Among them, Scale is the scaling multiple; 3-2 Get the candidate box set B = {b i |i=1,…,n}, extract the feature F, and obtain the attribute Obj_attr corresponding to the candidate box through the bidirectional LSTM, as shown in formula (3): Obj_attr = DoubleLSTM(F) (3) Where F is the feature of the corresponding candidate box obtained by the convolutional neural network method; 3-3 The object attribute Obj_attr and the subject and object regional features U are input into the predicate context predicatecontext to obtain the relation attribute Rel_attr, as shown in formula (4) (5): U=Union(b i ,b j ) (4) Rel_attr=Context(U, Obj_attr) (5) Among them, b i , b j The candidate box set B = {b i |i=1,…,n}, n is the i-th and j-th candidate region boxes, and U is the region feature obtained by two candidate region boxes.

4. The method for generating an image scene graph based on a knowledge graph according to claim 3, characterized in that The specific implementation process of step (3) is as follows: 4-1 Put the object attribute Obj_attr into the knowledge graph for retrieval. For each object attribute Obj_attr, obtain N1 groups of different corresponding relationship attributes Rel_attr and object attributes Sub_attr. The relationship attribute Rel_attr and the object attribute Sub_attr are taken as a group. The search results are in reverse order as the weight value, and the initial object attribute Obj_attr is combined as a screening triple (Obj_attr, Rel_attr I , Sub_attr I ) set, as shown in formula (6): Sub_attr I ,Rel_attr I =Query(Obj_attr,N1)0≤I≤N1 (6) Among them, N1 is the maximum number of triples searched, which is set to 7.

5. The method for generating an image scene graph based on a knowledge graph according to claim 4, characterized in that The specific implementation process of step (4) is as follows: 5-1 Put the relation attribute Rel_attr into the knowledge graph for retrieval. For each relation attribute Rel_attr, obtain N2 groups of corresponding object attributes Obj_attr and object attributes Sub_attr. The object attribute Obj_attr and the object attribute Sub_attr are taken as a group. The search results are in reverse order as the weight value, and the initial Rel_attr is combined as the screening triple (Obj_attr I , Rel_attr, Sub_attr I ) set, as shown in formula (7): Sub_attr I ,Obj_attrI=Query(Rel_attr,N2)0≤I≤N2 (7) Among them, N2 is the maximum number of triples searched, which is set to 7.

6. The method for generating an image scene graph based on a knowledge graph according to claim 5, characterized in that The specific implementation process of step (5) is as follows: 6-1 The triplet obtained from both sides (Obj_attr, Rel_attr I , Sub_attr I ) and (Obj_attr I , Rel_attr, Sub_attr I ), calculate any Ti obtained by the object attribute Obj_attr and Tj obtained by the relationship attribute Rel_attr, as shown in formulas (8) and (9); in order to prevent the combination of Ti and Tj from disturbing the original weight comparison, it is necessary to normalize Ti and Tj to obtain Ti′ and Tj′, as shown in formulas (10) and (11): Among them, β and γ are weight parameters, ε is a small decimal to prevent division by 0, μ and σ 2 are the mathematical expectation and variance of the normal distribution respectively; 6-2 requires a set of parameters to ensure that the normalization operation does not destroy the previous information. Finally, the two are combined and the maximum value is found to obtain the final T, as shown in formula (12) (13): T′=Ti′☉Tj' (12) T=Max(T′*g°+b)1≤i,j≤n (13) Among them, the gain g° is used as the gain and b is used as the bias; T′ represents the sum of Ti′ and Tj′ after normalization.

Images