Scene graph generation method and control system based on core knowledge reasoning

Abstract

The invention discloses a scene graph generation method based on core knowledge reasoning, and belongs to the technical field of deep learning. The method aims to solve the problem that in the prior art, excessive prediction still occurs when many meaningless labels or labels with little knowledge expression are predicted. According to the method, in the target detection process, a target bounding box and a scene layout in an image are extracted in a data set, and a rough entity label is obtained; then the two are respectively sent to core entity reasoning and core relation reasoning for knowledge extraction; the visual part carries out mining on visual information based on appearance and space in a scene graph, and subjects and guests in an image are extracted from coarse to fine so as to obtain a core subject; according to the common sense part, the subject-object pair is reconnected and then matched with the common sense in the context feature space, and the core relation of the subject-object pair is obtained. The method has the advantages that according to the core knowledge reasoning method, through a reasonable depolarization strategy, potential core knowledge is mined, so that the model generates a scene graph which is more accurate and rich in relation.

Description

Technical Field

[0001] The present invention relates to a scene graph generation method and a control system based on core knowledge reasoning, and belongs to the technical field of deep learning. Background Art

[0002] As a complex visual understanding task, scene graph generation has shown excellent results in tasks such as visual question answering, image retrieval, visual relationship detection, and image captioning. It separates entities and the relationships between entity pairs by modeling the image as a graph. Entities (including subjects and objects) are represented by nodes, and relationships are represented by directed edges, which together form triples. However, since most scene graphs contain complex scenes or diverse relationships, they cause perceptual defects in machine vision and seriously affect the reasoning ability of the model. Based on this, the model can only output relatively few entities or simple relationships, thereby ignoring a large amount of meaningful scene knowledge.

[0003] Knowledge reasoning, as the most important research topic in artificial intelligence at this stage, can realize the ability to integrate large amounts of scattered data into structured knowledge. Figure 1 It is generally modeled as a triple, which coincides with the form of scene graph knowledge extraction. In the process of knowledge engineering research, various forms of knowledge reasoning have been generated, such as common sense knowledge reasoning, language knowledge, etc. Experts have also summarized some knowledge bases, such as ConceptNet, Google Knowledge Graph, and YAGO. Most of these common knowledge representation forms have been tried by developers to introduce into SGG to guide model prediction.

[0004] At present, knowledge graphs have become the mainstay of intelligent recommendation systems, relationship extraction, question answering and other fields. The advantages of knowledge graphs in various fields attract researchers to try to explore their potential in scene graphs. Precisely because knowledge graphs have the same distribution and analysis methods as scene graphs, they are often used to assist scene graph generation tasks. However, although the triple form in knowledge graphs can analyze unstructured knowledge distribution to form structured representations, the knowledge in scene graphs is always structured and scene-specific. This often makes scene graphs that focus on knowledge graph research overly dependent on knowledge distribution and ignore visual information. Therefore, in order to fully explore the knowledge system in scene graphs, we focus on studying the intrinsic knowledge structure of scene graphs, including visual knowledge and common sense knowledge. This intrinsic visual and common sense feature together constitutes the "core knowledge" that we care about.

[0005] Previous work often predicts many meaningless (single entity) or poorly expressed knowledge (simple relationships) labels, and sometimes even over-predicts and produces many erroneous results. This is because they focus too simply on the visual information of the scene graph or the application of external knowledge. It can be seen that building an effective knowledge mining system to obtain comprehensive core knowledge is a key step in deeply understanding visual scenes. Summary of the invention

[0006] In response to the problems existing in the prior art, this application proposes a scene graph generation method and control system based on core knowledge reasoning, which involves an implementation of applying core knowledge in cognitive knowledge to scene graph generation in the field of deep learning. The solution of this application comprehensively considers the rich semantic features extracted by the model and the stable scene information extracted during the prediction process to train the deep model, which can effectively improve the core semantic representation and greatly improve the accuracy of the prediction; and the solution of this application effectively alleviates the problems of insufficient meaningful entity mining and missing long-tail relationship prediction.

[0007] The technical solution adopted by the present invention is a scene graph generation method based on core knowledge reasoning, comprising the following steps:

[0008] 1) Input the dataset into the pre-trained deep model to extract scene layout and entity labels;

[0009] 2) The extracted layout and visual images are fed into the core entity reasoning module, which performs global detection on the scene graph and mines entity nodes;

[0010] The input of the deep model is the data set, which is input into the model in the form of pictures. The pre-trained deep model is a commonly used model for target detection, which extracts the spatial position and entity label of the target in the picture. The visual picture mentioned here refers to the visual appearance features of the picture. The pure visual appearance combined with the spatial position information (layout) is sent to the core entity reasoning module to help the model quickly locate and extract the detailed features of the corresponding entity.

[0011] The core entity reasoning module obtains the core entity position P e , core entity feature label N e , the corresponding class score C;

[0012] 3) The core relational reasoning module combines common sense entities in an orderly manner to generate different subject-object pairs;

[0013] The joint entity passes through the subject attention head channel and the object attention head channel;

[0014] The combined entity is to combine the acquired entity features in the form of "subject-object" or "object-subject" to verify the semantic information of a certain feature when it is a subject or an object;

[0015] After locating the entities, the initial representations of the entities and predicates are calculated at the relational awareness layer according to their different attention scores;

[0016] The knowledge extraction layer further removes redundant targets, predicates, or triples;

[0017] The knowledge extraction layer calculates the cosine similarity between the subject and the object and outputs a relational metric matrix;

[0018] The knowledge extraction layer is used to filter the predicted predicate relations when extracting entities in the form of subjects or objects. It compares the corresponding predicate relationship distribution probability perceived by the relationship perception layer with the co-occurrence frequency of subject-object pairs in the data set obtained in the target extraction, thereby outputting more valuable visual entity features.

[0019] 4) The Bayesian inference process integrates the obtained core entities and core relations;

[0020] First, standardize the core entity features and core relationship features and map them into the same space;

[0021] Calculate the set of subjects, objects, and relationships in public space;

[0022] In this public domain, core knowledge is used as priors for refined reasoning;

[0023] Compute the maximum a posteriori distribution of subjects, relations, and objects.

[0024] Optimized, in the above scene graph generation method based on core knowledge reasoning, in step 1), the deep model is a pre-trained target recognition model ResNeXt-101-FPN, and the data set is input into the target detector for pre-training on the ImageNet data set, and the training data sets are all image data sets;

[0025] In step 1), the core entity reasoning module and the core relationship reasoning module are first trained separately on the target dataset Visual Genome dataset, and then jointly trained with the Bayesian relationship reasoning module.

[0026] Optimized, in the above scene graph generation method based on core knowledge reasoning, in step 2), the core knowledge is extracted by arranging and screening the samples with the highest confidence in the entities and relationships, and the core knowledge extraction process includes:

[0027] Input the original image into the pre-trained object detection model to obtain the annotation box and instance features;

[0028] The extracted basic image features are sent to the core entity reasoning module and the core relationship reasoning module to make a more detailed prediction of the sample probability;

[0029] The entity features and relationship features with high confidence are output as core knowledge after knowledge fusion.

[0030] Optimized, in the above scene graph generation method based on core knowledge reasoning, in step 3) and step 4), a cross entropy loss function is used and a priori factor λ is used for restriction;

[0031] The larger the λ, the more the model trusts the results of visual reasoning, and the smaller the λ, the more the model trusts the inherent prior knowledge inference.

[0032] Optimized, the scene graph generation method based on core knowledge reasoning, in step 2), P e , N e , C = FFN(Decoder(Q e , F e ));

[0033] Assume that undetected entities or relations are detected using Filled, the visual loss after core entity reasoning is L vis Defined as:

[0034]

[0035] in, represents the indicator function, L cls represents the cross entropy loss of all predictions, L box It consists of L1 loss and GIOU loss and is used to match successful predictions.

[0036] Optimized, in the above scene graph generation method based on core knowledge reasoning, in step 3), the knowledge extraction layer calculates the cosine similarity between the subject and the object to output a relationship measurement matrix that conforms to common sense:

[0037] M = cos_sim(sub, obj);

[0038] Where sub is the embedded representation of the subject node in the triple obtained by the knowledge extraction layer, and obj is the embedded representation of the object node in the triple obtained by the knowledge extraction layer;

[0039] The cross entropy loss based on the common sense relationship measurement matrix is ​​used to measure the cognitive semantic consistency between the generated scene graph and the real scene graph obtained in the core relationship reasoning stage; the cross entropy loss based on the common sense relationship measurement matrix is:

[0040]

[0041] in, and Defined as the cross entropy loss between the true label and the prediction, η obj and η rel is the weight parameter;

[0042] in e i and r ij Represent the entity and relationship labels under the real labels, respectively, and and They represent the entity and relationship labels predicted by the core relationship layer respectively.

[0043] Optimized, in the above scene graph generation method based on core knowledge reasoning, in step 4), the conditional distribution based on Bayes is:

[0044]

[0045] Where P I，δ (S), P I，δ (O) and P I，δ (R) is the probability of measurement observed from image I by the model trained with the parameters;

[0046] The maximum posterior distribution of subject S', relation R', and object O' is expressed as

[0047]

[0048] The cross entropy loss for the prediction of core entities is limited to

[0049]

[0050] The cross entropy loss for the prediction of core relationships is limited to

[0051]

[0052] Among them, e i represents the entity label under the groundtruth label, r ij They represent the relationship labels under the groundtruth labels, e′ i represents the predicted label of the final core entity obtained through Bayesian inference in the public domain, r′ ij Represents the predicted labels of the final core relations obtained through Bayesian inference in the public domain.

[0053] The beneficial effects of this application are:

[0054] In this application, the object detector Faster-RCNN is first used for pre-training to obtain a set of bounding boxes and corresponding feature maps to encode objects or object pairs in the image; in the core entity reasoning stage, the roles of different entities are associated to enhance the visual context information between entities; in the core relationship reasoning stage, the network re-encodes the entity order, jointly detects and recognizes objects and their relationships; finally, this prior information is used to predict predicates to complete the construction and reasoning of the graph.

[0055] Compared with these methods, the technical solution of the present application can make full use of the core visual information of the scene graph and combine it with common sense knowledge for knowledge modeling. The fusion of core knowledge can try to cross the inherent threshold of the scene graph to generate an accurate and semantically rich scene graph.

[0056] The technical solution of the present application proposes to construct a semantically rich and common-sense knowledge representation, namely core knowledge, in the entity and predicate prediction process, and uses the core knowledge as guiding information to assign fine-grained labels to the scene graph model to train a proprietary classifier, so that the model has the ability to predict complex scenes and dynamically changing environments; and according to the mechanism of core knowledge, a unique scene graph generation method based on core knowledge reasoning is proposed, thereby improving the under-prediction and long-tail distribution problems of the scene graph generation task while effectively ensuring basic predictions, and plays an important role in correcting the matching problem of core entities and core relationships.

[0057] The technical solution of this application fully considers that human perception of scene modality can be inspired by visual intuitive features and common sense experience, and comprehensively considers that the model fully utilizes the core visual information in the scene graph and combines common sense knowledge for knowledge modeling. A reasoning architecture based on core knowledge mining is proposed, which uses the model's processing ability for visual and textual knowledge to fully explore the potential factors that affect the generation of scene graphs, and attempts to cross the inherent threshold of scene graphs to generate accurate and semantically rich scene graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of the principle of the scene graph generation method based on core knowledge reasoning of this application;

[0059] Figure 2 A schematic diagram of the core entity reasoning framework used in the embodiment;

[0060] Figure 3 A schematic diagram of the core relational reasoning framework used in the embodiment;

[0061] Figure 4 This is a comparison chart of experimental data for three different tasks on the Visual Genome dataset in the embodiment;

[0062] Figure 5This is a comparison chart of experimental results of using three indicators under the guidance of different modules on the Visual Genome dataset in the embodiment;

[0063] Figure 6 The predicate distribution diagram of the embodiment compared with the BGNN method on the Visual Genome dataset;

[0064] Figure 7 This is a visualization diagram of the specific scene graph prediction results in the embodiment. DETAILED DESCRIPTION

[0065] The technical features of the present invention are further described below in conjunction with specific embodiments.

[0066] The basic concept of the scene graph generation method based on core knowledge reasoning in this application is:

[0067] The original image is input into a pre-trained deep model to extract the scene layout and entity labels, and a set of bounding boxes and corresponding feature maps are obtained. At this time, the visual features of each object are encoded.

[0068] In the core entity reasoning stage, the visual features in the first stage are globally searched and local information fused to extract the core entity features with interactive information, thereby outputting meaningful relational predicate representations.

[0069] In the core relationship reasoning stage, the network re-encodes the entity order to obtain richer relationship categories, sorts the obtained relationship categories from high to low according to confidence, and screens out semantically diverse triple features, namely core relationship features.

[0070] The core entities and core relationships are then fed into the knowledge fusion module for weighted fusion. After selecting representative and robust fusion features, the fused knowledge features are defined as prior knowledge in the embedding space, and Bayesian inference is performed for the core knowledge and corresponding predictions.

[0071] The core features and true labels are approximated using cross entropy loss. The process is repeated to gradually improve the feature fusion of core knowledge and true labels, thereby adapting to predictions in complex scenes and dynamically changing environments.

[0072] This method improves the under-prediction and long-tail distribution problems of scene graph generation tasks while effectively ensuring basic predictions, and plays an important role in correcting the matching problems of core entities and core relationships.

[0073] Specifically, the steps of the scene graph generation method based on core knowledge reasoning of the present application include:

[0074] Step 1) Input the dataset into the pre-trained deep model to extract scene layout and entity labels.

[0075] Among them, the deep model is the pre-trained target recognition model ResNeXt-101-FPN. The data set is input into the target detector for pre-training on the ImageNet data set. The training data sets are all image data sets.

[0076] In step 1), first, we train our core entity reasoning module and core relation reasoning module separately on the target dataset Visual Genome dataset, and then jointly train them with the Bayesian relation reasoning module.

[0077] Step 2) Send the extracted layout and visual images to the core entity reasoning module.

[0078] The core entity reasoning module performs global detection on the scene graph to mine as many entity nodes as possible to achieve maximum matching.

[0079] The visual Transformer encoder obtains the query Q of the i-th entity feature. e , key K e , value V e At the same time, query Q e It is fed to the visual Transformer encoding layer again. The entity feature F obtained by the encoder and decoder e With query Q e They are input into the feed-forward neural network FFN, and the core entity mining module outputs the core entity position P e , core entity feature label N e And the corresponding class score C is: P e , N e , C = FFN(Decoder(Q e , F e ));

[0080] Assume that undetected entities or relations are detected using Filled, the visual loss after core entity reasoning is L vis It can be defined as:

[0081]

[0082] in, represents the indicator function; L cls represents the cross entropy loss of all predicted entity categories; L box represents the loss of the bounding box, which consists of L1 loss and GIOU loss and is used to match successful predictions; b i and They represent the true bounding box of the i-th entity and the bounding box predicted after core entity reasoning.

[0083] Extraction of core knowledge:

[0084] The original image is input into the pre-trained target detection model to obtain the annotation box and instance features; the extracted basic image features are sent to the core entity reasoning module and the core relationship reasoning module to make a more detailed prediction of the sample probability. The high-confidence entity features and relationship features are output as the core knowledge after knowledge fusion.

[0085] The model predicts as many entities and relationship descriptions that fit the situation as possible, so that the reasoning conclusions can be more consistent with the actual scene diagram at the time.

[0086] Step 3) The core relational reasoning module first combines common sense entities in an orderly manner to generate different subject-object pairs.

[0087] Next, the joint entity passes through the subject attention head channel and the object attention head channel. After the entity is located, the initial representation of the entity and predicate is calculated at the relational awareness layer according to their different attention scores.

[0088] The knowledge extraction layer further removes redundant targets, predicates, or triples. The knowledge extraction layer calculates the cosine similarity cos_sim(·) function between the subject and the object to output a common sense relational metric matrix:

[0089] M = cos_sim(sub, obj);

[0090] Among them, sub and obj are the embedded representations of the subject and object nodes in the triples obtained by the knowledge extraction layer.

[0091] To measure the cognitive semantic consistency of the common sense relations obtained in the core relation reasoning stage between the generated scene graph and the real scene graph, we use the cross entropy loss based on the common sense relation metric matrix:

[0092]

[0093] in, and Defined as the cross entropy loss between the true label and the prediction; η obj and η rel is the weight parameter; e i and r ij Respectively represent the entity and relationship labels under the groundtruth label, and and They represent the entity and relationship labels predicted by the core relationship layer respectively.

[0094] Step 4) The Bayesian inference process integrates the obtained core entities and core relationships. First, standardize the core entity features and core relationship features and map them to the same space. Calculate the subject, object and relationship set in the public space. In this public domain, the core knowledge is used as a priori conditions for refined reasoning. Bayesian inference can further correct the bias in core entity reasoning and core relationship reasoning. The conditional distribution based on Bayes is as follows:

[0095]

[0096] Where P I，δ (S), P I，δ (O) and P I，δ (R) is the probability of measurement observed from image I by the model trained with parameters; S, R, O represent subject, relation and object features respectively; and Z S , Z R , Z O They represent the subject, relationship and object characteristics in public space respectively.

[0097] Then, by using the arg max(·) function to obtain the maximum value of the predicted distribution probability, the maximum posterior distribution of the output subject S', relation R', and object O' is calculated:

[0098]

[0099] In order to train the Bayesian reasoning process and improve the core knowledge expression, we impose a cross-entropy loss constraint on the prediction of core entities and core relationships.

[0100]

[0101] Among them, e i and r ij They represent the entity and relationship labels under the ground truth labels, respectively, and e′ i and r′ ij They respectively represent the predicted labels of the final core entities and core relations obtained through Bayesian inference in the public domain.

[0102] In addition, the prior factor λ is used to limit. The larger the λ, the more the model believes in the result of visual reasoning, and the smaller the λ, the more the model believes in the inherent prior knowledge inference. This effectively prevents the model from relying too much on visual information or common sense information to obtain overfitting prediction results. Finally, the visual reasoning loss L obtained by the core entity reasoning module is vis , the common sense reasoning loss L obtained by the core relational reasoning module com The loss L obtained by Bayesian inference bes_en , L bes_reCombined, the final loss L is:

[0103] L=λL vis +(1-λ)L com +L bes_en +L bes_re

[0104] When λ=0, it means that we only use the core relationship reasoning combined with Bayesian inference. At this time, the model's reasoning is mainly based on common sense and experience information. When λ=1, it means that the core relationship reasoning module is deleted and only the core entity reasoning module is used in conjunction with Bayesian to infer core knowledge. At this time, the model's inference is based on visual experience, and the inference conclusion can be more consistent with the actual scene diagram at that time.

[0105] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A scene graph generation method based on core knowledge reasoning, characterized by: The following steps are involved: 1) Input the dataset into the pre-trained deep model to extract scene layout and entity labels; 2) The extracted layout and visual images are fed into the core entity reasoning module, which performs global detection on the scene graph and mines entity nodes; The core entity reasoning module obtains the core entity position P e , core entity feature label N e , the corresponding class score C; 3) The entity labels obtained in step 1) and step 2) are combined with spatial features to form common sense entities, and the core relationship reasoning module combines them in an orderly manner to generate different subject-object pairs; The combined common sense entities are fed to the subject attention head channel and the object attention head channel in the form of subject-object and object-subject respectively; After locating the entities, the initial representations of the entities and predicates are calculated in the relation-aware layer according to their different attention scores; The knowledge extraction layer further removes redundant targets, predicates, or triples; The knowledge extraction layer calculates the cosine similarity between the subject and the object and outputs a relational metric matrix; 4) The Bayesian inference process integrates the obtained core entities and core relations; First, standardize the core entity features and core relationship features and map them into the same space; Calculate the set of subjects, objects, and relationships in public space; In this public domain, core knowledge is used as priors for refined reasoning; Compute the maximum a posteriori distribution of subjects, relations, and objects.

2. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 1), the deep model is a pre-trained target recognition model ResNeXt-101-FPN, and the data set is input into the target detector for pre-training on the ImageNet data set, and the training data sets are all image data sets; In step 1), the core entity reasoning module and the core relationship reasoning module are first trained separately on the target dataset Visual Genome dataset, and then jointly trained with the Bayesian relationship reasoning module.

3. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 2), the core knowledge is extracted by arranging and screening the samples with the highest confidence in the entities and relations. The core knowledge extraction process includes: Input the original image into the pre-trained object detection model to obtain the annotation box and instance features; The extracted basic image features are sent to the core entity reasoning module and the core relationship reasoning module to make a more detailed prediction of the sample probability; The entity features and relationship features with high confidence are output as core knowledge after knowledge fusion.

4. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 3) and step 4), the cross entropy loss function is used and the prior factor λ is used for restriction; The larger the λ, the more the model trusts the results of visual reasoning, and the smaller the λ, the more the model trusts the inherent prior knowledge inference.

5. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 2), P e ,N e ,C=FFN(Decoder(Q e ,F e )); Assume that undetected entities or relations are detected using Filled, the visual loss after core entity reasoning is L vis Defined as: in, represents the indicator function, L cls represents the cross entropy loss of all predictions, L box It consists of L1 loss and GIOU loss and is used to match successful predictions.

6. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 3), the knowledge extraction layer calculates the cosine similarity between the subject and the object to output a common sense relationship measurement matrix: M = cos_sim(sub, obj); Where sub is the embedded representation of the subject node in the triple obtained by the knowledge extraction layer, and obj is the embedded representation of the object node in the triple obtained by the knowledge extraction layer; The cross entropy loss based on the common sense relation metric matrix is ​​an important tool to measure the consistency of cognitive semantics between the generated scene graph and the real scene graph obtained in the core relation reasoning stage; The loss based on the common sense relationship measurement matrix is: in, and Defined as the cross entropy loss between the true label and the prediction, η obj and η rel is the weight parameter; Among them, e i and r ij Represent the entity and relationship labels under the real labels respectively, and They represent the entity and relationship labels predicted by the core relationship layer respectively.

7. The scene graph generation method based on core knowledge reasoning according to claim 1 is characterized in that: In step 4), the Bayesian conditional distribution is: Where P I，δ (S), P I，δ (O) and P I，δ (R) is the probability of measurement observed from image I by the model trained with the parameters; The maximum posterior distribution of the subject, relation, and object is expressed as The cross entropy loss for the prediction of core entities is limited to The cross entropy loss for the prediction of core relationships is limited to Among them, e i represents the entity label under the groundtruth label, r ij They represent the relationship labels under the groundtruth labels, e′ i represents the predicted label of the final core entity obtained through Bayesian inference in the public domain, r′ ij Represents the predicted labels of the final core relations obtained through Bayesian inference in the public domain.

Images