A visual common sense reasoning method and device and a storage medium
By constructing heterogeneous graphs and utilizing a structure-sensitive graph converter network for graph structure evolution, the problem of graph structure inability to evolve in visual commonsense reasoning models is solved, achieving higher reasoning accuracy and deeper feature extraction, thus improving the performance of visual commonsense reasoning.
Patent Information
- Application Number
- CN202210735418.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-27
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-06-27
AI Technical Summary
Existing visual commonsense reasoning models struggle to effectively model the correlations between distant words during the learning process, and their performance declines with increasing layers. They also fail to evolve the graph structure throughout the learning process, resulting in insufficient reasoning accuracy.
We construct answer-visual heterogeneous graphs and answer-question heterogeneous graphs, use a structure-sensitive graph converter network to evolve the graph structure, learn the representation of graph nodes through a multi-layer network, and use a scoring pooling mechanism for inference and prediction. We also use pre-trained BERT and ResNet101 to extract features.
It improves the accuracy of visual commonsense reasoning, better represents the correlation between modalities, and achieves deeper feature extraction and higher precision.
Smart Images
Figure CN115344675B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of computers, and relates to a video processing method, in particular to a visual common sense reasoning method and device and a storage medium. BACKGROUND
[0002] In recent years, visual-linguistic tasks such as visual question answering, visual localization, and visual description have attracted more and more attention from the academic and industrial communities. Most of them are still at the recognition level and can only provide appropriate predictions, but cannot provide reasons for making predictions. Taking visual question answering as an example, it is a four-classification task, which selects the correct option from the answers (usually phrases such as fresh and frozen) according to the given image and question. In order to conduct in-depth research, researchers gradually focus on reasoning tasks at the cognitive level and propose the visual common sense reasoning task. Given a question about an image, the visual common sense reasoning model not only needs to provide the correct answer, but also needs to provide a reasonable reason to prove the answer. The options of the visual common sense reasoning task are not simple phrases, but more complex visual and linguistic expressions, and the task is a challenging problem.
[0003] Current solutions for visual common sense reasoning mainly include models based on overall attention and models based on graph representation. The model based on overall attention processes the question and the answer as sequence data, but it is difficult to model the correlation between words far apart. The method based on graph representation can associate words or entities with each other by constructing edges between graph nodes without considering word order. However, the current methods based on graph representation have some disadvantages. First, the structure of the graph learned by these models is determined by the initial adjacency matrix, and the adjacency matrix cannot evolve during the entire learning process. In addition, these models can only use one layer of graph convolutional network, and the performance will decrease with the increase of the number of layers. These disadvantages hinder the model from learning the graph constructed in the visual common sense reasoning task more effectively and flexibly. SUMMARY
[0004] The purpose of the present application is to overcome the defects of the prior art and provide a visual common sense reasoning method, device and storage medium with high reasoning accuracy.
[0005] The purpose of the present application can be achieved by the following technical solutions:
[0006] A visual common sense reasoning method comprises the following steps:
[0007] obtain an inference prediction result by using a trained visual common sense reasoning model as input of the image region, the question text corresponding to the image region, and the candidate answer text;
[0008] The visual common sense reasoning model comprises:
[0009] A heterogeneous graph construction layer is configured to construct an answer-visual heterogeneous graph and an answer-question heterogeneous graph based on the image region, the question text corresponding to the image region, and the candidate answer text.
[0010] A heterogeneous graph updating layer is configured to use a structure-sensitive graph transformer network to evolve the heterogeneous graphs based on the answer-visual heterogeneous graph and the answer-question heterogeneous graph, and obtain inference sequence features.
[0011] A classification layer is configured to use a scoring pooling mechanism to obtain the inference prediction result based on the inference sequence features.
[0012] Further, the node features of the answer-visual heterogeneous graph and the answer-question heterogeneous graph are obtained based on visual regions and text words.
[0013] Further, the visual regions and the text words are subjected to feature extraction by using a pre-trained Bert and ResNet101 to obtain the node features.
[0014] Further, the adjacency matrix of the answer-visual heterogeneous graph and the answer-question heterogeneous graph is initialized by a similarity matrix of the node features.
[0015] Further, the evolution process of the heterogeneous graph updating layer comprises:
[0016] The answer-visual heterogeneous graph and the answer-question heterogeneous graph are processed by a first structure-sensitive graph transformer network to update the node features of the respective heterogeneous graphs, and obtain answer visual sequence features and answer question sequence features.
[0017] The answer visual sequence features and the answer question sequence features obtained by the first structure-sensitive graph transformer network are spliced with original answer features to obtain spliced sequence features, and a reasoning graph is constructed. The reasoning graph is processed by a second structure-sensitive graph transformer network to update the node features of the respective heterogeneous graphs, and obtain inference sequence features.
[0018] Further, the first structure-sensitive graph transformer network uses a multi-head self-attention mechanism to learn the increment of the graph structure encoding of the previous layer as priori, to obtain the context association, and to update the node features according to the evolved graph structure.
[0019] Furthermore, the second structure-sensitive graph converter network uses a multi-head self-attention mechanism to learn incremental graph structure encodings, thereby updating the node features based on the evolving graph structure.
[0020] Furthermore, the loss function used during the training of the visual commonsense reasoning model is expressed as follows:
[0021] loss = CE_loss + αBCE_loss
[0022]
[0023]
[0024] Where n is the number of samples, y i For tags, The predicted probability of activation by the softmax function. α is the predicted probability of activation by the sigmoid function, and α is the adjustment factor.
[0025] The present invention also provides a visual commonsense reasoning apparatus, including one or more processors, a memory, and one or more programs stored in the memory, said one or more programs including instructions for performing the visual commonsense reasoning method as described above.
[0026] The present invention also provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the visual commonsense reasoning method as described above.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] 1) This invention constructs answer-visual heterogeneous graphs and answer-question heterogeneous graphs, which can simultaneously represent intramodal and intermodal correlations and undergo structural evolution throughout the learning process. This allows for more appropriate representation of cross-modal information, thereby improving reasoning accuracy.
[0029] 2) This invention employs a multi-layer structure-sensitive converter, which uses the evolutionary adjacency matrix as graph structure information as a priori to effectively learn the representation and structure of graph nodes, thereby constructing a deeper graph model with stronger feature extraction capabilities.
[0030] 3) This invention uses a scoring pooling mechanism to obtain inference prediction results, more adaptively integrates graph features, selects valuable clues for inference, and ultimately achieves higher accuracy. Attached Figure Description
[0031] Figure 1 This is a schematic diagram illustrating the principle framework of the visual common sense reasoning model of this invention. Detailed Implementation
[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0033] This embodiment provides a visual commonsense reasoning method, including the following steps: obtaining the image region to be reasoned, as well as the question text and candidate answer text corresponding to the image region; using the image region, the question text and candidate answer text corresponding to the image region as input, and obtaining the reasoning prediction result through a trained visual commonsense reasoning model.
[0034] like Figure 1 As shown, the visual commonsense reasoning model includes: a heterogeneous graph construction layer, used to construct answer-visual heterogeneous graphs and answer-question heterogeneous graphs based on the image region, the corresponding question text, and the candidate answer text; a heterogeneous graph update layer, which uses a structure-sensitive graph converter network to evolve the heterogeneous graphs based on the answer-visual heterogeneous graphs and answer-question heterogeneous graphs to obtain reasoning sequence features; and a classification layer, which uses a scoring pooling mechanism based on the reasoning sequence features to obtain reasoning prediction results.
[0035] Specifically, in the heterogeneous graph construction layer, pre-trained BERT and ResNet101 are first used to extract features from the received visual regions and text words. Specifically, BERT is used to encode and extract features from the question text and candidate answer text, while ResNet101 is used to encode and extract features from the image regions. Then, using the extracted features as node features, answer-visual heterogeneous graphs and answer-question heterogeneous graphs are constructed. These heterogeneous graphs can simultaneously represent intramodal and intermodal correlations. The features involved in the answer-visual heterogeneous graph include M... av_vv M av_va M av_aa and M av_av The features involved in the answer-question heterogeneous graph include M. aq_qq M aq_qa M aq_aa and M aq_aq av and aq are used to distinguish between two heterogeneous graphs.
[0036] The similarity matrix of node features is obtained by calculating the dot product between the features of each node and then performing softmax activation. This matrix is used to initialize the adjacency matrix of the heterogeneous graph. Taking the answer-visual heterogeneous graph as an example, the adjacency matrix can be calculated as follows:
[0037]
[0038]
[0039]
[0040]
[0041] E av_v =softmax r ([M av_vv M av_va ])
[0042] E av_a =softmax r ([M av_aa M av_av ])
[0043] Among them, S i It is an image region feature. It is the word feature of the answer text, Softmax r It is the softmax function for the row direction.
[0044] In the heterogeneous graph update layer, evolutionary updates are performed through a first structure-sensitive graph converter network for context association and a second structure-sensitive graph converter network for inference. Unlike traditional converter models, the input of the structure-sensitive graph converter includes graph structure encoding, rather than position encoding. The specific evolutionary process includes:
[0045] The first structure-sensitive graph converter network processes the answer-visual heterogeneous graph and the answer-question heterogeneous graph respectively, updates the node features of each heterogeneous graph, and obtains the answer visual sequence features and the answer question sequence features. The first structure-sensitive graph converter network uses the graph structure encoding of the previous layer as a priori and uses a multi-head self-attention mechanism to learn the increment of the graph structure encoding, thereby obtaining the context association and updating the node features according to the evolving graph structure. When the graph structure does not need to evolve further, the increment tends to zero.
[0046] The visual sequence features of the answer and the sequence features of the answer question, obtained through the first structure-sensitive map converter network, are concatenated with the original answer features to obtain the concatenated sequence features. A reasoning graph is constructed, and the reasoning graph is processed by a second structure-sensitive graph converter network to update the node features of each heterogeneous graph, thereby obtaining the reasoning sequence feature V. cls The second structure-sensitive graph converter network also uses a multi-head self-attention mechanism to learn the incremental graph structure encoding, thereby updating the node features according to the evolving graph structure.
[0047] The aforementioned heterogeneous graph update layer performs the evolution of the heterogeneous graph. Taking the answer-visual heterogeneous graph as an example, the graph structure update mechanism can be defined as:
[0048]
[0049] The node feature update mechanism can be defined as:
[0050]
[0051] in, Value av_a These are the three feature matrices for multi-head attention.
[0052] In the classification layer, a scoring pooling mechanism is used to adaptively fuse map features to obtain f. cls Then, a fully connected layer is used for prediction.
[0053] The visual commonsense reasoning model described above is trained using a dataset. The specific process of visual commonsense reasoning includes:
[0054] S401: Given an image, a question, and four answers, select the most appropriate answer;
[0055] S402: Given the image, question, correct answer and four reasons provided directly in the dataset for selection, choose the most appropriate reason.
[0056] S403: If the results of S401 and S402 are both correct, then a visual common sense reasoning exercise has been successfully performed.
[0057] S404: Repeat steps S401, S402 and S403 multiple times to complete the prediction of all visual commonsense reasoning in the dataset.
[0058] The loss function used in training the above visual commonsense reasoning model is expressed as follows:
[0059] loss = CE_loss + αBCE_loss
[0060]
[0061]
[0062] Where n is the number of samples, y i For tags, The predicted probability of activation by the softmax function. α represents the predicted probability of activation by the sigmoid function, and α is a modulating factor. Both the softmax and sigmoid functions are applied in the classification layer.
[0063] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0064] To verify the performance of the present invention, the following experiment was designed.
[0065] In this embodiment, the model constructed in this invention (SGTEHG) was compared with other models on a visual commonsense benchmark dataset. Among them, ECMR is one of the best performing methods to date, and this invention still has a certain accuracy advantage compared with it, as shown in Table 1.
[0066] Table 1. Comparison of the SGTEHG model with other models on the visual commonsense benchmark dataset.
[0067]
[0068]
[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A visual commonsense reasoning method, characterized in that, Includes the following steps: Obtain the image region to be reasoned, as well as the question text and candidate answer text corresponding to the image region. Using the image region, the question text and candidate answer text corresponding to the image region as input, obtain the reasoning prediction result through a trained visual commonsense reasoning model. The visual commonsense reasoning model includes: A heterogeneous graph construction layer is used to construct an answer-visual heterogeneous graph and an answer-question heterogeneous graph based on the image region, the question text corresponding to the image region, and the candidate answer text; The heterogeneous graph update layer employs a structure-sensitive graph converter network to evolve the heterogeneous graph based on the answer-visual heterogeneous graph and the answer-question heterogeneous graph, thereby obtaining inference sequence features. In the classification layer, based on the inference sequence features, a scoring pooling mechanism is used to obtain the inference prediction results; The evolution process of the heterogeneous graph update layer includes: The first structure sensitive map converter network processes the answer-visual heterogeneous map and the answer-question heterogeneous map respectively, updates the node features of each heterogeneous map, and obtains the answer visual sequence features and the answer question sequence features. The visual sequence features of the answer and the sequence features of the answer question obtained by the first structure sensitive map converter network are concatenated with the original answer features to obtain concatenated sequence features, which constitute a reasoning graph. The reasoning graph is then processed by the second structure sensitive map converter network to update the node features of each heterogeneous graph and obtain the reasoning sequence features. The first structure-sensitive graph converter network uses the graph structure encoding of the previous layer as a priori, and uses a multi-head self-attention mechanism to learn the increment of the graph structure encoding, thereby obtaining the context association and updating the node features according to the evolving graph structure. The second structure-sensitive graph converter network uses a multi-head self-attention mechanism to learn incremental graph structure encodings, thereby updating the node features based on the evolving graph structure.
2. The visual commonsense reasoning method according to claim 1, characterized in that, The node features of the answer-visual heterogeneous graph and the answer-question heterogeneous graph are obtained based on visual regions and text word extraction.
3. The visual commonsense reasoning method according to claim 2, characterized in that, The visual regions and text words are used to extract features using pre-trained BERT and ResNet101 to obtain the node features.
4. The visual commonsense reasoning method according to claim 2, characterized in that, The adjacency matrices of the answer-visual heterogeneous graph and the answer-question heterogeneous graph are initialized by the similarity matrix of node features.
5. The visual commonsense reasoning method according to claim 1, characterized in that, The loss function used during the training of the visual commonsense reasoning model is expressed as follows: in, n For the sample size, For tags, The predicted probability of activation by the softmax function. The predicted probability of activation by the sigmoid function. It is a regulating factor.
6. A visual common sense reasoning device, characterized in that, It includes one or more processors, memory, and one or more programs stored in the memory, said one or more programs including instructions for performing the visual commonsense reasoning method as described in any one of claims 1-5.
7. A computer-readable storage medium, characterized in that, It includes one or more programs that are executed by one or more processors of an electronic device, the one or more programs including instructions for performing the visual commonsense reasoning method as described in any one of claims 1-5.