A Visual Commonsense Reasoning Method and System Based on Graph Attention Networks

Abstract

This invention discloses a visual commonsense reasoning method and system based on graph attention networks, relating to the interdisciplinary fields of computer vision and natural language processing. The method includes: training a constructed visual commonsense reasoning model using multiple sets of corresponding sample images, sample questions, and labeled sample answer candidates as a training set; the training process includes: acquiring visual and linguistic features of the sample images, sample questions, and sample answer candidates; using visual features as visual nodes and updating visual features using semantic information based on a graph attention mechanism; employing multimodal fusion to fuse linguistic features and updated visual features to obtain a joint visual and linguistic feature representation, thereby training the visual commonsense reasoning model; inputting the image to be reasoned, the question to be reasoned, and multiple reasoning answer candidates into the trained visual commonsense reasoning model, outputting the reasoning result, achieving accurate reasoning, and improving the performance of visual commonsense reasoning tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and natural language processing, and particularly relates to a visual common sense reasoning method and system based on a graph attention network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art that is known to those skilled in the art.

[0003] With the rapid development of computer vision and natural language processing, many visual-related researches gradually shift from bottom-level feature recognition to more complex and intelligent tasks, such as image description, visual question answering, video question answering, etc., involving computer vision, natural language processing, machine learning, and other research fields, aiming to realize high-level semantic cognition and expression of visual content. These specific tasks can usually be completed by people without any external assistance, but it is very difficult for computers to complete the tasks. People understand images usually according to existing common sense, combining objects and scenes in the image, and analyzing the meaning of the image through reasoning, while computers understand images limited to surface features, lacking semantic understanding of images.

[0004] In the prior art, visual common sense reasoning (VCR) is a multi-modal task that integrates common sense. This task not only needs to select the correct answer from the options according to the question, but also needs to select a reasonable reason according to the answer. Visual common sense reasoning includes three sub-tasks, including: Q→A: Given a natural language question Q, select the correct answer A from the answer candidate; QA→R: Given a natural language question and the answer to the question QA, select a reason R from the reason candidate; Q→AR: contains both Q→A and QA→R tasks, which need to select the correct answer A and reason R at the same time.

[0005] At present, the methods of visual common sense reasoning task can be divided into two categories: based on pre-training model and based on reasoning model. The pre-training model is pre-trained on the visual-linguistic dataset, and the pre-trained feature representation is used as the embedding feature of the task, which effectively improves the performance of visual-linguistic task; the reasoning model is to infer the answer by integrating visual and linguistic features. Although the performance of the pre-training model and the reasoning model is better than that of the early technology, due to their universality, the system is too complex, and when it comes to specific visual language tasks, there is a certain semantic gap, resulting in poor performance in completing visual language tasks.

[0006] Due to the difference between tasks, such as the attributes and common sense involved in visual question answering and visual common sense reasoning problems are not the same, it is difficult to solve all visual language problems using the above model architecture when the problem requires additional prior common sense, and the reasoning performance is poor. SUMMARY

[0007] To solve the above problems of the prior art, the present application provides a visual common sense reasoning method and system based on a graph attention network, which models the features of visual nodes and their adjacent nodes through a graph attention network to obtain the internal correlation between visual objects, then integrates natural language information into the visual features of visual objects, and further improves the semantic understanding ability of the model for images by strengthening the interaction between vision and language, thereby improving the performance of the visual common sense reasoning task.

[0008] In a first aspect, the present disclosure provides a visual common sense reasoning method based on a graph attention network, comprising:

[0009] A plurality of sets of sample images, sample questions and annotated sample answer candidates corresponding to each other are used as a sample training set to train the constructed visual common sense reasoning model; the training process comprises:

[0010] The language features of the sample questions and sample answer candidates are obtained, and the visual features of the visual objects in the sample images are obtained;

[0011] The visual features are used as visual nodes, the visual relationship features between each visual node in the sample images are learned based on a graph attention mechanism, the visual relationship features are updated using the semantic information of the sample questions and sample answer candidates, and the visual features are updated based on the updated visual relationship features;

[0012] Multi-modal fusion is used to fuse the updated visual features of the sample images, the language features of the sample questions and sample answer candidates, and obtain the joint feature representation of vision and language, so as to train the visual common sense reasoning model;

[0013] The to-be-reasoned image, the to-be-reasoned question and a plurality of reasoning answer candidates are input into the trained visual common sense reasoning model, and a reasoning result is output.

[0014] In a further technical solution, the language features of the sample questions and sample answer candidates are obtained, comprising:

[0015] A pre-trained language representation model is used to extract word embeddings of the natural language texts of the sample questions and sample answer candidates;

[0016] The extracted word embeddings are input into a bidirectional long short-term memory network to generate language features of the natural language texts of the sample questions and sample answer candidates.

[0017] Further, the technical scheme, the visual feature of the visual object in the sample image is obtained, and the method comprises the following steps:

[0018] The pre-trained ResNet101 network-based target detection model is used for target detection of the sample image, and the visual features of the multiple visual objects in the image are extracted.

[0019] Further, the technical scheme, the visual relationship feature between each visual node in the sample image is learned based on the graph attention mechanism, and the method comprises the following steps:

[0020] The visual features of the visual nodes are linearly transformed by using the shared weight matrix, the visual features of the visual nodes are mapped into a feature subspace, the mapping results of each visual node are obtained, and the mapping results are taken as the visual relationship features of each visual node.

[0021] Further, the technical scheme, the visual relationship feature is updated by using the semantic information of the sample question and the sample answer candidate, and the method comprises the following steps:

[0022] The word feature vectors of the visual words in the sample question and the sample answer candidate are connected with the visual relationship features of the visual nodes in the sample image, and the updated visual relationship features are obtained.

[0023] Further, the technical scheme, the visual feature is updated based on the updated visual relationship feature, and the method comprises the following steps:

[0024] The attention coefficient is calculated based on the updated visual relationship feature.

[0025] Based on the attention coefficient, the adjacent visual nodes of the visual node are weighted and summed, and the visual feature of the visual node fused with the weight features of the adjacent visual nodes is obtained.

[0026] Further, the technical scheme, the joint feature representation of the visual and the language is obtained, and the visual common sense reasoning model is trained based on the joint feature representation, and the method comprises the following steps:

[0027] The multilayer perceptron with the piecewise linear function is taken as the classifier, the joint feature representation of the visual and the language is taken as the input, and the cross-entropy loss between the real label and the predicted label of the sample answer candidate is taken as the input, and the visual common sense reasoning model is trained.

[0028] In a second aspect, the present disclosure provides a visual common sense reasoning system based on a graph attention network, comprising:

[0029] The visual common sense reasoning model construction and training module is used for taking a plurality of groups of sample image, sample question and labeled sample answer candidate corresponding to each other as a sample training set, training the constructed visual common sense reasoning model, and the training process comprises the following steps:

[0030] Obtain language features of the sample question and sample answer candidate, and obtain visual features of visual objects in the sample image;

[0031] Take the visual features as visual nodes, learn visual relationship features between each visual node in the sample image based on a graph attention mechanism, update the visual relationship features by using semantic information of the sample question and sample answer candidate, and update the visual features based on the updated visual relationship features;

[0032] Multi-modal fusion is adopted to fuse the updated visual features of the sample image, language features of the sample question and sample answer candidate, obtain joint feature representation of vision and language, and train the visual common sense reasoning model;

[0033] The reasoning module is configured to input the to-be-reasoned image, to-be-reasoned question and multiple reasoning answer candidates into the trained visual common sense reasoning model, and output a reasoning result.

[0034] In a third aspect, the disclosure further provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of the first aspect are completed.

[0035] In a fourth aspect, the disclosure further provides a computer-readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the steps of the method of the first aspect are completed.

[0036] The above one or more technical solutions have the following beneficial effects:

[0037] 1. The present application provides a visual common sense reasoning method and system based on a graph attention network, which models the features of visual nodes and their adjacent nodes through a graph attention network, obtains the internal correlation between visual objects, then integrates natural language information into the visual features of the visual objects, and by strengthening the interaction between vision and language, the semantic understanding ability of the model for images can be further improved, accurate reasoning is realized, and the performance of the visual common sense reasoning task is improved.

[0038] 2. The present application fully extracts information in the question and answer texts, integrates the text information into the image, and can better assist in understanding the semantic information of the image, providing favorable support for establishing the visual relationship between visual objects.

[0039] 3. The present application models the visual relationship between visual objects, can better understand and reason the image content, integrates the question and answer features to give the visual objects more fine-grained visual relationship features, and thus improves the accuracy of the model.

[0040] 4. The visual commonsense reasoning method proposed in this invention can effectively improve the understanding of images, gain a deeper understanding of the semantics between images, and effectively improve the performance of machine vision commonsense reasoning tasks. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0042] Figure 1 The flowchart is a visual commonsense reasoning method based on graph attention network as described in Embodiment 1 of the present invention;

[0043] Figure 2 This is an example diagram of the visual commonsense reasoning task in Embodiment 1 of the present invention. Detailed Implementation

[0044] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0046] Example 1

[0047] This embodiment provides a visual commonsense reasoning method based on graph attention networks, such as... Figure 1As shown, firstly, word feature vectors from questions and answer candidates are obtained through a pre-trained Bidirectional Encoder Representations from Transformer (BERT) model, and visual feature vectors of visual objects in the image are obtained through a Residual Network (ResNet). Then, visual objects in the image are encoded as visual nodes, and the features of these visual nodes and their connected nodes are modeled using a graph attention network to obtain an attention map containing the internal relationships between visual objects. Next, semantic information of questions and answer candidates is injected into this attention map, dynamically assigning higher weights to the relationships most relevant to each question and answer, thus giving visual objects more fine-grained visual relationship features. Multimodal fusion is used to fuse the semantic features of questions and answers with image semantic features into the response features, obtaining a fused response feature representation, i.e., a joint feature representation of vision and language. A piecewise linear function (Rectified Linear Function) is then used to further refine the representation. The ReLU multilayer perceptron is used as the classifier, and the joint feature representation of vision and language is input. The visual commonsense reasoning model is trained based on the cross-entropy loss between the true labels and predicted labels of the answer candidates. Finally, the image to be reasoned, the reasoning question corresponding to the image to be reasoned, and multiple candidate reasoning answers are input into the trained visual commonsense reasoning model, and the reasoning result is output.

[0048] The method described in this embodiment specifically includes:

[0049] First, the constructed visual commonsense reasoning model is trained using multiple sets of corresponding sample images, sample questions, and labeled sample answer candidates as a training set. In this embodiment, each set of sample images, sample questions, and sample answer candidates corresponds to each other, and the number of sample answer candidates is set to 4. Each sample answer candidate is labeled with an answer label relative to the sample image and sample question, that is, each sample answer candidate is labeled as a correct answer or an incorrect answer.

[0050] Then, the image to be reasoned, the question to be reasoned, and multiple candidate answers are input into the trained visual commonsense reasoning model, and the reasoning result is output.

[0051] The training process of the above visual commonsense reasoning model is as follows: Figure 1 As shown, it includes the following steps:

[0052] Step 1: Obtain the linguistic feature vectors of the sample questions and sample answer candidates, and obtain the visual feature vectors of the visual objects in the sample images.

[0053] In step 1 above, the visual commonsense reasoning task includes an image, a question, and answer candidates. The natural language question is based on the image. Both the question and the answer are sets of words (such as text words like [is], [are], [book], etc.) and visual words (such as [person1], [chair2], etc.). The visual words in the question and the answer correspond to objects or people in the image.

[0054] Therefore, firstly, for the sample images, a pre-trained object detection model based on the ResNet101 network is used to detect objects in the sample images, extracting the visual features of m objects in the image, that is, extracting m visual feature vectors v. i In other words, a sample image can be represented by a visual feature vector v. i The set represented by (i = 1, ..., m), where v i From bounding box b i and category tags i composition.

[0055] Secondly, in this embodiment, for the natural language sample questions and sample answer candidates corresponding to the sample images given in the sample training set, a pre-trained language representation model (Bidirectional Encoder Representations from Transformer, BERT) is used to extract word embeddings from the natural language text. Then, the extracted word embeddings are input into a Bi-directional Long Short-Term Memory (Bi-LSTM) network to generate word features of the natural language text, i.e., word feature vectors. Taking sample question q as an example, the question is represented by the word feature vector q. i The set represented by (i = 1, ..., n), where q i The feature of the i-th word in the corresponding sample question is defined by n, where n corresponds to the number of words in the sample question. Similarly, the sample answer candidate a can also be extracted. n Word feature vector a ni In this embodiment, n=4. Taking the first sample answer candidate as an example, the word feature vector a1 of the first sample answer candidate is extracted. 1i .

[0056] Step 2: To further enhance the understanding and reasoning of the sample image content, visual features are used as visual nodes. Based on a graph attention mechanism, visual relationship features between each visual node in the sample image are learned. The semantic information of the sample question and sample answer candidates is used to update the visual relationship features, and the visual features are updated based on the updated visual relationship features. In this way, the internal associations between visual objects are obtained, and natural language information is fused with visual features. By strengthening the interaction between vision and language, the semantic understanding ability of the visual commonsense reasoning model of images is further improved, thus enhancing the model's performance.

[0057] Specifically, each visual object in the sample image is encoded as a visual node, and multiple visual nodes are connected to construct a fully connected directed graph. A graph attention network is used to learn different weights for different neighboring visual nodes of each visual node through the attention mechanism, that is, different weights are assigned to each edge of each visual node to represent the visual relationship between two adjacent visual objects.

[0058] The graph attention network trains a shared weight matrix W∈R for all nodes. F* F′ is used to obtain the weights of each neighboring node. This weight matrix represents the relationship between the input F features and the output F′ features, acting as a mapping. When calculating the attention value, all nodes (including node i and its neighboring nodes N) are considered. i The representation of each is mapped using W, and the resulting vectors are concatenated. A feedforward neural network is then used to map the concatenated vector to real numbers and activated using LeakyReLU. After normalization, the final attention coefficients are obtained. These attention coefficients are then used to adjust the attention of neighboring nodes N. i We perform a weighted summation to obtain the output features of node i.

[0059] In this embodiment, the visual features of visual objects (i.e. visual nodes) in the sample image are used as the initial input of the graph attention network. In order to transform the input features into higher-level feature representations, the visual relationship features between each visual node in the sample image are learned based on the graph attention mechanism. That is, the visual features of the visual nodes are linearly transformed by the shared weight matrix, and the node features are mapped to a feature subspace to obtain the mapping result of each visual node, which is used as the visual relationship feature.

[0060] Specifically, firstly, the visual features v of m visual objects in the sample image are analyzed. i Perform a linear transformation (i.e., mapping) by applying a linear transformation to each visual node v. i ∈R B×N×F Perform a learnable, custom-initialized shared weight matrix w o ∈R B×F×F′A linear transformation (i.e., mapping) is performed to obtain the mapping result for each visual node, and this mapping result is used as the visual relationship feature of each visual node. Taking visual node i and its neighboring visual node j as an example, the mapping result (i.e., relationship feature) of visual node i and its neighboring visual node j is as follows:

[0061] V i =v i w0

[0062] V j =v j w0

[0063] In the above, F and F′ represent the number of input features and output features of the graph attention network, respectively. B is the number of training samples, i.e. the number of sample images during each training session, and N represents the number of features for each node.

[0064] Since the neighboring nodes of visual node i may include not only visual node j, but also visual node h, but the importance of these two visual nodes to visual node i is not necessarily equivalent, we calculate the attention coefficients of visual node i and its neighboring visual nodes to represent the importance of the features of different neighboring visual nodes to visual node i.

[0065] Before calculating the attention coefficients of visual node i and its neighboring visual nodes, i.e., before concatenating the corresponding mapping result vectors, to further improve the accuracy of reasoning, semantic information of the reasoning question is injected into the attention map. By integrating question features into visual relationship features, more fine-grained visual relationship features are assigned to visual objects. That is, for the sample question q corresponding to the sample image, the sample question q can be derived from the word feature vector q i The problem is represented by a set (i = 1, ..., n). Considering that the problem consists of words and visual words, where the visual words correspond to objects or people in the sample image, the word feature vectors of the visual words in the sample problem are concatenated with the visual relationship features of m visual objects (visual nodes) in the sample image to obtain the updated visual relationship features. Taking visual node i and its neighboring visual node j as an example, the updated visual relationship features of visual node i and its neighboring visual node j are as follows:

[0066] V i ′=[V i ||Q i ]

[0067] V j ′=[V j ||Q j ]

[0068] In the above formula, V i' represents the updated visual relation feature of visual node i, Q i V represents the set of word feature vectors for visual words related to visual node i in sample question q. Similarly, V... j ' represents the updated visual relation features of visual node j, Q j This represents the set of word feature vectors of visual words related to visual node j in sample problem q.

[0069] Using the same method described above, the word feature vectors of visual words in the sample answer candidates are connected with the visual relationship features of m visual objects (visual nodes) in the sample image, and the visual relationship features are updated again.

[0070] Using the above method, the updated visual relationship features are obtained by utilizing the semantic information of the sample questions and sample answer candidates. After encoding all visual nodes, the updated visual relationship features V are obtained. i The '' contains more granular visual features between visual objects, providing important semantic support for reasoning and fusion.

[0071] After obtaining the updated visual relationship features, the attention coefficient α is calculated based on these features. ij Taking visual node i and its neighboring visual node j as an example, the attention coefficient α between visual node i and its neighboring visual nodes... ij for:

[0072]

[0073] In the above formula, m is the initial matrix, and W ij Let b' be the directional matrix of the edges from visual node i to its neighboring visual node j. ij This represents the bias term of visual node i with respect to its neighboring visual node j.

[0074] After obtaining the attention coefficients, the neighboring visual nodes of visual node i are weighted and summed to obtain the visual features of visual node i fused with the weights of its neighboring visual nodes j, i.e.:

[0075]

[0076] Furthermore, to ensure that self-attention can stably represent visual nodes, this embodiment introduces a multi-head attention mechanism to improve the representational capability of the graph attention network model. For the output features of the intermediate layers, self-attention is calculated using K W values, and then the results obtained from the attention heads are concatenated to obtain the output vector. The final result is obtained by averaging the output vectors of each attention head. In this embodiment, multi-head attention is used to extend the graph attention network, and the parameters w are learned through k self-attention operations.k and The output features are concatenated together as v″ i The output feature representation is obtained, namely:

[0077]

[0078] Based on the above method, all visual objects are encoded, and the updated visual features cover the semantic relationships between visual objects. Unlike graph convolutional networks, graph attention networks effectively assign different feature weights to adjacent nodes.

[0079] Step 3: Employ multimodal fusion to fuse the updated visual features of the sample images, the linguistic features of the sample questions and sample answer candidates, and obtain a joint feature representation of visual and linguistic features.

[0080] Since the above process preserves the dimension of visual features, its joint information representation can be directly learned. This embodiment further uses a multimodal fusion method to fuse the semantic information of the sample question and sample answer candidates with the updated visual feature representation of each visual object, obtaining a joint feature representation of vision and language. Specifically, F represents the fusion function, and the joint feature representation of vision and language is Z. n =F(v″) i ,q i ,a ni ).

[0081] Step 4: Using a multilayer perceptron with a piecewise linear function as the classifier, the model is trained with the joint feature representation of vision and language as input and the cross-entropy loss between the true and predicted labels of the sample answer candidates as input.

[0082] In step 4 above, the joint visual and linguistic representations are input into a multilayer perceptron with a ReLU activation function, which outputs a predicted value using the joint visual and linguistic feature representations as input. Where ξ represents the trainable parameters and F′ represents the classification function.

[0083] A visual commonsense reasoning model is trained using cross-entropy as the loss function, employing the cross-entropy loss between the true and predicted labels of sample answer candidates. Here, `loss` represents the classification loss, and `f` represents the true value. The loss function represents the predicted value:

[0084]

[0085] The above visual commonsense reasoning model is trained based on sample images, sample questions, and sample answer candidates until the loss function converges, resulting in a fully trained visual commonsense reasoning model.

[0086] Finally, the image to be reasoned, the reasoning question, and multiple candidate reasoning answers are input into the trained visual commonsense reasoning model, and the reasoning result is output. The above scheme is further illustrated and verified by the following example.

[0087] The VCR task includes an image, a question, and answer candidates. The natural language question is based on objects in the image, and the question and answer candidates are sets of words and visual words. To conform to the BERT input format, name labels in the image are replaced with corresponding person names, and object labels are replaced with corresponding category names. Figure 2 Taking the question "What is [person2] wearing tuxedo?" and the answer "[person2] is coming from a formal party" as an example, replace [person2] with the corresponding person's name, input it into the Bert model, and input the word sequence into a Bi-LSTM with a hidden layer, which is encoded into a 512-dimensional feature vector.

[0088] The feature vectors of targets in the image are extracted using ResNet-101 and encoded as 2048-dimensional vectors. Each target in the image is defined by a bounding box. The feature vector of each image target is obtained using the RoiAlign method in the last layer of ResNet-101 and projected into a 512-dimensional embedding vector using a fully connected downsampling network.

[0089] Then, each visual object in the image (person2 and tuxedo in this example) is treated as a visual node, and a fully connected directed graph is constructed. The visual features of the visual nodes are used as the initial input to the graph attention layer and fed into the GAT to construct a visual relation representation. LeakyReLU is used as the activation function, and the number of multi-head attention is set to 3.

[0090] Secondly, by injecting semantic information of the question and a candidate answer into the directed graph, attention coefficients are constructed to imbue visual relationship features with relevant semantic information, thereby updating the visual features. Multi-head attention is then used to extend the graph attention network, learning semantically weighted information through self-attention operations. The output features are then concatenated to output the updated visual features.

[0091] Finally, a multimodal fusion method is used to further fuse semantic information with the updated visual features to obtain a joint visual and linguistic feature representation. This representation is then input into a multilayer perceptron to calculate the predicted value of this candidate answer as the correct answer output. Based on the above scheme, the correct candidate answer, "[person2] is coming from a formal party," is ultimately output.

[0092] Based on the above scheme, the reasoning process from QA to R can also be realized, which will not be elaborated here.

[0093] In summary, this embodiment provides a visual commonsense reasoning method based on graph attention networks. It uses graph attention networks to model the features of visual nodes and their neighboring nodes to obtain the internal relationships between visual objects. Then, natural language information is integrated into the visual features of visual objects. By strengthening the interaction between vision and language, the model's semantic understanding of images can be further improved, thus enhancing the performance of VCR tasks.

[0094] Example 2

[0095] This embodiment provides a visual commonsense reasoning system based on graph attention networks, including:

[0096] A visual commonsense reasoning model construction and training module is used to train the constructed visual commonsense reasoning model using multiple sets of corresponding sample images, sample questions, and labeled sample answer candidates as a sample training set; the training process includes:

[0097] Obtain the linguistic features of sample questions and sample answer candidates, and obtain the visual features of visual objects in sample images;

[0098] Using visual features as visual nodes, the visual relationship features between each visual node in the sample image are learned based on the graph attention mechanism. The semantic information of the sample question and sample answer candidates is used to update the visual relationship features, and the visual features are updated based on the updated visual relationship features.

[0099] Multimodal fusion is employed to integrate the updated visual features of sample images, the linguistic features of sample questions and sample answer candidates, and obtain joint feature representations of visual and linguistic features to train a visual commonsense reasoning model.

[0100] The reasoning module is used to input the image to be reasoned, the question to be reasoned, and multiple candidate answers into the trained visual commonsense reasoning model, and output the reasoning result.

[0101] Example 3

[0102] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps in the visual commonsense reasoning method based on graph attention networks as described above.

[0103] Example 4

[0104] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the visual commonsense reasoning method based on graph attention networks as described above.

[0105] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0106] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0108] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual common sense reasoning method based on a graph attention network, characterized in that, The method comprises the following steps: The method comprises the following steps: Obtaining language features of the sample question and the sample answer candidate, and obtaining visual features of visual objects in the sample image; A fully connected directed graph is constructed by taking the visual features as visual nodes and connecting multiple visual nodes, and visual relationship features between each visual node in the sample image are learned based on a graph attention mechanism; The visual relationship features are updated using semantic information of the sample question and the sample answer candidate, including connecting word feature vectors of visual words in the sample question and the sample answer candidate with visual relationship features of visual nodes in the sample image to obtain updated visual relationship features, and the formula is as follows: ； ； ； ； In the formula, denotes the custom initialization of the shared weight matrix, denotes a visual node i of a visual object in a sample image, denotes a neighboring visual node j of the visual node i of the visual object in the sample image, respectively represent the number of input features of the input graph attention network and the number of output features of the graph attention network output, B is the number of training samples, that is, the number of sample images at each training, and N represents the number of features of each node, denotes the mapping result of the visual node i and the neighboring visual node j, denotes the updated visual relationship feature of the visual node i, denotes a set of word feature vectors of visual words related to the visual node i in a sample question q; denotes the updated visual relationship feature of the visual node j, denotes a set of word feature vectors of visual words related to the visual node j in a sample question q; The visual features are updated based on the updated visual relationship features, including calculating an attention coefficient based on the updated visual relationship features, and the formula is as follows: ； wherein, is the attention coefficient of visual node i to its neighboring visual node j, m is the initial matrix, represents the directionality matrix of visual node i to its neighboring visual node j, represents the bias term of visual node i to its neighboring visual node j; The visual features are updated based on the attention coefficient; A multi-modal fusion is adopted to fuse the updated visual features of the sample image, language features of the sample question and the sample answer candidate, and joint feature representations of the visual and the language are obtained to train the visual common sense reasoning model; The image to be reasoned, the question to be reasoned and multiple reasoning answer candidates are input into the trained visual common sense reasoning model, and a reasoning result is output.

2. The visual common sense reasoning method based on a graph attention network according to claim 1, characterized in that, The language features of the sample question and the sample answer candidate are obtained by using a pre-trained language representation model to extract word embeddings of natural language texts of the sample question and the sample answer candidate. The language features of the sample question and the sample answer candidate are obtained by using a pre-trained language representation model to extract word embeddings of natural language texts of the sample question and the sample answer candidate. The visual features of the visual objects in the sample image are obtained by using a pre-trained target detection model based on a ResNet101 network to perform target detection on the sample image and extracting visual features of multiple visual objects in the image. 3.The visual common sense reasoning method based on the graph attention network according to claim 1, wherein, The visual relationship features between each visual node in the sample image are learned based on a graph attention mechanism by using a shared weight matrix to perform linear transformation on the visual features of the visual nodes, mapping the visual features of the visual nodes into a feature subspace to obtain mapping results of each visual node, and taking the mapping results as the visual relationship features of each visual node. The visual features are updated based on the attention coefficient by performing weighted summation on adjacent visual nodes of the visual node to obtain visual features of the visual node fused with weight features of the adjacent visual nodes. 4.The method of claim 1, wherein, The joint feature representations of the visual and the language are obtained to train the visual common sense reasoning model by taking a multi-layer perceptron with a piecewise linear function as a classifier, taking the joint feature representations of the visual and the language as input, and training the visual common sense reasoning model based on cross-entropy loss between real labels and predicted labels of the sample answer candidate.

7. A visual common sense reasoning system based on a graph attention network, comprising:

5. The visual common sense reasoning method based on graph attention network according to claim 1, characterized in that, ​ ​ 6. The visual common sense reasoning method based on graph attention network according to claim 1, characterized in that, ​ ​ ​ The visual common sense reasoning model construction and training module is configured to train the constructed visual common sense reasoning model by taking a plurality of sets of sample image, sample question and annotated sample answer candidate as a sample training set. The training process includes: Obtaining the language features of the sample question and the sample answer candidate, and obtaining the visual features of the visual objects in the sample image; Taking the visual features as visual nodes, connecting a plurality of visual nodes, constructing a fully connected directed graph, and learning the visual relationship features between each visual node in the sample image based on a graph attention mechanism; Updating the visual relationship features by using the semantic information of the sample question and the sample answer candidate, including connecting the word feature vectors of the visual words in the sample question and the sample answer candidate with the visual relationship features of the visual nodes in the sample image to obtain updated visual relationship features: ； ； ； ； In the formula, denotes the custom initialization of the shared weight matrix, denotes a visual node i of a visual object in a sample image, denotes a neighboring visual node j of the visual node i of the visual object in the sample image, respectively represent the number of input features of the input graph attention network and the output features of the graph attention network output, B is the number of training samples, that is, the number of sample images at each training, and N represents the number of features of each node, denotes the mapping result of the visual node i and the neighboring visual node j, denotes the updated visual relationship feature of the visual node i, denotes a set of word feature vectors of visual words related to the visual node i in a sample question q; denotes the updated visual relationship feature of the visual node j, denotes a set of word feature vectors of visual words related to the visual node j in a sample question q; Updating the visual features based on the updated visual relationship features, including calculating an attention coefficient based on the updated visual relationship features, including: ； wherein, is the attention coefficient of visual node i to its neighboring visual node j, m is the initial matrix, represents the directionality matrix of visual node i to its neighboring visual node j, represents the bias term of visual node i to its neighboring visual node j; Updating the visual features based on the attention coefficient; Using multi-modal fusion to fuse the updated visual features of the sample image, the language features of the sample question and the sample answer candidate, and obtaining the joint feature representation of the visual and language to train the visual common sense reasoning model; The reasoning module is configured to input the to-be-reasoned image, the to-be-reasoned question and a plurality of reasoning answer candidates into the trained visual common sense reasoning model, and output a reasoning result.

8. An electronic device, characterized by comprising: A computer program product comprising a memory and a processor and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of a visual common sense reasoning method based on a graph attention network according to any one of claims 1-6 are completed.

9. A computer readable storage medium characterized by, A computer program product for storing computer instructions, when the computer instructions are executed by a processor, the steps of a visual common sense reasoning method based on a graph attention network according to any one of claims 1-6 are completed.

Images