Visual question and answer method based on visual concept atlas

Abstract

A visual question and answer method based on a visual concept atlas comprises the steps of constructing the visual concept atlas comprising a concept hierarchical relationship, a visual relationship and visual features, and positioning related concepts of the visual concept atlas by using questions and images respectively. A trilinear interactive fusion module (CTI) is used for fusing input image-text pairs and local atlas features and visual features obtained by a visual concept atlas, answers of visual questions and answers are obtained through a classification head and a preset answer list, and the performance of visual questions and answers is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal artificial intelligence, relates to the cross-research direction of natural language processing and computer vision, and specifically relates to a visual question answering method based on a visual concept graph. Background Art

[0002] Visual question answering is an important research direction in the field of artificial intelligence, aiming to combine computer vision and natural language processing technologies so that machines can understand image content and answer related natural language questions. The existing visual question answering forms are roughly divided into two types: FFOE VQA and MC VQA. FFOE VQA, that is, Free-Form Opened-Ended VQA, refers to answering in a free form for a given input image-text pair; MC VQA, that is, Mulitiple Choice VQA, refers to selecting an answer from a given answer list for a given image and text.

[0003] Early visual question answering models were based on a two-stream architecture. After independently extracting visual (CNN) and text (RNN) features, modal fusion was performed, and then a joint model was built through a fully connected layer to generate an answer classification result. Due to the characteristic that the CNN convolutional neural network can only extract global image features, it is very easy to cause answer deviation when the question and answer involve the attributes of specific objects in the image. At the same time, simple cross-modal feature splicing and linear interaction ignore the fine-grained alignment of cross-modalities.

[0004] In recent years, a new research direction has emerged in the field of visual question answering, that is, the visual question answering task based on a knowledge base. For example Figure 3 , it is very difficult to simply rely on the input image-text pair to predict the answer. However, if additional common sense information "Fire hydrant can be used for fighting fires" can be added, the prediction probability of the correct answer can be greatly improved. Models such as ConceptBert and KRISP all improve the performance of visual question answering by integrating external knowledge bases. However, such knowledge bases still only utilize structured text knowledge and cannot utilize the rich and huge heterogeneous modal data at the present stage.

[0005] Introducing visual modality information into a knowledge-based visual question answering model can provide multi-angle descriptions and information supplementation for the common sense of the input image-text pair. In the literature [Ding Y, Yu J, Liu B, et al. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 5089-5098.], an end-to-end multimodal knowledge extraction and accumulation model was proposed, and three loss functions were proposed to learn the representations of triples, namely head entity features, tail entity features, and relationship features, and the multimodal knowledge of the accumulation model was trained and accumulated through in-domain and out-of-domain data. Another study [Yu J, Zhu Z, Wang Y, et al. Cross-modal knowledge reasoning for knowledge-based visual question answering [J]. Pattern Recognition, 2020, 108: 107563.] obtains supplementary evidence from multimodal information and decomposes the model into modules based on memory reasoning. These studies are all based on off-the-shelf common sense knowledge bases, which contain miscellaneous and excessive common sense information and it is difficult to ensure the isolation of noise. Summary of the Invention

[0006] To overcome the above-mentioned shortcomings of the prior art, in view of the current situation that the common sense knowledge base is miscellaneous and excessive and does not contain multimodal information, the present invention provides a visual question answering method based on a visual concept graph. By introducing a visual concept graph as the common sense knowledge base for the visual question answering task, this common sense knowledge base is more concise and contains visual information, so it can solve the problem that the picture in the visual question answering task may not contain answer information and improve the visual question answering performance.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A visual question answering method based on a visual concept graph, comprising the following steps:

[0009] Step 1, construct a visual concept graph VCG and a directed graph G, and store the triples of the visual concept graph VCG in the directed graph G; the visual concept graph VCG includes a concept hierarchy relationship, a visual relationship, and visual features;

[0010] Step 2, use the question and the image to respectively locate the relevant concepts of the Visual Concept Graph (VCG), and obtain the local graph feature Q K and the visual feature V K ;

[0011] Step 3, fuse the question feature Q and the visual feature V of the input text-image pair with the local graph feature Q K and the visual feature V K , and obtain the answer to the visual question through the classification head and a preset answer list.

[0012] In one embodiment, the Visual Concept Graph (VCG) is constructed by the following method:

[0013] Use a public dataset, retrieve the upper semantic concepts starting from each label until the independent starting concept is retrieved, and link all the concepts to obtain a hierarchical concept graph;

[0014] Use scene graph generation technology to process the images in the validation set corresponding to the public dataset, obtain visual relationship information triples, filter out positional and prepositional visual relationships, and retain action and comparative visual relationships as the visual relationships between concept nodes;

[0015] Set a screening rule, select the visual feature images of concept nodes that meet the rule, and generate corresponding visual features for the concept nodes.

[0016] In one embodiment, use the question to locate the relevant concepts of the visual concept graph and obtain the local graph feature. The implementation method is as follows:

[0017] Step 211, tokenize the question text, and calculate the word2vec word vectors w of the nouns and exclusive vocabulary in it;

[0018] Step 212, traverse the directed graph G in a loop, find the concept node with the highest similarity to w in it, and determine whether the similarity is higher than the threshold τ. If so, add the corresponding concept node to the list. If not, do not add it, and repeat Step 211;

[0019] Step 213, repeat Steps 211 - 212 until all the concept nodes in the Visual Concept Graph (VCG) are located;

[0020] Step 214, find the shortest path between the concept nodes with the highest similarity of the word vectors of nouns, named entities, and exclusive vocabulary in the question text and all the concept nodes on the path;

[0021] Step 215, use the concept nodes obtained in Step 212 as the subject in the Visual Concept Graph (VCG), retrieve the object downward, and then use the object as the subject to continue retrieving the object downward. Save all the retrieved triples to Gdata;

[0022] Step 216: Construct a GNN network, and use the local graph in Gdata as a directed graph to extract corresponding features, namely Q, in the GNN network. K 。

[0023] Compared with the prior art, the present invention introduces the hierarchy and visual relationships between visual features and concepts in the visual concept graph in the field of visual question answering, overcoming the problem that the picture may not contain answer information in the visual question answering task. By introducing a visual and hierarchical relationship knowledge base, additional common sense is added to the model on the basis of enhancing visual localization. And the visual concept graph can additionally provide relevant visual features of concept nodes, which is significantly different from the common sense knowledge bases cited by common models.

[0024] In addition, the present invention uses an improved fusion module. For the existing trilinear fusion module for input images, questions, and answers, the present invention modifies it to four inputs to adapt to the situation where the present invention needs to additionally fuse the visual features provided by the visual concept graph and the local graph features. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of the visual concept graph constructed by the present invention.

[0026] Figure 2 is the classification system of the visual relationships of the present invention.

[0027] Figure 3 is a schematic diagram showing the importance of visual question answering based on the knowledge base of the present invention.

[0028] Figure 4 is a schematic diagram of the visual question answering model based on the visual concept graph of the present invention.

[0029] Figure 5 is a schematic diagram of the process of locating local concept nodes in the visual concept graph through question text of the present invention.

[0030] Figure 6 is a schematic diagram of the process of locating the visual features of concept nodes in the visual concept graph through the input image of the present invention.

[0031] Figure 7 is a schematic diagram of the model for cross-attention between local graph features and visual concept features of the present invention.

[0032] Figure 8 is the question-answering image showing the experimental evaluation indexes of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The embodiments of the present invention will be described in detail below with reference to the drawings and examples.

[0034] According to the foregoing, the present invention needs to combine the constructed visual concept graph and use it as a common sense knowledge base for visual question answering. At the same time, the visual concept graph includes the hierarchical structure of common life concept nodes and the visual relationship information summarized and refined from the corresponding nodes in a large number of images. The nodes link the images and their features screened by certain screening conditions.

[0035] Reference Figure 1 , the visual question answering method based on the visual concept graph of the present invention mainly includes the following steps:

[0036] Step 1, construct a visual concept graph VCG.

[0037] In this embodiment, the COCO dataset is used, and 80 labels therein cover 11 common aspects in life. With the help of the hierarchical relationship of the WordNet knowledge base, retrieve the upper semantic concepts starting from these 80 labels until the independent starting concepts (synset without upper synonyms) are retrieved, and link all 80 concepts to obtain a hierarchical concept graph, which contains the hierarchical relationship between the concepts covering 11 aspects of life.

[0038] Next, use the scene graph generation technology to process the images in the COCO2014 validation set to obtain a large number of visual relationship information triples. By setting certain screening relationships, filter the position and prepositional visual relationships, and retain the action and comparative visual relationships as the visual relationships between concept nodes. The specific visual relationship classification is as Figure 2 shown.

[0039] Finally, set the following screening rules, that is, the selection restriction conditions for the visual feature images of concept nodes:

[0040] 1. There is only one visual entity in the image.

[0041] 2. The proportion of the bounding box of this visual entity exceeds 75%.

[0042] 3. Select at most 5 images for each category.

[0043] Based on the COCO2014 validation set as the basic dataset, select several images for each of the 80 labels, a total of 352 images, with an average of 4.4 images for each concept node. Select ResNet50 as the extraction model for visual features and generate corresponding visual features for the concept nodes.

[0044] So far, a visual concept graph VCG covering hierarchical relationships, visual relationships, and visual features can be obtained.

[0045] Step 2, as Figure 4This is the overall framework of the invention model. First, according to the hierarchical relationship of the Visual Concept Graph (VCG) and the triple of visual relationship information, a directed graph G is constructed for subsequent shortest path retrieval. The triples in the visual concept graph are stored in the directed graph G and saved in the graphml format.

[0046] Specifically, according to the triple information stored in the csv file containing hierarchical relationship triples, concept nodes are added to the initially created directed graph, and directed edges are added according to the hierarchical relationship in the visual concept graph. Then, according to the triples in the visual_relation.json file storing visual relationship triple information, first determine whether the triple and its subject and object exist in the directed graph. If not, add a directed edge for the visual relationship; if they exist, consider the next set of visual relationships.

[0047] All the subjects and objects in the directed graph G are stored in the file graph_data.graphml.

[0048] Step 3: Extract the feature vector of the question text.

[0049] In this embodiment, Faster r-cnn with ResNet152 as the backbone is used as the visual feature extractor. 100 bounding boxes are identified for each image, and a 2048-dimensional feature vector is generated for each bounding box. The processed results are shown in Table 1:

[0050] Table 1 Basic information and feature information table of images in the coco dataset

[0051] Image ID Height Width Number of bounding boxes Bounding box coordinates Image features 202373 426 640 100 (100,4) (100,2048) 48641 480 640 100 (100,4) (100,2048) 505010 640 597 100 (100,4) (100,2048) 505145 593 640 100 (100,4) (100,2048)

[0052] Bert is selected as the encoder for the question text to extract question features. In this embodiment, the input question text is subjected to feature extraction through bert and extracted as a 768-dimensional feature tensor

[0053] Step 4, as Figure 5 The process of locating the relevant local graph of the visual concept graph from the question text shown is mainly divided into three steps: word segmentation, similarity calculation, and location.

[0054] Word segmentation means splitting the question text into words. Specifically, for the input question text, the natural language processing library spacy library is used for word segmentation and splitting. First, the named entities and exclusive words in the input question text are extracted; to avoid omission, nouns are identified from the text again using the spacy library.

[0055] For example, the question text "a group of men bow their heads in front of tables offood" can be processed to obtain a noun list of ['group','men','heads','front','tables','food'] (['group','men','head','front','table','food']).

[0056] Step 5: Set the similarity and calculate the word2vec word vector w of the noun and the exclusive vocabulary.

[0057] In this embodiment, the similarity τ is set to 0.8, and the 300-dimensional Word2vec word vector of each word in the noun list is calculated.

[0058] Step 6: Loop through the directed graph G to find the concept node with the highest similarity to w, and determine whether the similarity is higher than the threshold τ. If so, add the corresponding concept node to the list; if not, do not add it, and repeat steps 4 and 5.

[0059] In this embodiment, the entities contained in the directed graph G are traversed word by word and the word vector similarity is detected, and the following formula is used:

[0060]

[0061] Where: v Q ——Word vector that needs to be traversed; v KG ——The word vector of each concept node in the visual concept graph.

[0062] The word with the highest similarity to the comparison word is selected, and its word vector similarity is determined to be greater than a threshold τ. If so, it is stored in the similar_result list. This allows the question text to be used to locate similar concept nodes in the visual concept graph while ensuring a high level of semantic similarity. Using the question text above as an example, this solution would produce a similar_result list of ['group', 'people', 'tableware', 'food'] (['group', 'people', 'tableware', 'food']).

[0063] Repeat steps 4-6 until all concept nodes in the visual concept graph VCG are located.

[0064] Step 7: Find the shortest path directly to the concept node with the highest similarity among nouns, named entities, and exclusive vocabulary word vectors in the question text and all concept nodes on the path.

[0065] In this embodiment, concept nodes in the similar_result list are selected pairwise, and the shortest path between two nodes in the directed graph G is retrieved by the nx.shortest_path method until all the shortest paths that do not form a cycle between pairwise concept nodes related to the problem text and all the concept nodes on the paths are found. For example, a list of shortest path triples shortest_path such as [{"subject": "domestic_animal", "relationship": "parent", "object": "dog"}, {"subject": "dog", "relationship": "beside", "object": "cardboard"}] ([{"subject": "livestock", "relationship": "parent-child", "object": "dog"}, {"subject": "dog", "relationship": "beside", "object": "cardboard"}]) can be obtained.

[0066] Step 8: Take the concept nodes obtained in step 212 as the subjects in the visual concept graph VCG, retrieve the objects downward, and then take the objects as the subjects again to continue retrieving the objects downward. Save all the retrieved triples to Gdata.

[0067] Locate the concept nodes in the visual concept graph according to the problem text. If there is a path that does not form a cycle between pairwise concept nodes, the relationships between all the concept nodes on this path and the concept nodes retrieved downward pairwise form the retrieved triples. For example, if the concept nodes "elephant" and "banana" are retrieved and there is a relationship "eat" between these two concept nodes, then <elephant, eat, banana> is the retrieved triple.

[0068] Therefore, in this embodiment, based on the retrieved shortest_path list, for all the concept nodes in it, retrieve their lower-level concept nodes by the G.successors method. If the shortest_path list is empty, it means that there is no path relationship between the concept nodes related to the problem text, and only the lower-level concept node relationships of all the concept nodes in the similar_result list need to be retrieved by the G.successors method. Finally, a list of local graph triples Gdata obtained by text location can be obtained.

[0069] Step 9: Design a GNN network to extract local graph features.

[0070] The GNN network designed in this embodiment includes a first convolutional layer, a second convolutional layer, and a pooling layer. Among them, the input dimension of the first convolutional layer is aligned with the dimension of the input triple list Gdata, and 64-dimensional feature representation is output after feature extraction; the second convolutional layer raises the 64-dimensional features to a 768-dimensional high-dimensional space; the pooling layer aggregates features along the node dimension and outputs 768-dimensional feature representation.

[0071] Table 2 GNN Network Layer Settings

[0072] Network layer name Network scale Convolutional layer 1 GCNConv(input_dim = len(Gdata), hidden_dim = 64) Convolutional layer 2 GCNConv(hidden_dim = 64, output_dim = 768) Pooling layer torch.mean(output, dim = 0, keepdim = True)

[0073] According to the constructed GNN network, the corresponding features, namely Q, are extracted from the local graph spectrum in Gdata as a directed graph in the GNN network. K 。

[0074] Step 10, use the image to locate the relevant concepts of the visual concept graph spectrum to obtain visual features.

[0075] The process of locating the local area of the visual concept graph spectrum by the image is as Figure 6 shown. In the annotation file of the COCO dataset, the bounding boxes and their category information of all images in the COCO dataset are stored. Select training pictures from them, and select several bounding boxes (no more than 5 at most). According to the label categories corresponding to the selected bounding boxes, locate the corresponding concept nodes in the visual concept graph spectrum VCG, and then select the visual features linked by the concept nodes, that is, the visual features V K 。

[0076] Step 11, fuse the features, and obtain the answer to the visual question through the classification head and the preset answer list.

[0077] In this step, there are three forms of the input for fusion, which are respectively:

[0078] Form 1, fuse the question feature Q and the visual feature V of the input image-text pair with the local graph spectrum feature Q K and the visual feature V K 。

[0079] Form 2, process Q and V through the multimodal pre-training model vilbert to obtain the question feature Q f and the visual feature V f , fuse Q f and V f with the local graph spectrum feature Q K and the visual feature V K 。

[0080] Vilbert uses the way that the text stream and the visual stream interact through the attention Transformer layer to cross-learn the joint representation. By interacting the features V and Q through the multi-modal pre-trained model Vilbert, the visual V f (dimension 1024) and the question feature Q f (dimension 768) can be obtained.

[0081] In Form Three, Q K and V K pass through the cross-attention layer twice respectively to achieve feature enhancement, and the locally mapped features Q KG and the visual features V KG after feature enhancement are obtained. Then, Q f and V f are fused with Q KG and V KG .

[0082] For example, Figure 7 in this invention, Q K and V K are respectively used as the V feature input. In this way, the model will not independently learn the correlations between the two features and the predicted answer, but will generate the locally mapped features based on the visual features and the visual features based on the locally mapped features. In this way, the model can learn in advance the mutual correlations between the two features provided by the visual concept map before the fusion module. Finally, the locally mapped features Q KG and the visual features V KG after interaction are obtained.

[0083] Furthermore, this invention uses an improved trilinear interaction fusion module for feature fusion. The improvement is to increase the input channels and modify the learnable parameters of the fourth input feature.

[0084] The CTI trilinear interaction fusion module is used to linearly interact and fuse the relationships among the image, question and answer features. The formula is as follows:

[0085] z T =(((Τ×1vec(M1))×2vec(M2))×3vec(M3))

[0086] In the formula: M1, M2 and M3 are the feature representations of the input image, question and answer, and t = 1, 2, 3; n t is the number of input feature channels, and d t is the channel dimension.

[0087] The present invention modifies the trilinear interaction fusion module to add a one-dimensional input M4, performs trilinear interaction on the features obtained in the present invention, and modifies the dimension of the learnable tensor T so that it can learn the features of M4. The formula is as follows:

[0088] z T = (((Τ × 1vec(M1)) × 2vec(M2)) × 3vec(M3)) × 4vec(M4))

[0089] In the formula: M1, M2, M3, and M4 are the joint representations of the input image, question, atlas, and visual features of the atlas nodes; T is a learnable tensor; vec(M t ) is a row vector in the feature representation, and t = 1, 2, 3, 4.

[0090] Step 12, the visual question answering form of the present invention adopts MC VQA, and a list of answers with a length of 3129 is set in advance. Therefore, according to the above experiments, the features after trilinear cross-fusion are passed through a 3129 classification head for category prediction. In the experiments of the present invention, a multi-classification loss function is selected, and the formula is as follows:

[0091]

[0092] In the formula: x represents the output of the model, y represents the true label, C represents the number of apologies, n represents the number of samples, y ij represents whether the i-th sample belongs to the j-th label. If it belongs, it is 1; if it does not belong, it is 0; x ij represents the probability that the i-th sample is predicted as the j-th label.

[0093] Description of implementation effect

[0094] To test the actual performance of the present invention, an ablation experiment is set in the VQA-v2 dataset, and 20 rounds of training are set. In the visual question answering task, there may be multiple reasonable answers to the same question. For example, given a picture and asking "What color is this?", there may be different answers such as "red" or "dark red", all of which should be reasonably evaluated. Therefore, when calculating the final score, multiple possible answers should be considered, rather than simply judging whether it matches the only standard answer.

[0095] For example, for Figure 8The image shown corresponds to the question "What do these giraffes have in common?" The dataset provides reference answers including "eating," "spots," and "tall." "Eating" has the highest weight, 1; "spots" and "tall" have lower weights, only 0.3. Therefore, the evaluation metric used in this experiment is score. This metric maps the classification result to a list of preset answers. If the classification result matches the preset answer, the score is calculated based on the corresponding weight; otherwise, no score is included.

[0096] Table 3 Ablation experiment table of visual question answering based on visual concept graph

[0097] Model Loss Score Baseline model 2.587 74.611 Baseline model + local graph 2.510 75.912 Baseline model + visual features 2.539 75.829 Baseline model + visual concept graph 2.335 78.655

[0098] Table 3 compares the performance of the baseline model, the model that introduces local graphs and visual features alone, and the model that introduces a visual concept graph as an external knowledge base. Introducing the visual concept graph significantly improves the model performance of the downstream visual question answering task by 4.044%, significantly outperforming the model that uses only the local graph or visual features alone (1.301% and 1.274%). The visual concept graph can not only establish a semantic association network through the concept ontology library, but also build a perceptual discrimination foundation based on visual prototype features, thereby effectively enhancing the robustness of the model in cross-scene understanding and multi-step logical reasoning tasks. Experimental results show that when the input image and text contain too much irrelevant local information for the concept node corresponding to the retrieved question text, or when the bounding box in the image has low relevance to the key content of the question, noise may be introduced, which may interfere with the accuracy of answer prediction to a certain extent.

[0099] Table 4 Visual question answering ablation experiment examples

[0100]

[0101] As can be seen in the second example in Table 4, the baseline model incorrectly predicts the answer as yes, likely due to overlapping visual features. Given the relationship between "elephant" and "elephant" in the visual concept graph, it is found that it does not encompass this relationship. Therefore, introducing the visual concept graph as an external knowledge base may help visual question answering predict the correct answer.

[0102] In summary, the core innovation of the present invention lies in constructing a dual knowledge base enhancement mechanism for the visual concept graph: at the semantic level, through the hierarchical relationship structure of the concept skeleton, in-depth mining of the attributes, relationships, and hierarchical structures of the concepts related to the image-text pairs is achieved; at the visual level, interpretable abstract conceptual visual features are introduced. Through ablation experiments by separately introducing the local graph, visual features, and visual concept graph, the results show that the visual concept graph, as an external knowledge base to assist the inference of the visual question answering task, not only outperforms the baseline model but also significantly outperforms the cases where only the local graph or visual features are introduced. Therefore, it can be concluded that the constructed visual concept graph can effectively support the visual question answering task based on the knowledge base.

[0103] However, simply introducing the visual concept graph may introduce noise, thus affecting the inference performance of the model. Therefore, it is necessary to further improve and enrich the visual concept graph and combine the graph refinement and optimization strategy to reduce the problems of noise interference and insufficient conceptual common sense and enhance the robustness and inference ability of the model.

Claims

1. A visual question answering method based on a visual concept graph, characterized in that, It includes the following steps: Step 1, construct a Visual Concept Graph (VCG) and a directed graph G, and store the triples of the VCG in the directed graph G; the VCG contains a concept hierarchy, visual relationships, and visual features; Step 2: Use the question and the image to respectively locate the relevant concepts of the Visual Concept Graph (VCG), and obtain the local graph feature Q K and the visual feature V K ; Step 3, fuse the problem feature Q and visual feature V of the input image-text pair with the local graph spectrum feature Q K and visual feature V K , and obtain the answer to the visual question and answer through the classification head and a preset answer list.

2. The visual question answering method based on a visual concept graph according to claim 1, wherein The VCG is constructed by the following method: Use a publicly available dataset, retrieve the upper semantic concepts starting from each label until the independent starting concept is retrieved, and link all the concepts to obtain a hierarchical concept graph; Use scene graph generation technology to process the images in the validation set corresponding to the publicly available dataset, obtain visual relationship information triples, filter out positional and prepositional visual relationships, and retain action and comparative visual relationships as the visual relationships between concept nodes; Set a filtering rule to select the visual feature images of concept nodes that meet the rule and generate corresponding visual features for the concept nodes.

3. The visual question answering method based on a visual concept graph according to claim 1 or 2, wherein The directed graph G is constructed by the following method: According to the hierarchical relationship of the VCG and the hierarchical relationship in the visual relationship information triples, add concept nodes to the initially created directed graph, and add directed edges according to the hierarchical relationship in the VCG; According to the visual relationship in the triples, first judge whether the triples and their subjects and objects exist in the directed graph. If not, add a directed edge for the visual relationship; If they exist, consider the next set of visual relationships.

4. The visual question answering method based on a visual concept graph according to claim 1, wherein Use the question to locate the relevant concepts of the VCG to obtain the local graph features. The implementation method is as follows: Step 211, tokenize the question text, and calculate the word2vec word vectors w of the nouns and exclusive words in it; Step 212, traverse the directed graph G in a loop, find the concept node with the highest similarity to w, and judge whether the similarity is higher than the threshold τ. If so, add the corresponding concept node to the list. If not, do not add it, and repeat Step 211; Step 213, repeat Steps 211 - 212 until all concept nodes in the VCG are located; Step 214, find the shortest path between the concept nodes with the highest similarity of the noun, named entity, and exclusive word vectors in the question text and all the concept nodes on the path; Step 215, use the concept nodes obtained in Step 212 as the subject in the VCG, retrieve the object downward, and then use the object as the subject to continue retrieving the object downward. Save all the retrieved triples to Gdata; Step 216, construct a GNN network, and use the local graph spectrum in Gdata as a directed graph to extract corresponding features, namely Q, in the GNN network K .

5. The visual question answering method based on a visual concept graph according to claim 4, characterized in that In Step 211, use the natural language processing library spacy to tokenize the question text, extract the command entities and exclusive words in the question text; and again use the spacy library to identify nouns from the text.

6. The visual question answering method based on a visual concept graph according to claim 4, wherein In Step 214, select concept nodes in the list pairwise, search for the shortest path between the two nodes in the directed graph G until all the shortest paths between the concept nodes related to the question text that do not form a loop and all the concept nodes on the path are found.

7. The visual question answering method based on a visual concept graph according to claim 4, characterized in that In step 215, concept nodes in the visual concept graph are located according to the problem text. If there are acyclic paths between any two concept nodes, the relationships between all concept nodes on the path and the concept nodes retrieved downward from them form the retrieved triples.

8. The visual question answering method based on a visual concept graph according to claim 1, wherein, The relevant concepts of the visual concept graph are located using the image to obtain the visual features. The implementation method is as follows: Step 221, in the annotation file of the input image, search for the bounding boxes and their corresponding labels, and locate the corresponding concept nodes in the Visual Concept Graph (VCG) through these labels to obtain the visual feature V K .

9. The visual question answering method based on a visual concept graph according to claim 1, characterized in that In step 3, the question feature Q and the visual feature V of the input image-text pair are interacted through a multi-modal pre-trained model to obtain the interacted question feature Q f and the visual feature V f ; Apply the local atlas feature Q K and the visual feature V K twice through the cross-attention layer respectively to achieve feature enhancement, obtaining the enhanced local atlas feature Q KG and the visual feature V KG ; Fuse Q f and V f with the local atlas feature Q K and the visual feature V K ; or, fuse Q f and V f with Q KG and V KG .

10. The visual question answering method based on a visual concept graph according to claim 1, characterized in that, In step 3, an improved trilinear interaction fusion module is used for feature fusion. The improvement is to add a one-dimensional input M4 to the trilinear interaction fusion module and modify the dimension of the learnable tensor T so that it can learn the features of M4. The formula is as follows: z T = (((Τ × 1vec(M1)) × 2vec(M2)) × 3vec(M3)) × 4vec(M4)) Where: M1, M2, M3, and M4 are the features of the input image, problem, atlas, and visual feature of the atlas node, and vec(M t ) is a row vector in the feature representation, where t = 1, 2, 3, 4.

Images