A multi-relation-aware heterogeneous graph visual question answering method fusing syntax trees

Abstract

The present application relates to a kind of fusion grammar tree multi-relation perception heterogeneous graph visual question and answer method, belong to computer vision and natural language processing technical field.This method constructs semantic relationship graph and spatial relationship graph, to make full use of the entity relationship in picture;To fully combine question grammar information and picture information, use noun word node as link, combine question grammar tree with entity semantic relationship graph and entity spatial relationship graph respectively, construct multi-modal heterogeneous graph structure that fuses multi-relation and multi-node;Design a kind of relationship perception heterogeneous graph attention mechanism based on multi-head attention, to fully extract the question representation and entity representation of relationship perception, realize the fine-grained feature interaction and implicit alignment of graph-text entity in modal and inter-modal.This method realizes the semantic alignment of question word and picture entity while also realizing the dynamic information interaction between entities.

Description

Technical Field

[0001] This invention relates to a multi-relation-aware heterogeneous graph visual question answering method, specifically a multi-relation-aware heterogeneous graph visual question answering method that integrates syntax trees, belonging to the fields of computer vision and natural language processing technology. Background Technology

[0002] In recent years, the rapid development of deep learning has driven leaps and bounds in the fields of computer vision and natural language processing. Visual question answering (VQA) technology combines visual inputs such as images and videos with related natural language questions to generate a natural language answer as output. It involves the intersection of multiple technical fields such as computer vision (CV), natural language processing (NLP), and knowledge representation and reasoning (KR).

[0003] Visual Question Answering (VQA) techniques need to overcome the challenges of general image understanding and question answering tasks, as well as the difficulties posed by using large-scale databases with mixed-quality inputs. Driven by the advent of deep learning (DL), the existence of advanced techniques in computer vision (CV) and natural language processing (NLP), and the availability of relevant large-scale datasets, VQA techniques have made significant progress. Currently, many different visual question answering models have been proposed for a comprehensive understanding of multimodal information. Initially, Antol et al. proposed the classic network architecture LSTM+VGG, which uses element-wise outer products to fuse question and image features to solve VQA tasks.

[0004] In attention-based visual question answering methods, Yang et al. proposed SAN, whose core idea is to use question semantics to search for the most relevant region in the image to the answer, and to use stacked attention layers in the process of repeatedly querying the image content using question semantics to gradually infer the final answer; Kumar et al. proposed DMN, which adjusts the attention size by using the input and the memory of previous iterations. This model uses a hierarchical recursive sequence model for reasoning when generating answers; Anderson et al. proposed BUTD, which uses a combination of bottom-up and top-down attention mechanisms. Bottom-up attention is used to extract regions of interest in the image, and top-down attention is used to determine the attention weight of each region of interest, thus computing the image-text joint representation more naturally.

[0005] In visual question answering methods based on bilinear pooling, Ben et al. proposed MUTAN, which uses the Tucker decomposition method based on multimodal tensors to effectively parameterize visual and text representations with bilinear parameters and uses a low-rank matrix factorization method to solve the dimensionality problem; Kim et al. proposed the BAN model, which considers the bilinear interaction between two sets of input channels and fuses visual and linguistic information by finding a bilinear attention distribution; Guo et al. proposed the BGN model, which uses a bilinear graph network to model the context of joint word and object embeddings. For complex reasoning, the BGN model can fully explore the implicit relationships between words.

[0006] In the field of image relation mining, Cadened et al. proposed MuRel, which introduces MuRel units to represent the interaction between the question region and the image region using rich vector representations. It uses pairwise combinations to model the relationship between regions, enabling more complex reasoning functions required for higher-level tasks. Hu et al. proposed the LCGN model, which achieves complex relation reasoning through iterative message passing conditioned on text input. Li et al. proposed the ReGAT model, which constructs a local scene graph for each image and models explicit and implicit relations through graph attention mechanisms, making it more capable of learning relation representations that are adaptive to the question.

[0007] However, existing research often overlooks the grammatical information in the question text. Introducing grammatical information offers two advantages: First, it enriches the question's content. For example, different parts of speech play different roles in different tasks; in visual question answering tasks, noun nodes typically provide more useful information, allowing the visual question answering system to understand the essence of the question more deeply. Second, grammatical information contains the dependencies between words.

[0008] By constructing a syntax tree, it is possible to establish the relationship between heterogeneous graph structures between text modalities and image modalities, construct a multimodal heterogeneous graph, and thus better achieve cross-modal content understanding. Summary of the Invention

[0009] The purpose of this invention is to address the problems and shortcomings of existing technologies. To overcome the technical issues of current visual question answering methods not fully considering grammatical and entity relationship information, resulting in poor performance when answering complex questions, this invention creatively proposes a multi-relation-aware heterogeneous graph visual question answering method that integrates syntax trees. This method utilizes the grammatical information of the question and the entity relationship information in the image to achieve a fine-grained understanding of the question and image content. This method is the first to construct a text-image heterogeneous graph structure, enabling more accurate joint feature mining of text and images.

[0010] The innovations of this invention include: designing a relation extraction network to explicitly extract semantic and spatial relations from images; constructing semantic and spatial relation graphs to fully utilize entity relations in images; using noun-type word nodes as links to combine the question syntax tree with the entity semantic relation graph and entity spatial relation graph respectively, in order to fully combine question grammar information and image information, constructing a multimodal heterogeneous graph structure that integrates multiple relations and multiple nodes; and designing a relation-aware heterogeneous graph attention network based on a multi-head attention mechanism to fully extract relation-aware question representations and entity representations, realizing fine-grained feature interaction within and between modalities and implicit alignment of image and text entities.

[0011] This invention is based on the following technical solution.

[0012] A multi-relation-aware heterogeneous graph visual question answering method integrating syntax trees includes the following steps:

[0013] Step 1: Extract image entity features and entity relationships, and construct entity semantic relationship graph and spatial relationship graph.

[0014] Specifically, it includes:

[0015] Step 1.1: The Faster R-CNN network can be used to identify a set of entity objects and a set of bounding box feature vectors from the image.

[0016] Step 1.2: A pre-trained Entity Relationship Extraction Network (REN) can be used to extract the semantic relationships between various entity objects from the image and calculate the spatial relationships between entity objects based on the bounding box feature vectors.

[0017] Step 1.3: Construct semantic relationship graphs and spatial relationship graphs based on entity semantic relationships and entity spatial relationships.

[0018] Step 2: Construct the problem syntax tree and encode the syntax tree information.

[0019] Specifically, it includes:

[0020] Step 2.1: The Stanford syntax parsing tool can be used to extract the grammatical dependencies and part-of-speech information of the question and construct the question syntax tree.

[0021] Step 2.2: We can use GloVe to initialize the text representation of each word node in the problem syntax tree, randomly initialize the part-of-speech representation vectors that the network can learn, and concatenate the two as the tree node representation.

[0022] Step 2.3: The GAT neural network can be used to learn the syntax tree and obtain grammar-structure-aware word representations. Finally, a bidirectional GRU neural network can be used to extract grammar-information-aware question representations and word representations.

[0023] Step 3: Construct a grammar-aware multimodal heterogeneous graph and extract fine-grained entity and problem representations for grammar information awareness.

[0024] Specifically, it includes:

[0025] Step 3.1: Using the noun nodes in the problem syntax tree as bridges, construct a multimodal heterogeneous graph that connects the problem syntax tree and the entity relationship graph.

[0026] Step 3.2: To flexibly aggregate different node features in multimodal heterogeneous graphs, the relation-aware heterogeneous graph attention network RAHAN can be used to extract grammatically informed entity representations from the multimodal heterogeneous graphs. This not only includes the semantic and spatial relation graphs of the image modality, but also combines the question syntax tree, which can guide the feature transfer between visual entities according to different relations, and finally obtain fine-grained entity representations with grammatically informed information.

[0027] Step 4: Calculate the joint features of the visual image and the question text.

[0028] To obtain a joint representation that integrates image information and question information, a top-down attention mechanism can be used to integrate question representation and entity representation that is aware of grammatical information.

[0029] Step 5: Predict the answer labels for relevant questions.

[0030] The probability distribution of the answer can be obtained through two fully connected layers.

[0031] Step 6: Train the neural network and use cross-entropy as the loss function to calculate the loss value.

[0032] Specifically, the number of iterations is preset (e.g., 30 times). If the number of iterations meets the preset number, the training process ends and the prediction of the rumor authenticity label is achieved. If the number of iterations does not meet the preset number, the process returns to step 1 and continues to the next round of training.

[0033] Thus, the multi-relation-aware heterogeneous graph visual question answering method integrating syntax trees was completed, enabling the prediction of answers to questions.

[0034] Beneficial effects

[0035] Compared with the prior art, the method of the present invention has the following advantages:

[0036] 1. This method proposes to utilize the syntax tree information of the problem to achieve a deeper understanding of the problem, and it performs better in understanding complex problems.

[0037] 2. This method is not limited to a single-modal graph structure. Instead, it uses noun nodes in the syntax tree to connect the syntax tree, entity semantic relation graph, and entity spatial relation graph, thus constructing a multimodal heterogeneous graph structure that includes word nodes, entity nodes, and various edge relations.

[0038] 3. This method designs a multi-relation-aware heterogeneous graph attention mechanism, which includes an intra-modal relation-aware attention mechanism and a multi-head attention mechanism for inter-modal questions. While realizing the semantic alignment of question words and image entities, it also realizes dynamic information interaction between entities. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention.

[0040] Figure 2 This is a block diagram illustrating the implementation of the method of the present invention. Detailed Implementation

[0041] The method of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] Example

[0043] The VQA2.O dataset was used as the training data.

[0044] like Figure 1 As shown, a multi-relation-aware heterogeneous graph visual question answering method that integrates syntax trees includes the following steps:

[0045] Step 1: Extract image entity features and entity relationships, and construct entity semantic relationship graph and spatial relationship graph.

[0046] Using the Faster R-CNN network, a set of entity objects is identified. Each entity object v i They all have a visual feature vector and a bounding box feature vector K∈[10, 100], d v =2048, d b =4. Feature vector b for each bounding box i = [x, y, w, h] corresponds to a four-dimensional spatial coordinate system, where (x, y) represents the coordinates of the top-left corner of the bounding box, and (w, h) represents the width w and height h of the bounding box.

[0047] For semantic relation extraction, the input image Region feature extraction is performed using a Faster R-CNN network to obtain v i v j and the joint bounding box v of the two ij Feature v i vj v ij Region feature filtering v′ is achieved through ROI pooling operations. i , v′ j , v′ ij Finally, the features are concatenated and input into a multilayer perceptron to obtain the relation probability distribution p. ij The calculation is as follows:

[0048]

[0049] v′ i , v′ j , v′ ij =R(v i v j v ij (2)

[0050] p ij =Softmax(W[v′) i ;v′ j ;v′ ij ]+b) (3)

[0051] in, Let R represent the Faster-RCNN network, R() represent the ROI pooling function, W represent the parameter matrix, and b represent the bias term.

[0052] For spatial relation extraction, spatial relations are defined as follows: <v i -v j > represents the geometric position of object j relative to object i. Spatial relationships can be categorized into multiple classes based on the IoU (Intersection over Union, a metric for measuring the accuracy of detecting corresponding objects in a given dataset), relative distance, and angle between the bounding boxes of the two objects. Given two entity objects v i and v j Their positions are represented by the center coordinates (x, y, y). i y i ), (x j y j To represent this, we get v. i and v j IoU between them, relative distance d ij and relative angle θ ij The relevant calculations are as follows:

[0053]

[0054]

[0055]

[0056]

[0057] Among them, d ij represents the distance between the centroids of two objects, and θ ij represents the angle between the vector from the i centroid of v j to the centroid of v diag and the positive x-axis. d i represents the length of the diagonal of the entire matrix containing v j and v hntSec . S i represents the size of the intersection area of v j and v Union . S

[0058] If v i completely covers v j or v i is completely covered by v j , then the spatial relationships <v i , v j > are "inside" and "cover" relationships respectively; if the IoU between v i and v j is ≥ 0.5, then the spatial relationship <v i , v j > is an "overlap" relationship; if the φ i between v j and v ij is < 0.5 and IoU < 0.5, then the category of the spatial relationship is selected according to the size of index; if the φ i between v j and v ij is > 0.5 and IoU < 0.5, it means that the spatial relationship <v i , v j > is too weak to construct the spatial relationship between v i and v j .

[0059] Step 2: Construct a problem syntax tree and encode the syntax tree information.

[0060] Specifically, taking the original problem sequence as input, through the Stanford syntax parsing tool, obtain the词性类别序列T = (t1, t2,..., t N ) and the set of dependency relationships between words Among them, the dependency relationship is represented by <q i , q j .

[0061] First, randomly initialize the part-of-speech tag dictionary (e.g., with a length of 42), obtain the part-of-speech tag vectors that the network can learn, and obtain the part-of-speech tag sequence T = (t1, t2, ..., t3) for the question. N ).

[0062] Then, the positional representation of the word P = (p1, p2, ..., p) is calculated using the positional encoding formula. N )

[0063] Then, the GloVe word embedding model is used to obtain the word representation sequence X = (x1, x2, ..., xn) consisting of the representations of each word in the problem. N ), Describing a vector space, d t d represents the vector dimension of the part-of-speech tag. p d represents the dimension of the vector representing the position. x The vector dimension representing the word representation.

[0064] Finally, the three different features are concatenated to form the grammar-aware feature X for each word. cat The calculation is as follows:

[0065] T=RandomInit(t1,t2,...,t N (8)

[0066] X = Glove(q1, q2, ..., q) N (9)

[0067]

[0068]

[0069]

[0070] Where RandomInit represents the random initialization function, Glove represents the Glove word embedding model, pos represents the position of the word in the sentence, and sin represents the sine function. This indicates vector concatenation.

[0071] Construct a problem syntax tree based on dependencies. Let X be the set of nodes of the problem syntax tree, where each node of the syntax tree is a word node, and its node representation set is X. cat =(x′1,x′2,...,x′ N )'

[0072] Step 3: Construct a grammar-aware multimodal heterogeneous graph and extract fine-grained entity and problem representations for grammar information awareness.

[0073] From the problem syntax tree Extracting grammar-aware question representations and grammar-aware word representations. Since a tree structure can be viewed as a special type of graph structure, the question syntax tree is first constructed. The input is fed into a grammar relation-aware graph neural network to learn grammar structure-aware node representations. Based on grammatical dependencies... i q j Depending on the category, relation-aware attention weights are adaptively calculated, and a multi-head attention mechanism is used to focus on the features of different vector spaces of each node.

[0074]

[0075]

[0076] in, Let α represent the node features after aggregation of node i, M represent the number of multi-head attention heads, and α represent the node features after aggregation. ij This represents the attention weight score, and σ represents the sigmoid function. W represents the number of neighboring nodes of node i. dir(i，j) b represents the relation parameter matrix from node i to node j. dep(i，j) b′ represents the bias term in the relationship between node i and node j. dep(i，j) Let V represent another relational bias term between node i and node j, U represent the parameter matrix for calculating attention weight scores, T represent matrix transpose, and V represent the other relational bias term between node i and node j. dir(i，j) Let represent the parameter matrix from node i to node j, and exp represent the exponential function.

[0077] Representation of tree nodes The problem representation h is further extracted from the bidirectional GRU neural network to perceive sequence information. t :

[0078] h t =BiGRU(X) * (15)

[0079] BiGRU stands for Bidirectional GRU Neural Network.

[0080] Step 4: Calculate the joint features of the visual image and the question text.

[0081] ​RAHAN is a relation-aware heterogeneous graph attention model. This network consists of an intramodal relation-aware attention mechanism and an intermodal multi-head self-attention mechanism. It aims to extract grammatically informed entity representations from multimodal heterogeneous graphs. It not only includes semantic and spatial relation graphs of the image modality, but also combines question syntax trees, which can guide the feature transfer between visual entities according to different relations.

[0082] heterogeneous graph The input to RAHAN first uses an intramodal relation-aware attention mechanism to calculate relation-aware attention weights in the semantic relation graph, spatial relation graph, and problem syntax tree, respectively. Then, based on relation-aware attention, neighbor features are aggregated to the current node, and the graph attention modules of the three relation spaces are trained separately.

[0083]

[0084]

[0085] in, represents the features of node i after aggregation through the RAHAN network; z can be a node representation in the problem syntax tree, or an entity representation in the semantic relation graph or spatial relation graph; rel(i,j) can represent the syntactic dependencies in the problem syntax tree, or a semantic relation or a spatial relation. Where W {·} V {·} It is a parameter matrix, U is a projection matrix, and b {·} , b′ {·} It is an offset item.

[0086] The input to the multi-head attention module of the modal query is the target aggregate entity representation. The output z of the semantic relation graph attention module sem The output z of the spatial relationship diagram module should be noted. spa and noun-type node representations that have already been assigned grammatical information. C represents the number of noun nodes in the sentence. First, a multi-head attention mechanism is used to calculate the cross-modal attention score between each noun node and the visual entity node, achieving inter-modal entity feature alignment and aggregation to obtain a multi-relational perceptual visual entity multi-modal representation:

[0087]

[0088]

[0089] Among them, z dep β represents the node features after multi-head attention aggregation across modalities. tj e represents the attention score aggregated at node i.j Represents the features of noun nodes, v t This represents the visual features aggregated through the RAHAN network; It is a projection matrix.

[0090] Get v t Syntax-aware aggregate representation z dep Subsequently, a problem-aware attention mechanism based on an additive model is designed to achieve z sem z spa z dep Fusion:

[0091]

[0092]

[0093] Among them, z out z represents the node representation of the final output of the network. rel This represents the parameter matrix corresponding to different dependencies; sem, spa, and dep represent syntactic relations, spatial relations, and dependency relations, respectively; γ rel Indicates the parameters corresponding to different dependencies; z i represents the attention score corresponding to different dependencies; q is the problem representation of syntactic information perception. It is a parameter matrix. It is an offset item.

[0094] Step 5: Predict the answer labels for relevant questions.

[0095] Visual entity representation based on problem representation q and K multi-relation perception. By fusing these two features through a top-down attention mechanism, a joint representation J is obtained, which is then used to predict the final answer, yielding the probability distribution of the answer:

[0096]

[0097]

[0098] Where f is the top-down attention fusion method, Θ is the trainable parameter of the multimodal fusion method, and MLP is a two-layer feedforward neural network that serves as the final classifier. This represents the final probability output for different answers.

[0099] Step 6: Train the neural network and use cross-entropy as the loss function to calculate the loss value.

[0100] Determine if the number of iterations is equal to the preset number of iterations (30 times). If the preset number of iterations is met, end the training process. If not, return to step 1 to continue the next round of iteration training.

[0101] Figure 2 The following is a block diagram illustrating the implementation of this method. The VQA scores of the prediction results obtained using this method are shown in Table 1.

[0102] Table 1 shows the experimental results of the comparison model on the VQA2.0 dataset.

[0103]

Claims

1. A multi-relation-aware heterogeneous graph visual question answering method of fusing syntax trees, characterized in that, The method comprises the following steps: Step 1: extracting image entity features and entity relationships, and constructing an entity semantic relationship graph and a spatial relationship graph; Step 1.1: identifying a set of entity objects and a set of bounding box feature vectors from the picture; Step 1.2: extracting semantic relationships between each entity object from the picture, and calculating the spatial relationship between the entity objects based on the bounding box feature vectors; Step 1.3: constructing a semantic relationship graph and a spatial relationship graph based on the entity semantic relationship and the entity spatial relationship; Step 2: constructing a question syntax tree and encoding the syntax tree information; Step 2.1: extracting the syntax dependency information and the part-of-speech information of the question from the question, and constructing a question syntax tree; Step 2.2: initializing the text representation of each word node of the question syntax tree, randomly initializing the network-learnable part-of-speech representation vector, and splicing the two to obtain the tree node representation; Step 2.3: learning the syntax tree to obtain the word representation with syntax structure perception, and finally extracting the question representation and the word representation with syntax information perception; Step 3: constructing a syntax-aware multi-modal heterogeneous graph, and extracting fine-grained entity representation and question representation with syntax information perception; Step 3.1: taking the noun nodes in the question syntax tree as a bridge to construct a multi-modal heterogeneous graph connecting the question syntax tree and the entity relationship graph; Step 3.2: extracting the entity representation with syntax information perception from the multi-modal heterogeneous graph, which not only includes the semantic relationship graph and the spatial relationship graph of the picture modality, but also combines the question syntax tree to guide the feature transmission between visual entities according to different relationships, and finally obtains the fine-grained entity representation with syntax information perception; Step 4: calculating the joint features of the visual picture and the question text; Step 5: predicting the answer label of the related question; Step 6: training the neural network, and using cross-entropy as the loss function to calculate the loss value; The number of iterations is set in advance, if the number of iterations meets the set number, the training process is ended, and the rumor authenticity label is predicted; if the number of iterations does not meet the set number, return to step 1 and continue the next round of training.

2. The multi-relation-aware heterogeneous graph visual question answering method of fusing syntax trees according to claim 1, wherein, In step 1, a Faster R-CNN network is used to identify a set of entity objects Each entity object v i has a visual feature vector and a bounding box feature vector Each bounding box feature vector b i = [x, y, w, h] corresponds to a four-dimensional space coordinate, (x, y) represents the coordinates of the upper left corner of the bounding box, (w, h) represents the width w and height h of the bounding box; For semantic relation extraction, input image Region feature extraction is performed by a Faster R-CNN network to obtain v i , v j , and a joint bounding box v ij of the two i Features v j , v ij are obtained by ROI pooling operation to realize region feature screening v′ i , v′ j , v′ ij Finally, the features are spliced and input into a multilayer perceptron to obtain a relation probability distribution p ij , which is calculated as follows: v′ i , v′ j , v′ ij = R(v i , v j , v ij ) (2) p ij = Softmax(W[v i ; v j ; v ij ] + b) (3) wherein, represents a Faster-RCNN network, R() represents an ROI pooling function, W represents a parameter matrix, and b represents a bias term. For spatial relation extraction, the spatial relation is defined as <v i -v j > which represents the geometric position of object j relative to object i; according to the intersection over union IoU, the relative distance and the angle of the two object bounding boxes; given two entity objects v i and v j , whose positions are represented by the center coordinates (x i , y i ), (x j , y j ) respectively, the IoU between v i and v j , the relative distance d ij and the relative angle θ ij are obtained: where d ij denotes the distance between the two object centroids, θ ij denotes the angle between the vector from v i centroid to v j centroid and the positive x-axis, d diag denotes the length of the diagonal of the entire matrix containing v i and v j , S IntSec denotes the size of the intersection area between v i and v j , and S Union denotes the size of the union area, and index denotes the category of the spatial relationship. If v i Full coverage v j Or v i Completely v j Coverage, then spatial relationship <v i v j > These represent "inside" and "covering" relationships, respectively; if v i and v j If the IoU between them is ≥0.5, then the spatial relationship is... <v i v j > indicates an "overlapping" relationship; if v i and v j φ between ij If v < 0.5 and IoU < 0.5, then the category to which the spatial relationship belongs is selected based on the index size; if v i and v j φ between ij If the IoU is greater than 0.5 and the value of v is less than 0.5, do not construct v. i With v j Spatial relationships between them.

3. The method of claim 1, wherein the fusion of syntax tree and multi-relation aware heterogeneous graph visual question answering is characterized by, In Step 2, the original question sequence As input, the part-of-speech (POS) tag sequence T = (t1, t2,..., t N ) and the dependency set D = (d1, d2,..., d where dependency is represented by <q i , q j > is obtained by Stanford Parser. First, the part-of-speech representation dictionary is randomly initialized to obtain the part-of-speech representation vectors learnable by the network, and the part-of-speech representation sequence T = (t1, t2, …, t N ) of the question is obtained. N ) Then, the position representation P = (p1, p2, …, p N ) of the word is calculated by the position encoding formula. After that, the Glove word embedding model is used to obtain a word representation sequence X = (x1, x2, …, xT) composed of representations of each word in the question N , denotes a vector space, d t denotes the vector dimension of the part-of-speech representation, d p denotes the vector dimension of the position representation, d x denotes the vector dimension of the word representation; Finally, the three different features are concatenated as the syntax-aware feature X for each word cat , which is computed as: T = RandomInit(t1, t2,..., t N ) (8) X = Glove(q1, q2,..., q N ) (9) wherein RandomInit denotes a random initialization function, Glove denotes a Glove word embedding model, pos denotes a position of the word in the sentence, sin denotes a sine function, denotes vector concatenation; Building a question syntax tree according to dependency relations A set of nodes representing a question syntax tree, where each node of the syntax tree is a word node, whose node representation set is X cat = (x'1, x'2,..., x'n), N ​ 4. The method of claim 1, wherein the fusion of syntax tree and multi-relation aware heterogeneous graph visual question answering is characterized by, from the question syntax tree in step 3 a syntax information-aware question representation and a syntax information-aware word representation, a set of nodes representing the question syntax tree, a set of dependency relations between words; First, the problem syntax tree is input into a syntax relation-aware graph neural network to learn syntax structure-aware node representations; according to the different categories of syntax dependency relations i , q j > categories, the relation-aware attention weight is adaptively calculated, and the multi-head attention mechanism is used to focus on the features of different vector spaces of each node: wherein, represents the node features of the aggregated i-node, M represents the number of multi-head attention heads, and a ij represents the attention weight score, and σ represents the sigmoid function, represents the number of neighbor nodes of the i-node, W dir(i,j) represents the relationship parameter matrix from the i-node to the j-node, b dep(i,j) represents the relationship bias term between the i-node and the j-node, b' dep(i,j) represents another relationship bias term between the i-node and the j-node, U represents the parameter matrix for calculating the attention weight score, T represents the matrix transpose, and V dir(i,j) represents the parameter matrix from the i-node to the j-node, and exp represents the exponential function; Representations of tree nodes In the input bidirectional GRU neural network, further extract sequence information perception problem representation h t : h t = BiGRU(X * ) (15) Wherein, BiGRU represents a bidirectional GRU neural network.

5. The method of claim 1, wherein the fusion of syntax tree and multi-relation aware heterogeneous graph visual question answering is characterized by, In step 4, RAHAN is a relationship-aware heterogeneous graph attention model, which is composed of a relationship-aware attention mechanism within the modality and a multi-head self-attention mechanism between the modalities; The isomeric graph is input into the RAHAM, and first, the inter-modal relationship perception attention mechanism calculates the relationship perception attention weight in each relationship space inside the semantic relationship graph, the spatial relationship graph and the question syntax tree respectively, and then aggregates the neighbor features to the current node based on the relationship perception attention. The graph attention modules of the three relationship spaces are trained separately: wherein, represents the i-node features aggregated through the RAHAN network; z is the entity representation in the node representation / semantic relationship graph and spatial relationship graph of the question syntax tree; rel(i,j) represents the syntax dependency relationship / semantic relationship / space relationship in the question syntax tree; wherein, W {·} , V {·} is a parameter matrix, U is a projection matrix, b {·} , b′ {·} is a bias term; The input of the multi-head attention module between modalities is a target aggregated entity representation The output z of the semantic relation graph attention module sem The output z of the spatial relation graph attention module spa And the nominal node representation that has been endowed with syntax-related information C represents the number of nominal nodes in the sentence; First, the multi-head attention mechanism is used to calculate the cross-modal attention score of each noun node and the visual entity node, to realize the alignment and aggregation of the entity features between the modalities to obtain the multi-relationship-aware visual entity multi-modal representation: where z dep denotes the node features aggregated by inter-modal multi-head attention, β tj denotes the attention score aggregated by node i, e j denotes the node features of the noun node, v t denotes the visual features aggregated by the RAHAN network; is a projection matrix; get v t syntax-aware aggregated representation z dep After that, design problem-aware attention mechanism based on additive model, realize z sem , z spa , z dep fusion: where z out denotes the node representation output by the network, z rel denotes the parameter matrix corresponding to different dependency relations; sem, spa, dep denote syntax relation, spatial relation, and dependency relation, respectively; γ rel denotes the parameter corresponding to different dependency relations; z i denotes the attention score corresponding to different dependency relations; q is the syntax information-aware question representation, is a parameter matrix, is a bias term.

6. The method of claim 1, wherein the method of fusing syntax trees of multi-relation-aware heterogeneous graph visual question answering is characterized by, In step 5, based on the question representation q and K multi-relation aware visual entity representations By fusing these two features through a top-down attention mechanism to get a joint representation J, and predicting the final answer, getting the probability distribution of the answer: where f is a top-down attention fusion method, Θ is the trainable parameters of the multi-modal fusion method, MLP is a two-layer feed-forward neural network as the final classifier; The final probability output representing different answers.

Images