A graph neural network-based chart question answering method

Abstract

The application discloses a kind of chart question and answer method based on graph neural network, comprising the following steps: a) using visual graph neural network and bidirectional long short-term memory network respectively modeling chart and text two kinds of modal characteristics, then the chart and text two kinds of characteristics representation obtained after splicing are carried out cross-modal feature fusion;B) using 1X1 convolution kernel and bidirectional GRU to the cross-modal characteristic representation obtained after splicing is carried out first stage feature fusion, obtain low-order cross-modal characteristic representation;C) using multi-head self-attention mechanism and densely connected graph convolution network to the low-order cross-modal characteristic representation is carried out second stage feature fusion, extract cross-modal semantic dependency, obtain high-order cross-modal characteristic representation;D) high-order cross-modal characteristic representation is input into a fully connected network, finally input classifier again to predict question answer.The application compared with prior art has strong feature extraction capability, cross-modal information fusion degree is high, can improve chart question and answer accuracy to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of visual question answering related to multimodal data fusion research, specifically to a graph question answering method based on graph neural networks. Background Technology

[0002] Visual question answering is a challenging and practical task in the field of artificial intelligence. It is a multimodal task, primarily involving an interdisciplinary research direction of computer vision and natural language processing. Chart question answering (CQA) is a type of visual question answering task, a multimodal reasoning task used to handle chart images and text-based questions. Chart question answering refers to the process where, given a chart and its associated question, the goal is to combine the visual information of the chart with the textual content of the question, obtaining the answer through feature fusion and reasoning.

[0003] Early research on graph-based question answering primarily employed simple combinations of neural networks and single cross-modal interaction methods. For example, the question text representation modeled by a unidirectional long short-term memory network was concatenated with image features extracted by a convolutional neural network before being input into a classifier. Another approach, based on simple relational networks, treated each pixel in the feature map obtained through a convolutional neural network as an object, then paired the feature vectors of every two pixels to construct a relational feature pair containing all relationships between pixels. A fully connected network was then used to infer the relationships between each pair of objects, and finally, these relational pairs were summed to predict the final answer. These models based on simple feature combinations often only modeled low-level image and text information and contained a lot of redundant information, resulting in poor performance in practice.

[0004] In recent years, researchers have designed some novel algorithms to improve the performance of chart question answering tasks. For example, traditional attention mechanisms with multiple iterations are used to collect features of key areas in charts; other methods use pre-trained open-source optical character recognition (OCR) to mine character information in charts, then locate it in the question, and use a pre-trained ResNet-152 to extract chart feature maps. Finally, spatial attention mechanisms are used to add the feature maps as hidden layer information to a long short-term memory network to obtain text representations. However, the above methods have difficulty modeling the structured features in charts, especially the contrast relationships between elements within the chart, such as size, proportion, and line trends. They also have difficulty extracting high-order semantic relationships between charts and question keywords, and cross-modal features cannot be fully interacted. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a chart question-answering method based on graph neural networks. To model the structured features in charts, especially the comparative relationships between elements within the chart, such as size, proportion, and line trends, this method uses a visual graph neural network to extract chart features. To extract the high-order semantic relationships between the chart and question keywords, enabling full interaction and fusion of cross-modal features, this method proposes a two-stage cross-modal fusion process.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A graph question answering method based on graph neural networks includes the following steps:

[0008] (1) Visual graph neural network and bidirectional long short-term memory network are used to extract feature representations of image and text modalities respectively, and the two modal feature representations are aligned and concatenated, including the following sub-steps:

[0009] (1.1) The scientific charts collected from the FigureQA dataset, namely vertical bar charts, horizontal bar charts, line charts, pie charts and scatter plots, along with the questions corresponding to the scientific charts and the manually annotated answers, are used as the training set and the validation set. The ratio of the training set to the validation set is 8:2. The manually annotated answers are the answers given to the questions.

[0010] (1.2) Visual graph neural networks are used to model the visual information of scientific charts. First, the entire scientific chart is decomposed into multiple sub-charts by gridding. Then, the RGB three-channel matrix of each sub-chart is flattened in the channel direction, and each sub-chart is represented by a vector. Each sub-chart is regarded as a node. The K nearest neighbor algorithm is used to calculate the K nearest nodes to the current node, and the current node is established with these K nodes. This process is iterated to construct the dependency matrix between each sub-chart and other sub-charts. Then, all sub-chart representation vectors and dependency matrix are input into the graph neural network to model the feature representation of the image.

[0011] (1.3) Preprocess the input text; the word vectors corresponding to each word in the question are obtained using BERT, and then the entire question is input into a bidirectional long short-term memory network for sequence modeling to obtain the text feature representation;

[0012] (1.4) Align the channel dimension of the image feature representation with the word dimension of the text feature representation and then concatenate them to obtain the cross-modal feature representation.

[0013] (2) Perform first-stage feature fusion on the cross-modal feature representation to obtain a low-order cross-modal feature representation, including the following sub-steps:

[0014] (2.1) Perform the first-stage feature fusion on the cross-modal feature representation; use a convolutional neural network with a 1x1 kernel to extract features from the cross-modal feature representation;

[0015] (2.2) Input the cross-modal feature representation extracted by the convolutional neural network with a convolutional kernel of 1x1 into the bidirectional GRU network to perform cross-modal feature fusion and obtain a low-order cross-modal feature representation.

[0016] (3) Perform a second-stage feature fusion on the low-order cross-modal feature representation to obtain the high-order cross-modal feature representation, including the following sub-steps:

[0017] (3.1) Perform a second-stage feature fusion on the low-order cross-modal feature representation; use a multi-head self-attention mechanism to generate multiple self-attention matrices based on the number of heads for the low-order cross-modal feature representation obtained after the first-stage feature fusion; perform average pooling on the multiple self-attention matrices to obtain the interactive attention weight matrix;

[0018] (3.2) Input the low-order cross-modal feature representation obtained after the first stage feature fusion and the interaction attention weight matrix into a densely connected deep graph convolutional network to model the semantic dependency relationship between text and image, and obtain a mid-order cross-modal feature representation.

[0019] (3.3) Perform average pooling on the intermediate-order cross-modal feature representation to obtain the higher-order cross-modal feature representation.

[0020] (4) Input the higher-order cross-modal feature representation into the classifier to obtain the prediction result, including the following sub-steps:

[0021] (4.1) Input the high-order cross-modal feature representation into a fully connected network, and then input it into a classifier to obtain the prediction result.

[0022] Compared with the prior art, the present invention, employing the above technical solution, has the following beneficial effects:

[0023] This invention proposes a chart question answering method based on graph neural networks. This invention uses visual graph neural networks to model the dependency relationships between different sub-image blocks in a chart image, which can better extract the structured features in the chart, especially the contrast relationships between elements within the chart, such as size, proportion, and line trend.

[0024] The two-stage cross-modal feature fusion mechanism proposed in this invention enables cross-modal features to interact fully, and can better uncover the high-order semantic relationships between charts and question keywords. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention;

[0026] Figure 2 is a flowchart of an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0028] like Figure 1 As shown, the graph question answering method based on graph neural networks of the present invention includes the following steps:

[0029] (1) Visual graph neural network and bidirectional long short-term memory network are used to extract feature representations of image and text modalities respectively, and the two modal feature representations are aligned and concatenated, including the following sub-steps:

[0030] (1.1) The scientific charts collected from the FigureQA dataset, namely vertical bar charts, horizontal bar charts, line charts, pie charts and scatter plots, and the questions corresponding to the scientific charts and the manually annotated answers, are used as the training set and the validation set, with a ratio of 8:2; wherein, the manually annotated answers are the answers given to the questions.

[0031] (1.2) Visual graph neural networks are used to model the visual information of scientific charts. First, the entire scientific chart is decomposed into multiple sub-charts by gridding. Then, the RGB three-channel matrix of each sub-chart is flattened in the channel direction, and each sub-chart is represented by a vector. Each sub-chart is regarded as a node. The K nearest neighbor algorithm is used to calculate the K nearest nodes to the current node, and the current node is established with these K nodes. This process is iterated to construct the dependency matrix between each sub-chart and other sub-charts. Then, all sub-chart representation vectors and dependency matrix are input into the graph neural network to model the feature representation of the image.

[0032] (1.3) Preprocess the input text; the word vector corresponding to each word in the question is obtained using BERT, and then the entire question is input into a bidirectional long short-term memory network for sequence modeling to obtain the text feature representation.

[0033] (1.4) Align the channel dimension of the image feature representation with the word dimension of the text feature representation and then concatenate them to obtain the cross-modal feature representation.

[0034] (2) Perform first-stage feature fusion on the cross-modal feature representation to obtain a low-order cross-modal feature representation, including the following steps:

[0035] (2.1) Perform the first stage of feature fusion on the cross-modal feature representation; use a convolutional neural network with a 1x1 kernel to extract features from the cross-modal feature representation.

[0036] (2.2) Input the cross-modal feature representation extracted by the convolutional neural network with a convolutional kernel of 1x1 into the bidirectional GRU network to perform cross-modal feature fusion and obtain a low-order cross-modal feature representation.

[0037] (3) Perform a second-stage feature fusion on the low-order cross-modal feature representation to obtain the high-order cross-modal feature representation, including the following sub-steps:

[0038] (3.1) Perform a second-stage feature fusion on the low-order cross-modal feature representation; use a multi-head self-attention mechanism to generate multiple self-attention matrices based on the number of heads for the low-order cross-modal feature representation obtained after the first-stage feature fusion; perform average pooling on the multiple self-attention matrices to obtain the interactive attention weight matrix.

[0039] (3.2) The low-order cross-modal feature representation obtained after the first stage feature fusion and the interaction attention weight matrix are input into a densely connected deep graph convolutional network to model the semantic dependency relationship between text and image, and obtain a mid-order cross-modal feature representation.

[0040] (3.3) Perform average pooling on the intermediate-order cross-modal feature representation to obtain the higher-order cross-modal feature representation.

[0041] (4) Input the higher-order cross-modal feature representation into the classifier to obtain the prediction result, including the following sub-steps:

[0042] (4.1) Input the high-order cross-modal feature representation into a fully connected network, and then input it into a classifier to obtain the prediction result. Example

[0043] See Figure 2 In this embodiment, scientific charts and corresponding questions collected from the FigureQA dataset are preprocessed and then input into a visual graph neural network and a bidirectional long short-term memory network to obtain two modal features: chart and text. These two feature representations are then concatenated and subjected to cross-modal feature fusion. A 1x1 convolutional kernel and a bidirectional GRU are used for the first-stage feature fusion of the concatenated cross-modal feature representation to obtain a low-order cross-modal feature representation. A multi-head self-attention mechanism and a densely connected graph convolutional network are used for the second-stage feature fusion of the low-order cross-modal feature representation to extract cross-modal semantic dependencies. Average pooling is then performed to obtain a high-order cross-modal feature representation. The high-order cross-modal feature representation vector is then input into a fully connected network, and the output is the question-answering prediction result.

[0044] The above description is only a preferred embodiment of the present invention. Modifications may be made within the scope defined by the claims of the present invention, but all such modifications shall fall within the protection scope of the present invention.

Claims

1. A graph neural network-based graph question answering method, characterized in that, The method comprises the following specific steps: Step 1: the visual graph neural network and the bidirectional long short-term memory network are used to extract the image and the text two modal feature representations respectively, and the two modal feature representations are aligned and spliced to obtain a cross-modal feature representation; Step 2: the cross-modal feature representation is subjected to first-stage feature fusion to obtain a low-order cross-modal feature representation; Step 3: the low-order cross-modal feature representation is subjected to second-stage feature fusion to obtain a high-order cross-modal feature representation; Step 4: the high-order cross-modal feature representation is input into a classifier to obtain a question and answer prediction result; wherein: The step 1 is specifically: 1.1: scientific charts such as vertical column charts, horizontal column charts, line charts, pie charts, scatter charts and questions and answers annotated by artificial labeling corresponding to the scientific charts are taken as a training set and a verification set, and the training set and the verification set are collected in a ratio of 8:2; wherein the artificial labeling is an answer given to the question; 1.2: the visual information of the scientific chart is modeled by using a visual graph neural network; first, the whole scientific chart is grid processed and decomposed into a plurality of sub-charts, then the RGB three-channel matrix of each sub-chart is flattened in the channel direction, and each sub-chart is represented as a vector; each sub-chart is regarded as a node, the K nearest neighbor algorithm is used to calculate the K nodes closest to the current node, and the current node and the K nodes are established corresponding dependency relationship; iteration is performed to construct the dependency relationship matrix between each sub-chart and other sub-charts, and then all sub-chart representation vectors and dependency relationship matrices are input into a graph neural network to model the feature representation of the image; 1.3: the input text is preprocessed; the word vector corresponding to each word in the question is obtained by using BERT, and then the whole question is input into a bidirectional long short-term memory network for sequence modeling to obtain a text feature representation; 1.4: the channel dimension of the image feature representation is aligned with the word dimension of the text feature representation, and then spliced to obtain a cross-modal feature representation; The step 2 is specifically: 2.1: the cross-modal feature representation is subjected to first-stage feature fusion; a convolutional neural network with a convolution kernel of 1x1 is used to extract the cross-modal feature representation; 2.2: the cross-modal feature representation extracted by the convolutional neural network with the convolution kernel of 1x1 is input into a bidirectional GRU network for cross-modal feature fusion to obtain a low-order cross-modal feature representation; The step 3 is specifically: 3.1: the low-order cross-modal feature representation is subjected to second-stage feature fusion; the low-order cross-modal feature representation obtained after the first-stage feature fusion is used for multi-head self-attention mechanism, a plurality of self-attention matrices are generated according to the number of heads; the plurality of self-attention matrices are subjected to average pooling to obtain an interactive attention weight matrix; 3.2: the low-order cross-modal feature representation obtained after the first-stage feature fusion and the interactive attention weight matrix are input into a densely connected deep graph convolutional network to model the semantic dependency relationship between the text and the image to obtain a medium-order cross-modal feature representation; 3.3: the medium-order cross-modal feature representation is subjected to average pooling to obtain a high-order cross-modal feature representation.

Images