Attention mechanism based multi-view relationship network graph question answering method and system

Abstract

The application provides a multi-view relationship network chart question and answer method based on an attention mechanism, and the method comprises the following steps: S1, acquiring a data set required to be processed; S2, inputting a chart image and a corresponding text question in the data set as input items respectively; S3, configuring a fusion reasoning algorithm model to output a final result. By improving the image encoder model of the traditional relationship network, an effective transformer attention model CoT module is introduced, and the problem that the extraction capability of RN for image feature information is limited is solved; a novel multi-view relationship module is proposed, the pairing process is improved based on pixel points and channel information, the problem that RN regards all feature vectors as equally important without highlighting more effective relationship image feature pairs is solved, and the problem that the overall information of each channel of the image is ignored in the traditional RN pairing process is solved.

Description

Technical Field

[0001] This invention belongs to the technical field of information processing, specifically relating to a multi-view relationship network graph question-answering method and system based on an attention mechanism. Background Technology

[0002] Relationship networks, as one of the best-performing benchmark models for graph question answering tasks, were first proposed by Google's DeepMind team to improve visual reasoning tasks and have demonstrated powerful performance on the CLEVR dataset.

[0003] In recent years, many studies have addressed the problems existing in relational networks. Jialong Zou et al. proposed a relational network driven by an affinity mechanism, which proposes an affinity measurement method between feature vector pairs. The aim is to rank all feature vector pairs in an image based on affinity, thereby removing half of the feature vector pairs with lower affinity and thus improving the performance of the relational network. However, brute-force removal of half of the feature vector pairs often leads to the loss of some graph information; therefore, a more efficient relational pairing method needs to be found.

[0004] Furthermore, traditional relationship networks perform relationship pairing based on the pixel dimension, which often leads to the neglect of overall information for each channel. In other words, the feature map of each channel typically represents the overall image features of a particular dimension of the graph. If we consider the feature map of each channel as an object, pairing relationships between objects at the channel dimension might provide more dimensional information for image question answering inference, thereby improving the performance of the relationship network.

[0005] In view of this, it is very meaningful to propose a multi-view relational network graph question answering method and system based on attention mechanism. Summary of the Invention

[0006] To address the problems of inefficiency in existing relationship matching methods and insufficient performance of relationship networks, this invention provides a multi-view relationship network graph question-answering method and system based on an attention mechanism to solve the aforementioned technical deficiencies.

[0007] Firstly, this invention proposes a multi-view relational network graph question-answering method based on an attention mechanism, which includes the following steps:

[0008] S1. Obtain the dataset to be processed;

[0009] S2. Input the charts and images in the dataset and the corresponding text questions as input items respectively;

[0010] S3. Configure the fusion inference algorithm model to output the final result.

[0011] Preferred options also include:

[0012] S31. Configure the multi-view relationship module to assign different weights to each relationship feature;

[0013] S32. Obtain the intermediate features F from the two MLP layers. M And so on, to get the final result.

[0014] Preferably, the chart image is used to extract image features using an image coding model based on the CoT module, and the feature map... Represented as:

[0015] X i =Conv(X i-1 X0=I,i∈{1,2,3}

[0016] X i =CoTBlock(X) i-1 ), i∈{4,5}

[0017] Where X0 is the initial input image, Conv represents the convolutional layer, and CoTBlock represents the CoT module.

[0018] Preferably, the text problem obtains text vectors through a bidirectional LSTM model, represented as:

[0019] h t =LSTM(x t ), 1≤t≤T

[0020] Where LSTM represents the LSTM model, h t x represents the representation vector of each word after passing through the LSTM model. t This represents the vocabulary vector after one-hot encoding.

[0021] Preferably, the final feature O of the relation pairing in S3 is represented as follows:

[0022]

[0023]

[0024]

[0025] Where, p (i,j),(u,v) It is the concatenation of the corresponding object vector, position information, and problem representation vector q, denoted as p. (i,j),(u,v) =[f i,j ,i,j,f u,v ,u,v,q];l(k),(w) It is the concatenation of the corresponding object vector and the problem representation vector q, denoted as l. (k),(w) =[f k ,f w ,q];g θ This indicates a shared MLP layer, and d=256 represents the output dimension of the MLP layer.

[0026] Further preferred embodiments include: constructing a wise aggregation submodule, which compresses the feature set O through max pooling and average pooling operations to obtain two description vectors. and

[0027]

[0028]

[0029] Use a shared MLP layer to compute the attention graph for each relation pair. Multiplying O by A yields attention-based object pair features O. a ;

[0030] Attention-based object pair features O a It is expressed as follows:

[0031]

[0032] O a =O*A

[0033] Where σ represents the sigmoid activation function, and W0 and W1 represent the weight matrices of the two MLP layers.

[0034] Further preferred embodiments include: focusing the attention-treated object pair features O a By averaging, we obtain the intermediate feature F. M ∈R d .

[0035] Secondly, this invention also discloses a multi-view relational network graph question-answering system based on an attention mechanism, comprising:

[0036] Image representation learning module: used to extract image features;

[0037] CoT module: used to extract more effective image features;

[0038] Problem representation learning module: used for extracting text features;

[0039] Multi-view relationship module: used for pixel-based relationship pairing and channel-based relationship pairing;

[0040] The wise aggregation submodule is used to assign different weights to each relation feature.

[0041] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0042] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0043] Compared with the prior art, the beneficial results of the present invention are as follows:

[0044] (1) By improving the traditional relational network image encoder model and introducing the effective transformer attention model CoT module, the problem of the limited ability of RN to extract image feature information is solved.

[0045] (2) The present invention also proposes a novel multi-view relationship module, which improves the pairing process based on pixel and channel information, and solves the problem that RN treats all feature vector pairs as equally important and does not highlight the role of more effective relationship image feature pairs, and the problem that the traditional RN pairing process ignores the overall information of each channel of the image. Attached Figure Description

[0046] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0047] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied;

[0048] Figure 2 This is a flowchart illustrating an attention-based multi-view relationship network graph question-answering system according to an embodiment of the present invention.

[0049] Figure 3 This is a flowchart illustrating the attention-based multi-view relationship network graph question-answering method according to an embodiment of the present invention.

[0050] Figure 4This is a schematic diagram of the structure of the attention-based multi-view relational network graph question-answering method according to an embodiment of the present invention;

[0051] Figure 5 This is a diagram illustrating the overall framework of the MVARN model in the attention-based multi-view relational network graph question answering method of this invention.

[0052] Figure 6 This is a structural diagram of the CoT module in the attention-based multi-view relational network graph question answering method according to an embodiment of the present invention;

[0053] Figure 7 This is a structural diagram of the multi-view relationship module in the attention-based multi-view relationship network graph question answering method according to an embodiment of the present invention;

[0054] Figure 8 This is a structural diagram of the wiseaggregation submodule in the attention-based multi-view relational network graph question answering method of this invention.

[0055] Figure 9 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0056] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0057] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0058] Figure 1 An exemplary system architecture 100 for processing information, or for processing information, to which embodiments of the present invention can be applied, is shown.

[0059] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0061] Terminal devices 101, 102, and 103 can be various electronic devices with communication functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0062] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain processing results (such as verification success information used to indicate that the verification request is a valid request).

[0063] It should be noted that the information processing method provided in the embodiments of the present invention is generally executed by server 105, and correspondingly, the device for processing information is generally disposed in server 105. Furthermore, the information sending method provided in the embodiments of the present invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the device for sending information is generally disposed in terminal devices 101, 102, and 103.

[0064] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0065] Figure 2 The following is a flowchart illustrating an embodiment of the present invention that discloses a multi-view relationship network graph question-answering system based on an attention mechanism, as shown in the figure. Figure 2 As shown, it includes:

[0066] Image representation learning module 21: used for extracting image features;

[0067] CoT module 22: Used to extract more effective image features;

[0068] Problem representation learning module 23: used for extracting text features;

[0069] Multi-view relationship module 24: used for pixel-based relationship pairing and channel-based relationship pairing;

[0070] Wise aggregation submodule 25: Used to assign different weights to each relation feature.

[0071] Figure 3 The following is a flowchart illustrating an embodiment of the present invention that discloses a multi-view relationship network graph question-answering method based on an attention mechanism, as shown in the figure. Figure 3 and Figure 4 As shown, the method includes the following steps:

[0072] S1. Obtain the dataset to be processed;

[0073] S2. Input the charts and images in the dataset and the corresponding text questions as input items respectively;

[0074] S3. Configure the fusion inference algorithm model to output the final result;

[0075] Specifically, the fusion inference algorithm model includes the MLP splitter model.

[0076] S31. Configure the multi-view relationship module to assign different weights to each relationship feature;

[0077] S32. Obtain the intermediate features F from the two MLP layers. M And so on, to get the final result.

[0078] Specifically, this invention proposes a novel graph question-answering model based on relational networks to improve the performance of graph question-answering tasks, named Multi-view Attention Relation Network (MVARN). This invention uses the FigureQA graph question-answering dataset task as an example. The specific algorithmic details of the MVARN model are described below:

[0079] The overall framework diagram of the MVARN model is as follows: Figure 5 As shown, the framework comprises three modules: an image representation learning module, a problem representation learning module, and a multi-view relationship module.

[0080] In the image representation learning module, an image encoding model based on the CoT module is used to extract image features. The CoT module integrates context information mining and self-attention learning into a unified architecture, thereby extracting more useful feature map information. The structure diagram of the CoT module is shown below. Figure 6 As shown.

[0081] This image coding model can be divided into five stages, and Table 1 shows the specific structure of the image coding model based on the CoT module. In this invention, the feature maps output from the five stages are labeled as X1, X2, ..., X5, respectively.

[0082]

[0083] Table 1. Specific structure of the image coding model based on the CoT module.

[0084] Specifically, the first three stages also use convolutional layers with a kernel size of 3×3, a kernel number of 64, and a stride of 2. The resulting feature maps are obtained in this way. The dimensions are all from the previous feature map X. i-1 ∈R H*W*C Half of it. Feature map The formula is expressed as follows:

[0085] X i =Conv(X i-1 X0=I,i∈{1,2,3}

[0086] Where X0 is the initial input image and Conv represents the convolutional layer.

[0087] Furthermore, the CoT module is used in the last two stages, as shown in Table 1. Specifically, given the input feature map X∈R... H*W*C The keys, queries, and values ​​are represented as K = X, Q = X, V = XW, respectively. v The CoT module first performs 3×3 sets of convolutions on all adjacent keys within a 3×3 grid to contextualize each key representation. Then, K... 1 It is considered a static context representation of X. Then, with K... 1 Conditioned by the concatenation of W and Q, the attention matrix is ​​obtained through two 1×1 convolutions (where W... θ With ReLU activation function, W δ (No activation function):

[0088] A = [K] 1 ,Q]W θ W δ

[0089] Furthermore, based on A, feature map K2 It is calculated in a typical self-attention manner by aggregating all values ​​V:

[0090] K 2 =V*A

[0091] Among them, feature map K 2 This can be viewed as a dynamic context representation. The final output can be considered as K using an attention mechanism. 1 and K 2 The fusion. Then, the last two stages can be expressed by the following formula:

[0092] X i =CoTBlock(X) i-1 ), i∈{4,5}

[0093] Here, CoTBlock represents the CoT module. It's worth noting that each stage is followed by a Batch Normalization layer, which helps prevent overfitting and accelerates model training. Finally, the feature map F = X5 is obtained from the image encoding model based on the CoT module.

[0094] In the problem representation learning module, similar to traditional RN models, this invention combines all existing words to form a dictionary. This invention uses a simple bidirectional LSTM model with 512 hidden units to obtain the text vectors for the problem representation learning module:

[0095] h t =LSTM(x t ), 1≤t≤T

[0096] Here, LSTM represents the LSTM model, h t It is the representation vector of each word after passing through the LSTM model, x t The word vector after one-hot encoding is represented in this invention, where the representation vector of the last word is considered as the final problem representation vector, which can be expressed as q = h. T ∈R 512 .

[0097] In the multi-view relationship module, objects in multiple views are paired to make relationship features more specific, and a wise aggregation submodule is proposed to assign different weights to each relationship feature to improve inference accuracy.

[0098] In the initial relational network, the feature vector of each pixel in the feature map F is treated as a set containing n objects, and the relational pairings between the objects are calculated. However, in the field of computer vision, each channel of the feature map is often regarded as capturing a specific feature of a certain dimension of the image. Therefore, this invention proposes a multi-view relational module, namely, pixel-based relational pairing and channel-based relational pairing. The structure diagram of the multi-view relational module is as follows. Figure 7 As shown.

[0099] Specifically, in a pixel-based view, the set of objects can be represented as F1 = {f i，j |1≤i,j≤n}∈R (H*w)×C , where f i，j ∈R C Let F represent the i-th row and j-th column of feature map F, where n = H = W = 8 and C = 64. Furthermore, the set of object pairs can be represented as:

[0100] P = {p (i，j)，(u，v) |1≤i,j,u,v≤n}

[0101] Where, p (i，j)，(u，v) It is the concatenation of the corresponding object vector, position information, and problem representation vector q, denoted as p. (i，j)，(u，v) =[f i，j i, j, f u，v [u, v, q]. Each object is paired with all objects, including itself, represented as: p (1，1)，(1，1) p (1，1)，(1，2) , ..., p (n，n)，(n，n) .

[0102] In a channel-based view, the set of objects can be represented as F2 = {f k |1≤k≤C}∈R C×(H*w) , where f k ∈R h*w Let F represent the feature vector of the k-th channel after flattening. Furthermore, the set of object pairs in this view can be represented as:

[0103] L={l (k)，(w) |1≤k,w≤C}

[0104] Among them, l (k)，(w) It is the concatenation of the corresponding object vector and the problem representation vector q, denoted as l. (k)，(w) =[f k f w It is worth noting that because the features in the channel dimension have no positional relationship, positional information is not included in this view.

[0105] Furthermore, each object pair in both views is processed using a shared MLP layer to generate a feature representation of the relationship between the corresponding objects. These two feature dimensions are then concatenated to generate the final feature O representing the relationship pair. This process can be represented by the following formula:

[0106]

[0107]

[0108]

[0109] Among them, g θ This indicates a shared MLP layer, where d=256 represents the output dimension of the MLP layer. It is the range of the final feature O obtained by concatenating the relationship features based on the two dimensions of pixels and channels.

[0110] Because the importance of each feature in relational reasoning differs for each relation pair in O, this invention proposes a novel wise aggregation submodule to assign different weights to each feature, thereby improving the accuracy of reasoning. The structure diagram of the wise aggregation submodule is shown below. Figure 8 As shown.

[0111] In the wise aggregation submodule, max pooling and average pooling operations are first used to compress the feature set O, resulting in two description vectors. and

[0112]

[0113]

[0114] Then, a shared MLP layer is used to compute the attention graph for each relation pair. Finally, multiplying O by A yields the attention-based object pair feature O. a In this process, an MLP layer is used to calculate the importance of each object pair's features for relational reasoning, thereby generating an attention map. The larger the weight in the attention map, the more important the object pair it represents is to the features, and vice versa. This process can be represented by the following formula:

[0115]

[0116] O a =O*A

[0117] Here, σ represents the sigmoid activation function, and W0 and W1 represent the weight matrices of the two MLP layers. Then, this invention applies the attention-processed object pair features O... a The intermediate feature F is obtained by averaging. M ∈R d .

[0118] Finally, two MLP layers are used to process the intermediate features F. M The final result is obtained by using a 256 ReLU unit layer and a 25% dropout layer.

[0119] Specifically, in this embodiment of the invention, the experimental results are analyzed using the FigureQA chart question-answering dataset task as an example.

[0120] The overall accuracy performance was compared with other algorithm models. First, to verify the effectiveness of the proposed MVARN model for graph question answering tasks, this embodiment compared its performance with other advanced algorithms in recent years on the FigureQA dataset, including QUES (trained with only text), IMG+QUES (CNN+LSTM model), RN model, FigureNet model, LEAF-Net model, and ARN model.

[0121] Among them, QUES, IMG+QUES, and RN models were the earliest benchmark models used on the FigureQA dataset, with the RN model performing best. The FigureNet model proposes a multi-module algorithm framework to solve chart question-answering problems; this module is pre-trained on a large amount of labeled data to extract corresponding values ​​of features and elements, such as image colors, then obtains feature vectors, which are concatenated with the question's embedding representation; finally, inference is performed through a multi-layer fully connected neural network. However, due to limitations in model design, the FigureNet model can only be applied to bar charts and pie charts.

[0122] The LEAF-Net model utilizes many open-source pre-trained models. First, it identifies character information in images through optical character recognition, then locates it in the embedding problem. Simultaneously, it obtains image feature maps using a pre-trained ResNet. Finally, it adds these feature maps as hidden layer information to the LSTM text encoder using a spatial attention mechanism to obtain sentence-level representations. The ARN model proposes an affinity-driven relational network model and improves the image encoding module using the Unet model.

[0123] Table 2 shows the experimental results of the proposed MVARN model on two validation sets of the FigureQA dataset, comparing it with the methods described above. It can be seen that the model of this embodiment significantly outperforms the RN benchmark model. Specifically, it achieves 10.77% and 11.6% higher accuracy on the two validation sets, respectively. Furthermore, the overall accuracy of MVARN is also better than other state-of-the-art algorithms, exceeding the accuracy of the second-best ARN model by 1.68% and 1.19% on the two validation sets, respectively. This also demonstrates the effectiveness of the model of this embodiment in relation pairing based on multiple views. From a time and space complexity perspective, since the MVARN network uses a multi-view approach, it requires more training time and memory. However, compared to other networks, the results are better, so the increased time and space complexity is acceptable.

[0124] Model Validation set 1 Validation set 2 QUES - 50.01％ IMG+QUES 59.41％ 57.14％ RN 76.39％ 72.54％ FigureNet 83.90％ - LEAF-Net - 81.15％ ARN 85.48％ 82.95％ MVARN(ours) 87.16％ 84.14％

[0125] Table 2 compares the experimental results of other algorithms on the FigureQA dataset.

[0126] Performance comparison with other algorithm models on chart types. Furthermore, this embodiment of the invention also compares the accuracy differences with other algorithms on each chart type. Tables 3 and 4 show the comparison results on two validation sets. On both validation sets, it can be seen that the model of this embodiment of the invention achieves the best performance on all chart types except Vertical Bar. Specifically, compared with the ARN model, on the four chart types Horizontal Bar, Pie and Line, and DotLine, the accuracy of the MVARN model is improved by 0.20%, 5.60%, 3.85%, and 2.34% respectively on validation set 1; and by 0.61%, 6.44%, 2.85%, and 3.24% respectively on validation set 2. The FigureNet model achieves an average accuracy of approximately 83.9% on the three chart types of validation set 1, while the model of this embodiment of the invention achieves an accuracy of 89.68%, which is significantly higher than FigureNet.

[0127] Furthermore, it was found that the model in this embodiment of the invention exhibits significantly higher inference performance on bar charts and pie charts than on line charts. Analysis suggests this may be because line charts contain more complex information, and the questions are also more complex and harder to answer, thus bar charts and pie charts are more accurate, while line charts are relatively less accurate.

[0128] Model Vertical Bar Horizontal Bar Pie Line Dot Line Overall IMG+QUES 61.98％％ 62.44％ 59.63％ 57.07％ 57.35％ 59.41％ RN 85.71％ 80.60％ 82.56％ 69.53％ 68.51％ 76.39％ FigureNet 87.36％ 81.57％ 83.13％ - - - ARN 92.49％ 91.20％ 84.25％ 81.31％ 81.03％ 85.48％ MVARN 87.84％ 91.40％ 89.85％ 85.16％ 83.37％ 87.16％

[0129] Table 3 compares the performance of other algorithms on various problems in the FigureQA dataset validation set 1.

[0130] Model Vertical Bar Horizontal Bar Pie Line Dot Line Overall IMG+QUES 58.60％ 58.05％ 55.97％ 56.37％ 56.97％ 57.14％ RN 77.35％ 77.00％ 74.16％ 67.90％ 69.40％ 72.54％ ARN 90.46％ 89.56％ 80.29％ 77.60％ 78.28％ 82.95％ MVARN 84.19％ 90.17％ 86.73％ 80.45％ 81.52％ 84.24％

[0131] Table 4 compares the performance of other algorithms on various problems in the FigureQA dataset validation set 2.

[0132] Ablation experiment results for each module of the algorithm model. To further verify the importance of each module of the model in this embodiment of the invention, an ablation experiment was conducted by removing a portion of the model. The specific results of the ablation experiment are shown in Table 5. Here, "no CoT module" indicates that the CoT module was not used in the image encoding module, and "no multi-view relationship module" indicates that the multi-view relationship module was not used in this paper; instead, the relationship features were simply averaged.

[0133] As shown in Table 5, the accuracy of the model without the CoT module decreased by nearly 5% on both validation sets. Compared to the complete model on both validation sets, the accuracy of the model without the multi-view relation module decreased by approximately 4.75% and 3.85%, respectively. Based on the results of the ablation study, it is clear that using all modules can improve the overall performance of the model.

[0134] ablation model Validation set 1 Validation set 2 No CoT module 81.45％ 79.30％ No multi-view relationship module 82.41％ 80.39％ MVARN (Full Model) 87.16％ 84.24％

[0135] Table 5 shows the ablation experiments of the MVARN model on the FigureQA dataset.

[0136] Experimental results of the algorithm model on various problem types. Finally, the embodiments of this invention demonstrate the accuracy of the MVARN model on each problem type, as shown in Table 6. As can be seen from the table, the prediction accuracy of the MVARN model varies across different problem types. For example, the accuracy is above 90% for the three problem types "Is X the maximum?", "Is X less than Y? (bar, pie)", and "Is X greater than Y? (bar, pie)", while the accuracy for the two problem types "Is X the smoothest?" and "Is X the roughest?" is just above 60%.

[0137] Analysis of this invention's embodiments leads to the conclusion that questions like "Is X the smoothest?" are extremely complex, even for humans. Furthermore, this invention also found that even for the same question, the accuracy of the MVARN model varies depending on the type of chart. For example, the question "Is X less than Y?" achieves 92.66% accuracy on bar and pie charts in validation set 1, but only 85.80% accuracy on line charts. This further verifies that line charts are more difficult to answer than bar and pie charts because they contain more complex visual information. It also verifies that the model's performance differs across different chart types.

[0138]

[0139]

[0140] Table 6. Accuracy of the MVARN model on various questions in the FigureQA dataset.

[0141] Furthermore, to verify the scalability of the MVARN model in graph question answering tasks, this embodiment of the invention also conducted experimental verification on another open-source dataset, DVQA. Table 7 shows the comparison results of the MVARN model with other models.

[0142] As can be seen from the table, for algorithm models without OCR (Object Recognition System), the MVARN model outperforms the best baseline model SANDY (No OCR) by 10.1% and 9.77% in accuracy on the Test-Familiar and Test-Novel test sets, respectively, and outperforms the current best-performing ARN model by 1.62% and 1.4% in accuracy on the Test-Familiar and Test-Novel test sets, respectively.

[0143] For the Oracle-based algorithm model, the MVARN model outperformed the best benchmark model, SANDY (Oracle), by 24.37% and 24.49% in accuracy on the Test-Familiar and Test-Novel test sets, respectively, and outperformed the current best-performing ARN model by 1.42% and 1.53% on the Test-Familiar and Test-Novel test sets, respectively.

[0144] Overall, the MVARN model performs exceptionally well in DVQA, significantly outperforming its baseline model, SANDY. Notably, the MVARN model shows a much greater improvement with the Oracle version than with the OCR-free version, and all experimental results show higher performance with the Oracle version compared to the version without OCR.

[0145]

[0146] Table 7 Comparison of the MVARN model with other models on the DVQA dataset.

[0147] This invention improves the algorithm based on Relational Networks (RN) to more effectively extract visual information presented by charts. It improves the traditional image encoder model of Relational Networks by introducing an effective Transformer Attention (CoT) module, thus solving the problem of limited image feature extraction capability of RNs.

[0148] This invention also proposes a novel multi-view relationship module that improves the pairing process based on pixel and channel information. This solves the problem that RN treats all feature vector pairs as equally important and fails to highlight the more effective relationship image feature pairs, and the problem that the traditional RN pairing process ignores the overall information of each channel of the image.

[0149] The following is for reference. Figure 9 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 The diagram shows the structure of a computer device 600 (a server or terminal device). Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0150] like Figure 9 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 603 or programs loaded from storage section 609 into random access memory (RAM) 606. Various programs and data required for the operation of device 600 are also stored in RAM 604. The CPU 601, GPU 602, ROM 603, and RAM 604 are interconnected via bus 605. Input / output (I / O) interface 606 is also connected to bus 605.

[0151] The following components are connected to I / O interface 606: an input section 607 including a keyboard, mouse, etc.; an output section 608 including an LCD, speakers, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card, such as a LAN card or modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to I / O interface 606 as needed. A removable medium 612, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 611 as needed so that computer programs read from it can be installed into storage section 609 as needed.

[0152] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 610, and / or installed from removable medium 612. When the computer program is executed by central processing unit (CPU) 601 and graphics processing unit (GPU) 602, the functions defined in the methods of the present invention are performed.

[0153] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0154] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0156] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0157] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the method described in the first aspect.

[0158] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A multi-view relational network graph question-answering method based on an attention mechanism, characterized in that, The method includes the following steps: S1. Obtain the dataset to be processed; S2. Input the chart images and corresponding text questions in the dataset as input items respectively. For the chart images, an image encoding model with CoT module is used to extract image features. For the text questions, a bidirectional LSTM model is used to extract text features and obtain the text question representation vector. S3. Configure the fusion inference algorithm model to process the image features and text question representation vectors and output the final result. The fusion inference algorithm model includes a multi-view relationship module and a wise aggregation sub-module. The multi-view relationship module performs pixel-based and channel-based dual-view relationship pairing and concatenation to obtain a relationship feature set. The wise aggregation sub-module assigns different attention weights to each relationship feature in the relationship feature set, and then processes the attention-weighted features through the MLP layer to obtain the question answering result. The relationship feature set obtained by the multi-view relationship module O It is expressed as follows: in, It corresponds to the object vector, position information, and problem representation vector. q The splicing is represented as ; These correspond to the object vector and the problem representation vector. q The splicing is represented as ; Indicates a shared MLP layer. d =256 indicates the output dimension of the MLP layer.

2. The multi-view relational network graph question answering method based on attention mechanism according to claim 1, characterized in that, Also includes: S31. Configure the multi-view relationship module to assign different weights to each relationship feature; S32. Obtain intermediate features from two MLP layers. And so on, to get the final result.

3. The method according to claim 1, characterized in that, The image coding model that integrates the CoT module extracts feature maps of image features. Represented as: in, This is the initial input image, Conv represents the convolutional layer, and CoTBlock represents the CoT module.

4. The method according to claim 1, characterized in that, The process of extracting text features using the bidirectional LSTM model is represented as follows: in, LSTM Represents the LSTM model. This represents the representation vector of each word after passing through the LSTM model. This represents the vocabulary vector after one-hot encoding.

5. The method according to claim 1, characterized in that, The wise aggregation submodule assigns different attention weights to each relation feature, including: The relation feature set is compressed using max pooling and average pooling operations. O This yields two description vectors. and : Use a shared MLP layer to compute the attention graph for each relation pair. , O and A Multiplication yields attention-based object pair features. ; Attention-based object pair features It is expressed as follows: in, Represents the sigmoid activation function. and This represents the weight matrix of the two MLP layers.

6. The method according to claim 5, characterized in that, Also includes: Features of object pairs after attention By averaging, we can obtain the intermediate features. .

7. A multi-view relational network graph question-answering system based on an attention mechanism, characterized in that, For implementing the method as described in any one of claims 1-6, comprising: Image representation learning module: used to extract image features; CoT module: used to extract more effective image features; Problem representation learning module: used for extracting text features; Multi-view relationship module: used for pixel-based relationship pairing and channel-based relationship pairing; The wise aggregation submodule is used to assign different weights to each relation feature.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images