A capsule network structure suitable for numerical reasoning tasks

By introducing capsule network structure and bidirectional attention mechanism into the machine reading comprehension model, the problem of identifying legal numerical values ​​and distinguishing overlapping numerical semantics is solved, which significantly improves the accuracy of numerical inference tasks, especially in digital tasks and date calculations. .

CN113609272BActive Publication Date: 2024-11-19BEIJING NORMAL UNIV AT ZHUHAI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110926468.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-12
Publication Date
2024-11-19
Estimated Expiration
2041-08-12

AI Technical Summary

Technical Problem

Existing methods to improve the numerical reasoning capabilities of machine reading comprehension models are difficult to identify legal numerical values ​​distributed in multiple sentences and distinguish overlapping numerical semantics, resulting in insufficient accuracy.

Method used

Using a capsule network structure suitable for numerical inference tasks, including input layer, Roberta encoding layer, inference layer and problem prediction layer, legal numbers are identified through Roberta encoder and bidirectional attention network, and using the capsule network to gather the underlying features to construct a graph Structures conduct inference and control information flow through gated mechanisms to predict answers.

Benefits of technology

It effectively improves the model's ability to identify legal values ​​and distinguish overlapping numerical semantics, and improves the accuracy of numerical inference tasks, especially in digital tasks, date calculations and multi-segment extraction tasks, which outperforms traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure QLYQS_1
    Figure QLYQS_1
  • Figure QLYQS_2
    Figure QLYQS_2
  • Figure QLYQS_3
    Figure QLYQS_3
Patent Text Reader

Abstract

The present invention discloses a capsule network structure suitable for numerical reasoning tasks. The capsule network structure is provided with an input layer, a Robert encoding layer, an inference layer, a capsule network layer and a question prediction layer in sequence. A RoBerta neural network is introduced, which can identify legal numbers and important words. In addition, a bidirectional attention mechanism is introduced to finely capture the relationship between questions and related documents. At the same time, the capsule network is used to aggregate the underlying feature semantic features into high-level features, which is different from max-pooling, self-attention and other methods, and effectively extracts overlapping numerical semantic information. In addition, a gating mechanism is introduced to control the information flow of the capsule network and the RoBerta network, so as to participate in the prediction of the results and improve the efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of natural language processing in computer science, and in particular to a capsule network structure suitable for numerical reasoning tasks. [Background technology]

[0002] There are few existing methods to improve the numerical reasoning ability of machine reading comprehension models. The main method is to extract numbers from texts and questions, then use the type of numbers and the numerical relationship of numbers to construct a directed graph, and then use graph neural networks (graph attention networks) to calculate and reason on the directed graph, and finally use the calculation results to assist in the prediction of answers.

[0003] Although existing methods can improve the accuracy of machine reading comprehension to a certain extent, there are still two difficulties that have not been solved: 1) How to identify legal numerical values ​​distributed in multiple sentences. 2) For each question, how to distinguish overlapping numerical semantics. [Summary of the invention]

[0004] To solve the above problems, a capsule network structure suitable for numerical reasoning tasks is proposed;

[0005] A capsule network structure suitable for numerical reasoning tasks, characterized in that: the capsule network structure is provided with an input layer, a Robert coding layer, a reasoning layer, a capsule network layer and a question prediction layer in sequence;

[0006] The specific implementation steps are as follows;

[0007] S1. Question prediction layer: The description text and question are used as the input S of the whole model, which is divided into question sequence S q and the document sequence S d ;

[0008] S2, then use the Roberta encoder to encode the input layer to obtain the encoded representation H of the input layer;

[0009] S3, the inference layer and the capsule network layer take the encoded representation as input for computation;

[0010] The results of S4, the inference layer, and the capsule network layer are used as the input of the question prediction layer, and the final answer P(ans) is obtained through the question prediction layer.

[0011] Furthermore, the input layer adopts S q represents the problem sequence, S d Represents a sequence of documents;

[0012] The document sequence S d Contains m words; the question sequence S q Contains n words;

[0013] Use S for S d and S q The combined result is expressed as follows:

[0014]

[0015] Furthermore, the Roberta encoding layer uses the Roberta network as a basic encoder to encode the input S into H, that is:

[0016] H=RoBerta-Encoder(S) (4)

[0017] The outputs of the last four layers of the Roberta network structure are used as the input of the subsequent layers, namely:

[0018] RoBerta out =[H N-3 , H N-2 , H N-1 , H N ] (5).

[0019] Furthermore, a bidirectional attention network is provided inside the Roberta encoding layer, namely:

[0020]

[0021]

[0022]

[0023] in The hidden layer vector representing the problem, The hidden layer vector representing the document; the Attention mechanism uses the attention algorithm in the table to multiply the hidden layer vector by different matrices W Q , W K and W V As query, key and value vectors respectively.

[0024] Furthermore, the capsule of the capsule network is a set of vectors, wherein the direction of the capsule represents a specific type of digital feature, and the length of the capsule represents the possibility of the existence of these features;

[0025] The active capsules from the high-level layers are predicted through matrix transformation, and the high-level capsules are clustered by the low-level capsules, while the low-level capsules have more abstract features. When multiple predictions are consistent, the high-level capsules will take effect;

[0026] In the task of numerical reasoning, all the underlying semantic information extracted by the Roberta encoding layer is projected into a certain number of underlying capsules. Each vector represents a capsule, and each vector is activated by the nonlinear function Squash function. The Squash function can reduce the length of the vector to the range of 0 to 1, indicating the probability of the capsule, that is:

[0027]

[0028]

[0029] After the lower-level capsules capture the numerical semantics, the higher-level capsules It is obtained by collecting the information of the underlying capsules, namely:

[0030]

[0031] in, Represents the parameter, c uj represents the coupling coefficient obtained by dynamic routing;

[0032] Then, the mean-pooling operation is used to collect information from different capsule layers, namely:

[0033]

[0034] MP stands for mean-pooling operation. In order to fuse the information extracted by Roberta and capsule network, sigmoid is first applied to automatically calculate the weights of the two, namely:

[0035]

[0036]

[0037] Furthermore, the reasoning layer further enhances the reasoning ability of the model. It extracts all the numbers in the text and questions and constructs a graph structure. Each number represents a point, and the edges are divided into two types according to the size relationship of the values:

[0038] A) Greater than edge: When two nodes v i , v j ∈V, if v i >v j , that is, to establish a directed edge

[0039] B) Less than or equal to edge: When two nodes v i , v j ∈V, if v i ≤v j, that is, to create a directed edge

[0040] Furthermore, after constructing the graph, the inference layer first uses Roberta's output H for all nodes. N I(v)) is initialized, where I(v) represents the index position in the sentence. Then, the graph neural network is used for reasoning; and for each numerical node v i A weighted score is calculated, namely:

[0041]

[0042] Where W v represents the weight matrix, b v represents bias;

[0043] Then the information of each node is propagated to its neighboring nodes. Specifically, each node is updated according to the following formula:

[0044]

[0045] in is node v i Information representation; ji Represents and edge ji relevant relationship; Represents the specific transformation matrix of the relationship; N i ={j|(v j , v i )∈E} represents the node v i Neighbor nodes of

[0046] In order to fuse a node with its neighbor nodes, its information representation is fused with itself, namely:

[0047]

[0048] Where W f and b f is the weight matrix;

[0049] Expressing the above formula as a whole reasoning process, we can conclude that:

[0050] H G =Multi-Reasoning(G, V, H N ) (18).

[0051] Furthermore, before answering a question, the question-answering layer needs to predict the type of question, and then predict the answer based on the question type;

[0052] Specifically, using H G and To predict the type of question, that is:

[0053]

[0054] Where g represents the softmax function, which is used to calculate the probability of each type;

[0055] After obtaining the question type, predict the final answer, that is:

[0056] P(ans)=∑ type P(type)P(ans|type) (20).

[0057] The present invention proposes a new capsule network structure to solve two difficult problems in numerical reasoning tasks, namely, 1) how to identify legal numerical values ​​distributed in multiple sentences; 2) for each question, how to distinguish overlapping numerical semantics; for the first problem, the present invention is provided with a RoBerta neural network to identify legal numbers and important words. In addition, a bidirectional attention mechanism is set up to finely capture the relationship between questions and related documents; for the second problem, the capsule network is used to aggregate the underlying feature semantic features into high-level features. Different from max-pooling, self-attention and other methods, this method effectively extracts overlapping numerical semantic information; in addition, a gating mechanism is provided to control the information flow of the capsule network and the RoBerta network, so as to participate in the prediction of the results;

[0058] The present invention was tested on the DROP numerical reasoning dataset and judged by EM and F1 Score. Compared with the traditional NumNet+ network, the proposed capsule structure uses its unique dynamic routing mechanism to improve the feature screening ability and further improve the performance of the model. Compared with previous methods, it has a certain degree of performance improvement in digital tasks, segment extraction, date calculation, and multi-segment extraction tasks.

[0059] This paper introduces a bidirectional attention mechanism, which can enhance the semantic understanding and information fusion between questions and sentence pairs compared to previous attention mechanisms. Therefore, it has a certain degree of performance improvement in digital tasks, segment extraction, date calculation, and multi-segment extraction tasks.

[0060] The invention is an improvement on the model structure. The capsule network and gating mechanism can help improve the ability to extract complex information. There are no special requirements for the input text. Therefore, it has a certain versatility and can be applied to complex machine reading comprehension tasks.

Brief Description of the Drawings

[0061] Figure 1 Schematic diagram of the capsule network structure of the present invention;

[0062] Figure 2 This is a model architecture diagram of the present invention;

[0063] Figure 3 This is a schematic diagram of parameter settings of the first embodiment of the present invention;

[0064] Figure 4 This is a comparison chart of experimental result data of Example 1 of the present invention;

[0065] Figure 5 This is a data comparison chart of the F1 score value of the date class processed in Example 1 of the present invention. [Specific implementation method]

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] A capsule network structure suitable for numerical reasoning tasks, characterized in that: the capsule network structure is provided with an input layer, a Robert coding layer, a reasoning layer, a capsule network layer and a question prediction layer in sequence;

[0068] The specific implementation steps are as follows;

[0069] S1. Question prediction layer: The description text and question are used as the input S of the whole model, which is divided into question sequence S q and the document sequence S d ;

[0070] S2, then use the Roberta encoder to encode the input layer to obtain the encoded representation H of the input layer;

[0071] S3, the inference layer and the capsule network layer take the encoded representation as input for computation;

[0072] The results of S4, the inference layer, and the capsule network layer are used as the input of the question prediction layer, and the final answer P(ans) is obtained through the question prediction layer.

[0073] Furthermore, the input layer adopts S q represents the problem sequence, S d Represents a sequence of documents;

[0074] The document sequence S d Contains m words; the question sequence S q Contains n words;

[0075] Use S for S d and S qThe combined result is expressed as follows:

[0076]

[0077] Furthermore, the Roberta encoding layer uses the Roberta network as a basic encoder to encode the input S into H, that is:

[0078] H=RoBerta-Encoder(S) (4)

[0079] The outputs of the last four layers of the Roberta network structure are used as the input of the subsequent layers, namely:

[0080] RoBerta out =[H N-3 , H N-2 , H N-1 , H N ] (5).

[0081] Furthermore, a bidirectional attention network is provided inside the Roberta encoding layer, namely:

[0082]

[0083]

[0084]

[0085] in The hidden layer vector representing the problem, The hidden layer vector representing the document; the Attention mechanism uses the attention algorithm in the table to multiply the hidden layer vector by different matrices W Q , W K and W V As query, key and value vectors respectively.

[0086] Furthermore, the capsule of the capsule network is a set of vectors, wherein the direction of the capsule represents a specific type of digital feature, and the length of the capsule represents the possibility of the existence of these features;

[0087] The active capsules from the high-level layers are predicted through matrix transformation, and the high-level capsules are clustered by the low-level capsules, while the low-level capsules have more abstract features. When multiple predictions are consistent, the high-level capsules will take effect;

[0088] In the task of numerical reasoning, all the underlying semantic information extracted by the Roberta encoding layer is projected into a certain number of underlying capsules. Each vector represents a capsule, and each vector is activated by the nonlinear function Squash function. The Squash function can reduce the length of the vector to the range of 0 to 1, indicating the probability of the capsule, that is:

[0089]

[0090]

[0091] After the lower-level capsules capture the numerical semantics, the higher-level capsules It is obtained by collecting the information of the underlying capsules, namely:

[0092]

[0093] in, Represents the parameter, c uj represents the coupling coefficient obtained by dynamic routing;

[0094] Then, the mean-pooling operation is used to collect information from different capsule layers, namely:

[0095]

[0096] MP stands for mean-pooling operation. In order to fuse the information extracted by Roberta and capsule network, sigmoid is first applied to automatically calculate the weights of the two, namely:

[0097]

[0098]

[0099] Furthermore, the reasoning layer further enhances the reasoning ability of the model. It extracts all the numbers in the text and questions and constructs a graph structure. Each number represents a point, and the edges are divided into two types according to the size relationship of the values:

[0100] A) Greater than edge: When two nodes v i , v j ∈V, if v i >v j , that is, to establish a directed edge

[0101] B) Less than or equal to edge: When two nodes v i , v j ∈V, if v i ≤v j, that is, to create a directed edge

[0102] Furthermore, after constructing the graph, the inference layer first uses Roberta's output H for all nodes. N (I(v)) is initialized, where I(v) represents the index position in the sentence. Then, the graph neural network is used for reasoning; and for each numerical node v i A weighted score is calculated, namely:

[0103]

[0104] Where W v represents the weight matrix, b v represents bias;

[0105] Then the information of each node is propagated to its neighboring nodes. Specifically, each node is updated according to the following formula:

[0106]

[0107] in is node v i Information representation; ji Represents and edge ji relevant relationship; Represents the specific transformation matrix of the relationship; N i ={j|(v j , v i )∈E} represents the node v i Neighbor nodes of

[0108] In order to fuse a node with its neighbor nodes, its information representation is fused with itself, namely:

[0109]

[0110] Where W f and b f is the weight matrix;

[0111] Expressing the above formula as a whole reasoning process, we can conclude that:

[0112] H G =Multi-Reasoning(G,V,H N ) (18).

[0113] Furthermore, before answering a question, the question-answering layer needs to predict the type of question, and then predict the answer based on the question type;

[0114] Specifically, using H G and To predict the type of question, that is:

[0115]

[0116] Where g represents the softmax function, which is used to calculate the probability of each type;

[0117] After obtaining the question type, predict the final answer, that is:

[0118] P(ans)=∑ type P(type)P(ans|type) (20).

[0119] The experimental data in this embodiment uses the Drop dataset, which is a public dataset for numerical reasoning tasks.

[0120] The answers in the Drop dataset are manually annotated and contain 77,409 training samples, 9,536 validation samples, and 9,622 test samples.

[0121] The performance of the model was evaluated by using the two indicators EM and F1. The model of this embodiment was implemented in Python, using the pytorch neural network framework, and hyperparameter tuning was performed on the validation set by using the grid search method. The Roberta model with 24 layers, 16 attention heads, and 1024 dimensions was used as the basic encoder.

[0122] And use Adam optimizer for optimization, with batch size of 16 and learning rate of 5e-5;

[0123] L2 regularization was set to 5e-5, gradient clipping value was set to 1.0, and three inferences were performed on the GNN. The size of the text and question was limited to 400 and 50, respectively;

[0124] Specific parameter settings are as follows Figure 3 As shown;

[0125] In order to compare the experimental results, several public models were selected as the model baselines of the present invention, including semantic parsing models (Syn Dep, OpenIE, SRL), traditional reading comprehension models (BiDAF, QANet, BERT), and numerical reasoning reading comprehension models (NAQANet, MTMSN, GENBERT, NumNet, NumNet+). The experimental results are as follows: Figure 4 As shown;

[0126] From the experimental results, the model of the present invention surpasses the semantic parsing model due to its ability to model long texts without manually designing rules. Compared with the traditional reading comprehension model, the model of the present invention has achieved significant improvement due to the improvement of numerical reasoning ability;

[0127] Besides this, the model in the present invention outperforms all numerical reasoning models;

[0128] In addition, the analysis was conducted according to the problem categories, such as Figure 5 As shown;

[0129] from Figure 5 It can be concluded that all models are poor at processing date problems, which is due to the small number of samples in the training set. In addition, it can be concluded that thanks to the capsule network and the bidirectional attention mechanism, the model of the present invention performs better than other models.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A capsule network structure suitable for numerical reasoning tasks, characterized by: The capsule network structure is sequentially provided with an input layer, a Robert coding layer, an inference layer, a capsule network layer and a question prediction layer; The specific implementation steps are as follows; S1. Question prediction layer: The description text and question are used as the input S of the whole model, which is divided into question sequence S q and the document sequence S d ; S2, then use the Roberta encoder to encode the input layer to obtain the encoded representation H of the input layer; S3, the inference layer and the capsule network layer take the encoded representation as input for computation; The results of S4, the inference layer and the capsule network layer are used as the input of the question prediction layer, and the final answer P(ans) is obtained through the question prediction layer; The Roberta encoding layer has a bidirectional attention network inside, namely: in The hidden layer vector representing the problem, The hidden layer vector representing the document; the Attention mechanism uses the attention algorithm in the table to multiply the hidden layer vector by different matrices W Q , W K and W V As query, key and value vectors respectively; The capsule of the capsule network is a set of vectors, where the direction of the capsule represents a specific type of digital feature, and the length of the capsule represents the possibility of the existence of these features; The active capsules from the high-level layers are predicted through matrix transformation, and the high-level capsules are clustered by the low-level capsules, while the low-level capsules have more abstract features. When multiple predictions are consistent, the high-level capsules will take effect; In the task of numerical reasoning, all the underlying semantic information extracted by the Roberta encoding layer is projected into a certain number of underlying capsules. Each vector represents a capsule, and each vector is activated by the nonlinear function Squash function. The Squash function can reduce the length of the vector to the range of 0 to 1, indicating the probability of the capsule, that is: After the lower-level capsules capture the numerical semantics, the higher-level capsules It is obtained by collecting the information of the underlying capsules, namely: in, Represents the parameter, c uj represents the coupling coefficient obtained by dynamic routing; Then, the mean-pooling operation is used to collect information from different capsule layers, namely: MP stands for mean-pooling operation. In order to fuse the information extracted by Roberta and capsule network, sigmoid is first applied to automatically calculate the weights of the two, namely: The reasoning layer further enhances the reasoning ability of the model. It extracts all the numbers in the text and question and builds a graph structure. Each number represents a point, and the edges are divided into two types according to the size relationship of the values: A) Greater than edge: When two nodes v i , v j ∈V, if v i >v j , that is, to create a directed edge B) Less than or equal to edge: When two nodes v i , v j ∈V, if v i ≤v j , that is, to create a directed edge After the graph is constructed, the inference layer first uses Roberta’s output H N (I(v)) is initialized, where I(v) represents the index position in the sentence. Then, the graph neural network is used for reasoning; and for each numerical node v i A weighted score is calculated, namely: Where W v represents the weight matrix, b v represents bias; Then the information of each node is propagated to its neighboring nodes. Specifically, each node is updated according to the following formula: in is node v i Information representation; ji Represents and edge ji relevant relationship; Represents the specific transformation matrix of the relationship; N i ={j|(v j , v i )∈E} represents the node v i Neighbor nodes of In order to fuse a node with its neighbor nodes, its information representation is fused with itself, namely: Where W f and b f is the weight matrix; Expressing the above formula as a whole reasoning process, we can conclude that: H G =Multi-Reasoning(G,V,H N ) (18); Before answering a question, the question prediction layer needs to predict the type of question and then predict the answer based on the question type. Specifically, using H G and To predict the type of question, that is: Where g represents the softmax function, which is used to calculate the probability of each type; After obtaining the question type, predict the final answer, that is: P(ans)=∑ type P(type)P(ans|type) (20)。 2. A capsule network structure suitable for numerical reasoning tasks according to claim 1, characterized in that: The input layer uses S q represents the problem sequence, S d Represents a sequence of documents; The document sequence S d Contains m words; the question sequence S q Contains n words; Use S for S d and S q The combined result is expressed as follows:

3. The capsule network structure suitable for numerical reasoning tasks according to claim 1, characterized in that: The Roberta encoding layer uses the Roberta network as the basic encoder and encodes the input S into H, that is: H=RoBerta-Encoder(S) (4) The outputs of the last four layers of the Roberta network structure are used as the input of the subsequent layers, namely: RoBerta out =[H N-3 ,H N-2 ,H N-1 ,H N ] (5)。