An attention-based multiple-choice machine reading comprehension method

Abstract

The present invention belongs to the field of natural language processing, and specifically relates to an attention-based multiple-choice machine reading comprehension method, comprising: incrementally training a Chinese pre-trained model BERT-wwm-ext using a sentence selection dataset S to obtain a continued pre-trained model D; introducing a multi-head attention mechanism to fuse option information, question and article information, and enhance the accuracy of the model's answers; and optimizing the model parameters using a weighted moving average. By continuing to pre-train the pre-trained model BERT-wwm-ext, the present invention makes the model more adaptable to horizontal fields. At the same time, the introduction of the multi-head attention mechanism can fully utilize the information between articles, questions and options, and then optimize the model parameters through weighted moving average, so that the model can more accurately answer reading comprehension questions.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to an attention-based multiple-choice machine reading comprehension method. Background Art

[0002] Machine reading comprehension is a key research area in natural language processing. With the integration of various deep learning technologies in recent years, it has garnered increasing attention. Multiple-choice machine reading comprehension is one such task. Its mission is to help humans quickly focus on relevant information from massive amounts of text and make optimal choices, thereby reducing the cost of manual information acquisition and increasing the effectiveness of information retrieval.

[0003] Currently, mainstream machine reading comprehension technologies are based on large-scale pre-trained models, such as BERT and ROBERT. These models leverage better initialization parameters and utilize richer and more robust datasets, resulting in improved accuracy on target tasks. However, stacking more data increases the model's load, causing it to be cluttered with useless information, impairing generalization, and slowing convergence. In summary, the combination of traditional models and data cannot meet the needs of current application scenarios. Summary of the Invention

[0004] To solve the above problems in the prior art, the present invention proposes an attention-based multiple-choice machine reading comprehension method, which includes:

[0005] S1: Obtain reading comprehension data and divide the reading comprehension data into multiple choice datasets and sentence selection datasets;

[0006] S2: Perform a secondary masking operation on the article questions and options in the sentence selection dataset, and input the masked data into the BERT-wwm-ext pre-trained model for incremental training to obtain the pre-trained model D;

[0007] S3: Add a multi-head attention mechanism to the pre-trained model D to calculate the mutual attention weight a between each option ij , get the option's attention output vector Attention(a ij );

[0008] S4: The attention output vector is integrated into the original option representation through the gated channel attention mechanism to obtain the candidate option set options;

[0009] S5: Perform similarity matching between the options and the article, and between the question and each sentence in the article, retain the k sentences with the highest relevance between the article and the options, and combine the k sentences into the article P';

[0010] S6: Construct fully connected information based on the candidate option set options, article P' and question, and predict the final answer to the question through the constructed fully connected linear layer.

[0011] Preferably, the process of performing incremental training on the BERT-wwm-ext pre-training model includes: inputting sentence selection data into the BERT-wwm-ext model for one round of continued pre-training to obtain a pre-training model M; randomly selecting words from the options, masking them with [MASK], and filling the correct answer sentences into the article, using the masked words as options to obtain new text data; repeating the above process, inputting all new data texts into the model M to obtain the current pre-training model D.

[0012] Furthermore, the iterative training process of the BERT-wwm-ext model includes: given a paragraph P and n options a1, a2...a n , replace the spaces in P with a special tag from the vocabulary [unusedNum], where Num ranges from 0 to the number of spaces minus 1; for each a in the answer choice i , a i and P with label [SEP] as the input sequence; the input sequence of length l is input into BERT, and the learning rate of the model is decayed by the weighted moving average optimization method to obtain a hidden representation H and trainable parameters Perform the dot product to get The probability t of a blank appearing in the current option is logarithmically calculated through softmax, and the option with the largest probability is selected as the prediction for that space.

[0013] Furthermore, the weighted moving average optimization method is used to decay the learning rate of the model, including: for n weight data [θ1, θ2, ... θ n ], the weighted average formula is:

[0014] v t =β·v t-1 +(1-β)·θ t

[0015] Among them, v t represents the t-th shadow weight, β is a hyperparameter, v t-1 represents the t-1th shadow weight, θ t Represents the tth weight data.

[0016] Preferably, calculate the mutual attention weight a between each option ij The formula is:

[0017]

[0018]

[0019] m=1,2,3...head

[0020] Among them, a ij Represents the i-th option vector o i Compared with the j-th option vector p j Attention weight, Attention(a ij ) m represents the attention output vector calculated by the option vector and the comparison option vector, softmax() represents the softmax function, and W1, W2, and W3 are all trainable weight matrices.

[0021] Preferably, integrating the attention output vector into the original option representation via a gated channel attention mechanism includes:

[0022]

[0023]

[0024]

[0025]

[0026] Beneficial effects of the present invention:

[0027] This paper continues to pre-train the pre-trained model BERT-wwm-ext by using sentence selection data, making the model more adaptable to horizontal fields. At the same time, the introduction of the multi-head attention mechanism can make full use of the information between articles, questions and options, and then optimize the model parameters through weighted moving average, so that the model can more accurately answer reading comprehension questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of the attention-based multiple-choice machine reading comprehension method of the present invention;

[0029] Figure 2 Schematic diagram of the attention model structure of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] An attention-based multiple-choice machine reading comprehension method, such as Figure 1 As shown, the method includes:

[0032] S1: Obtain reading comprehension data and divide the reading comprehension data into multiple choice datasets and sentence selection datasets;

[0033] S2: Perform a secondary masking operation on the article questions and options in the sentence selection dataset, and input the masked data into the BERT-wwm-ext pre-trained model for incremental training to obtain the pre-trained model D;

[0034] S3: Add a multi-head attention mechanism to the pre-trained model D to calculate the mutual attention weight a between each option ij , get the option's attention output vector Attention(a ij );

[0035] S4: The attention output vector is integrated into the original option representation through the gated channel attention mechanism to obtain the candidate option set options;

[0036] S5: Perform similarity matching between the options and the article, and between the question and each sentence in the article, retain the k sentences with the highest relevance between the article and the options, and combine the k sentences into the article P';

[0037] S6: Construct fully connected information based on the candidate option set options, article P' and question, and predict the final answer to the question through the constructed fully connected linear layer.

[0038] The incremental training process for the BERT-wwm-ext pre-trained model involves applying a secondary masking strategy to sentence selection data. First, the sentence selection data is fed into the BERT-wwm-ext model for one round of pre-training, resulting in pre-trained model M. Next, a word from the selected options is masked with [MASK], and the correct answer sentence is inserted into the article, with the masked word as the option. This process is repeated to generate new text data. The resulting new text data is fed into model M to produce the current pre-trained model D.

[0039] The iterative training process of the BERT-wwm-ext model includes: given a paragraph P and n options a1, a2...a n , replace the spaces in P with a special tag from the vocabulary [unusedNum], where Num ranges from 0 to the number of spaces minus 1; for each a in the answer choice i , a iand P with label [SEP] as the input sequence; the input sequence of length l is input into BERT, and the learning rate of the model is decayed by the weighted moving average optimization method to obtain a hidden representation H and trainable parameters Perform the dot product to get The probability t of a blank appearing in the current option is logarithmically calculated through softmax, and the option with the largest probability is selected as the prediction for that space.

[0040] Preferably, the weighted moving average optimization method is used to decay the learning rate of the model, including: for n weight data [θ1, θ2, ... θ n ], the weighted average formula is:

[0041] v t =β·v t-1 +(1-β)·θ t

[0042] Among them, v t represents the t-th shadow weight, β is a hyperparameter, v t-1 represents the t-1th shadow weight, θ t Represents the tth weight data.

[0043] A multi-head attention mechanism, such as Figure 2 As shown, a multi-head attention mechanism is used to process the data, specifically including: defining a ij Represents the i-th option vector o i Compared with the j-th option vector p j Attention weight, Attention(a ij ) m represents the attention output vector calculated by the option vector and the contrast option vector, which is calculated as follows:

[0044]

[0045] m=1,2,3...head

[0046] Among them, a ij Represents the i-th option vector o i Compared with the j-th option vector p j Attention weight, Attention(a ij ) m represents the attention output vector calculated by the option vector and the comparison option vector, softmax() represents the softmax function, and W1, W2, and W3 are all trainable weight matrices.

[0047] Concatenate all the resulting vectors as the final result:

[0048]

[0049] In the formula Represents a vector concatenation operation.

[0050] The obtained attention interaction representation is integrated into the original representation through the gating mechanism. The specific calculation process is:

[0051]

[0052]

[0053]

[0054]

[0055] in, represents the set of attention output vectors for the i-th option, a i represents the i-th option, Indicates option a i The matrix obtained by concatenating the attention output vectors of outi Indicates option a i Compared with other options, tanh represents the hyperbolic tangent function, W4, W5, and W6 represent trainable weight matrices, and sigm represents the sigmoid function.

[0056] The expression of the sigmoid function is:

[0057]

[0058] remember That is the set of all candidate options.

[0059] The accuracy of the model is improved by calculating the similarity between words in the passage, question, and options. The specific operation is to only calculate the weighted average of the word vectors in the sentence and then remove the common part of the average vector on the principal component.

[0060] Specifically, similarity matching detection is performed on each sentence in the option and the article, and similarity matching detection is performed on each sentence in the question and the article, including: calculating the weighted average of the word vectors in the sentence and removing the common value of the average vector on the principal component; given a sentence s, calculating the weighted average of each word w in the sentence; obtaining the semantic information in the sentence s; filtering the semantic information according to the weighted average, deleting the components in the direction of the semantic information, and obtaining the key information; comparing the similarity of all the key information to obtain k sentences; when calculating the weighted average, it is necessary to calculate the generation probability of the word in the sentence, and calculate the weight of each word according to the generation probability, that is, the weight of the word is expressed as Where a is a hyperparameter, p(w) is the frequency of word w in the corpus, and the corpus is a database composed of random data.

[0061] Calculating the probability of a word in a sentence includes: given a topic vector t s , the generation probability of word w in sentence s is:

[0062]

[0063]

[0064]

[0065] Where α and β are hyperparameters, t o is the public topic vector, v w is the word vector corresponding to each word; is the normalization constant, The relationship between words and common topics and sentences.

[0066] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multiple-choice machine reading comprehension method based on attention, characterized in that: include: S1: Obtain reading comprehension data and divide the reading comprehension data into multiple choice datasets and sentence selection datasets; S2: Perform a secondary masking operation on the article questions and options in the sentence selection dataset, and input the masked data into the BERT-wwm-ext pre-trained model for incremental training to obtain the pre-trained model D; The incremental training process for the BERT-wwm-ext pre-trained model includes: inputting sentence selection data into the BERT-wwm-ext model for one round of iterative pre-training to obtain the pre-trained model M; randomly selecting words from the options, masking them with [MASK], and filling the correct answer sentences into the article, using the masked words as options to obtain new text data; repeating the above process, inputting all new data text into model M to obtain the current pre-trained model D; The iterative training process of the BERT-wwm-ext model includes: given a paragraph P and n options a1, a2...a n , replace the spaces in P with a special tag from the vocabulary [unusedNum], where Num ranges from 0 to the number of spaces minus 1; for each a in the answer choice i , a i and P with label [SEP] as the input sequence; the input sequence of length l is input into BERT, and the learning rate of the model is decayed by the weighted moving average optimization method to obtain a hidden representation H and trainable parameters Perform the dot product to get The probability t of a blank in the current option is calculated logarithmically using softmax, and the option with the highest probability is selected as the blank prediction; S3: Add a multi-head attention mechanism to the pre-trained model D to calculate the mutual attention weight a between each option in the multiple choice dataset ij , get the option's attention output vector Attention(a ij ); S4: The attention output vector is integrated into the original option representation through the gated channel attention mechanism to obtain the candidate option set options; S5: Perform similarity matching tests on each sentence in the options and the article, and on each sentence in the question and the article, retaining the k sentences with the highest relevance between the article and the options and the question, and combining these k sentences into the article P'; S6: Construct fully connected information based on the candidate option set options, article P' and question, and predict the final answer to the question through the constructed fully connected linear layer.

2. The attention-based multiple-choice machine reading comprehension method according to claim 1, characterized in that: The weighted moving average optimization method is used to decay the learning rate of the model, including: for n weight data [θ1, θ2, ... θ n ], the weighted average formula is: v t =β·v t-1 +(1-β)·θ t Among them, v t represents the t-th shadow weight, β is a hyperparameter, v t-1 represents the t-1th shadow weight, θ t Represents the tth weight data.

3. The attention-based multiple-choice machine reading comprehension method according to claim 1, characterized in that: Calculate the mutual attention weight a between each option ij The formula is: m=1,2,3...head Among them, a ij Represents the i-th option vector o i Compared with the j-th option vector p j Attention weight, Attention(a ij ) m represents the attention output vector calculated by the option vector and the comparison option vector, softmax() represents the softmax function, and W1, W2, and W3 are all trainable weight matrices.

4. The attention-based multiple-choice machine reading comprehension method according to claim 1, characterized in that: The attention output vector is incorporated into the original option representation through the gated channel attention mechanism including: in, represents the set of attention output vectors for the i-th option, a i Indicates that the i-th option, Indicates option a i The matrix obtained by concatenating the attention output vectors of , c represents the gate unit, Indicates option a i Compared with other options, tanh represents the hyperbolic tangent function, W4, W5, and W6 represent trainable weight matrices, and sigm represents the sigmoid function.

5. The attention-based multiple-choice machine reading comprehension method according to claim 1, characterized in that: Perform similarity matching tests on each option and sentence in the article, and on each question and sentence in the article. The tests include: calculating the weighted average of the word vectors in the sentence and removing the common values ​​of the average vector on the principal component; given a sentence s, calculating the weighted average of each word w in the sentence; obtaining the semantic information in sentence s; filtering the semantic information based on the weighted average, removing the components that are in the direction of the semantic information, and obtaining the key information; performing similarity comparison on all key information to obtain k sentences; when calculating the weighted average, the weight of each word is expressed as Where a is a hyperparameter, p(w) is the frequency of word w in the corpus, and the corpus is a database composed of random data.

Images