Machine reading question and answer understanding method based on adversarial training

Abstract

The invention discloses a machine reading question and answer understanding method based on adversarial training. According to the method, firstly, MacBERT is used for carrying out word embedding on an input natural language to generate vector representation, then adversarial training is used for adding tiny disturbance on word vectors according to gradient changes to generate adversarial samples, finally, original samples and the adversarial samples are input into a BiLSTM network to further extract text context features, and finally, prediction answers are output. An adversarial training module is added on the basis of an existing machine reading understanding model, and the robustness and generalization of the whole model are improved by utilizing an adversarial sample attack network. The result shows that the accuracy of the method on a CMRC2018 data set and a DRCD data set is higher than that of a current mainstream machine reading understanding method. The method aims at enabling a machine to understand natural languages, and is applied to various fields such as information retrieval, intelligent question and answer systems and machine translation.

Description

Technical Field

[0001] The invention designs a machine reading comprehension question answering method based on adversarial training, which relates to the field of natural language processing. Background Art

[0002] Machine reading comprehension technology aims to enable machines to understand human natural language. In order to evaluate the machine's ability to understand a natural language text, a common method is to give a text C and ask a question Q about the content of the text. The machine extracts, generates or selects the appropriate answer A based on the text C and question Q. With the emergence of large-scale data sets, high-computing computers and deep learning models in recent years, the research on machine reading comprehension has been greatly promoted and has been applied in many fields such as information retrieval, intelligent question-answering systems, and machine translation.

[0003] In the early days, due to the limitations of computer level and data set size, machine reading comprehension systems were mainly based on manually formulated rules and were greatly restricted by application fields. Later, the emergence of deep learning brought machine reading comprehension to a new stage. Hermann et al. proposed The Attentive Reader and The Impatient Reader models based on traditional deep learning models, adding attention mechanisms to the deep learning network LSTM. Seo et al. proposed the Bidirectional Attention Flow BiDAF network, adding a bidirectional attention layer to integrate question and context representation. In 2017, the Google team proposed the Transformer model, which achieved excellent results in multiple tasks using only the multi-head attention mechanism. On this basis, Devlin et al. proposed the BERT pre-training model, which greatly improved the performance of the machine reading comprehension model. The excellent performance of the BERT pre-training model in the field of NLP has led more and more scholars to improve and innovate the pre-training model. Liu et al. proposed the RoBERTa model, which used more training data, larger batch size and longer training time during pre-training. Later, Cui et al. also improved the pre-training method and proposed the MacBERT model. Yang et al. improved the BERT model structure and proposed XLNET, which added a dual-stream attention mechanism and related technologies of transformer-xl. After that, various pre-trained models such as ALBERT and spanBERT were proposed. At present, most machine reading comprehension models are optimized based on BERT or BERT variant pre-trained models, but there are some problems in the actual machine reading comprehension question-answering applications. Different people may express the same question differently when asking questions, and there may be abbreviations and grammatical errors in the text. Therefore, the single text and question expression in the machine reading comprehension task will lead to poor generalization ability and low robustness of the model. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a machine reading comprehension question answering method based on adversarial training. The network first uses MacBERT to embed words into text and converts natural language into vector representation. Then, adversarial training is introduced to add different small perturbations to each word vector according to the gradient to generate adversarial samples. After adversarial training of the model, the adversarial samples will make the model adapt to noise, improve the model's robustness and generalization ability, and then introduce a bidirectional long short-term memory network to enable the model to better extract the features of the context in the text and questions.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] The machine reading comprehension question answering method based on adversarial training includes the following steps:

[0007] Step 1: Concatenate the text containing the answer and the question and send them to MacBERT's word embedding network to convert the input text into a vector representation, where each word and character is converted into a 768-dimensional vector, and the input text length is padded to 512 words. Finally, the vector representation dimension of each question and answer is (1,512,768);

[0008] Step 2: Input the generated vector representation into the encoding module of MacBERT for iterative training, and then input it into the BiLSTM network to further extract context features;

[0009] Step 3: Calculate the small perturbation to be added based on the forward propagation loss and the backward propagation gradient, and add it to the original vector representation generated in step 1 to obtain the adversarial sample;

[0010] Step 4: Perform the same operation as step 2 on the adversarial sample generated in step 3, calculate the forward propagation loss value and back-propagation gradient of the adversarial sample, accumulate the generated adversarial gradient to the gradient generated by back-propagation of the original word vector, and restore the word vector to the original word vector;

[0011] Step 5: Connect the fully connected layer to generate the starting and ending positions of the predicted answer in the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is the structure of the machine reading comprehension question answering method based on adversarial training proposed by the present invention.

[0013] Figure 2 This is the word embedding module structure based on MacBERT proposed in the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further described below in conjunction with the accompanying drawings:

[0015] Figure 1 This is the structure of the machine reading comprehension question-answering method based on adversarial training proposed in the present invention. First, the text containing the answer and the question are concatenated and sent to the word embedding network of MacBERT to convert the input text into a vector representation, where each word and character is converted into a 768-dimensional vector, and the input text length is padded to 512 words. Finally, the vector representation dimension of each question and answer is (1,512,768); then the generated vector representation is input into the encoding module of MacBERT for iterative training, and then input into the BiLSTM network to further extract context features. The BiLSTM expression is shown in formula (1)(2)(3):

[0016]

[0017]

[0018] h t =o t *tanh(C t ) (3)

[0019] where f t 、i t , o t are the forget gate, memory gate and output gate at time t, W f , W i and W o is the corresponding weight coefficient, b f 、b i and b o is the corresponding offset, h t-1 is the hidden layer state at time t-1, x t is the input at time t, is the temporary cell state, C t is the current cell state, h t is the hidden layer state at time t, and σ is the Sigmoid activation function.

[0020] In order to achieve the best perturbation effect, perturbation is added in the direction where the forward propagation loss increases the most. The expression is shown in formula (4):

[0021] max(-logp(y|x+r;θ)) (4)

[0022] Where y is the true label, x is the original word vector, r is the added small perturbation, θ is the model constant, and the perturbation r that increases the loss the most adv The calculation formula is shown in formula (5) (6):

[0023]

[0024]

[0025] Where ε is the intensity factor and g is the partial derivative of the loss function loss with respect to x, i.e. the gradient.

[0026] The process of adding perturbations for adversarial training is as follows:

[0027] 1) Calculate the forward propagation loss and the backward propagation gradient for each original word vector x sent into the network;

[0028] 2) Calculate the disturbance r to be added based on the gradient adv , added to x to get the adversarial sample a = x + r adv ;

[0029] 3) Calculate the forward propagation loss and back-propagation gradient of adversarial example a, and add the generated adversarial gradient to the gradient generated by the back-propagation of the original word vector in 1);

[0030] 4) Restore the word vector to the original word vector x;

[0031] 5) Update the network parameters according to the accumulated gradient generated in 3).

[0032] After adversarial training, the network model is more robust and has stronger generalization ability. It has good resistance to errors caused by typos, abbreviations, and grammatical errors that may appear in question-answering tasks, which improves the performance of the network. Finally, the fully connected layer is connected to generate the starting and ending positions of the predicted answer in the text.

[0033] Figure 2This is a structural diagram of the word embedding module based on MacBERT proposed in the present invention. MacBERT first performs a word segmentation operation on the input text, divides the complete sentence into individual words and characters, and then inputs the word embedding layer (EmbeddingsLayer). The word embedding layer includes three embedding representations, namely word embeddings (token embeddings), segment embeddings (segmentembeddings) and position embeddings (position embeddings). Word embedding converts each word (token) of the input text into a vector representation. First, a special symbol [CLS] is inserted at the starting position of the input text, and a special symbol [SEP] is inserted at the end of the sentence and between two sentences. Finally, each word and special symbol is converted into a 768-dimensional vector, so the word embedding vector dimension of the input text is (1,n,768), where n is the total number of words and symbols in the input text, and 1 is the assumed batch_size size. Segment embedding is to mark the input text at the sentence level, which is used to distinguish the order of two sentences in a sentence pair. Segment embedding has only two vector representations. The first vector A is assigned to all the words and symbols of the first sentence, and the second vector B is assigned to all the words and symbols of the second sentence. Then the first vector A is assigned to all the words and symbols of the third sentence. The assignment is performed alternately. Finally, the dimension of the segment embedding vector of the input text is (1, n, 768). Position embedding is to represent the characters at different positions of the input text by vectors. First, let MacBERT learn a vector representation at each position to encode the sequence order information, and make a (512, 768) vector table. The first character of the input text is embedded as the 768-dimensional vector in the first row of the vector table, and the second character of the input text is embedded as the vector in the second row of the vector table. And so on. The dimension of the position embedding vector of the input text is also (1, n, 768). BERT limits the length of the input text. After inserting the special characters [CLS] and [SEP], the number of characters must not exceed 512. If it exceeds, it must be truncated, and if it is less than, it must be supplemented. Finally, the generated word embedding, segment embedding, and position embedding are added together to obtain an embedding representation of the input text size of (1,512,768). The vector obtained by word embedding is then input into the encoder layer for training. The encoding layer of MacBERT is improved from the Transformer model and consists of 12 layers of Transformer encoder modules stacked together, where each layer of the encoder module consists of a multi-head attention layer and a feed-forward network layer.

[0034] In order to verify the effectiveness of the proposed method, an experimental analysis was conducted on the CMRC2018 Chinese machine reading comprehension dataset and compared with BERT base 、RoBERTa wwm-ext-base 、MacBERT base Basic models such as Hybrid-Attention, ChineseBERT, BERT-NTC, and ERNIE2.0 base , ESPReader and other models proposed by other scholars. The experimental results are shown in Table 1.

[0035] Table 1

[0036]

[0037] From Table 1, we can see that the accuracy of the model in this paper is improved after BiLSTM is introduced on the basis of MacBERT, which shows that the performance of the model is improved by using BiLSTM to integrate context features. After adding adversarial training on the basis of MacBERT, the accuracy is not only higher than the three traditional models, but also higher than the five advanced models proposed by other scholars. This shows that adversarial training enables the model to continuously optimize its performance under the attack of adversarial samples, improves the robustness and generalization ability of the model, and after adding BiLSTM and adversarial training at the same time, the performance of the model is further improved.

[0038] In order to verify the universality of the method of the present invention, we also conducted experiments on the DRCD Traditional Chinese machine reading comprehension dataset. The experimental results are shown in Table 2.

[0039] Table 2

[0040]

Claims

1. A machine reading comprehension question answering method based on adversarial training, characterized in that The following steps are involved: Step 1: Concatenate the text containing the answer and the question and send them to MacBERT's word embedding network to convert the input text into a vector representation, where each word and character is converted into a 768-dimensional vector, and the input text length is padded to 512 words. Finally, the vector representation dimension of each question and answer is (1,512,768); Step 2: Input the generated vector representation into the encoding module of MacBERT for iterative training, and then input it into the BiLSTM network to further extract context features. The encoding layer of MacBERT is improved from the Transformer model and consists of 12 layers of Transformer encoder modules stacked together. Each layer of the encoder module consists of a multi-head attention layer and a feed-forward network layer. Step 3: Calculate the small perturbation r to be added based on the forward propagation loss and the backward propagation gradient adv , the calculation formula is as follows, where ε is the intensity factor, g is the partial derivative of the loss function loss with respect to x, and the generated small perturbation r adv Add the original vector representation generated in step 1 to get the adversarial sample: Step 4: Repeat the operation of step 2 for the adversarial sample generated in step 3, calculate the forward propagation loss value and back-propagation gradient of the adversarial sample, accumulate the generated adversarial gradient to the gradient generated by back-propagation of the original word vector, and then restore the word vector to the original word vector; Step 5: Connect the fully connected layer to generate the starting and ending positions of the predicted answer in the text.

Images