A Reading Comprehension Bias Removal Method Based on Pre-trained Models

Abstract

This invention provides a reading comprehension bias removal method based on a pre-trained model, comprising: constructing a text representation layer based on the BERT embedding layer of a pre-trained model; setting the model input as sentence pairs consisting of an article and a question; mapping the input sentences to sentence vectors; performing two random regularization operations on the positional encoding to obtain two random sub-models and calculating the KL divergence, evaluating the difference in their output distribution, and updating the sentence vector input to the BERT encoder; passing the output of the last hidden layer of the BERT encoder through a fully connected layer and a classifier, calculating the prediction loss of the samples using the sum of cross-entropy loss and KL divergence; optimizing the pre-training parameters based on the prediction loss of the samples during model training to remove reading comprehension positional bias. This invention utilizes dropout technology to regularize the positional encoding, and by lowering the KL divergence of the random sub-model distributions generated by the two dropouts during training, it constrains randomness, improves the model's robustness to positional bias, and restores the accuracy of answer prediction.

Description

Technical Field

[0001] This invention belongs to the field of machine reading comprehension in natural language processing, and specifically relates to a reading comprehension bias removal method based on a pre-trained model. Background Technology

[0002] In recent years, the development of deep learning technology has continuously driven research innovation in the field of natural language processing. Machine reading comprehension is an important topic in natural language processing, aiming to use algorithms to enable machines to understand the semantics of articles and answer related questions. The massive growth of internet information in the era of big data has promoted the practical application value of reading comprehension tasks, including but not limited to: improving search engine query performance, using intelligent customer service to provide simple question-and-answer services, thereby improving user experience and platform service efficiency.

[0003] Mainstream reading comprehension models are primarily based on end-to-end neural networks and pre-training plus fine-tuning. End-to-end neural networks encode the input text sequence using CNN or RNN network architectures and capture the information between the text and the questions using attention or self-attention mechanisms. With deeper research into text feature representation, pre-trained language models based on large-scale corpora have further propelled the development of machine reading comprehension technology. The BERT pre-trained model [Kenton JDMWC, Toutanova L K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [C] / / Proceedings of NAACL-HLT. 2019:4171-4186.] innovatively incorporated mask prediction and next sentence prediction pre-training tasks to better capture semantic information and achieved excellent results on SQuAD1.1, surpassing human performance on evaluation metrics, thus ushering in a new era of pre-trained language models.

[0004] However, the extractive reading comprehension model, which relies on predicting the position of the answer, has certain drawbacks. Experiments have shown that when trained on skewed datasets, the model's predictive performance drops significantly, confirming a serious bias in models based on predicting the starting position of the answer—a phenomenon known as position bias. This indicates that although existing models achieve above-human accuracy on many standard evaluation sets, their prediction results may decline sharply on specially processed or slightly perturbed samples. The paper [Ko M, Lee J, Kim H, et al. Lookat the First Sentence: Position Bias in Question Answering[C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP).2020:1109-1121.] proposes an ensemble-based debiasing approach, which uses a product of expert to combine a predefined bias model and the log-likelihood of a pre-trained model, where the bias model is defined as a pre-computed answer prior.

[0005] However, the integrated model balances positional biases through an additional bias model, without fully considering the robustness optimization of the target model itself. Summary of the Invention

[0006] The technical problem this invention aims to solve is to alleviate the positional bias problem in the reading comprehension training process. This invention proposes a reading comprehension debiasing method based on a pre-trained model. Based on the pre-trained model BERT, this invention utilizes dropout technology to regularize positional encoding and then constrains randomness by lowering the KL divergence of the random sub-model distributions generated by two dropouts during training. This improves the model's robustness to positional bias while restoring answer prediction accuracy.

[0007] To achieve the above objectives, this invention provides a reading comprehension bias removal method based on a pre-trained model, comprising the following steps:

[0008] A text representation layer is built based on the pre-trained BERT embedding layer. The model input is set as sentence pairs consisting of article-question pairs, and the input sentences are mapped to sentence vectors.

[0009] The positional encoding is subjected to two random regularization operations to obtain two random sub-models. The KL divergence is calculated to evaluate the difference in their output distribution and update the sentence vector input to the BERT encoder.

[0010] The output of the last hidden layer of the BERT encoder is passed through a fully connected layer and a classifier. The prediction loss of the sample is calculated by the sum of cross-entropy loss and KL divergence. During the model training process, the pre-training parameters are optimized based on the loss value to remove reading comprehension positional bias.

[0011] Furthermore, the positional encoding is subjected to two random regularization operations, including:

[0012] The dropout technique is used to randomly reset the initial positional codes of BERT to 0 twice with probability p, resulting in two positional code vectors with weakened positional information. The average of these two vectors is then added to the remaining codes to obtain the updated sentence vector, expressed by the formula:

[0013]

[0014] Where L represents the maximum length of the input sequence; d represents the length of each token vector; E L×d This represents the updated sentence vector; This represents the token embedding composed of each token and its corresponding vocabulary. This indicates the original position code. This indicates the segment embedding that distinguishes between the question and the article; Dropout(·) indicates a random zeroing operation.

[0015] Furthermore, L is 512 and d is 768.

[0016] Furthermore, the formula for calculating the KL divergence is:

[0017]

[0018] Where i represents the i-th training sample; D KL (·) indicates the calculation of KL divergence; P θ (y|e pos ) and P′ θ (y|e pos The positional encoding represents the different distributions generated by the two different sub-models after the dropout operation; since the KL divergence itself is asymmetric, the positions of these two distributions need to be swapped to finally obtain the averaged KL divergence.

[0019] Furthermore, the BERT encoder is a Transformer encoder structure, comprising 12 stacked encoding modules, each encoding module having a hidden state size of 768, and including 12 attention heads.

[0020] Furthermore, the loss function includes cross-entropy and KL divergence of regularized positional encoding. The ratio of cross-entropy to KL divergence of regularized positional encoding is determined by an adjustable hyperparameter α. The specific formula for calculating the sum of cross-entropy loss and total training loss is as follows:

[0021]

[0022] L = L CE +α*L KL

[0023] Where i represents the i-th training sample; N represents the number of samples; This indicates the true starting position of the answer in the i-th sample; represents the true end position of the answer in the i-th sample; ps represents the probability that each position in the predicted text sequence is the starting position of the answer; p s α represents the probability that each position in the predicted text sequence is the end position of the answer; α is an adjustable hyperparameter representing the ratio of cross-entropy to the KL divergence of the regularized positional encoding; L CE L represents the cross-entropy loss value. KL represents the KL divergence value of the regularized positional encoding; L represents the total training loss sum.

[0024] Furthermore, the dataset used for model training is the extractive reading comprehension dataset SQuAD1.1 and its subsets divided according to the position of the paragraph containing the answer. The ratio α of KL divergence to cross-entropy is 3 when the best accuracy is achieved, and the probability p of dropping out to zero is 0.1.

[0025] The beneficial effects achieved by this invention are as follows:

[0026] This invention addresses the positional bias problem in machine reading comprehension tasks. Using the pre-trained BERT model, it regularizes the positional encoding of its embedding layers to enhance the model's focus on answer positional information. Simultaneously, it incorporates the KL divergence of the evaluation model distribution into the loss function to constrain the performance impact of random regularization. Simulation experiments demonstrate that this invention effectively improves the accuracy of reading comprehension answer prediction under biased training datasets while maintaining performance under unbiased conditions, significantly enhancing the robustness of reading comprehension models based on pre-trained models. Attached Figure Description

[0027] Figure 1 This is a model structure diagram of a reading comprehension bias removal method based on a pre-trained model provided in an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of the loss training process in a reading comprehension bias removal method based on a pre-trained model provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will now be further described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0030] This invention provides a reading comprehension debiasing method based on a pre-trained model. Using the pre-trained BERT model, regularization is applied to the positional encoding of its embedding layers to generalize the model's focus on answer positional information. Simultaneously, the KL divergence of the evaluation model distribution is added to the loss function to constrain the performance impact of random regularization. Simulation experiments demonstrate that this invention can effectively improve the accuracy of reading comprehension answer prediction under biased training datasets while maintaining answer prediction performance under unbiased conditions, thus significantly improving the robustness of reading comprehension models based on pre-trained models.

[0031] This invention provides a reading comprehension bias removal method based on a pre-trained model, the specific implementation process of which is as follows:

[0032] The experimental environment consisted of a Windows 10 64-bit operating system, an Intel i7-8700 CPU, an NVIDIA GeForce RTX 2070 GPU, and 16GB of memory. The experiment was implemented on the deep learning framework PyTorch and was developed using Python.

[0033] The hyperparameter settings for the experiment were as follows: maximum question length was set to 64, maximum predicted answer length was set to 30, number of hidden layers was set to 12, number of hidden units was set to 768, initial learning rate of Adam optimizer was set to 3e-5, batch size was set to 6, number of training iterations per round was 2, and dropout probability used in position encoding was an adjustable parameter, with optimal debiasing effect obtained at 0.1.

[0034] This invention provides a reading comprehension bias removal method based on a pre-trained model, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0035] Step 1: Build a text representation layer based on the pre-trained BERT embedding layer. Set the model input as sentence pairs consisting of article-question pairs and map the input sentences into sentence vectors.

[0036] Step 2: To generalize the model's focus on location information, the location encoding is subjected to two random regularization operations, i.e., dropout is performed with probability p to randomly set the values ​​to 0, resulting in two location encoding vectors with weakened location information. Their mean is then added to the remaining encodings to obtain the updated sentence vector, expressed by the formula:

[0037]

[0038] Where L represents the maximum length of the input sequence, set to 512; d represents the length of each token vector, set to 768; E L×d This represents the updated sentence vector. This represents the Tokenembedding, which consists of a list of terms corresponding to each token. This represents the original position code. The Segmentembedding function distinguishes between questions and articles, while Dropout(·) represents a random zeroing operation.

[0039] This results in two distinct sub-model distributions generated by the positional encoding after the dropout operation. Their KL divergence is then calculated to assess the differences in their output distributions. The KL divergence calculation formula is as follows:

[0040]

[0041] Where i represents the i-th training sample, D KL (·) indicates the calculation of KL divergence, P θ (y|e pos ) and P′ θ (y|e pos The positional encoding represents the different distributions generated by the two different sub-models after the dropout operation. Since the KL divergence itself is asymmetric, the positions of these two distributions need to be swapped to finally obtain the averaged KL divergence.

[0042] The updated sentence vector is input into the Transformer encoder structure, which consists of 12 stacked encoding modules, each with a hidden state size of 768, and includes 12 attention heads.

[0043] Step 3: The input sentence pair vector is processed through a 12-layer BERT encoding module, and the last hidden layer is retained as the feature vector M = {m1, ..., m}. L Let L represent the maximum length of the input sequence. Pass this sequence through a fully connected layer and a Softmax classifier to predict the starting position of the answer, and calculate the cross-entropy loss L_s across the predicted answer span. CE In order to constrain the output distribution of the random sub-model and keep different outputs consistent, a robust regularization term, KL divergence, is added to the loss function. The sum of KL divergence and cross-entropy loss is used as the total loss L of the calculated sample. The pre-training parameters are optimized based on the loss value to remove the reading comprehension position bias problem, thereby improving the accuracy of obtaining appropriate answers when the question scenario in the actual question answering environment is different from the distribution of the question answering data used for model training.

[0044] The specific formula for calculating the cross-entropy loss is as follows:

[0045]

[0046] Where i represents the i-th training sample, and N represents the number of samples. This represents the true starting position of the answer in the i-th sample. p represents the true ending position of the answer in the i-th sample. s p represents the probability that each position in the predicted text sequence is the starting position of the answer. s This represents the probability that each position in the predicted text sequence is the end position of the answer.

[0047] The formula for calculating the total training loss is:

[0048] L = L CE +α*L KL

[0049] Where α is an adjustable hyperparameter representing the ratio of cross-entropy to the KL divergence of regularized positional encoding, and L... CE L represents the cross-entropy loss value. KL represents the KL divergence value of the regularized positional encoding, and L represents the total training loss sum.

[0050] The dataset used in the experiment was the extractive reading comprehension dataset SQuAD1.1 and its subsets based on the position of the answer within the paragraph. Specifically, k represents the k-th sentence in the paragraph where the answer is located, and k can take values ​​of 1, 2, 3, 4, and k≥5. Therefore, the subsets are denoted as SQuAD. train k＝1 SQuAD train k＝2 SQuAD train k＝3 SQuAD train k＝4 SQuAD train k＝5，6，... .

[0051] First, in SQuAD, which has the largest number of samples... train k＝1 Simulation experiments were conducted, the network was trained multiple times, and the model's answer prediction accuracy was recorded. The improved model achieved a 1.24% improvement in F1 accuracy compared to the previous model. The improved model was then applied to other subsets of the dataset, specifically SQuAD. train k＝2 SQuAD train k＝3 SQuAD traink＝4 SQuAD train k＝5，6，... The F1 scores improved by 0.38%, 0.46%, 0.37%, and 0.15% respectively, demonstrating that the pre-trained model-based debiasing method effectively improves the bias problem in extractive reading comprehension. To further evaluate the performance of this debiasing method, the model was trained on the full SQuAD1.1 dataset, resulting in a 0.08% improvement in accuracy. This shows that the pre-trained model-based reading comprehension debiasing method still performs well on unbiased datasets. The debiasing method proposed in this invention effectively alleviates the reading comprehension bias problem, improves the robustness of reading comprehension models based on pre-trained models, and is better suited for question-answering scenarios.

[0052] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A pre-training model-based reading comprehension debiasing method, characterized in that: Comprising the following steps: A text representation layer is constructed based on a pre-trained model BERT embedding layer, an article-question sentence pair is set as the input of the model, and the input sentence is mapped to a sentence vector; Two random regularization operations are performed on the position encoding to obtain two random sub-models and calculate the KL divergence to evaluate the difference of the output distribution, and the sentence vector is updated to input the BERT encoder; The output of the last hidden layer of the BERT encoder is passed through a fully connected layer and a classifier, and the prediction loss of the sample is calculated by the sum of the cross-entropy loss and the KL divergence, and the pre-training parameters are optimized according to the prediction loss of the sample during the model training process to remove the position bias of the reading comprehension; The dropout technique is used to perform two random 0 operations on the position encoding initialized by BERT with a probability p to obtain two position encoding vectors with weak position information, and the mean of the two vectors is added to the remaining encoding to obtain an updated sentence vector, which is represented by the formula: ； wherein, denotes the maximum length of the input sequence; denotes the length of each token vector representation; denotes the updated sentence vector, denotes the Token embedding of each token against the vocabulary, denotes the position encoding before improvement, denotes the Segment embedding to distinguish questions from articles, denotes the random zeroing operation.

2. The reading comprehension debiasing method based on a pre-trained model according to claim 1, characterized in that: The L is 512, and the d is 768.

3. The reading comprehension debiasing method based on a pre-trained model according to claim 1, characterized in that: The KL divergence calculation formula is: ； wherein represents the first group of training samples, represents the calculation of the KL divergence, and represents the position encoding after the dropout operation, two different sub-models produce different distributions, due to the asymmetry of the KL divergence itself, the positions of the two distributions need to be exchanged, and finally the average processed KL divergence is obtained.

4. The reading comprehension debiasing method based on a pre-trained model according to claim 1, characterized in that: The BERT encoder is a Transformer encoder structure, including 12 stacked encoding modules, the hidden state size of each encoding module is 768, and 12 attention heads are included.

5. The reading comprehension debiasing method based on a pre-trained model according to claim 1, characterized in that: The loss function comprises a cross-entropy and a KL divergence of the regularized position encoding, a ratio of the cross-entropy and the KL divergence of the regularized position encoding being adjustable by a hyperparameter The specific calculation formula of the cross-entropy loss and the total training loss sum is: ； ； wherein, denotes the set of training samples, denotes the number of samples, denotes the true start position of the answer in the sample, denotes the true end position of the answer in the sample, denotes the probability that each position of the predicted text sequence is the start position of the answer, denotes the probability that each position of the predicted text sequence is the end position of the answer, is a tunable hyperparameter that scales the cross-entropy with the KL divergence of the regularized position encoding, denotes the cross-entropy loss value, denotes the KL divergence value of the regularized position encoding, denotes the total training loss sum.

6. The reading comprehension debiasing method based on a pre-trained model according to claim 1, characterized in that: The data set used for model training is the extractive reading comprehension data set SQuAD1.1 and its sub data set divided according to the position of the answer in the article paragraph, and the corresponding KL divergence and cross entropy ratio when the best accuracy is reached is 3, and the dropout random 0 probability is is 0.1.

Images