A method to optimize machine reading comprehension ability based on brain-like attention mechanism

Abstract

The present invention discloses a method for optimizing machine reading comprehension ability based on brain-like attention mechanism, which collects reading eye movement data through cognitive science experiments, takes fixation duration as an indicator, quantifies the reading attention distribution of the human brain; extracts bottom-up text features, visual features and top-down question-related features, and uses linear regression method to fit the reading attention distribution of the human brain; finally, by maximizing the model accuracy and minimizing the cross entropy loss function between the word attention weight of the output layer of the attention model and the estimated value of the word attention distribution of the human brain, the reading comprehension model parameters are updated by back propagation to obtain the optimal model attention mechanism. The present invention not only effectively improves the accuracy of machine reading comprehension tasks, but also can be used to explain model reading comprehension behavior, providing a basis for further optimization of machine reading comprehension systems.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a method for optimizing machine reading comprehension ability based on a brain-like attention mechanism. Background Art

[0002] Reading comprehension, as a classic information processing problem, can be divided into two modes: sequential browsing and search-based reading. The latter is usually aimed at meeting specific information needs and is more common in real scenarios. This problem-oriented reading comprehension involves screening and integrating long-term dynamic information flows, and the human brain has limited cognitive resources for processing information. Therefore, the human brain will use attention mechanisms to selectively allocate cognitive resources and conduct in-depth processing of key information related to reading problems. In recent years, the field of artificial intelligence has also introduced the concept of attention mechanisms in artificial neural networks. For text sequence information processing problems, the attention model assigns different weights to each unit in the sequence based on its importance, thereby effectively improving the model's natural language understanding ability.

[0003] Machine reading comprehension requires machines to read and understand human natural language texts, and on this basis, answer questions related to text information. This task is usually used to measure the machine's natural language comprehension ability and is one of the research hotspots in the field of natural language processing. A typical machine reading comprehension system generally includes four modules: embedding coding, feature extraction, article-question interaction, and answer prediction (ref: Neural Machine Reading Comprehension: Methods and Trends): Embedding coding is responsible for converting natural language articles and questions into vectors of fixed dimensions for machine processing. Common encoding methods include one-hot representation, distributed word vectors (Glove, Word2Vec), and context-based word representation methods such as ELMo, GPT, and Bert. The feature extraction module is used to extract context information. Common neural network models include RNN\CNN and transformer structures based on multi-head self-attention mechanisms. The interaction module is responsible for capturing the relationship between articles and questions. It usually uses an attention mechanism to calculate the semantic relevance between each word in the article and question, thereby simulating the behavior of humans in reading comprehension, repeated reading, and mining key information. Finally, the prediction layer predicts the final answer based on the information accumulated by the first three modules according to different task types. In recent years, deep learning reading comprehension models based on attention mechanisms have been proposed. Such models can better mine the contextual semantic information of texts and effectively improve machine reading comprehension capabilities. At the same time, a series of large-scale machine reading comprehension datasets have been released, providing a data basis for training and evaluating deep network models.

[0004] However, a study (Jia & Liang) pointed out that the existing machine reading comprehension system has a significantly reduced effect on adversarial samples with interference sentences, while humans are able to perform normal reading comprehension without being affected by such adversarial attacks, which to some extent shows that the existing model cannot truly understand natural language like humans. On the other hand, the training of most current neural network reading comprehension models relies on the optimization of answer prediction accuracy, while ignoring the review and correction of the model's intermediate results - the attention mechanism, resulting in the model's inability to have human-like reading comprehension capabilities at the mechanism level. Summary of the invention

[0005] The purpose of the present invention is to provide a method for optimizing machine reading comprehension ability based on a brain-like attention mechanism to address the shortcomings of existing methods.

[0006] The object of the present invention is achieved through the following technical solution: a method for optimizing machine reading comprehension ability based on a brain-like attention mechanism, comprising the following steps:

[0007] Step 1: Obtain reading text and corresponding reading comprehension questions, collect eye movement data of human reading, and quantify the eye movement data to obtain the distribution of human reading attention.

[0008] Step 2: Perform feature engineering on the reading text, select top-down task relevance and bottom-up statistical values ​​as features, and obtain the text feature vector for predicting human attention distribution.

[0009] Step 3: Use the text feature vector corresponding to the corpus as the input of the linear regression model, and the human reading attention distribution on the articles in the corpus as the observation value to model the human attention distribution of the article word sequence;

[0010] Step 4: Use the predicted human attention distribution as a brain-like attention mechanism and introduce an artificial neural network attention model. By minimizing the difference in attention distribution between the human brain and the model on the text, the model is guided to generate a problem-related brain-like attention score calculation mechanism.

[0011] Step 5: Input the question and the corresponding reading text into the trained brain-like attention mechanism machine reading model, and calculate the answer based on the output layer.

[0012] Furthermore, the step 1 is implemented by the following sub-steps:

[0013] (1.1) Select and adapt reading comprehension questions according to the needs of cognitive science experiments to form reading comprehension corpus suitable for specific experimental paradigms.

[0014] (1.2) Based on the experimental paradigm of "reading the questions first, then reading the article, and finally looking at the questions and answering them", a cognitive science experiment was conducted to record the eye movement data of the subjects during the reading comprehension process.

[0015] (1.3) For each article, all the words it contains constitute the article word sequence P = {t1, t2, ..., t n}, where n is the length of the article sequence, t1, t2, ..., t n Denote different words t in the article P, obtain the fixation time of the article word sequence of all subjects who correctly answered the corresponding questions, calculate the average fixation time on each unit interest area (here, words are used as unit interest areas, and the method is not limited to words when implemented), and use it as the human attention distribution on the article word sequence in is the weight for each word t.

[0016] Furthermore, the step 2 is implemented by the following sub-steps:

[0017] (2.1) Extract word-level features for the article word sequence: select part-of-speech tagging, named entity recognition, and word length as features F 1 , Feature F 2 , Feature F 3 ;

[0018] (2.2) Extracting chapter-level features for article word sequences: Obtain the corpus vocabulary based on all the words contained in the articles and questions in the corpus, and calculate the frequency of each word in each article as feature F4. The specific calculation process is as follows: represents the frequency of the jth word in the i-th article, d represents the number of articles in the corpus, and n represents the number of words in the corresponding article;

[0019]

[0020] (2.3) Extract visual features from article sequences: Use natural language processing tools to segment, sentence, and stem the article text to obtain the sentence-word sequence of the article The first digit of the superscript indicates the paragraph number of the article corresponding to the word, the second digit indicates the sentence number of the article corresponding to the word, p and s are the number of paragraphs and sentences in the article respectively; the paragraph number, sentence number and word number are selected as the feature F 5 , Feature F 6 , Feature F 7 ;

[0021] (2.4) Extracting semantic similarity features for question sequence and article sequence: Using natural language processing tools, stemming and removing stop words are performed on the article text and question text, and the article word sequence P = {t1, t2, ..., t n} and question word sequence Q = {q1, q2, ..., q m}, where m is the length of the question sequence, q1, q2, ..., q m Represent different words q in the question text; select a GloVe vector with a dimension of 300 to embed the words in the article word sequence and the question word sequence, and weighted average all vectors in the question sequence to obtain the question sequence vector; use cosine similarity to calculate the similarity between each word in the article word sequence and the question sequence vector as feature F 8 ;

[0022] (2.5) Extract question-related features from question text: select part-of-speech tagging, named entity recognition, and question type as features F respectively. 9 , Feature F 10 , Feature F 11 .

[0023] Furthermore, the step three is implemented by the following sub-steps:

[0024] (3.1) Using a linear regression model, the feature value of each word extracted in step 3 is used as input to model the human brain attention distribution during reading comprehension and obtain the predicted value of the human reading comprehension attention score for each word in the corresponding article.

[0025]

[0026] in, Represents word t i The eigenvalue corresponding to the kth feature, k is the 11 different text features in step 3, β k and b are model parameters, initialized to random values;

[0027] (3.2) The human attention distribution observations obtained in step 1 As the model optimization goal, the model parameters β are updated by minimizing the mean square error MSE. k and b, thus obtaining a linear regression model trained to predict human attention distribution:

[0028]

[0029] Among them, d and n are the number of articles and the number of words in the corresponding articles respectively. and They represent the observed and predicted values ​​of the human brain attention weight of the jth word in the i-th article respectively.

[0030] Furthermore, the step 4 is implemented by the following sub-steps:

[0031] (4.1) Preprocess the articles and question texts in the corpus, input the trained attention model, output through the attention layer, and obtain the model's attention distribution prediction value for the article word sequence The specific calculation formula is as follows:

[0032]

[0033] in, represents the attention weight prediction value of each word in the article, P represents the input text sequence, and W and b are model parameters.

[0034] (4.2) For the same set of article-question data, calculate the human attention distribution weight of each word according to the method described in step 4 Through the cross entropy loss function Minimize the difference between the human brain's attention distribution and the attention model's attention distribution:

[0035]

[0036] (4.3) For the same set of article-question data, the correct answer to the question is y i , the attention model predicts the answer to be Through the cross entropy loss function Maximize the model’s reading comprehension accuracy:

[0037]

[0038] (4.4) For each set of data in the corpus, obtain its overall cross entropy loss function Loss total , through all the article-question data in the training set corpus, minimize the model's answer error rate and the difference between the model's attention distribution and the brain-like attention distribution, so as to guide the model to generate a problem-related brain-like attention score calculation mechanism:

[0039]

[0040] Furthermore, the question and the corresponding article are input into the trained reading comprehension model, and through the calculation of the brain-like attention mechanism, the attention representation used to perform specific tasks in the output layer is obtained, which can be used to obtain the final answer.

[0041] The beneficial effect of the present invention is that the present invention collects reading eye movement data through cognitive science experiments, takes fixation duration as an indicator, quantifies the reading attention distribution of the human brain; extracts bottom-up text features, visual features and top-down question-related features, and uses linear regression method to fit the reading attention distribution of the human brain; finally, by maximizing the model accuracy and minimizing the cross entropy loss function between the word attention weight of the output layer of the attention model and the estimated value of the word attention distribution of the human brain, the reading comprehension model parameters are updated by back propagation to obtain the optimal model attention mechanism. This reading comprehension model based on brain-like attention mechanism not only effectively improves the accuracy of machine reading comprehension tasks, but also can be used to explain model reading comprehension behavior, providing a basis for further optimization of machine reading comprehension systems. DETAILED DESCRIPTION

[0042] The present invention provides a method for optimizing machine reading comprehension ability based on a brain-like attention mechanism, comprising the following steps:

[0043] Step 1: Obtain reading text and corresponding reading comprehension questions, collect eye movement data of human reading, and quantify the eye movement data to obtain human reading attention distribution. This step 1 is achieved through the following sub-steps:

[0044] (1.1) Select and adapt reading comprehension questions according to the needs of cognitive science experiments to form reading comprehension corpus suitable for specific experimental paradigms.

[0045] (1.2) Based on the experimental paradigm of "reading the questions first, then reading the article, and finally looking at the questions and answering them", a cognitive science experiment was conducted to record the eye movement data of the subjects during the reading comprehension process.

[0046] (1.3) For each article, all the words it contains constitute the article word sequence P = {t1, t2, ..., t n}, where n is the length of the article sequence. Obtain the fixation time of the article word sequence of all subjects who correctly answered the corresponding questions, and calculate the average fixation time on each unit interest area (here, words are used as unit interest areas, and the method is not limited to words when implemented) as the human attention distribution on the article word sequence

[0047] Step 2: Perform feature engineering on the reading text, select top-down task relevance and bottom-up statistical values ​​as features, and obtain the text feature vector for predicting human attention distribution. This step is achieved through the following sub-steps:

[0048] (2.1) Extract word-level features for the article word sequence: select part-of-speech tagging, named entity recognition, and word length as features F 1 , Feature F 2 , Feature F3 .

[0049] (2.2) Extract paragraph-level features for article word sequences: Obtain the corpus vocabulary based on all the words contained in the articles and questions in the corpus, and calculate the frequency of each word in each article as the feature F 4 The specific calculation process is as follows: is the frequency of the jth word in the i-th article, d represents the number of articles in the corpus, and n is the number of words in the corresponding article.

[0050]

[0051] (2.3) Extract visual features from article sequences: Use natural language processing tools to segment, sentence, and stem the article text to obtain the sentence-word sequence of the article The first digit of the superscript indicates the paragraph number of the article corresponding to the word, the second digit indicates the sentence number of the article corresponding to the word, p and s are the number of paragraphs and sentences in the article respectively; the paragraph number, sentence number and word number are selected as the feature F 5 , Feature F 6 , Feature F 7 .

[0052] (2.4) Extracting semantic similarity features for question sequence and article sequence: Using natural language processing tools, stemming and removing stop words are performed on the article text and question text, and the article word sequence P = {t1, t2, ..., t n} and question word sequence Q = {q1, q2, ..., q m}, where m is the length of the question sequence, q1, q2, ..., q m Represent different words q in the question text; select a GloVe vector with a dimension of 300 to embed the words in the article word sequence and the question word sequence, and weighted average all vectors in the question sequence to obtain the question sequence vector; use cosine similarity to calculate the similarity between each word in the article word sequence and the question sequence vector as feature F8.

[0053] (2.5) Extract question-related features from question text: select part-of-speech tagging, named entity recognition, and question type as features F respectively. 9 , Feature F 10 , Feature F 11 .

[0054] Step 3: Use the text feature vector corresponding to the corpus as the input of the linear regression model, and the human reading attention distribution on the articles in the corpus as the observation value to model the human attention distribution of the article word sequence; this step is achieved through the following sub-steps:

[0055] (3.1) Using a linear regression model, the feature value of each word extracted in step 3 is used as input to model the human brain attention distribution during reading comprehension and obtain the predicted value of the human reading comprehension attention score for each word in the corresponding article.

[0056]

[0057] in, Represents word t i The eigenvalue corresponding to the kth feature, k is the 11 different text features in step 3, β k and b are model parameters, initialized to random values.

[0058] (3.2) The human attention distribution observations obtained in step 1 As the model optimization goal, the model parameters β are updated by minimizing the mean square error MSE. k and b, thus obtaining a linear regression model trained to predict human attention distribution:

[0059]

[0060] Among them, d and n are the number of articles and the number of words in the corresponding articles respectively. and They represent the observed and predicted values ​​of the human brain attention weight of the jth word in the i-th article respectively.

[0061] Step 4: Use the predicted human attention distribution as a brain-like attention mechanism and introduce an artificial neural network attention model to guide the model to generate a brain-like attention score calculation mechanism related to the question by minimizing the difference in attention distribution between the human brain and the model on the text. This step is achieved through the following sub-steps:

[0062] (4.1) Preprocess the articles and question texts in the corpus, input the trained attention model, output through the attention layer, and obtain the model's attention distribution prediction value for the article word sequence The specific calculation formula is as follows:

[0063]

[0064] in, represents the attention weight prediction value of each word in the article, P represents the input text sequence, and W and b are model parameters.

[0065] (4.2) For the same set of article-question data, calculate the human attention distribution weight of each word according to the method described in step 4 Through the cross entropy loss function Minimize the difference between the human brain's attention distribution and the attention model's attention distribution:

[0066]

[0067] (4.3) For the same set of article-question data, the correct answer to the question is y i , the attention model predicts the answer to be Through the cross entropy loss function Maximize the model’s reading comprehension accuracy:

[0068]

[0069] (4.4) For each set of data in the corpus, obtain its overall cross entropy loss function Loss total , through all the article-question data in the training set corpus, minimize the model's answer error rate and the difference between the model's attention distribution and the brain-like attention distribution, so as to guide the model to generate a problem-related brain-like attention score calculation mechanism:

[0070]

[0071] Step 5: Input the question and the corresponding article into the trained reading comprehension model. Through the calculation of the brain-like attention mechanism, the attention representation used to perform specific tasks in the output layer is obtained, which can be used to obtain the final answer.

Claims

1. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism, characterized in that: The following steps are involved: Step 1: Obtain reading text and corresponding reading comprehension questions, collect eye movement data of human reading, and quantify the eye movement data to obtain human reading attention distribution; Step 2: Perform feature engineering on the reading text, select top-down task relevance and bottom-up statistical values ​​as features, and obtain the text feature vector for predicting human attention distribution; Step 3: Use the text feature vector corresponding to the corpus as the input of the linear regression model, and the human reading attention distribution on the articles in the corpus as the observation value to model the human attention distribution of the article word sequence; Step 4: Use the predicted human attention distribution as a brain-like attention mechanism and introduce an artificial neural network attention model. By minimizing the difference in attention distribution between the human brain and the model on the text, the model is guided to generate a brain-like attention score calculation mechanism related to the question. Step 5: Input the question and the corresponding reading text into the trained brain-like attention mechanism machine reading model, and calculate the answer based on the output layer.

2. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism according to claim 1, characterized in that: The step 1 is implemented by the following sub-steps: (1.1) Select and adapt reading comprehension questions according to the needs of cognitive science experiments to form reading comprehension corpus suitable for specific experimental paradigms; (1.2) Based on the experimental paradigm of "read the questions first, then read the article, and finally look at the questions and answer them", we conducted cognitive science experiments and recorded the eye movement data of the subjects during the reading comprehension process; (1.3) For each article, all the words it contains constitute the article word sequence P = {t1, t2, ..., t n }, where n is the length of the article sequence, t1, t2, ..., t n Denote different words t in the article P, obtain the fixation time of the article word sequence of all subjects who correctly answered the corresponding questions, calculate the average fixation time on each unit interest area (here, words are used as unit interest areas, and the method is not limited to words when implemented), and use it as the human attention distribution on the article word sequence in is the weight for each word t.

3. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism according to claim 1, characterized in that: The step 2 is implemented by the following sub-steps: (2.1) Extract word-level features for the article word sequence: select part-of-speech tagging, named entity recognition, and word length as features F 1 , Feature F 2 , Feature F 3 ; (2.2) Extract paragraph-level features for article word sequences: Obtain the corpus vocabulary based on all the words contained in the articles and questions in the corpus, and calculate the frequency of each word in each article as the feature F 4 The specific calculation process is as follows: represents the frequency of the jth word in the i-th article, d represents the number of articles in the corpus, and n represents the number of words in the corresponding article; (2.3) Extract visual features from article sequences: Use natural language processing tools to segment, sentence, and stem the article text to obtain the sentence-word sequence of the article The first digit of the superscript indicates the paragraph number of the article corresponding to the word, the second digit indicates the sentence number of the article corresponding to the word, p and s are the number of paragraphs and sentences in the article respectively; the paragraph number, sentence number and word number are selected as the feature F 5 , Feature F 6 , Feature F 7 ; (2.4) Extracting semantic similarity features for question sequence and article sequence: Using natural language processing tools, stemming and removing stop words are performed on the article text and question text, and the article word sequence P = {t1, t2, ..., t n } and question word sequence Q = {q1, q2, ..., q m }, where m is the length of the question sequence, q1, q2, ..., q m Represent different words q in the question text; select a GloVe vector with a dimension of 300, embed the words in the article word sequence and the question word sequence, and take the weighted average of all vectors in the question sequence to obtain the question sequence vector; Using cosine similarity, we calculate the similarity between each word in the article word sequence and the question sequence vector as feature F 8 ; (2.5) Extract question-related features from question text: select part-of-speech tagging, named entity recognition, and question type as features F respectively. 9 , Feature F 10 , Feature F 11 .

4. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism according to claim 1, characterized in that: The step three is implemented by the following sub-steps: (3.1) Using a linear regression model, the feature value of each word extracted in step 3 is used as input to model the human brain attention distribution during reading comprehension and obtain the predicted value of the human reading comprehension attention score for each word in the corresponding article. in, Represents word t i The eigenvalue corresponding to the kth feature, k is the 11 different text features in step 3, β k and b are model parameters, initialized to random values; (3.2) The human attention distribution observations obtained in step 1 As the model optimization goal, the model parameters β are updated by minimizing the mean square error MSE. k and b, thus obtaining a linear regression model trained to predict human attention distribution: Among them, d and n are the number of articles and the number of words in the corresponding articles respectively. and They represent the observed and predicted values ​​of the human brain attention weight of the jth word in the i-th article respectively.

5. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism according to claim 1, characterized in that: The step 4 is implemented by the following sub-steps: (4.1) Preprocess the articles and question texts in the corpus, input the trained attention model, and output through the attention layer to obtain the model's attention distribution prediction value for the article word sequence The specific calculation formula is as follows: in, represents the attention weight prediction value of each word in the article, P represents the input text sequence, and W and b are model parameters; (4.2) For the same set of article-question data, calculate the human attention distribution weight of each word according to the method described in step 4 Through the cross entropy loss function Minimize the difference between the human brain's attention distribution and the attention model's attention distribution: (4.3) For the same set of article-question data, the correct answer to the question is y i , the attention model predicts the answer to be Through the cross entropy loss function Maximize the model’s reading comprehension accuracy: (4.4) For each set of data in the corpus, obtain its overall cross entropy loss function Loss total , through all the article-question data in the training set corpus, minimize the model's answer error rate and the difference between the model's attention distribution and the brain-like attention distribution, so as to guide the model to generate a problem-related brain-like attention score calculation mechanism:

6. A method for optimizing machine reading comprehension ability based on brain-like attention mechanism according to claim 1, characterized in that: The question and the corresponding article are input into the trained reading comprehension model. Through the calculation of the brain-like attention mechanism, the attention representation used to perform specific tasks in the output layer is obtained, which can be used to obtain the final answer.

Images