A Selective Reading Comprehension Method Based on Multi-View Graph Encoding within a Joint Learning Framework
By employing a multi-view graph encoding method within a collaborative learning framework, the problem of low answer prediction accuracy in selective reading comprehension is addressed. Through multi-view modeling and information fusion, efficient answer prediction is achieved even without evidence annotation, thereby improving the performance of selective reading comprehension.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-22
- Publication Date
- 2026-03-10
AI Technical Summary
Existing selective reading comprehension methods struggle to effectively capture the multi-faceted relationships between documents, questions, and candidate answers during the answer selection process, resulting in low accuracy in answer prediction, especially when there is no evidence annotation.
We employ a multi-view graph encoding method under a joint learning framework. By constructing a statistical word co-occurrence graph, an inter-sentence distance graph, and a semantic similarity graph, we encode documents, questions, and candidate answers from multiple perspectives. We then use a graph convolutional neural network for information fusion and perform joint optimization by combining an evidence extraction module and an answer prediction module.
It improves the accuracy of answer prediction for selective reading comprehension tasks, especially in the absence of evidence annotation, significantly enhancing the effectiveness of answer prediction and reducing error propagation problems in the pipeline framework.
Smart Images
Figure CN115906862B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a selective reading comprehension method based on multi-view graph coding under a joint learning framework, and belongs to the technical field of natural language processing. BACKGROUND
[0002] Machine reading comprehension is a task of correctly answering a given question through understanding of a natural language document. Common machine reading comprehension tasks can be divided into four categories according to different question and answer forms: cloze test, multiple-choice, answer fragment extraction and free-form answer. The multiple-choice, namely the selective reading comprehension, requires a machine to automatically select one item as the correct answer to a given question according to analysis of a document. Since the answer is not directly extracted from the document but selected from multiple candidate answers, the relationship between the question, the document and the candidate answers needs to be analyzed to select the answer. The task requires the machine to have certain analysis and reasoning capabilities, and relevant research work is difficult, so it has attracted extensive attention from the academic field and is an important direction of machine reading comprehension tasks.
[0003] The core of the selective reading comprehension is to model the relevance among a document, a question and candidate answers, and select an option most consistent with the original expression as the answer to the question. According to different answer selection methods, the selective reading comprehension can be divided into exclusion method, correlation ranking method and matching method. Parikh and Sai proposed an answer exclusion model based on a neural network, which simulates the human reading comprehension method, calculates the document representation perceived by the question, constructs an elimination gate, calculates the relevance between the perceived document representation and the candidate answers, deletes the least relevant option through the relevance judgment, repeatedly iterates the exclusion of irrelevant options and obtains the final answer. Ran proposed an answer selection method based on an option comparison network, which adds comparison and analysis of the relationship between the candidate answers in the answer selection process, encodes each option into a vector, compares the vectors one by one through an attention mechanism and uses the correlation relationship to assist the answer selection. Duan proposed a multi-angle common matching model for selective reading comprehension, uses a multi-angle matching mechanism to obtain the relevance among a document, a question and candidate answers, weights the document representation by using the relevance, enhances the relevant document candidate fragments and further selects the correct answer based on the optimized document representation. SUMMARY
[0004] The application provides a selective reading comprehension method based on multi-view graph coding under a joint learning framework, which is used for solving the selective reading comprehension problem in the machine reading comprehension task and has achieved good results in the selective reading comprehension task.
[0005] The technical solution of this invention is: a selective reading comprehension method based on multi-view graph encoding under a joint learning framework, the specific steps of which are as follows:
[0006] Step 1: Analyze sentence S i Question and answer pairs (Q, A) j The document and question-answer pair representations are obtained through the pre-trained language model BERT, which are h respectively. i With h′ j The former is the i-th sentence S in the document. i The latter represents the relationship between question Q and the j-th candidate answer A. j The concatenated representation; using a pre-trained language model to encode the document and question-answer pairs (question and each candidate answer) to obtain the input form for BERT, this process can be represented as:
[0007] h′1, h′2, ..., h′ m h1, h2, ..., h n =BERT((Q,A1),(Q,A2),...,(Q,A m S1, S2, ..., S n )
[0008] Where h′ j Indicates candidate answer A j The representation of (i = 1, 2, ..., m) after BERT semantic encoding, h i S represents sentence S in the document i The representation of (i = 1, 2, ..., n) after BERT semantic encoding, (Q, A) j ) represents the question-and-answer pair obtained by concatenating "[SEP]";
[0009] Step 2: Construct a multi-view graph encoding module to construct graphs of sentences, questions and candidate answers in the document from three main different perspectives: statistical characteristics, relative distance and deep semantics. The three graphs constructed are a statistical word co-occurrence graph, an inter-sentence distance relationship graph and a semantic similarity relationship graph.
[0010] As a further aspect of the present invention, in step Step 2, the statistical word co-occurrence graph based on statistical characteristics is constructed based on tf-idf. Specifically, firstly, the tf-idf value of the question sentence is calculated, and words with high values are selected as important words. All sentences and question-answer pairs in the document are used as nodes. When important words co-occur between two nodes, an edge is connected between the two nodes.
[0011] As a further scheme of the present application, in the step Step2, for the inter-sentence distance relationship graph based on relative distance, a Gaussian distribution is used to measure the influence of the spatial distance between sentences in a document on the relevance, and the relevance of two sentences S i The distance between the sentences S j is calculated using the formula σ represents a hyperparameter, and a sentence distance threshold is set. When the distance value between two sentences exceeds the threshold, a link is connected between the two sentences.
[0012] As a further scheme of the present application, in the step Step2, for the semantic similarity relationship graph based on deep semantics, the cosine similarity is used to calculate the similarity between two sentences, and a similarity threshold is set. When the similarity between two nodes exceeds the threshold, a link is connected between the two nodes.
[0013] As a further scheme of the present application, in the step Step3, a two-layer perceptron is used to fuse the representations obtained from the three different perspective graphs to obtain a final representation.
[0014] Step3, using a graph convolutional neural network to convolve the three graphs constructed in Step2 respectively:
[0015]
[0016] where A u is an adjacency matrix, and A u =A word , A dist , A simi ; f u (·) is a graph convolution operation, and l is the number of graph convolution layers; is the representation of all sentences.
[0017] A two-layer perceptron is used to fuse the representations obtained from the three different perspective graphs, and the representation is represented as:
[0018]
[0019] where W a , b a , W, b are trainable parameters of the model, and || is a feature concatenation operation.
[0020] Step4, constructing an evidence extraction module, using a binary classifier to judge the possibility of each sentence in the document as evidence according to the information provided by the document and the question, and the process can be described as:
[0021]
[0022] where P 为 is the probability of the sentence Si evidence probability; is the representation of the sentence Si after graph convolution; || is the feature concatenation operation; W c , b c , W d , b d are trainable parameters of the model, where P can be both as the evidence probability and weighted to the representation of the sentence to improve the effect of answer prediction;
[0023] In Step 4, the constructed evidence extraction module is used to judge the possibility of each sentence in the document as the evidence information of answering the question. A softmax-based classifier is used to perform binary classification (evidence or non-evidence) on each sentence, and the classification probability is used as the possibility and weighted on the representation obtained after graph convolution and information fusion to assist the subsequent answer prediction.
[0024] Step 5, construct an answer prediction module, and use the probability of each sentence as evidence information obtained by the evidence extraction module as a weight to assist in answer prediction. The process is described as follows:
[0025]
[0026] K, V = W f (P i · H i ) + b f
[0027]
[0028] wherein represents the representation of the candidate answer A j after graph convolution; C is the representation obtained after cross-attention; W e , b e , W f , b f are trainable parameters of the model.
[0029] Finally, the output of the answer prediction module is obtained as follows:
[0030] P'1, P'2,..., P' m = softmax(W h h· C + b j )
[0031] wherein P' is the probability of the candidate answer A j as an answer; C is the representation obtained after cross-attention; W h , b h are trainable parameters of the model.
[0032] Wherein, the answer prediction module constructed fuses the question and answer pair representation obtained after the graph convolution information and the representation weighted by the evidence prediction probability, and the process is realized through cross attention.
[0033] In the joint learning framework, the evidence extraction and the answer prediction are optimized synchronously, the accuracy of the answer prediction is improved under the assistance of the evidence extraction, and the prediction of the evidence prediction module on the probability of the sentence as evidence is adjusted according to the feedback of the answer prediction result.
[0034] The beneficial effects of the present application are:
[0035] 1. The present application jointly encodes the document, question and candidate answer from multiple different perspectives through the multi-perspective graph encoding network, captures the correlation between the sentences in the document and between the document sentences and the question sentences from three perspectives of statistical characteristics, relative distance and deep semantics, fully excavates the potential evidence information, and obtains the document encoding perceived by the question and answer pair.
[0036] 2. The present application further judges whether each sentence in the document is an evidence sentence through a binary classifier to realize the function of the evidence extraction module; finally, an answer prediction module is constructed, the document encoding obtained by the evidence extraction module is weighted and selectively fused by using the probability of the document sentence as evidence, and the two modules are trained simultaneously under the joint learning framework, so that the target of the answer prediction is realized. The present application has achieved good results on the selective reading comprehension task.
[0037] 3. The method for modeling the document, question and candidate answer from multiple perspectives is effective; the training is carried out under the joint learning framework, the evidence extraction and the answer prediction are optimized jointly, and the joint learning framework can alleviate the error propagation problem in the pipeline framework.
[0038] 4. The method proposed in the present application can effectively model the relationship between the sentences in the document without evidence annotation, and improve the accuracy of the answer prediction. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The present application proposes a selective reading comprehension model based on multi-perspective graph encoding under a joint learning framework. DETAILED DESCRIPTION
[0040] As Figure 1 shown, a selective reading comprehension method based on multi-perspective graph encoding under a joint learning framework, the specific steps of the method are as follows:
[0041] S1, encode the document and the question-answer pair (question and each candidate answer) using a pre-trained language model; specifically, use the Transformers framework proposed by Wolf et al., and use bert_base with a hidden vector length of 768 as the initial model, and optimize the hyperparameters consistent with the original model;
[0042] S2, construct a multi-view graph encoding module to graph the sentences in the document, the questions and the candidate answers from three different main angles; specifically, the maximum length of each sentence is 64, if it exceeds 64, it is split into two sentences of the same length, the maximum number of sentences is 20, and the part greater than 20 is discarded, which is less than 1.8%;
[0043] The three angles are statistical properties, relative distance and deep semantics, and the three graphs constructed are statistical word co-occurrence graph, inter-sentence distance relationship graph and semantic similarity relationship graph;
[0044] The specific graphing methods of the three angles are as follows:
[0045] Step 2.1, the specific method of constructing the statistical word co-occurrence graph is: the statistical word co-occurrence graph based on statistical properties is constructed based on tf-idf, specifically, first calculate the tf-idf value of the question Q, select the words with a value greater than the threshold value 0.4 as important words; all sentences and question-answer pairs in the document are nodes, when important words co-occur between two nodes, a edge is connected between the two nodes. Its adjacency matrix is defined as:
[0046]
[0047] Where represents the set of words with high tf-idf value of sentence S i , represents that the i-th sentence S i and the j-th sentence S j have important word co-occurrence;
[0048] Step 2.2, for the inter-sentence distance relationship graph based on relative distance, use Gaussian distribution to measure the influence of the spatial distance between sentences in the document on the correlation;
[0049] The specific construction method of the inter-sentence distance relationship graph is: all sentences and question-answer pairs in the document are nodes in the graph; calculate the distance value between the nodes, for example, the distance between the sentence S i and the sentence S j uses the formula σ represents a hyperparameter, and a threshold value of the distance between sentences in the document is set, when the distance value exceeds the threshold value, a edge is connected between the two nodes, and its adjacency matrix is defined as:
[0050]
[0051] where δ d = 0.6 is the inter-sentence distance threshold;
[0052] Step 2.3, the specific construction method of the semantic similarity graph based on deep semantic is: all sentences and question-answer pairs in the document are nodes in the graph; the cosine similarity is used to calculate the similarity between two sentences, that is, the cosine similarity between all nodes is calculated; a semantic similarity threshold is set, when the cosine similarity between two nodes (sentences) is greater than the similarity threshold, a edge is connected between the two nodes, and its adjacency matrix is defined as:
[0053]
[0054] where sim(·) represents the calculation of cosine similarity, sim(S i ,S j ) represents the semantic similarity between sentences S i and S j ; δ s = 0.8 is the similarity threshold.
[0055] The GCN module in the dgl library is used to encode the three graphs constructed, where the number of GCN layers is 2 and the length of the hidden state is 256;
[0056] S3, use the graph convolutional neural network to convolve the three graphs constructed in Step 2 respectively:
[0057]
[0058] where A u is the adjacency matrix, and A u = A word , A dist , A simi ; f u (·) is the graph convolution operation, and l is the number of graph convolution layers; is the representation of all sentences.
[0059] A two-layer perceptron is used to fuse the representations obtained from the three different perspective graphs, which is represented as:
[0060]
[0061] where W a , b a , W, and b are trainable parameters of the model, and || is a feature concatenation operation.
[0062] Step 4, build an evidence extraction module, according to the information provided by the document and the question, use a binary classifier to judge the possibility of each sentence in the document as evidence;
[0063] The loss calculation of the evidence extraction module is defined as follows:
[0064]
[0065] where y i is the evidence label of the sentence S i , P i is the probability of the model classifying the sentence S i as evidence.
[0066] S5, an answer prediction module is constructed, and the probability of each sentence obtained by the evidence extraction module as evidence information is used as a weight to assist answer prediction. Model training uses an AdamW optimizer, the joint loss hyperparameter setting is alpha=0.25, the learning rate is 4e-5, the training batch size is 8, a total of 6 rounds of training are performed, and the model with the best accuracy is selected for saving every 15,000 steps, wherein the loss calculation of the evidence extraction module, the answer prediction module and the joint learning is as follows:
[0067] The loss calculation of the answer prediction module is defined as follows:
[0068]
[0069] where P′ j is the probability of the model judging that the candidate answer A j is the correct answer, and y A is the correct answer.
[0070] The evidence extraction and answer prediction tasks are trained in a joint learning framework, and the parameters of the two tasks are obtained through joint optimization. The calculation formula of the joint loss is as follows:
[0071]
[0072] where alpha is a hyperparameter.
[0073] The present application proposes a method of multi-angle graph encoding selective reading comprehension under a joint learning framework, which is respectively used in the scene with evidence labels (joint learning framework applicable: MAGE-JF and MAGE) and the scene without evidence labels (joint learning framework not applicable: MAGE). The methods used for comparative experimental effects are as follows:
[0074] BIDAF: a model that uses multi-stage, hierarchical processing to capture features of different granularities of the original text. Long Short-Term Memory (LSTM) is used as its encoder, and a bidirectional attention flow mechanism is used to model the relationship between the question and the answer.
[0075] BIDAF*: Replaces the traditional word embeddings in BiDAF with ELMO, a language model trained on unsupervised data, resulting in better performance.
[0076] BERT_doc: A multi-layered Transformer encoder pre-trained on a large amount of unlabeled data and outperforms state-of-the-art models on many natural language processing tasks. BERT_doc represents the input of only the document as reference information without evidence labels when performing reading comprehension.
[0077] BERT_evi: Encodes the document, question, and candidate answer using BERT. BERT_evi represents the input of annotated evidence sentences as reference information with evidence labels when performing reading comprehension. (Only the ReCO dataset has evidence labels, and the RACE dataset does not have this baseline model experiment.)
[0078] For the pipeline framework model, we train a pipeline machine reading comprehension model based on evidence extraction and a machine reading comprehension model based on evidence generation:
[0079] Pipeline_ex: To compare the evidence extraction pipeline framework with the joint learning framework of evidence extraction assisted answer prediction, we design a pipeline that uses MAGE to encode the document, the evidence extraction module and the answer prediction module are independent, and the answer prediction is based on evidence extraction.
[0080] Pipeline_gen: A pipeline framework based on an encoder-decoder framework is designed. To compare the performance of the pipeline framework and the joint learning framework, this baseline method encodes the document through MAGE, then uses an LSTM decoder to generate evidence, and performs answer prediction on the generated evidence.
[0081] Table 1 shows the accuracy of BiDAF, BERT, and the test set validation set of the invention on the ReCO public dataset on the selective reading comprehension task with / without evidence labels (MAGE-JF / MAGE) and other methods.
[0082] Table 1 Selective reading comprehension accuracy of different models
[0083] Model \ Dataset ReCO_Dev (%) ReCO_Test (%) Random 33.3 33.3 BiDAF 55.8 56.4 BiDAF* 57.5 58.9 BERT_evi 76.3 77.3 BERT_doc 61.4 61.1 Pipeline_ex 62.5 62.9 Pipeline_gen 61.9 62.1 MAGE 64.3 64.7 MAGE-JF 65.1 65.4
[0084] From Table 1, it can be seen that the present application is improved by 9.3 points compared with BiDAF, which shows that modeling the document and the question from the semantic perspective is far from enough to answer the question, and multi-perspective modeling of the document, the question and the candidate answer greatly promotes the answer prediction of the selective reading comprehension; compared with the baseline method based on BERT using the document as the reference information input, the present application is improved by 3.7 points, which shows the effectiveness of the method of modeling the document, the question and the candidate answer from multiple perspectives; the present application is improved by 2.6 points compared with the method of the pipeline framework, and under the condition that the evidence extraction method and the answer prediction method are the same, the present application is trained in the joint learning framework, and the evidence extraction and the answer prediction are jointly optimized, while the two tasks are trained respectively in the pipeline framework, so it can be seen that the joint learning framework can alleviate the problem of error propagation in the pipeline framework.
[0085] In order to evaluate the effectiveness of the present application in the no evidence label scenario, experiments are carried out on the RACE data set, and compared with other mainstream methods, and the experimental results are shown in Table 2.
[0086] Table 2 Accuracy rate table of selective reading comprehension in no evidence label scenario
[0087] Model \ Dataset RACE (%) RACE-M (%) RACE-H (%) Random 25 25 25 BiDAF_doc 38.2 40.2 41.8 BiDAF*_doc 40.9 43.6 43.3 BERT_doc 65.0 71.7 62.3 MAGE 66.1 72.1 64.5
[0088] From Table 2, it can be seen that MAGE achieves the best performance on RACE-M, RACE-H and RACE compared with other methods, which shows the effectiveness of MAGE. From the experimental results, it can be seen that the method MAGE of the present application is improved by 2.2 points in performance compared with BERT_doc, which fully shows the effectiveness of modeling the document, the question and the candidate answer from multiple perspectives on answer prediction, that is, the potential evidence information in the document can be effectively mined by encoding the document from multiple perspectives, which proves that the method proposed in the present application can effectively model the relationship between sentences in the document without evidence annotation, and improve the accuracy of answer prediction. From Table 3, it can be seen that the improvement effect of RACE-H is higher than that of RACE-M, but according to the construction rules of the data set, the text prediction difficulty in RACE-H is higher than that in RACE-M. From this point, it can be seen that multi-perspective graph encoding can better mine evidence information from more complex text to assist answer prediction.
[0089] Table 3 Multi-perspective graph encoding module ablation experiment
[0090] Model \ Dataset ReCO (%) RACE-H (%) BERT_doc 61.1 62.3 + spatial graph 61.4 62.3 + statistical graph 61.9 62.4 + semantic graph 62.1 62.6 + spatial & statistical graph 62.4 62.9 + spatial & semantic graph 63.3 63.1 + statistical & semantic graph 63.9 63.4 MAGE 64.3 64.7
[0091] Table 3 shows the effectiveness of each composition method in the multi-view graph encoding module, where '+' represents adding the composition method of the specified view on the basis of the BERT encoder. As can be seen from the second to fourth rows of Table 3, the three different view graphs are ranked as follows in terms of the final answer prediction performance: semantic graph > statistical graph > spatial graph. By analyzing the experimental results, it can be seen that the reason why the spatial graph has the smallest contribution is that BERT is used as the basic encoder, which has a position embedding operation, so the spatial position information between the sentences of the document has been considered. At the same time, semantic information is an important part of judging the relevance of the document and the question, so the semantic graph has the largest contribution to the answer prediction performance. Since the BERT pre-training language model can effectively capture semantic information, the semantic graph constructed based thereon has good representation ability.
[0092] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.
Claims
1. A method of selective reading comprehension based on multi-view graph encoding under a federated learning framework, characterized in that: The specific steps of the method are as follows: Step 1, using a pre-trained language model to encode the document and the question-answer pair, the question-answer pair including the question and each candidate answer; Step 2, constructing a multi-view graph encoding module to graph the sentences in the document, the question and the candidate answer from three different angles; Step 3, using a graph convolutional neural network to convolve the three graphs constructed in Step 2 respectively, and performing information fusion; Step 4, constructing an evidence extraction module to determine the possibility of each sentence in the document as evidence according to the information provided by the document and the question; Step 5, constructing an answer prediction module to use the probability of each sentence as evidence information obtained by the evidence extraction module as a weight to assist answer prediction according to the candidate answer information; In Step 2, the document and the question-answer pair are graphed from three different angles, namely statistical characteristics, relative distance and deep semantics, and three graphs are constructed, namely a statistical word co-occurrence graph, an inter-sentence distance relationship graph and a semantic similarity relationship graph.
2. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In the step Step1, the sentence and the question-answer pair are represented by the pre-trained language model BERT, respectively and , where the former is the representation of the th sentence in the document, and the latter is the representation of the question concatenated with the th candidate answer .
3. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 2, the statistical word co-occurrence graph based on statistical characteristics is constructed based on tf-idf, specifically, the tf-idf value of the question sentence is calculated, the words with high value are selected as important words, all sentences in the document and the question-answer pair are taken as nodes, and an edge is connected between two nodes when important words co-occur between the two nodes.
4. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 2, for the inter-sentence distance relationship graph based on relative distance, a Gaussian distribution is used to measure the impact of spatial distance between sentences within a document on relevance, and the relationship between sentences in the document is calculated. With sentences The distance between them is calculated using the formula. , This represents a hyperparameter and sets a threshold for the distance between sentences. When the distance between two sentences exceeds the threshold, an edge is connected.
5. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 2, the semantic similarity relationship graph based on deep semantics calculates the similarity between two sentences using cosine similarity, and sets a similarity threshold, when the similarity between two nodes exceeds the threshold, an edge is connected between the two nodes.
6. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 3, a two-layer perceptron is used to fuse the representations obtained from the three different view graphs to obtain the final representation.
7. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 4, the constructed evidence extraction module is used to determine the possibility of each sentence in the document as evidence information to answer the question, a softmax-based classifier is used to perform binary classification on each sentence, which is evidence or non-evidence, and the classification probability is used as the possibility to weight the representation obtained after graph convolution and information fusion to assist subsequent answer prediction.
8. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 5, the constructed answer prediction module uses the question-answer pair representation obtained after graph convolution information fusion and the representation weighted by the evidence prediction probability to predict the answer, which is achieved through cross-attention.
9. The method of claim 1, wherein the method is based on multi-view graph encoding under a federated learning framework. In Step 5, under the framework of joint learning, the evidence extraction and answer prediction parts are optimized simultaneously, the accuracy of answer prediction is improved with the assistance of evidence extraction, and the prediction of the probability of sentences as evidence by the evidence prediction module is adjusted according to the feedback of the answer prediction result.