A self-supervised textbook question answering method based on heterogeneous hypergraphs

Abstract

This invention relates to a self-supervised textbook question-answering method based on heterogeneous hypergraphs, belonging to the field of education. It includes: text retrieval: obtaining the most relevant background knowledge based on the question and candidate answers for each question; construction of a heterogeneous hypergraph for textbook question answering: representing the question, candidate answers, background knowledge, and their relationships using a heterogeneous hypergraph; self-supervised pre-training with text matching: using the results of text retrieval as a supervisory signal to enhance the model's ability to model the relationships between questions and knowledge points; and fine-tuning: fine-tuning the pre-trained heterogeneous hypergraph neural network using labeled data. This invention solves the problem of modeling high-order relationships between questions, answers, and knowledge points in textbook question answering; it employs self-supervised text matching pre-training to reduce the model's dependence on a large number of labeled textbook question-answering data samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of educational applications, and in particular to a self-supervised textbook question answering method based on a heterogeneous hypergraph. BACKGROUND

[0002] Textbook question answering refers to the task of automatically answering questions in textbooks using natural language processing and machine learning techniques. This task is of great significance for the development and utilization of educational resources, as it can help students better understand the content of the textbook and improve their learning efficiency. However, due to the high complexity and non-structure of the data in textbook question answering, how to build an effective model to solve this problem has always been a difficult problem faced by academia and industry. Current textbook question answering methods can be divided into two categories: deep learning-based methods and knowledge graph and graph neural network-based methods.

[0003] Deep learning-based methods focus on extracting text features and graph features through deep neural networks, and predicting the correct answer through feature fusion methods. In recent years, with the rise of Transformer in the multi-modal field, Xu et al. proposed a new model MoCA based on Transformer, which integrates multi-stage domain pre-training and multi-modal cross-attention, solving the feature fusion problem of a large number of multi-modal inputs. However, deep learning-based methods often have difficulty directly processing and modeling complex relationships and structures, such as dependency relationships between concepts, hierarchical relationships, and reasoning chains, due to the use of deep neural networks such as LSTM, CNN, and Transformer.

[0004] In contrast, knowledge graph and graph neural network-based methods can better capture and model these complex relationships. Ma et al. proposed a relation-aware fine-grained reasoning (RAFR) network that uses graph neural networks to extract knowledge from graphs, proposed a relationship detection algorithm based on semantic dependency relationships and relative positions between nodes within the graph, and used text background knowledge to assist in the generation of visual graphs. Based on the teaching graph and the question-guided attention, the problem graph node is reasoned, realizing fine-grained knowledge reasoning. Daesik Kim et al. represent concepts and entities in textbooks as nodes and the relationships between them as edges, combining text and image information in textbooks to form a multi-modal background knowledge graph, and proposed a fusion GCN (f-GCN) to extract features from the graph. SUMMARY

[0005] The technical problem to be solved by the present application is:

[0006] In order to solve the problem of high-order correlation modeling among the question, the answer and the knowledge point in the textbook question and answer, the application provides a self-supervised textbook question and answer method based on a heterogeneous hypergraph.

[0007] In order to solve the above technical problems, the technical scheme adopted by the application is:

[0008] A self-supervised textbook question and answer method based on a heterogeneous hypergraph, characterized by comprising the following steps:

[0009] Step 1: Obtain a textbook question and answer data set Where N represents the number of questions, q i represents the ith question, A i =[a1,...,a k ] represents the k candidate answers of the ith question, and C represents all background knowledge.

[0010] Step 2: For each question in the data set D, use a text retrieval method to retrieve the background knowledge with the highest relevance from C, and the data set after text retrieval can be represented as Where c i represents a piece of background knowledge with the highest relevance to q i .

[0011] Step 3: According to the association relationship among the retrieved question, answer and background knowledge, a textbook question and answer heterogeneous hypergraph is constructed.

[0012] Step 4: Use a deep neural network model to initialize the features of the nodes in the heterogeneous hypergraph constructed in step 3.

[0013] Step 5: Construct a heterogeneous hypergraph neural network to learn the heterogeneous hypergraph after the node feature initialization in step 4.

[0014] Step 5.1: Construct a positive and negative sample pair according to the text retrieval result in step 2.

[0015] Step 5.2: Calculate the similarity of the text of each sample pair constructed in step 5.1, and optimize the heterogeneous hypergraph neural network according to the binary cross entropy.

[0016] Step 6: Connect a multilayer perceptron as a prediction head behind the heterogeneous hypergraph neural network in step 5, use the spliced neural network as a question and answer model, and perform parameter fine-tuning, and further optimize the parameters of the model through cross-entropy loss.

[0017] Step 7: In the test process, for any question node q, the features thereof are fused with all candidate answer node features of the question respectively, and then input into a prediction head, and finally the confidence of each candidate answer is obtained through a softmax function, and the one with the highest confidence is the correct answer.

[0018] Further technical solutions of the present application: the construction mode of the textbook question and answer heterogeneous supergraph in step 3 is specifically:

[0019] If multiple answers come from the candidate answer set of the same question, the answers are connected;

[0020] If an answer exists in the candidate answer set of a question, the answer and the question are connected;

[0021] If a background knowledge is the result of text retrieval of a question, the background knowledge and the question are connected;

[0022] If multiple background knowledges come from the same course, the background knowledges are connected.

[0023] Further technical solutions of the present application: the deep neural network model used in step 4 is RoBERTa-base, YOLO and ResNet101; for answer nodes, background knowledge nodes and non-diagram question nodes, the open source pre-trained RoBERTa-base model is used to obtain initial sequence features; for diagram question nodes, visual features are fused on the basis of text sequence features; the feature initialization formula of the question node in the diagram question is as follows:

[0024]

[0025]

[0026]

[0027] In the formula, represents the text data of the i-th question, represents the diagram data of the i-th question, and DU(·) represents a diagram understanding module, represents Hadamard product operation, represents the modeling of the text sequence feature of the question node, the representation vector obtained after modeling, d represents the vector dimension, represents the modeling of the text sequence feature of the question node, the representation vector obtained after modeling, represents a multi-modal fusion representation vector, which is the fusion result of the question text and the question diagram;

[0028] The diagram understanding module uses a ResNet101 model to learn the visual features of the diagram in the question node. The kth region detected in the image The feature vector of the kth region detected in the image The feature vector of the kth region detected in the image The feature vector of the kth region detected in the image The feature vector of the kth region detected in the image The specific calculation formula is as follows:

[0029]

[0030]

[0031] In the formula, is a learnable parameter matrix, and μ represents the number of regions detected by YOLO in the image.

[0032] The further technical solution of the application is: the construction of positive and negative sample pairs in step 5.1, which is specifically:

[0033] All questions are sorted according to the course order, and for each question, there is a positive sample pair and a negative sample pair; first, the feature fusion of the question node q i and all candidate answer nodes is performed as query i , wherein M represents the number of options; the calculation formula of query i is as follows:

[0034]

[0035] Then the query i and its most relevant background knowledge ci are taken as a positive sample pair (query i , c i ), and the irrelevant background knowledge c j is taken as a negative sample pair (query i , c j ), wherein N represents the number of questions.

[0036] The further technical solution of the application is: in step 5.2, the cosine similarity is used to calculate the similarity of the text of each sample pair, which is specifically:

[0037] s(i,j)=Cosine similarity(query i ,c j ) (7) Adopting binary classification cross-entropy as a loss function to optimize the heterogeneous hypergraph neural network, and the loss calculation formula is as follows:

[0038]

[0039] The further technical solution of the present application is: fine-tuning the model parameters in step 5, specifically:

[0040] For any question node q i , its features are fused with the features of all candidate answer nodes of the question respectively; finally, a multi-layer perceptron is added as a prediction head, and the model parameters are fine-tuned through cross-entropy, and the calculation formula is as follows:

[0041]

[0042]

[0043]

[0044] In the formula, N represents the number of questions, M represents the number of options for each question, Softmax(·) is an activation function, MLP(·) represents a multi-layer perceptron, and || represents a splicing operation.

[0045] A computer system, characterized in comprising: one or more processors, a computer readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0046] A computer readable storage medium, characterized in storing computer executable instructions, the instructions being executed to implement the above method.

[0047] The beneficial effects of the present application are:

[0048] The self-supervised textbook question and answer method based on heterogeneous hypergraph provided by the present application firstly designs a new heterogeneous hypergraph structure to accurately represent the association relationship between questions, answers and knowledge points. Through the heterogeneous hypergraph, the complex relationship in the data can be more comprehensively and accurately captured. Secondly, a text matching self-supervised task is introduced to pre-train the model. The task uses the results of text retrieval as a supervision signal, so that the model can extract more rich semantic information between questions and knowledge points. In this way, we can effectively use unlabeled data to improve the performance of the model. Finally, the network is fine-tuned on the data with correct answer annotations to further optimize the performance and generalization ability of the model.

[0049] Compared with existing technologies, this invention effectively solves the problem of modeling high-order associations among questions, answers, and knowledge points in textbook question-answering by combining hypergraph representation and graph neural networks, and introducing graph self-supervised learning. This invention employs self-supervised text matching pre-training to reduce the model's dependence on a large number of labeled textbook question-answering data samples. The method demonstrates excellent performance in experiments, providing a powerful solution for question-answering tasks in the education field. Attached Figure Description

[0050] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0051] Figure 1 This is a general framework diagram of the method described in the example of the present invention.

[0052] Figure 2 This is a schematic diagram of a heterogeneous hypergraph for textbook question-and-answer formats involved in this invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0054] This invention provides a self-supervised textbook question answering method based on heterogeneous hypergraphs, consisting of four parts: text retrieval, construction of a heterogeneous hypergraph for textbook question answering, self-supervised pre-training of text matching, and model fine-tuning. The overall method framework is as follows: Figure 1 As shown, it includes the following steps:

[0055] 1) Text retrieval: Based on the question and candidate answers for each question, obtain the background knowledge segment with the highest relevance to it;

[0056] 2) Construction of heterogeneous hypergraphs for textbook question answering: Using heterogeneous hypergraphs to represent questions, candidate answers, background knowledge, and the relationships between them;

[0057] 3) Text matching self-supervised pre-training: Using the results of text retrieval as a supervision signal to enhance the model's ability to model the relationship between questions and knowledge points;

[0058] 4) Fine-tuning: Fine-tuning the pre-trained heterogeneous hypergraph neural network using labeled data.

[0059] The specific steps described above are as follows:

[0060] 1. Data acquisition and text retrieval

[0061] 1.1 Obtaining the textbook question-and-answer dataset

[0062] The data used in this invention consists of multiple-choice questions from the CK12-QA dataset. The dataset contains N samples, which can be represented as follows: Where q i Let A represent the i-th question. i =[a1,…,a k Let ] represent the k candidate answers to the i-th question, and C represent all the background knowledge.

[0063] 1.2 Text Retrieval

[0064] This invention preprocesses data using common text retrieval methods, namely information retrieval methods. After information retrieval preprocessing, the dataset can be considered as... Where c i This represents the textual background knowledge segment in C that is most relevant to the i-th question.

[0065] 2. Construction of heterogeneous hypergraphs for textbook question-and-answer sessions

[0066] 2.1 Heterogeneous Hypergraph Structure Design

[0067] This invention establishes three node types: question, answer, and background knowledge, and models four hyperedge relationships: <question-answer>, <answer-answer>, <question-background knowledge>, and <background knowledge-background knowledge>. It constructs a heterogeneous hypergraph for textbook question-and-answer systems to enrich the associations between questions, answers, and background knowledge. The specific construction method is as follows:

[0068] ① If multiple answers come from the same set of candidate answers for the same question, then these answers are concatenated;

[0069] ② If an answer exists in the candidate answer set of a question, then connect that answer to the question;

[0070] ③ If a certain background knowledge is the result of a text search for a certain question, then the background knowledge is linked to the question;

[0071] ④ If multiple pieces of background knowledge come from the same course, then connect these pieces of background knowledge;

[0072] The final constructed heterogeneous hypergraph is as follows Figure 2 As shown.

[0073] 2.2 Node Feature Initialization

[0074] For answer nodes and background knowledge nodes, the initial sequence features are obtained using the open-source pre-trained RoBERTa-base model. For question nodes, the dataset includes two types: questions without graphs and questions with graphs. For question nodes without graphs, the initial sequence features are still obtained using the open-source pre-trained RoBERTa-base model. For question nodes with graphs, visual features are incorporated into the text sequence features. The feature initialization formula for question nodes with graphs is as follows:

[0075]

[0076]

[0077]

[0078] In the formula, This represents the text data for the i-th question. This represents the chart data for the i-th question, and DU(·) represents the chart understanding module. This represents the Hadamah product operation. Indicates to The representation vector obtained after modeling, where d represents the vector dimension. Indicates to The representation vector obtained after modeling This represents the multimodal fusion representation vector, which is the result of fusing the question text and the question chart.

[0079] In this invention, the graph understanding module is divided into two parts: object detection and feature extraction. Specifically, it utilizes the ResNet101 model to learn YOLO's graph understanding capabilities. The kth region detected The feature vectors are then used to transform their coordinates using a fully connected layer. The projection is a d-dimensional position vector, and then the arithmetic mean of the eigenvectors is taken to obtain the chart. Feature representation of the k-th region Finally, the feature representations of all regions are summed to obtain the feature representation of the entire chart. The specific calculation formula is as follows:

[0080]

[0081]

[0082] In the formula, This is a learnable parameter matrix, where μ represents the number of regions detected by YOLO in the graph.

[0083] 3. Self-supervised pre-training for text matching

[0084] 3.1 Construction of positive and negative sample pairs

[0085] Sort all questions according to the course order. For each question, there is one positive sample pair and one negative sample pair. First, sort the question node q... i and all its candidate answer nodes Perform feature fusion as query i , where M represents the number of options. query i The calculation formula is as follows:

[0086]

[0087] Then query i And its most relevant background knowledge c i As positive sample pairs (query) i ,c i ), and unrelated background knowledge c j As negative sample pairs (query) i ,c j ),in N represents the number of questions.

[0088] 3.2 Self-supervised learning for text matching

[0089] The confidence score for each sample pair is calculated using cosine similarity, as shown in the following formula:

[0090] s(i,j)=Cosine similarity(query i ,c j (18)

[0091] Then, a heterogeneous hypergraph neural network model is constructed, and binary cross entropy (BCE) is used as the loss function to optimize the model. The loss calculation formula is as follows:

[0092]

[0093] 4. Model fine-tuning

[0094] This invention treats textbook question-and-answer as a classification problem. For any question node q... i The features are then fused with the features of all candidate answer nodes for the question. Finally, a multilayer perceptron is added as the prediction head, and the model parameters are fine-tuned using cross-entropy, calculated as follows:

[0095]

[0096]

[0097]

[0098] In the formula, N represents the number of questions, M represents the number of options for each question, and in the data used in this invention, there are 7 options for questions without charts and 4 options for questions with charts. Softmax(·) is an activation function, MLP(·) represents a multilayer perceptron, and this invention uses two fully connected layers. || represents the concatenation operation.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A self-supervised textbook question answering method based on heterogeneous hypergraphs, characterized in that Comprising the following steps: Step 1: Obtain the textbook question-and-answer dataset ,in Indicates the number of questions. Indicates the first One question, Indicates the first The question 10 candidate answers This represents all the background information; Step 2: For the dataset For each question in the text, use text retrieval methods from The most relevant piece of background knowledge was retrieved from the text, and the dataset after text retrieval can be represented as follows: ,in Represents all background knowledge Middle and the first The most relevant piece of background information for each question; Step 3: According to the association relationship between the searched question, answer and background knowledge, a textbook question and answer heterogeneous hypergraph is constructed; the construction method of the textbook question and answer heterogeneous hypergraph in step 3 is specifically as follows: If multiple answers come from the candidate answer set of the same question, the answers are connected; If an answer exists in the candidate answer set of a question, the answer and the question are connected; If a background knowledge is the result of text retrieval of a question, the background knowledge and the question are connected; If multiple background knowledges come from the same course, the background knowledges are connected; Step 4: A deep neural network model is used to initialize the features of the nodes in the heterogeneous hypergraph constructed in step 3; the deep neural network model used in step 4 is RoBERTa-base, YOLO and ResNet101; for answer nodes, background knowledge nodes and graphless question nodes, the initial sequence features are obtained by using the open source pre-trained RoBERTa-base model; for graph question nodes, visual features are integrated on the basis of text sequence features; the feature initialization formula of the question node in the graph question is as follows: (1) (2) (3) In the formula, text data representing the first question, graph data representing the first question, a graph understanding module, a Hadamard product operation, a representation vector obtained after modeling, a representation vector obtained after modeling, a vector dimension, a representation vector obtained after modeling, a representation vector obtained after modeling, a multi-modal fusion representation vector, which is a fusion result of the question text and the question graph. The chart understanding module utilizes a ResNet101 model to learn feature vectors for regions detected by YOLO in a chart The chart understanding module utilizes a ResNet101 model to learn feature vectors for regions detected by YOLO in a chart The chart understanding module utilizes a ResNet101 model to learn feature vectors for regions detected by YOLO in a chart The chart understanding module utilizes a ResNet101 model to learn feature vectors for regions detected by YOLO in a chart The chart understanding module utilizes a ResNet101 model to learn feature vectors for regions detected by YOLO in a chart (4) (5) In the formula, is a learnable parameter matrix, represents the number of regions detected by YOLO in the graph; Step 5: A heterogeneous hypergraph neural network is constructed to learn the heterogeneous hypergraph after the node feature initialization in step 4; Step 5.1: According to the results of text retrieval in step 2, a positive and negative sample pair is constructed; the construction of the positive and negative sample pair in step 5.1 is specifically as follows: All the questions are sorted by course order, for each question, there is a positive sample pair and a negative sample pair; first, the question node and all its candidate answer nodes are fused as , where represents the number of answer options for each question; The calculation formula is as follows: (6) Then we will and its most relevant background knowledge as positive sample pairs and irrelevant background knowledge as negative sample pairs where , denotes the number of questions; Step 5.2: The similarity of the text of each sample pair constructed in step 5.1 is calculated, and the heterogeneous hypergraph neural network is optimized according to the binary cross entropy; in step 5.2, the cosine similarity is used to calculate the similarity of the text of each sample pair, specifically as follows: (7) The binary classification cross entropy is used as the loss function to optimize the heterogeneous hypergraph neural network, and the loss calculation formula is as follows: (8) Step 6: A multilayer perceptron is connected behind the heterogeneous hypergraph neural network in step 5 as a prediction head, the spliced neural network is used as a question and answer model, and the model parameters are fine-tuned through cross entropy loss to further optimize the parameters of the model; wherein, the model parameter fine-tuning is specifically as follows: For any one question node The characteristics are fused with all candidate answer node characteristics of the question respectively; finally, a multi-layer perceptron is added as a prediction head, and the model parameters are fine-tuned through cross entropy, and the calculation formula is as follows: (9) (10) (11) wherein, denotes the number of questions, denotes the number of answer options for each question, is an activation function, denotes a multi-layer perceptron, denotes a concatenation operation; Step 7: During the test process, for any one question node , its features are fused with the features of all candidate answer nodes of this question respectively, and then input into the prediction head, and finally the confidence of each candidate answer is obtained through the softmax function, and the one with the highest confidence is the correct answer.

2. A computer system, characterized by Comprise: One or more processors, a computer readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of claim 1.

3. A computer-readable storage medium, characterized in that A computer executable instruction is stored, which is used to implement the method of claim 1 when executed.

Images