Multi-task knowledge point labeling method fusing comparative learning

By integrating the methods of comparative learning and multi-task learning, combined with two-way long and short-term memory neural networks, the semantic information and feature matching degree of the problem and answers was solved, and the problem of low accuracy of knowledge point annotation in the existing technology was achieved. Efficient knowledge point labeling.

CN118964610APending Publication Date: 2024-11-15NINGXIA TEACHERS UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410944271.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When the existing knowledge point annotation model uses the shallow semantic information of the test questions, it is impossible to accurately capture the connection information between the test questions and the knowledge points, resulting in low accuracy of the annotation results.

Method used

The method of fusion contrast learning, momentum update, multi-task learning and two-way long and short-term memory neural network is adopted to improve the accuracy of knowledge point annotation by constructing question-solving text embedding, extracting semantic information, capturing feature matching degree and integrating prediction.

Benefits of technology

The accuracy of knowledge point annotation and model performance are improved, and the ability to capture the semantic representation of the test questions is enhanced by considering the one-to-one correspondence between questions and answers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964610A_ABST
    Figure CN118964610A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of knowledge point labeling in an intelligent education system, and provides a multi-task mathematical knowledge point labeling method fusing comparative learning, which comprises the following steps of: (1) constructing question-answer text embedding; (2) extracting semantic information of questions-answers; (3) capturing a question-answer feature matching degree; and (4) marking questions and solving knowledge points. According to the method, deep learning technologies such as comparative learning, momentum updating, multi-task learning and a bidirectional long-short-term memory neural network are fused, the corresponding matching relation between the question and the answer is considered, and the matching degree between the question and the answer is extracted by obtaining text semantic feature representation and utilizing a comparative learning framework network, so that the matching degree of the question and the answer is improved. And finally, integrating prediction of questions and answers to carry out knowledge point labeling, and improving the performance of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of knowledge point labeling in intelligent education systems, and specifically relates to a multi-task mathematics knowledge point labeling method integrating contrastive learning. Technical Background

[0002] With the popularization of intelligent education, personalized adaptive learning has become the focus of the education field, and it is necessary to push appropriate educational learning resources to learners and establish a good index structure for test resources by marking knowledge points.

[0003] Constructing a model for test question knowledge point annotation is an important means to achieve the classification of educational resources. At present, there have been some studies on test question knowledge point annotation methods. From the perspective of the method itself and the specific implementation effect of the method, the existing models can be divided into two types according to technical means: specific models based on traditional machine learning methods and general text classification models in the field of natural language processing.

[0004] The test question annotation method based on machine learning inputs the extracted test question features into the machine learning algorithm, so that the model can learn the shallow test question text features and use them as the annotation basis. However, this type of test question annotation method only uses the shallow semantic information of the test question, and the expression perception of the knowledge semantics of the test question is not accurate enough, especially the acquisition of the connection information between the test question and the knowledge point is not complete, which leads to the low accuracy of the test question annotation results.

[0005] The researchers integrated deep learning and natural language processing to develop a neural network-based knowledge point annotation model, which greatly improved the drawbacks of models based on machine learning methods. The research paradigm of this type of model is to first extract word vectors from the test text, then fuse the extracted text vectors with the feature vectors related to the labels, and input them into a convolutional or recurrent neural network for parameter training, and then input the obtained test text representation vector into a classifier for test question classification, and finally output the category as the knowledge point corresponding to the test question.

[0006] However, although existing studies have great advantages in the semantic representation of test question context, they have completed the knowledge point labeling task, which depends on the semantic information captured by the model. In fact, the relationship between questions and answers in the test questions can also help label knowledge points. Questions and answers are in a one-to-one correspondence, and they always belong to the same knowledge point. Summary of the invention

[0007] The purpose of the present invention is to address the shortcomings of the existing knowledge point labeling model and propose a multi-task knowledge point labeling method that integrates contrastive learning, momentum updating, multi-task learning, bidirectional long short-term memory neural network (BiLSTM) and other cutting-edge deep learning technologies. Taking into account the corresponding matching relationship between questions and answers, the text semantic feature representation is obtained, and the matching degree between questions and answers is extracted using a contrastive learning framework network. Finally, the predictions of questions and answers are integrated to perform knowledge point labeling, thereby improving the performance of the model.

[0008] To achieve the purpose of the invention, the present invention adopts the following technical solutions.

[0009] The present invention provides a multi-task knowledge point labeling method integrating contrastive learning, comprising the following steps:

[0010] (1) Construct question-answer text embedding: data cleaning, and text segmentation and embedding.

[0011] (2) Extracting the semantic information of the question-answer: construct two text encoders with the same structure but different parameters; use a bidirectional long short-term memory neural network to capture the text semantic information; and construct three pooling layers to extract semantic information and obtain text features.

[0012] (3) Capturing the matching degree of question-answer features: define positive and negative samples, set up a queue to record all answer samples, and calculate the comparison loss between the question-answer text features in the same question and the answer text features in other questions.

[0013] (4) Labeling question-answer knowledge points: Construct two classifiers to predict the probability vectors of the question and answer respectively based on the feature vectors in step (2), integrate the contrastive learning task and the knowledge point labeling task training model, and integrate the two predictions to obtain the final labeled knowledge points.

[0014] In the above technical solution, the step (1) of constructing the question-answer text embedding includes:

[0015] (1-1) Data preprocessing and word segmentation: Clean the data and use the tokenizer dictionary for word segmentation.

[0016] (1-2) Word embedding: embed the word segmentation results of the question and answer respectively, and convert the knowledge points into 0-1 vectors.

[0017] In the above technical solution, the step (2) of extracting the semantic information of the question-answer includes:

[0018] (2-1) Construct a bidirectional long short-term memory neural network encoder: construct two question encoders and answer encoders with the same structure but different parameters, and input the embedding of the question-answer text obtained in step (1) into the encoders to capture the question-answer semantic information.

[0019] (2-2) Feature enhancement: Construct three pooling operations to enrich the question-answer feature representation.

[0020] (2-3) Update the parameters of the answer encoder: Use the momentum update algorithm to update the parameters of the question encoder during iterative training.

[0021] In the above technical solution, the capture question-answer feature matching degree described in step (3) includes:

[0022] (3-1) Storing positive and negative samples: Define positive and negative samples of questions and answers. In the same test question, the question and the answer constitute the positive sample, and the question and the answer to other test questions constitute the negative sample. Set up a first-in-first-out queue as a dictionary to store and query the text features of the answer.

[0023] (3-2) Calculate the problem-answer matching degree: Use the queue to calculate the problem-answer matching degree, and predict the positive sample label of the problem in all the data in the queue for the calculation of the comparison loss.

[0024] (3-3) Construct contrastive loss: classify the answers to the questions and use the loss as one of the losses for training the encoder so that the encoder matches the positive samples during the encoding process.

[0025] In the above technical solution, the labeled question-answering knowledge points described in step (4) include:

[0026] (4-1) Constructing a knowledge point prediction network: Construct two linear networks, input the question-answer text features obtained in step (2), and obtain the probability vectors predicted by the two linear networks respectively.

[0027] (4-2) Setting loss function and multi-task learning: Set the knowledge point labeling loss function, adopt multi-task learning method, inject the contrast loss of step (3) into the loss to train the entire network.

[0028] (4-3) Labeling knowledge points: During the prediction process, the probability vectors obtained by the two linear networks are integrated to correct each other’s predictions and label the knowledge points of the test questions.

[0029] Compared with the prior art, the present invention has the following outstanding substantive features and significant progress:

[0030] 1. The present invention designs a brand-new multi-task knowledge point labeling method based on contrastive learning, constructs a question encoder and an answer encoder, which have the same structure but different parameters, and are used to extract question text features and answer text features respectively. In addition, a dynamic update algorithm is used to update the answer encoder parameters. Finally, a prediction layer is set to label the knowledge points of the question and answer respectively and correct each other's predictions, thereby improving the knowledge point labeling accuracy of the model.

[0031] 2. The present invention takes into account the one-to-one correspondence between questions and answers in the knowledge point labeling task, introduces contrastive learning, calculates the matching degree of questions and answers, explores the association between questions and answers in the same test question, and the difference between questions and answers in different test questions, so that the features of the question encoder and the answer encoder can be similar when extracting information between positive samples, and the features between negative samples can be different.

[0032] 3. The present invention incorporates the contrastive learning task and the knowledge point labeling task into the definition of loss and trains the encoder at the same time, so that the model can also identify the relationship between questions and answers in the knowledge point labeling task. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of the method of the present invention.

[0034] Figure 2 Schematic diagram of model loss calculation in the method of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] The embodiment of the present invention provides a multi-task knowledge point labeling method integrating contrastive learning, such as Figure 1 As shown, the following steps are included:

[0037] (1) Constructing Question-Answer Text Embedding

[0038] (1-1) Data cleaning and word segmentation

[0039] First, we preprocess the data, count the length of the questions and answers in the sample, and set a reasonable length to ensure that most of the text is covered to prevent a large number of padding labels from appearing during word embedding. In addition, we remove the question-answer pairs where the answer or question text is empty, and obtain Q = {q1, q2, ..., q M},R={r1,r2,...,r M} respectively represent the set of question texts and answer texts of M test questions, q i , r iThey represent the question-answer text in the same test question, Y = {y1, y2, ..., y M} represents the set of knowledge points of M test questions. Use Bert-tokenizer to segment test questions Q and R and extract their word indexes.

[0040] (1-2) Word vector embedding:

[0041] Use embedding for word embedding instead of the word vector embedding method in Bert to prevent the gradient problem and overfitting problem caused by Bert word embedding. The formula is:

[0042] q=embedding(q i )

[0043] r = embedding(r i )

[0044] In addition, the MultiLabelBinarizer function is used to encode the knowledge points so that the knowledge points are converted into 0-1 vectors.

[0045] (2) Extracting semantic information of question-answer

[0046] (2-1) Construct two text encoders with the same structure but different parameters, denoted as Encoder q ,Encoder r

[0047] BiLSTM is used as a text encoder to obtain the semantic information of the text. BiLSTM combines information flows in two directions and considers the context information of the current word. It has a good effect on feature extraction in knowledge point labeling tasks.

[0048] In order to obtain the hidden state of the question text, the question text q i The specific calculation formula for t words is:

[0049]

[0050] Where W xh , W hh is the model weight, b h is the bias, tanh is the activation function, are the forward and backward hidden states respectively.

[0051] Then and Connect to get the hidden state H with previous and next information t .

[0052]

[0053] Finally, a linear operation is set up to obtain the final feature with semantic information, which is recorded as

[0054]

[0055] Where W q is the model weight, b q is the model parameter.

[0056] (2-2) Feature enhancement:

[0057] Construct three pooling operations to enhance the semantic information features and enrich the feature representation, and concatenate the outputs of the three pooling operations to obtain features with semantic information; the specific calculation form is as follows:

[0058]

[0059] Among them, max, avg, and att represent the maximum pooling operation, the average pooling operation, and the attention pooling operation respectively. The maximum pooling operation is to obtain the maximum value in the feature, the average pooling operation is to obtain the average value, and the calculation formula of the attention pooling operation is as follows:

[0060]

[0061] α=softmax(HV T )

[0062]

[0063] Its weight matrix is ​​W, V, the bias vector is b, the attention score is α, and the output of the final pooling operation is q att .

[0064] The calculation of the answer text r in steps (2-1) and (2-2) is the same as that of q.

[0065] (2-3) Update the solution encoder parameters

[0066] Considering that training two encoders at the same time will lead to difficulty in convergence and easily cause local minima during training, momentum update is used to solve the above problems. q As the gradient update iteration, for Encoder r We will Encoder q Parameters passed to Encoder r , the specific calculation formula is as follows:

[0067] θ r →mθ k +(1-m)θ q

[0068] where θ q ,θ r Encoder q ,Encoder r model parameter, m∈[0,1).

[0069] (3) Capturing the matching degree of question-answer features, such as Figure 2 As shown, the following steps are used to define the calculation of positive and negative samples and contrast loss

[0070] (3-1) Storing positive and negative samples

[0071] Bundle Recorded as an anchor sample, the same test question with the characteristics of the answer text Recorded as anchor point positive sample, The negative sample representing the anchor point is also recorded as q i . Let the anchor point be q and the positive sample be r + , each q can only be connected with r + In traditional contrastive learning, a query dictionary is usually set for matching positive and negative samples. Each time during the matching process, the anchor point will query the dictionary for positive samples. The process of the anchor point querying the dictionary for positive samples will consume a lot of space and time. In order to solve this problem, a first-in-first-out queue is set to simulate the dictionary for positive and negative sample queries.

[0072] During the training process, positive sample pairs always appear in pairs. q With Encoder r In the output, the negative sample pairs are stored in the Queue. After each calculation of the matching degree of the positive and negative samples, 0 An enqueue operation and a dequeue at the end of the queue.

[0073] (3-2) Calculating the problem-solution matching degree

[0074] The essence of the contrastive learning task is the process of an anchor point querying positive samples in the sample space, and using the dot multiplication method to calculate the matching degree between the question and the answer in the sample space. The specific calculation formula is

[0075]

[0076] Among them, τ is a designed trainable hyperparameter, K is the length of the queue, and the obtained σ is essentially the prediction of q in all positive and negative samples (the anchor point is classified as a positive sample). What is done is to predict that q belongs to a certain sample in all the data in the queue, r + is a positive sample, r i is a negative sample.

[0077] (3-3) Construct contrast loss:

[0078] Based on the output of (3-2), in order to allow Encoder q During the training process, the cross entropy function is used to allow the encoder to incorporate the similarity characteristics of the questions and answers in the positive samples, so that the model can recognize that the predictions between positive samples belong to the same knowledge point.

[0079] Before calculating the loss each time, a dequeue and enqueue operation is performed to enqueue the positive sample r + In the hope that during the training process, the encoder can recognize that the anchor point and the positive sample belong to the same category, and the negative sample and the anchor point belong to different categories. Then the cross entropy function is used to calculate the loss of the contrastive learning task. At the same time, a real positive sample is introduced. The sample value corresponds to the index value 0 of the positive sample in the queue. The sample feature is recorded as By taking advantage of the characteristics of the queue, in each iteration, the positive sample will belong to the first position in the index of the queue due to the queue operation, so the following calculation formula is used:

[0080]

[0081] Here K is the length of the queue, loss c The loss of the network structure for comparative learning.

[0082] (4) Marking questions and answering knowledge points

[0083] (4-1) Constructing a knowledge point prediction network

[0084] Construct two linear layers for knowledge point prediction and transform the output obtained by the encoder The (question-answer) feature is input into the prediction layer, and two predicted knowledge points are obtained, which are respectively recorded as question prediction and answer prediction. q ,pred r , the calculation formula is:

[0085]

[0086] Among them, W and b are the linear layer network parameters. The two linear layer parameters are independent of each other and both participate in the gradient update.

[0087] (4-2) Setting the multi-task learning loss function

[0088] Considering that when labeling knowledge points, the number of predicted knowledge points is usually much smaller than the total number of knowledge points, the softmax function and cross entropy loss function are used to generalize the multi-classification problem to the multi-label classification problem. Specifically, 0 is set as the threshold. The score of each target class needs to be greater than 0, and the score of each non-target class needs to be less than 0. That is, 0 minus the maximum value of each target class score and the maximum value of each non-target class score minus 0 need to be greater than 0. Therefore, the classification loss calculation formula of the model based on multi-task learning is:

[0089]

[0090] Where Ω neg With Ω pos They represent the target class and non-target class sample sets respectively, and z is the score of each knowledge point.

[0091] The loss functions of the obtained questions and answers are recorded as loss q , loss r At the same time, in order to enable the model to learn the loss in the contrastive learning task, the contrastive loss obtained by (3-3) is recorded as loss c As one of the total losses, the model is trained and the total loss is calculated as:

[0092] Loss = loss q +loss r +loss c

[0093] (4-3) Marking knowledge points

[0094] In traditional knowledge point annotation, question answering is usually used as a feature to predict knowledge points. In this method, the upper network structure outputs two predictions of question-answering: q ,pred r , in order to make the two sides correct each other’s annotation results, pred q ,pred r The weights are added together to obtain a probability vector that combines the predictions of the two neural networks.

[0095] pred=pred q +pred r

[0096] In addition, according to the characteristics of the loss function set in (4-2), there is no need to construct a threshold. For the probability vector, only the value greater than 0 in the prediction vector is needed as the labeling result to obtain the final labeling result. The calculation formula is:

[0097]

[0098] If the probability vector is greater than 0, it is set to 1, otherwise it is set to 0, where 1 means that the question and the answer are labeled together.

[0099] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0100] It is easy for those skilled in the art to understand that the above is to explain the technical features of the present invention. Under the premise of not departing from the present invention, several improvements and modifications made are also protected by the present invention, so the protection scope of the present invention should be based on the content defined by the claims of this application.

Claims

1. A multi-task knowledge point labeling method integrating contrastive learning, characterized in that The method comprises the following steps: (1) Construct question-answer text embedding: data cleaning, and text segmentation and embedding; (2) Extracting semantic information of the question-answer: construct two text encoders with the same structure but different parameters; use a bidirectional long short-term memory neural network to capture text semantic information; construct three pooling layers to extract semantic information and obtain text features; (3) Capturing the matching degree of question-answer features: define positive and negative samples, set up a queue to record all answer samples, and calculate the comparison loss between the question-answer text features in the same question and the answer text features in other questions; (4) Labeling question-answer knowledge points: Construct two classifiers to predict the probability vectors of the question and answer respectively based on the feature vectors in step (2), integrate the contrastive learning task and the knowledge point labeling task training model, and integrate the two predictions to obtain the final labeled knowledge points.

2. The multi-task knowledge point labeling method of fusion contrastive learning according to claim 1 is characterized in that The construction of the question-answer text embedding described in step (1) specifically includes: (1-1) Data preprocessing and word segmentation: clean the data and use the tokenizer dictionary for word segmentation; (1-2) Word embedding: The word segmentation results of the question and answer are embedded separately, and the knowledge points are converted into 0-1 vectors.

3. The multi-task knowledge point labeling method of fusion contrastive learning according to claim 1 is characterized in that The semantic information of extracting the question-answer described in step (2) specifically includes: (2-1) Constructing a bidirectional long short-term memory neural network encoder: construct two question encoders and answer encoders with the same structure but different parameters, and input the embedding of the question-answer text obtained in step (1) into the encoders to capture the question-answer semantic information; (2-2) Feature enhancement: construct three pooling operations to enrich the question-answer feature representation; (2-3) Update the parameters of the answer encoder: Use the momentum update algorithm to update the parameters of the question encoder during iterative training.

4. The multi-task knowledge point labeling method of fusion contrastive learning according to claim 1 is characterized in that The capture question-answer feature matching described in step (3) specifically includes: (3-1) Storing positive and negative samples: Define positive and negative samples of question-answer. In the same test question, the question and the answer constitute the positive sample, and the question and the answer of other test questions constitute the negative sample. Set up a first-in-first-out queue as a dictionary to store and query the text features of the answer; (3-2) Calculate the matching degree of question-answer: Use the queue to calculate the matching degree of question-answer, and predict the positive sample label of the question in all the data in the queue for the calculation of comparison loss; (3-3) Construct contrastive loss: classify the answers to the questions and use the loss as one of the losses for training the encoder so that the encoder matches the positive samples during the encoding process.

5. The multi-task knowledge point labeling method of fusion contrastive learning according to claim 1 is characterized in that The labeling question-answering knowledge points described in step (4) specifically include: (4-1) Constructing a knowledge point prediction network: Construct two linear networks, input the question-answer text features obtained in step (2), and obtain the probability vectors predicted by the two linear networks respectively; (4-2) Setting loss function and multi-task learning: Setting the knowledge point labeling loss function, adopting the multi-task learning method, injecting the contrast loss of step (3) into the loss to train the entire network; (4-3) Labeling knowledge points: During the prediction process, the probability vectors obtained by the two linear networks are integrated to correct each other’s predictions and label the knowledge points of the test questions.