Answer evaluation method, language model training method and related device

Abstract

This application provides an answer evaluation method, a language model training method, and related devices. The answer evaluation method includes: constructing a test sentence based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein the first mask represents the semantic relevance between the standard answer and the student answer; and inputting the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask. In this way, the application can unify the training objectives of upstream and downstream tasks to improve the accuracy of the evaluation prediction results.

Description

Technical Field

[0001] This application belongs to the field of natural language processing, and specifically relates to an answer evaluation method, a language model training method and related devices. Background Art

[0002] Natural language processing (NLP) is a key research area in artificial intelligence. Its core goal is to teach machines to understand the natural language used by humans in everyday situations. Common NLP tasks include machine translation, automatic text summarization, text classification, text question answering, reading comprehension, text semantic analysis, speech recognition, language modeling, and optical character recognition (OCR). With the continuous maturity of NLP technology in recent years, especially the rise of large-scale unsupervised language models, deep learning models have demonstrated performance comparable to or even superior to humans in many tasks.

[0003] Education is one of the areas where natural language processing technology is being applied. Models can understand questions, compare standard answers with student responses, and predict student scores. This can significantly reduce the pressure on teachers and alleviate the uneven distribution of educational resources.

[0004] However, existing methods primarily adhere to the basic framework of natural language processing tasks, resulting in inconsistent training objectives for the pre-training phase and the downstream fine-tuning task. The pre-training phase aims to predict masked words, while the downstream task aims to score student responses. This is a common drawback of large-scale pre-trained language models, stemming from the direct application of existing frameworks without considering the specificities of educational scenarios. Summary of the Invention

[0005] The present application provides an answer evaluation method, a language model training method and related devices to unify the training objectives of upstream and downstream tasks and improve the accuracy of evaluation prediction results.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: to provide an answer evaluation method, including: constructing a test sentence based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein, the first mask represents the semantic correlation between the standard answer and the student answer; the test sentence is input into the trained language model to obtain an evaluation prediction result corresponding to the first mask.

[0007] To solve the above technical problems, another technical solution adopted in this application is: providing a language model training method, including: constructing multiple first training sentences based on a first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions; wherein a second mask is set in the training questions and / or the training answers and / or the training answers, and the second mask has a corresponding true value label; inputting the first training sentence into the language model to obtain a first prediction result corresponding to the first mask, and a second prediction result corresponding to the second mask; constructing multiple second training sentences based on multiple first training sentences; wherein at least some of the second training sentences include the first prediction result, the training questions, the training answers corresponding to the training questions, and the training answers corresponding to the training questions; and the training questions and / or the training answers and / or the training answers are set with the second mask; inputting the second training sentence into the language model to obtain a third prediction result corresponding to the second mask; obtaining a total loss value based at least on the second prediction result and the third prediction result; and adjusting the training parameters of the language model based on the total loss value.

[0008] In order to solve the above technical problems, another technical solution adopted in this application is: to provide an answer evaluation device, including: a construction module, used to construct a test sentence based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein, the first mask represents the semantic correlation between the standard answer and the student answer; a prediction module, connected to the construction module, used to input the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask.

[0009] In order to solve the above technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the answer evaluation method or the language model training method described in any of the above embodiments.

[0010] In order to solve the above technical problems, another technical solution adopted in this application is: providing a storage device storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the answer evaluation method or the language model training method described in any of the above embodiments.

[0011] Different from the existing technology, the beneficial effect of the present application is that the answer evaluation method provided by the present application includes constructing a test sentence based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein the first mask represents the semantic correlation between the standard answer and the student answer; the test sentence is input into the trained language model to obtain the evaluation prediction result corresponding to the first mask. The test sentence constructed in this solution is input into the trained language model to perform a cloze task to predict the evaluation prediction result corresponding to the first mask, that is, whether the student's answer is correct or incorrect relative to the standard answer. The existing method adopts an obvious serial mechanism, where the language model is first trained on the pre-training task and then trains the downstream task, which severs the connection between the upstream and downstream tasks; while in the present application, through the construction of the above-mentioned type of test sentence, the training objectives of the upstream and downstream tasks can be cleverly unified, and the downstream right and wrong classification task is also converted into a cloze task (i.e., predicting the word corresponding to the first mask), solving the problem of the "pre-training-fine-tuning paradigm" of the existing method and improving the accuracy of the final evaluation prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0013] Figure 1 This is a structural diagram of an implementation method of the first stage pre-training in the prior art;

[0014] Figure 2 This is a structural diagram of an implementation method of the second stage fine-tuning in the prior art;

[0015] Figure 3 A flowchart of an implementation method for answering and evaluating this application;

[0016] Figure 4 This is a flowchart of an implementation method of a language model training method of the present application;

[0017] Figure 5 This is a schematic diagram of the structure of an embodiment of the evaluation device for answering this application;

[0018] Figure 6 This is a schematic structural diagram of an embodiment of an electronic device of the present application;

[0019] Figure 7 This is a structural diagram of an embodiment of the storage device of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] Large-scale language models excel at semantic understanding, but they struggle to capture real-world student responses and corresponding scores. Existing methods typically train language models on large-scale unlabeled data, then fine-tune them using a small set of educational data to enable the trained language models to score student responses.

[0022] At present, the language model adopts the Transformer (i.e. encoder) architecture, which has good performance in many natural language processing tasks. The Transformer series of language models are based on the self-attention mechanism. Each input word will interact deeply with the rest of the words, and the global information will be integrated into the feature vector corresponding to the [CLS] position (i.e. the beginning placeholder). In view of the characteristics of this deep interaction, individual words in long texts are masked [MASK]. The language model can predict the masked words on the premise of understanding the semantics of the text, just like doing a cloze task. Generally speaking, when performing downstream tasks, existing methods will use the global feature vector [CLS] as input and connect to a suitable decoder to complete it. For example, for educational correction tasks, a 0-1 classifier can be connected. The existing method can be divided into the following two stages:

[0023] The first stage of pre-training: Figure 1 As shown, Figure 1 This is a structural diagram of an implementation method of the first stage of pre-training in the prior art. Large-scale Internet text data (preferably texts related to educational scenarios) is used to construct a cloze task, that is, individual words are masked, and the language model is allowed to predict the masked words based on the context. Since the masked words are known and do not rely on manual annotation, the language model training at this stage is also called "self-supervised pre-training." Quantitative change leads to qualitative change. After pre-training with a sufficiently large amount of text data, the language model has mastered the ability to understand contextual semantics.

[0024] Second stage fine-tuning: Figure 2 As shown, Figure 2This is a schematic diagram of the structure of a second-stage fine-tuning implementation in the prior art. By inputting the "question-standard answer-student answer" data into a pre-trained language model, we can obtain a fused feature. This fused feature is then input into a scoring module to obtain the corresponding score.

[0025] As can be seen from the above content, existing methods mainly follow the basic framework of natural language processing tasks (pre-training and fine-tuning paradigm), but do not consider the particularity of grading tasks in educational scenarios. This leads to the following problems:

[0026] Question 1: Cloze-type pre-training tasks can indeed help language models understand global semantic information, but this is far from enough. In ordinary Internet texts, for example, when masking "Suzhou", the language model can predict the masked words through the repeated mention of "Jinji Lake" in the context. However, in educational grading tasks, it is unreasonable to apply the existing pre-training framework and splice in the form of "question-standard answer-student answer" (such as "3 sulfur atoms-3S-35"). No matter which word is masked, it is difficult to complete it through contextual reasoning, because it is impossible to predict divergent student answers when the student scores are unknown.

[0027] Problem 2: The training objectives of the pre-training phase and the downstream fine-tuning task are inconsistent: the pre-training phase aims to predict masked words, while the downstream task aims to score student responses. This is a common drawback of large-scale pre-trained language models, stemming from the direct application of existing frameworks without considering the specificities of educational scenarios.

[0028] To address at least some of the above issues, see Figure 3 , Figure 3 This is a flowchart of an implementation method of the answer evaluation method for this application, which includes:

[0029] S101: Constructing a test sentence based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein the first mask represents the semantic relevance between the standard answer and the student answer.

[0030] Specifically, the implementation process of the above step S101 can be: splicing "test question - standard answer corresponding to the test question - student answer corresponding to the test question", and the splicing method is consistent with the existing method, that is, using [CLS] and [SEP] as the beginning placeholder and separation placeholder respectively. Figure 1The difference is that we manually constructed a template, namely "done [MASK1] → right / wrong", and the [MASK1] in the above "done [MASK1]" can be called the first mask, and the evaluation prediction result corresponding to the first mask is the first evaluation prediction result or the second evaluation prediction result. For example, the first evaluation prediction result is "right", and the second evaluation prediction result is "wrong"; for another example, the first evaluation prediction result is "0", and the second evaluation prediction result is "1", and the subsequent processor can determine whether the student's answer is correct based on the "0" and "1". Of course, in other embodiments, the template containing the first mask can also be other, for example, directly "[MASK1]", and this application is not limited to this.

[0031] Alternatively, the test statement in this application can be: [CLS] did [MASK1] [SEP] [test question] [SEP] [standard answer corresponding to the test question] [SEP] [student answer corresponding to the test question] [SEP]. For simplicity, the separators are omitted below for easier reading. It should be noted that this application does not specify the order of the first mask, test question, standard answer corresponding to the test question, and student answer corresponding to the test question in the test statement.

[0032] S102: Input the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask.

[0033] Specifically, the language model in this application can be any one of the existing technologies, and the language model also adopts the Transformer architecture. For example, the language model can be a BERT language model, etc.

[0034] In the above design, inputting the constructed test sentence into the trained language model is equivalent to doing a cloze task to predict the evaluation prediction result corresponding to the first mask, that is, whether the student's answer is correct or wrong relative to the standard answer. The existing method adopts an obvious serial mechanism, in which the language model is first trained on the pre-training task and then trained on the downstream task, which severs the connection between the upstream and downstream tasks; while in this application, through the construction of the above-mentioned type of test sentence, the training objectives of the upstream and downstream tasks can be cleverly unified, and the downstream right and wrong classification task can also be converted into a cloze task (that is, predicting the word corresponding to the first mask), solving the problem of the "pre-training-fine-tuning paradigm" of the existing method and improving the accuracy of the final evaluation prediction result.

[0035] In one embodiment, after obtaining the evaluation prediction result in step S102, the answer evaluation method is completed. In this case, the answer evaluation method has the advantages of fewer steps and faster speed, and is suitable for online real-time correction scenarios; for example, taking pictures to solve problems, real-time scoring, etc.

[0036] In another embodiment, the evaluation prediction result is a first evaluation prediction result or a second evaluation prediction result, wherein the first evaluation prediction result indicates that the semantic relevance between the standard answer and the student's answer is consistent (i.e., the semantic relevance between the standard answer and the student's answer is the same), and the second evaluation prediction result indicates that the semantic relevance between the standard answer and the student's answer is inconsistent (i.e., the semantic relevance between the standard answer and the student's answer is different); for example, the first evaluation prediction result is "correct" and the second evaluation prediction result is "incorrect". After the above step S102, the following may also be included:

[0037] A. Obtain a third global feature vector h_3 based on the evaluation prediction result and the test sentence, obtain a fourth global feature vector h_4 based on the first evaluation prediction result and the test sentence, and obtain a fifth global feature vector h_5 based on the second evaluation prediction result and the test sentence. Specifically, the third global feature vector h_3 obtained after step S102 can be recorded, the first mask in the test sentence can be replaced with the first evaluation prediction result and input into the language model to obtain a fourth global feature vector h_4, and the first mask in the test sentence can be replaced with the second evaluation prediction result and input into the language model to obtain a fifth global feature vector h_5.

[0038] B. Obtain a first similarity between the third global eigenvector h_3 and the fourth global eigenvector h_4, and obtain a second similarity between the third global eigenvector h_3 and the fifth global eigenvector h_5. Specifically, the first similarity and the second similarity may be cosine similarities, where a larger cosine similarity indicates that the two eigenvectors are more similar.

[0039] C. In response to the first similarity being greater than the second similarity, determining that the student's answer is correct.

[0040] In the above design method, the semantic similarity between different templates and actual answers is compared over multiple rounds, which is suitable for offline scenarios that require high problem-solving accuracy and are insensitive to delays.

[0041] The following is a detailed description of the language model training process mentioned in step S102. Figure 4 , Figure 4 This is a flow chart of an embodiment of a method for training a language model of the present application, which specifically includes:

[0042] S201: Construct multiple first training sentences based on a first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions; wherein a second mask is set in the training questions and / or the training answers and / or the training answers, and the second mask has a corresponding true value label.

[0043] Specifically, the specific implementation process of the above step S201 can be:

[0044] A. Construct multiple first training texts based on the first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions.

[0045] For example, the format of the first training text can be: [CLS] did [MASK1] [training topic] [training answer corresponding to the training topic] [training answer corresponding to the training topic]. For example, if the training topic is "What is the speed of the car?", the training answer corresponding to the training topic is "30 km / h", and the training answer corresponding to the training topic is "20 kilometers per hour", then the first training text can be: [CLS] did [MASK1] [What is the speed of the car?] [30 km / h] [20 kilometers per hour].

[0046] B. Setting a second mask for the training questions and / or training answers and / or training responses contained in the first training text.

[0047] For example, part of the training questions and / or training answers and / or training responses contained in the first training text may be randomly masked (for example, 15% of the content may be randomly masked).

[0048] In some cases, when the randomly masked part of the content contains key information of the training questions and / or training answers and / or training answers, for example, when the key information includes at least one of numbers and chemical formulas, since it is difficult to accurately predict numbers, chemical formulas, etc. through the language model, the corresponding masking strategy can be pre-set during masking so that the second mask will not cover the key information of the training questions and / or training answers and / or training answers during random masking.

[0049] For example, the first training sentence after setting the second mask is: [CLS] did [MASK1] [What is the speed of the car? ][30KM / H][20 kilometers per [MASK2]].

[0050] S202: Inputting the first training sentence into the language model to obtain a first prediction result corresponding to the first mask and a second prediction result corresponding to the second mask.

[0051] Specifically, for the first mask, the corresponding predicted vocabulary consists only of "right" and "wrong." For the second mask, the corresponding vocabulary is the full vocabulary of existing methods (typically containing over 20,000 common words). After the first training sentence is fed into the language model, the model simultaneously determines whether the training answer is correct (i.e., [MASK1] → Right / Wrong) when performing a regular cloze task (i.e., predicting the second mask).

[0052] S203: Construct multiple second training sentences based on multiple first training sentences; wherein, at least some of the second training sentences include a first prediction result, a training question, a training answer corresponding to the training question, and a training answer corresponding to the training question; and a second mask is set in the training question and / or the training answer and / or the training answer.

[0053] Specifically, in one application scenario, assume that the first prediction result corresponding to the first mask in step S202 is "true," but the ground-truth label corresponding to the first mask is "false." Directly using the first prediction result to construct the second training sentence will result in error accumulation. To reduce the probability of error accumulation and ensure the training effect of the language model, a certain probability (e.g., 50%) can be used to select either the first prediction result or the ground-truth label corresponding to the first prediction result to construct the second training sentence.

[0054] Specifically, the specific implementation process of the above-mentioned step S203 can be: A. Obtain a labeled training set and an unlabeled training set; wherein, the first training sentences contained in the labeled training set are set with a true value label corresponding to the first mask, and the first training sentences contained in the unlabeled training set are not set with a true value label corresponding to the first mask. B. For some of the first training sentences in the labeled training set and all the first training sentences in the unlabeled training set, replace the first mask in the first training sentence with the corresponding first prediction result to obtain a second training sentence; and for the remaining first training sentences in the labeled training set, replace the first mask in the first training sentence with the corresponding true value label to obtain a second training sentence. It should be noted that for the second training sentence corresponding to the first training sentence, the position of the second mask is the same.

[0055] S204: Input the second training sentence into the language model to obtain a third prediction result corresponding to the second mask.

[0056] Specifically, the second training sentence is input into the language model to perform a cloze task. The cloze task only needs to predict the content of one mask, that is, the language model randomly masks out part of the content, which is also the second mask.

[0057] S205: Obtain a total loss value based on at least the second prediction result and the third prediction result.

[0058] Specifically, before the above step S205, for example, between step S202 and step S203, the following steps may also be included:

[0059] For labeled data, in response to the first training sentence being set with a true value label corresponding to the first mask, a first sub-loss is obtained based on the first prediction result and the true value label of the first mask, and a second sub-loss is obtained based on the second prediction result and the true value label of the second mask, and the sum of the first sub-loss and the second sub-loss is used as the first loss. Optionally, the first sub-loss and the second sub-loss can be cross-entropy losses.

[0060] For example, taking [CLS] doing [MASK1] [What is the speed of the car?] [30KM / H] [20 kilometers per [MASK2] → [right, hour]] as an example, the specific formula can be written as: loss1 = -y 对 *logP 对 -y 小时 *logP 小时 ; Among them, y i represents the true value label (for example, 0 or 1), P i represents the predicted probability.

[0061] For unlabeled data, in response to the fact that the first training sentence is not provided with a true value label corresponding to the first mask, a second sub-loss is obtained based on the second prediction result and the true value label of the second mask, and the second sub-loss is used as the first loss. That is, the loss at the first mask can be ignored for unlabeled data. Since the method provided in this application focuses on comparing the differences at the semantic level between the two reasonings before and after the language model, it has no necessary relationship with whether the student has answered the question. Of course, if there is a "right / wrong" true value label, it can help the model to be better trained. Therefore, based on semantic invariance, the training method provided in this application can make full use of large unlabeled corpus for training.

[0062] In addition, before the above step S205, for example, between the above step S204 and step S203, the method further includes: obtaining a second loss based on the third prediction result and the true value label of the second mask. Optionally, the second loss may be a cross entropy loss.

[0063] Furthermore, the specific implementation process of the above step S204 includes: obtaining a total loss value based on the first loss and the second loss. For example, the sum of the first loss and the second loss is taken as the total loss.

[0064] Of course, in other embodiments, other losses may be introduced, for example, KL divergence loss, wherein KL divergence is an indicator that measures the degree of matching between two probability distributions. The greater the difference between the two distributions, the greater the KL divergence. Specifically, before the above-mentioned step S204, it may also include: obtaining the first global feature vector h_1 of the current first training sentence based on the first prediction result and the second prediction result, and obtaining the second global feature vector h_2 of the current second training sentence based on the third prediction result; obtaining the third loss, namely KL divergence loss, based on the first global feature vector h_1 of the first training sentence and the second global feature vector h_2 of the second training sentence corresponding to the first training sentence. The above-mentioned first global feature vector h_1 can be obtained by recording the global feature variables of the language model at the current stage after the execution of step S202 is completed, and the above-mentioned second global feature vector h_2 can be obtained by recording the global feature variables of the language model at the current stage after the execution of step S204 is completed.

[0065] Furthermore, the implementation process of the above step S205 may be: obtaining a total loss value based on the first loss, the second loss, and the third loss. For example, the sum of the first loss, the second loss, and the third loss is taken as the total loss.

[0066] Here we first review the two major features of the semantic model: a) mask a small number of words in the text, and the language model can complete the missing words through understanding of global semantics; b) global semantic features are encoded to the position corresponding to the beginning of the sentence [CLS]. Based on the above two features, this application proposes a metric learning scheme for semantic invariance. In the input template we constructed, there is a first mask, and the language model will predict the difference between the training answers and the training answers in the input text in the first stage (i.e., step S201-step S202). Furthermore, in the second stage (i.e., step S203-step S204), after we fill the template (i.e., after filling the first mask), if the prediction result of the first stage is correct, due to semantic invariance, the global feature vectors (placeholder [CLS]) obtained by the two inferences should be consistent. Based on this, we propose to use KL divergence to align the global feature vectors ([CLS]) of the two previous and subsequent inferences, which is recorded as the third loss. After adjusting the training parameters of the language model based on the total loss related to the third loss, the language model can be more accurate and have better results.

[0067] S206: Adjust the training parameters of the language model based on the total loss value.

[0068] Specifically, the process of adjusting the training parameters of the language model based on the total loss value can be found in the prior art and will not be described in detail here. In addition, after the above step S206, the following steps may be performed: determining whether a preset condition (e.g., the total loss value converges, or the number of training times reaches a threshold, etc.) is met. If so, training is stopped and the trained language model is output; if not, the next round of training is continued.

[0069] In summary, unlike the existing methods that adopt the "pre-training-fine-tuning" training paradigm, this application unifies the upstream and downstream training objectives by constructing a template, and proposes to use the semantic invariance of the language model to design a two-stage reasoning framework. The second stage can play a role in assisting training and reviewing the reasoning results of the first stage. Specifically, the training goal of the first stage (i.e., the above-mentioned steps S201-S202) is similar to the cloze task in the pre-training stage of the existing method, and the words corresponding to the first mask and the second mask are predicted. In the second stage (i.e., the above-mentioned steps S203-S204), the prediction results of the first stage are filled into the template and then input into the language model. If the prediction results of the first stage are correct, due to semantic invariance, the global feature vectors (placeholders [CLS]) obtained by the two reasonings should be consistent. By comparing the two prediction results, it is possible to determine whether the student's answer has been scored. It is worth noting that the "first stage-second stage" in this application is significantly different from the "pre-training-fine-tuning" paradigm of existing methods: existing methods adopt an obvious serial mechanism, in which the language model is first trained on the pre-training task and then trained on the downstream task, which severs the connection between the upstream and downstream tasks; while the two-stage reasoning in this application is a joint training, which cleverly unifies the upstream and downstream tasks; at the same time, the second stage is a review of the results of the first stage to ensure the credibility of the language model prediction results.

[0070] See also Figure 5 , Figure 5 FIG1 is a schematic diagram of the structure of an embodiment of the answer evaluation device of the present application. The answer evaluation device includes a first construction module 10 and a first prediction module 12.

[0071] Specifically, the first construction module 10 is configured to construct a test sentence based on a first mask, a test question, a corresponding standard answer, and a corresponding student answer. The first mask represents the semantic relevance between the standard answer and the student answer. The first prediction module 12 is connected to the first construction module 10 and is configured to input the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask.

[0072] Furthermore, the answer evaluation device provided in the present application may also include a training module 14, which is connected to the prediction module 12; wherein the training module 14 includes a second construction module, a second prediction module, a third construction module, a third prediction module and an adjustment module connected in sequence; wherein the second construction module is used to construct a plurality of first training sentences based on the first mask, the training questions, the training answers corresponding to the training questions, and the training answers corresponding to the training questions; wherein the training questions and / or the training answers and / or the training answers are provided with a second mask, and the second mask has a corresponding true value label; the second prediction module is used to input the first training sentence into the language model to obtain a label corresponding to the first mask. The corresponding first prediction result, and the second prediction result corresponding to the second mask; the third construction module is used to construct multiple second training sentences based on the multiple first training sentences; wherein, at least some of the second training sentences include the first prediction result, training questions, training answers corresponding to the training questions and training answers corresponding to the training questions; and the training questions and / or training answers and / or training answers are provided with a second mask; the third prediction module is used to input the second training sentence into the language model to obtain a third prediction result corresponding to the second mask; the adjustment module is used to obtain a total loss value based on at least the second prediction result and the third prediction result, and adjust the training parameters of the language model based on the total loss value.

[0073] In one embodiment, the adjustment module is further configured to obtain, in response to the first training statement being provided with a true value label corresponding to the first mask, a first sub-loss based on the first prediction result and the true value label of the first mask, and a second sub-loss based on the second prediction result and the true value label of the second mask, and use the sum of the first sub-loss and the second sub-loss as the first loss; or, in response to the first training statement not being provided with a true value label corresponding to the first mask, obtain a second sub-loss based on the second prediction result and the true value label of the second mask, and use the second sub-loss as the first loss; obtain a second loss based on the third prediction result and the true value label of the second mask; and obtain a total loss value based on the first loss and the second loss.

[0074] Furthermore, before the step of obtaining the total loss value based on the first loss and the second loss, it includes: obtaining the first global feature vector of the current first training sentence based on the first prediction result and the second prediction result, and obtaining the second global feature vector of the current second training sentence based on the third prediction result; obtaining the third loss based on the first global feature vector of the first training sentence and the second global feature vector of the second training sentence corresponding to the first training sentence; the step of obtaining the total loss value based on the first loss and the second loss includes: obtaining the total loss value based on the first loss, the second loss and the third loss.

[0075] In another embodiment, the above-mentioned second construction module is specifically used to construct multiple first training texts based on the first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions; a second mask is set for the training questions and / or training answers and / or training answers contained in the first training text, and the second mask does not cover the key information of the training questions and / or training answers and / or training answers; wherein the key information includes at least one of numbers and chemical formulas.

[0076] The above-mentioned third construction module is specifically used to obtain a labeled training set and an unlabeled training set; wherein, the first training statements contained in the labeled training set are set with a true value label corresponding to the first mask, and the first training statements contained in the unlabeled training set are not set with a true value label corresponding to the first mask; for some of the first training statements in the labeled training set and all the first training statements in the unlabeled training set, the first mask in the first training statement is replaced with the corresponding first prediction result to obtain a second training statement; and for the remaining first training statements in the labeled training set, the first mask in the first training statement is replaced with the corresponding true value label to obtain a second training statement.

[0077] Optionally, the evaluation prediction result is a first evaluation prediction result or a second evaluation prediction result; the first evaluation prediction result represents that the semantic correlation between the standard answer and the student's answer is consistent, and the second evaluation prediction result represents that the semantic correlation between the standard answer and the student's answer is inconsistent; the answer evaluation device provided in this application also includes a comparison module, which is connected to the first prediction module 12, and is used to obtain a third global feature vector based on the evaluation prediction result and the test sentence, obtain a fourth global feature vector based on the first test result evaluation prediction result and the test sentence, and obtain a fifth global feature vector based on the second test result evaluation prediction result and the test sentence; obtain a first similarity between the third global feature vector and the fourth global feature vector, and obtain a second similarity between the third global feature vector and the fifth global feature vector; in response to the first similarity being greater than the second similarity, determine that the student's answer is correct.

[0078] See also Figure 6 , Figure 6Schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device includes: a memory 22 and a processor 20 coupled to each other, the memory 22 stores program instructions, and the processor 20 is used to execute the program instructions to implement any of the above-mentioned answer evaluation methods or language model training methods. Specifically, electronic devices include but are not limited to: desktop computers, laptops, tablet computers, servers, etc., which are not limited here. In addition, the processor 20 can also be called a CPU (Center Processing Unit). The processor 20 may be an integrated circuit chip with signal processing capabilities. The processor 20 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 20 can be implemented by an integrated circuit chip.

[0079] See also Figure 7 , Figure 7 This is a structural diagram of an embodiment of a storage device of the present application. The storage device 30 stores program instructions 300 that can be executed by a processor. The program instructions 300 are used to implement any of the above-mentioned answer evaluation methods or language model training methods.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0081] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for evaluating an answer, characterized in that: include: Constructing a test statement based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein the first mask represents the semantic relevance between the standard answer and the student answer; Inputting the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask; Among them, the step of training the language model includes: constructing multiple first training sentences based on a first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions; wherein, a second mask is set in the training questions and / or the training answers and / or the training answers, and the second mask has a corresponding true value label; inputting the first training sentence into the language model to obtain a first prediction result corresponding to the first mask, and a second prediction result corresponding to the second mask; constructing multiple second training sentences based on multiple first training sentences; inputting the second training sentence into the language model to obtain a third prediction result corresponding to the second mask; obtaining a total loss value based on at least the second prediction result and the third prediction result; and adjusting the training parameters of the language model based on the total loss value.

2. The answer evaluation method according to claim 1, characterized in that: At least part of the second training sentence includes the first prediction result, the training question, the training answer corresponding to the training question and the training answer corresponding to the training question; and the second mask is set in the training question and / or the training answer and / or the training answer.

3. The answer evaluation method according to claim 2, characterized in that: Before the step of obtaining a total loss value based at least on the second prediction result and the third prediction result, the method includes: In response to the first training sentence being set with a true value label corresponding to the first mask, a first sub-loss is obtained based on the first prediction result and the true value label of the first mask, and a second sub-loss is obtained based on the second prediction result and the true value label of the second mask, and the sum of the first sub-loss and the second sub-loss is used as the first loss; or, in response to the first training sentence not being set with a true value label corresponding to the first mask, a second sub-loss is obtained based on the second prediction result and the true value label of the second mask, and the second sub-loss is used as the first loss; Obtaining a second loss based on the third prediction result and the true value label of the second mask; The step of obtaining a loss value based at least on the second prediction result and the third prediction result includes: obtaining the total loss value based on the first loss and the second loss.

4. The answer evaluation method according to claim 3, characterized in that: Before the step of obtaining the total loss value based on the first loss and the second loss, the method includes: obtaining a first global feature vector of the current first training sentence based on the first prediction result and the second prediction result, and obtaining a second global feature vector of the current second training sentence based on the third prediction result; obtaining a third loss based on the first global feature vector of the first training sentence and the second global feature vector of the second training sentence corresponding to the first training sentence; The step of obtaining the total loss value based on the first loss and the second loss includes: obtaining the total loss value based on the first loss, the second loss and the third loss.

5. The answer evaluation method according to claim 2, characterized in that: The step of constructing a plurality of first training sentences based on the first mask, the training questions, the training answers corresponding to the training questions, and the training answers corresponding to the training questions includes: Constructing a plurality of first training texts based on the first mask, the training questions, the training answers corresponding to the training questions, and the training answers corresponding to the training questions; A second mask is set for the training questions and / or training answers and / or training responses contained in the first training text, and the second mask does not cover the key information of the training questions and / or training answers and / or training responses; wherein the key information includes at least one of numbers and chemical formulas.

6. The answer evaluation method according to claim 2, characterized in that: The step of constructing a plurality of second training sentences based on a plurality of the first training sentences comprises: Obtaining a labeled training set and an unlabeled training set; wherein the first training sentences included in the labeled training set are set with true value labels corresponding to the first mask, and the first training sentences included in the unlabeled training set are not set with true value labels corresponding to the first mask; For some of the first training statements in the labeled training set and all of the first training statements in the unlabeled training set, the first mask in the first training statement is replaced with the corresponding first prediction result to obtain a second training statement; and for the remaining first training statements in the labeled training set, the first mask in the first training statement is replaced with the corresponding true value label to obtain a second training statement.

7. The answer evaluation method according to claim 1, characterized in that: The evaluation prediction result is a first evaluation prediction result or a second evaluation prediction result; the first evaluation prediction result indicates that the semantic relevance between the standard answer and the student's answer is consistent, and the second evaluation prediction result indicates that the semantic relevance between the standard answer and the student's answer is inconsistent; After the step of inputting the test sentence into the trained language model to obtain the evaluation prediction result corresponding to the first mask, the method further includes: Obtaining a third global feature vector based on the evaluation prediction result and the test sentence, obtaining a fourth global feature vector based on the first evaluation prediction result and the test sentence, and obtaining a fifth global feature vector based on the second evaluation prediction result and the test sentence; Obtaining a first similarity between the third global feature vector and the fourth global feature vector, and obtaining a second similarity between the third global feature vector and the fifth global feature vector; In response to the first similarity being greater than the second similarity, it is determined that the student's answer is correct.

8. A method for training a language model, characterized in that: include: Constructing a plurality of first training sentences based on a first mask, a training question, a training answer corresponding to the training question, and a training answer corresponding to the training question; wherein a second mask is set in the training question and / or the training answer and / or the training answer, and the second mask has a corresponding true value label; Inputting the first training sentence into the language model to obtain a first prediction result corresponding to the first mask and a second prediction result corresponding to the second mask; Constructing a plurality of second training sentences based on the plurality of first training sentences; wherein at least some of the second training sentences include the first prediction result, the training question, the training answer corresponding to the training question, and the training answer corresponding to the training question; and the second mask is set in the training question and / or the training answer and / or the training answer; Inputting the second training sentence into the language model to obtain a third prediction result corresponding to the second mask; Obtaining a total loss value based at least on the second prediction result and the third prediction result; The training parameters of the language model are adjusted based on the total loss value.

9. A device for evaluating an answer, characterized in that: include: A construction module, configured to construct a test statement based on a first mask, a test question, a standard answer corresponding to the test question, and a student answer corresponding to the test question; wherein the first mask represents a semantic correlation between the standard answer and the student answer; a prediction module, connected to the construction module, configured to input the test sentence into the trained language model to obtain an evaluation prediction result corresponding to the first mask; Among them, the step of training the language model includes: constructing multiple first training sentences based on a first mask, training questions, training answers corresponding to the training questions, and training answers corresponding to the training questions; wherein, a second mask is set in the training questions and / or the training answers and / or the training answers, and the second mask has a corresponding true value label; inputting the first training sentence into the language model to obtain a first prediction result corresponding to the first mask, and a second prediction result corresponding to the second mask; constructing multiple second training sentences based on multiple first training sentences; inputting the second training sentence into the language model to obtain a third prediction result corresponding to the second mask; obtaining a total loss value based on at least the second prediction result and the third prediction result; and adjusting the training parameters of the language model based on the total loss value.

10. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the answer evaluation method described in any one of claims 1 to 7 or the language model training method described in claim 8.

11. A storage device, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the answer evaluation method described in any one of claims 1 to 7 or the language model training method described in claim 8.

Images