Training method and device of judgment model in judgment scene, equipment and storage medium

Abstract

The invention provides a training method and device for a judgment model in a judgment scene, equipment and a storage medium, and relates to the technical field of automatic evaluation of education and teaching, artificial intelligence and the like. According to the specific implementation scheme, a first training data set of dichotomous labels is obtained, wherein first training data comprises questions, standard answers, student answers and classification labels for identifying the student answers to be correct or wrong; generating a second training data set of a three-classification label based on the first training data set of the two-classification label; the second training data comprises questions, standard answers, student answers and classification labels for identifying the student answers to be correct, wrong or unknown; the error judgment probability of the second training data corresponding to the unknown classification label is greater than the error judgment probability of the second training data corresponding to the correct classification label and the error classification label; and based on the second training data set of the three-classification label, training a three-classification judgment model. According to the technology disclosed by the invention, the accuracy of the three-classification judgment model can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of automated evaluation of education and teaching and artificial intelligence and other technical fields, and in particular to a training method, device, equipment and storage medium for a judgment model in a judgment scenario. Background Art

[0002] With the continuous advancement of educational informatization, the evaluation methods in educational and teaching scenarios have also undergone tremendous changes. More and more automated processing tools are applied to the evaluation and assessment of education and teaching. For example, the judgment model based on neural network model training has gained favor for its advantages of high efficiency, convenience, and saving manpower and material resources, and is widely used in the detection of student performance.

[0003] When judging questions, the existing judgment model inputs the question, the standard answer and the student's answer into the judgment model. Based on the question and the standard answer, the judgment model predicts whether the student's answer is correct or wrong. However, due to the limited capabilities of the model and the low accuracy of judging correct and wrong answers, it is necessary to ensure the accuracy of the model's prediction of one category (correct or wrong). For example, when ensuring the accuracy of the model's judgment of correct answers, manual review can be implemented for the other category, that is, manual review can be implemented for questions judged as wrong. Summary of the invention

[0004] The present disclosure provides a training method, apparatus, device and storage medium for a question judgment model in a question judgment scenario.

[0005] According to one aspect of the present disclosure, a method for training a judgment model in a judgment scenario is provided, comprising:

[0006] Obtain a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect;

[0007] Based on the first training data set with two-category labels, a second training data set with three-category labels is generated; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown; wherein the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label;

[0008] Based on the second training data set with the three-category labels, a three-category question determination model is trained.

[0009] According to another aspect of the present disclosure, a training device for a judgment model in a judgment scenario is provided, comprising:

[0010] An acquisition module is used to acquire a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect;

[0011] A generating module is used to generate a second training data set with three-category labels based on the first training data set with two-category labels; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown; wherein the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label;

[0012] A training module is used to train a three-category judgment model based on a second training data set of the three-category labels.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0019] According to the technology disclosed in the present invention, the accuracy of the trained three-category judgment model can be effectively improved.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0026] Figure 5 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0029] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0030] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0031] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1As shown, this embodiment provides a method for training a judgment model in a judgment scenario, which may specifically include the following steps:

[0032] S101, obtaining a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect;

[0033] In this embodiment, the first training data set of binary classification labels can be obtained by collecting questions for evaluating students in teaching scenarios, standard answers for teachers to refer to for judging questions, and student answers made by students.

[0034] S102, generating a second training data set with three-category labels based on the first training data set with two-category labels; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown;

[0035] Among them, the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label. In other words, the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label; at the same time, the error probability of the second training data corresponding to the unknown classification label is also greater than the error probability of the second training data corresponding to the wrong classification label.

[0036] Compared with the first training data set of two-category labels, the second training data set of three-category labels in this embodiment also includes second training data with unknown classification labels. In the actual application of educational and teaching scenarios, when judging questions manually, there is no situation where the student's answer is unknown based on the standard answer. Therefore, the training data of unknown labels cannot be collected, and thus the second training data set of three-category labels cannot be collected. Therefore, in this embodiment, the second training data set of three-category labels can only be generated based on the first training data set of two-category labels.

[0037] The error judgment in this embodiment refers to an error in judgment. For example, based on the question and the standard answer, the student's answer should be judged to be correct in theory, but it is judged to be wrong during the judgment, which is an error judgment. Conversely, based on the question and the standard answer, the student's answer should be judged to be wrong in theory, but it is judged to be correct during the judgment, which is also an error judgment. For some questions, standard answers and student answers, it is very easy to make an error judgment during the judgment.

[0038] In this embodiment, the error probability of the second training data corresponding to the unknown classification label, the correct classification label and the incorrect classification label respectively may refer to the error probability of judging whether the student's answer is correct based on the questions and standard answers in the corresponding second training data.

[0039] For example, the probability of error judgment of each second training data can be detected by using a plurality of different pre-trained judgment models. The plurality of different judgment models here can be trained using different training data. Further optionally, the plurality of different judgment models can also be implemented using different neural network models. For example, for the same second training data, N judgment models judge at the same time. If M judgment models judge correctly, and NM judgment models judge incorrectly. Then the probability of error judgment of the second training data can be (NM) / N.

[0040] Optionally, for another example, in a question-judging task performed on the second training data set, it can be considered that the probability of incorrectly judging the second training data is greater than the probability of incorrectly judging the second training data.

[0041] S103. Based on the second training data set with three-category labels, train a three-category question-determination model.

[0042] In this embodiment, the trained three-category judgment model can judge whether the student's answer is correct, wrong or unknown based on the question and the standard answer.

[0043] Because in the second training data set, the probability of error judgment of the second training data corresponding to the unknown classification label is greater than the probability of error judgment of the second training data corresponding to the correct classification label and the probability of error judgment of the second training data corresponding to the wrong classification label. Conversely, that is to say, in the second training data set, the probability of error judgment of the second training data corresponding to the correct classification label and the second training data corresponding to the wrong classification label is very small. By setting the second training data with a larger probability of error judgment as an unknown classification label, the accuracy and reliability of the training data can be further improved. Then, using such a second training data set to train the three-category judgment model can effectively improve the accuracy of the three-category model prediction, including the accuracy of the predicted correct classification label and the wrong classification label data. Of course, the accuracy of the prediction of unknown classification labels is also very high. However, for data with unknown classification labels, due to uncertainty about whether they are correct or not, manual participation in the review is required later.

[0044] In the prior art, since the accuracy of the judgment model prediction is not high, in order to ensure the accuracy of the two-category judgment model predicting one category (such as correct or wrong), the judgment model predicts the other category (such as wrong or correct) manually for review. When the student answers are evenly distributed or the amount of data for the category that needs manual review is large, the amount of data for manual review will be too large, resulting in the entire judgment task being time-consuming and labor-intensive, and the judgment efficiency is low.

[0045] In order to improve the efficiency of question judgment, in this embodiment, the three-category question judgment model is trained by adopting the above method. Since the probability of wrong judgment of the second training data corresponding to the unknown classification label in the second training data set of the three-category label is greater than the probability of wrong judgment of the second training data corresponding to the correct classification label and the probability of wrong judgment of the second training data corresponding to the wrong classification label, based on such a second training data set, the accuracy of the data of the correct classification label and the wrong classification label predicted by the trained three-category question judgment model can be effectively improved. In this way, in the subsequent manual review, only the questions with the predicted results of unknown classification labels can be manually reviewed, which can effectively reduce the amount of manual review and effectively improve the efficiency of question judgment.

[0046] The training method of the question judgment model in the question judgment scenario of this embodiment, by adopting the above-mentioned scheme, can train a three-category question judgment model with higher accuracy, and then in the subsequent execution of question judgment tasks, it can effectively improve the accuracy of question judgment, and facilitate the subsequent manual review of only the questions with unknown classification labels, which can effectively reduce the amount of manual review and effectively improve the efficiency of question judgment.

[0047] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the training method of the judgment model in the judgment scenario of this embodiment, in the above Figure 1 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 2 As shown, this embodiment provides a method for training a judgment model in a judgment scenario, which may specifically include the following steps:

[0048] S201, obtaining a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect;

[0049] Refer to the above Figure 1 The specific implementation of step S101 of the illustrated embodiment will not be described in detail here.

[0050] S202, using the first training data set with binary classification labels to train a binary classification question-judgment model; the binary classification question-judgment model has a binary classification capability and can identify whether a student's answer is correct or incorrect based on the question and the standard answer;

[0051] S203, based on the binary classification model, obtaining multiple target test data from the test data set; the target test data is the test data in the test data set with a higher probability of error than other test data;

[0052] In this embodiment, based on the binary classification judgment model, multiple target test data with a higher probability of misjudgment can be obtained from the test data set. These target test data may correspond to test data with unknown classification labels.

[0053] For example, when the step S203 is implemented specifically, any of the following implementation methods may be adopted:

[0054] The first implementation method may specifically include the following steps:

[0055] (a1) Use a binary classification model to predict the predicted category of each test data in the test data set;

[0056] (b1) detecting whether the predicted category of each test data is consistent with the corresponding classification label, and obtaining a first detection result;

[0057] For example, the first test result obtained can identify whether the predicted category of each test data in the test data set is consistent with the corresponding classification label. For example, 1 can be used to indicate consistency and 0 can be used to indicate inconsistency. If the test data set includes 1000 test data, the corresponding first test result can include 1000 0s and 1s. Each 0 or 1 corresponds to a test result of a test data.

[0058] (c1) Based on the first detection result, a plurality of test data whose predicted categories are inconsistent with corresponding classification labels are obtained from the test data set as a plurality of target test data.

[0059] If the predicted category of the test data is consistent with the corresponding classification label, it means that the binary classification judgment model has made a correct judgment. If the predicted category of the test data is inconsistent with the corresponding classification label, it means that the binary classification judgment model has made an incorrect judgment. The incorrectly judged test data can be manually reviewed to improve the accuracy of the judgment. In this embodiment, the incorrectly judged test data is obtained as the target test data, which is convenient for the subsequent construction of training data with unknown classification labels, so that the three-category judgment model can learn the ability to recognize unknown classification labels based on the training data with unknown classification labels.

[0060] The second implementation method may specifically include the following steps:

[0061] (a2) Use a binary classification model to predict the probability of the two categories of each test data in the test data set;

[0062] Specifically, when the binary classification judgment model tests each test data, the test data is input into the binary classification judgment model, and the binary classification judgment model can output the predicted probabilities of the test data being two categories. Usually, the category with a large predicted probability can be selected as the predicted category of the test data.

[0063] (b2) based on the predicted probabilities of the two categories of each test data, obtaining the difference between the predicted probabilities of the two categories of each test data;

[0064] (c2) detecting whether the difference between the predicted probabilities of the two categories of each test data is less than a preset threshold, and obtaining a second detection result;

[0065] Specifically, the smaller the difference between the predicted probabilities of the two categories, the greater the probability that the binary classification model will make an error in judging the test data.

[0066] Similarly, similar to the first detection result, the obtained second detection result can identify whether the difference in the predicted probabilities of the two categories of each test data in the test data set is less than a preset threshold value. For example, 1 can be used to indicate that the difference in the predicted probabilities of the two categories is not less than the preset threshold value, and 0 can be used to indicate that the difference in the predicted probabilities of the two categories is less than the preset threshold value. The preset threshold value of this embodiment can be set according to experience or actual needs. For example, it can be set to 0.1 or 0.15, etc., which is not limited here.

[0067] (d2) Based on the second detection result, a plurality of test data pieces in which the difference between the prediction probabilities of the two categories is less than a preset threshold are obtained from the test data set as a plurality of target test data pieces.

[0068] If the difference between the predicted probabilities of the two categories of the test data is not less than the preset threshold, it means that the two-category judgment model is basically correct. If the difference between the predicted probabilities of the two categories of the test data is less than the preset threshold, it means that the two-category judgment model is very likely to make an incorrect judgment. Furthermore, the test data that is very likely to be judged incorrectly is used as the target test data to facilitate the subsequent construction of training data with unknown classification labels, so that the three-category judgment model can learn the ability to recognize unknown classification labels based on the training data with unknown classification labels.

[0069] The above two implementations can both accurately and efficiently obtain multiple test data with a high probability of error from the test data set as multiple target test data. In practical applications, other methods can also be used to obtain multiple target test data, which will not be described one by one here.

[0070] S204, based on the plurality of target test data, identifying first training data with unknown classification labels in the first training data set, and obtaining a second training data set with three classification labels;

[0071] Since the target test data is the test data with a higher probability of misjudgment, it can be used as training data with unknown classification labels. However, the amount of data in the test data set is too small, resulting in less target test data being obtained, which is insufficient to fully train the judgment model. Therefore, in this embodiment, it is necessary to further identify the first training data with unknown classification labels in the first training data set based on multiple target test data, so that more test data with unknown classification labels can be obtained, and the second training data set with three classification labels can be obtained.

[0072] That is to say, compared with the first training data set, the second training data set is equivalent to changing the classification label of the first training data with a higher probability of error in the first training data set from correct or wrong to unknown; while the classification label of the first training data with a lower probability of error remains unchanged. Each first training data in the first training data set obtained in this way corresponds to a second training data in the second training data set. The first training data and the second training data that correspond to each other have exactly the same questions, standard answers, and student answers; while the labels may be the same or different.

[0073] Since the multiple target test data are test data with a higher probability of being misjudged than other test data in the test data set, it can be considered that the first training data with unknown classification labels identified in the first training data set based on the multiple target test data is also the first training data with a higher probability of being misjudged than other first training data. In this way, by identifying the training data with unknown classification labels, the probability of each classification label being misjudged can be reduced, and then when the three-class classification judgment model is subsequently trained based on the second training data set, the accuracy of the trained three-class classification judgment model can be effectively improved.

[0074] For example, when the step S204 is specifically implemented, it may include the following steps:

[0075] (a3) for each target test data, based on the target test data, obtaining matching first training data from the first training data set as first training data of the third category;

[0076] (b3) The category labels of the first training data of the third category in the first training data set are set to unknown, thereby obtaining a second training data set with three category labels.

[0077] In this embodiment, the target test data matches the first training data, which can be considered that the target test data and the first training data have a very high similarity, for example, greater than a preset similarity threshold. Specifically, when performing similarity matching, only the first concatenated text obtained by concatenating the question and the standard answer in the target test data is obtained, and semantic matching is performed with the second text obtained by concatenating the question and the standard answer in the first training data. If the semantic similarity between the two is greater than the preset similarity threshold, the two can be considered to match.

[0078] Alternatively, step (a3), when implemented specifically, may further include the following steps:

[0079] (a4) for each piece of target test data, obtaining a first feature expression of the target test data;

[0080] (b4) obtaining a second feature expression of each first training data in the first training data set;

[0081] Specifically, the first characteristic expression of the target test data can be obtained by concatenating the question of the target test data and the standard answer to obtain a first concatenated text; then obtaining the characteristic expression of the first concatenated text as the first characteristic expression of the target test data. Specifically, the first concatenated text can be embedded (embedding) to obtain a corresponding embedded expression vector as the first characteristic expression of the corresponding target test data. In this way, the target test data can be accurately and effectively expressed.

[0082] Similarly, the second characteristic expression of each first training data in the first training data set can be to concatenate the questions and standard answers in each first training data in the first training data set to obtain the second concatenated text corresponding to each first training data; obtain the characteristic expression of each second concatenated text as the second characteristic expression of the corresponding first training data, and obtain the second characteristic expression of each first training data. Similarly, the second characteristic expression can also be an embedded expression vector of the second concatenated text. Similarly, by adopting this method, each first training data in the first training data set can also be accurately and effectively expressed.

[0083] (c4) calculating the distance between the first feature expression and each second feature expression;

[0084] Since both the first feature expression and the second feature expression are vectors, the distance between the first feature expression and each second feature expression can also reflect the vector similarity between the first feature expression and the second feature expression. The smaller the distance between the two feature expressions, the higher the vector similarity, that is, the higher the similarity between the corresponding target test data and the questions and standard answers in the first training data.

[0085] (d4) detecting whether the distance between the first feature expression and each second feature expression is less than a preset distance threshold; if so, executing step (e4); otherwise, not performing any processing for the time being, and continuing to process the next first training data or the next target test data;

[0086] (e4) taking the first training data corresponding to the second feature expression whose distance to the first feature expression is less than a preset distance threshold as the first training data of the third category.

[0087] For each target test data, the first training data matching the target test data is obtained in accordance with steps (a4)-(e4) as the first training data of the third classification. In this way, the first training data matching the target test data can be obtained efficiently and accurately from the first training data set, which provides an effective guarantee for the subsequent identification of training data with unknown classification labels in the second training data set.

[0088] Steps S202-S204 are as above Figure 1 An implementation of step S102 of the illustrated embodiment. This implementation, with the aid of the test data set, can accurately and efficiently obtain a second training data set of three-category labels with a very low error rate.

[0089] Optionally, in one embodiment of the present disclosure, other methods can be used to generate a second training data set with three-category labels. For example, a two-category judgment model trained with other training data sets can be used to detect each first training data in the first training data set in turn, and obtain the first training data that can be identified as an unknown label from the first training data set according to the above (a1)-(c1) or (a2)-(d2) method of obtaining target test data, and identify it to obtain the second training data set. Of course, other methods can also be used to generate a second training data set with three-category labels, which will not be repeated here one by one.

[0090] Further optionally, before step (d4), a preset distance threshold may be configured. For example, it may be configured based on experience. Or in this embodiment, the following steps may be used for configuration:

[0091] (a5) setting the classification labels of multiple target test data in the test data set to unknown;

[0092] (b5) Calculate the average feature expression of the test data of each classification label in the three classification labels in the test data set;

[0093] Specifically, according to the relevant calculation method of the feature expression of the test data in the above embodiment, the feature expression of each test data in each classification label is obtained. Then the average of the feature expressions of all the test data in the classification label is taken as the average feature expression of the test data of the classification label. (c5) A first distance between the average feature expression of the test data with unknown classification labels and the average feature expression of the test data with correct classification labels is obtained in the test data set, and a second distance between the average feature expression of the test data with unknown classification labels and the average feature expression of the test data with incorrect classification labels is obtained;

[0094] The first distance between the average feature expression of the test data with unknown classification labels and the average feature expression of the test data with correct classification labels can be obtained according to the distance calculation method between vectors. The calculation method of the second distance is similar and will not be repeated here.

[0095] (d5) Selecting the smaller one between the first distance and the second distance as the preset distance threshold.

[0096] That is to say, the first distance can be considered as the standard value of the distance between two test data with unknown classification labels and correct classification labels, and the second distance can be considered as the standard value of the distance between two test data with unknown classification labels and incorrect classification labels. If the distance between two test data is less than the smaller of the first distance and the second distance, it can be considered that the distance between the corresponding two test data is too small and the similarity is too high. Based on this, the smaller of the first distance and the second distance can be selected as the preset distance threshold.

[0097] The preset distance threshold set in the above manner is very reasonable and accurate.

[0098] S205. Based on the second training data set with three-category labels, train a three-category question-determination model.

[0099] Since the second training data set of three-category labels used are all training data with very low probability of error judgment, the trained three-category judgment model can be a model with very high accuracy. In this way, when judging questions based on the three-category judgment model, the accuracy of correct and incorrect answers can be effectively improved, and only questions with unknown classification labels can be manually reviewed later, thereby effectively reducing the amount of manual review.

[0100] Since both the questions and answers in the question-judgment scenario include a large amount of language and text, the question-judgment model of this embodiment can be implemented based on a large language model (LLM for short).

[0101] The training method of the question judgment model in the question judgment scenario of this embodiment, by adopting the above-mentioned scheme, can train a three-category question judgment model with higher accuracy, and then in the subsequent execution of question judgment tasks, it can effectively improve the accuracy of question judgment. Subsequently, only the questions with unknown classification labels can be submitted to manual review, which can effectively reduce the amount of manual review and effectively improve the efficiency of question judgment.

[0102] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; Figure 3 As shown, this embodiment provides a training device 300 for a judgment model in a judgment scenario, including:

[0103] An acquisition module 301 is used to acquire a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect;

[0104] A generating module 302 is used to generate a second training data set with three-category labels based on the first training data set with two-category labels; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown; wherein the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label;

[0105] The training module 303 is used to train a three-category question determination model based on the second training data set of the three-category labels.

[0106] Compared with the first training data set of two-category labels, the second training data set of three-category labels in this embodiment also includes second training data with unknown classification labels. In the actual application of educational and teaching scenarios, training data with unknown labels cannot be collected, and thus the second training data set of three-category labels cannot be collected. Based on this, in this embodiment, the second training data set of three-category labels is generated based on the first training data set of two-category labels.

[0107] In the second training data set of the three-category label, the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label. Based on this, the data with a higher probability of error can be divided into the second training data of the unknown classification label, and the error probability of the second training data corresponding to the correct classification label and the second training data corresponding to the wrong classification label will be further reduced. In this way, the accuracy and reliability of the training data in the second training data set can be improved. Then, using such a second training data set to train the three-category judgment model can effectively improve the accuracy of the three-category model prediction, including the accuracy of the predicted correct classification label and the wrong classification label data. Of course, the accuracy of the prediction of unknown classification labels is also very high. However, for samples with unknown classification labels, due to uncertainty about whether they are correct or not, manual participation in the review is required later.

[0108] The training device 300 for the question judgment model in the question judgment scenario of this embodiment, by adopting the above-mentioned modules, realizes the implementation principle and technical effect of the training of the question judgment model in the question judgment scenario, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0109] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure; the training device 400 of the judgment model in the judgment scenario of this embodiment, in the above Figure 3 Based on the illustrated embodiment, the technical solution of the present disclosure is further described in more detail. Figure 4 As shown, the training device 400 for the judgment model in the judgment scenario of this embodiment includes the above Figure 3 Modules with the same functions and the same names shown are: acquisition module 401 , generation module 402 and training module 303 .

[0110] like Figure 4 As shown, the generation module 402 of this embodiment includes:

[0111] A training unit 4021 is used to train a binary classification question-determination model using the first training data set of the binary classification labels; the binary classification question-determination model has a binary classification capability and can identify whether a student's answer is correct or wrong based on the question and the standard answer;

[0112] The acquisition unit 4022 is used to acquire multiple target test data from the test data set based on the binary classification judgment model; the target test data is the test data in the test data set whose probability of error judgment is greater than that of other test data;

[0113] The generating unit 4023 is configured to identify first training data with unknown classification labels in the first training data set based on the multiple target test data, and obtain a second training data set with three classification labels.

[0114] Further optionally, in one embodiment of the present disclosure, the acquiring unit 4022 is configured to:

[0115] Using the binary classification model, predict the predicted category of each test data in the test data set;

[0116] Detecting whether the predicted category of each of the test data is consistent with the corresponding classification label, and obtaining a first detection result;

[0117] Based on the first detection result, a plurality of test data whose predicted categories are inconsistent with corresponding classification labels are obtained from the test data set as the plurality of target test data.

[0118] In this implementation, the acquisition unit 4022 can screen multiple target test data based on the first detection result. Specifically, the predicted category of the test data is consistent with the corresponding classification label, indicating that the two-category judgment model is correct. If the predicted category of the test data is inconsistent with the corresponding classification label, it means that the two-category judgment model is wrong. In this embodiment, the wrongly judged test data is obtained as the target test data, which is convenient for the subsequent construction of training data with unknown classification labels, so that the three-category judgment model can learn the ability to recognize unknown classification labels based on the training data of unknown classification labels.

[0119] Further optionally, in one embodiment of the present disclosure, the acquiring unit 4022 is configured to:

[0120] Using the binary classification model, predict the prediction probabilities of the two categories of each test data in the test data set;

[0121] Based on the predicted probabilities of the two categories of each test data, obtaining the difference between the predicted probabilities of the two categories of each test data;

[0122] Detect whether the difference between the predicted probabilities of the two categories of each test data is less than a preset threshold, and obtain a second detection result;

[0123] Based on the second detection result, a plurality of test data pieces in which the difference between the prediction probabilities of two categories is less than a preset threshold are obtained from the test data set as the plurality of target test data pieces.

[0124] Taking into account the actual application scenario, the smaller the difference between the predicted probabilities of the two categories, the greater the probability that the binary classification judgment model will make an error in judging the test data. Based on this, a preset threshold can be configured. If the difference between the predicted probabilities of the two categories of the test data is not less than the preset threshold, it means that the binary classification judgment model makes a basically correct judgment. If the difference between the predicted probabilities of the two categories of the test data is less than the preset threshold, it means that the binary classification judgment model is very likely to make an error. Furthermore, the test data that is very likely to be judged incorrectly is used as the target test data to facilitate the subsequent construction of training data for unknown classification labels, so that the three-category judgment model can learn the ability to recognize unknown classification labels based on the training data for unknown classification labels.

[0125] Further optionally, in one embodiment of the present disclosure, the generating unit 4023 is used to:

[0126] For each of the target test data, based on the target test data, obtaining matching first training data from the first training data set as first training data of a third category;

[0127] The category labels of the first training data of the third category in the first training data set are set to unknown, so as to obtain a second training data set of the three-category labels.

[0128] Since the target test data is the test data with a higher probability of misjudgment, it can be used as training data with unknown classification labels. However, the amount of data in the test data set is too small, resulting in even less target test data being obtained, which is insufficient to fully train the judgment model. Therefore, in this embodiment, it is necessary to further identify the first training data with unknown classification labels in the first training data set based on multiple target test data to obtain the second training data set, so as to ensure the richness of the test data with unknown classification labels in the second training data set.

[0129] Further optionally, in one embodiment of the present disclosure, the generating unit 4023 is used to:

[0130] For each piece of the target test data, obtaining a first feature expression of the target test data;

[0131] Obtaining a second feature expression of each of the first training data in the first training data set;

[0132] Calculating the distance between the first feature expression and each of the second feature expressions;

[0133] Detecting whether the distance between the first feature expression and each of the second feature expressions is less than a preset distance threshold;

[0134] If so, the first training data corresponding to the second feature expression whose distance to the first feature expression is less than a preset distance threshold is used as the first training data of the third category.

[0135] The smaller the distance between the first feature expression and the second feature expression, the higher the similarity between the two feature expressions. By obtaining a second feature expression whose distance from the first feature expression is less than a preset distance threshold, first training data similar to the target test data can be retrieved from the first training data set to construct training data with unknown classification labels.

[0136] Further optionally, in one embodiment of the present disclosure, the generating unit 4023 is used to:

[0137] Splicing the questions and standard answers of the target test data to obtain a first spliced ​​text;

[0138] Acquire a feature expression of the first concatenated text as a first feature expression of the target test data;

[0139] The generating unit 4023 is used for:

[0140] splicing the questions and standard answers in each of the first training data in the first training data set to obtain a second spliced ​​text corresponding to each of the first training data;

[0141] The feature expression of each second concatenated text is obtained as the second feature expression of the corresponding first training data, so as to obtain the second feature expression of each first training data.

[0142] Further optionally, in one embodiment of the present disclosure, the generating unit 4023 is further configured to:

[0143] Configure the preset distance threshold.

[0144] Further optionally, in one embodiment of the present disclosure, the generating unit 4023 is further configured to:

[0145] Setting the classification labels of the plurality of target test data in the test data set to unknown;

[0146] Calculate the average feature expression of the test data of each classification label in the three classification labels in the test data set;

[0147] Obtaining a first distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with correct classification labels in the test data set, and a second distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with incorrect classification labels;

[0148] A smaller one is selected from the first distance and the second distance as the preset distance threshold.

[0149] The preset distance threshold in this embodiment can be set based on experience. Alternatively, the above implementation method can be used to analyze the three types of classification labels in the test data set to obtain the average feature expression of each type of classification label. Then, the first distance between the average feature expression of the test data with unknown classification labels and the average feature expression of the test data with correct classification labels, and the second distance between the average feature expression of the test data with unknown classification labels and the average feature expression of the test data with incorrect classification labels can be further obtained. The smaller one is taken as the preset distance threshold. The preset distance threshold set by this setting method is very reasonable and accurate.

[0150] The training device 400 for the question judgment model in the question judgment scenario of this embodiment, by adopting the above-mentioned modules, realizes the implementation principle and technical effect of the training of the question judgment model in the question judgment scenario, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0151] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0152] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0153] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0154] like Figure 5As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0155] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the above-mentioned methods of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0157] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0162] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a judgment model in a judgment scenario, comprising: Obtain a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect; Based on the first training data set with two-category labels, a second training data set with three-category labels is generated; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown; wherein the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label; Based on the second training data set with the three-category labels, a three-category question determination model is trained.

2. The method according to claim 1, wherein: Generating a second training data set of three-category labels based on the first training data set of two-category labels includes: The first training data set of the binary classification labels is used to train a binary classification question-determining model; the binary classification question-determining model has a binary classification capability and can identify whether a student's answer is correct or incorrect based on the question and the standard answer; Based on the binary classification model, multiple target test data are obtained from the test data set; the target test data is the test data in the test data set whose probability of error is greater than that of other test data; Based on the multiple target test data, first training data with unknown classification labels are identified in the first training data set to obtain the second training data set with three classification labels.

3. The method according to claim 2, wherein: Based on the binary classification model, multiple target test data are obtained from the test data set, including: Using the binary classification model, predict the predicted category of each test data in the test data set; Detecting whether the predicted category of each of the test data is consistent with the corresponding classification label to obtain a first detection result; Based on the first detection result, a plurality of test data whose predicted categories are inconsistent with corresponding classification labels are obtained from the test data set as the plurality of target test data.

4. The method according to claim 2, wherein: Based on the binary classification model, multiple target test data are obtained from the test data set, including: Using the binary classification model, predict the prediction probabilities of the two categories of each test data in the test data set; Based on the predicted probabilities of the two categories of each test data, obtaining the difference between the predicted probabilities of the two categories of each test data; Detect whether the difference between the predicted probabilities of the two categories of each test data is less than a preset threshold, and obtain a second detection result; Based on the second detection result, a plurality of test data pieces in which the difference between the prediction probabilities of two categories is less than a preset threshold are obtained from the test data set as the plurality of target test data pieces.

5. The method according to any one of claims 2 to 4, wherein: Based on the plurality of target test data, first training data with unknown classification labels are identified in the first training data set to obtain a second training data set with three classification labels, including: For each of the target test data, based on the target test data, obtaining matching first training data from the first training data set as first training data of a third category; The category labels of the first training data of the third category in the first training data set are set to unknown, so as to obtain a second training data set of the three-category labels.

6. The method according to claim 5, wherein: For each of the target test data, based on the target test data, obtaining matching first training data from the first training data set as first training data of a third category, including: For each piece of the target test data, obtaining a first feature expression of the target test data; Obtaining a second feature expression of each of the first training data in the first training data set; Calculating the distance between the first feature expression and each of the second feature expressions; Detecting whether the distance between the first feature expression and each of the second feature expressions is less than a preset distance threshold; If so, the first training data corresponding to the second feature expression whose distance to the first feature expression is less than a preset distance threshold is used as the first training data of the third category.

7. The method according to claim 6, wherein: Acquiring a first characteristic expression of the target test data includes: Splicing the questions and standard answers of the target test data to obtain a first spliced ​​text; Acquire a feature expression of the first concatenated text as a first feature expression of the target test data; Wherein, obtaining the second feature expression of each of the first training data in the first training data set includes: splicing the questions and standard answers in each of the first training data in the first training data set to obtain a second spliced ​​text corresponding to each of the first training data; The feature expression of each second concatenated text is obtained as the second feature expression of the corresponding first training data, so as to obtain the second feature expression of each first training data.

8. The method according to claim 6, wherein: Before detecting whether the distance between the first feature expression and each of the second feature expressions is less than a preset distance threshold, the method further includes: Configure the preset distance threshold.

9. The method according to claim 8, wherein: Configuring the preset distance threshold includes: Setting the classification labels of the plurality of target test data in the test data set to unknown; Calculate the average feature expression of the test data of each classification label in the three classification labels in the test data set; Obtaining a first distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with correct classification labels in the test data set, and a second distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with incorrect classification labels; A smaller one is selected from the first distance and the second distance as the preset distance threshold.

10. A training device for a judgment model in a judgment scenario, comprising: An acquisition module is used to acquire a first training data set with binary classification labels, wherein each first training data in the first training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct or incorrect; A generating module is used to generate a second training data set with three-category labels based on the first training data set with two-category labels; each second training data in the second training data set includes a question, a standard answer, a student answer, and a classification label indicating whether the student answer is correct, wrong, or unknown; wherein the error probability of the second training data corresponding to the unknown classification label is greater than the error probability of the second training data corresponding to the correct classification label and the error probability of the second training data corresponding to the wrong classification label; A training module is used to train a three-category judgment model based on a second training data set of the three-category labels.

11. The device according to claim 10, wherein: The generating module comprises: A training unit, used to train a binary classification question-determining model using the first training data set of the binary classification labels; the binary classification question-determining model has a binary classification capability and can identify whether a student's answer is correct or incorrect based on the question and the standard answer; An acquisition unit is used to acquire multiple target test data from the test data set based on the binary classification judgment model; the target test data is the test data in the test data set whose probability of error judgment is greater than that of other test data; A generating unit is used to identify first training data with unknown classification labels in the first training data set based on the multiple target test data, so as to obtain a second training data set with the three classification labels.

12. The device according to claim 11, wherein The acquisition unit is used to: Using the binary classification model, predict the predicted category of each test data in the test data set; Detecting whether the predicted category of each of the test data is consistent with the corresponding classification label to obtain a first detection result; Based on the first detection result, a plurality of test data whose predicted categories are inconsistent with corresponding classification labels are obtained from the test data set as the plurality of target test data.

13. The device according to claim 11, wherein the acquisition unit is configured to: Using the binary classification model, predict the prediction probabilities of the two categories of each test data in the test data set; Based on the predicted probabilities of the two categories of each test data, obtaining the difference between the predicted probabilities of the two categories of each test data; Detect whether the difference between the predicted probabilities of the two categories of each test data is less than a preset threshold, and obtain a second detection result; Based on the second detection result, a plurality of test data pieces in which the difference between the prediction probabilities of two categories is less than a preset threshold are obtained from the test data set as the plurality of target test data pieces.

14. The device according to any one of claims 11 to 13, wherein: The generating unit is used for: For each of the target test data, based on the target test data, obtaining matching first training data from the first training data set as first training data of a third category; The category labels of the first training data of the third category in the first training data set are set to unknown, so as to obtain a second training data set of the three-category labels.

15. The device according to claim 14, wherein: The generating unit is used for: For each piece of the target test data, obtaining a first feature expression of the target test data; Obtaining a second feature expression of each of the first training data in the first training data set; Calculating the distance between the first feature expression and each of the second feature expressions; Detecting whether the distance between the first feature expression and each of the second feature expressions is less than a preset distance threshold; If so, the first training data corresponding to the second feature expression whose distance to the first feature expression is less than a preset distance threshold is used as the first training data of the third category.

16. The device according to claim 15, wherein: The generating unit is used for: Splicing the questions and standard answers of the target test data to obtain a first spliced ​​text; Acquire a feature expression of the first concatenated text as a first feature expression of the target test data; The generating unit is used for: splicing the questions and standard answers in each of the first training data in the first training data set to obtain a second spliced ​​text corresponding to each of the first training data; The feature expression of each second concatenated text is obtained as the second feature expression of the corresponding first training data, so as to obtain the second feature expression of each first training data.

17. The device according to claim 15, wherein: The generating unit is further used for: Configure the preset distance threshold.

18. The device according to claim 17, wherein: The generating unit is further used for: Setting the classification labels of the plurality of target test data in the test data set to unknown; Calculate the average feature expression of the test data of each classification label in the three classification labels in the test data set; Obtaining a first distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with correct classification labels in the test data set, and a second distance between an average feature expression of test data with unknown classification labels and an average feature expression of test data with incorrect classification labels; A smaller one is selected from the first distance and the second distance as the preset distance threshold.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images