A scoring method and related model training method and related device

Abstract

The application discloses a scoring method and a related model training method and related device. The method comprises the following steps: obtaining a target text corresponding to to-be-scored answer information, the to-be-scored answer information being obtained by answering a question, and the target text comprising reference text information and to-be-filled scoring information, the reference text information comprising at least one target scoring example of the question and the to-be-scored answer information; and based on the reference text information in the target text, the to-be-filled scoring information in the target text is predicted and filled to obtain a scoring result of the to-be-scored answer information. In the foregoing manner, the application can score the answer information, and the accuracy of the scoring is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular to a scoring method and related model training method and related device. BACKGROUND

[0002] Natural language processing technology has been widely applied to various aspects of daily life, such as text classification, speech recognition, machine translation, etc. The education field is one of the landing scenarios of natural language processing technology, so as to be able to cope with large-scale question scoring through natural language processing technology. However, in the education scenario, due to the difference in semantic expression, there will be differences between the correct answer and the standard answer, so when using natural language processing technology to cope with large-scale question scoring, the scoring result obtained is not accurate enough.

[0003] In summary, how to use natural language processing technology to accurately score a large number of answerers' answers is of great significance. SUMMARY

[0004] The technical problem solved by the present application is to provide a scoring method and related model training method and related device, which can score answer information and has high accuracy of scoring results.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide a scoring method, which comprises: obtaining a target text corresponding to to-be-scored answer information, the to-be-scored answer information being obtained by answering a question, the target text comprising reference text information and to-be-filled scoring information, the reference text information comprising at least one target scoring example of the question and the to-be-scored answer information; based on the reference text information in the target text, predicting and filling the to-be-filled scoring information in the target text to obtain a scoring result of the to-be-scored answer information.

[0006] Among them, at least one target scoring example includes a target scoring example corresponding to a correct reference answer and a target scoring example corresponding to an incorrect reference answer, wherein each target scoring example contains a reference answer of the question and a reference score of the reference answer; and / or, the to-be-filled scoring information in the target text is provided with a mask label, and the mask label indicates that it needs to be filled; and / or, the reference text information further comprises the question.

[0007] Among them, the step of predicting and filling the to-be-filled scoring information in the target text based on the reference text information in the target text is performed by a prediction model.

[0008] Wherein, before predicting and filling the to-be-filled scoring information in the target text based on the reference text information in the target text, the method further comprises: obtaining a plurality of sample texts, the sample texts comprising a sample question, at least one sample scoring example, sample answer information to be scored, and sample scoring results, at least one part of the sample question, the sample scoring example, the sample answer information, and the sample scoring results of each sample text being a to-be-filled part; and training the prediction model using the plurality of sample texts to obtain a prediction model capable of predicting the to-be-filled part.

[0009] Wherein, the to-be-filled parts of the at least two sample texts are different; and / or, one part of the sample question, the sample answer information, and the sample scoring results of each sample text is a to-be-filled part.

[0010] Wherein, obtaining a plurality of sample texts comprises: obtaining at least one set of question data, wherein each set of question data comprises a question, at least one scoring example of the question, answer information of the question, and scoring results of the answer information; using each set of question data to form an original text; and for each original text, performing at least two mask processing on the original text to obtain a sample text corresponding to each mask processing, each mask processing being mask processing on different parts of the question, the scoring example, the answer information, and the scoring results of the original text, the masked part being a to-be-filled part.

[0011] Wherein, training the prediction model using the plurality of sample texts to obtain a prediction model capable of predicting the to-be-filled part comprises: predicting and filling the masked part in each sample text using the prediction model to obtain a prediction filling result; and adjusting network parameters of the prediction model using a prediction difference of each sample text, the prediction difference being a difference between the prediction filling result and original content of the masked part.

[0012] Wherein, before obtaining the target text corresponding to the to-be-scored answer information, the method further comprises: using a standard answer of the question as a correct reference answer of the question, and obtaining an incorrect reference answer of the question using a preset strategy; and generating at least two target correction examples as target scoring examples using the correct reference answer and the incorrect reference answer of the question.

[0013] Wherein, after predicting and filling the to-be-filled scoring information in the target text based on the reference text information in the target text, the method further comprises: obtaining feedback information of a user on a scoring result of the to-be-scored answer information obtained by the prediction model; and updating the target scoring example of the question according to the feedback information.

[0014] To solve the above technical problems, another technical solution adopted by the present application is to provide a scoring model training method, which comprises: obtaining a plurality of sample texts, the sample texts comprising sample questions, at least one sample scoring example, sample answer information to be scored, and sample scoring results, each sample scoring example containing a reference answer of a sample question and a reference score of the reference answer, and at least one part of the sample question, the sample scoring example, the sample answer information, and the sample scoring result of each sample text being a part to be filled; and training a prediction model using the plurality of sample texts to obtain a prediction model capable of predicting the part to be filled, wherein the prediction model is used to predict and fill the part to be filled in the sample text.

[0015] To solve the above technical problems, another technical solution adopted by the present application is to provide a scoring device, which comprises: an obtaining module configured to obtain a target text corresponding to answer information to be scored, the answer information to be scored being obtained by answering a question, and the target text comprising reference text information and scoring information to be filled, the reference text information comprising at least one target scoring example of the question and the answer information to be scored; and a filling module configured to predict and fill the scoring information to be filled in the target text based on the reference text information in the target text to obtain a scoring result of the answer information to be scored.

[0016] To solve the above technical problems, another technical solution adopted by the present application is to provide a scoring model training device, which comprises: a sample obtaining module configured to obtain a plurality of sample texts, the sample texts comprising sample questions, at least one sample scoring example, sample answer information to be scored, and sample scoring results, each sample scoring example containing a reference answer of a sample question and a reference score of the reference answer, and at least one part of the sample question, the sample scoring example, the sample answer information, and the sample scoring result of each sample text being a part to be filled; and a sample filling module configured to train a prediction model using the plurality of sample texts to obtain a prediction model capable of predicting the part to be filled, wherein the prediction model is used to predict and fill the part to be filled in the sample text.

[0017] To solve the above technical problems, another technical solution adopted by the present application is to provide a scoring device, which comprises: an obtaining module configured to obtain a target text corresponding to answer information to be scored, the answer information to be scored being obtained by answering a question, and the target text comprising reference text information and scoring information to be filled, the reference text information comprising at least one target scoring example of the question and the answer information to be scored; and a filling module configured to predict and fill the scoring information to be filled in the target text based on the reference text information in the target text to obtain a scoring result of the answer information to be scored.

[0018] To solve the above technical problems, another technical solution adopted by the present application is to provide a computer readable storage medium for storing program instructions, the program instructions being executable to implement the above scoring method and / or the above model training method.

[0019] The aforementioned scheme first obtains the target text corresponding to the answer to be scored. This target text contains reference text information and the scoring information to be filled. Then, based on the reference text information, the scoring result of the answer to be scored is obtained. The scheme of this application can directly predict and fill in the scoring information to be filled based on the reference text information to obtain the scoring result of the answer to be scored. Furthermore, the reference text information in this application contains at least one target scoring example, which can be used as a reference when predicting and filling in the scoring information to be filled, thus improving the accuracy of the scoring result. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an embodiment of the scoring method provided in this application;

[0021] Figure 2 This is a schematic diagram of the framework of an embodiment of the scoring method provided in this application;

[0022] Figure 3 This is a partial flowchart of an embodiment of the scoring method provided in this application;

[0023] Figure 4 This is a schematic diagram of different masked portions of sample text in one embodiment of the scoring method provided in this application;

[0024] Figure 5 yes Figure 3 The flowchart of step S210 shown is a schematic diagram of one embodiment;

[0025] Figure 6 yes Figure 3 The flowchart of step S220 shown is a schematic diagram of one embodiment;

[0026] Figure 7 This is a partial flowchart of an embodiment of the scoring method provided in this application;

[0027] Figure 8 This is a flowchart illustrating an embodiment of the scoring model training method provided in this application;

[0028] Figure 9 This is a schematic diagram of the framework of an embodiment of the scoring device provided in this application;

[0029] Figure 10 This is a schematic diagram of the framework of an embodiment of the training device for the scoring model provided in this application;

[0030] Figure 11 This is a schematic diagram of the framework of an embodiment of the scoring device provided in this application;

[0031] Figure 12This is a schematic diagram of the framework of the computer-readable storage medium provided in this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] It should be noted that the scoring method provided in this application is applicable to scoring any online or paper-based quizzes, and can score relevant answer information for any subject and any question type, such as multiple-choice questions, fill-in-the-blank questions, and short-answer questions. Furthermore, the reference text information in the target text of this application includes at least one target scoring example, which can be used as a reference when predicting and filling in the scoring information to be filled, thus improving the accuracy of the scoring results.

[0034] At the same time, it should be noted that due to differences in semantic expression, there will be differences between the standard answer and the correct answer. Furthermore, seemingly similar answers can convey very different meanings. For example, in the question "The function of food preservatives," the standard answer A, "Prevents food spoilage," and the correct answer B, "Enables food to be preserved for a long time," although seemingly different, convey the same meaning. Similarly, in the question "The function of filter paper," the answer A, "Filtering," and the answer B, "Suction filtration," although seemingly similar, convey very different meanings. Therefore, when scoring question answers, simply comparing the answer with the standard answer is not accurate enough. Based on this, to improve the accuracy of scoring question answers, this application can further provide multiple scoring examples for reference during the scoring process. Furthermore, answers with significant differences can be selected from existing answer information as scoring examples to increase the reference value of the scoring examples, thereby improving the accuracy of the scoring results. For example, answers A, B, C, and D are all correct responses to a question. Answers A and B are almost identical, except that answer B contains more function words. Answers C and D appear significantly different from answer A, but are all correct. Therefore, answers A, C, and D can be selected as scoring examples. Compared to having only one standard answer, having multiple scoring examples increases the amount of information available for reference during scoring. Therefore, the method described in this application can improve the accuracy of scoring question responses to a certain extent, thus enabling faster and more accurate scoring results when scoring a large number of questions.

[0035] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the scoring method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 1 The sequence of processes shown is limited.

[0036] S110: Obtain the target text corresponding to the answer information to be scored. The answer information to be scored is obtained by answering the target question. The target text includes reference text information and scoring information to be filled. The reference text information includes at least one target scoring example of the target question and the answer information to be scored.

[0037] The method in this embodiment is used to obtain a scoring result for the answer information to be scored based on reference text information. The target text corresponding to the answer information to be scored is text composed of reference text information and scoring information to be filled in.

[0038] In one embodiment, the reference text information includes at least one target scoring example for the target question and answer information to be scored. Each target scoring example includes a reference answer to the target question and a reference score for that answer. In other words, the target text is composed of at least one target scoring example, answer information to be scored, and scoring information to be filled in. In another embodiment, the reference text information, in addition to including at least one target scoring example and answer information to be scored, also includes the target question. That is, the acquired target text is composed of, for example... Figure 2 The text consists of four main parts: the target question, at least one target scoring example, the answer to be scored, and the scoring information to be filled. In this embodiment, the target question may only contain target question information, or it may be as follows: Figure 2 The text shown includes both the target question information and the corresponding standard answer. The specific target text and the content included in the target question can be determined based on the actual scoring results. Furthermore, the order of the components in the target text can be random or arranged in a specific order; no specific restrictions are imposed here.

[0039] The answer information to be scored is obtained by answering the target question. In this embodiment, the answer information to be scored is the object to be scored. The scoring information to be filled is the part to be filled with the scoring result corresponding to the answer information. In this embodiment, the scoring information to be filled can be predicted and filled directly. In some embodiments, in order to distinguish the part to be filled, the part to be filled is marked, for example, by setting a mask label for the part to be filled, such as... Figure 2 The "[MASK]" in the text, or for example, leaving the part that needs to be filled blank.

[0040] Wherein, at least one target scoring example serves as a reference example when scoring the answer information to be scored. "At least one target scoring example" means that the number of target scoring examples can be one or more, depending on the actual situation. For example, for objective questions with a unique answer (multiple-choice questions or fill-in-the-blank questions with a unique answer), there can be one target scoring example. However, for subjective questions (fill-in-the-blank questions or essay questions with non-unique answers), due to the divergent thinking of the respondent and differences in semantic expression, there will be differences between the standard answer and the correct answer. To reduce erroneous scoring caused by differences in semantic expression, there can be multiple target scoring examples. In one embodiment, such as... Figure 2 As shown, the target scoring example includes a reference answer to the target question and a reference score corresponding to the reference answer. In other words, the target scoring example includes both the reference answer information and the score corresponding to the reference answer information.

[0041] In one embodiment, at least one target score example contains only target score examples corresponding to correct reference answers; in another embodiment, at least one target score example includes, for example, Figure 2 The example shown is a target score corresponding to a correct reference answer (e.g.) Figure 2 The example of "1 point for NaOH" and the target score for an incorrect reference answer (e.g.) Figure 2 (e.g., "0 points for 3Fe"). The correct reference answer can be the standard answer or a correct answer selected from the respondent's responses that is semantically identical or similar to the standard answer. Incorrect responses can be randomly selected incorrect answers, or responses from respondents that appear similar to the correct answer but have different semantic expressions. For example, the answers "filtering" and "vacuum filtration," or "solid to liquid" and "liquid to solid," may appear similar but convey significantly different meanings. Understandably, in some embodiments, respondent responses that appear similar to the correct answer but have different semantic expressions can be selected as incorrect reference answers. The existence of these incorrect reference answers can effectively reduce scoring errors caused by incorrect answers that appear similar to the correct answer but actually have different semantic expressions. Furthermore, in some embodiments, considering situations where respondents do not know how to answer, such as giving random answers or not answering at all in the corresponding answer area, the random or no-answer content can be added as incorrect reference answers to at least one target scoring example to further improve the accuracy of the scoring. For example, blank spaces without text information can be added as incorrect reference information to at least one target scoring example. The specific composition of the reference answers and reference scores in at least one target scoring example can be determined according to the question type and target question, etc., and is not specifically limited here.

[0042] It should be noted that the components of the target text can be directly input content, or images of each component can be obtained first through scanning or taking pictures, and then OCR (Optical Character Recognition) technology can be used to recognize the content of each component image. The specific method of obtaining the content of each component in the target text can be determined according to the actual situation, and no specific limitation is made here.

[0043] S120: Based on the reference text information in the target text, predict and fill in the scoring information to be filled in the target text to obtain the scoring result of the answer information to be scored.

[0044] In some embodiments, after obtaining the target text, some feature extraction algorithms can be used to extract features from the answer information to be scored and the reference answer information in at least one target scoring example, and then compare them to obtain the target scoring result corresponding to the answer information to be scored.

[0045] In other implementations, after acquiring the target text, a prediction model can be used to predict and fill in the rating information to be filled in the target text based on reference text information, thereby obtaining the rating result of the answer to be rated. Specifically, when the prediction model scores the answer to be rated in the target text, it can compare the answer to be rated with reference answer information in at least one target rating example, and predict the rating of the answer to be rated based on the comparison result with the reference answer and the reference rating corresponding to the reference answer, thereby obtaining the target rating result corresponding to the answer to be rated. For example, if the answer to be rated is the same as a reference answer in at least one target example, or the difference between the two is less than a preset threshold, it indicates that the two are the same or highly similar, and the reference rating corresponding to the reference answer can be used as the rating result of the answer to be rated. Here, the prediction model used to obtain the rating result is a prediction model trained on a neural network, specifically a natural language processing model, which can be, but is not limited to, a Transformer model or an LSTM (Long Short-Term Memory) model.

[0046] In this context, the score information to be filled in the target text is the part to be filled. In some embodiments, this part to be filled is performed by the prediction model; that is, the step of predicting and filling the score information to be filled in the target text based on reference text information in the target text is performed by the prediction model. In some embodiments, to help the prediction model know that the target information to be filled belongs to a part of the target text, i.e., to know what the information to be filled is, a mask label can be set for the part to be predicted and filled in the target text, so that the prediction model can identify the part to be predicted and filled in the target text by recognizing the mask label. For example, in some embodiments, a mask label can be set in advance for the score information part to be filled in the target text. When the prediction model obtains the input target text, it predicts and fills in the part with the mask label to obtain the corresponding score result. The mask label can be in the form of letters, symbols, graphics, etc., or it can be any combination of letters, symbols, graphics, etc. For example, the mask label is as follows: Figure 2 The [MASK] shown.

[0047] In this embodiment, the target text corresponding to the answer information to be scored is first obtained. This target text includes reference text information and the scoring information to be filled. Then, based on the reference text information, the scoring result of the answer information to be scored is obtained. This embodiment can directly predict and fill in the scoring information to be filled based on the reference text information to obtain the scoring result of the answer information to be scored. At the same time, the reference text information in this application contains at least one target scoring example, which can be used as a reference when predicting and filling in the scoring information to be filled. Therefore, the accuracy of the scoring result can be improved.

[0048] Please see Figure 3 , Figure 3 This is a partial flowchart illustrating an embodiment of the scoring method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 3 The illustrated process sequence is limited. In this embodiment, before step S120, which predicts and fills in the scoring information to be filled in the target text based on the reference text information to obtain the scoring result of the answer information to be scored, the following steps are also included:

[0049] S210: Obtain several sample texts, wherein the sample texts include sample questions, at least one sample scoring example, sample answer information to be scored, and sample scoring results, and at least one part of the sample question, sample scoring example, sample answer information, and sample scoring results of each sample text is a part to be filled.

[0050] This embodiment is used to train a prediction model using several sample texts to obtain a prediction model that can predict the part to be filled.

[0051] like Figure 4 As shown, the sample texts in this embodiment consist of four main parts: sample questions, at least one scoring example, the answer information to be scored, and sample scoring results. For a detailed discussion of these four components, please refer to the descriptions of the target question, at least one target scoring example, the answer information to be scored, and the scoring information to be filled in step S110. These details will not be repeated here. The sample questions in the acquired sample texts can be from any subject and of any question type. It should be noted that acquiring the sample texts in this step is for training the prediction model using these sample texts to obtain a trained prediction model capable of predicting and filling in the scoring information corresponding to the answer information to be scored in step S110. Furthermore, "several sample texts" refers to multiple sample texts; the required number of sample texts can be determined based on the training effect of the prediction model, and is not specifically limited here.

[0052] In this method, among the acquired sample texts, at least one part of the sample question, sample scoring example, sample answer information, and sample scoring result in each sample text is the part to be filled. That is, during the training of the prediction model, not only the sample scoring result part can be predicted and filled, but also the sample question, sample scoring example, and sample answer information parts. This allows the trained prediction model to predict and fill multiple parts of the input sample, resulting in better prediction performance. Of course, in some embodiments, the part to be filled in the sample text may only be the scoring result part, thus training only the prediction model to predict the scoring result; the specific part to be filled by the model can be determined based on the input sample text.

[0053] Specifically, please refer to Figure 5 , Figure 5 yes Figure 3 The flowchart shown is a schematic diagram of an embodiment of step S210. In this embodiment, before step S210, the following steps are also included:

[0054] S211: Obtain at least one set of question data, wherein each set of question data includes a question, at least one scoring example of the question, answer information of the question, and scoring results of the answer information.

[0055] This embodiment is used to compose original text using question data, and then to obtain sample text corresponding to each masking process by performing at least two masking processes on the original text.

[0056] Each set of question data includes at least one question, at least one example of a score for the question, the answer information for the question, and the score result for the answer information. The question data can be online or printed data; there is no specific limitation here.

[0057] S212: Use the data from each set of questions to form an original text.

[0058] If the question data is online, the original text can be directly assembled using preset computer program instructions. If the question data is printed data, an image of the question data needs to be obtained first by scanning or photographing, and then OCR technology is used to recognize the text from the image of the question data. The text of the question data is then assembled into the original text. The method of assembling the original text using the question data can be referred to the method of assembling the target text in step S110, which will not be elaborated here.

[0059] S213: For each original text, perform at least two masking processes on the original text to obtain sample text corresponding to each masking process. Each masking process masks different parts of the original text, including the question, scoring example, answer information, and scoring result. The masked parts are the parts to be filled.

[0060] Masking, as described in step S120, involves setting mask labels for the portions that need to be masked, for example, setting a mask label like... Figure 4 The [MASK] shown is used to cover the content in the original text that needs to be masked. The content to be masked and the proportion of the masked content to the total original text content can be determined according to the training purpose and the training strategy adopted. In this embodiment, for each original text, at least two masking processes can be performed on the original text to obtain sample text corresponding to each masking process. Each masking process masks different parts of the original text, including the question, scoring example, answer information, and scoring result. The masked part is the part to be filled.

[0061] In other words, in the sample text obtained by performing at least two masking processes on the original text, at least two parts contain mask labels. Specifically, the sample text corresponds to at least two parts of the original text: the question section, the scoring example section, the answer information section, and the scoring result section. Here, "at least two parts" can refer to two, three, or four parts, such as... Figure 4 As shown, the sample text contains masked labels for three parts: sample question, sample answer information, and sample scoring result. Sample text A contains a masked label for the sample question part, sample text B contains a masked label for the sample answer information part, and sample text C contains a masked label for the sample scoring result part. It should be noted that the sample texts obtained are for training a prediction model that can predict the parts to be filled, especially a prediction model that can predict the scoring result. Therefore, the sample texts used for model training must include sample text C, that is, they must include training samples with masked labels in the scoring result part. In other words, if at least two parts are two separate parts, then those two parts must contain training samples of the sample scoring result. For example, the training samples can be sample texts A and C, sample texts B and C, or sample texts C and the scoring example part with masked labels.

[0062] After obtaining several sample texts, step S220 is performed to train the prediction model using the sample texts to obtain a prediction model that can predict the part to be filled.

[0063] S220: Train the prediction model using several sample texts to obtain a prediction model that can predict the part to be filled.

[0064] Understandably, to train a predictive model that can predict ratings, sample texts containing the rating results are essential training samples. However, to enable the predictive model to obtain more accurate rating results, the model can be trained in a more diverse manner so that it can have better rating capabilities through various training methods.

[0065] In some embodiments, to enable the prediction model to achieve better prediction results, the parts to be filled by the model in at least two sample texts can be different, so that the model can have the ability to predict multiple parts of the sample text. For example... Figure 4 As shown, in sample text A, part of the sample question is to be filled in; in sample text B, part of the sample answer information is to be filled in; and in sample text C, part of the sample scoring results is to be filled in. Figure 4 As can be seen, sample texts A, B, and C all contain parts to be filled. Although the parts to be filled are different, when any of the sample texts A, B, and C is input into the prediction model, the prediction model can give a prediction result corresponding to the part to be filled in the input sample text. Therefore, the goal of training the prediction model with sample texts A, B, and C is the same, which is to obtain the prediction result of the part to be filled. Only the parts to be filled in the training samples are different.

[0066] Furthermore, in some embodiments, to enable the prediction model to obtain more accurate scoring results, a portion of the sample question, sample answer information, and sample scoring result of each sample text can be used as the part to be filled in by the model. That is, the prediction model can be trained using sample texts A, B, and C, such as... Figure 4As shown, in the process of training the prediction model using sample text A, the prediction model can predict the part to be filled in by using context and subject knowledge, so as to train the model's ability to understand the context, thereby enabling the model to score based on the answer information. During the training of the prediction model using sample text B, given a scoring result, the prediction model predicts the corresponding answer information. This training process helps the model understand the scoring logic. For example, the prediction model can learn that the closer the generated answer information is to a certain scoring example, the closer the scoring result is to the reference score in that scoring example. Therefore, using sample text B helps the model understand the scoring logic, thereby helping the model understand the relationship between the scoring result and the answer information. This allows the prediction model, trained using sample text B, to accurately score based on the answer information and the scoring example. Therefore, the goal of training the model using sample texts A, B, and C is unified: to obtain the prediction result of the part to be filled. At the same time, during the training process of the prediction model using sample texts A and B, the model can understand the context and the scoring logic. Therefore, the prediction model trained using sample texts A, B, and C can combine the context and the scoring logic to predict the score, resulting in a stronger predictive ability and more accurate prediction results.

[0067] It should be noted that a predictive model capable of predicting scoring results can be trained using only the sample text C, or the sample texts B and C, or the sample texts A and C. The specific type of sample used to train the predictive model can be determined based on the difficulty and effectiveness of model training, and no specific restrictions are made here.

[0068] Specifically, please refer to Figure 6 , Figure 6 yes Figure 3 The flowchart of one embodiment of step S220 is shown. In this embodiment, S220 further includes steps S221 and S222:

[0069] S221: Use the prediction model to predict and fill in the masked part of each sample text to obtain the prediction filling result.

[0070] When sample text is obtained, the prediction model can be used to predict and fill in the masked part of each sample text to obtain the prediction filling result corresponding to each sample text.

[0071] S222: Adjust the network parameters of the prediction model using the prediction difference for each sample text. The prediction difference is the difference between the predicted filling result and the original content of the masked part.

[0072] It is understandable that the training performance of a prediction model is closely related to its prediction variance. Prediction variance is the difference between the predicted imputation result and the original content of the masked portion. A smaller prediction variance indicates a better training performance. When training a prediction model using several sample texts, the prediction variance of each sample text can be used to adjust the model's network parameters. For example, if the training samples include sample texts A, B, and C, the prediction variance of each sample text can be calculated, and the network parameters can be adjusted using these variances. Alternatively, the prediction variance of only sample text C can be used to adjust the network parameters. The specific method used to adjust the network parameters depends on the ease of parameter adjustment and the training effect it brings; no specific limitations are imposed here.

[0073] Please see Figure 7 , Figure 7 This is a partial flowchart of an embodiment of the scoring method provided in this application. Before step S110, the following steps are also included:

[0074] S310: Use the standard answer to the target question as a correct reference answer to the target question, and use a pre-set strategy to obtain incorrect reference answers to the target question.

[0075] In this implementation, during the initial scoring of the responses to the target question, due to the lack of response data for the target question, the standard answer to the target question can be used as a correct reference answer in the scoring examples. A preset strategy is then employed to obtain incorrect reference answers for the target question. This preset strategy can use randomly generated strings or incorrect response information from other grading examples to obtain incorrect reference answers for the target question.

[0076] S320: Using the correct and incorrect reference answers to the target question, generate at least two target grading examples as target scoring examples.

[0077] Using the correct and incorrect reference answers to the target question obtained in step S310, at least two target grading examples are generated and used as target scoring examples. After obtaining the target text, the scoring information to be filled is predicted and filled based on the respondent's answer information and the target scoring examples to obtain the scoring result of the answer information to be scored.

[0078] In some embodiments, after predictively filling in the rating information to be filled in the target text based on reference text information in the target text, the rating method further includes: obtaining feedback information on the user's rating result for the answer to be rated, and updating the target rating example for the target question according to the feedback information. It is understood that, for example, in the early stages of rating the answer to the target question, due to the lack of grading examples, when comparing the answerer's information with the standard answer to obtain the rating result, there may be grading errors due to differences in semantic expression. For example, the answerer's answer itself may be incorrect, but the prediction model gives the corresponding rating result as a correct answer. Therefore, when feedback from the user regarding the error in the prediction model's rating result is obtained, the answerer's answer can be updated to the rating example for the incorrect answer in the target question according to the user's feedback information.

[0079] In other embodiments, after obtaining the scoring result of the information to be scored based on the reference text information in the target text, the obtained scoring result is updated to the scoring examples of the corresponding incorrect and correct answers. Preferably, the scoring result and the corresponding reference answer of the scoring example are updated to the corresponding scoring example if the difference between them is large.

[0080] Please see Figure 8 , Figure 8 This is a flowchart illustrating an embodiment of the scoring model training method provided in this application. In this embodiment, the scoring model is the aforementioned prediction model, and the scoring model training method includes:

[0081] S410: Obtain several sample texts, each sample text including a sample question, at least one sample rating example, sample answer information to be rated, and sample rating result. Each sample rating example contains a reference answer to the sample question and a reference rating for the reference answer. At least one part of the sample question, sample rating example, sample answer information, and sample rating result of each sample text is a part to be filled.

[0082] Each sample scoring example includes a reference answer to the sample question and a reference score for the reference answer. For details, please refer to the relevant description of at least one scoring example in step S110, which will not be repeated here.

[0083] S420: Train the prediction model using several sample texts to obtain a prediction model that can predict the part to be filled. The prediction model is used to predict and fill the part to be filled in the sample text.

[0084] For details regarding the sample text in step S410 and the training of the prediction model using several sample texts in step S420, please refer to the relevant descriptions of steps S210 and S220 respectively. Further details will not be elaborated here.

[0085] Please see Figure 9 , Figure 9 This is a schematic diagram of a framework of an embodiment of the scoring device provided in this application. In this embodiment, the scoring device 90 includes: an acquisition module 91 and a filling module 92. The acquisition module 91 is used to acquire the target text corresponding to the answer information to be scored. The answer information to be scored is obtained by answering a target question. The target text includes reference text information and scoring information to be filled. The reference text information includes at least one target scoring example of the target question and the answer information to be scored. Each target scoring example includes a reference answer to the target question and a reference score for the reference answer. The filling module 92 is used to predict and fill in the scoring information to be filled in the target text based on the reference text information in the target text to obtain the scoring result of the answer information to be scored.

[0086] In some embodiments, the at least one target score example obtained by the acquisition module 91 includes a target score example corresponding to a correct reference answer and a target score example corresponding to an incorrect reference answer; and / or, the target score result in the acquired target text is provided with a mask label, the mask label indicating that it needs to be filled, and the acquired reference information also includes the target question.

[0087] In some embodiments, the step of the filling module 92 predicting and filling the rating information to be filled in the target text based on the reference text information in the target text is performed by the prediction model.

[0088] In some embodiments, before the filling module 92 predicts and fills the rating information to be filled in the target text based on the reference text information in the target text, it further includes: acquiring several sample texts using the acquisition module 91, wherein the acquired sample texts include sample questions, at least one sample rating example, sample answer information to be rated, and sample rating results, and at least one part of the sample question, sample rating example, sample answer information, and sample rating results of each sample text is the part to be filled; and training the prediction model using the several sample texts to obtain a prediction model that can predict the part to be filled.

[0089] In some embodiments, the parts to be filled by the model in at least two sample texts acquired by the acquisition module 91 are different; and / or, one part of the sample question, sample answer information and sample score result of each sample text is the part to be filled by the model to be predicted.

[0090] In some embodiments, the acquisition module 91 acquires several sample texts, including: acquiring at least one set of question data, wherein each set of question data includes a question, at least one scoring example of the question, answer information of the question, and scoring result of the answer information; using each set of question data to form an original text; for each original text, performing at least two masking processes on the original text to obtain sample text corresponding to each masking process, wherein each masking process masks different parts of the question, scoring example, answer information, and scoring result of the original text, and the masked parts are the parts to be filled.

[0091] In some embodiments, the filling module 92 trains the prediction model using several sample texts, including: using the prediction model to predict and fill the masked portion of each sample text to obtain a prediction filling result; and using the prediction difference of each sample text to adjust the network parameters of the prediction model, wherein the prediction difference is the difference between the prediction filling result and the original content of the masked portion.

[0092] In some embodiments, before acquiring the target text using the acquisition module 91, the method further includes: using the standard answer of the target question as the correct reference answer of the target question, and using a preset strategy to obtain the incorrect reference answer of the target question; using the correct reference answer and the incorrect reference answer of the target question to generate at least two target grading examples as target scoring examples.

[0093] In some embodiments, after the filling module 92 predicts and fills the rating information to be filled in the target text based on the reference text information in the target text, it further includes obtaining feedback information on the user's rating result of the answer information to be rated obtained by the prediction model; and updating the target rating example of the target question according to the feedback information.

[0094] Please see Figure 10 , Figure 10 This is a schematic diagram of the framework of an embodiment of the training device for the scoring model provided in this application. In this embodiment, the training device 100 for the scoring model includes a sample acquisition module 101 and a sample filling module 102. The sample acquisition module 101 is used to acquire a plurality of sample texts, which include sample questions, at least one sample scoring example, sample answer information to be scored, and sample scoring results. Each sample scoring example includes a reference answer to the sample question and a reference score for the reference answer. At least one part of the sample question, sample scoring example, sample answer information, and sample scoring result of each sample text is a part to be filled by the model. The sample filling module 102 is used to train the prediction model using the plurality of sample texts, wherein the prediction model is used to predict and fill in the parts to be filled in the sample texts.

[0095] Please seeFigure 11 , Figure 11 This is a schematic diagram of a framework of an embodiment of the scoring device provided in this application. In this embodiment, the scoring device 110 includes a memory 111 and a processor 112.

[0096] Processor 112 can also be referred to as CPU (Central Processing Unit). Processor 112 may be an integrated circuit chip with signal processing capabilities. Processor 112 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 112 can be any conventional processor 112, etc.

[0097] The memory 111 in the scoring device 110 is used to store the program instructions required for the processor 112 to run.

[0098] The processor 112 is used to execute program instructions to implement the methods provided in any embodiment of the scoring method and the scoring model training method described above, as well as any non-conflicting combination thereof.

[0099] Please see Figure 12 , Figure 12 This is a schematic diagram of the framework of the computer-readable storage medium provided in this application. The computer-readable storage medium 120 of this application embodiment stores program instructions 121, which, when executed, implement the methods provided in any of the above embodiments and any non-conflicting combinations. The program instructions 121 can form a program file and be stored in the computer-readable storage medium 120 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 120 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0100] The aforementioned scheme first obtains the target text corresponding to the answer to be scored. This target text contains reference text information and the scoring information to be filled. Then, based on the reference text information, the scoring result of the answer to be scored is obtained. The scheme of this application can directly predict and fill in the scoring information to be filled based on the reference text information to obtain the scoring result of the answer to be scored. Furthermore, the reference text information in this application contains at least one target scoring example, which can be used as a reference when predicting and filling in the scoring information to be filled, thus improving the accuracy of the scoring result.

[0101] Optionally, to further improve the accuracy of the prediction model's score prediction, the model is trained using three sample texts: sample text A, sample text B, and sample text C. The parts to be filled in by the model differ among these three sample texts, but the training objective is the same: to train the model to obtain the prediction results for the parts to be filled. Furthermore, sample texts A and B, during the training process, help the model understand the context and the scoring logic. Therefore, the model trained using sample texts A, B, and C will provide more accurate score predictions.

[0102] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0103] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A scoring method, characterized in that, The method includes: Obtain the target text corresponding to the answer to be scored, wherein the answer to be scored is obtained by answering the target question, and the target text includes reference text information and scoring information to be filled in. The reference text information includes at least one target scoring example of the target question and the answer to be scored; the at least one target scoring example includes target scoring examples corresponding to correct reference answers and target scoring examples corresponding to incorrect reference answers, wherein each target scoring example contains a reference answer to the target question and a reference score for the reference answer; the incorrect reference answers include reference answers that are similar to the correct reference answer text but have different semantics. Based on the reference text information in the target text, the scoring information to be filled in the target text is predicted and filled in to obtain the scoring result of the answer information to be scored. The method further includes, after predicting and filling in the rating information to be filled in the target text based on the reference text information in the target text, the method further includes: Obtain feedback information on the user's rating of the answer to be rated; Based on the feedback information, update the target score example for the target question.

2. The method according to claim 1, characterized in that, The rating information to be filled in the target text is masked, and the mask indicates that it needs to be filled. And / or, the reference text information may also include the target title.

3. The method according to claim 1, characterized in that, The step of predicting and filling in the rating information to be filled in the target text based on the reference text information in the target text is performed by the prediction model.

4. The method according to claim 3, characterized in that, Before predicting and filling the rating information to be filled in the target text based on the reference text information in the target text, the method further includes: Obtain several sample texts, each sample text including a sample question, at least one sample scoring example, sample answer information to be scored, and sample scoring result, wherein at least one part of the sample question, sample scoring example, sample answer information, and sample scoring result of each sample text is a part to be filled; The prediction model is trained using the aforementioned sample texts to obtain a prediction model capable of predicting the part to be filled.

5. The method according to claim 4, characterized in that, At least two of the sample texts have different parts to be filled by the model; And / or, a portion of the sample question, sample answer information, and sample score result of each sample text is the portion to be filled by the model to be predicted.

6. The method according to claim 4, characterized in that, The acquisition of several sample texts includes: Obtain at least one set of question data, wherein each set of question data includes a question, at least one scoring example of the question, answer information of the question, and scoring results of the answer information; Use the data from each set of questions to create an original text; For each of the original texts, at least two masking processes are performed on the original texts to obtain the sample texts corresponding to each masking process. Each masking process is to mask different parts of the question, scoring example, answer information and scoring result of the original text. The parts that have been masked are the parts to be filled.

7. The method according to claim 6, characterized in that, The step of training the prediction model using the aforementioned sample texts to obtain a prediction model capable of predicting the part to be filled includes: The prediction model is used to predict and fill in the masked portion of each sample text to obtain the prediction filling result. The network parameters of the prediction model are adjusted using the prediction difference for each sample text, where the prediction difference is the difference between the prediction fill result and the original content of the portion processed by the mask.

8. The method according to claim 1, characterized in that, Before obtaining the target text corresponding to the answer to be scored, the method further includes: The standard answer to the target question is used as a correct reference answer to the target question, and a preset strategy is used to obtain incorrect reference answers to the target question. Using the correct and incorrect reference answers to the target question, at least two target grading examples are generated as target scoring examples.

9. A method for training a scoring model, characterized in that, The method includes: Obtain several sample texts, each sample text including a sample question, at least one sample rating example, sample answer information to be rated, and sample rating result. Each sample rating example includes a reference answer to the sample question and a reference rating for the reference answer. At least one part of the sample question, sample rating example, sample answer information, and sample rating result in each sample text is a part to be filled in. The at least one sample rating example includes a sample rating example corresponding to a correct reference answer and a sample rating example corresponding to an incorrect reference answer. The prediction model is trained using the aforementioned sample texts to obtain a prediction model capable of predicting the part to be filled. The prediction model is used to predict and fill in the part of the sample text to be filled. The trained prediction model, after obtaining the target text corresponding to the answer to be scored, predicts and fills in the scoring information to be filled in the target text based on reference text information in the target text, thus obtaining the scoring result of the answer to be scored. The answer to be scored is obtained by answering a target question. The target text includes reference text information and the scoring information to be filled. The reference text information includes at least one target scoring example of the target question and the answer to be scored. The at least one target scoring example includes target scoring examples corresponding to correct reference answers and target scoring examples corresponding to incorrect reference answers. Each target scoring example contains a reference answer to the target question and a reference score for that reference answer. Incorrect reference answers include reference answers that are similar to the correct reference answer text but semantically different.

10. A scoring device, characterized in that, The device includes: The acquisition module is used to acquire the target text corresponding to the answer information to be scored. The answer information to be scored is obtained by answering a target question. The target text includes reference text information and scoring information to be filled in. The reference text information includes at least one target scoring example of the target question and the answer information to be scored. The at least one target scoring example includes target scoring examples corresponding to correct reference answers and target scoring examples corresponding to incorrect reference answers. Each target scoring example contains a reference answer to the target question and a reference score for the reference answer. The incorrect reference answers include reference answers that are similar to the correct reference answer text but have different semantics. The filling module is used to predict and fill the scoring information to be filled in the target text based on the reference text information in the target text, so as to obtain the scoring result of the answer information to be scored; The device is further configured to, after the filling module predicts and fills the rating information to be filled in the target text based on the reference text information in the target text, obtain feedback information on the user's rating result of the answer to be rated; and update the target rating example of the target question according to the feedback information.

11. A scoring model training device, characterized in that, The device includes: The sample acquisition module is used to acquire several sample texts, each sample text including a sample question, at least one sample rating example, sample answer information to be rated, and sample rating result. Each sample rating example includes a reference answer to the sample question and a reference rating for the reference answer. At least one part of the sample question, sample rating example, sample answer information, and sample rating result in each sample text is a part to be filled. The at least one sample rating example includes a sample rating example corresponding to a correct reference answer and a sample rating example corresponding to an incorrect reference answer. A sample filling module is used to train a prediction model using the plurality of sample texts to obtain a prediction model capable of predicting the part to be filled. The prediction model is used to predict and fill in the part of the sample text to be filled. The trained prediction model, after obtaining the target text corresponding to the answer to be scored, predicts and fills in the scoring information to be filled in the target text based on reference text information in the target text, thereby obtaining the scoring result of the answer to be scored. The answer to be scored is obtained by answering a target question. The target text includes reference text information and the scoring information to be filled. The reference text information includes at least one target scoring example of the target question and the answer to be scored. The at least one target scoring example includes target scoring examples corresponding to correct reference answers and target scoring examples corresponding to incorrect reference answers. Each target scoring example contains a reference answer to the target question and a reference score for that reference answer. Incorrect reference answers include reference answers that are similar to the correct reference answer text but semantically different.

12. A scoring device, characterized in that, Including interconnected memory and processor, The memory stores program instructions; The processor is configured to execute program instructions stored in the memory to implement the method of any one of claims 1-8, or to implement the method of claim 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the method of any one of claims 1-8, or to implement the method of claim 9.

Images