Review method, review device, electronic device, and computer-readable storage medium

Abstract

The present application discloses a review method, a review device, an electronic device, and a computer-readable storage medium. The review method comprises: obtaining a mathematical text to be scored; wherein the mathematical text comprises a standard answer and a user's answer content, and the mathematical text comprises a mathematical formula and text information; wherein the mathematical text is a text after word segmentation processing; inputting the mathematical text into a scoring prediction model, and using the scoring prediction model to recognize the text information and mathematical formula in the mathematical text; wherein the scoring prediction model is obtained by training with an incomplete mathematical text; wherein the incomplete mathematical text comprises a masked mathematical formula and masked text information; and scoring the user's answer content based on the recognition result and outputting the score rate. The present application can make predictions based on the user's answer content well, thereby meeting the demand for automatic review of answer questions.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a review method, a review device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of Internet technology, the number of online education users is increasing day by day, and a large amount of manpower is needed to cope with large-scale question review. Among them, for multiple-choice questions, efficient review can be achieved through simple answer matching. However, due to the subjectivity of the question-solving process and the diversity of solutions, different teachers also have a certain degree of subjectivity in the review results, which is not conducive to students' objective evaluation of themselves and also increases the workload of teachers.

[0003] Existing automatic math problem grading methods mainly calculate scores based on the degree of match between the standard answer and the student's answer. For example, math algebraic type questions are graded by substituting the true value into the student's answer and matching it with the standard answer. Alternatively, a large number of manually defined rules are used to structure the student's answer and extract the conclusion, and finally the student's score is calculated based on the degree of match between the conclusion.

[0004] However, the existing scoring prediction models cannot accurately understand the rules of mathematical statements in mathematical texts, so they need to manually structure and extract conclusions from standard answers and student answers in advance, which will still greatly increase labor costs and cannot meet the needs of automatic review of answer questions. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a review method, a review device, an electronic device and a computer-readable storage medium, which can solve the problem of high labor costs caused by reviewing answers in the prior art.

[0006] To solve the above technical problems, the first technical solution adopted in the present application is to provide a review method, including: obtaining a mathematical text to be scored; wherein the mathematical text includes a standard answer and a user's answer content, and the mathematical text includes mathematical formulas and text information; wherein the mathematical text is a text after word segmentation processing; inputting the mathematical text into a scoring prediction model, and using the scoring prediction model to recognize the text information and mathematical formulas in the mathematical text; wherein the scoring prediction model is trained using an incomplete mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information; scoring the user's answer content based on the recognition result and outputting the score rate.

[0007] Among them, the step of obtaining mathematical texts to be scored specifically includes: obtaining multiple mathematical texts to be scored; inputting the mathematical texts into the scoring prediction model, and using the scoring prediction model to identify the text information and mathematical formulas in the mathematical texts, including: inputting each mathematical text into the scoring prediction model, and using the scoring prediction model to identify the text information and mathematical formulas in each mathematical text to obtain multiple recognition results; the step of scoring the user's answer content based on the recognition results and outputting the score rate includes: scoring all multiple recognition results through the scoring prediction model, outputting multiple score rates, and obtaining the mean of the multiple score rates through the scoring prediction model; performing Gaussian distribution fitting on the mean of the multiple score rates and the score rate corresponding to each mathematical text through the scoring prediction model, and judging whether to refuse to review each mathematical text based on the comparison result of each variance obtained with the second set threshold; wherein, in response to the variance of a single mathematical text being greater than the second set threshold, refusing to score the single mathematical text.

[0008] Among them, the method of training using incomplete mathematical text specifically includes: obtaining a first sample data set, each first sample data is an incomplete mathematical text; using the first sample data set to perform mathematical formula and text prediction training on a preset language model to obtain a first model; obtaining a second sample data set, each second sample data includes a question, a standard answer, a user's answer content, and evaluation information of the user's answer content; and the second sample data includes mathematical formulas and text information; using the second sample data set to perform scoring training on the first model to obtain a scoring prediction model.

[0009] Among them, the step of obtaining a first sample data set, each first sample data is an incomplete mathematical text, including: obtaining an original sample set, wherein each original sample includes a mathematical expression and text information; performing word segmentation processing on the mathematical expression and text information in each original sample to divide the mathematical expression and text information into multiple word segments; partially masking the word segments corresponding to the mathematical expression and the word segmentation corresponding to the text information to obtain the first sample data set.

[0010] Among them, the steps of using the first sample data set to perform mathematical formula and text prediction training on the preset language model to obtain the first model include: constructing the mathematical formula in the original sample to generate a formula parse tree; traversing the formula parse tree to obtain the position information of each node of the formula parse tree in the original sample, and storing the formula parse tree and the corresponding position information; using the first sample data and the position information to perform mathematical formula and text prediction training on the preset language model to obtain the first model.

[0011] Among them, the step of using the first sample data and position information to perform mathematical formula and text prediction training on the preset language model to obtain the first model includes: predicting the content and position of the masked mathematical formula and masked text information in the first sample data through the preset language model to obtain the first prediction information; using the first prediction information, position information, and each first sample data to perform mathematical formula and text prediction training on the preset language model to obtain the first model.

[0012] Among them, the step of constructing the mathematical formula in the original sample to generate a formula parse tree includes: obtaining at least one variable and at least one operator in the word segment corresponding to the mathematical formula; adding the variable and the operator to the first stack and the second stack respectively, and converting the mathematical formula into reverse Polish notation according to the priority of the operator to generate a formula parse tree based on the reverse Polish notation.

[0013] Among them, the preset language model includes an input layer, a convolutional downsampling layer, a bidirectional encoder, a fully connected layer and an output layer in sequence; the step of predicting the content and position of the masked mathematical formula and the masked text information in the first sample data through the preset language model to obtain the first prediction information includes: inputting the first sample data into the input layer for processing to obtain the word vector, position vector and paragraph vector corresponding to the word segmentation; inputting the word vector, position vector and paragraph vector into the convolutional downsampling layer for feature extraction to obtain the feature vector; inputting the feature vector into the bidirectional encoder for multi-dimensional information extraction, and inputting the extracted information into the fully connected layer for content and position prediction to obtain the first prediction information; using the first prediction information, position information, and each first sample data to perform mathematical formula and text prediction training on the preset language model to obtain the first model, specifically including: judging whether the first prediction information matches the content and position of the masked mathematical formula and the masked text information based on the position information and the first sample data, and adjusting the model parameters of the preset language model based on the matching result to obtain the first model.

[0014] Among them, the step of using the second sample data set to score train the first model to obtain a scoring prediction model includes: predicting the user's answer content and the standard answer through the first model to obtain second prediction information; using the second prediction information and the judgment information to score predict the first model to obtain a scoring prediction model.

[0015] Among them, the step of predicting the user's answer content and the standard answer through the first model to obtain the second prediction information includes: inputting the questions, standard answers and user's answer content in the second sample data into the first model, so that the first model makes a prediction based on the degree of match between the user's answer content and the standard answer to obtain the second prediction information; using the second prediction information and the judgment information to train the first model for score prediction to obtain the score prediction model, includes: comparing the second prediction information with the judgment information, and adjusting the model parameters of the first model based on the comparison results to obtain the score prediction model.

[0016] Among them, the step of predicting the user's answer content and the standard answer through the first model to obtain the second prediction information includes: inputting the questions, standard answers, user answer content and evaluation information of the user answer content in each second sample data into the first model, so that the first model makes predictions based on the degree of matching between the user's answer content and the standard answer, and obtains all the second prediction information of each second sample data; using the second prediction information and the evaluation information to train the first model for score prediction, and the step of obtaining the score prediction model includes: obtaining the mean of all the second prediction information through the first model; fitting the mean and the evaluation information of the user's answer content corresponding to each second sample data with a Gaussian distribution through the first model, and adjusting the model parameters of the first model based on the comparison results of the obtained variances with the first set threshold value, so as to obtain the score prediction model.

[0017] In order to solve the above technical problems, the second technical solution adopted in the present application is to provide a review device, including: an acquisition module, used to acquire a mathematical text to be scored; wherein the mathematical text includes a standard answer and a user's answer content, and the mathematical text includes mathematical formulas and text information; wherein the mathematical text is a text after word segmentation processing; a recognition module, used to input the mathematical text into a scoring prediction model, and use the scoring prediction model to recognize the text information and mathematical formulas in the mathematical text; wherein the scoring prediction model is trained using an incomplete mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information; a scoring module, used to score the user's answer content based on the recognition result and output the score rate.

[0018] In order to solve the above technical problems, the third technical solution adopted in the present application is to provide an electronic device, including: a memory, used to store program data, and the program data implements the steps in the review method as mentioned above when executed; a processor, used to execute the program data stored in the memory to implement the steps in the review method as mentioned above.

[0019] In order to solve the above technical problems, the fourth technical solution adopted in the present application is to provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the review method as described above are implemented.

[0020] The beneficial effects of the present application are as follows: Different from the prior art, the present application provides a review method, a review device, an electronic device and a computer-readable storage medium, which recognize and score data texts including mathematical formulas and text information through a scoring prediction model, and the scoring prediction model is obtained through training of incomplete mathematical texts, has a certain understanding ability for mathematical texts containing mathematical formulas, and can better make autonomous predictions based on the content of user answers, thereby reducing labor costs and meeting the needs of automatic review of answer questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flowchart of the first implementation method of the review method of this application;

[0023] Figure 2 This is the method for obtaining the score prediction model in this application;

[0024] Figure 3 yes Figure 2 S21 is a flow chart of a specific implementation method;

[0025] Figure 4 yes Figure 2 S22 is a flow chart of a specific implementation method;

[0026] Figure 5 It is a structural diagram of a formula analysis tree corresponding to a digital formula of the present application;

[0027] Figure 6 yes Figure 5 Schematic diagram of the structure of the formula parse tree after some nodes are covered;

[0028] Figure 7 This is a structural diagram of an implementation method of a preset language model of the present application;

[0029] Figure 8 yes Figure 2 A schematic diagram of a process of a first specific implementation method of S24;

[0030] Fig. 9 yes Figure 2 A schematic flow chart of a second specific implementation method of S24;

[0031] Fig.10 It is a flow chart of the second implementation method of the review method of this application;

[0032] Fig.11 It is a structural schematic diagram of an implementation method of the review device of the present application;

[0033] Fig.12 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0034] Fig.13 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless otherwise clearly indicated above, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0037] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0038] It should be understood that the terms "include", "comprises" or any other variations used herein are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0039] See also Figure 1 , Figure 1 Schematic diagram of the first embodiment of the review method of this application. Figure 1 As shown, in this embodiment, the method includes:

[0040] S11: Obtain the mathematics text to be graded; wherein the mathematics text includes the standard answer and the user's answer content, and the mathematics text includes mathematical formulas and text information; wherein the mathematics text is the text after word segmentation processing.

[0041] Among them, the standard answer is the full-mark solution process corresponding to the question, and the user's answer content is the student's answer content.

[0042] Among them, mathematical formula refers to the general term for using mathematical language and mathematical symbols to express a certain relationship, a certain operation or a certain property. It includes both formulas and other equations expressed in algebraic expressions about a certain conclusion, and a group of algebraic expressions connected by mathematical symbols.

[0043] In this implementation, the mathematics text is a text token (segmentation) sequence obtained by concatenating three parts: the question obtained after word segmentation processing, the standard answer, and the user's answer content.

[0044] Specifically, during the word segmentation process, mathematics-related dictionaries can be used to maintain the integrity of mathematical terms as much as possible. For example, the text information includes a text "parallelogram", and this small text corresponds to 5 characters, but because "parallelogram" is a separate mathematical term, this text is directly processed into a token during the word segmentation process. For another example, when a mathematical formula is subjected to word segmentation, each operator and variable in the mathematical formula is processed as a token. Here, a specific embodiment is used to illustrate that the mathematical formula (-a+b)c>10, after the word segmentation process, includes a total of 8 tokens, namely "-", "a", "+", "b", "*", "c", ">" and "10".

[0045] In this embodiment, when performing word segmentation, mathematical terms must also be standardized to facilitate subsequent model training. For example, the multiplication sign in the LaTeX formula and the names from different encoding systems are constrained to be unified symbols. Among them, LaTeX is a typesetting system suitable for typesetting large papers and inputting mathematical formulas.

[0046] S12: Input the mathematical text into a scoring prediction model, and use the scoring prediction model to identify text information and mathematical formulas in the mathematical text; wherein the scoring prediction model is trained using incomplete mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information.

[0047] In this embodiment, the score prediction model identifies the standard answer and the user's answer content based on the input text token sequence.

[0048] Specifically, see Figure 2 , Figure 2 This is the method for obtaining the score prediction model in this application. Figure 2 As shown, in this embodiment, the method of using incomplete mathematical text for training specifically includes:

[0049] S21: Acquire a first sample data set, each first sample data is an incomplete mathematical text.

[0050] See also Figure 3 , Figure 3 yes Figure 2 S21 is a flow chart of a specific implementation method. Figure 3 As shown, in this embodiment, the step of obtaining the first sample data set, each of which is an incomplete mathematical text, specifically includes:

[0051] S211: Acquire an original sample set, wherein each original sample includes a mathematical formula and text information.

[0052] In this implementation, the original sample is a mathematics text including a question, a standard answer, and a user's answer content.

[0053] In this implementation, the original sample set includes a massive amount of original samples.

[0054] Specifically, a large number of original samples can be efficiently obtained in the following two ways: First, a large number of math problem solving questions and the corresponding full-mark solution process information and user answer content can be obtained from the Internet through web crawling. Second, pictures of math problem solving questions and answers can be obtained by taking photos, scanning math exercise books and math test papers, and then the pictures can be converted into math texts including math formulas and text information through OCR (Optical Character Recognition) technology.

[0055] S212: Perform word segmentation processing on the mathematical expressions and text information in each original sample to divide the mathematical expressions and text information into a plurality of word segments.

[0056] In this embodiment, after a large amount of original samples are obtained, the mathematical expressions and text information in the original samples are segmented to divide the mathematical expressions and text information into multiple tokens.

[0057] S213: Partially mask the segmented words corresponding to the mathematical formula and the segmented words corresponding to the text information to obtain a first sample data set.

[0058] In this implementation, the first sample data set is used to pre-train a preset language model to obtain a language model that can understand mathematical texts.

[0059] Specifically, pre-training needs to achieve two training goals. The first is to enable the language model to predict the specific content of the masked token based on the context of the masked token in the text information, that is, to complete the training goal of the masked language model. The second is to enable the language model to predict the position of the masked parent and child nodes in the sequence of the masked token in the mathematical formula, that is, to be able to predict the substructure of the mathematical formula, so it is necessary to partially mask the word segmentation corresponding to the mathematical formula and the word segmentation corresponding to the text information.

[0060] In this embodiment, the tokens corresponding to the set proportion of text information and mathematical formulas in the original sample are masked. In a preferred embodiment, the set proportion is 15%, and 15% of the tokens corresponding to the text information and 15% of the tokens corresponding to the mathematical formulas are masked. In other embodiments, the set proportion can also be 10%, 20% or other proportions, which is not limited in this application.

[0061] S22: Using the first sample data set to perform mathematical formula and text prediction training on a preset language model to obtain a first model.

[0062] See also Figure 4 , Figure 4 yes Figure 2 S22 is a flow chart of a specific implementation method. Figure 4 As shown, in this embodiment, the steps of using the first sample data set to perform mathematical formula and text prediction training on the preset language model to obtain the first model specifically include:

[0063] S221: Construct the mathematical formula in the original sample to generate a formula parsing tree.

[0064] In this embodiment, at least one variable and at least one operator in the word segment corresponding to the mathematical expression are obtained, the variables and the operators are added to the first stack and the second stack respectively, and the mathematical expression is converted into reverse Polish notation according to the priority of the operator, so as to generate a formula parse tree based on the reverse Polish notation. The formula in the formula parse tree refers not only to the formula, but also to a general mathematical expression.

[0065] Among them, the reverse Polish notation is also called the suffix expression, which means writing the operand in front and the operator in the back.

[0066] Among them, variables are alphabetic characters representing numbers, and operators can be roughly divided into five types: arithmetic operators, connection operators, relational operators, assignment operators, and logical operators. Operators with higher priority are operators that are operated first and are child nodes; operators with lower priority are operators that are operated later and are parent nodes.

[0067] Here we continue to use the mathematical formula (-a+b)c>10 for explanation. In this mathematical formula, the variables are "a", "b" and "c", and the operators include "-", "+", "*" and ">", where "-", "+" and "*" are arithmetic operators, and ">" is a relational operator. In this mathematical formula, "-" and "+" are the operators that are operated first and are child nodes, and "*" is the operator that is operated later and is the parent node.

[0068] Specifically, see Figure 5 , Figure 5 It is a schematic diagram of the structure of the formula analysis tree corresponding to a digital formula of the present application. Figure 5 As shown in the figure, each circle in the formula parsing tree and the variables or operators it contains are a token.

[0069] S222: Traverse the formula parsing tree to obtain the position information of each node of the formula parsing tree in the original sample, and store the formula parsing tree and the corresponding position information.

[0070] In this embodiment, after obtaining the formula parse tree corresponding to each mathematical expression in the original sample, the formula parse tree is traversed in order to calculate the position information of the parent node or child node corresponding to each token in the mathematical expression in the original sequence of the original sample, and the formula parse tree and the corresponding position information are stored.

[0071] Among them, in-order traversal (LDR) is a type of binary tree traversal, also known as middle root traversal or in-order tour. In a binary tree, in-order traversal first traverses the left subtree, then visits the root node, and finally traverses the right subtree.

[0072] S223: Perform mathematical formula and text prediction training on a preset language model using the first sample data and the position information to obtain a first model.

[0073] In this embodiment, a preset language model is used to predict the content and position of the masked mathematical formula and the masked text information in the first sample data. After obtaining the first prediction information, the first prediction information, the position information, and each first sample data are used to perform mathematical formula and text prediction training on the preset language model to obtain the first model.

[0074] It can be understood that the main function of the position information is to be used for supervised learning and is not input into the preset language model. After the preset language model predicts the content and position of the masked mathematical formula and the masked text information in the first sample data and obtains the first prediction information, the position information is used to prompt the preset language model so that the preset language model knows whether the prediction result is accurate, thereby deciding whether to adjust the parameters of the preset language model.

[0075] Specifically, see Figure 6 , Figure 6 yes Figure 5 The schematic diagram of the structure of the formula parse tree after some nodes are covered. Figure 6 As shown, token "a" and token "b" are child nodes that need to be calculated first, and the masked token "+" is an operator that needs to be added. After the addition operation is completed, the multiplication operation can be performed through the token "*". Because the token "+" is masked, the preset language model needs to learn how to determine the position of the parent node or child node corresponding to the masked token "+" in the original sequence during training, and determine the specific content of the token. After the preset language model generates the first prediction information, the stored position information is used to prompt the preset language model so that the preset language model knows whether the content and position of this prediction are accurate, thereby deciding whether to adjust the parameters of the preset language model.

[0076] In the prior art, when mathematical text is segmented, the mathematical formula and text information are separated into two parts for processing, and the mathematical formula is attached to the end of the text information in the sample data. The model cannot accurately embed the mathematical formula into the text information during processing, and its ability to understand the mathematical text is relatively weak.

[0077] Different from the prior art, this embodiment does not change the position of the mathematical formula in the original sequence when performing word segmentation processing. Instead, it performs structural analysis on the mathematical formula by constructing a formula parsing tree and in-order traversal, and calculates the position information of the parent node or child node corresponding to each token in the mathematical formula corresponding to each formula parsing number in the original input sequence, so that the preset language model can predict the substructure of the mathematical formula embedded in the text information, thereby deepening the ability to understand mathematical texts.

[0078] In this embodiment, the preset language model is a model based on BERT (Bidirectional Encoder Representations from Transformers). In order to clearly explain the structure and training method of the preset language model, as shown in FIG. Figure 7 As shown, Figure 7 It is a structural diagram of an implementation method of a preset language model of the present application.

[0079] In this implementation, the preset language model 20 includes an input layer 21, a convolutional downsampling layer 22, a bidirectional encoder 23, a fully connected layer 24 and an output layer 25 in sequence.

[0080] The bidirectional encoder 23 is composed of 12 layers of Transformer. Specifically, the more layers of the preset language model 20, the better the effect index, so this embodiment selects a 12-layer Transformer as the bidirectional encoder 23 according to the effect index. In other embodiments, different layers of Transformer can be selected according to different requirements, such as a 24-layer Transformer, etc., which is not limited in this application.

[0081] In this embodiment, the first sample data is input into the input layer 21 for processing to obtain the word vector, position vector and paragraph vector corresponding to the word segmentation. Specifically, the text token sequence obtained by splicing the three parts of the question, the standard answer and the user's answer content obtained after the word segmentation of the first sample data is used as the input of the input layer 21 to map each token in the sequence to the corresponding word vector, position vector and paragraph vector.

[0082] In this embodiment, in order to reduce the amount of calculation of the subsequent bidirectional encoder 23, the word vector, position vector and paragraph vector are input into the convolution downsampling layer 22 for feature extraction to obtain a feature vector. Specifically, by downsampling the input sequence through a multi-layer convolutional neural network, the length of the input sequence can be shortened while alleviating information loss, thereby greatly reducing the subsequent computational complexity. In other embodiments, the input sequence can also be downsampled by structures such as maximum pooling or average pooling, which is not limited in this application.

[0083] The feature vector is input into the bidirectional encoder 23 for multi-dimensional information extraction, and the extracted information is input into the fully connected layer 24 for content and position prediction to obtain the first prediction information. Specifically. The bidirectional encoder 23 uses the feature vector (the result of downsampling) as input, and extracts multi-dimensional information (high-level information extraction) from the feature vector through a multi-head attention mechanism, which can broaden the richness of information to further deepen the preset language model's ability to understand mathematical texts. The output layer 25 calculates the output (first prediction information) of the preset language model 20 based on the high-level information provided by the fully connected layer 24.

[0084] Furthermore, based on the position information and the first sample data, it is determined whether the first prediction information matches the content and position of the masked mathematical formula and the masked text information, and based on the matching result, the model parameters of the preset language model 20 are adjusted to obtain the first model.

[0085] It can be understood that the main function of the position information is to supervise the learning of the preset language model 20, and it is not input into the preset language model 20. After the preset language model 20 predicts the content and position of the masked mathematical formula and the masked text information in the first sample data and obtains the first prediction information, the position information is used to prompt the preset language model 20 so that the preset language model 20 knows whether the prediction result is accurate, thereby deciding whether to adjust the parameters of the preset language model 20.

[0086] Through the above-mentioned training of the preset language model 20, not only can the first model finally obtained learn to understand the basic grammatical rules of mathematical texts, but also the first model can pay attention to the role of each component (token) in the mathematical formula, so that the first model can understand the content of the mathematical text more accurately.

[0087] S23: Acquire a second sample data set, each second sample data includes a question, a standard answer, a user's answer content, and evaluation information of the user's answer content; and the second sample data includes mathematical formulas and text information.

[0088] In this embodiment, the second sample data may be sample data consisting of each original sample plus the evaluation information of the corresponding user answer content, or may be sample data carrying the evaluation information of the user answer content obtained by re-crawling the network or taking photos.

[0089] Among them, the evaluation information of the user's answer content, namely the score label, is the real score rate (the ratio of the real score to the full score) obtained by reviewing the user's answer content based on the standard answer.

[0090] S24: Using the second sample data set to perform scoring training on the first model to obtain a scoring prediction model.

[0091] In this embodiment, the first model is used to predict the user's answer content and the standard answer to obtain second prediction information, and the second prediction information and the judgment information are used to perform score prediction training on the first model to obtain a score prediction model.

[0092] Among them, the training goal of the first model can be to predict the corresponding score rate based only on information such as the input question, standard answer, and user answer content, or it can be to predict the Gaussian distribution to which each second sample data belongs based on the input question, standard answer, user answer content, and evaluation information of the user answer content.

[0093] Specifically, see Figure 8 , Figure 8 yes Figure 2 The schematic diagram of the process of the first specific implementation method of S24 in FIG. Figure 8 As shown, in this embodiment, the training goal of the first model is to predict the corresponding score rate based only on the input question, the standard answer, and the user's answer content. The method specifically includes:

[0094] S2411: Input the questions, standard answers and user answers in the second sample data into the first model, so that the first model makes predictions based on the degree of matching between the user answers and the standard answers, and obtains second prediction information.

[0095] In this implementation, the second prediction information is the predicted score rate obtained by the first model based on the matching degree between the user's answer content and the standard answer.

[0096] In this implementation, the data output by the output layer of the first model only includes the second prediction information.

[0097] S2412: Compare the second prediction information with the evaluation information, and adjust the model parameters of the first model based on the comparison result to obtain a score prediction model.

[0098] In this implementation, the main function of the evaluation information is to supervise the learning of the first model. It is not input into the first model. The first model only predicts the score based on the user's answer content and the degree of match between the standard answer and the first model. After obtaining the second prediction information, the evaluation information is used to prompt the first model so that the first model knows whether the prediction result is accurate, thereby deciding whether to adjust the parameters of the first model and obtaining a scoring prediction model.

[0099] Please continue to see Fig. 9 , Fig. 9 yes Figure 2The schematic diagram of the second specific implementation method of S24 in FIG. Fig. 9 As shown, in this embodiment, the training goal of the first model is to predict the Gaussian distribution to which each second sample data belongs based on the input question, the standard answer, the user's answer content, and the judgment information of the user's answer content. The method specifically includes:

[0100] S2421: Input the questions, standard answers, user answer content and evaluation information of the user answer content in each second sample data into the first model, so that the first model makes predictions based on the degree of matching between the user answer content and the standard answer, and obtains all second prediction information of each second sample data.

[0101] In this implementation, the second prediction information is the predicted score rate obtained by the first model based on the matching degree between the user's answer content and the standard answer.

[0102] In this implementation manner, the second prediction information corresponding to all the second sample data in the second sample data set is obtained in order to subsequently fit the Gaussian distribution.

[0103] S2422: Obtain the mean of all second prediction information through the first model.

[0104] In this implementation, the average prediction score rate of all the second prediction information is obtained through the first model.

[0105] S2423: Gaussian distribution fitting is performed on the mean and the evaluation information of the user's answer content corresponding to each second sample data through the first model, and the model parameters of the first model are adjusted based on the comparison results of the obtained variances with the first set threshold to obtain a score prediction model.

[0106] In this implementation, the data output by the output layer of the first model includes the mean of all the second prediction information and the standard deviation corresponding to each second sample data.

[0107] In this implementation, the following formula is fitted with a Gaussian distribution using the first model:

[0108]

[0109] Where p(y) is the probability density function; y is a random variable; μ is the mathematical expectation, which is the location parameter of the Gaussian distribution and describes the central tendency position of the Gaussian distribution. The Gaussian distribution is completely symmetrical with y=μ as the axis of symmetry, and the mean, median and mode of the Gaussian distribution are the same, all equal to μ; σ is the standard deviation, which describes the degree of dispersion of the data distribution of the Gaussian distribution and determines the amplitude of the distribution. The larger σ is, the more dispersed the data distribution is, and the smaller σ is, the more concentrated the data distribution is; σ 2 is the variance.

[0110] In this implementation, μ is the average predicted score rate of all second prediction information, and y is the evaluation information of the user's answer content corresponding to each second sample data, that is, the true score rate (score label) of each second sample data.

[0111] Specifically, by fitting a Gaussian distribution to each second sample data, the variance (the square of the standard deviation) of each second sample data can be obtained. If the variance corresponding to a certain second sample data is greater than the first set threshold, it indicates that the second prediction information predicted by the first model based on the second sample data is not accurate enough, and the model is not confident in the result of the scoring, and needs to be rejected for correction.

[0112] In this implementation, the first set threshold is determined through a validation set.

[0113] Specifically, the sample data in the validation set may be part of the second sample data in the second sample data set, or may be additionally obtained sample data, and its structure is the same as that of the second sample data. The sample data in the validation set is input into the first model, and the variance of all sample data in the entire sample set is obtained by the above formula, and all variances are sorted from large to small. After sorting, all variance data are divided into ten equal parts, and a group of data in the first group of the ten equal parts is selected, that is, a group of data including the largest variance, and the smallest variance is selected from the group of data, and the variance is used as the first set threshold.

[0114] It can be understood that selecting the variance with a numerical value in the top 10% as the first set threshold can enable the first model to reject approximately 10% of the second sample data in the second sample data set, thereby ensuring the prediction confidence of the first model.

[0115] It can be understood that the first set threshold should not be too small in order to prevent overfitting and improve the generalization performance of the first model, so as to ensure the consistency of the first model tested in the validation set and the test set.

[0116] In this implementation, if the variance corresponding to a certain second sample data is smaller than the first set threshold, it indicates that the second prediction information predicted by the first model based on the second sample data is relatively accurate, and the system is confident in the result of this scoring and does not reject this correction. However, the parameters of the first model can be appropriately adjusted according to the degree of match between the second prediction information and the score label, and the above operation can be repeated to obtain a more accurate scoring prediction model.

[0117] Through the above training process, a scoring prediction model with relatively strong scoring ability can be obtained. However, due to the use of a large amount of labeled data during training, the scoring accuracy of the scoring prediction model for the trained answer type is significantly higher than that for the untrained answer type. If an untrained answer type is added later and the scoring prediction model is retrained with the answer content corresponding to the answer type, it will result in very high computational costs.

[0118] To solve the above-mentioned cost problem, this implementation fixes most of the parameters of the scoring prediction model on the basis of obtaining a relatively reliable scoring prediction model through the above-mentioned training. During the training process of the data corresponding to the newly added question types, by only adjusting the parameters of the fully connected layer in the scoring prediction model, the scoring prediction model can be effectively adapted to the newly added data while maximizing the scoring accuracy of the scoring prediction model on the trained question types.

[0119] This embodiment uses the first sample data set to perform prediction training on the preset language model, so that the constructed first model can learn the rules implicit in mathematical sentences to more accurately extract and identify different mathematical texts. Furthermore, by performing scoring training on the first model through the second sample data set, the obtained scoring prediction model can better predict based on the user's answer content, thereby meeting the needs of automatic review of answer questions. In addition, by fine-tuning the trained scoring prediction model, the scoring prediction model can also effectively adapt to the newly added different types of answer questions, thereby further improving the application scope of the scoring prediction model.

[0120] S13: Score the user's answer based on the recognition result and output the score rate.

[0121] In this embodiment, the scoring prediction model determines the degree of match between the user's answer content and the standard answer based on the identified standard answer and the user's answer content, and then scores the user's answer content based on the degree of match and outputs a predicted score rate.

[0122] Different from the existing technology, this embodiment uses a scoring prediction model to identify and score data texts including mathematical formulas and text information. The scoring prediction model is obtained through training of incomplete mathematical texts, has a certain understanding ability for mathematical texts containing mathematical formulas, and can better make autonomous predictions based on the content of user answers, thereby reducing labor costs and meeting the needs of automatic review of answers.

[0123] The above review method can score a single input mathematics text, but cannot determine the confidence of a single correction. Based on this, the present application provides another review method.

[0124] Specifically, see Fig.10 , Fig.10 Schematic diagram of the second embodiment of the review method of this application. Fig.10 As shown, in this embodiment, the score prediction model is obtained by training through the above-mentioned training method, and the review method includes:

[0125] S41: Acquire multiple mathematics texts to be scored.

[0126] In this embodiment, the mathematical text includes the standard answer and the user's answer content, and the mathematical text includes mathematical formulas and text information; wherein the mathematical text is the text after word segmentation processing.

[0127] The math text is a text token sequence obtained by concatenating the question, the standard answer, and the user's answer content after word segmentation.

[0128] S42: Input each mathematical text into a scoring prediction model, and use the scoring prediction model to recognize text information and mathematical formulas in each mathematical text to obtain multiple recognition results.

[0129] In this embodiment, the rating prediction model identifies multiple corresponding standard answers and user answer contents based on multiple input text token sequences to obtain multiple recognition results.

[0130] S43: Score all the multiple recognition results through the scoring prediction model, output multiple score rates, and obtain the average of the multiple score rates through the scoring prediction model.

[0131] S44: Gaussian distribution fitting is performed on the means of multiple score rates and the score rates corresponding to each mathematical text through the score prediction model, and based on the comparison results of the obtained variances with the second set threshold, it is determined whether to refuse to review each mathematical text; wherein, in response to the variance of a single mathematical text being greater than the second set threshold, the single mathematical text is refused to be scored.

[0132] In this implementation, the following formula is fitted with a Gaussian distribution through a score prediction model:

[0133]

[0134] Where p(y) is the probability density function; y is a random variable; μ is the mathematical expectation; σ is the standard deviation; σ 2 is the variance.

[0135] In this implementation, μ is the average of multiple scoring rates, and y is the scoring rate corresponding to each mathematics text to be scored.

[0136] Specifically, by fitting a Gaussian distribution to each mathematical text to be scored, the variance (the square of the standard deviation) of each mathematical text to be scored can be obtained. If the variance corresponding to a mathematical text to be scored is greater than the second set threshold, it indicates that the score rate predicted by the scoring prediction model based on the mathematical text to be scored is not accurate enough, and the model is not confident in the result of the scoring and needs to refuse to grade it.

[0137] In this implementation manner, the method for determining the second set threshold is consistent with the above, and will not be repeated here.

[0138] Different from the prior art, this embodiment enables the scoring prediction model to make better predictions based on the user's answer content, thereby meeting the needs of automatic review of answer questions. In addition, by refusing to correct some mathematics texts, the scoring accuracy of the scoring prediction model can be improved, thereby improving the accuracy of the review method.

[0139] Correspondingly, the present application provides a review device.

[0140] See also Fig.11 , Fig.11 Schematic diagram of the structure of an embodiment of the review device of the present application. Fig.11 As shown, the review device 50 includes an acquisition module 51 , an identification module 52 and a scoring module 53 .

[0141] In this embodiment, the acquisition module 51 is used to acquire the mathematical text to be scored; wherein the mathematical text includes the standard answer and the user's answer content, and the mathematical text includes mathematical formulas and text information; wherein the mathematical text is the text after word segmentation processing.

[0142] The recognition module 52 is used to input the mathematical text into the scoring prediction model, and use the scoring prediction model to recognize the text information and mathematical formulas in the mathematical text; wherein the scoring prediction model is trained using incomplete mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information.

[0143] The scoring module 53 is used to score the user's answer content based on the recognition result and output the score rate.

[0144] For the specific review process, please refer to the relevant text descriptions in S11~S13, S21~S24, S211~S213, S221~S223, S2411~S2412 and S2421~S2423, which will not be repeated here.

[0145] Different from the prior art, this embodiment obtains the mathematical text to be scored through the acquisition module 51, and recognizes the text information and mathematical formulas in the mathematical text through the recognition module 52, and the recognition process is performed by a scoring prediction model that has a certain understanding of the mathematical text, and can accurately extract and recognize different mathematical texts. Furthermore, the scoring module 53 scores the user's answer content based on the recognition result and outputs the score rate, which can better predict based on the user's answer content, thereby reducing labor costs and meeting the needs of automatic review of answer questions.

[0146] Correspondingly, the present application provides an electronic device.

[0147] See also Fig.12 , Fig.12 Schematic diagram of the structure of an electronic device of the present application. Fig.12 As shown, the electronic device 60 includes a memory 61 and a processor 62 .

[0148] In this embodiment, the memory 61 is used to store program data, and when the program data is executed, the steps in the review method described above are implemented; the processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps in the review method described above.

[0149] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in the review method as described above. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by multiple integrated circuit chips.

[0150] Different from the prior art, the present embodiment obtains the math text to be scored through the processor 62, and recognizes the text information and math formulas in the math text, and the recognition process is performed by a scoring prediction model that has a certain understanding of the math text, and can accurately extract and recognize different math texts. Furthermore, based on the recognition result, the user's answer content is scored and the score rate is output, which can better predict according to the user's answer content, thereby reducing labor costs and meeting the needs of automatic review of answer questions.

[0151] Correspondingly, the present application provides a computer-readable storage medium.

[0152] See also Fig.13 , Fig.13 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present invention.

[0153] The computer-readable storage medium 70 includes a computer program 701 stored on the computer-readable storage medium 70, and the computer program 701 implements the steps in the review method as described above when executed by the processor. Specifically, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 100. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium 70, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present application. The aforementioned computer-readable storage medium 70 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0158] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A review method, It is characterized in that include: A mathematics text to be graded is obtained; wherein the mathematics text includes a standard answer and a user's answer content, and the mathematics text includes a mathematical formula and text information; wherein the mathematics text is a text that has been processed by word segmentation; The mathematical text is input into a scoring prediction model, and the scoring prediction model is used to identify the text information and the mathematical formula in the mathematical text; wherein the scoring prediction model is obtained by training using an incomplete mathematical text, and the scoring prediction model has the ability to understand the basic grammatical rules of the mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information; the method of training using an incomplete mathematical text specifically includes: A first sample data set is obtained, each first sample data is the incomplete mathematical text; wherein the first sample data is obtained by partially masking the original sample; Using the first sample data set to perform mathematical formula and text prediction training on a preset language model to obtain a first model, including: constructing the mathematical formula in the original sample to generate a formula parse tree; traversing the formula parse tree to obtain position information of each node of the formula parse tree in the original sample, and storing the formula parse tree and the corresponding position information; using the first sample data and the position information to perform mathematical formula and text prediction training on the preset language model to obtain the first model; A second sample data set is obtained, each second sample data includes a question, a standard answer, a user's answer content, and evaluation information of the user's answer content; and the second sample data includes a mathematical formula and text information; Using the second sample data set to perform scoring training on the first model to obtain the scoring prediction model; The user's answer content is scored based on the recognition result and the score rate is output.

2. The review method according to claim 1, It is characterized in that The step of obtaining the mathematics text to be scored specifically includes: Acquire a plurality of mathematics texts to be scored; The step of inputting the mathematical text into a scoring prediction model and using the scoring prediction model to identify the text information and the mathematical formula in the mathematical text comprises: Inputting each of the mathematical texts into the score prediction model, and using the score prediction model to recognize the text information and the mathematical formula in each of the mathematical texts to obtain a plurality of recognition results; The step of scoring the user's answer content based on the recognition result and outputting the score rate includes: Scoring all the multiple recognition results through the scoring prediction model, outputting multiple score rates, and obtaining the average of the multiple score rates through the scoring prediction model; The scoring prediction model is used to perform Gaussian distribution fitting on the mean of the multiple scoring rates and the scoring rates corresponding to the respective mathematics texts, and based on the comparison results of the obtained variances with the second set threshold, it is determined whether to refuse to review the respective mathematics texts; wherein, in response to the variance of a single mathematics text being greater than the second set threshold, the scoring of the single mathematics text is refused.

3. The review method according to claim 1, It is characterized in that The step of acquiring a first sample data set, each first sample data being the incomplete mathematical text, comprises: Acquire an original sample set, wherein each original sample includes the mathematical formula and the text information; Performing word segmentation processing on the mathematical formula and the text information in each original sample to divide the mathematical formula and the text information into a plurality of word segments; The word segments corresponding to the mathematical formula and the word segments corresponding to the text information are partially masked to obtain the first sample data set.

4. The review method according to claim 3, It is characterized in that The step of performing mathematical formula and text prediction training on the preset language model using the first sample data and the position information to obtain the first model includes: Predicting the content and position of the masked mathematical formula and the masked text information in the first sample data by using the preset language model to obtain first prediction information; The first prediction information, the position information, and each of the first sample data are used to perform mathematical formula and text prediction training on the preset language model to obtain the first model.

5. The review method according to claim 3, It is characterized in that The step of constructing the mathematical formula in the original sample to generate a formula parsing tree includes: Obtain at least one variable and at least one operator in the word segment corresponding to the mathematical formula; The variables and the operators are added to the first stack and the second stack respectively, and the mathematical formula is converted into reverse Polish notation according to the priority of the operators, so as to generate the formula parse tree based on the reverse Polish notation.

6. The review method according to claim 4, It is characterized in that The preset language model includes an input layer, a convolutional downsampling layer, a bidirectional encoder, a fully connected layer and an output layer in sequence; The step of predicting the content and position of the masked mathematical formula and the masked text information in the first sample data by using the preset language model to obtain first prediction information includes: Inputting the first sample data into the input layer for processing to obtain a word vector, a position vector, and a paragraph vector corresponding to the word segmentation; Inputting the word vector, the position vector and the paragraph vector into the convolutional downsampling layer for feature extraction to obtain a feature vector; Inputting the feature vector into the bidirectional encoder to extract multi-dimensional information, and inputting the extracted information into the fully connected layer to predict the content and the position, to obtain the first prediction information; The step of performing mathematical formula and text prediction training on the preset language model using the first prediction information, the position information, and each of the first sample data to obtain the first model specifically includes: Based on the position information and the first sample data, it is determined whether the first prediction information matches the content and position of the masked mathematical formula and the masked text information, and the model parameters of the preset language model are adjusted based on the matching result to obtain the first model.

7. The review method according to claim 6, It is characterized in that The step of using the second sample data set to perform scoring training on the first model to obtain the scoring prediction model includes: Predicting the user's answer content and the standard answer by using the first model to obtain second prediction information; The first model is trained for score prediction using the second prediction information and the evaluation information to obtain the score prediction model.

8. The review method according to claim 7, It is characterized in that The step of predicting the user's answer content and the standard answer by using the first model to obtain second prediction information includes: Inputting the question, the standard answer and the user's answer content in the second sample data into the first model, so that the first model makes a prediction based on the matching degree between the user's answer content and the standard answer, thereby obtaining the second prediction information; The step of performing score prediction training on the first model using the second prediction information and the evaluation information to obtain the score prediction model comprises: The second prediction information is compared with the evaluation information, and the model parameters of the first model are adjusted based on the comparison result to obtain the score prediction model.

9. The review method according to claim 7, It is characterized in that The step of predicting the user's answer content and the standard answer by using the first model to obtain second prediction information includes: Inputting the question, the standard answer, the user's answer content, and the evaluation information of the user's answer content in each of the second sample data into the first model, so that the first model makes a prediction based on the matching degree between the user's answer content and the standard answer, and obtains all the second prediction information of each of the second sample data; The step of performing score prediction training on the first model using the second prediction information and the evaluation information to obtain the score prediction model comprises: Obtaining a mean value of all the second prediction information through the first model; The first model is used to perform Gaussian distribution fitting on the mean and the evaluation information of the user's answer content corresponding to each of the second sample data, and the model parameters of the first model are adjusted based on the comparison results of the obtained variances with the first set threshold to obtain the score prediction model.

10. A review device, It is characterized in that include: An acquisition module is used to acquire a mathematical text to be graded; wherein the mathematical text includes a standard answer and a user's answer content, and the mathematical text includes a mathematical formula and text information; wherein the mathematical text is a text after word segmentation processing; A recognition module is used to input the mathematical text into a scoring prediction model, and use the scoring prediction model to recognize the text information and the mathematical formula in the mathematical text; wherein the scoring prediction model is obtained by training with incomplete mathematical text, and the scoring prediction model has the ability to understand the basic grammatical rules of mathematical text; wherein the incomplete mathematical text includes masked mathematical formulas and masked text information; the method for training with incomplete mathematical text specifically comprises: obtaining a first sample data set, each first sample data is the incomplete mathematical text; wherein the first sample data is obtained by partially masking the original sample; using the first sample data set to perform mathematical formula and text prediction on a preset language model The method comprises: constructing the mathematical formula in the original sample to generate a formula parse tree; traversing the formula parse tree to obtain the position information of each node of the formula parse tree in the original sample, and storing the formula parse tree and the corresponding position information; performing mathematical formula and text prediction training on the preset language model using the first sample data and the position information to obtain the first model; obtaining a second sample data set, each second sample data including a question, a standard answer, a user's answer content and evaluation information of the user's answer content; and the second sample data including mathematical formulas and text information; performing scoring training on the first model using the second sample data set to obtain the scoring prediction model; The scoring module is used to score the user's answer content based on the recognition result and output a score rate.

11. An electronic device, It is characterized in that include: A memory for storing program data, wherein the program data, when executed, implements the steps of the review method according to any one of claims 1 to 9; A processor is used to execute the program data stored in the memory to implement the steps in the review method according to any one of claims 1 to 9.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the review method according to any one of claims 1 to 9 are implemented.

Images