Automatic correcting and scoring method and system based on data driving

Abstract

The invention provides an automatic correction scoring method and system based on data driving, and relates to the field of electric digital data processing, and the method comprises the steps: obtaining to-be-corrected homework data and a corresponding question type; matching correction modes according to the question types, wherein the correction modes comprise a direct correction mode and a retrieval correction mode; if the mode is the direct correction mode, inputting the homework data into the trained direct correction model to obtain a correction result; if the mode is the retrieval correction mode, a preset number of student answers with the highest similarity and corresponding scores are retrieved from a database according to the homework data; and inputting the homework data, the student answer and the corresponding score into a scoring model to obtain a correction result. By implementing the method, the most suitable correction mode can be automatically matched according to the complexity of the question, the ability of evaluating the diversity and depth of answers is improved, and then the accuracy and efficiency of automatic correction are improved.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a data-driven automatic grading method and system. Background Art

[0002] In the current education assessment system, automated grading technology is a key tool for improving teaching efficiency and reducing the burden on teachers. With the development of information technology, particularly the application of big data and artificial intelligence, automated grading systems have begun to be widely used in the grading of various exams and assignments. This not only speeds up grading but also ensures consistent grading.

[0003] In related technologies, automated grading relies primarily on rule-based systems that evaluate students' responses based on pre-defined answers or scoring criteria. For example, for simple questions like multiple-choice or fill-in-the-blank questions, rule-based systems automatically assign scores based on the accuracy of keywords or calculation formulas. This approach works well for questions with simple structures and clear standard answers.

[0004] However, grading complex question types such as short-answer questions and essay questions usually requires a deeper understanding and analysis. Automatic grading technology based on rule systems often finds it difficult to accurately capture the diversity and depth of students' answers, resulting in lower grading accuracy. Summary of the Invention

[0005] This application provides a data-driven automatic grading method and system to address the problem that rule-based automatic grading systems have limitations when handling complex question types and are difficult to effectively evaluate the diversity and depth of students' answers.

[0006] In a first aspect, the present application provides a data-driven automatic grading method, which is applied to an automatic grading system. The method comprises: Obtain the homework data to be graded and the corresponding question types, which include fill-in-the-blank questions, short-answer calculation questions, and solution questions; Match the correction mode according to the question type, which includes direct correction mode and search correction mode; If it is detected that the question type matches the direct grading mode, the homework data is input into the trained direct grading model to obtain the grading result; If it is detected that the question type matches the search and correction mode, a preset number of student answers and corresponding scores with the highest similarity are retrieved from the database based on the homework data; The homework data, the student's answer and the corresponding score are input into the scoring model to obtain the grading result.

[0007] Through the above-described embodiment, the automatic grading system automatically selects the most appropriate grading mode based on the question type: direct grading mode or search grading mode. Direct grading mode is suitable for questions with relatively fixed answer structures and allows for quick and accurate grading. Search grading mode, on the other hand, targets complex questions that require deeper understanding and analysis. It scores answers by searching for existing answers that are highly similar to the answer to be graded. This effectively improves the ability to assess the diversity and depth of answers, thereby enhancing the accuracy and efficiency of automatic grading.

[0008] In some embodiments, before the step of inputting the homework data into the trained direct grading model to obtain the grading result if it is detected that the question type matches the direct grading mode, the method further includes: Obtaining original training data, which includes question stems, reference answers, question scores, student answers, and student scores; Preprocessing the original training data to obtain pre-training data, wherein the pre-processing includes data deduplication, data screening, data verification, and data expansion, and the pre-training data includes a training set and a test set; The pre-trained data is used to train a direct grading model to predict the scores of the input homework data and obtain the grading results. The homework data includes the question stem, reference answers, question scores, and student answers.

[0009] Through the above embodiments, the automatic grading system ensures the quality and diversity of training data through preprocessing steps such as data deduplication, screening, verification and expansion, thereby enhancing the prediction accuracy and generalization ability of the direct grading model, enabling it to better understand and predict the answer scores of different students.

[0010] In some embodiments, after the step of using the pre-training data to train a direct grading model for predicting scores for the inputted homework data and obtaining grading results, the following steps are further included: Use the trained direct correction model to score each student's answer on the test set and give a corresponding confidence score; Setting multiple confidence threshold ranges, where the multiple confidence threshold ranges cover all the confidence scores; Calculate the scoring accuracy corresponding to each confidence threshold range; Filtering out one or more target confidence threshold ranges whose scoring accuracy exceeds a preset accuracy threshold; The minimum confidence score in the target confidence threshold range is determined as the lowest confidence threshold, and the lowest confidence threshold is used to screen the correction results.

[0011] Through the above embodiments, the automatic grading system can select the optimal confidence threshold interval by scoring on the test set and calculating the accuracy under different confidence threshold ranges, which is used for scoring screening in actual grading, thereby improving the accuracy and reliability of grading.

[0012] In some embodiments, the step of retrieving a preset number of student answers and corresponding scores with the highest similarity from the database based on the homework data specifically includes: Input the homework data into the similarity model to obtain the vector representation corresponding to the student's answer; Searching a vector database to determine a preset number of target vector representations that have the highest similarity to the vector representation, wherein the vector database stores vector representations corresponding to multiple student answers, and the similarity is determined by the distance between two vectors; A preset number of student answers and corresponding scores are determined based on the target vector representation.

[0013] Through the above-described embodiment, the automated grading system, by utilizing a similarity model and a vector database, can accurately locate multiple student answers that are most similar to the assignment data to be graded. This method can quickly locate the most relevant student answers within a vast answer library, providing a more realistic grading basis, thereby improving grading accuracy and efficiency.

[0014] In some embodiments, the steps of inputting the homework data into a similarity model to obtain vector representations corresponding to the student's answers; and searching a vector database to determine a preset number of target vector representations having the highest similarity to the vector representations specifically include: Inputting the homework data into a first similarity model and a second similarity model respectively to obtain a first vector representation and a second vector representation corresponding to the student's answer, wherein the complexity of the second similarity model is higher than that of the first similarity model; Searching a vector database to determine a first preset number of target first vector representations having the highest similarity to the first vector representation, wherein the plurality of first vector representations stored in the vector database are obtained by processing the plurality of student answers using the first similarity model; Inputting the first target vector representation into a second similarity model to obtain a second target vector representation; A search is performed in the target second vector representation to determine a second preset number of target second vector representations with the highest similarity to the second vector representation, where the second preset number is smaller than the first preset number.

[0015] Through the above embodiment, the automatic grading system uses the first similarity model to screen out multiple historical student answers that are highly similar to the student answers, and uses it to perform incremental training on the second similarity model, thereby improving the screening accuracy of the second similarity model, improving the system's ability to understand the content and structure of the answers, and thereby improving the accuracy of the grading results.

[0016] In some embodiments, before the step of inputting the homework data into the similarity model to obtain the vector representation corresponding to the student's answer, the step further includes: Obtain historical answer data, which includes question stems and student answers; Preprocess the historical answer data to obtain historical training data. The preprocessing includes data deduplication, data screening, data verification, and data expansion. Use deep learning technology to perform scoring point matching analysis on the historical training data, and automatically generate students' problem-solving ideas and scoring point sequences; Based on the problem-solving ideas and scoring point sequence, a similarity model is trained to output the vector representation corresponding to each student's answer.

[0017] Through the above embodiment, the automatic grading system analyzes historical answer data through deep learning technology and automatically generates students' problem-solving ideas and scoring points. This not only provides high-quality input data for the training of the similarity model, but also enhances the model's depth of understanding of students' answers, and improves the personalization and accuracy of grading.

[0018] In some embodiments, after the step of retrieving a preset number of student answers and corresponding scores with the highest similarity from the database based on the homework data if it is detected that the question type matches the search and correction mode, the method further includes: Use the preset number of student answers and corresponding scores to construct a scoring dataset; Use the scoring dataset to incrementally train the grading model to obtain a single-question scoring model; The single-question scoring model is structurally adjusted to obtain a scoring model, which is used to predict scores based on input homework data and corresponding search data to obtain grading results.

[0019] Through the above-mentioned embodiment, the automatic grading system, in retrieval and grading mode, uses the retrieved answers and scores to incrementally train the scoring model. This method can continuously optimize and update the scoring model, making it more accurately adapt to the changes in different questions and student answers, thereby improving the system's adaptability and scoring accuracy.

[0020] In a second aspect, the present application provides an automatic grading system, the automatic grading system comprising: one or more processors and a memory; The memory is coupled to the one or more processors, and is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions to enable the automatic grading system to implement a data-driven automatic grading and scoring method provided in the above embodiment, which will not be described in detail here.

[0021] In a third aspect, the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on an automatic grading system, the automatic grading system can implement a data-driven automatic grading and scoring method provided in the above embodiment, which will not be described in detail here.

[0022] Fourthly, the present application provides a computer program product. When the computer program product runs on an automatic grading system, the automatic grading system can implement a data-driven automatic grading and scoring method provided in the above embodiment, which will not be described in detail here.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. The automatic grading system automatically selects the most appropriate grading mode based on the question type, either direct grading or search grading. For complex questions, it grades existing answers that are highly similar to the answer to be graded. Furthermore, by using a similarity model and vector database, the system continuously conducts incremental training during actual grading. This not only improves grading accuracy and efficiency, but also enhances the system's understanding of answer content and structure, significantly improving the accuracy of grading results.

[0024] 2. The automated grading system ensures the quality and diversity of training data through preprocessing steps such as data deduplication, screening, verification, and expansion, enhancing the predictive accuracy and generalization capabilities of the direct grading model. Furthermore, the system leverages deep learning technology to analyze historical answer data, automatically generating student problem-solving strategies and scoring points. This provides high-quality input data for similarity model training, enhancing the model's understanding of student answers and ultimately improving the personalization and accuracy of grading.

[0025] 3. The automated grading system scores the test set and calculates accuracy under different confidence thresholds, selecting the optimal confidence threshold range for scoring and screening during actual grading. This approach optimizes the scoring model, making it more accurately adaptable to the variability of different questions and student answers, improving the system's adaptability and scoring accuracy, thereby effectively enhancing the reliability of grading. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a data-driven automatic grading and correction method in an embodiment of the present application; Figure 2 This is a flow chart of the automatic correction process performed by the automatic correction system in the direct correction mode according to an embodiment of the present application; Figure 3 This is a flow chart of the automatic correction process performed by the automatic correction system in the retrieval correction mode according to an embodiment of the present application; Figure 4 It is a schematic diagram of the physical device structure of the automatic correction system in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in this application refers to any or all possible combinations comprising one or more of the listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0029] For ease of understanding, the following is a flow chart of the method provided in this implementation. Figure 1 FIG. 1 is a flow chart of a data-driven automatic grading and correction method in an embodiment of the present application.

[0030] S101. Obtain homework data to be graded and corresponding question types.

[0031] The automatic grading system obtains the data of homework to be graded submitted by students from the teaching platform or homework submission channel. Among them, the data of homework to be graded usually exists in various forms such as text, pictures, audio, etc. At the same time, the system also obtains the question type information corresponding to the data of homework to be graded. The question type refers to the structural form and answer requirements of the question. Common question types include fill-in-the-blank questions, short-answer calculation questions, and answer questions. Specifically, the automatic grading system can obtain question type information in a variety of ways. Optionally, the teacher directly marks the question type when asking the question; optionally, the system automatically determines the question type by analyzing the keywords and sentence structure in the question stem using natural language processing technology. There is no limitation here.

[0032] Understandably, fill-in-the-blank questions leave blanks in the question stem, requiring students to fill in the blanks with words, numbers, symbols, and other content. The answers to fill-in-the-blank questions are typically brief and fixed. Simple calculation questions require students to analyze and calculate the problem, and then write down the calculation process and results. Solution questions require students to think thoroughly about the problem and provide a complete reasoning process and conclusion. Compared to fill-in-the-blank questions, simple calculation questions and solution questions offer more open-ended answers and more flexible expression.

[0033] S102. Match the grading mode according to the question type.

[0034] After obtaining the question type information, the automatic grading system selects an appropriate grading mode based on the characteristics of the question type. The grading mode refers to the strategy and process the system uses to analyze and score student answers. Different question types correspond to different grading modes. The present embodiment proposes two grading modes: direct grading mode and search grading mode.

[0035] Specifically, the direct grading mode is suitable for questions with relatively fixed answers and clear evaluation criteria, such as fill-in-the-blank questions. In this mode, the system simply matches the answer with the standard answer and directly gives the correctness and score of the answer.

[0036] The retrieval and grading mode is suitable for questions with diverse answer formats and complex evaluation criteria, such as short-answer calculation questions and essay questions. This mode relies on a large amount of historical answer data and uses techniques such as similarity matching to retrieve student answers from the database that are similar to the answers to be graded. The answers to be graded are then comprehensively evaluated and graded based on their scores.

[0037] Furthermore, in actual use, the automatic grading system can flexibly combine the two modes described above. For example, in one embodiment, for a question that combines both fill-in-the-blank and short-answer sections, the direct grading mode can be used to determine the score of the fill-in-the-blank section, and then the search grading mode can be used to evaluate the score of the short-answer section. Finally, the scores of the two sections are weighted and combined to obtain the final answer score.

[0038] For example, for the question "Calculate and fill in the blanks: When x=1, the value of the function f(x)=2x+3 is ____. Explain your calculation process.", the system first determines that the question contains fill-in-the-blank requirements and explanation requirements, and is a combination of fill-in-the-blank and answer questions. Therefore, the system first uses the direct correction mode to compare the numerical value filled in by the student with the standard answer "5". If they are the same, the fill-in-the-blank part is judged to be full marks, otherwise no points. Then, the system uses the retrieval correction mode to analyze the student's description of the problem-solving process, and matches the similarity with the excellent problem-solving processes in the historical data to obtain the score of the answer part. Finally, the scores of the fill-in-the-blank and answer parts are weighted averaged according to the weight of the question score to obtain the student's final score.

[0039] S103: If it is detected that the question type matches the direct grading mode, the homework data is input into the trained direct grading model to obtain the grading result.

[0040] When the automatic grading system detects that the question to be graded is a fill-in-the-blank type that can be directly graded, the system will use direct grading mode. In this mode, the system first loads a pre-trained direct grading model, then inputs the homework data into the direct grading model, outputs the score corresponding to the student's answer, and thus obtains the grading result.

[0041] In direct grading mode, the appropriate model architecture can be selected based on the specific application scenario. For relatively simple question types, a neural network architecture based on pre-trained language models such as BERT can be used to build the framework for the direct grading model. Since student scores are generally discrete values ​​(such as 0, 1, 3, 5, 7, 10, etc.), the model's output layer can be designed as either a regression task to predict a continuous score value, or a classification task to predict the category to which the score belongs. In the classification setting, multi-task learning can be further employed, with a separate output layer for each question. These output layers share a common encoder, effectively preventing feature confounding and interference between different questions and improving the model's generalization. In the classification setting, the model's output layer can be a unified structure or a multi-task approach to predict a floating-point value representing the score. Of course, in addition to traditional neural network methods, large models (such as the GPT series) can also be used to build direct grading models. Information such as the question stem, reference answers, question scores, student answers, and student scores can be integrated into a valid input format to guide model training and prediction.

[0042] S104: If it is detected that the question type matches the search and correction mode, a preset number of student answers and corresponding scores with the highest similarity are retrieved from the database based on the homework data.

[0043] When the automatic grading system detects a subjective question, such as a short-answer or essay question, the system uses a search-based grading model. Because the answers are flexible and diverse, making it difficult to set a standard answer, the system uses a search-based grading model. This model uses a massive amount of historical answer data and similarity matching techniques to identify the most similar reference answers to the question being graded. The system then uses these scores to evaluate and score the question being graded.

[0044] Specifically, the automatic grading system constructs historical training data through the questions and student answers stored in the database, uses deep learning technology to perform scoring point matching analysis on the historical training data, automatically generates students' problem-solving ideas and scoring point sequences, and then trains a similarity model based on the problem-solving ideas and scoring point sequences, outputs the vector representation corresponding to each student's answer, and stores it in the vector database.

[0045] After receiving the homework data, it is input into the similarity model and its corresponding vector representation is output. By calculating the distance between it and other vector representations in the vector database (such as Euclidean distance, etc.), the preset number of student answers and corresponding scores with the highest similarity are determined.

[0046] S105: Input the homework data, student answers and corresponding scores into the grading model to obtain the grading results.

[0047] Specifically, the automatic grading system first uses the preset number of student answers and corresponding scores retrieved in step S104 to construct a scoring data set; then uses the scoring data set to perform incremental training on the direct grading model to obtain a single-question scoring model; structurally adjusts the single-question scoring model, adds new layers or modules to the existing model architecture, such as adding an additional attention layer to process the relationship between multiple answers, or introduces a new embedding layer to integrate additional inputs such as scores and similar answers, modifies the input format, ensures that the model can receive and correctly process the question stem, reference answers, question scores, student answers, and the retrieved preset number of similar answers and their scores to obtain a scoring model, so that the scoring model is used to predict scores based on the input homework data and the corresponding retrieval data to obtain grading results.

[0048] In the above embodiment, the automatic grading system automatically selects the most appropriate grading mode based on the question type: direct grading mode or search grading mode. Direct grading mode is suitable for questions with relatively fixed answer structures and allows for quick and accurate grading. Search grading mode, on the other hand, targets complex questions that require deeper understanding and analysis. It scores answers that are highly similar to the answer being graded, effectively improving the ability to assess the diversity and depth of answers, thereby enhancing the accuracy and efficiency of automatic grading.

[0049] The following is a more detailed description of the process of the method provided by this embodiment. Figure 2 FIG. 1 is a flow chart of automatic correction performed by the automatic correction system in the direct correction mode according to an embodiment of the present application.

[0050] S201: Obtain original training data and perform preprocessing to obtain pre-training data.

[0051] Specifically, the automatic grading system can obtain raw training data from multiple channels, such as open-source homework data platforms and the system's own historical database, to train the direct grading model. Raw training data includes question stems, reference answers, question scores, student answers, and student scores.

[0052] After acquiring the raw training data, the automated grading system performs pre-processing operations such as data deduplication, data screening, data validation, and data expansion. During the data deduplication phase, the system uses text features to deduplicate the raw training data, filtering out duplicate questions and answers to ensure data uniqueness. During the data screening phase, the system utilizes language models (such as BERT) to perform linguistic analysis on the text data, selecting questions and answers that are grammatically coherent, well-formulated, and consistent with human conventions. For liberal arts data, the system calculates the language model perplexity to filter out low-quality text with high perplexity. For science questions, in addition to calculating perplexity, the system also verifies the grammatical correctness of formulas in the questions and answers to ensure professionalism and accuracy. During the data validation phase, the system utilizes machine learning techniques, such as large language models, to analyze the obtained grading data (including student answers and corresponding scores) to determine whether the student answers are relevant, complete, include key steps, and whether the final result is correct. Based on this information, the system calculates the grading confidence level for each answer. Grading data with low confidence levels is discarded. The system also incorporates filtering rules, such as those that exclude students who receive full marks despite clearly incorrect answers or receive 0 points despite correct key steps and results, ensuring the reliability of the graded data. During the data expansion phase, the system leverages existing high-quality data using semi-supervised learning to automatically generate more student answers and corresponding scores, enriching the training data. Furthermore, the system conducts in-depth analysis of question stems and reference answers according to pre-set extraction rules, extracting key information such as key points and scoring criteria to guide subsequent scoring and feedback.

[0053] It should be noted that if the automatic grading system does not obtain enough original training data, it is necessary to manually construct the data. The construction method is: extract all the full-mark student answers, use the students' full-mark answers to simulate the reference answers, construct new original training data, and ensure that the number of question types with the lowest scores in the new original training data accounts for no less than 5%.

[0054] S202: Use the pre-training data to train a direct grading model to predict scores for the input homework data and obtain grading results.

[0055] The automatic grading system uses BERT as its base model and uses pre-training data to train the direct grading model. Since questions are generally scored discretely—for example, out of a total score of 10, the teacher's grading scores may be discrete values ​​such as 0, 1, 3, 5, 7, or 10—the automatic grading system can choose to train the direct grading model using a classification training method. Specifically, a multi-task model architecture can be designed. In this architecture, all questions share a common encoding layer (such as BERT), but each question has its own independent classification layer. This prevents cross-influence between questions with different scores and improves model accuracy. The output of the classification layer is the question's score category.

[0056] Of course, the automatic grading system can also choose to use regression training to train a direct grading model. The model output is a floating-point value representing the predicted student score. The direct grading model trained using regression can be a unified model or, like the classification model, use a multi-task structure with a separate output layer for each question. This is not a limitation here.

[0057] In addition, large-scale model technology can be used to train direct grading models. In this approach, information such as the question stem, reference answers, question scores, student answers, and student scores is constructed into a valid input format (prompt) and fed into the large language model. The large model understands the relationships between this information and generates appropriate scoring results based on the context. This allows the trained direct grading model to automatically grade new student assignments and generate grading results.

[0058] In the above embodiment, the automatic grading system ensures the quality and diversity of the training data through preprocessing steps such as data deduplication, screening, verification and expansion, thereby enhancing the prediction accuracy and generalization ability of the direct grading model, enabling it to better understand and predict the answer scores of different students.

[0059] S203. Use the trained direct grading model to score each student's answer on the test set and give a corresponding confidence score.

[0060] Once the direct grading model is trained, the automated grading system uses a new test dataset to evaluate its performance. The test dataset consists of a large number of real student assignments, which were completely invisible during the training phase. The system uses the trained direct grading model to predict each student's answer in the test dataset, assigning a score between 0 and 100.

[0061] In addition to the prediction score, the automatic grading model assigns a confidence score to each student's answer, indicating the model's confidence in the prediction. The confidence score is typically a real number between 0 and 1, with values ​​closer to 1 indicating greater confidence in the model's prediction. The specific calculation method for this score depends on the model type used. For example, for a logistic regression model, the output of the sigmoid function can be used as the confidence score; for a neural network model, the output probability of the softmax layer can be used as the confidence score. This is not a limitation here.

[0062] S204: Set multiple confidence threshold ranges, and calculate the scoring accuracy corresponding to each confidence threshold range.

[0063] After using the direct grading model to predict the test set, the automated grading system further analyzes the relationship between confidence scores and prediction accuracy. Specifically, the automated grading system divides the confidence score from 0 to 1 into several continuous intervals, such as [0, 0.5), [0.5, 0.7), [0.7, 0.9), [0.9, 1], etc. The automated grading system then counts the number of examples within each confidence interval, as well as the number of examples for which the direct grading model's predicted scores are completely consistent with the manually annotated scores. Dividing the latter by the former yields the scoring accuracy corresponding to that confidence interval.

[0064] S205: Filter out one or more target confidence threshold ranges whose scoring accuracy exceeds a preset accuracy threshold.

[0065] After calculating the scoring accuracy for each confidence interval, the automated grading system further selects a target confidence interval that meets the accuracy requirements. Specifically, the system can set a preset accuracy threshold, such as 90%. The system then iterates through all previously obtained confidence intervals and identifies any intervals whose corresponding accuracy exceeds the preset threshold (i.e., 90%). These intervals are then determined as the target confidence threshold range.

[0066] S206: Determine the minimum confidence score in the target confidence threshold range as the lowest confidence threshold.

[0067] During actual operation, the automatic grading system automatically determines whether each student assignment requires manual review based on the confidence score of the direct grading model's prediction results. Specifically, if the confidence score of the direct grading model's prediction results is greater than or equal to the minimum confidence threshold, it is determined that no manual review is required and the prediction results are directly used as the final grading results. Conversely, if the confidence score of the direct grading model's prediction results is less than the minimum confidence threshold, it is determined that manual review is required, and the corresponding assignment data is marked and collected for subsequent manual analysis and processing.

[0068] In the above embodiment, the automatic grading system can select the optimal confidence threshold interval by scoring on the test set and calculating the accuracy under different confidence threshold ranges for use in scoring screening in actual grading, thereby improving the accuracy and reliability of grading.

[0069] The following is a more detailed description of the process of the method provided by this embodiment. Figure 3 FIG. 1 is a flow chart of automatic correction performed by the automatic correction system in the retrieval correction mode according to an embodiment of the present application.

[0070] S301: Obtain historical answer data and preprocess it to obtain historical training data.

[0071] The historical answer data includes the question stem and the student's answer. The acquisition method and pre-processing method of the historical answer data are basically the same as those in step S201 and will not be repeated here.

[0072] S302. Use deep learning technology to perform scoring point matching analysis on historical training data to automatically generate students' problem-solving ideas and scoring point score sequences.

[0073] The automatic grading system uses deep learning technology to match scoring points and analyze problem-solving ideas on historical training data, and then extracts key information from the text corresponding to the student's answer, including the steps to solve the problem, the formulas and theorems used, the logic of the argument, etc., and determines the score of each key step.

[0074] Specifically, the automatic grading system uses pre-trained language models such as BERT, RoBERTa, etc. to unitize and vectorize student answers to obtain semantic representations of different fragments of student answers. Then, based on the reference answers and scoring criteria of the questions, the matching algorithm designed by technical personnel is used to find the key fragments in the student answers that correspond to the scoring points, and then determine whether the fragment is complete, correct and scored. At the same time, the deep learning model infers the student's thinking and logical order of solving the problem based on the contextual connection between the answer fragments. Finally, the system organizes the results of the matching analysis into a standardized data format to obtain the feature representation of the student's answer corresponding to the historical training data, including the knowledge point coverage vector (reflecting the type and proportion of knowledge points involved in the answer), the problem-solving path sequence (representing the key steps and logical order of problem-solving), and the scoring point score sequence (whether each key step is correct and the score).

[0075] S303: Based on the problem-solving ideas and scoring point score sequence, a similarity model is trained to output the vector representation corresponding to each student's answer.

[0076] In step S302, the automated grading system obtains a structured feature representation of each student's answer, including the solution path, knowledge point coverage, and scoring sequence. The system then uses these features to train a similarity model. By learning a mapping function, it converts the unstructured text of the student's answer into a semantic vector representation. The distance between these vectors reflects the degree of similarity between the answers in terms of solution path, knowledge point coverage, and other aspects.

[0077] Specifically, the automatic grading system can use the Siamese Network architecture to train the similarity model. The Siamese Network consists of two sub-networks with the same structure, each processing a pair of inputs. During training, the system samples answer pairs from historical training data, one part of which is different students' answers to the same question, and the other part is students' answers to different questions. For pairs of answers to the same question, the similarity model learns to map them to vector space positions that are closer; for pairs of answers to different questions, the model should map them to positions that are farther away. In this way, the similarity model can learn a semantic embedding space and encode features such as problem-solving ideas and knowledge point coverage into vector representations.

[0078] In practice, the automatic grading system can be based on pre-trained language models such as BERT, RoBERTa, and XLNet. The system designs a twin network structure based on these language models and fine-tunes it using historical training data to adapt it to the task of judging the similarity of student answers. During training, a contrastive loss function can be used as the optimization objective. This loss function encourages the vector distance between similar answer pairs to be less than a threshold, while the distance between different answer pairs is greater than a threshold. This allows the similarity model to grasp the semantic characteristics of student answers.

[0079] In the above embodiment, the automatic grading system analyzes historical answer data through deep learning technology and automatically generates students' problem-solving ideas and scoring points. This not only provides high-quality input data for the training of the similarity model, but also enhances the model's depth of understanding of students' answers and improves the personalization and accuracy of grading.

[0080] S304: Input the homework data into the first similarity model and the second similarity model respectively to obtain the first vector representation and the second vector representation corresponding to the student's answer.

[0081] Specifically, when a student submits a new assignment, the system first inputs the assignment data into the first similarity model. This model is a lightweight model that uses traditional text representation methods, such as TF-IDF and Word2Vec, to convert the student's answer into a first-level vector representation. This first vector representation primarily captures shallow features of the answer, such as keywords and word frequency, and aims to quickly filter out a subset of candidate answers that are similar to the answer to be graded in terms of problem-solving ideas and knowledge point coverage.

[0082] At the same time, the system feeds the homework data into a second similarity model. This model, an advanced model based on deep learning, further extracts the semantic features and problem-solving ideas of the answers from the subset of candidate answers selected by the first similarity model, transforming the student answers into a more refined and high-dimensional second vector representation.

[0083] S305 : Search in the vector database to determine a first preset number of target first vector representations having the highest similarity to the first vector representation.

[0084] After obtaining the first vector representation of the student's answer to be graded, the automatic grading system quickly retrieves a batch of similar answers (i.e., the target first vector representation) from the vector database to provide candidate matching targets for the second similarity model.

[0085] During system initialization, each student's answer is converted into a vector representation using a primary similarity model (such as TF-IDF) and stored in the vector database. When a new student answer arrives for grading, the system first generates its first vector representation. Then, using this vector as the center, it performs a K-nearest neighbor (KNN) search in the database to find the K closest historical answer vectors (i.e., the target first vector representation). K is a preset hyperparameter that controls the number of answers screened in the initial round.

[0086] It's important to note that a vector database is a database system specifically designed for processing high-dimensional vector data. It supports various operations on vectors, particularly similarity (nearest neighbor) retrieval. Available vector databases include Faiss, Annoy, and HNSW. These databases utilize hashing and tree indexing techniques to efficiently search for similar vectors.

[0087] S306 : Input the historical operation data corresponding to the first target vector representation into the second similarity model to obtain a second target vector representation.

[0088] The automatic grading system inputs a first preset number (eg, 100) of historical homework data retrieved in step S305 that have the highest similarity to the homework data to be graded into the second similarity model for processing, and obtains a target second vector representation corresponding to each set of historical homework data.

[0089] S307 : Search the target second vector representation to determine a second preset number of target second vector representations having the highest similarity to the second vector representation.

[0090] The automatic grading system also inputs the to-be-graded homework data into the second similarity model to obtain its corresponding second vector representation. Then, from the target second vector representation obtained in step S306, a second preset number (e.g., 5) of target second vector representations having the highest similarity with the second vector representation corresponding to the to-be-graded homework data are retrieved.

[0091] Common similarity calculation methods include Euclidean distance, cosine similarity, etc. The system calculates the similarity between the vector of the homework to be graded and each target second vector to obtain a similarity score, and then selects the top N vectors with the highest scores as the final selected target second vector representation.

[0092] For example, in step S306, the system obtains 100 target second vector representations, each of which is 768-dimensional. Simultaneously, the second vector representation of the homework to be graded is A, also 768-dimensional. The system calculates the similarity between A and these 100 vectors using the cosine similarity formula, obtaining 100 similarity scores. The system then selects the five vectors with the highest scores as the final target second vector representations.

[0093] In the above embodiment, the automatic grading system uses the first similarity model to screen out multiple historical student answers that are highly similar to the student answers, and uses it to perform incremental training on the second similarity model, thereby improving the screening accuracy of the second similarity model, improving the system's ability to understand the content and structure of the answers, and thereby improving the accuracy of the grading results.

[0094] S308. Determine a preset number of student answers and corresponding scores based on a second preset number of target second vector representations.

[0095] Because each target second vector representation is generated from a previous student's answer, the automated grading system can establish a mapping relationship between the target second vector representation and the student's answer. Once the target second vector representation is determined, this mapping relationship can be used to find the corresponding student answer. Each student answer is then assigned a score, resulting in a predetermined number of student answers and corresponding scores.

[0096] S309: Construct a scoring dataset using a preset number of student answers and corresponding scores.

[0097] The automated grading system organizes a preset number (e.g., five) of student answers and corresponding scores obtained in step S308 into a specific data format (e.g., JSON) to form a scoring dataset. Each sample in this dataset is a pairing of a student answer and a score. Furthermore, the system can also include metadata, such as question number and question type, in each sample to distinguish scoring data for different questions.

[0098] S310. Use the scoring dataset to perform incremental training on the direct grading model to obtain a single-question scoring model.

[0099] The automatic grading system divides the scoring data set constructed in step S309 into a training set and a validation set according to a certain ratio, such as 8:2. Then, the system loads the pre-trained direct grading model, which is usually an end-to-end scoring model based on deep learning, such as LSTM, BERT, etc. Next, the system uses the student answers in the training set as input and the corresponding scores as labels to perform incremental training on the direct grading model. Incremental training refers to adjusting the parameters of the model through learning new data on the basis of the original direct grading model, so that it can better adapt to the scoring requirements of the current question. During the training process, the system also uses the validation set to evaluate the performance of the model so as to stop in time and prevent overfitting. When the performance indicators of the model on the validation set (such as MSE, Accuracy, etc.) converge, the system obtains a single-question scoring model.

[0100] S311. Structural adjustment is performed on the single-question scoring model to obtain a scoring model, which is used to obtain grading results based on homework data, student answers, and corresponding scores.

[0101] New layers or modules can be added to the architecture of the existing single-question scoring model, such as adding an additional attention mechanism to handle the relationship between multiple retrieved data (retrieval of a preset number of student answers and corresponding scores with the highest similarity from the database). At the same time, the input format of the existing single-question scoring model is modified to ensure that the scoring model can simultaneously receive and correctly process the assignment data (including the question stem, reference answers, question scores, student answers) as well as the retrieved multiple student answers and their scores. After adjusting the structure, the model is fine-tuned using the scoring dataset to adapt it to the new input format and learn to use the added contextual information (such as the retrieved data) to improve its prediction quality. The scoring model will be able to comprehensively utilize the content of the student answer itself, as well as similar student answers and their scores retrieved from the database, to make more accurate scoring predictions for the assignment data, thereby obtaining the final grading results.

[0102] In the above embodiment, the automatic grading system, in retrieval and grading mode, uses the retrieved answers and scores to incrementally train the grading model. This approach continuously optimizes and updates the grading model, making it more accurately adaptable to different questions and student answers, thereby improving the system's adaptability and grading accuracy.

[0103] The automatic correction system of the embodiment of the present invention is applied to electronic equipment. Figure 4 A schematic diagram of the architecture of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0104] It should be noted that Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0105] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions (computer programs) or by controlling related hardware through instructions (computer programs), and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. The electronic device of this embodiment includes a storage medium and a processor, wherein the storage medium stores a plurality of instructions, which can be loaded by the processor to execute any step of the method provided in the embodiment of the present invention.

[0106] Specifically, the storage medium and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other via one or more signal lines. The storage medium stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware. The processor executes various functional applications and data processing by running the software programs and modules stored in the storage medium. The storage medium can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The storage medium is used to store programs, and the processor executes the programs after receiving the execution instructions.

[0107] Furthermore, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip having signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., which may implement or execute the various methods, steps, and logic flow diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] Since the instructions stored in the storage medium can execute the steps of any method provided in the embodiments of the present invention, the beneficial effects of any method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0109] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data-driven automatic grading method, applied to an automatic grading system, characterized in that: The method comprises: Obtaining homework data to be graded and corresponding question types, wherein the question types include fill-in-the-blank questions, short-answer calculation questions, and solution questions; Matching a correction mode according to the question type, the correction mode including a direct correction mode and a search correction mode; If it is detected that the question type matches the direct grading mode, the homework data is input into the trained direct grading model to obtain a grading result; If it is detected that the question type matches the search and correction mode, a preset number of student answers and corresponding scores with the highest similarity are retrieved from the database according to the homework data; The homework data, the student answers and the corresponding scores are input into a grading model to obtain a grading result.

2. The method according to claim 1, characterized in that Before the step of inputting the homework data into the trained direct correction model to obtain the correction result if it is detected that the question type matches the direct correction mode, the method further includes: Obtaining original training data, wherein the original training data includes question stems, reference answers, question scores, student answers, and student scores; Preprocessing the original training data to obtain pre-training data, wherein the pre-processing includes data deduplication, data screening, data verification and data expansion, and the pre-training data includes a training set and a test set; The pre-trained data is used to train a direct grading model for predicting scores for input homework data to obtain grading results. The homework data includes question stems, reference answers, question scores, and student answers.

3. The method according to claim 2, characterized in that After the step of using the pre-training data to train the direct grading model for score prediction of the inputted homework data to obtain the grading result, the method further includes: Use the trained direct grading model to score each student’s answer on the test set and give a corresponding confidence score; Setting a plurality of confidence threshold ranges, wherein the plurality of confidence threshold ranges cover all of the confidence scores; Calculate the scoring accuracy corresponding to each confidence threshold range; Screening out one or more target confidence threshold ranges whose scoring accuracy exceeds a preset accuracy threshold; The minimum confidence score in the target confidence threshold range is determined as the lowest confidence threshold, and the lowest confidence threshold is used to screen the correction results.

4. The method according to claim 1, characterized in that The step of retrieving a preset number of student answers and corresponding scores with the highest similarity from the database based on the homework data specifically includes: Inputting the homework data into a similarity model to obtain a vector representation corresponding to the student's answer; Searching in a vector database to determine a preset number of target vector representations with the highest similarity to the vector representation, wherein the vector database stores vector representations corresponding to multiple student answers, and the similarity is determined by the distance between two vectors; A preset number of student answers and corresponding scores are determined based on the target vector representation.

5. The method according to claim 4, characterized in that The step of inputting the homework data into the similarity model to obtain vector representations corresponding to the student answers; searching in the vector database to determine a preset number of target vector representations having the highest similarity to the vector representation specifically includes: Inputting the homework data into a first similarity model and a second similarity model respectively to obtain a first vector representation and a second vector representation corresponding to the student's answer, wherein the complexity of the second similarity model is higher than that of the first similarity model; Searching in a vector database to determine a first preset number of target first vector representations having the highest similarity to the first vector representation, wherein the plurality of first vector representations stored in the vector database are obtained by processing the plurality of student answers by the first similarity model; Inputting the historical operation data corresponding to the first vector representation of the target into a second similarity model to obtain a second vector representation of the target; A search is performed in the target second vector representation to determine a second preset number of target second vector representations having the highest similarity to the second vector representation, where the second preset number is smaller than the first preset number.

6. The method according to claim 4, characterized in that Before the step of inputting the homework data into the similarity model to obtain the vector representation corresponding to the student's answer, the method further includes: Acquire historical answer data, wherein the historical answer data includes question stems and student answers; Preprocessing the historical answer data to obtain historical training data, wherein the preprocessing includes data deduplication, data screening, data verification, and data expansion; Use deep learning technology to perform scoring point matching analysis on the historical training data, and automatically generate students' problem-solving ideas and scoring point scoring sequences; Based on the problem-solving ideas and scoring point score sequence, a similarity model is trained to output a vector representation corresponding to each student's answer.

7. The method according to claim 1, characterized in that After the step of retrieving a preset number of student answers and corresponding scores with the highest similarity from a database according to the homework data if it is detected that the question type matches the search and correction mode, the method further includes: Constructing a scoring data set using the preset number of student answers and corresponding scores; Using the scoring data set to incrementally train the direct grading model to obtain a single-question scoring model; The single-question scoring model is structurally adjusted to obtain a scoring model, and the scoring model is used to predict scores based on input homework data and corresponding search data to obtain a grading result.

8. An automatic marking system, characterized in that: The automatic marking system includes: one or more processors and memory; The memory is coupled to the one or more processors, and is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the automatic correction system to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on an automatic marking system, the automatic marking system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on an automatic marking system, the automatic marking system is enabled to perform the method according to any one of claims 1 to 7.

Images