Exercise automatic fine-grained labeling method, system and application based on three-stage progressive training framework

Abstract

This invention discloses an automatic fine-grained annotation system for exercises based on a three-stage progressive training framework. It innovatively employs a stage isolation mechanism of "data augmentation – independent pre-training – fusion optimization" to solve three major technical challenges: ambiguous fine-grained label boundaries, long-tailed sample distribution, and conflicting results from heterogeneous models. The core of the system includes: a data augmentation module, a language model annotation module, a statistical feature annotation module, and an annotation result fusion module. This three-stage progressive framework, through strict stage isolation and a dynamic fusion mechanism at the result level, achieves decoupling between synthetic data utilization and real data evaluation, and adaptive integration of the advantages of heterogeneous models, significantly improving the accuracy and robustness of fine-grained annotation.

Description

Technical Field

[0001] This invention belongs to the field of computer science and technology, and relates to a method, system and application of automatic fine-grained annotation of exercises based on a three-stage progressive training framework. Background Technology

[0002] In the field of education, exercise annotation is a key step in achieving intelligent education. By analyzing the content of exercises, identifying the knowledge points or other attributes (such as grade level, subject, etc.) they cover, and attaching tags to the exercise data, students and teachers can quickly identify exercise characteristics and effectively organize learning content. However, traditional exercise annotation methods have some drawbacks:

[0003] 1. Inefficiency: Manual analysis of each problem is time-consuming and costly. 2. Inconsistency issues: Different annotators may have inconsistent understandings of the same problem. 3. Poor scalability: It is difficult to quickly adapt to large-scale annotation needs.

[0004] Existing automated exercise annotation technologies can be categorized into three types:

[0005] 1. Machine learning methods based on single statistical features (such as TF-ID and XGBoost). These methods rely on manual feature engineering, making it difficult to capture the deep semantics of the exercise text, and their accuracy drops significantly when there are many label categories.

[0006] 2. Deep learning methods based on pre-trained language models (such as BERT and RoBERTa). Although these methods can model semantics, they have the following drawbacks: (1) They require a large amount of training data and perform poorly on labels with scarce samples; (2) They have difficulty classifying exercises with a wide variety of labels and small semantic differences between labels; (3) A single model is difficult to adapt to the diverse features of different disciplines and different question types at the same time.

[0007] 3. Multi-model fusion methods. Existing fusion schemes mostly use fixed weights or simple averaging, which cannot dynamically adjust the contribution of each model according to the characteristics of the sample data, and lack data augmentation mechanisms tailored to the characteristics of the exercise data.

[0008] The core challenge that this invention emphasizes in distinguishing fine-grained annotation from coarse-grained annotation lies in:

[0009] 1. Blurred label boundaries: Similar labels (such as "quadratic equation" and "economic problems involving quadratic equations") share a large number of semantic features, requiring the capture of subtle discrimination signals.

[0010] 2. Refined label sample scarcity (long-tail distribution): The more refined the label, the fewer the number of samples, leading to a higher risk of model overfitting and poor generalization ability.

[0011] 3. A single model struggles to accommodate multiple discriminative features: Text semantic understanding and statistical pattern mining represent two heterogeneous cognitive modes, and simply combining them can easily lead to feature conflicts or weight imbalances. Summary of the Invention

[0012] Purpose of the invention: To address the core problems in existing fine-grained exercise annotation technologies, such as the difficulty in discrimination due to blurred label boundaries, the scarcity of refined label samples (long-tail distribution), and the difficulty in effectively integrating the results of semantic understanding models and statistical models, this invention provides an automatic fine-grained exercise annotation method, system, and application based on a three-stage progressive training framework. Through the innovative mechanism of "stage isolation, heterogeneous independent optimization, and dynamic fusion of results," it achieves efficient integration of the advantages of heterogeneous models and effective utilization of synthetic data.

[0013] Core innovation – a three-stage progressive training framework:

[0014] Phase 1: Data Augmentation Phase (Addressing Sample Scarcity and Boundary Ambiguity)

[0015] A controlled augmentation mechanism involving multi-agent collaboration (coarse-grained similarity label extraction, fine-grained similarity label extraction, label summary generation, and exercise generation and checking) is used to generate synthetic samples with label boundary awareness. This stage is strictly isolated; the synthetic data is used only for training the basic annotation module in the second stage and does not participate in the fusion training in the third stage.

[0016] Phase Two: Labeling and Training Phase (Achieving Independent Optimization of Heterogeneous Models)

[0017] The language model annotation module and the statistical feature annotation module are trained independently and in parallel.

[0018] The language model learns deep semantics from the original text and outputs a label probability distribution;

[0019] The statistical model learns structured patterns based on statistical feature matrices (keyword frequency, TF-IDF, image features, keyword combinations) and outputs a probability distribution in a uniform form.

[0020] The two modules have independent feature spaces and a unified output format, laying the foundation for result fusion.

[0021] Phase 3: Fusion Training Phase (Achieving dynamic fusion at the result level)

[0022] The annotation result fusion module learns a set of dynamic weights based on real samples that did not participate in augmentation, and then performs weighted fusion on the probability distribution vectors output by each annotation module. The fusion strategy is driven by real data, achieving adaptive fusion where "good models contribute more and poor models cause less interference".

[0023] This invention achieves decoupling of synthetic and real data evaluation, independent optimization and result coordination of semantic understanding and statistical regularity, and unification of fine-grained label boundary clarification and long-tail sample augmentation. Experiments show that the proposed method achieves an accuracy of 0.89 on single-label datasets and 0.92 on multi-label datasets, significantly outperforming traditional methods. To achieve the above objectives, the technical solution adopted in this invention is as follows:

[0024] Overview of the Invention's Technical Principles: This invention breaks through the limitations of traditional end-to-end training or simple phased training, proposing a three-stage progressive training framework of "data augmentation - independent pre-training - fusion optimization". Its core innovation lies in systematically solving the problems of data scarcity, heterogeneous model result conflict, and fusion failure in fine-grained annotation through a strict phase isolation mechanism and a dynamic fusion mechanism at the result level.

[0025] Principle 1: Stage isolation and data flow control mechanism: The data augmentation module only runs in the first stage, and its output only flows to the second stage; the third stage only uses real samples to train the fusion weights, avoiding the contamination of the fusion strategy by synthetic data.

[0026] Principle 2, Heterogeneous Independent Optimization and Unified Result Mechanism: The language model and the statistical model have independent feature spaces and independent optimization objectives, and both output label probability distribution vectors, realizing the conversion of heterogeneous features into homogeneous prediction results.

[0027] Principle 3, dynamic fusion mechanism at the result level: The fusion operation is delayed to the prediction result level, and fusion is performed through learnable dynamic weights, with the weights optimized based on real samples using cross-entropy loss.

[0028] Principle 4, Controlled Augmentation Mechanism with Label Boundary Awareness: Through coarse-grained and fine-grained two-layer label screening and label overview constraints, combined with the generation-inspection closed loop, it is ensured that augmented samples do not cross the label boundary.

[0029] The automatic fine-grained annotation system for exercises proposed in this invention addresses the technical problems of ambiguous fine-grained label boundaries, long-tailed sample distribution, and conflicting heterogeneous features by employing a three-stage progressive training framework.

[0030] Data augmentation stage: The data augmentation module generates synthetic samples based on a controlled augmentation mechanism of multi-agent collaboration and merges them with real samples.

[0031] In the annotation training phase, the language model annotation module (learning deep semantics) and the statistical feature annotation module (learning statistical patterns) are trained separately. The two modules are optimized independently, and the feature space is kept consistent through a unified statistical feature extraction module.

[0032] Fusion training phase: The training annotation result fusion module, based on real samples that did not participate in data augmentation, fuses the probability outputs of each annotation module with learnable dynamic weights to obtain the optimal fusion weights.

[0033] The entire system includes the following modules:

[0034] The data augmentation module is used to generate augmented exercise data based on the original labeled training dataset through multi-agent interaction, thereby improving the sample richness of the training dataset. The data augmentation module includes a coarse-grained similar label extraction agent, a fine-grained similar label extraction agent, a label overview extraction agent, an exercise generation agent, and an exercise checking agent. The data augmentation module only runs during the labeled training phase, and its output is merged with the original labeled training dataset to form the augmented labeled training dataset.

[0035] The exercise statistical feature matrix extraction module is used to extract statistical features from exercise data and construct a feature matrix through statistical analysis methods. The statistical features include keyword feature frequency, TF-IDF vector, image features, and keyword combination features. This module receives the augmented labeled training dataset in the labeled training stage, the fused training dataset in the fused training stage, and the exercise data to be labeled in the labeled stage, and outputs the corresponding statistical feature matrix.

[0036] The language model annotation module is used to perform semantic understanding and label prediction on exercise data based on a deep learning language model. The language model annotation module includes a pre-trained language model, a word segmenter, and a trainable classification head. During the annotation training phase, the system is trained on the augmented annotated training dataset. During the fusion training phase and the annotation phase, the system outputs the label probability distribution vectors of the fusion training dataset or the exercise data to be annotated, respectively. Multiple language model annotation modules can be configured simultaneously in the system.

[0037] A statistical feature annotation module is used for label prediction based on a statistical feature matrix. The statistical feature annotation module includes a statistical feature annotation model. During the annotation training phase, the model is trained based on the statistical feature matrix corresponding to the augmented annotation training dataset. During the annotation phase, the model outputs the label probability distribution vector of the exercise data to be annotated. Multiple statistical feature annotation modules can be configured simultaneously in the system.

[0038] The annotation result fusion module is used to fuse the label probability distribution vectors output by all language model annotation modules and statistical feature annotation modules in the system, generate the fused label probability distribution vector, and complete the final annotation. The annotation result fusion module includes a set of learnable fusion weights, the number of which is equal to the total number of all annotation modules in the system. This module runs during the fusion training phase and the annotation phase, but does not participate in the annotation training phase.

[0039] The operation of the system described in this invention includes the following four stages:

[0040] In the data augmentation stage, only the data augmentation module is run. It generates augmented exercise data based on the original labeled training dataset and merges the output with the original labeled training dataset to form the augmented labeled training dataset.

[0041] During the annotation training phase, only the language model annotation module and the statistical feature annotation module are trained independently; the data augmentation module and the annotation result fusion module are not involved.

[0042] During the fusion training phase, only the annotation result fusion module is trained; the data augmentation module is not involved.

[0043] The above three stages constitute the core innovation of this invention: the progressive three-stage training method.

[0044] In the annotation phase, which is the actual application phase of the system, the data augmentation module does not participate. The system receives the exercise data to be annotated and outputs the final annotation results.

[0045] The data augmentation stage of this invention includes the following steps:

[0046] The original labeled exercise dataset was divided into a labeled training dataset, a fused training dataset, and a labeled test dataset in an 8:1:1 ratio.

[0047] Input the labeled training dataset into the data augmentation module to generate an augmented labeled training dataset, and merge it with the original labeled training dataset to form an augmented labeled training dataset.

[0048] The annotation training phase of this invention includes the following steps:

[0049] Input the augmented labeled training dataset into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix;

[0050] All statistical feature annotation modules in the system are trained based on the aforementioned statistical feature matrix;

[0051] All language model annotation modules in the system are trained based on the augmented labeled training dataset.

[0052] The fusion training phase of this invention includes the following steps:

[0053] Input the fused training dataset into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix;

[0054] Input the statistical feature matrix into all statistical feature annotation modules to obtain the label probability distribution vector output by each module;

[0055] Input the fused training dataset into all language model annotation modules and obtain the label probability distribution vectors output by each module;

[0056] A labeling result fusion module is trained based on the label probability distribution vectors output by all labeling modules.

[0057] The annotation stage described in this invention includes the following steps:

[0058] Input the exercise data to be labeled into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix;

[0059] Input the statistical feature matrix into all statistical feature annotation modules to obtain the label probability distribution vector output by each module;

[0060] Input the exercise data to be labeled into all language model labeling modules and obtain the label probability distribution vector output by each module;

[0061] Input the label probability distribution vectors output by all annotation modules into the annotation result fusion module to generate the final annotation result of the exercise data to be annotated.

[0062] The labeled exercise dataset described in this invention contains at least the following fields:

[0063] The ID field is used to uniquely identify the exercise data;

[0064] The Content field is used to store the text content of the exercises.

[0065] The operation of the data augmentation module described in this invention includes the following steps:

[0066] We extract agents using coarse-grained similarity labels, extracting coarse-grained similarity labels for each label from the exercise dataset based on the literal semantics of the label text;

[0067] The agent is extracted using fine-grained similarity labels. Several examples are extracted for each label and its coarse-grained similarity labels. Fine-grained similarity labels are extracted from the coarse-grained similarity labels through comparative analysis.

[0068] The agent is extracted using the tag overview. Several example problems are extracted for each tag and its fine-grained similar tags. The similarities and differences between the tag problems and the fine-grained similar tag problems are analyzed to generate the tag overview.

[0069] A problem-generating agent is used to randomly select labeled problems and fine-grained similar labeled problems. Based on the label overview information, the labeled problems are data augmented to generate new problems.

[0070] The agent is checked using exercises, and newly generated exercises are checked against the label overview to see if they meet the augmentation criteria. Exercises that meet the criteria are added to the augmented dataset.

[0071] The coarse-grained similarity label extraction agent of this invention is based on label semantics and selects up to 5 candidate labels that are similar to the target label from all exercise labels.

[0072] The fine-grained similarity label extraction agent of this invention extracts up to three fine-grained similarity labels by comparing the differences between label exercises and coarse-grained similarity label exercises, based on the coarse-grained similarity labels.

[0073] The tag overview extraction agent described in this invention extracts two example questions for each tag and its fine-grained similar tags, and generates a tag overview that includes the characteristics of the tag and the similarities and differences with the fine-grained similar tags through comparative analysis.

[0074] The exercise generation agent of this invention expands the number of exercises under each label in the training set to the average number of exercises level where the number of exercises under each label is less than the average number of exercises. On this basis, it further expands 10 exercises under each label. The exercise generation agent randomly selects one label exercise based on the label overview information, 3 label exercises and 3 fine-grained similar label exercises, modifies and enhances it, and generates a new exercise.

[0075] The exercise checking agent described in this invention checks whether newly generated exercises conform to the label overview information and whether there are cases that directly mention the label information. Exercises that do not conform to the overview information or directly mention the label information are discarded, while those that do are retained and added to the augmented dataset.

[0076] The feature extraction process of the exercise statistical feature matrix extraction module of the present invention includes:

[0077] Keyword frequency is obtained by statistically analyzing the occurrence frequency of each keyword in the exercise text.

[0078] The TF-IDF values ​​of each keyword in the exercise text are calculated using the TF-IDF algorithm, and a TF-IDF vector is constructed.

[0079] Extract image type, content description, and key element features from exercises containing images;

[0080] Keyword combinations are selected based on the relevance between the exercise content and the tags to form keyword combination characteristics.

[0081] The feature extraction process remains consistent throughout the annotation training phase, the fusion training phase, and the annotation phase.

[0082] It should be noted that the statistical feature matrix extraction module for the exercises maintains a consistent workflow across the three stages of the system, differing only in the input data: in the labeled training stage, the input is the augmented labeled training dataset; in the fused training stage, the input is the fused training dataset; and in the labeled stage, the input is the exercise data to be labeled. The feature extraction algorithms, parameter configurations, and output formats are identical across all three stages, ensuring consistency in the feature space and the reusability of the module.

[0083] The statistical feature annotation module of this invention uses the XGBoost model, and the training of the XGBoost model includes the following steps:

[0084] Initialization steps: Set initial predicted values ​​for all samples;

[0085] Iterative training steps: For each iteration, calculate the first-order gradient and second-order ladder of the loss function with respect to the current predicted value; construct a decision tree based on the gradient and second-order gradient, and select the optimal split node by calculating the split gain; weight and accumulate the predicted values ​​of the newly constructed decision tree to the current model's predicted value.

[0086] Convergence judgment steps: Stop training when the preset maximum number of iterations is reached or the model converges;

[0087] The XGBoost model contains multiple decision trees, each corresponding to the prediction of a label, and the objective function includes training loss and regularization term.

[0088] The training of the language model annotation module described in this invention includes multiple iterations until a preset number of training rounds is reached or the loss function value satisfies the convergence condition; a single training round includes:

[0089] Forward propagation steps: The exercise text is processed by a word segmenter to obtain a word sequence, which is then input into a pre-trained language model to obtain the features of the last hidden layer; for models with CLS tags, the features corresponding to the CLS tags are extracted, and for models without CLS tags, average pooling is performed on the last hidden layer; the extracted features are processed by an activation function and a linear classification head to output a label probability distribution vector.

[0090] Backpropagation steps: Calculate the loss between the predicted output and the true label based on the cross-entropy loss function, and update the model parameters using the gradient descent algorithm.

[0091] Among these, training continues until the preset number of training rounds is reached, the loss value is less than the preset threshold, or the loss does not decrease for several consecutive rounds.

[0092] The process of generating label probability distribution vectors by the statistical feature annotation module of the present invention includes:

[0093] Obtain the feature vector and initial predicted value of the sample to be predicted;

[0094] The feature vectors are input into the set of decision trees obtained through training, and the original prediction scores are calculated based on the weights output by the leaf nodes of each decision tree.

[0095] The original predicted scores are converted into a normalized label probability distribution vector using the Softmax function.

[0096] The training of the annotation result fusion module described in this invention includes multiple iterations until the loss function value meets the convergence condition or reaches the preset number of training rounds; a single training round includes:

[0097] Forward propagation steps: The learnable parameter vector is converted into normalized weights for each annotation module using the Softmax function. The label probability distribution vectors output by each annotation module are weighted and summed based on the weights to generate the fused label probability distribution vector.

[0098] Backpropagation steps: Calculate the loss between the fused label probability distribution vector and the real label based on the cross-entropy loss function, and update the learnable parameter vector through the gradient descent algorithm.

[0099] Based on the above, this invention proposes an automatic fine-grained annotation method for exercises based on the fusion of data-augmented statistical features and language models, comprising the following steps:

[0100] Data augmentation steps: Based on the original labeled training dataset, augmented exercise data is generated through multi-agent interaction, and then merged to form the augmented labeled training dataset;

[0101] Feature extraction steps: Extract the statistical feature matrix of the exercises from the augmented labeled training dataset. The statistical features include keyword feature frequency, TF-IDF vector, image features, and keyword combination features.

[0102] The steps for training the labeled model are as follows: Based on the augmented labeled training dataset and its statistical feature matrix, the language model labeling module and the statistical feature labeling module are trained independently.

[0103] Fusion training steps: Based on the fusion training dataset, train the annotation result fusion module to learn the optimal fusion weights of each annotation module;

[0104] Annotation steps: Input the exercise data to be annotated into each trained module, and output the final annotation results through the annotation result fusion module.

[0105] Based on the above, this invention also proposes the application of an automatic fine-grained annotation system for exercises in automatic annotation, educational document classification, and automatic summarization of knowledge points. Attached Figure Description

[0106] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a functional diagram of all modules of the system of the present invention.

[0108] Figure 2 This is a detailed flowchart of the data augmentation module of this invention.

[0109] Figure 3 This is a schematic diagram of the three-stage process described in this invention.

[0110] Figure 4 This is a schematic diagram of the training and inference (testing) process of the language model annotation module described in this invention.

[0111] Figure 5 This is a schematic diagram of the training and inference (testing) process of the annotation result fusion module described in this invention. Detailed Implementation

[0112] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other. The processes, conditions, and experimental methods for implementing this invention, except as specifically mentioned below, are all common knowledge and general information in the field, and this invention does not impose any particular limitations.

[0113] The automatic fine-grained annotation method and system technical solution for exercises based on a three-stage progressive training framework proposed in this invention are as follows:

[0114] The first component of the technical solution: an automatic fine-grained annotation system for exercises.

[0115] This invention provides an automatic fine-grained annotation system for exercises based on a three-stage progressive training framework. The system includes the following five core modules:

[0116] The data augmentation module operates only during the system's annotation and training phase. Its core function is to generate high-quality augmented exercise data through a multi-agent collaborative workflow. This module receives the original labeled training dataset as input and outputs an augmented labeled training dataset, which is then merged with the original dataset to form the augmented labeled training dataset, used for training subsequent modules. The data augmentation module comprises five components: a coarse-grained similarity label extraction agent, a fine-grained similarity label extraction agent, a label overview extraction agent, an exercise generation agent, and an exercise checking agent. These agents collaborate to complete the data augmentation task.

[0117] The exercise statistical feature matrix extraction module runs in all three phases of the system (annotation training phase, fusion training phase, and annotation phase), responsible for extracting multidimensional statistical features from the exercise data and constructing a feature matrix. It's important to note that the feature extraction process is identical across all three phases, differing only in the input data source: in the annotation training phase, the input is the augmented annotated training dataset; in the fusion training phase, the input is the fusion training dataset; and in the annotation phase, the input is the exercise data to be annotated. The feature extraction algorithms, parameter configurations, and output formats remain consistent across all three phases to ensure consistency in the feature space and module reusability. The statistical features extracted by this module include four categories: keyword feature frequency, TF-IDF vectors, image features, and keyword combination features.

[0118] The language model annotation module and the statistical feature annotation module, collectively referred to as the annotation module, are the core prediction units of the system. The language model annotation module is based on a pre-trained language model and focuses on deep semantic understanding of the exercise text, including the pre-trained language model, a word segmenter, and a trainable classification head. The statistical feature annotation module is based on the XGBoost gradient boosting tree model and focuses on modeling statistical features. Both modules are trained independently during the annotation training phase and run in parallel during the fusion training and annotation phases, outputting label probability distribution vectors respectively. Multiple language model annotation modules and multiple statistical feature annotation modules can be configured simultaneously in the system to improve its generalization performance.

[0119] The annotation result fusion module is responsible for integrating the prediction results from various annotation modules. This module contains a set of learnable fusion weights, the number of which is equal to the total number of annotation modules in the system. After Softmax normalization, the label probability distribution vectors output by each module are weighted and summed to generate the final fused label probability distribution vector, which is used to determine the final label of the exercise. This module only runs during the fusion training and annotation phases, and does not participate in the annotation training phase. During the fusion training phase, the module trains the weights based on the fusion training dataset composed of original real samples that have not undergone data augmentation; during the annotation phase, inference is performed based on the trained dynamic weights.

[0120] The system operates in three phases: the annotation training phase, in which only the data augmentation module, the exercise statistical feature matrix extraction module, the language model annotation module, and the statistical feature annotation module participate, while the annotation result fusion module does not participate; the fusion training phase, in which only the exercise statistical feature matrix extraction module, the language model annotation module, the statistical feature annotation module, and the annotation result fusion module participate, while the data augmentation module does not participate; and the annotation phase, which is the actual application phase of the system, in which the data augmentation module does not participate, and the remaining modules work together to complete the exercise annotation.

[0121] The second component of the technical solution: System operation phase

[0122] The automatic fine-grained annotation system for exercises as described in technical solution one is characterized by three stages from initial system setup to actual annotation participation:

[0123] Phase 1: Data Augmentation Phase

[0124] The data augmentation stage only runs the data augmentation module, which generates augmented exercise data based on the original labeled training dataset, and outputs the data that is merged with the original labeled training dataset to form the augmented labeled training dataset.

[0125] The data augmentation phase follows these steps:

[0126] Step 1: Select and divide the existing labeled exercise dataset into labeled training dataset, fused training dataset, and labeled test dataset in an 8:1:1 ratio, which correspond to the subsequent stages 1 (labeled training stage), 2 (fused training stage), and 3 (labeling stage), respectively.

[0127] Step 2: Input the labeled training dataset into the data augmentation module to obtain the augmented labeled training dataset, and merge it with the original labeled training dataset to form the augmented labeled training dataset.

[0128] Phase Two: Labeling and Training Phase

[0129] The annotation training phase only trains the annotation modules on the annotation training dataset, and each annotation module is trained independently. The annotation result fusion module does not participate in this phase.

[0130] The process for each module in the annotation training phase follows these steps:

[0131] Step 1: Input the augmented labeled training dataset into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the augmented labeled training dataset.

[0132] Step 2: Train all statistical feature annotation modules in the system using the statistical feature matrix of the augmented labeled training dataset.

[0133] Step 3: Train all language model annotation modules in the system using the augmented labeled training dataset.

[0134] Phase Three: Integration Training Phase

[0135] The fusion training phase only trains the annotation fusion module on the fusion training dataset. The data augmentation module does not participate in this phase.

[0136] The process for each module in the fusion training phase follows these steps:

[0137] Step 1: Input the fused training dataset into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the fused training dataset.

[0138] Step 2: Input the statistical feature matrix of the fused training dataset into all statistical feature annotation modules in the system to obtain the label probability vector of the fused training dataset output by all statistical feature annotation modules.

[0139] Step 3: Input the fused training dataset into all language model annotation modules in the system to obtain the label probability vectors output by all language model annotation modules.

[0140] Step 4: Merge the label probability vector training annotation results of the training dataset output by all modules into a single module.

[0141] Phase Four: Labeling Phase

[0142] The annotation phase is the practical application phase of the system, where the annotation results are output for the input exercise data. The data augmentation module does not participate in this phase.

[0143] The process for each module in the annotation phase follows these steps:

[0144] Step 1: Input the data of exercises to be labeled (test data / real data) into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the exercises to be labeled.

[0145] Step 2: Input the statistical feature matrix of the exercises to be labeled into all statistical feature labeling modules in the system to obtain the label probability vector of the exercises to be labeled output by all statistical feature labeling modules.

[0146] Step 3: Input the exercises to be labeled into all language model annotation modules in the system to obtain the label probability vectors of the exercises to be labeled output by all language model annotation modules.

[0147] Step 4: Input the label probability vectors of the exercises to be labeled from all modules into the labeling result fusion module to obtain the final labeling result of the exercises to be labeled.

[0148] The third component of the technical solution: Dataset field requirements

[0149] The automatic fine-grained annotation system for exercises as described in technical solution 2 is characterized in that, in step one of stage one, the labeled exercise dataset contains at least the following required fields, while other fields are compatible:

[0150] Field 1: ID, a unique identifier for the exercise data.

[0151] Field 2: content, the text content of the exercise data.

[0152] The fourth component of the technical solution: Detailed process of the data augmentation module

[0153] The data augmentation module described in step two of phase one of technical solution two is characterized in that the specific interaction steps of each component of the data augmentation module are as follows:

[0154] Step 1, Coarse-grained Similarity Label Extraction: A coarse-grained similarity label extraction agent is used to extract coarse-grained similarity labels for each label in the exercise dataset according to the literal meaning of the label text. Specifically, in this coarse-grained similarity label extraction stage, the coarse-grained extraction agent selects up to 5 candidate labels that are similar to the label from all exercise labels based on the label semantics.

[0155] Step two, fine-grained similarity tag extraction: The fine-grained similarity tag extraction agent extracts several example questions for each tag and several example questions for each tag's coarse-grained similarity tags. The fine-grained similarity tag extraction agent then uses these example questions to further extract fine-grained similarity tags from the coarse-grained similarity tags. Specifically, in this stage, the fine-grained extraction agent, based on the extracted coarse-grained similarity tags, provides pre-defined tags and corresponding exercises to further extract up to three finer-grained similarity tags.

[0156] Step 3, Tag Overview Generation:

[0157] A tag summary extraction agent is used to extract several example problems for each tag and its fine-grained similar tags. The agent analyzes the similarities and differences between the tag-specific exercises and the fine-grained tag-specific exercises to generate a tag summary. Specifically, the tag summary extraction agent compares the semantic information of the tag, the data information of the corresponding exercise, and the exercise information under the fine-grained similar tags of that tag to obtain the tag summary. The tag summary includes the characteristics of the tag and its similarities and differences with the corresponding fine-grained similar tags, serving as contextual instructions for the exercise generation agent.

[0158] Step 4, Exercise Generation: The labeled exercises are augmented using an exercise generation agent. This agent randomly selects labeled exercises and fine-grained similar labeled exercises. Based on the labeled exercises, enhancements are made using the summary information of the labels. Specifically, labels with fewer exercises under each label than the average number of exercises in the training set are augmented to the average number of exercises. After augmentation to the average, 10 more exercises are added to each label. The exercise generation agent receives the label summary information, three labeled exercises, and three fine-grained similar labeled exercises. It randomly selects one labeled exercise for modification and enhancement, resulting in a newly generated enhanced exercise. The modification methods include expanding the question's explanation, restating the question's background, and modifying the answer.

[0159] Step 5, Exercise Checking: The exercise checking agent checks whether the newly generated exercises meet the augmentation criteria by comparing them with the label summary. If they do, the generated exercises are retained and added to the training set. Specifically, the exercise checking agent receives new exercises and label summary information, checks whether the exercises match the label summary information and whether they directly mention the label information. Exercises that do not match the summary information or directly mention the label information are discarded; otherwise, they are retained and added to the augmented dataset. Fifth component of the technical solution: Detailed process of statistical feature extraction.

[0160] The exercise statistical feature matrix extraction module described in step one of stage two of technical solution two is characterized in that the process of using the exercise statistical feature matrix extraction module is the same in all three stages of the system (step one of stage two, step one of stage two, and step one of stage three). The specific extraction process is as follows:

[0161] Step 1: Obtain the keyword feature frequency by statistically analyzing the frequency of occurrence of each keyword in the exercise text, thus providing basic text feature information.

[0162] Step 2: Calculate the TF-IDF value of each keyword in the exercise text using the TF-IDF algorithm program, construct a TF-IDF vector, and focus on the key information in the text.

[0163] Step 3: Extract features from the images in the exercises that contain pictures, including image type, content description, and key elements.

[0164] Step 4: The keyword combination features are used to filter keyword combinations based on the relationship between the exercise content and the tag relevance.

[0165] The sixth component of the technical solution: Detailed implementation of the statistical feature annotation module.

[0166] The statistical feature annotation module described in step two of phase two of technical solution two is characterized in that the annotation model of the statistical feature annotation module is XGBoost.

[0167] XGBoost (eXtreme Gradient Boosting) is an efficient machine learning algorithm based on Gradient Boosting Decision Trees (GBDT). It constructs a strong learner by combining a series of weak learners (decision trees). The goal of XGBoost is to optimize model performance by minimizing a loss function. The objective function typically consists of two parts: training loss (such as mean squared error, log loss, etc.) and a regularization term (used to prevent overfitting).

[0168] In the exercise annotation, XGBoost contains M tree groups, where M is the number of label sets (the length of the label probability vector), and each tree group has K trees, with each tree group focusing on the annotation of a specific label.

[0169] The training process for XGBoost is as follows:

[0170] Step 1: Initial Predictions: Typically, the initial predicted values ​​for all samples are set to a constant (such as the mean or median of the target value) as the starting point for the first round of iterations.

[0171] Step 2, Iterative Training: XGBoost builds the model step by step through multiple iterations, generating a new decision tree in each iteration and continuously optimizing the objective function.

[0172] (1) Calculate the gradient and second derivative: For each sample, calculate the first derivative (gradient) and second derivative (Hessian) of the loss function with respect to the current prediction value.

[0173] (2) Constructing a decision tree: A greedy algorithm is used to select the optimal split point by calculating the split gain of each feature. The split gain measures the degree of improvement of the objective function after splitting. Nodes are continuously split from the root node until the maximum depth is reached or the split gain is less than a threshold. Pruning is performed by limiting the depth of the tree, the number of leaf nodes, or setting a minimum split gain to prevent overfitting.

[0174] (3) Update model predictions: After each decision tree is trained, its predictions are weighted according to the learning rate and added to the predictions of the current model.

[0175] Step 3, End of Iteration: The training process ends when the preset maximum number of iterations is reached, the model converges (the improvement of the objective function is less than the threshold), or the training error reaches a satisfactory level.

[0176] The seventh component of the technical solution: Detailed implementation of the language model annotation module

[0177] As described in step three of phase two of technical solution two, the language model annotation module is characterized in that the language model annotation module undergoes multiple rounds of training until a preset number of training rounds is reached, or the output cross-entropy loss is less than a preset threshold, or the cross-entropy loss value does not decrease after more than 10 rounds.

[0178] The training process for a single round is as follows:

[0179] Forward propagation (which is identical in all three phases of the system):

[0180] Step 1: The question text is segmented by a tokenizer to obtain tokens.

[0181] Step 2: Extract information from the obtained tokens through the last hidden layer of the natural language processing model. For models with CLS-tokens, extract the CLS-tokens; for models without CLS-tokens, apply average pooling to the last hidden layer.

[0182] Step 3: Process the text signal through a linear layer to obtain the output signal.

[0183] 2. Backpropagation (performed only during the annotation training phase):

[0184] Step 1: Calculate the cross-entropy loss of the current output by combining the output signal and the label using the cross-entropy loss function.

[0185] Step 2: Backpropagate the error, calculate the error layer by layer from back to front, and perform gradient calculation.

[0186] Step 3: Parameter update, using gradient descent to update network parameters.

[0187] The eighth component of the technical solution: Statistical feature annotation module prediction process

[0188] The statistical feature annotation module described in step two of phase two of technical solution two is characterized in that the statistical feature annotation module generates the label probability vector in the following manner, and the process of generating the label probability vector using the statistical feature annotation module is the same in both phases of the system (step two of phase two and step two of phase three):

[0189] Step 1: Obtain the initial prediction value: Obtain the feature vector of the sample to be predicted, and extract the initial prediction value determined during the XGBoost training phase as the baseline value.

[0190] Step 2, Tree-based cumulative calculation: Input the feature vector into a set of decision trees obtained through multiple rounds of iterative training, where each decision tree maps the feature vector to the corresponding leaf node based on the optimal split node determined during the training phase, and obtains the output weight of each leaf node; the output weights of all decision trees are weighted and accumulated with the initial prediction value according to the learning rate to obtain the original prediction score vector of the sample.

[0191] Step 3, Probability Transformation: The original predicted score vector is nonlinearly transformed using a preset link function to map the scores in the real number domain to the probability interval [0,1], generating a normalized label probability distribution vector. The link function is the Softmax function.

[0192] The ninth component of the technical solution: Detailed implementation of the annotation result fusion module.

[0193] The annotation result fusion module described in step four of phase three of technical solution two is characterized in that the annotation result fusion module undergoes multiple rounds of training until the loss is less than a preset threshold or a specified number of training rounds is reached.

[0194] The training steps for a single round are as follows:

[0195] 1. Forward propagation (completely identical in the fusion training and annotation phases):

[0196] Step 1: Use the Softmax function to convert the learnable parameter vector into a probability distribution to obtain the normalized weights of each annotation module.

[0197] Step 2: Sum the label probability vectors output by each annotation module to obtain the final probability vector. The label with the highest probability in the final probability vector is taken as the predicted label.

[0198] 2. Backpropagation (performed only during the fusion training phase) is the same as in Component VII.

[0199] Component 10 of the technical solution: Application Area

[0200] The automatic fine-grained annotation method for exercises as described in any one of technical solutions one through nine, or the application of the exercise annotation system as described in technical solution one in exercise annotation, document classification, and knowledge point summarization.

[0201] Example 1: System Overall Architecture

[0202] like Figure 1As shown, this embodiment provides an automatic fine-grained annotation system for exercises based on a three-stage progressive training framework. The system includes a data augmentation module, an exercise statistical feature matrix extraction module, a language model annotation module, a statistical feature annotation module, and an annotation result fusion module.

[0203] The data augmentation module operates only during the system's annotation and training phase. Its core function is to generate high-quality augmented exercise data through a multi-agent collaborative workflow. This module receives the original labeled training dataset as input and outputs an augmented labeled training dataset, which is then merged with the original dataset to form the augmented labeled training dataset, used for training subsequent modules. The data augmentation module comprises five components: a coarse-grained similarity label extraction agent, a fine-grained similarity label extraction agent, a label overview extraction agent, an exercise generation agent, and an exercise checking agent. These agents collaborate to complete the data augmentation task.

[0204] The exercise statistical feature matrix extraction module runs in all three phases of the system (annotation training phase, fusion training phase, and annotation phase), responsible for extracting multidimensional statistical features from the exercise data and constructing a feature matrix. It's important to note that the feature extraction process is identical across all three phases, differing only in the input data source: in the annotation training phase, the input is the augmented annotated training dataset; in the fusion training phase, the input is the fusion training dataset; and in the annotation phase, the input is the exercise data to be annotated. The feature extraction algorithms, parameter configurations, and output formats remain consistent across all three phases to ensure consistency in the feature space and module reusability. The statistical features extracted by this module include four categories: keyword feature frequency, TF-IDF vectors, image features, and keyword combination features.

[0205] The language model annotation module and the statistical feature annotation module, collectively referred to as the annotation module, are the core prediction units of the system. The language model annotation module is based on a pre-trained language model and focuses on deep semantic understanding of the exercise text, including the pre-trained language model, a word segmenter, and a trainable classification head. The statistical feature annotation module is based on the XGBoost gradient boosting tree model and focuses on modeling statistical features. Both modules are trained independently during the annotation training phase and run in parallel during the fusion training and annotation phases, outputting label probability distribution vectors respectively. Multiple language model annotation modules and multiple statistical feature annotation modules can be configured simultaneously in the system to improve its generalization performance.

[0206] The annotation result fusion module is responsible for integrating the prediction results from various annotation modules. This module contains a set of learnable fusion weights, the number of which is equal to the total number of annotation modules in the system. After Softmax normalization, the label probability distribution vectors output by each module are weighted and summed to generate the final fused label probability distribution vector, which is used to determine the final label of the exercise. This module only runs during the fusion training and annotation phases, and does not participate in the annotation training phase. During the fusion training phase, the module trains the weights based on the fusion training dataset composed of original real samples that have not undergone data augmentation; during the annotation phase, inference is performed based on the trained dynamic weights.

[0207] The system operates in three phases: the annotation training phase, in which only the data augmentation module, the exercise statistical feature matrix extraction module, the language model annotation module, and the statistical feature annotation module participate, while the annotation result fusion module does not participate; the fusion training phase, in which only the exercise statistical feature matrix extraction module, the language model annotation module, the statistical feature annotation module, and the annotation result fusion module participate, while the data augmentation module does not participate; and the annotation phase, which is the actual application phase of the system, in which the data augmentation module does not participate, and the remaining modules work together to complete the exercise annotation.

[0208] Example 2: Detailed Implementation of Multi-Agent Data Augmentation

[0209] like Figure 2 As shown, this embodiment describes in detail the workflow of the data augmentation module. This module employs a five-agent collaborative architecture, including a coarse-grained similarity label extraction agent, a fine-grained similarity label extraction agent, a label overview extraction agent, a problem generation agent, and a problem checking agent.

[0210] Step S201: Coarse-grained similarity tag extraction

[0211] The coarse-grained similarity label extraction agent selects candidate labels that are semantically similar to the target label from all exercise labels based on the literal semantics of the label text. Specifically, the coarse-grained similarity label extraction agent selects up to 5 candidate labels that are similar to the target label from all exercise labels based on the label semantics.

[0212] In one specific implementation, the coarse-grained similarity label extraction agent uses the following prompt:

[0213] "You are a middle school math teacher. You need to find up to 5 knowledge points similar to the target knowledge point from a given list of candidate knowledge points. Output only one list, where each element must be from the candidate knowledge point list; no other content is required."

[0214] For example: ['Quadratic equation in one variable', 'Economic problems involving quadratic equations in one variable', 'Maximum value of a quadratic function']\n

[0215] Target knowledge points: {Target knowledge points}\n

[0216] Candidate knowledge point list: {candidate knowledge points}

[0217] Step S202: Fine-grained similarity tag extraction

[0218] The fine-grained similarity label extraction agent further refines the similarity label set by comparing and analyzing exercise examples, building upon the coarse-grained similarity labels. For each target label and its coarse-grained similarity labels, two example questions (including the question stem and explanation) are randomly selected. The agent compares the similarities and differences between these example questions and selects up to three fine-grained similarity labels from the coarse-grained similarity labels that are most similar to the target label.

[0219] The core of this step lies in using specific exercise examples to distinguish subtle differences between labels. For example, "quadratic equation" and "economic problems involving quadratic equations" are highly similar in literal meaning, but by comparing their actual exercises, it can be found that the latter specifically refers to equation problems involving economic applications. This subtle difference can only be accurately identified through specific examples. In one specific implementation, the fine-grained similarity label extraction agent uses the following prompt words:

[0220] "You are a middle school math teacher. You need to find at most 23 and at least 1 knowledge point that is strongly similar to a target knowledge point from a given list of candidate knowledge points. Two example problems, including the problem statement and explanation, will be provided for both the target and candidate knowledge points. The final output should be a list where all elements are from the candidate knowledge point list; do not output any other content."

[0221] For example: ['Quadratic Equation', 'Economic Problems Related to Quadratic Equations']\n

[0222] Target knowledge points: {Target knowledge points}\n

[0223] Candidate knowledge point list: {candidate knowledge points}\n

[0224] Example 1 and Example 2 for the target knowledge point.

[0225] Example 1 and Example 2 of {Candidate Knowledge Point 1}

[0226] Example 1 and Example 2 of {Candidate Knowledge Point 2}

[0227] ..."

[0228] Step S203: Tag Overview Generation

[0229] The tag overview extracts exercise instances based on the target tag and its fine-grained similar tags from the agent, generating a structured tag overview. This overview includes two parts: first, an introduction to the core concepts of the tags; and second, the specific characteristics of the tags in the application of the exercises, especially the key differences from fine-grained similar tags.

[0230] The process of generating the tag overview is as follows: For the target tag and each of its fine-grained similar tags, two example questions are extracted; the agent compares and analyzes these example questions to identify the common features of the target tag questions and the differences between them and the similar tag questions; and a tag overview described in natural language is generated based on the analysis results.

[0231] In one specific implementation, the tag summary extraction agent uses the following prompt words:

[0232] "You are a middle school math teacher. You need to compare a target knowledge point with its similar knowledge points (each knowledge point will be given two example questions, including the question stem and explanation). Generate a knowledge point overview, including an introduction to the knowledge point and the characteristics of its application in exercises."

[0233] Note! Only output a summary of the target knowledge points; do not output any extra content!

[0234] For example, the knowledge point introduction: Economic problems involving systems of inequalities mainly involve the calculation and optimization of economic indicators such as profit, cost, selling price, and loss rate. Typically, it requires establishing a system of inequalities based on the problem's conditions and solving for the range of variables that satisfy a specific economic objective (such as minimum profit, maximum cost, etc.). These types of problems often appear in practical scenarios such as commodity sales and investment decisions, requiring students to master profit formulas, profit rate calculations, and the modeling and solving of inequalities.

[0235] Characteristics of Exercise Application: Core Relationship: Inequalities are usually established based on formulas such as "Profit = Selling Price − Cost" and "Profit Margin = Profit / Cost × 100%".

[0236] Variable setting: Unknowns need to be clearly defined (such as the percentage increase in selling price, unit price, etc.), and changes in losses, costs, etc. should be expressed reasonably.

[0237] Inequality construction: Determine the direction of the inequality sign based on keywords such as "not less than" and "at least", and integrate multiple constraints in combination with the conditions of the problem.

[0238] Practical significance verification: The range of the solution must conform to reality (e.g., the percentage increase in selling price cannot be negative).

[0239] Calculation precision: The result may need to be accurate to a specific number of decimal places.

[0240] Differences from similar knowledge points (solution selection problems):

[0241] Economic problems focus on calculating economic indicators such as profit and cost, while solution selection problems focus on finding feasible solutions under constraints (such as the number of equipment and transportation capacity), and usually involve integer solutions.

[0242] Target knowledge points: {Target knowledge points}

[0243] List of similar knowledge points: {Fine-grained tag list}

[0244] Example 1 and Example 2 for the target knowledge point.

[0245] Example 1 and Example 2 of {Fine-grained similarity knowledge point 1}

[0246] Example 1 and Example 2 of {Fine-grained similarity knowledge point 2}

[0247] ..."

[0248] Step S204: Exercise Generation

[0249] The exercise generation agent generates new exercise data based on the label summary information. The data generation strategy follows these principles: First, calculate the average number of exercises for each label in the training set; for labels with a number of exercises below the average, generate new exercises until the average is reached; on this basis, generate an additional 10 exercises for each label to enhance data diversity.

[0250] When generating exercises, the agent receives the following inputs: an overview of the current label, three example questions for the current label, and three example questions with fine-grained similar labels. The agent randomly selects one example question from the current label as a base and uses a randomly chosen modification strategy (expanding the original question analysis, restating the question's background, or modifying the question's answer) to generate new exercises. By introducing example questions with fine-grained similar labels as a reference, the agent can better grasp the boundary between the current label and similar labels, avoiding the generation of ambiguous or erroneous samples.

[0251] In one specific implementation, the exercise-generating agent uses the following prompt words:

[0252] "You are a middle school math teacher. Now you need to generate a new problem by referring to the example problems. The key point is that the new problem can test the relevant knowledge points. The input includes the knowledge points and an overview of the knowledge points, 3 example problems of the knowledge points, and 3 example problems similar to the knowledge points."

[0253] The output should only contain the question stem and the solution text, without any other content. In particular, avoid emphasizing the knowledge points related to the question.

[0254] Please use the following method to generate a new question: {"Expand the question based on the original question's analysis", "Restate the question's background", or "Modify the question's answer"}.

[0255] Knowledge Points: {Tags}

[0256] Knowledge Point Overview: {Tag Overview}

[0257] Exercises to be revised: {Exercises to be revised}

[0258] Example 1 and Example 2 for the target knowledge point.

[0259] Example 1 and Example 2 of {Fine-grained similarity knowledge point 1}

[0260] {Fine-grained similarity knowledge point 2} Example 1 {Example 1}, Example 2 {Example 2}”

[0261] Step S205: Exercise Check

[0262] The exercise checking agent performs quality checks on the newly generated exercises. The check criteria include two aspects: first, whether the generated exercises conform to the description of the label overview information, that is, whether they accurately reflect the core features of the target label; second, whether there are any cases in the exercises that directly mention the label information (such as the knowledge point name appearing explicitly in the question stem). Such cases will make the questions too straightforward and reduce the training value.

[0263] Exercises that pass the test are retained and added to the augmented dataset; exercises that fail are discarded. This checking mechanism effectively filters out low-quality or erroneous augmented samples, ensuring the reliability of the training data.

[0264] In one specific implementation, the exercise checking agent uses the following prompt words:

[0265] "You are a middle school math teacher. You have a practice problem that needs to be checked. The check includes two parts: whether the problem matches the description of the knowledge point being tested, and whether the problem directly provides the knowledge point content. If the problem does not match the knowledge point description or directly provides the knowledge point content, output 'False'; otherwise, output 'True'. The output is in \box{}."

[0266] For example: \box{True}”

[0267] Through the collaborative work of the above five steps, the data augmentation module can generate high-quality, diverse, and accurately labeled augmented exercise data, effectively expanding the training set size and improving the problem of class imbalance.

[0268] Example 3: Detailed Implementation of Statistical Feature Extraction

[0269] This embodiment details the workflow of the exercise statistical feature matrix extraction module. As mentioned earlier, the feature extraction process is identical in the annotation training phase, the fusion training phase, and the annotation phase. This unified feature extraction process is described in detail below.

[0270] This module extracts four types of statistical features from the exercise data and constructs a multidimensional feature matrix.

[0271] (1) Keyword feature frequency extraction

[0272] The exercise text was segmented using a Chinese word segmentation tool (such as Jieba), and the frequency of each keyword in the exercises was counted to form a keyword frequency vector. For example, for exercises involving "quadratic function graphs", keywords such as "parabola", "vertex", and "axis of symmetry" tend to have higher frequencies.

[0273] (2) TF-IDF vector extraction

[0274] Based on keyword frequency, the Term Frequency-Inverse Document Frequency (TF-IDF) value is calculated. TF-IDF comprehensively considers both the frequency (TF) of a keyword in the current exercise and the prevalence (IDF) of that keyword in the entire dataset, effectively highlighting distinctive keywords and reducing the weight of common words. The calculation formula is as follows:

[0275]

[0276] in, Keywords In the exercises word frequency in The total number of exercises in the dataset. For keywords The number of exercises.

[0277] (3) Image feature extraction

[0278] For exercises containing images, multidimensional features of the images are extracted, including: image type (geometric figures, function graphs, statistical charts, etc.), content description (shape of the figure, line features, region division, etc.), and key elements (coordinate axes, function curves, annotation points, etc.). These visual features are extracted using image recognition algorithms or pre-trained visual models and transformed into numerical vectors.

[0279] (4) Keyword combination feature extraction

[0280] Keyword combinations highly correlated with specific tags are manually constructed based on domain knowledge. These combinations reflect the tag's signature linguistic patterns. For example, the combination of "parabola" and "area" strongly indicates the tag "area of ​​a quadratic function," while the combination of "profit," "cost," and "selling price" indicates the tag "economic issues." Keyword combination features are quantified using Boolean values ​​or frequency of occurrence, enhancing the model's sensitivity to specific tag recognition.

[0281] The above four types of features are vectorized and then concatenated to form a complete statistical feature matrix, which serves as the input to the statistical feature annotation module.

[0282] Example 4: Detailed Implementation of the Language Model Annotation Module

[0283] like Figure 4 As shown in the figure, this embodiment describes in detail the structure and training process of the language model annotation module. This module includes a pre-trained language model, a word segmenter, and a trainable classification head.

[0284] Module composition

[0285] The pre-trained language model is responsible for deep semantic encoding of the exercise text. This embodiment configures multiple heterogeneous language models simultaneously to improve generalization ability, including:

[0286] BGE-large-zh-v1.5: An encoder model focused on Chinese semantic understanding;

[0287] Roberta-xml: A RoBERTa variant optimized for long text processing;

[0288] BGE-m3: Supports multi-language, multi-modal encoder models;

[0289] MathBERTa: A model specifically designed for pre-training on mathematical texts, adept at handling LaTeX formulas;

[0290] Qwen-Math-2.5-Instructor: A fine-tuning model for instructions in large-scale mathematical domains.

[0291] For models like MathBERTa that specialize in processing mathematical formulas, the input text is preprocessed to retain only the LaTeX formula portion and remove other natural language descriptions in order to highlight the mathematical structure features.

[0292] The tokenizer converts the exercise text into a sequence of tokens that the model can process. Different language models are equipped with corresponding tokenizers, such as the BPE tokenizer and the WordPiece tokenizer.

[0293] The trainable classification head is a newly added fully connected layer whose output dimension is equal to the size of the label set. The classification head receives the text feature representation output by the language model and maps it to the predicted score of each label.

[0294] The language model annotation module undergoes multiple rounds of training until a preset number of training rounds is reached, the output cross-entropy loss is less than a preset threshold, or the cross-entropy loss value does not decrease after more than 10 rounds. The training steps for a single round are as follows:

[0295] 1. Forward Propagation: The forward propagation steps of the language model annotation module are exactly the same in all three stages of the system described in the patent (Step 5 in Stage 1, Step 3 in Stage 2, and Step 3 in Stage 3).

[0296] Step 1: The question text is segmented by a tokenizer to obtain tokens, represented as follows:

[0297] ,

[0298] Step 2: Extract information from the obtained tokens through the last hidden layer of the natural language processing model;

[0299] For models with CLS-tokens, extract the CLS-tokens to obtain the processed text signal:

[0300] ,

[0301] For models without CLS-tokens, average pooling is applied to the last hidden layer to obtain the processed text signal:

[0302] ,

[0303] Step 3: Process the text signal through a linear layer to obtain the output signal:

[0304]

[0305] 2. Backpropagation: The main difference between the language model annotation module in the annotation training phase and the other two phases is that the annotation training phase requires backpropagation to train the language model after forward propagation, while the fusion training phase and annotation phase do not have this step and use standard backpropagation.

[0306] Example 5: Detailed Implementation of the Statistical Feature Annotation Module

[0307] This embodiment details the implementation of the statistical feature annotation module. This module employs the XGBoost (eXtremeGradient Boosting) algorithm to predict labels based on the exercise's statistical feature matrix.

[0308] XGBoost is an efficient machine learning algorithm based on Gradient Boosting Decision Trees (GBDT). It constructs a strong learner by combining a series of weak learners (decision trees). The goal of XGBoost is to optimize model performance by minimizing a loss function. The objective function typically consists of two parts: training loss (such as mean squared error, log loss, etc.) and a regularization term (used to prevent overfitting).

[0309] In the exercise annotation, XGBoost contains M tree groups, where M is the number of label sets (the length of the label probability vector), and each tree group has K trees, with each tree group focusing on the annotation of a specific label.

[0310] 1. Training process: The training process of XGBoost is as follows:

[0311] Step 1: Initial Predictions: Typically, the initial predicted values ​​for all samples are set to a constant (such as the mean or median of the target value) as the starting point for the first round of iterations.

[0312] Step 2, Iterative Training: XGBoost builds the model step by step through multiple iterations, generating a new decision tree in each iteration and continuously optimizing the objective function.

[0313] (1) Calculate the gradient and second derivative

[0314] For each sample, the first derivative (gradient) and second derivative (Hessian) of the loss function with respect to the current prediction are calculated. These derivatives reflect how well the current model fits the sample and how sensitive it is to model adjustments.

[0315] (2) Constructing a decision tree

[0316] Split Node Selection: XGBoost employs a greedy algorithm to select the optimal split point by calculating the split gain for each feature. The split gain measures the improvement of the objective function after splitting, and its calculation formula is as follows:

[0317]

[0318] in, and These are the gradient sums of the left and right subtrees, respectively. and is the sum of the corresponding second derivatives, λ is the regularization parameter, and γ is the leaf node penalty term.

[0319] Tree growth: Starting from the root node, nodes are continuously split until the maximum depth is reached or the splitting gain is less than a threshold. XGBoost supports both depth-first and breadth-first tree growth strategies.

[0320] Pruning: Trees are pruned by limiting tree depth, the number of leaf nodes, or setting a minimum split gain to prevent overfitting.

[0321] (3) Update model predictions

[0322] After each decision tree is trained, its predictions are added to the current model's predictions. The update formula is:

[0323]

[0324] in, This is the predicted value after the t-th iteration, where η is the learning rate (step size), used to control the contribution of each tree to the final predicted value. It is the predicted value of the t-th tree.

[0325] Step 3: End of iteration:

[0326] The training process ends when the preset maximum number of iterations is reached, the model converges (the improvement of the objective function is less than the threshold), or the training error reaches a satisfactory level.

[0327] 2. Prediction process

[0328] The process of generating label probability vectors using the statistical feature annotation module is the same in both stages of the system (Step 2 in Stage 2 and Step 2 in Stage 3):

[0329] Step 1: Obtain the initial prediction value: Obtain the feature vector of the sample to be predicted, and extract the initial prediction value determined during the XGBoost training phase as the baseline value;

[0330] Step 2, Tree Set Accumulation Calculation: Input the feature vector into the decision tree set obtained by multiple rounds of iterative training, wherein each decision tree maps the feature vector to the corresponding leaf node based on the optimal split node determined in the training phase, and obtains the output weight of each leaf node; the output weights of all decision trees are weighted and accumulated with the initial prediction value according to the learning rate to obtain the original prediction score vector of the sample.

[0331] Step 3, Probability Transformation: The original predicted score vector is transformed nonlinearly through a preset link function to map the real number domain scores to the probability interval [0,1], generating a normalized label probability distribution vector.

[0332] The cumulative calculation in step two follows the following formula:

[0333]

[0334] in, The original predicted score, The initial predicted value, For the total number of decision trees, For the first Tree samples The leaf node output weights, The learning rate determined for the training phase.

[0335] In step three, the link function is the Softmax function.

[0336]

[0337] in The total number of categories, For the first The original predicted score for the class.

[0338] Example 6: Detailed Implementation of the Annotation Result Fusion Module

[0339] like Figure 5 As shown, this embodiment details the structure and training process of the annotation result fusion module. It should be noted that the forward propagation process of this module is completely identical in the fusion training phase and the annotation phase; the only difference is the inclusion of a backpropagation parameter update step in the fusion training phase.

[0340] This module is responsible for integrating the prediction results from all annotation modules in the system (including multiple language model annotation modules and statistical feature annotation modules). The module contains a set of learnable weights. ,in This represents the total number of labeled modules in the system.

[0341] 1. Forward Propagation: The forward propagation steps of the annotation result fusion module are exactly the same in both the fusion training phase and the annotation phase (Phase 2, Step 4; Phase 3, Step 4).

[0342] Step 1: Use the Softmax function to transform the learnable parameter vector Convert to probability distribution:

[0343]

[0344] in, It is a learnable parameter vector (N is the number of labeled modules). It is the first Normalized weights of each annotation module.

[0345] Step 2: Perform a weighted summation of the label probability vectors output by each annotation module:

[0346]

[0347] in, This is the final probability vector. This is the label probability vector output by the i-th annotation module. For each question, the label with the highest probability in the final probability vector is taken as the predicted label, and this is used as the final prediction result.

[0348]

[0349] 2. Backpropagation: The main difference between the fusion training and annotation phases in the fusion result fusion module is that the fusion training phase requires backpropagation to train the learnable parameter vectors after forward propagation, while the annotation phase does not have this step and uses standard backpropagation.

[0350] Example 7: Detailed Implementation of System Operation Process

[0351] like Figure 3 As shown, this embodiment describes the complete operation flow of the system, including three stages:

[0352] Phase 1: Labeling and Training Phase

[0353] Step T1: Select and divide the existing labeled exercise dataset into labeled training dataset, fused training dataset and labeled test dataset in an 8:1:1 ratio, corresponding to stage one (labeled training stage), stage two (fused training stage) and stage three (labeling stage) respectively.

[0354] Step T2: Input the labeled training dataset into the data augmentation module to obtain the augmented labeled training dataset, and merge it with the original labeled training dataset to form the augmented labeled training dataset.

[0355] Step T3: Input the augmented labeled training dataset into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the augmented labeled training dataset.

[0356] Step T4: Train all statistical feature annotation modules in the system using the statistical feature matrix of the augmented labeled training dataset.

[0357] Step T5: Train all language model annotation modules in the system using the augmented labeled training dataset.

[0358] Phase Two: Integration Training Phase

[0359] Step T5: Input the fused training dataset into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the fused training dataset. It should be noted that the feature extraction process in this step is exactly the same as step T3 in stage one, only the source of the input data is different.

[0360] Step T6: Input the statistical feature matrix of the fused training dataset into all statistical feature annotation modules in the system to obtain the label probability vector of the fused training dataset output by all statistical feature annotation modules.

[0361] Step T7: Input the fused training dataset into all language model annotation modules in the system to obtain the label probability vectors output by all language model annotation modules.

[0362] Step T8: Fuse the label probability vector training annotation results of the fusion training dataset output by all modules into a fusion module.

[0363] Phase 3: Labeling Phase

[0364] Step T9: Input the exercise data to be labeled (test data / real data) into the exercise statistical feature matrix extraction module to obtain the statistical feature matrix of the exercises to be labeled. It should be noted that the feature extraction process in this step is exactly the same as step T3 in stage one and step T5 in stage two, except that the input is the exercise data to be labeled.

[0365] Step T10: Input the statistical feature matrix of the exercise to be labeled into all statistical feature labeling modules in the system to obtain the label probability vector of the exercise to be labeled output by all statistical feature labeling modules.

[0366] Step T11: Input the exercise data to be labeled into all language model annotation modules in the system to obtain the label probability vector of the exercise to be labeled output by all language model annotation modules.

[0367] Step T12: Input the label probability vectors of the unannotated exercise data output by all modules into the annotation result fusion module to obtain the final annotation result of the unannotated exercise data. It should be noted that the forward propagation process in this step is exactly the same as step T8 in stage two, but the backpropagation parameters are not updated.

[0368] Example 8

[0369] The method described in this invention is used to annotate single-label Chinese math exercise data (diaa dataset). This single-label Chinese exercise data consists of junior high school math exercises, with each exercise corresponding to only one label. The exercise text includes HTML tags and LaTeX formulas. All exercises cover 155 knowledge point labels. Using the system proposed in this invention, an accuracy of 0.68 can be achieved compared to traditional automatic methods on more difficult subsets of the dataset, and an accuracy of 0.89 can be achieved on more common subsets.

[0370] Example 9

[0371] The method described in this invention is used to annotate multi-label Chinese math exercise data (CCNU dataset). The single-label Chinese exercise data consists of elementary and junior high school math exercises. One exercise may correspond to multiple labels. The exercise text includes HTML tags and LaTeX formulas. All exercises cover 98 knowledge point labels. Using the system proposed in this invention, an accuracy of 0.92 can be achieved on the entire dataset.

[0372] Example 10

[0373] The method described in this invention is used to annotate single-tag English course titles. The course titles are written in English, and each course corresponds to one topic tag, for a total of 10 topic tags. The system proposed in this invention achieves an accuracy rate of 0.79.

[0374] Comparative Example 1

[0375] In Example 1, the single-label Chinese math problem data was labeled using methods such as TextCNN, BiLSTM, and BERT. The accuracy obtained was significantly lower than the method proposed in this system, achieving only 0.58.

[0376] Comparative Example 2

[0377] In Example 2, the multi-label Chinese math problem data was labeled using methods such as TextCNN, BiLSTM, and BERT. The accuracy obtained was significantly lower than the method proposed in this system, achieving only 0.87.

[0378] Compared with traditional text classification methods, the results of this invention are better than those of traditional text classification methods on all indicators, as shown in the table below:

[0379]

[0380] Where Top@k is a parameter for multi-label datasets, representing the selection of the k most likely labels as output labels in the model output.

[0381] The ACC indicator is an accuracy metric, and its calculation formula is as follows:

[0382]

[0383] TP represents the true cases, which are the number of samples that are actually positive but are predicted as positive by the model in the results.

[0384] TN stands for True Negative Instances, which is the number of samples that are actually negative but are predicted as negative by the model.

[0385] FP stands for false positive, which is the number of samples that are actually negative but are predicted as positive by the model.

[0386] FN stands for false negatives, which is the number of samples that are actually positive but are predicted as negative by the model.

[0387] The Precision metric is a precision rate indicator, calculated using the following formula:

[0388]

[0389] The recall metric is a measure of the rate of return to service, and its calculation formula is as follows:

[0390]

[0391] The F1-Score is the F1 score indicator, calculated using the following formula:

[0392]

[0393] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A three-stage progressive training framework-based automatic fine-grained labeling system for exercises, characterized in that, include: The system includes a data augmentation module, a problem statistical feature matrix extraction module, at least one language model annotation module, at least one statistical feature annotation module, and an annotation result fusion module; among which, The data augmentation module runs only during the annotation training phase. It is used to generate augmented exercise data based on the original annotation training dataset through a controlled augmentation mechanism involving multi-agent collaboration. The augmented exercise data is then merged with the original annotation training dataset to form the augmented annotation training dataset. The exercise statistical feature matrix extraction module is used to extract statistical features from the input exercise data and construct a feature matrix in the annotation training stage, fusion training stage, and annotation stage, using a unified feature extraction process, algorithm, and parameter configuration; the statistical features include keyword feature frequency, TF-IDF vector, image features, and keyword combination features; The language model annotation module and the statistical feature annotation module are heterogeneous and independently optimized dual-path annotation modules. They are trained independently and in parallel during the annotation training phase, their feature spaces are isolated from each other, and both output a label probability distribution vector with the same dimension as the label set size. The language model annotation module is trained on the augmented annotation training dataset during the annotation training phase. It is used to perform deep semantic understanding of the exercise text and output label probability distribution vectors. The language model annotation module includes a pre-trained language model, a word segmenter, and a trainable classification head. The statistical feature annotation module completes training based on the statistical feature matrix corresponding to the augmented annotation training dataset during the annotation training phase. It is used to learn the structured patterns of the exercises based on the statistical feature matrix and output the label probability distribution vector. The annotation result fusion module only operates during the fusion training and annotation phases, and does not participate in the annotation training phase. The annotation result fusion module contains a set of learnable dynamic weights, the number of which is equal to the total number of all annotation modules in the system. During the fusion training phase, the annotation result fusion module completes weight training only based on the fusion training dataset composed of original real samples that have not participated in data augmentation. During the annotation phase, based on the trained dynamic weights, the module performs weighted fusion of the label probability distribution vectors output by each annotation module to generate a fused label probability distribution vector and output the final annotation result.

2. The homework auto-fine-grained labeling system of claim 1, wherein, The operation of the system includes a data augmentation phase, a labeling training phase, a fusion training phase, and a labeling phase for actual labeling applications, which are executed sequentially. The data augmentation phase only runs the data augmentation module, the annotation training phase only trains the language model annotation module and the statistical feature annotation module independently, and the fusion training phase only trains the annotation result fusion module. The data augmentation module does not participate in the operation during the annotation phase. The data augmentation module includes a coarse-grained similar label extraction agent, a fine-grained similar label extraction agent, a label overview extraction agent, an exercise generation agent, and an exercise checking agent that work in sequence.

3. The homework auto-fine-grained labeling system of claim 2, wherein, The data augmentation phase includes the following steps: The original labeled exercise dataset was divided into a labeled training dataset, a fused training dataset, and a labeled test dataset in an 8:1:1 ratio. Input the labeled training dataset into the data augmentation module to generate an augmented labeled training dataset, and merge it with the original labeled training dataset to form an augmented labeled training dataset. And / or, The annotation training phase includes the following steps: Input the augmented labeled training dataset into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix; All statistical feature annotation modules in the system are trained based on the aforementioned statistical feature matrix; Train all language model annotation modules in the system based on the augmented labeled training dataset; And / or, The fusion training phase includes the following steps: Input the fused training dataset into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix; Input the statistical feature matrix into all statistical feature annotation modules to obtain the label probability distribution vector output by each module; Input the fused training dataset into all language model annotation modules and obtain the label probability distribution vectors output by each module; Based on the label probability distribution vectors output by all annotation modules, the learnable dynamic weights of the annotation result fusion module are trained with cross-entropy loss as the optimization objective. And / or, The annotation stage includes the following steps: Input the exercise data to be labeled into the exercise statistical feature matrix extraction module to generate the corresponding statistical feature matrix; Input the statistical feature matrix into all statistical feature annotation modules to obtain the label probability distribution vector output by each module; Input the exercise data to be labeled into all language model labeling modules and obtain the label probability distribution vector output by each module; Input the label probability distribution vectors output by all annotation modules into the annotation result fusion module to generate the final annotation result of the exercise data to be annotated.

4. The homework auto-grading system of claim 3, wherein, The labeled exercise dataset contains at least an ID field and a Content field. The ID field is used to uniquely identify the exercise data, and the Content field is used to store the text content of the exercise.

5. The automatic fine-grained annotation system for exercises according to claim 1, characterized in that, The operation of the data augmentation module includes the following steps: We extract agents using coarse-grained similarity labels, extracting coarse-grained similarity labels for each label from the exercise dataset based on the literal semantics of the label text; The agent is extracted using fine-grained similarity labels. Several examples are extracted for each label and its coarse-grained similarity labels. Fine-grained similarity labels are extracted from the coarse-grained similarity labels through comparative analysis. The agent is extracted using the tag overview. Several example problems are extracted for each tag and its fine-grained similar tags. The similarities and differences between the tag problems and the fine-grained similar tag problems are analyzed to generate the tag overview. A problem-generating agent is used to randomly select labeled problems and fine-grained similar labeled problems. Based on the label overview information, the labeled problems are data augmented to generate new problems. The agent is checked using exercises, and newly generated exercises are checked against the label overview to see if they meet the augmentation criteria. Exercises that meet the criteria are added to the augmented dataset.

6. The automatic fine-grained annotation system for exercises according to claim 5, characterized in that, The coarse-grained similarity label extraction agent selects up to 5 candidate labels that are similar to the target label from all exercise labels based on label semantics. The fine-grained similarity label extraction agent, based on the coarse-grained similarity labels, extracts up to three fine-grained similarity labels by comparing the differences between the label exercises and the coarse-grained similarity label exercises; And / or, The tag overview extraction agent extracts two example questions for each tag and its fine-grained similar tags, and generates a tag overview that includes the characteristics of the tag and the similarities and differences with the fine-grained similar tags through comparative analysis. And / or, The exercise generation agent expands the number of exercises under each label in the training set that is less than the average number of exercises to the average number of exercises level, and further expands each label by 10 exercises. Based on the label overview information, 3 label exercises and 3 fine-grained similar label exercises, the exercise generation agent randomly selects a label exercise for modification and enhancement to generate a new exercise. And / or, The exercise checking agent checks whether newly generated exercises conform to the label overview information and whether there are any direct references to the label information. Exercises that do not conform to the overview information or directly reference the label information are discarded, while those that do are retained and added to the augmented dataset.

7. The automatic fine-grained annotation system for exercises according to claim 1, characterized in that, The feature extraction process of the exercise statistical feature matrix extraction module includes: Keyword frequency is obtained by statistically analyzing the occurrence frequency of each keyword in the exercise text. The TF-IDF values ​​of each keyword in the exercise text are calculated using the TF-IDF algorithm, and a TF-IDF vector is constructed. Extract image type, content description, and key element features from exercises containing images; Keyword combinations are selected based on the relevance between the exercise content and the tags to form keyword combination characteristics.

8. The automatic fine-grained annotation system for exercises according to claim 1, characterized in that, The statistical feature annotation module uses the XGBoost model, which contains multiple decision trees, each decision tree corresponding to the prediction of a label. The objective function of the model includes training loss and regularization term. And / or, The training of the language model annotation module includes multiple iterations until a preset number of training rounds is reached, the loss function value is less than a preset threshold, or the loss does not decrease for multiple consecutive rounds. A single round of training includes: Forward propagation steps: The exercise text is processed by a word segmenter to obtain a word sequence, which is then input into a pre-trained language model to obtain the features of the last hidden layer; for models with CLS tags, the features corresponding to the CLS tags are extracted, and for models without CLS tags, average pooling is performed on the last hidden layer; the extracted features are processed by an activation function and a linear classification head to output a label probability distribution vector. Backpropagation steps: Calculate the loss between the predicted output and the true label based on the cross-entropy loss function, and update the model parameters using the gradient descent algorithm; And / or, The training of the annotation result fusion module includes multiple iterations until the loss function value meets the convergence condition or reaches the preset number of training rounds; a single training round includes: Forward propagation steps: The learnable parameter vector is converted into normalized weights for each annotation module using the Softmax function. The label probability distribution vectors output by each annotation module are weighted and summed based on the weights to generate the fused label probability distribution vector. Backpropagation steps: Calculate the loss between the fused label probability distribution vector and the real label based on the cross-entropy loss function, and update the learnable parameter vector through the gradient descent algorithm.

9. A method for automatic fine-grained annotation of exercises based on a three-stage progressive training framework, characterized in that, Includes the following steps: S1, Data Augmentation Stage: The original labeled exercise dataset is divided into labeled training dataset, fused training dataset and labeled test dataset according to the proportion. Based on the labeled training dataset, augmented exercise data is generated through a multi-agent collaborative process of coarse-grained similar label extraction, fine-grained similar label extraction, label summary generation, exercise generation and exercise checking. The augmented exercise data is then merged with the original labeled training dataset to form the augmented labeled training dataset. The fusion training dataset consists of original real samples that have not undergone data augmentation; S2, Independent Pre-training Stage: Based on the augmented labeled training dataset, the language model annotation module and the statistical feature annotation module are trained independently and in parallel. The language model annotation module learns deep semantics from the original text of the exercises and outputs a label probability distribution vector. The statistical feature annotation module learns structured patterns from the statistical feature matrix of the exercises and outputs a label probability distribution vector. The statistical feature matrix is ​​constructed through a unified feature extraction process, including keyword feature frequency, TF-IDF vector, image features, and keyword combination features. S3, Fusion Optimization Stage: Based on the fusion training dataset, obtain the label probability distribution vectors output by each annotation module, and use cross-entropy loss as the optimization objective to train the learnable dynamic weights of the annotation result fusion module to obtain the optimal fusion weights; S4, Automatic Labeling Stage: Input the exercise data to be labeled into the trained language model labeling module and statistical feature labeling module respectively, obtain the label probability distribution vector output by each module, and perform weighted fusion based on the optimal fusion weight through the labeling result fusion module to generate the final labeling result.

10. The automatic fine-grained annotation system for exercises according to any one of claims 1-8, or the automatic fine-grained annotation method for exercises according to claim 9, is applied in automatic annotation of exercises, classification of educational documents, and automatic summarization of knowledge points.

Images