Intelligent marking method and system for test literacy cognitive hierarchy based on multi-dimensional tensor

Abstract

The application discloses a multi-dimensional tensor-based test literacy cognitive level intelligent labeling method and system, in order to solve the problem of multi-dimensional literacy rating performance collapse caused by expert labeling scarcity, loss of mathematical symbol semantics and long tail distribution in the prior art, a three-dimensional tensor space is constructed by fusing course standards; the first data set is used as a few-sample example, and a diagnostic thinking chain is combined to drive a large language model to generate structured pseudo-labels for unlabeled test questions in the second data set; through the incremental pre-training of the mask language model by injecting mathematical symbol prior and combining the cost-sensitive weighting mechanism based on real slot statistics, the lightweight training of the high-dimensional multi-task joint rating network is realized; finally, the active backflow closed loop of low confidence samples is executed through confidence evaluation. The application greatly improves the detection rate and rating accuracy of high-order long tail literacy under very small samples, significantly reduces the labeling cost, and realizes the engineering intelligent evaluation of cross-version massive question banks.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of smart education and natural language processing, specifically to an intelligent annotation method and system for test literacy cognitive levels based on multidimensional tensors. Background Technology

[0002] With the deepening development of educational informatization and smart education, modern educational evaluation is undergoing a profound transformation from "knowledge transmission" to "core competency-based" evaluation. Online education platforms and intelligent question bank systems have accumulated a vast amount of digital learning resources, such as test questions and lesson plans. In the field of junior high school mathematics, a comprehensive and accurate analysis of the "core competencies" behind these test resources, such as abstract thinking, reasoning, and modeling concepts, and their corresponding "cognitive levels," is of crucial guiding significance for conducting in-depth student ability diagnosis, personalized learning path recommendations, and precise teaching research.

[0003] Deeply integrating educational measurement theory with the "Compulsory Education Mathematics Curriculum Standards," and constructing a multi-dimensional tensor space encompassing "test items - subject literacy - literacy cognitive level" covering core competencies and cognitive depth in junior high school mathematics, is a prerequisite for achieving intelligent educational assessment. Currently, mainstream methods for labeling test items with core competencies mainly rely on subjective manual grading by senior educational research experts or traditional text classification techniques. Traditional text classification techniques typically only output flat, one-dimensional labels, failing to achieve a quantitative representation of educational theory at the data structure level. Furthermore, when faced with complex real-world application scenarios, existing technologies reveal the following significant shortcomings.

[0004] For example, the existing technology CN118585918A discloses an intelligent systematic annotation method and system for core competencies across multiple disciplines and educational stages. This scheme extracts semantic features through a multi-task shared layer and then constructs an independent classification layer for each core competency dimension to perform binary classification prediction. However, this method has the following shortcomings: First, it can only determine whether a question involves a certain core competency, outputting a single-dimensional label of "yes" or "no," and cannot further quantify the cognitive depth that students should achieve in that competency, which is far from sufficient for accurately diagnosing students' abilities. Second, its parallel multi-classification head architecture, when facing multi-competency, multi-level joint prediction tasks, leads to a linear increase in network parameters with the number of classification nodes, not only causing parameter redundancy and exacerbating the training difficulty under small sample conditions, but also making it difficult to capture the intrinsic relationships between different competencies and levels. Another example is the proposed method for constructing and annotating tensors of questions, knowledge, and abilities for knowledge measurement, which introduces the concept of a three-dimensional tensor of "question-knowledge-ability." This scheme, based on the Q-matrix and Bloom's Taxonomy of Cognitive Domain Educational Objectives, divides knowledge point mastery into six general cognitive ability levels. However, its shortcomings are as follows: First, its tensor construction, based on Bloom's general cognitive objective classification, fails to deeply integrate the nine core competencies system unique to mathematics as stipulated in the "Compulsory Education Mathematics Curriculum Standards," lacking subject-specific relevance and guidance. Second, its technical implementation relies on Q-matrix theory, and when faced with massive amounts of unlabeled test questions, it lacks an effective data augmentation mechanism to overcome the bottleneck of scarce expert-labeled samples. Third, it also fails to provide an effective solution for extreme long-tail distributions, i.e., the extreme scarcity of samples at a few high-order competency levels.

[0005] Furthermore, the existing technology CN117744783B discloses a human-in-the-loop knowledge tracking method, which calculates the confidence level of the prediction results of the knowledge tracking model and then submits low-confidence samples for manual review. However, this method, when applied to the field of student knowledge tracking, processes the interaction data between students and test questions, rather than the semantic understanding and annotation of the test question content. Its confidence assessment object is the predicted result of student performance, which is fundamentally different from the confidence assessment of the multi-dimensional literacy labels of the test questions themselves in this application.

[0006] In addition to the shortcomings of the existing technical solutions mentioned above, this application has also discovered and overcome the following technical difficulties during the implementation process:

[0007] (1) High-quality, fine-grained labeled data is extremely scarce, making it difficult to drive deep learning models. Unlike conventional shallow "knowledge point" labeling, multi-dimensional "core competencies-cognitive level" labeling has extremely high teaching and research thresholds and costs. In practical applications, teaching and research experts can often only perform fine-grained labeling on a very small number of test questions. However, major smart education platforms face tens of thousands or even hundreds of thousands of unlabeled test questions from multiple versions of textbooks, including those from Beijing Normal University, People's Education Press, and East China Normal University. Due to the lack of sufficient high-quality supervised data, conventional deep learning networks are prone to overfitting on very small samples and cannot achieve effective generalization to massive cross-version question banks.

[0008] (2) General-purpose language models suffer from significant vertical domain barriers and loss of semantic meaning in mathematical symbols. Existing automatic annotation schemes mostly attempt to introduce basic pre-trained language models (BERT, RoBERTa, etc.) to process test texts. However, these general-purpose foundational models are mainly pre-trained on open-domain corpora and lack a deep understanding of the context specific to junior high school mathematics. Especially when faced with mathematical test questions containing a large number of complex LaTeX formulas and geometric symbols, general-purpose word segmenters often forcibly break down complete mathematical expressions, resulting in a large loss of underlying mathematical logic semantics, which seriously affects the model's accurate extraction of high-order core competencies such as geometric intuition and computational ability features.

[0009] (3) High-dimensional fine-grained classification leads to bloated network architecture and prediction collapse under long-tail distribution. Fine-grained labeling of mathematical core competencies is a typical high-dimensional multi-task classification problem. For example, the nine core competencies plus the four core competency cognitive levels constitute 36 fine-grained classification nodes. Using the traditional parallel multi-classification head architecture not only leads to redundant network parameters and exacerbates the training difficulty under small sample sizes, but also, the competency labels of real test questions usually exhibit an extreme long-tail imbalanced distribution.

[0010] (4) Lack of confidence quantification and dynamic closed-loop iteration mechanism for massive question banks. In the routine automatic labeling process of massive question banks, existing models usually directly output a single prediction label conclusion, lacking a quantitative assessment of the uncertainty of their own prediction results. When faced with difficult questions that exceed the model's generalization boundary or a dynamically updated question bank, the system cannot automatically identify fuzzy samples at the decision boundary, making it difficult to form an active learning feedback loop, which limits the engineering reliability and continuous evolution capability of the labeling system. Summary of the Invention

[0011] The main objective of this application is to provide an intelligent annotation method for test literacy cognitive levels based on multidimensional tensors, including the following steps:

[0012] S1: Construct a three-dimensional educational tensor evaluation space based on the subject curriculum standards, which includes test questions, subject literacy, and literacy cognition level. Obtain a first dataset that is anchored by experts through tensor instantiation based on the three-dimensional educational tensor evaluation space. At the same time, collect a large number of unlabeled test questions to form a second dataset.

[0013] S2: Based on the first dataset as a few sample examples, a three-stage deduction including analysis process, dimension matching and result assembly is performed on the unlabeled test items to be labeled in the second dataset using a large language model combined with a diagnostic thinking chain strategy to generate structured pseudo-label data. After data filtering and format conversion, a pseudo-label tensor augmentation dataset is obtained.

[0014] S3: Clean the test text, explicitly register mathematical formulas and geometric symbols in LaTeX format as special tags to the word segmenter of the pre-trained encoder language model and expand the embedding layer, perform domain-adaptive incremental pre-training of the masked language model on the educational corpus containing the pseudo-label tensor augmentation dataset, and obtain a customized base model.

[0015] S4: Using the customized base model as the semantic representation base, construct a feature dimension folding architecture as a multi-task sharing and classification layer, calculate the total loss of the multi-task by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network by using the full training set composed of the first dataset and the pseudo-label tensor augmentation dataset.

[0016] S5: Input the test text into the fine-tuned joint rating network, output the classification labels of each core competency dimension covered by the test text and their corresponding probabilities, and perform automatic discovery and active learning feedback of low confidence samples based on confidence assessment.

[0017] In one embodiment, step S1 includes:

[0018] Based on the "Compulsory Education Mathematics Curriculum Standards," a three-dimensional educational tensor evaluation space is constructed, encompassing test items, subject literacy, and literacy cognitive levels. The first dimension represents the test item dimension, symbolizing specific test item entities in the question bank. The second dimension represents the subject literacy dimension, defining nine core competencies including abstract thinking, computational ability, geometric intuition, spatial reasoning, reasoning ability, data literacy, modeling ability, application awareness, and innovative awareness. The third dimension represents the literacy cognitive level dimension, defining four continuous quantitative levels for each subject literacy dimension: level zero, representing no involvement of that competency; and levels one, two, and three, representing progressive abilities. The literacy cognitive level refers to the examination level of a test item on a specific subject literacy dimension, categorized based on cognitive depth, knowledge integration, transfer and application, and innovation requirements, serving as a fine-grained, hierarchical evaluation dimension representing core competencies.

[0019] Obtain the first dataset anchored by experts through tensor instantiation based on the three-dimensional education tensor evaluation space;

[0020] A second dataset was constructed by collecting a large number of unlabeled test questions covering at least four different textbook versions.

[0021] In one embodiment, the specific process of obtaining the first dataset includes:

[0022] Two annotators independently and double-blindly instantiated and anchored a small number of sampled test items based on the three-dimensional education tensor evaluation space. The anchoring results were compared, and test items with discrepancies were reviewed and arbitrated to obtain the first dataset with higher education research logic consistency.

[0023] In one embodiment, step S2 includes:

[0024] A predetermined number of test questions are randomly selected from the first dataset, and their test texts and corresponding tensor instantiation anchor labels are extracted to construct a small sample example set. At the same time, a diagnostic thinking chain system prompt word is constructed. The system prompt word forces the large language model to follow the three-stage inference logic of analysis process, dimension matching and result assembly.

[0025] The large language model application programming interface is invoked, and the unlabeled test questions in the second dataset and the few sample example set are input together. The large language model then performs the following three-stage deduction process in sequence: internally reviewing the knowledge points tested in the unlabeled test questions; deriving the logical complexity based on the key steps of problem-solving and anchoring the core competency dimensions and their corresponding cognitive level levels in the three-dimensional education tensor evaluation space; and outputting labeled data that conforms to the preset JSON structure.

[0026] Data filtering and format conversion are performed on the JSON structured label data output by the large language model, including parsing the JSON data, identifying and deleting invalid samples with missing target literacy label fields, empty dictionaries, or empty lists; the retained valid label data is mapped to numerical pseudo-label vectors corresponding to the three-dimensional education tensor evaluation space to obtain the pseudo-label tensor augmentation dataset.

[0027] In one embodiment, step S3 includes:

[0028] The test text is cleaned to remove irrelevant hypertext markup language tags and noise characters; a special character set containing mathematical formulas and geometric symbols in LaTeX format is loaded from a preset mathematical symbol library, and the special character set is dynamically registered as a special tag into the base word segmenter of the pre-trained encoder language model;

[0029] The word embedding layer of the pre-trained encoder language model is expanded based on the registration results of the special tags;

[0030] The test texts in the pseudo-labeled tensor augmentation dataset are mixed with the remaining massive amount of unlabeled test texts in the second dataset to construct an educational corpus for mathematics. A dynamic masking strategy is used to randomly mask the text sequences in the educational corpus with a preset probability, and unsupervised incremental pre-training of the masked language model is performed with the prediction of the real words at the masked positions as the optimization objective to obtain the customized base model.

[0031] In one embodiment, the step of expanding the word embedding layer of the pre-trained encoder language model based on the registration result of the special tag includes:

[0032] Extracting high-frequency formulas and geometric characters from junior high school mathematics to construct a special tag word list;

[0033] The special tag vocabulary is dynamically registered to the base word segmenter of the pre-trained encoder language model, and the word embedding layer is expanded using a formula, which is:

[0034]

[0035] in For the original embedded layer, Based on the vocabulary list, For the newly added special marker vocabulary;

[0036] A dynamic masking strategy is adopted to randomly mask educational scenario corpora containing test questions, and the log-likelihood loss is calculated with the real words at the predicted mask positions as the optimization objective. Unsupervised incremental pre-training of the masked language model is then performed.

[0037] In one embodiment, the specific process of constructing the feature dimension folding architecture in S4 includes:

[0038] Obtain the global pooled semantic feature vector of the test questions output by the customized base model. ;

[0039] The semantic feature vector is processed through a single-branch fully connected network layer. The linear mapping is represented by a one-dimensional flattening tensor, specifically as follows:

[0040] ;

[0041] The one-dimensional flattened tensor is physically folded into a two-dimensional joint logarithmic probability tensor using a tensor reshaping operation, specifically represented as follows:

[0042] ;

[0043] in, This is the weight matrix of the single-branch fully connected network layer. For bias parameters, The total number of core competency dimensions in the three-dimensional education tensor assessment space. The number of levels of cognitive level of core competencies.

[0044] In one embodiment, the specific process of calculating the total multi-task loss in S4 using a cost-sensitive weighting mechanism includes:

[0045] A distribution scan of the label slots in the full training set was performed to count the actual frequency of occurrence of each core competency cognitive level. ;

[0046] Based on the statistical results, using the formula Dynamically calculate the cost-sensitive penalty weights for different cognitive levels of core competencies. ,in This represents the total number of evaluation nodes in the entire training set.

[0047] The calculated penalty weights for each level The loss function is transformed into a tensor and injected into the multi-task cross-entropy loss function. A high penalty gradient is assigned to the high-order level of the long-tail core competencies, which accounts for a very small proportion, to calculate the total loss of the multi-tasks. During backpropagation, the parameters of the customized base model are frozen, and only the parameters of the feature dimension folding architecture are updated.

[0048] In one embodiment, the specific process of automatically discovering and actively learning backflow of low-confidence samples based on confidence assessment in S5 includes:

[0049] Perform a Softmax activation operation on the joint logarithmic probability tensor output of the joint rating network forward propagation, and extract the highest predicted probability of the test item in each core competency dimension as the prediction confidence of that dimension.

[0050] Perform a maximum value index operation on the category dimension to obtain each predicted core competency covered by the test questions and its corresponding core competency cognitive level label;

[0051] Determine whether the predicted core competency cognitive level label is a non-zero level. If so, and the corresponding prediction confidence is lower than the preset safety threshold, then determine that the current test question is a low-confidence fuzzy sample.

[0052] The system automatically intercepts the low-confidence fuzzy samples, extracts their original test question text and current prediction output, and appends them to the local low-confidence database for expert review or as targeted incremental data for the next round of model iteration.

[0053] A test item literacy cognitive hierarchy intelligent annotation system based on multidimensional tensor joint optimization, the system comprising:

[0054] The data processing and tensor construction module is used to construct a three-dimensional educational tensor evaluation space involving test questions, subject literacy, and literacy cognitive level, and to collect real question banks to form a first dataset containing a small number of manually labeled test questions and a second dataset containing a large number of unlabeled test questions.

[0055] The diagnostic thinking chain pseudo-label enhancement module is used to reason and label the unlabeled test questions in the second dataset based on the first dataset as a reference example, using a large language model combined with the diagnostic thinking chain strategy. After data filtering and format conversion, the pseudo-label tensor augmentation dataset is obtained.

[0056] The mathematical prior and incremental pre-training module is used to clean and preprocess the test text, explicitly inject mathematical symbol priors, and use the pre-trained encoder language model to perform domain-adaptive incremental pre-training of the masked language model on the educational scenario corpus containing the pseudo-label tensor augmentation dataset to obtain a customized base model.

[0057] The dimension folding and cost-sensitive training module is used to construct a feature dimension folding architecture as a multi-task sharing and classification layer using the customized base model as a semantic representation base, calculate the total loss of the multi-tasks by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network using the first dataset and the pseudo-label tensor augmentation dataset.

[0058] The active learning and low-confidence feedback module is used to input the questions to be tested into the fine-tuned joint rating network, output the classification labels and probabilities of each core competency, and perform automatic discovery and active learning feedback of low-confidence samples based on confidence assessment.

[0059] Therefore, this application has the following beneficial effects:

[0060] I. Breakthrough in high-dimensional long-tail prediction bottleneck. By using a feature dimension folding architecture and a cost-sensitive weighting mechanism based on real slot statistics, the joint optimization of 36 fine-grained nodes is achieved in one step, solving the problems of redundant parameters and "all-negative prediction collapse" in existing parallel multi-classification heads. The macro F1 score jumps from 0.5375 to 0.5980, and the detection rate of high-order long-tail literacy is significantly improved.

[0061] II. Overcoming the challenge of small sample annotation. By leveraging a diagnostic mindset to drive a large language model, a high-quality pseudo-label augmentation dataset can be generated with only about 300 precisely labeled test questions. Compared to existing methods that rely on large-scale labeled data, the cost of manual annotation is reduced by more than two orders of magnitude.

[0062] Third, eliminating semantic barriers of mathematical symbols. By dynamically registering LaTeX symbols and performing domain-adaptive incremental pre-training, the model is given a deep understanding of mathematical logic. Ablation experiments show that removing this module reduces the macro F1 score to 0.5461.

[0063] IV. Constructing a continuously evolving data closed loop. Introducing a confidence-driven active learning feedback mechanism, low-confidence fuzzy samples are automatically identified and fed back, forming an iterative system of automatic annotation, problem discovery, and correction and retraining, which significantly improves engineering reliability and long-term availability. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a system flowchart of an intelligent annotation method for test literacy cognitive levels based on multidimensional tensors;

[0066] Figure 2 This is a schematic diagram of the three-dimensional tensor space structure of test questions, subject literacy, and literacy cognitive level provided in the embodiments of this application;

[0067] Figure 3 This is a schematic diagram of the data augmentation and pseudo-label generation architecture based on diagnostic thinking chain provided in the embodiments of this application;

[0068] Figure 4 This is a schematic diagram of the joint rating network structure based on feature dimension folding provided in the embodiments of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0071] To address the shortcomings of existing technologies, this application provides an intelligent annotation method for test item literacy cognitive levels based on multidimensional tensors, including steps S1-S5, as described above. Figure 1 , Figure 1 This is a system flowchart for an intelligent annotation method for test literacy cognitive levels based on multidimensional tensors.

[0072] S1: Construct a three-dimensional educational tensor evaluation space based on the subject curriculum standards, which includes test questions, subject literacy, and literacy cognition level. Obtain a first dataset that is anchored by experts through tensor instantiation based on the three-dimensional educational tensor evaluation space. At the same time, collect a large number of unlabeled test questions to form a second dataset.

[0073] S2: Based on the first dataset as a few sample examples, a three-stage deduction including analysis process, dimension matching and result assembly is performed on the unlabeled test items to be labeled in the second dataset using a large language model combined with a diagnostic thinking chain strategy to generate structured pseudo-label data. After data filtering and format conversion, a pseudo-label tensor augmentation dataset is obtained.

[0074] S3: Clean the test text, explicitly register mathematical formulas and geometric symbols in LaTeX format as special tags to the word segmenter of the pre-trained encoder language model and expand the embedding layer, perform domain-adaptive incremental pre-training of the masked language model on the educational corpus containing the pseudo-label tensor augmentation dataset, and obtain a customized base model.

[0075] S4: Using the customized base model as the semantic representation base, construct a feature dimension folding architecture as a multi-task sharing and classification layer, calculate the total loss of the multi-task by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network by using the full training set composed of the first dataset and the pseudo-label tensor augmentation dataset.

[0076] S5: Input the test text into the fine-tuned joint rating network, output the classification labels of each core competency dimension covered by the test text and their corresponding probabilities, and perform automatic discovery and active learning feedback of low confidence samples based on confidence assessment.

[0077] Specifically, in this embodiment, the present application provides an intelligent annotation method for test item literacy cognitive levels based on multidimensional tensors, including steps S1 to S5. (Refer to...) Figure 1 , Figure 1 This is a system flowchart of the intelligent annotation method for test item literacy cognitive levels based on multidimensional tensors provided in this application embodiment. The specific implementation process of this embodiment will be described in detail below with reference to the accompanying drawings.

[0078] Step S1: Constructing a three-dimensional educational tensor evaluation space and preparing the dataset.

[0079] Targeting the junior high school mathematics education scenario, this paper deeply integrates educational measurement theory with the "Compulsory Education Mathematics Curriculum Standards" to innovatively construct a three-dimensional tensor space of "Test Items - Subject Literacy - Literacy Level". The first dimension is the test item dimension, corresponding to specific test items in the question bank; the second dimension is the subject literacy dimension, defining nine core literacy dimensions including abstract ability, computational ability, geometric intuition, spatial reasoning, reasoning ability, data literacy, modeling ability, application awareness, and innovative awareness; the third dimension is the literacy level dimension, defining four continuous quantitative levels, where level 0 represents no involvement of that literacy, and levels 1 to 3 represent progressive abilities. In this application, "literacy level" refers to the examination level of a test item on a certain subject literacy dimension, which is divided according to cognitive depth, knowledge integration, transfer and application degree, and innovation requirements. It is different from the general cognitive level in the general sense, but rather a graded evaluation dimension that serves the fine-grained labeling of core literacy.

[0080] In the dataset construction phase, a double-blind annotation and arbitration mechanism was employed to obtain a high-quality first dataset. Specifically, two experts with extensive teaching and research backgrounds rigorously followed the aforementioned tensor system to conduct back-to-back independent tensor node anchoring for sampled test questions. After annotation, consistency comparisons were performed, and test questions with discrepancies in tensor dimensions or cognitive levels were ultimately labeled through review and arbitration, resulting in approximately 324 manually labeled test questions with extremely high teaching and research logical consistency, constituting the first dataset. Simultaneously, approximately 44,441 unlabeled test questions were collected from four major textbook systems—Beijing Normal University Press, People's Education Press, East China Normal University Press, and Yuanshi Press—to form the second dataset.

[0081] Step S2: Pseudo-label data augmentation based on diagnostic thinking chain.

[0082] To overcome the bottleneck of limited sample sizes hindering the development of deep networks, this step utilizes typical questions from the first dataset as examples of limited sample size. When calling the large language model's application programming interface (API), a carefully constructed diagnostic thought chain system uses prompts to force constraints on the model's output logic. For example... Figure 3 As shown, the large language model needs to perform three stages of deduction sequentially: the first stage is the analysis process, which internally examines the knowledge points tested in the question stem; the second stage is dimension matching, which deduces the logical complexity based on the key steps of problem-solving and anchors the nine core competencies and their corresponding levels 0 to 3 in the three-dimensional tensor space; the third stage is result assembly, which outputs only the labeled data that conforms to the preset JSON structure. After the model inference outputs, data filtering and format conversion are performed, the JSON data is parsed and invalid samples with empty dictionaries or empty lists in the competency label fields are deleted, and the retained valid data is mapped into numeric pseudo-label vectors, thereby obtaining a high-quality pseudo-label tensor augmentation dataset at low cost.

[0083] Step S3: Injecting domain-adaptive incremental pre-training with mathematical symbol priors.

[0084] Middle school math exam questions contain a large number of complex LaTeX-formatted mathematical formulas and geometric symbols, which general-purpose word segmenters tend to fragment, leading to a loss of mathematical and semantic meaning. This step loads a special character set from a pre-defined mathematical symbol library and dynamically registers it as a special marker into the base word segmenter of the pre-trained encoder language model, expanding the word embedding layer according to the formulas. Subsequently, a domain corpus is constructed by mixing the pseudo-labeled tensor augmentation dataset with the remaining massive amount of unlabeled exam text from the second dataset. A dynamic masking strategy is used to randomly mask the text sequence with a 15% probability, aiming to predict the true words at the masked positions. Unsupervised incremental pre-training of the masked language model is then performed to obtain a customized base model for mathematical scenarios.

[0085] Step S4: Feature dimension folding and cost-sensitive joint fine-tuning training.

[0086] This step uses a customized base model as the semantic representation base, extracting the pooling vector corresponding to the [CLS] position in the output of the last layer of the model as the deep semantic feature vector of the test question. Unlike the parallel multi-classification head architecture of existing technologies, this step constructs a shared network consisting of single-branch fully connected layers. A one-dimensional flattened tensor is obtained through linear mapping, and a tensor reshaping operation is used to fold the one-dimensional flattened tensor into a log-probability tensor that represents the joint distribution of the nine literacy skills and the cognitive levels of the four core literacy skills. Simultaneously, to address the problem of all-negative prediction collapse caused by the extreme long-tail distribution of literacy labels, the frequency of occurrence of each literacy cognitive level slot in the full training set is globally statistically analyzed. Cost-sensitive penalty weights are dynamically calculated according to a formula, assigning high penalty gradients to the long-tail high-order levels with a very small proportion. During the fine-tuning training phase, the parameters of the customized base model are frozen, and only the multi-task cross-entropy loss function with injected penalty weights is used to update the parameters of the single-branch fully connected layers, completing the training of the joint rating network.

[0087] Step S5: Confidence-driven structured output and active learning feedback.

[0088] The test text to be labeled is input into the fine-tuned joint rating network. A single forward propagation computes the log-probability tensor of the multi-competency, multi-level joint distribution. A Softmax activation operation is performed on the core competency cognitive level dimension to obtain the predicted probability distribution for each competency dimension. A maximum value index operation is then used to obtain the predicted core competency cognitive level label for each competency dimension. Simultaneously, the highest predicted probability value for each competency dimension is extracted as the prediction confidence. If any competency dimension has a non-zero predicted core competency cognitive level label at a level below a preset confidence threshold, the test text is classified as a low-confidence sample. The system automatically extracts its question stem and current prediction output, appending them to the local low-confidence database for subsequent expert review or as targeted incremental training data for the next round of model iterations, forming a continuously optimized active learning feedback loop.

[0089] In one embodiment, step S1 includes:

[0090] Based on the "Compulsory Education Mathematics Curriculum Standards," a three-dimensional educational tensor evaluation space is constructed, encompassing test items, subject literacy, and literacy cognitive levels. The first dimension represents the test item dimension, symbolizing specific test item entities in the question bank. The second dimension represents the subject literacy dimension, defining nine core competencies including abstract thinking, computational ability, geometric intuition, spatial reasoning, reasoning ability, data literacy, modeling ability, application awareness, and innovative awareness. The third dimension represents the literacy cognitive level dimension, defining four continuous quantitative levels for each subject literacy dimension: level zero, representing no involvement of that competency; and levels one, two, and three, representing progressive abilities. The literacy cognitive level refers to the examination level of a test item on a specific subject literacy dimension, categorized based on cognitive depth, knowledge integration, transfer and application, and innovation requirements, serving as a fine-grained, hierarchical evaluation dimension representing core competencies.

[0091] Obtain the first dataset anchored by experts through tensor instantiation based on the three-dimensional education tensor evaluation space;

[0092] A second dataset was constructed by collecting a large number of unlabeled test questions covering at least four different textbook versions.

[0093] Specifically, in this embodiment, this application deeply integrates educational assessment theory with the "Compulsory Education Mathematics Curriculum Standards" to construct a three-dimensional tensor space of "Test Items - Subject Literacy - Literacy Level" (ILL) to achieve fine-grained quantitative representation of core competencies. In the ILL tensor, the first dimension is the test item dimension (Item), representing the test item entities in the question bank; the second dimension is the subject literacy dimension (Literacy, where...) The specific definitions and examination content are shown in Table 1:

[0094] Table 1 Evaluation Index System for Core Competencies in Junior High School Mathematics

[0095]

[0096] The third dimension is the level of literacy and cognitive ability (Level, where...) For each of the above qualities, four levels are defined: 0, 1, 2, and 3.

[0097] Level 0: The test questions do not assess this core competency;

[0098] Level 1: Basic understanding and simple application, requiring only single-step substitution or direct recognition;

[0099] Level 2: Comprehensive understanding and complex application, requiring cross-module convergence of knowledge points and multi-step logical deduction;

[0100] Level 3: Deep migration and innovative applications require building extremely complex implicit logic chains or solving open and unconventional problems.

[0101] Learning resources were collected from real-world teaching and research scenarios. The construction of the first dataset essentially involved teaching and research experts performing three-dimensional instantiation in tensor space. An extremely rigorous double-blind annotation mechanism was employed: two experts with extensive teaching and research backgrounds rigorously followed the aforementioned tensor system to conduct back-to-back independent node anchoring of sampled test questions. After annotation, consistency checks were performed. For test questions with discrepancies in tensor dimensions or cognitive levels, final determination was made through review and arbitration, resulting in approximately 324 meticulously labeled test questions with extremely high logical consistency in teaching and research, forming the high-quality first dataset. Simultaneously, approximately 44,441 unlabeled test questions were collected from textbooks published by Beijing Normal University Press, People's Education Press, East China Normal University Press, and Yuanshi Press, forming the second dataset.

[0102] The core symbols involved in the embodiments of this application are shown in Table 2.

[0103] Table 2. Symbol table related to the embodiments of this application

[0104]

[0105] In one embodiment, the specific process of obtaining the first dataset includes:

[0106] Two annotators independently and double-blindly instantiated and anchored a small number of sampled test items based on the three-dimensional education tensor evaluation space. The anchoring results were compared, and test items with discrepancies were reviewed and arbitrated to obtain the first dataset with higher education research logic consistency.

[0107] Specifically, in this embodiment, the detailed process of obtaining the first dataset in step S1 is as follows:

[0108] The construction of the first dataset is essentially a process in which educational research experts instantiate and anchor three-dimensional nodes—test questions, subject literacy, and literacy cognition levels—within the three-dimensional educational tensor evaluation space. The quality of this annotation directly determines the upper limit of the training effect for all subsequent models. To ensure the authority and consistency of the annotation results, this embodiment employs a strict independent double-blind annotation and disagreement arbitration mechanism.

[0109] First, two experts with extensive experience in junior high school mathematics teaching and research were selected as annotators. Before the annotation began, the two experts received standardized training on tensor annotation specifications to ensure that both parties reached a consensus on the definition of the nine core competencies and the criteria for classifying competency cognitive levels from zero to three. Subsequently, a small number of representative test questions were extracted from a real question bank, covering different knowledge modules such as algebra, geometry, statistics and probability, as well as different difficulty levels.

[0110] Secondly, two annotators entered a back-to-back, independent, double-blind annotation phase. In an isolated environment without communication, each annotator, based on a pre-constructed three-dimensional educational tensor assessment space, annotated the tensor nodes of each sampled question. Specifically, for each question, the annotator needed to determine whether it involved assessment across the nine subject competency dimensions, and further determine the core competency cognitive level level for those competency dimensions. The annotation results were recorded in the form of structured tensor labels.

[0111] Secondly, after the double-blind annotation is completed, the system automatically compares the anchoring results of the two annotators on a question-by-question and dimension-by-dimensional basis. For questions where the two annotation results are completely consistent across all nine competency dimensions and the corresponding core competency cognitive level, they are directly marked as high-confidence samples and included in the first dataset. For questions where there is disagreement on any competency dimension or cognitive level, the system marks them as disputed samples and submits them to a third expert with higher educational research authority for review and arbitration. The arbitration expert makes a final ruling based on the curriculum standards and the question's design intent, combined with the reasons given by both annotators, and determines the final tensor label for the question.

[0112] Through the three-stage process of double-blind labeling, consistency comparison, and disagreement arbitration described above, this embodiment ultimately obtained approximately 324 manually labeled test questions with extremely high teaching and research logical consistency and label accuracy, constituting the first dataset. Although this dataset is limited in size, the label quality is extremely high, providing a reliable benchmark for subsequent small-sample examples of diagnostic thinking chains.

[0113] In one embodiment, step S2 includes:

[0114] A predetermined number of test questions are randomly selected from the first dataset, and their test texts and corresponding tensor instantiation anchor labels are extracted to construct a small sample example set. At the same time, a diagnostic thinking chain system prompt word is constructed. The system prompt word forces the large language model to follow the three-stage inference logic of analysis process, dimension matching and result assembly.

[0115] The large language model application programming interface is invoked, and the unlabeled test questions in the second dataset and the few sample example set are input together. The large language model then performs the following three-stage deduction process in sequence: internally reviewing the knowledge points tested in the unlabeled test questions; deriving the logical complexity based on the key steps of problem-solving and anchoring the core competency dimensions and their corresponding cognitive level levels in the three-dimensional education tensor evaluation space; and outputting labeled data that conforms to the preset JSON structure.

[0116] Data filtering and format conversion are performed on the JSON structured label data output by the large language model, including parsing the JSON data, identifying and deleting invalid samples with missing target literacy label fields, empty dictionaries, or empty lists; the retained valid label data is mapped to numerical pseudo-label vectors corresponding to the three-dimensional education tensor evaluation space to obtain the pseudo-label tensor augmentation dataset.

[0117] Specifically, in this embodiment, to address the problem that small samples cannot drive deep networks, typical questions from the first dataset are used as examples of few samples. This embodiment focuses on illustrating the processing mechanism of this application when facing extreme long-tail literacy data. In junior high school math exam questions, Level 3 (deep transfer and innovative application) of "innovation awareness" and "model concept" are among the very few high-frequency points where points are lost. Traditional models are very prone to full negative prediction collapse at such nodes.

[0118] In this embodiment, the entire training set is first scanned to obtain the frequency distribution matrix of the 36 rating nodes. Assume that in one statistical analysis, the frequency of occurrence of "Computational Ability_Level 1" is... The number of times "Innovation Awareness_Level 3" appeared .

[0119] The penalty weight is dynamically calculated based on the formula. For "Innovation Awareness_Level 3", its weight will be adaptively amplified to an extremely high level (e.g., (times the benchmark weight).

[0120] When fine-tuning the network, this weight matrix is ​​transformed into a PyTorch tensor-injected multi-task joint cross-entropy loss function. When the model predicts a level 0 for the question "Innovation Awareness_Level 3", an extremely large gradient penalty will occur. Experiments show that after introducing this mechanism, the recall rate of long-tail core competencies surges, breaking the limitations of long-tail data.

[0121] When calling the large language model API, the model output is constrained by structured prompts. The large model should output the following in sequence: [Analysis Process] (Internal review of the knowledge points in the question stem). [Dimensional Matching] (Deducing complexity based on key problem-solving steps and matching the 9 competencies in Table 1 with the 0-3 level) [Result Assembly] (Pure JSON Output).

[0122] After obtaining the output, a filtering mechanism is executed to parse the JSON. If the `literacy_labels` field is missing, an empty dictionary, or an empty list, the entire sample is removed. Finally, a high-quality pseudo-label tensor augmentation dataset is obtained at low cost.

[0123] This embodiment uses a Python program to construct a batch API call stream. During the call process, the system carefully designs and issues the following system prompts:

[0124] "You are a seasoned junior high school math education expert. Your task is to label the given test questions with core competencies based on the provided examples."

[0125] Analysis process: Internally review the knowledge points tested in the question (such as functions, geometric proofs, etc.).

[0126] Dimension matching: Based on the key steps of problem-solving, the logical complexity is deduced, and 9 core competencies are matched and their levels (levels 1 to 3) are strictly determined.

[0127] Result assembly: Only output labeled results that conform to the JSON structure. The inclusion of inference process text in the output is strictly prohibited.

[0128] Example for reference: [Insert Few-Shot data extracted from the first dataset here].

[0129] After receiving the JSON response, the program executes the filter_empty_labels script to filter out dirty data with empty literacy labels and map them to generate a numeric pseudo-label vector in the form of [0,0,0,1,0,3,0,0,0]. Figure 2 This paper illustrates the construction logic of a three-dimensional tensor evaluation space consisting of "test items - subject literacy - literacy cognitive level". Based on Bloom's Taxonomy of Educational Objectives and the Q-matrix of cognitive diagnosis, and combined with the "Compulsory Education Mathematics Curriculum Standards (2022 Edition)", this framework integrates test item entities with nine subject literacy dimensions and progressive literacy levels from 0 to 3 into a unified quantitative structure. This achieves the transformation from abstract educational theory to a computable tensor representation, providing a standardized data foundation for subsequent intelligent annotation and tiered evaluation.

[0130] In one embodiment, step S3 includes:

[0131] The test text is cleaned to remove irrelevant hypertext markup language tags and noise characters; a special character set containing mathematical formulas and geometric symbols in LaTeX format is loaded from a preset mathematical symbol library, and the special character set is dynamically registered as a special tag into the base word segmenter of the pre-trained encoder language model;

[0132] The word embedding layer of the pre-trained encoder language model is expanded based on the registration results of the special tags;

[0133] The test texts in the pseudo-labeled tensor augmentation dataset are mixed with the remaining massive amount of unlabeled test texts in the second dataset to construct an educational corpus for mathematics. A dynamic masking strategy is used to randomly mask the text sequences in the educational corpus with a preset probability, and unsupervised incremental pre-training of the masked language model is performed with the prediction of the real words at the masked positions as the optimization objective to obtain the customized base model.

[0134] Specifically, in this embodiment, the test texts in the pseudo-labeled tensor augmentation dataset obtained above are merged with the remaining massive amount of real unlabeled test texts in the second dataset to construct a domain corpus specifically for junior high school mathematics.

[0135] To prevent the pre-trained model's word segmenter from breaking down complex LaTeX mathematical formulas and losing their mathematical semantics, a specific formula character set is loaded from a predefined mathematical symbol library, registered as a special tag, and the word embedding layer dimension of the basic pre-trained language model is expanded simultaneously.

[0136] The processed word segmenter is used to randomly mask the text in the domain corpus with a set probability, and unsupervised masked language model reconstruction training is performed. This enables the base model to deeply understand the syntactic structure and logical connections unique to junior high school mathematics in the hidden feature space, eliminating the distributional bias between general natural language corpora and the vertical mathematical domain. Figure 3 As shown in the figure, this diagram illustrates the pseudo-label augmentation process based on a diagnostic thinking chain. First, a small number of expert-labeled data are used to construct Few Shot examples, driving a large language model to sequentially perform a three-step reasoning process on a massive number of unlabeled test questions: "analyzing the knowledge points in the question stem—deducing logical complexity to match the cognitive levels of the nine literacy skills and the core literacy skills at levels 1-3—assembling structured JSON." The output is tensor data containing literacy labels. Subsequently, by parsing the JSON and removing invalid samples with empty label fields, data filtering and format conversion are achieved, ultimately obtaining a high-quality pseudo-label tensor augmentation dataset. This provides low-cost, logically consistent training support for downstream small-sample fine-tuning.

[0137] In one embodiment, the step of expanding the word embedding layer of the pre-trained encoder language model based on the registration result of the special tag includes:

[0138] Extracting high-frequency formulas and geometric characters from junior high school mathematics to construct a special tag word list;

[0139] The special tag vocabulary is dynamically registered to the base word segmenter of the pre-trained encoder language model, and the word embedding layer is expanded using a formula, which is:

[0140]

[0141] in For the original embedded layer, Based on the vocabulary list, For the newly added special marker vocabulary;

[0142] A dynamic masking strategy is adopted to randomly mask educational scenario corpora containing test questions, and the log-likelihood loss is calculated with the real words at the predicted mask positions as the optimization objective. Unsupervised incremental pre-training of the masked language model is then performed.

[0143] Specifically, in this embodiment, the junior high school math test questions contain a large number of complex LaTeX symbols (such as...). , This step involves extracting high-frequency mathematical symbols to form a proprietary vocabulary (such as cleaned_formulas.txt), registering it into the word segmenter of a pre-trained language model, and simultaneously expanding the embedding layer.

[0144]

[0145] Mix pseudo-labeled text with massive amounts of text from a second dataset. For the input sequence... The masked sequence is generated with a 15% probability. Let the set of masked positions be . Reconstruct the original words using the model and calculate the log-likelihood loss of the masked language model (MLM):

[0146]

[0147] After thorough training, save the parameters of the customized base model for junior high school mathematics scenarios. .

[0148] In one embodiment, the specific process of constructing the feature dimension folding architecture in S4 includes:

[0149] Obtain the global pooled semantic feature vector of the test questions output by the customized base model. ;

[0150] The semantic feature vector is processed through a single-branch fully connected network layer. The linear mapping is represented by a one-dimensional flattening tensor, specifically as follows:

[0151] ;

[0152] The one-dimensional flattened tensor is physically folded into a two-dimensional joint logarithmic probability tensor using a tensor reshaping operation, specifically represented as follows:

[0153] ;

[0154] in, This is the weight matrix of the single-branch fully connected network layer. For bias parameters, The total number of core competency dimensions in the three-dimensional education tensor assessment space. The number of levels of cognitive level of core competencies.

[0155] Specifically, in this embodiment, the segmented sequence is input to a customized base, and the last layer [CLS] vector is extracted as the global semantic feature of the test question. (like ). Build a single-branch fully connected layer (weight) bias ),Will After mapping, dimensionality folding is performed through tensor reshaping, outputting a joint log-probability tensor:

[0156]

[0157]

[0158] To address the problem of an excessive long-tail distribution of "Level 0" labels, the training focuses on the cognitive level of the target audience. True slot frequency Perform statistical analysis and calculate cost-sensitive weights:

[0159]

[0160] For the first in the input batch The sample, which is in the th , A true label of personal qualities Define the cost-sensitive multi-task cross-entropy total loss:

[0161]

[0162] The base parameters are frozen, and backpropagation is used for fine-tuning only at the classification layer. (See figure.) Figure 4This paper presents a fine-tuning architecture for a joint rating network based on feature dimension folding and cost-sensitive weighting. The test text is pre-trained in a customized foundation model in the mathematics domain to extract globally pooled semantic vectors. These vectors are then mapped to a 1D 36D log-probability tensor through a single-branch fully connected layer. Tensor reshaping is then used to physically fold the vectors into a 9×4 2D joint distribution matrix, achieving a compact representation of multiple literacy levels and cognitive hierarchy. During training, cost-sensitive penalty weights based on label slot statistics are introduced, injecting multi-task cross-entropy total loss. This forces the network to focus on a few high-order literacy levels under extreme long-tail distributions, thereby overcoming the bottleneck of all-negative prediction collapse and improving fine-grained rating accuracy.

[0163] In one embodiment, the specific process of calculating the total multi-task loss in S4 using a cost-sensitive weighting mechanism includes:

[0164] A distribution scan of the label slots in the full training set was performed to count the actual frequency of occurrence of each core competency cognitive level. ;

[0165] Based on the statistical results, using the formula Dynamically calculate the cost-sensitive penalty weights for different cognitive levels of core competencies. ,in This represents the total number of evaluation nodes in the entire training set.

[0166] The calculated penalty weights for each level The loss function is transformed into a tensor and injected into the multi-task cross-entropy loss function. A high penalty gradient is assigned to the high-order level of the long-tail core competencies, which accounts for a very small proportion, to calculate the total loss of the multi-tasks. During backpropagation, the parameters of the customized base model are frozen, and only the parameters of the feature dimension folding architecture are updated.

[0167] Specifically, in this embodiment, the detailed process of calculating the total multi-task loss in step S4 using the cost-sensitive weighting mechanism is described below:

[0168] First, the distribution of real-label data was scanned on the full training set, which consisted of a mixture of the first dataset and the pseudo-label tensor augmented dataset. Since the three-dimensional education tensor assessment space contains nine subject literacy dimensions and four literacy cognitive level levels, totaling thirty-six fine-grained classification nodes, the system traversed every question in the full training set, counting the actual frequency of each core literacy cognitive level. The statistical results show that level zero labels occupy the vast majority of slots, while level three labels, representing higher-order core literacy cognitive levels, appear with extremely low frequencies, exhibiting a typical extreme long-tail distribution characteristic.

[0169] Secondly, based on the statistical results, the system uses the formula:

[0170]

[0171] Dynamically calculate the cost-sensitive penalty weights for different cognitive levels of core competencies. In the formula, The total number of assessment nodes in the full training set is the product of the number of test questions and the number of subject literacy dimensions. The level of core competency cognitive level is represented by a value of four. For the first The frequency of a level in real data. The physical meaning of this formula is that the lower the frequency of a level, the higher its penalty weight, thus incentivizing the model to focus on scarce, long-tail, high-order competencies. Taking the third level of the "innovation awareness" competency as an example, if its frequency is only a few dozen times, its penalty weight will be adaptively amplified to hundreds of times the base weight.

[0172] Finally, the calculated penalty weights for each level are transformed into tensor form and injected into the multi-task cross-entropy loss function. Specifically, when calculating the total loss for each training batch, the prediction error for different core competency cognitive level levels is multiplied by the corresponding penalty weight, causing the model to generate a larger gradient update for erroneous predictions of long-tail high-order levels. During the backpropagation phase, the system freezes all parameters of the customized base model and only updates the parameters of the single-branch fully connected layer in the feature dimension folding architecture. This differential fine-tuning strategy not only protects the mathematical and semantic understanding capabilities gained by the base model through incremental pre-training but also forces the classification layer to accurately capture the discriminative features of minority high-order competencies under the guidance of the cost-sensitive mechanism, fundamentally solving the problem of all-negative prediction collapse under long-tail distribution.

[0173] After the model's forward propagation, Argmax is executed on the last dimension to obtain the prediction levels for each literacy, and the confidence matrix is ​​extracted simultaneously.

[0174]

[0175] If any non-zero predictive competence of a certain test question If the sample is below a preset threshold, it will be automatically added to the local low_confidence_data.json database as a "low-confidence fuzzy sample" to build a dynamic learning loop.

[0176] In one embodiment, the specific process of automatically discovering and actively learning backflow of low-confidence samples based on confidence assessment in S5 includes:

[0177] Perform a Softmax activation operation on the joint logarithmic probability tensor output of the joint rating network forward propagation, and extract the highest predicted probability of the test item in each core competency dimension as the prediction confidence of that dimension.

[0178] Perform a maximum value index operation on the category dimension to obtain each predicted core competency covered by the test questions and its corresponding core competency cognitive level label;

[0179] Determine whether the predicted core competency cognitive level label is a non-zero level. If so, and the corresponding prediction confidence is lower than the preset safety threshold, then determine that the current test question is a low-confidence fuzzy sample.

[0180] The system automatically intercepts the low-confidence fuzzy samples, extracts their original test question text and current prediction output, and appends them to the local low-confidence database for expert review or as targeted incremental data for the next round of model iteration.

[0181] Specifically, in this embodiment, the detailed process of automatically discovering and actively learning backflow of low-confidence samples based on confidence assessment in step S5 is described below:

[0182] First, after fine-tuning the joint rating network based on the annotated test text input, a single forward propagation is used to compute the joint log-probability tensor output by the dimensionality-folding architecture. A Softmax activation operation is then performed on this joint log-probability tensor along the core competency cognitive level dimensions, transforming the original log-probability values ​​into normalized probability distributions for each competency dimension from level zero to level three. Based on this, for each core competency dimension, the highest probability value in its corresponding probability distribution is extracted as the prediction confidence for that dimension. This prediction confidence quantifies the degree of certainty with which the model predicts the current competency dimension.

[0183] Secondly, a maximum value indexing operation is performed on the core competency cognitive level dimension to obtain the predicted core competency cognitive level labels for each competency dimension. Thus, the joint rating network outputs two key pieces of information for each test question: first, the predicted core competency cognitive level labels for each of the nine core competency dimensions; and second, the prediction confidence level for each competency dimension.

[0184] Next, the system executes low-confidence judgment logic on the prediction results. Specifically, it iterates through the prediction outputs of all nine competency dimensions and determines whether two conditions are met simultaneously: First, the predicted core competency cognitive level label for that competency dimension is a non-zero level, meaning the model determines that the question does indeed test a certain cognitive depth of that competency; second, the prediction confidence corresponding to the predicted core competency cognitive level label is lower than a preset safety threshold, for example, a threshold set to 0.6. If a question meets both of the above conditions simultaneously on at least one competency dimension, the system determines that the question is a low-confidence fuzzy sample.

[0185] Finally, the system automatically performs an active learning backflow operation. For questions identified as low-confidence fuzzy samples, the system intercepts their direct insertion into the database, automatically extracts the original question stem text and all prediction results output by the current model, and packages them into a local, independent low-confidence database. The data in this low-confidence database can be periodically reviewed by platform researchers who can provide manual confirmation labels, and can also serve as targeted incremental data for the next round of model iteration training. By accurately backflowing low-confidence samples into the training set, the model continuously strengthens its ability to learn from fuzzy boundary samples and long-tail higher-order competencies, forming a continuous evolutionary closed loop of "automatic labeling - problem identification - expert correction - model retraining," significantly improving the system's engineering reliability and long-term availability in a massive dynamic question bank environment.

[0186] A test item literacy cognitive hierarchy intelligent annotation system based on multidimensional tensor joint optimization, the system comprising:

[0187] The data processing and tensor construction module is used to construct a three-dimensional educational tensor evaluation space involving test questions, subject literacy, and literacy cognitive level, and to collect real question banks to form a first dataset containing a small number of manually labeled test questions and a second dataset containing a large number of unlabeled test questions.

[0188] The diagnostic thinking chain pseudo-label enhancement module is used to reason and label the unlabeled test questions in the second dataset based on the first dataset as a reference example, using a large language model combined with the diagnostic thinking chain strategy. After data filtering and format conversion, the pseudo-label tensor augmentation dataset is obtained.

[0189] The mathematical prior and incremental pre-training module is used to clean and preprocess the test text, explicitly inject mathematical symbol priors, and use the pre-trained encoder language model to perform domain-adaptive incremental pre-training of the masked language model on the educational scenario corpus containing the pseudo-label tensor augmentation dataset to obtain a customized base model.

[0190] The dimension folding and cost-sensitive training module is used to construct a feature dimension folding architecture as a multi-task sharing and classification layer using the customized base model as a semantic representation base, calculate the total loss of the multi-tasks by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network using the first dataset and the pseudo-label tensor augmentation dataset.

[0191] The active learning and low-confidence feedback module is used to input the questions to be tested into the fine-tuned joint rating network, output the classification labels and probabilities of each core competency, and perform automatic discovery and active learning feedback of low-confidence samples based on confidence assessment.

[0192] Specifically, in this embodiment, this application provides an intelligent annotation system for test item literacy cognitive levels based on multidimensional tensor joint optimization. This system is based on the same application concept as the aforementioned method embodiment and performs the steps in the above method embodiment accordingly. The specific structure and function of each module of the system in this embodiment will be described in detail below.

[0193] The data processing and tensor construction module is used to construct a three-dimensional educational tensor evaluation space of "test questions - subject literacy - literacy cognitive level" based on the "Compulsory Education Mathematics Curriculum Standards," and to manage the first dataset, which is anchored by tensor instantiation by experts, and the second dataset, which consists of a large number of unlabeled test questions. This module includes a tensor structure definition unit and a dataset management unit. The tensor structure definition unit configures the subject literacy dimension into nine core competencies, including abstract thinking, computational ability, geometric intuition, spatial reasoning, reasoning ability, data literacy, modeling ability, application awareness, and innovative awareness. It also divides the core competency cognitive level dimension into a zero level representing non-compliance aspects and three consecutive quantitative levels representing progressively higher competencies. The dataset management unit stores and manages the first dataset, consisting of approximately 324 high-quality manually labeled test questions obtained through double-blind annotation and a disagreement arbitration mechanism, and the second dataset, consisting of 44,441 unlabeled test questions covering four textbook versions.

[0194] The diagnostic thinking chain pseudo-label enhancement module, connected to the data processing and tensor construction module, is used to reason and label the unlabeled test items in the second dataset based on the first dataset as a few-sample example, utilizing a large language model combined with a diagnostic thinking chain strategy. This module internally includes a thinking chain prompt construction unit, a large language model invocation unit, and a data filtering and transformation unit. The thinking chain prompt construction unit generates system prompts containing three stages of logic: analysis process, dimension matching, and result assembly. The large language model invocation unit sends the unlabeled test items and few-sample examples to the large language model via an application programming interface (API) and receives the JSON-formatted label data output by the model. The data filtering and transformation unit parses the JSON data, removes invalid samples with missing or empty literacy label fields, and maps the valid label data into numerical pseudo-label vectors corresponding to the three-dimensional tensor space, ultimately outputting a pseudo-label tensor augmented dataset.

[0195] The mathematical prior and incremental pre-training module, connected to the diagnostic thinking chain pseudo-label enhancement module, is used to clean and preprocess the test text and explicitly inject mathematical symbol priors. This module includes a special symbol registration unit, an embedding layer expansion unit, and a masked language model pre-training unit. The special symbol registration unit loads a set of special characters containing LaTeX-formatted mathematical formulas and geometric symbols from a pre-defined mathematical symbol library and dynamically registers them as special tags to the word segmenter of the pre-trained encoder language model. The embedding layer expansion unit adjusts the word embedding layer dimension based on the registration results. The masked language model pre-training unit mixes the pseudo-labeled tensor augmentation dataset with the second dataset to construct a domain corpus, and performs unsupervised incremental pre-training of the masked language model using a dynamic masking strategy to obtain a customized base model.

[0196] The dimension folding and cost-sensitive training module, connected to the mathematical prior and incremental pre-training module, is used to construct a feature dimension folding architecture and fine-tune the joint rating network using a customized base model as the semantic representation base. This module includes a semantic feature extraction unit, a dimension folding classification unit, and a cost-sensitive fine-tuning unit. The semantic feature extraction unit extracts the pooling vectors corresponding to positions in the customized base model as deep semantic features of the test questions. The dimension folding classification unit performs linear mapping of semantic features through a single-branch fully connected layer and uses tensor reshaping operations to fold the one-dimensional flattened tensor into a log-probability tensor that represents the nine literacy skills and the joint distribution of their four literacy cognitive levels. The cost-sensitive fine-tuning unit performs global statistics on the frequency of occurrence of each literacy cognitive level slot in the full training set, calculates the cost-sensitive penalty weights, injects them into the multi-task cross-entropy loss function, and updates the parameters of the classification layer while freezing the base parameters.

[0197] The active learning and low-confidence feedback module, connected to the dimensionality folding and cost-sensitive training module, is used to input the test questions to be predicted into the fine-tuned joint rating network, output the classification labels and probabilities of each core competency, and perform automatic discovery and active learning feedback of low-confidence samples. This module internally includes a joint prediction output unit, a confidence assessment unit, and a low-confidence feedback unit. The joint prediction output unit performs a single forward propagation and calculates the predicted core competency cognitive level labels and corresponding confidence levels for each competency dimension; the confidence assessment unit determines whether any competency dimension has a predicted label with a non-zero level and a confidence level lower than a preset threshold; the low-confidence feedback unit automatically extracts the test question stems and predicted outputs that meet the judgment conditions and appends them to the local low-confidence database, forming a continuously optimized closed-loop iterative mechanism.

[0198] To scientifically and rigorously verify the effectiveness of the method in this application, the method provided in this application was validated against three classic deep learning text annotation baseline methods on a strictly consistent reserved test set.

[0199] The specific baselines include: 1) Vanilla baseline: adopts a general open source base and a traditional multi-class head parallel architecture, using standard cross-entropy loss; 2) Focal Loss baseline: introduces a focus loss function on the basis of Vanilla in order to alleviate class imbalance; 3) Oversampling baseline: replicates and expands long-tail high-order literacy samples at the data level.

[0200] To comprehensively measure the model's performance in complex educational assessment scenarios, this embodiment selects three core evaluation indicators:

[0201] 1) Level Accuracy: Measures the proportion of times the model makes completely correct predictions across all 36 rating points and all test items;

[0202] 2) Macro-F1 Score: This score is the harmonic mean of precision and recall across all competencies and levels. It assigns equal weight to all categories and is extremely challenging in terms of the model's ability to identify rare "long-tail categories" (such as innovation awareness at level 3).

[0203] 3) Overall Detection Rate: The task is reduced to binary classification (only judging whether the test question contains a certain competency without considering the level), which measures the model's ability to discover and recall implicit core competencies.

[0204] The comparison results on the independent test set are shown in Table 3.

[0205] Table 3 Comparison of experimental performance on test sets for different annotation methods

[0206]

[0207] Detailed experimental results and technical mechanism analysis:

[0208] First, it breaks through the performance bottleneck under high-dimensional long-tailed distributions. As shown in Table 3, while the Vanilla baseline achieves a rank accuracy of 83.29%, its Macro-F1 score, which measures the detection capability of rare higher-order literacy, is only 0.5375. This indicates that traditional architectures tend to conservatively predict "level 0" when facing extremely imbalanced tensor data. Although introducing FocalLoss and oversampling techniques can improve the Macro-F1 to the 0.54-0.56 range to some extent, oversampling easily leads to overfitting (manifested as a decrease in detection rate). In contrast, the method in this application achieves a Macro-F1 score of 0.5980, a significant leap. This fully demonstrates that the proposed method of "dynamically calculating cost-sensitive weights based on real statistical slots" combined with a "feature dimension folding tensor architecture" not only reduces parameter redundancy but also forces the network to transcend the local optima of all-negative predictions, accurately capturing rare higher-order literacy features.

[0209] Second, it significantly improves the ability to extract deep features of complex mathematical logic and implicit literacy. In terms of the overall literacy detection rate, the method in this application achieves the highest rate of 89.56%. Traditional models, when processing test questions containing complex LaTeX geometric and algebraic expressions, suffer from fragmented mathematical semantics due to segmentation defects, thus missing implicit literacy within the test questions. This application, through "MLM incremental pre-training with injected mathematical prior special symbols," enables the base model to truly understand the logical conjunctions and formula syntax unique to junior high school mathematics in the hidden feature space, significantly reducing the false negative rate of literacy detection.

[0210] To further verify the independent contribution of the three core modules proposed in this application in dealing with high-dimensional long-tailed educational tensors, this embodiment designed three variant models for ablation studies, and the results are shown in Table 4.

[0211] Table 4 Comparison of Ablation Test Performance of Core Components

[0212]

[0213] Analysis of ablation test results:

[0214] 1) The "accuracy paradox" of removing the mind chain data augmentation (w / o Diag-CoT): When relying on only a very small number of 324 manually labeled samples, the model exhibits highly deceptive behavior. Its rank accuracy abnormally rises to 0.8586, but the Macro-F1 score collapses catastrophically (dropping to 0.4496). This indicates that with extremely small samples, the network completely abandons fitting highly difficult and scarce literacy, outputting "level 0" indiscriminately. This confirms that the pseudo-label tensor distilled from the large model is the absolute cornerstone for activating the high-order mapping ability of the small model.

[0215] 2) Boundary degradation after removing cost-sensitive weights: After removing these weights, the overall detection rate of the model remained high at 89.17% (indicating that the model can still understand the test points), but the Macro-F1 dropped to 0.5634. This proves that the lack of asymmetric gradient protection causes the model to hesitate to give high-depth judgments such as "level 3" when making fine-grained judgments, highlighting the key role of the weighting mechanism in fine-grained partitioning.

[0216] 3) Removing semantic fragments from the Math-Aware pre-training: When reverting to a general text base, the Macro-F1 score drops to 0.5461. This demonstrates that the tokenizer's fragmentation of formulas disrupts the ability to trace mathematical logic links, proving the necessity of building a bottom-level semantic library for vertical domains.

[0217] The aforementioned experiments powerfully demonstrate that traditional single fine-tuning methods face severe bottlenecks under extreme engineering conditions, including complex tasks with up to 36 fine-grained tensor rating nodes and extremely limited expert-calibrated data. This application, however, comprehensively utilizes "thinking chain data augmentation" to overcome sample bottlenecks, "mathematical prior fine-tuning" to bridge domain gaps, and "dimensional folding and cost-sensitive weighting" to combat long-tail distributions, constructing a logically rigorous and high-performance tensor rating network. All three core indicators of this application achieve comprehensive leadership, fully validating the effectiveness, cutting-edge nature, and extremely high industrialization value of this technical solution in the field of automated assessment for smart education.

[0218] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0219] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0220] It should be particularly noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent annotation of test item literacy cognitive levels based on multidimensional tensors, characterized in that, Includes the following steps: S1: Construct a three-dimensional educational tensor evaluation space based on the subject curriculum standards, which includes test questions, subject literacy, and literacy cognition level. Obtain a first dataset that is anchored by experts through tensor instantiation based on the three-dimensional educational tensor evaluation space. At the same time, collect a large number of unlabeled test questions to form a second dataset. S2: Based on the first dataset as a few sample examples, a three-stage deduction including analysis process, dimension matching and result assembly is performed on the unlabeled test items to be labeled in the second dataset using a large language model combined with a diagnostic thinking chain strategy to generate structured pseudo-label data. After data filtering and format conversion, a pseudo-label tensor augmentation dataset is obtained. S3: Clean the test text, explicitly register mathematical formulas and geometric symbols in LaTeX format as special tags to the word segmenter of the pre-trained encoder language model and expand the embedding layer, perform domain-adaptive incremental pre-training of the masked language model on the educational corpus containing the pseudo-label tensor augmentation dataset, and obtain a customized base model. S4: Using the customized base model as the semantic representation base, construct a feature dimension folding architecture as a multi-task sharing and classification layer, calculate the total loss of the multi-task by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network by using the full training set composed of the first dataset and the pseudo-label tensor augmentation dataset. S5: Input the test text into the fine-tuned joint rating network, output the classification labels of each core competency dimension covered by the test text and their corresponding probabilities, and perform automatic discovery and active learning feedback of low confidence samples based on confidence assessment; The specific process of constructing the feature dimension folding architecture in S4 includes: Obtain the global pooled semantic feature vector of the test questions output by the customized base model. ; The semantic feature vector is processed through a single-branch fully connected network layer. The linear mapping is represented by a one-dimensional flattening tensor, specifically as follows: ； The one-dimensional flattened tensor is physically folded into a two-dimensional joint logarithmic probability tensor using a tensor reshaping operation, specifically represented as follows: ； in, This is the weight matrix of the single-branch fully connected network layer. For bias parameters, The total number of core competency dimensions in the three-dimensional education tensor assessment space. The number of levels for core competencies in cognitive level; The specific process of calculating the total multi-task loss using the cost-sensitive weighting mechanism in step S4 includes: A distribution scan of the label slots in the full training set was performed to count the actual frequency of occurrence of each core competency cognitive level. ; Based on the statistical results, using the formula Dynamically calculate the cost-sensitive penalty weights for different cognitive levels of core competencies. ,in This represents the total number of evaluation nodes in the entire training set. The calculated penalty weights for each level The loss function is transformed into a tensor and injected into the multi-task cross-entropy loss function. A high penalty gradient is assigned to the high-order level of the long-tail core competencies, which accounts for a very small proportion, to calculate the total loss of the multi-tasks. During backpropagation, the parameters of the customized base model are frozen, and only the parameters of the feature dimension folding architecture are updated.

2. The method according to claim 1, characterized in that, Step S1 includes: A three-dimensional educational tensor evaluation space is constructed, encompassing test items, subject literacy, and literacy awareness levels. The first dimension represents the test item dimension, indicating specific test item entities within the question bank. The second dimension represents the subject literacy dimension, defining nine core competencies including abstract thinking, computational ability, geometric intuition, spatial reasoning, reasoning ability, data literacy, modeling ability, application awareness, and innovative awareness. The third dimension represents the literacy awareness level dimension, defining four continuous quantitative levels for each subject literacy dimension: level zero, representing no involvement of that competency, and levels one, two, and three, representing progressively increasing competencies. The literacy awareness level refers to the assessment level of a test item within a specific subject literacy dimension, categorized based on cognitive depth, knowledge integration, transfer and application, and innovation requirements, serving as a fine-grained, hierarchical evaluation dimension representing core competencies. Obtain the first dataset anchored by experts through tensor instantiation based on the three-dimensional education tensor evaluation space; A second dataset was constructed by collecting a large number of unlabeled test questions covering at least four different textbook versions.

3. The method according to claim 2, characterized in that, The specific process of obtaining the first dataset includes: Two annotators independently and double-blindly instantiated and anchored a small number of sampled test items based on the three-dimensional education tensor evaluation space. The anchoring results were compared, and test items with discrepancies were reviewed and arbitrated to obtain the first dataset with higher education research logic consistency.

4. The method according to claim 1, characterized in that, Step S2 includes: A predetermined number of test questions are randomly selected from the first dataset, and their test texts and corresponding tensor instantiation anchor labels are extracted to construct a small sample example set. At the same time, a diagnostic thinking chain system prompt word is constructed. The system prompt word forces the large language model to follow the three-stage inference logic of analysis process, dimension matching and result assembly. The large language model application programming interface is invoked, and the unlabeled test questions in the second dataset and the few sample example set are input together. The large language model then performs the following three-stage deduction process in sequence: internally reviewing the knowledge points tested in the unlabeled test questions; deriving the logical complexity based on the key steps of problem-solving, and anchoring the core competency dimensions and their corresponding competency cognitive level levels in the three-dimensional education tensor evaluation space; and outputting labeled data that conforms to the preset JSON structure. Data filtering and format conversion are performed on the JSON structured label data output by the large language model, including parsing the JSON data, identifying and deleting invalid samples with missing target literacy label fields, empty dictionaries, or empty lists; the retained valid label data is mapped to numerical pseudo-label vectors corresponding to the three-dimensional education tensor evaluation space to obtain the pseudo-label tensor augmentation dataset.

5. The method according to claim 1, characterized in that, Step S3 includes: The test text is cleaned to remove irrelevant hypertext markup language tags and noise characters; a special character set containing mathematical formulas and geometric symbols in LaTeX format is loaded from a preset mathematical symbol library, and the special character set is dynamically registered as a special tag into the base word segmenter of the pre-trained encoder language model; The word embedding layer of the pre-trained encoder language model is expanded based on the registration results of the special tags; The test texts in the pseudo-labeled tensor augmentation dataset are mixed with the remaining massive amount of unlabeled test texts in the second dataset to construct an educational corpus for mathematics. A dynamic masking strategy is used to randomly mask the text sequences in the educational corpus with a preset probability, and unsupervised incremental pre-training of the masked language model is performed with the prediction of the real words at the masked positions as the optimization objective to obtain the customized base model.

6. The method according to claim 5, characterized in that, The step of expanding the word embedding layer of the pre-trained encoder language model based on the registration results of the special tags includes: Extracting high-frequency formulas and geometric characters from junior high school mathematics to construct a special tag word list; The special tag vocabulary is dynamically registered to the base word segmenter of the pre-trained encoder language model, and the word embedding layer is expanded using a formula, which is: ； in For the original embedded layer, Based on the vocabulary list, For the newly added special marker vocabulary; A dynamic masking strategy is adopted to randomly mask educational scenario corpora containing test questions, and the log-likelihood loss is calculated with the real words at the predicted mask positions as the optimization objective. Unsupervised incremental pre-training of the masked language model is then performed.

7. The method according to claim 1, characterized in that, The specific process of automatically discovering and actively learning backflow of low-confidence samples based on confidence assessment in S5 includes: Perform a Softmax activation operation on the joint logarithmic probability tensor output of the joint rating network forward propagation, and extract the highest predicted probability of the test item in each core competency dimension as the prediction confidence of that dimension. Perform a maximum value index operation on the category dimension to obtain each predicted core competency covered by the test questions and its corresponding core competency cognitive level label; Determine whether the predicted core competency cognitive level label is a non-zero level. If so, and the corresponding prediction confidence is lower than the preset safety threshold, then determine that the current test question is a low-confidence fuzzy sample. The system automatically intercepts the low-confidence fuzzy samples, extracts their original test question text and current prediction output, and appends them to the local low-confidence database for expert review or as targeted incremental data for the next round of model iteration.

8. A test item literacy cognitive level intelligent annotation system based on multidimensional tensor joint optimization, characterized in that, The system includes: The data processing and tensor construction module is used to construct a three-dimensional educational tensor evaluation space involving test questions, subject literacy, and literacy cognitive level, and to collect real question banks to form a first dataset containing a small number of manually labeled test questions and a second dataset containing a large number of unlabeled test questions. The diagnostic thinking chain pseudo-label enhancement module is used to reason and label the unlabeled test questions in the second dataset based on the first dataset as a reference example, using a large language model combined with the diagnostic thinking chain strategy. After data filtering and format conversion, the pseudo-label tensor augmentation dataset is obtained. The mathematical prior and incremental pre-training module is used to clean and preprocess the test text, explicitly inject mathematical symbol priors, and use the pre-trained encoder language model to perform domain-adaptive incremental pre-training of the masked language model on the educational scenario corpus containing the pseudo-label tensor augmentation dataset to obtain a customized base model. The dimension folding and cost-sensitive training module is used to construct a feature dimension folding architecture as a multi-task sharing and classification layer using the customized base model as a semantic representation base, calculate the total loss of multi-tasks by combining the cost-sensitive weighting mechanism, and complete the fine-tuning training of the joint rating network using the full training set composed of the first dataset and the pseudo-label tensor augmentation dataset. The specific process of constructing the feature dimension folding architecture includes: Obtain the global pooled semantic feature vector of the test questions output by the customized base model. ; The semantic feature vector is processed through a single-branch fully connected network layer. The linear mapping is represented by a one-dimensional flattening tensor, specifically as follows: ； The one-dimensional flattened tensor is physically folded into a two-dimensional joint logarithmic probability tensor using a tensor reshaping operation, specifically represented as follows: ； in, This is the weight matrix of the single-branch fully connected network layer. For bias parameters, The total number of core competency dimensions in the three-dimensional education tensor assessment space. The number of levels for core competencies in cognitive level; The specific process of calculating the total loss of multiple tasks using a cost-sensitive weighting mechanism includes: A distribution scan of the label slots in the full training set was performed to count the actual frequency of occurrence of each core competency cognitive level. ; Based on the statistical results, using the formula Dynamically calculate the cost-sensitive penalty weights for different cognitive levels of core competencies. ,in This represents the total number of evaluation nodes in the entire training set. The calculated penalty weights for each level The loss function is converted into a tensor and injected into the multi-task cross-entropy loss function. A high penalty gradient is assigned to the high-order level of the long-tail core competencies, which accounts for a very small proportion, so as to calculate the total loss of the multi-task. The parameters of the customized base model are frozen during backpropagation, and only the parameters of the feature dimension folding architecture are updated. The active learning and low-confidence feedback module is used to input the questions to be tested into the fine-tuned joint rating network, output the classification labels and probabilities of each core competency, and perform automatic discovery and active learning feedback of low-confidence samples based on confidence assessment.

Images