Student personalized skill formation evaluation method and system based on text mining

Abstract

The present application provides a kind of student individualized skill formation evaluation method and system based on text mining, the method includes step 1: obtaining the original text of formative evaluation homework, and it is preprocessed, obtain the homework text after preprocessing;Step 2: the homework text after preprocessing is classified, and the homework category to which homework text belongs is obtained;Step 3: for the homework category to which homework text belongs, analysis engine follows corresponding analysis rule set to analyze homework text;Step 4: output evaluation report.The present application can formative evaluation on student's experimental report, project homework and other formative evaluation homework such as this kind of unstructured, involves professional field knowledge, comprehensive and engineering text homework, to identify the professional skill defects of student in technical thinking, problem solving and professional expression ability and other aspects embodied in formative evaluation homework.

Description

Technical Field

[0001] This invention relates to the field of educational technology, specifically to a method and system for personalized formative assessment of students' skills based on text mining. Background Technology

[0002] With the acceleration of industrial upgrading and technological iteration, the modern vocational education system has placed higher demands on the cultivation of highly skilled technical personnel. Formative assessment outcomes, such as students' lab reports and project assignments, are key vehicles for measuring their vocational skills (including their technical thinking, problem-solving, and professional communication abilities). However, in the current teaching practices of many vocational schools, there are significant shortcomings in the evaluation of students' lab reports and project assignments (hereinafter collectively referred to as formative assessment assignments), which hinders the precise improvement of the quality of talent cultivation.

[0003] A major reason for the significant shortcomings in the evaluation process of these formative assessment assignments is that a teacher typically needs to evaluate dozens or even hundreds of assignments. Therefore, it is difficult to have sufficient time for in-depth and detailed analysis of each assignment, resulting in often superficial and vague evaluations. For example, comments on student work may be simplified to highly generalized phrases such as "unclear logic," "insufficient content," or "incorrect format." Alternatively, the evaluation may only reflect a score or grade.

[0004] This broad and vague assessment of students' formative assignments triggered a series of negative effects, including:

[0005] For students, the evaluation results do not provide them with accurate information about the types and specific locations of their vocational skills deficiencies as reflected in their formative assessment assignments. They also cannot obtain clear and actionable improvement paths, making it difficult for them to identify their specific deficiencies in knowledge application, process thinking, or critical reflection. Ultimately, they fall into a predicament of repeated trial and error and slow growth.

[0006] For teachers, this not only reduces their sense of efficacy and value in providing teaching feedback, but also makes it difficult for them to accurately grasp the overall weaknesses of the class, thus making it difficult to make targeted adjustments to their teaching design.

[0007] For teaching administrators, broad and vague evaluation data makes it difficult to assess and continuously improve teaching quality with solid and detailed data support.

[0008] Currently, research in the field of educational data mining and learning analytics is very active, and various related methods and platforms have been put into use. These methods and platforms each have their own characteristics, such as:

[0009] 1. Online learning platforms such as Coursera, edX, and Xueyin Online: They use data to analyze students' video completion rates, quiz scores, forum participation, etc., to predict academic risks and push resources. However, they rely mainly on structured data such as clickstreams and scores, which is not suitable for unstructured text assignments such as formative assessment assignments such as lab reports and project assignments.

[0010] 2. ETS (the organization that creates the TOEFL and GRE questions) e-rater system: It can automatically score essays through assessments of grammar, vocabulary, article structure, logic, etc., but it mainly assesses general writing skills and is not suitable for formative assessment assignments such as lab reports and project assignments that involve professional domain knowledge.

[0011] 3. AI-powered coding assignment grading tools: These tools are widely used in computer science education. They can automatically check the correctness, efficiency, and style of code, and even perform code similarity detection to prevent cheating. However, these tools are only suitable for evaluating software programs and are not suitable for text-based assignments such as lab reports and project assignments.

[0012] 4. Natural Language Processing (NLP) technology: Some studies have attempted to use NLP to analyze students' discussion forum posts and learning logs to assess students' engagement, emotional state, or concept mastery. However, these studies are usually fragmented and focus on a specific type of text or a particular aspect of ability. They are not well-suited for comprehensive and engineered assignments such as lab reports and project assignments. Summary of the Invention

[0013] To address at least one of the above technical problems, this invention provides a text mining-based method and system for formative assessment of students' personalized skills. This method enables formative assessment of unstructured, comprehensive, and engineering-oriented textual assignments such as students' lab reports and project assignments, which involve professional domain knowledge. The aim is to identify students' deficiencies in professional skills, such as technical thinking, problem-solving, and professional expression.

[0014] A first aspect of the present invention provides a method for personalized formative assessment of student skills based on text mining, comprising:

[0015] Step 1: Obtain the original text of the formative assessment assignment and preprocess it to obtain the preprocessed assignment text;

[0016] Step 2: Classify the preprocessed task texts to obtain the task category to which the task text belongs;

[0017] Step 3: Based on the job category to which the job text belongs, trigger the analysis engine to analyze the job text according to the corresponding set of analysis rules;

[0018] Step 4: Output the evaluation report.

[0019] In any of the above schemes, the preprocessing of the original text in step 1 includes: data desensitization and cleaning, and text standardization.

[0020] In any of the above-mentioned schemes, the data desensitization and cleaning includes: identifying data involving personal privacy information (such as student names, student IDs, contact information, etc.) in the original text and replacing it with a unified anonymous identifier; detecting and correcting content in the original text that affects subsequent text processing (such as content that is unrecognizable), and correcting it to recognizable content.

[0021] In any of the above-mentioned schemes, the text standardization includes: converting non-text format documents into UTF-8 encoded structured plaintext through optical character recognition (OCR) and document structure parsing, stripping all layout rendering instructions, and retaining only the text content and its logical structure to complete the data standardization before analysis; for professional terms with multiple expressions for the same concept, normalizing all expressions into standard terms based on a thesaurus established by the domain knowledge base.

[0022] In any of the above schemes, the preferred method is that in step 2, the preprocessed job text is classified by a text classifier to obtain the job category to which the job text belongs.

[0023] In any of the above schemes, the assignment category in step 2 is one of the following: experimental report, design scheme, and case analysis.

[0024] In any of the above schemes, preferably, in step 3, the analysis engine performs the following on the categorized job text:

[0025] Step 31: Determine the corresponding predefined structured form based on the job category;

[0026] Step 32: For each information slot in the predefined structured form, extract information, that is, identify and extract relevant text information fragments from the preprocessed job text; and establish a mapping relationship between the information slots and the text information fragments to obtain a structured data object;

[0027] Step 33: Based on the type of information slot in the structured data object, input the text information fragments corresponding to the information slots into the corresponding defect recognition model to obtain a list of defects in the job text;

[0028] Step 34: For each defect in the defect list, perform a severity assessment.

[0029] Preferably, for any of the above schemes, the information slots in the predefined structured form for assignment texts of the type of experiment report include: problem phenomenon, analysis process, tools used, solution, and experiment conclusion; for assignment texts of the type of design scheme, the information slots in the predefined structured form include: project requirements, technical solution, feasibility analysis, and innovation points; for assignment texts of the type of case analysis, the information slots in the predefined structured form include: case background, key issue identification, analysis process, decision and arguments.

[0030] In any of the above schemes, the preferred method is that, in step 32, for each information slot in the structured form, firstly, based on the characteristics of the target information slot, a sequence labeling or reading comprehension / question-answering information extraction model is determined for that information slot. Then, for that information slot, information is extracted using the determined information extraction model until the information extraction for each information slot is completed.

[0031] The preferred approach for any of the above schemes is to use a reading comprehension / question-answering information extraction model for information slots with relatively stable structures and concentrated answers, and a sequence labeling information extraction model for information slots with loose structures and discrete answers.

[0032] The preferred approach for any of the above schemes is to use the BIO annotation system for sequence labeling, which labels each word in the preprocessed job text.

[0033] In any of the above schemes, the preferred approach is to use a fine-tuned pre-trained language model BERT for reading comprehension / question answering, treating each information slot in a predefined structured form as a question, and finding the text information fragment that serves as the answer from the pre-processed assignment text.

[0034] In any of the above schemes, preferably, in step 33, based on the structured data object, a defect identification process is initiated. This defect identification process uses information slots within the structured data object as the basic processing unit, and specifically includes:

[0035] Step 331: Based on the predefined routing configuration table, route the text information fragment corresponding to each information slot in the structured data object to at least one of the semantic defect scanning model, process specification inspection model, and hard specification audit model;

[0036] Step 332: The semantic defect scanning model, process specification inspection model, and hard specification audit model identify defects in the received text information fragments, obtain the defects existing in the input text information fragments, and establish the association relationship between the defect type and the information slots of the defect source, the defect description, and the text fragments corresponding to the defect.

[0037] Step 333: The result fusion module performs multi-source evidence fusion on the mapping relationship established by the semantic defect scanning model, process specification inspection model, and hard specification audit model to generate a defect list.

[0038] In any of the above schemes, in step 331, the routing configuration table defines the mapping relationship between information slots of different semantic types and semantic defect scanning models, process specification inspection models, and hard specification audit models. Accordingly, the text information fragments corresponding to each information slot will be routed to at least one of the above models to identify the defects they have.

[0039] Preferably, for any of the above schemes, a mapping relationship is established between the information slots whose function is to present reasoning, analysis, and summarization and the semantic defect scanning model; a mapping relationship is established between the information slots that describe operation sequences, methods, schemes, and implementation paths and the process specification inspection model; and a mapping relationship is established between the information slots that have high requirements for objectivity and accuracy, or all information slots, and the hard specification audit model.

[0040] In any of the above schemes, the defect identification process of the semantic defect scanning model in step 332 includes:

[0041] Step 332A1: The BERT encoder converts the text information fragments corresponding to the information slots into context-dependent word vector sequences;

[0042] Step 332A2: For the word vector sequence, the CRF layer acts as a decoder to learn the transition constraints between labels and outputs the globally optimal label sequence;

[0043] Step 332A3: Scan the global optimal label sequence to find all segments with non-O labels, and establish a mapping relationship between the defect type, defect description, corresponding information slot, and corresponding text segment for segments with non-O labels.

[0044] In any of the above schemes, the defect identification process of the process specification inspection model in step 332 includes:

[0045] Step 332B1: Retrieve the "best practice" or "standard procedure" text corresponding to the input text information fragment from the domain standard knowledge base;

[0046] Step 332B2: Using the Sentence-BERT model, encode the input text information fragments and their corresponding "best practices" or "standard procedures" texts into sentence vectors respectively;

[0047] Step 332B3: Calculate the cosine similarity between the corresponding sentence vectors;

[0048] Step 332B4: Determine whether there is a defect based on whether the cosine similarity value is lower than the set threshold, and if there is a defect, establish a mapping relationship between defect type, defect description (including student description fragment, standard description fragment, and similarity score), and corresponding information slot.

[0049] In any of the above schemes, the defect identification process of the rigid standard audit model in step 332 includes:

[0050] Step 332C1: Based on a predefined rule set, scan the input text information fragments. If the rule set requirements are not met, trigger a defect.

[0051] Step 332C2: Based on the triggered defect, establish a mapping relationship between defect type, violated rule, and corresponding text fragment.

[0052] In any of the above schemes, preferably, in step 333, the result fusion module performs multi-source evidence fusion on the mapping relationship established by the semantic defect scanning model, process specification inspection model, and hard specification audit model based on a fusion algorithm of heuristic rules and confidence comparison, and generates a defect list with a uniform format, including:

[0053] Step 3331: Perform evidence aggregation, that is, merge different defective evidence pointing to the same text location;

[0054] Step 3332: Conflict resolution is performed. When the hard specification audit model determines that a defect exists but the semantic defect scanning model does not find a defect, the determination result of the hard specification audit model shall prevail (i.e., the principle of prioritizing certainty is followed). When the defect scanning results of the semantic defect scanning model and the process specification inspection model contradict each other, the scanning result of the semantic defect scanning model shall be given higher weight (i.e., the principle of prioritizing deep understanding is followed).

[0055] Step 3333: Generate a standardized defect list. Each record in the list includes: defect type, defect description, defect source information slot, corresponding text fragment, and initial severity score.

[0056] In any of the above schemes, in step 34, for the established defect list, each defect in the defect list is scored using an ordered classification model or a regression model, and the severity score of each defect in the defect list is updated using the score of each defect.

[0057] In any of the above schemes, the preferred option is that, in step 4, an evaluation report is output based on the results of step 3.

[0058] Preferably, in any of the above schemes, the evaluation report includes scores for various defects and a detailed list of defects.

[0059] Preferably, in any of the above embodiments, the evaluation report also includes personalized learning suggestions.

[0060] A second aspect of the present invention provides a text mining-based formative assessment system for personalized student skills, including a processor for running a computer program to perform the text mining-based formative assessment method for personalized student skills.

[0061] The text mining-based formative assessment method and system for personalized student skills of the present invention has the following beneficial effects:

[0062] 1. It can conduct formative assessments on students' unstructured, comprehensive, and engineering-oriented text-based assignments, such as lab reports and project assignments, to identify students' deficiencies in professional skills, such as technical thinking, problem-solving, and professional expression.

[0063] 2. Students can intuitively understand their skill deficiencies and their specific locations based on the evaluation report, thereby obtaining clear and actionable improvement paths;

[0064] 3. Teachers can accurately grasp the weaknesses in the class as a whole, making it difficult to make targeted adjustments to the teaching design; at the same time, school administrators can obtain more solid and detailed data support for teaching quality assessment.

[0065] 4. Considering the differences between different formative assessment tasks, the tasks are first classified, then different information extraction contents are triggered according to the classification results, and finally, defects are identified through different defect identification models according to the differences in the extracted contents, thus realizing intelligent routing and scheduling; at the same time, the evaluation results are more targeted and personalized. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the overall process of a preferred embodiment of the text mining-based formative assessment method for personalized student skills according to the present invention.

[0067] Figure 2 For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 The flowchart of step 3 in the embodiment shown is illustrated.

[0068] Figure 3 For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 A flowchart illustrating step 33 of step 3 in the illustrated embodiment.

[0069] Figure 4For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 The illustrated embodiment is a schematic diagram of the semantic defect scanning model process.

[0070] Figure 5 For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 The flowchart of the process specification inspection model in the embodiment shown is illustrated.

[0071] Figure 6 For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 The illustrated embodiment is a schematic diagram of the rigid standard audit model process.

[0072] Figure 7 For example, the text mining-based formative assessment method for personalized student skills according to the present invention... Figure 1 The flowchart of the result fusion module in the embodiment shown is illustrated.

[0073] Figure 8 This is a schematic diagram of the Pipeline architecture of a preferred embodiment of the text mining-based formative assessment method for personalized student skills according to the present invention. Detailed Implementation

[0074] To better understand the present invention, the present invention will be described in detail below with reference to specific embodiments.

[0075] Example 1

[0076] like Figure 1 As shown, a text mining-based method for formative assessment of students' personalized skills includes:

[0077] Step 1: Obtain the original text of the formative assessment assignment and preprocess it to obtain the preprocessed assignment text;

[0078] Step 2: Classify the preprocessed task texts to obtain the task category to which the task text belongs;

[0079] Step 3: Based on the job category to which the job text belongs, trigger the analysis engine to analyze the job text according to the corresponding set of analysis rules;

[0080] Step 4: Output the evaluation report.

[0081] Step 1, the preprocessing of the original text includes: data anonymization and cleaning, and text standardization. Specifically, data anonymization and cleaning includes: identifying data in the original text containing personal privacy information (such as student names, student IDs, contact information, etc.) and replacing it with a unified anonymous identifier; detecting and correcting content in the original text that affects subsequent text processing (such as unrecognizable content) (such as spelling errors, grammatical errors, non-standard symbols, unrecognizable fonts, etc.), and correcting it to recognizable content to ensure the smooth execution of subsequent text processing. Specific methods can refer to existing technologies (such as replacing with placeholders, prompting manual review and correction, etc.), which are not detailed in this application. However, it should be understood that in order to accurately identify skill deficiencies in subsequent processing, over-processing (such as automatic context-based correction and completion) should be avoided as much as possible during this preprocessing process to prevent masking skill deficiencies in the original text. The text standardization includes: converting non-text format documents into UTF-8 encoded structured plaintext through optical character recognition (OCR) and document structure parsing, stripping all layout rendering instructions, retaining only the text content and its logical structure, and completing the data standardization before analysis; for professional terms with multiple expressions for the same concept, based on the synonym mapping table established by the domain knowledge base, normalizing all expressions into standard terms, such as normalizing "motor", "electric motor", "electric motor" etc. into "electric motor"; it should be noted that by normalizing into standard terms, the following effects can be achieved: (1) purifying and condensing the feature space, specifically manifested as: greatly reducing the feature dimensions that subsequent text processing models need to learn, making the features denser and more representative, allowing subsequent text processing models to focus more on learning the deep relationships between concepts, rather than the surface word changes, from This improves the generalization ability and training efficiency of the model; (2) Ensures the consistency of downstream tasks, specifically: ① For the rule engine, it can be safely programmed to check "electric motor" rather than "motor" or "electric motor" or "electric motor"; ② For the subsequent semantic matching model, when performing step comparison, the normalized standard terms in the job text are compared with the normalized standard terms in the standard library to calculate the similarity, and the result is more accurate; ③ For the subsequent defect statistics, the number of defects involving the same term will be summarized and counted, rather than scattered under the entries of multiple expressions of a certain standard term, and the statistical results are more intuitive and accurate; (3) Realizes "understanding" rather than "matching", that is, it enables computer processing to move from string processing to semantic understanding, and enables the computer to understand that different expressions are talking about the same thing in the context.

[0082] After preprocessing, non-text formats such as images and PDFs, or even those containing images, are converted into text documents that can be recognized and processed, thus ensuring the smooth progress of subsequent processing.

[0083] In step 2, the preprocessed assignment texts are classified using a text classifier to determine their respective assignment categories. The assignment category can be one of the following: experimental report, design scheme, or case analysis.

[0084] In this application, a lightweight pre-trained language model based on the Transformer architecture is preferably used as the text classifier. Through supervised domain-adaptive fine-tuning on a small, high-quality manually labeled dataset, this classifier can classify pre-processed assignment text into one of three predefined categories—"experiment report," "design plan," or "case analysis"—with high confidence, based on the semantic content and structural features of the full assignment text, rather than solely relying on the title or keywords. This text classifier achieves high accuracy and robustness while maintaining model efficiency.

[0085] The job category to which the assignment text belongs provides the foundation for intelligent routing and scheduling in subsequent data processing. Different types of assignments have significantly different text structures and contained information, thus the focus of defect analysis also differs greatly. To conduct targeted defect analysis, the analysis engine is triggered to perform defect analysis on different categories of assignments according to different analysis rule sets, based on the assignment category to which the assignment belongs. Specifically: in step 3, when the assignment text belongs to an experiment report, the analysis engine follows the experiment report analysis rule set; when the assignment text belongs to a design scheme, the analysis engine follows the design scheme analysis rule set; and when the assignment text belongs to a case study, the analysis engine follows the case study analysis rule set. Different analysis rule sets differ in their core analysis objectives, information extraction focus, defect identification emphasis, and evaluation criteria, as shown in Table 1.

[0086] Table 1. Examples of comparisons between different analysis rule sets

[0087]

[0088] More specifically, such as Figure 2 As shown, in step 3, the analysis engine performs the following on the categorized job text:

[0089] Step 31: Determine the corresponding predefined structured form based on the job category;

[0090] Step 32: For each information slot in the predefined structured form, extract information, that is, identify and extract relevant text information fragments from the preprocessed job text; and establish a mapping relationship between the information slots and the text information fragments to obtain a structured data object;

[0091] Step 33: Based on the type of information slot in the structured data object, input the text information fragments corresponding to the information slots into the corresponding defect recognition model to obtain a list of defects in the job text;

[0092] Step 34: For each defect in the defect list, perform a severity assessment.

[0093] It should be noted that the predefined structured forms include several information slots, and the types of information slots in the corresponding predefined structured forms differ for different assignment categories. As shown in Table 1, for assignment texts of the experiment report type, the information slots in the corresponding predefined structured forms include: problem phenomenon, analysis process, tools used, solution, and experiment conclusion; for assignment texts of the design scheme type, the information slots in the corresponding predefined structured forms include: project requirements, technical solution, feasibility analysis, and innovative points; for assignment texts of the case analysis type, the information slots in the corresponding predefined structured forms include: case background, key issue identification, analysis process, and decision and arguments.

[0094] After determining the structured form that matches the assignment category, information extraction is performed for each information slot in the structured form. In step 32, a semantic information extraction method based on deep learning in natural language processing is used. Specifically, for each information slot in the structured form, firstly, based on the characteristics of the target information slot, the appropriate information extraction model, either sequence labeling or reading comprehension / question-answering, is determined for that information slot. Then, information extraction is performed for that information slot using the determined information extraction model until information extraction for each information slot is completed. More specifically, for information slots with relatively stable structures and concentrated answers, such as experimental conclusion information slots, a reading comprehension / question-answering information extraction model is used; for information slots with loose structures and discrete answers, such as analysis process and tool usage information slots, a sequence labeling information extraction model is used. Sequence labeling uses the BIO labeling system to label each word in the preprocessed assignment text. Reading comprehension / question-answering uses a finely tuned pre-trained language model BERT, treating each information slot in the predefined structured form as a question and finding the text information fragments that serve as the answer from the preprocessed assignment text. It should be understood that the information extraction method in step 32 can also be adjusted, such as by integrating sequence labeling and reading comprehension / question answering information extraction models to construct a hybrid information extraction model for information extraction, so as to improve the robustness and accuracy of information extraction.

[0095] For example, consider the following preprocessed job text:

[0096] "When completing the experiment of the sound and light alarm system, I found that the buzzer was not sounding the alarm as expected. I suspected that the drive current was insufficient, so I used a digital multimeter to measure the voltage of the transistor pin connected to the buzzer and found that the voltage was only 1.5V, which was lower than the 3.3V specified in the datasheet. Therefore, I modified the program and increased the duty cycle of the PWM output. Finally, the buzzer sounded successfully, and the problem was solved. This experiment made me deeply understand the coordination between drive circuit design and software control."

[0097] The text classifier categorized it as an experiment report. The information slots in the corresponding structured form for this type of assignment are: problem phenomenon, analysis process, tools used, solution, and experiment conclusion. For these information slots, sequence labeling and reading comprehension / question-answering techniques were used for information extraction, resulting in the structured data objects shown in Table 2.

[0098] Table 2 Examples of Structured Data Objects

[0099]

[0100] After step 32, a long assignment text is extracted into a structured data object, providing a focused "data field" for defect analysis in the subsequent step 33.

[0101] In step 33, based on the structured data object, a defect identification process is initiated. This process uses information slots within the structured data object as the basic processing unit, such as... Figure 3 As shown, it specifically includes:

[0102] Step 331: Based on the predefined routing configuration table, route the text information fragment corresponding to each information slot in the structured data object to at least one of the semantic defect scanning model, process specification inspection model, and hard specification audit model;

[0103] Step 332: The semantic defect scanning model, process specification inspection model, and hard specification audit model identify defects in the received text information fragments, obtain the defects existing in the input text information fragments, and establish the association relationship between the defect type and the information slots of the defect source, the defect description, and the text fragments corresponding to the defect.

[0104] Step 333: The result fusion module performs multi-source evidence fusion on the mapping relationship established by the semantic defect scanning model, process specification inspection model, and hard specification audit model to generate a defect list.

[0105] In step 331, the routing configuration table defines the mapping relationship between information slots of different semantic types and semantic defect scanning models, process specification inspection models, and hard specification audit models. Accordingly, the text information fragments corresponding to each information slot will be routed to at least one of the above models to identify their defects. Specifically: (1) For information slots whose function is to present reasoning, analysis, and summary, such as the analysis process information slot and experiment conclusion information slot in the experimental report, and the analysis process information slot and decision and argument information slot in the case analysis, a mapping relationship will be established with the semantic defect scanning model. Therefore, the text information fragments corresponding to this type of information slot will be input into the semantic defect scanning model to identify defects at the semantic level such as concepts and logic; (2) For information slots that describe operation sequences, methods, schemes, and implementation paths, such as the tool information slot and solution information slot in the experimental report, and the technical solution information slot in the design scheme, a mapping relationship will be established with the process specification inspection model. Therefore, the text information fragments corresponding to this type of information slot will be input into the semantic defect scanning model to identify defects at the semantic level such as concepts and logic; Information fragments will be input into the process specification inspection model to compare with the standard process and identify specification defects; (3) For information slots with high requirements for objectivity and accuracy (such as problem phenomenon information slots and tool usage information slots in the experimental report), or, all information slots will establish a mapping relationship with the hard specification audit model. Therefore, the text information fragments corresponding to such information slots will be input into the hard specification audit model to screen for defects in the corresponding text information fragments that violate explicit rules (such as the description of the corresponding text information fragments of the problem phenomenon information slots in the experimental report is not objective and accurate, and the tool name and model recorded in the corresponding text of the tool usage information slots in the experimental report are not complete and standardized, etc.). In step 331, the driving method based on the routing configuration table makes it possible to adjust the routing strategy in step 33 without modifying the core code. Only the routing configuration table needs to be adjusted adaptively, which greatly enhances the scalability and maintainability of the method. Table 3 shows an example of a core component of the routing configuration table.

[0106] Table 3. Examples of the core components of a routing configuration table

[0107]

[0108] In step 332, the semantic defect scanning model, the process specification inspection model, and the hard specification audit model perform multi-dimensional defect scanning on the received text information fragments in parallel, as detailed below.

[0109] The semantic defect scanning model employs a BERT+Conditional Random Field (CRF) sequence labeling model to perform deep understanding and fine-grained labeling of received text information fragments. This aims to identify and locate defects in knowledge concepts (such as labeling incorrect or confusing core terms), logical structure (such as identifying missing logical connectors and broken causal chains), and communication expression (such as labeling serious grammatical errors or extremely vague expressions). Figure 4 As shown, the process of defect identification by the semantic defect scanning model includes:

[0110] Step 332A1: The BERT encoder converts the text information fragments corresponding to the information slots into context-dependent word vector sequences;

[0111] Step 332A2: For the word vector sequence, the CRF layer acts as a decoder to learn the transition constraints between labels and outputs the globally optimal label sequence;

[0112] Step 332A3: Scan the global optimal label sequence to find all segments with non-O labels, and establish a mapping relationship between the defect type, defect description, corresponding information slot, and corresponding text segment for segments with non-O labels.

[0113] The process specification inspection model employs the Sentence-BERT model to perform a similarity-based specification comparison on received text information fragments. This identifies and locates process specification defects (such as missing steps or incorrect sequence), tool and method defects (such as inappropriate tool selection or incorrect measurement methods), and problem-solving defects (such as significant differences from validated successful solutions, rendering the solution infeasible). Figure 5 As shown, the process of defect identification by the process specification inspection model includes:

[0114] Step 332B1: Retrieve the "best practice" or "standard procedure" text corresponding to the input text information fragment from the domain standard knowledge base;

[0115] Step 332B2: Using the Sentence-BERT model, encode the input text information fragments and their corresponding "best practices" or "standard procedures" texts into sentence vectors respectively;

[0116] Step 332B3: Calculate the cosine similarity between the corresponding sentence vectors;

[0117] Step 332B4: Determine whether there is a defect based on whether the cosine similarity value is lower than the set threshold, and if there is a defect, establish a mapping relationship between defect type, defect description (including student description fragment, standard description fragment, and similarity score), and corresponding information slot.

[0118] The rigid compliance audit model, based on a rule engine that matches regular expressions and keyword patterns, performs deterministic rule verification on received text information fragments to identify and locate rigorous and detailed defects (such as missing data or omitted units) and process compliance defects (such as omitted security regulations). Figure 6 As shown, the process of defect identification in the rigid standard audit model includes:

[0119] Step 332C1: Based on a predefined rule set, scan the input text information fragments. If the rule set requirements are not met, trigger a defect.

[0120] Step 332C2: Based on the triggered defect, establish a mapping relationship between defect type, violated rule, and corresponding text fragment.

[0121] like Figure 7 As shown, in step 333, the result fusion module, based on a fusion algorithm combining heuristic rules and confidence comparison, performs multi-source evidence fusion on the mapping relationships established by the semantic defect scanning model, process specification inspection model, and hard specification audit model, generating a defect list with a unified format, including:

[0122] Step 3331: Perform evidence aggregation, that is, merge different defective evidence pointing to the same text location;

[0123] Step 3332: Conflict resolution is performed. When the hard specification audit model determines that a defect exists but the semantic defect scanning model does not find a defect, the determination result of the hard specification audit model shall prevail (i.e., the principle of prioritizing certainty is followed). When the defect scanning results of the semantic defect scanning model and the process specification inspection model contradict each other, the scanning result of the semantic defect scanning model shall be given higher weight (i.e., the principle of prioritizing deep understanding is followed).

[0124] Step 3333: Generate a standardized defect list. Each record in the list includes: defect type, defect description, defect source information slot, corresponding text fragment, and initial severity score.

[0125] In step 34, for the established defect list, each defect in the defect list is scored using an ordered classification model or a regression model, and the severity score of each defect in the defect list is updated with the score of each defect.

[0126] In step 4, an evaluation report is output based on the results of step 3. The evaluation report includes scores for various defects and a detailed list of defects; more preferably, the evaluation report also includes personalized learning suggestions and a visualized skill profile.

[0127] Example 2

[0128] A text mining-based formative assessment system for personalized student skills includes a processor for running a computer program to perform the text mining-based formative assessment method for personalized student skills.

[0129] Example 3

[0130] This embodiment is similar to Embodiment 1, except that, preferably, the method further includes: Step 5: Performing statistical analysis on the multiple evaluation reports output in Step 4, such as counting the frequency of each type of defect and counting the information slots with the most defects, so that teachers can have a comprehensive understanding of students' overall skill mastery and common weaknesses in this assignment, facilitating timely adjustments to teaching focus and the design and implementation of remedial teaching plans. The results of the statistical analysis can be presented in a list format or in a visually intuitive way such as a heat map or radar chart; this application does not impose any limitations.

[0131] Example 4

[0132] This embodiment is similar to the previous embodiment, except that some of the models used in the text mining-based student personalized skills formative assessment method are described in this embodiment.

[0133] The method uses multiple models, including:

[0134] Text classifier: Used to classify the types of assignments students submit;

[0135] Sequence labeling: Used in the information extraction process to label the preprocessed job text so as to extract relevant text fragments for information slots;

[0136] The pre-trained language model BERT is used in the information extraction process to extract text information fragments that can answer information slot questions from the pre-processed task text.

[0137] Semantic defect scanning model: used in the defect identification process, it uses a combination of BERT and CRF to identify defects in the text information fragments corresponding to the information slots;

[0138] Process specification inspection model: used in the defect identification process, including the Sentence-BERT model, to identify defects in the text information fragments corresponding to the information slots;

[0139] Hard-core standard audit model: Used in the defect identification process, this rule engine, based on regular expressions and keyword pattern matching, identifies defects in the text information fragments corresponding to the information slots.

[0140] I. The above model needs to be trained before it can be used. The sample dataset used to train the above model in this application includes the following aspects.

[0141] 1. Student homework text

[0142] Student assignment texts refer to various forms of textual academic outputs produced by students during the learning process, including:

[0143] ① Structured assignment text

[0144] Such texts typically follow a fixed framework and specifications, with strict formatting and clearly defined content elements, facilitating standardized, fine-grained information extraction and comparison. They mainly include:

[0145] Experiment / training report: Includes standardized sections such as experiment objective, procedures, results and analysis;

[0146] Project report / design plan: covering project requirements, technical approach, implementation process, and summary evaluation;

[0147] Operation procedure description: A procedural description of specific operation steps;

[0148] Code comments and design documents: In software and embedded systems courses, logical explanations and design intent statements accompany the code.

[0149] Design drawing instructions: In engineering courses, this refers to the technical explanations and annotations of drawings.

[0150] ② Semi-structured and unstructured assignment texts

[0151] This type of text has a free format and focuses on continuous argumentation. Although it lacks a fixed framework, it better reflects students' comprehensive thinking process, reflective ability, and depth of knowledge integration. It mainly includes:

[0152] Case study reports: demonstrating students' analytical and decision-making abilities in real-world problems;

[0153] Learning summary and reflection: reflects students' learning trajectory, metacognitive strategies, and degree of knowledge internalization;

[0154] Weekly Journal / Learning Log: Continuously record the small gains, confusions, and insights during the learning process;

[0155] Classroom discussion records: These reflect students' immediate thinking activity, communication, collaboration, and critical thinking skills.

[0156] 2. Key context metadata

[0157] To ensure the accuracy and interpretability of the analysis results, key metadata associated with student assignment texts is collected as important context for the analysis model. This key contextual metadata mainly includes:

[0158] Course Name and Related Professional Field: Domain knowledge graph and skill standards used for anchoring analysis (e.g., mechanical manufacturing, electronic information, nursing, e-commerce, etc.).

[0159] Assignment requirements / task sheets: serve as a benchmark for evaluating students' completion of assignments and mastery of skills;

[0160] Scoring criteria and competency dimensions: providing rubric support and judgment basis for the model to identify "skill deficiencies";

[0161] By integrating multi-type, multi-granular text data and contextual information, a research dataset capable of supporting multi-level, multi-dimensional skill defect identification is constructed.

[0162] 3. Domain knowledge and standard library

[0163] To ensure the analytical model has accurate judgment criteria, domain knowledge and standard libraries are used as benchmark reference data for comparison and evaluation. Domain knowledge and standard libraries mainly include:

[0164] ① Domain Knowledge Graph and Skills Standards Base: Integrating industry standards, course outlines and textbooks, it defines the core concepts, principles, operating procedures and safety standards in each professional field, providing authoritative basis for identifying "knowledge concept defects" and "process standard defects";

[0165] ② Excellent Examples and Typical Defect Case Library: Composed of high-quality assignment texts and assignment fragments containing typical errors, serving as a collection of high-quality positive samples and defective negative samples, providing key reference standards for model training and semantic matching.

[0166] Second, before training the model, the above sample dataset needs to be preprocessed, including data anonymization and cleaning, text standardization, and optional text augmentation. Data anonymization and cleaning, and text standardization are the same as or similar to the methods in Example 1. Text augmentation addresses the problem of insufficient high-quality samples under certain skill labels. Specifically, for assignment texts already labeled as high-quality, techniques such as back-translation or dictionary-based synonym replacement are used to generate new texts with consistent semantics but diverse expressions, while maintaining the core semantics and skill features of the original text. This aims to balance the dataset distribution, expand the number of positive samples, thereby enhancing the model's generalization ability on scarce categories and preventing overfitting.

[0167] Third, after preprocessing, manual annotation is required, especially for skill defect labels. This application constructs a well-structured and operable skill defect labeling system that can be recognized and understood by machine learning models.

[0168] Skill deficiency labels include professional skill dimension deficiency labels and general ability dimension deficiency labels.

[0169] 1. Deficiency tags in professional skills dimension

[0170] The professional skills dimension deficiency tags are directly linked to the knowledge system and practical norms of a specific discipline. Their dimension division and tag definition are closely aligned with industry standards and workflows, aiming to accurately pinpoint students' specific weaknesses in professional practice. These include:

[0171] A1 Knowledge Concept Deficiencies:

[0172] Definition: Incorrect or superficial understanding of the core concepts and basic principles of a discipline;

[0173] Tags: A1.1 Conceptual error, A1.2 Conceptual confusion, A1.3 Unclear explanation of principles;

[0174] Typical textual representations include statements that contradict accepted theories or confuse the connotation and denotation of similar concepts.

[0175] A2 process specification defects:

[0176] Definition: Failure to follow standard operating procedures or safety regulations;

[0177] Tags: A2.1 missing steps, A2.2 incorrect order, A2.3 omission of safety specifications;

[0178] Typical textual characteristics: Key steps in the standard process are omitted or reversed in order; necessary safety precautions are not mentioned in the experiment / operation report.

[0179] A3 tool method defects:

[0180] Definition: A deviation exists in the selection or use of tools, instruments, or methods to solve a problem;

[0181] Tags: A3.1 Inappropriate tool selection, A3.2 Incorrect measurement method, A3.3 Limited data analysis methods;

[0182] Typical textual representations include: selecting inappropriate tools or software in the problem context; and describing incorrect measurement or calculation methods.

[0183] A4 Problem Solving Deficiencies:

[0184] Definition: When analyzing the causes of a problem and designing a solution, the approach is one-sided or the solution is infeasible;

[0185] Tags: A4.1 One-sided causal analysis, A4.2 Infeasible solution, A4.3 Failure to consider multiple possibilities;

[0186] Typical textual characteristics: Fault diagnosis is based on a single perspective; proposed solutions are difficult to implement under technical, cost, or practical constraints.

[0187] 2. Defect Tags in General Capability Dimension

[0188] The general competency dimension's deficiency labels transcend specific subjects, reflecting common weaknesses among students in thinking, expression, and work habits. These include:

[0189] B1 logic structure defect:

[0190] Definition: The textual narrative lacks a clear logical thread and hierarchical structure;

[0191] Tags: B1.1 Logically confused, B1.2 Disorganized, B1.3 Lack of causal relationship;

[0192] Typical textual characteristics: extensive use of "then...then..." for simple listing; lack of logical connectors such as "firstly, secondly, therefore"; causal inferences are invalid or jump around.

[0193] B2's meticulous and detailed flaws:

[0194] Definition: Lack of accuracy and completeness in data recording, description, and presentation;

[0195] Tags: B2.1 Missing data, B2.2 Omitted units, B2.3 Vague description;

[0196] Typical textual characteristics: key experimental data were not recorded; physical quantities lacked units; imprecise terms such as "approximately", "more or less", and "somewhat bright" were used.

[0197] B3 Communication and Expression Deficiencies:

[0198] Definition: Failure to communicate effectively using standard and accurate language;

[0199] Tags: B3.1 Inappropriate use of terminology, B3.2 Incoherent sentences, B3.3 Non-standard format;

[0200] Typical textual characteristics: incorrect use of technical terms in a specific context; incomplete or lengthy sentence components; failure to comply with standard report writing format requirements.

[0201] IV. Multi-level Feature Representation and Model-Driven Learning

[0202] To achieve accurate assessment of students' skill levels, it is necessary to construct feature representations that comprehensively capture information from surface vocabulary to deep semantics. Unlike traditional methods that rely on manually designed statistical features, this application employs a task-driven deep representation learning paradigm. By designing high-level semantic tasks such as sequence labeling (information extraction), semantic matching (normativity checking), and defect classification, a fine-tuned pre-trained language model (such as BERT) is guided to automatically learn and form optimal multi-level feature representations while solving these specific tasks. These deep features, rich in linguistic information and generated adaptively by the model, form the foundation for subsequent intelligent defect diagnosis. The feature learning primarily covers the following three levels:

[0203] 1. Representation learning at the lexical and terminological levels

[0204] Vocabulary is the cornerstone of professional competence expression. Through domain-adaptive pre-training and fine-tuning, the system enables the model to deeply understand the precise semantics of professional terms and their contextual usage.

[0205] Core implementation path: In the terminology normalization and domain fine-tuning stage, the model's massive parameter learning maps technical terms (such as "PWM" and "Kirchhoff's laws") to specific regions in a high-dimensional semantic space and understands their co-occurrence relationship with related concepts (such as "ADC" and "sampling rate").

[0206] Related skill defect identification: The lexical semantic knowledge obtained by the model directly supports the semantic defect scanning model in accurately identifying knowledge concept defects (such as conceptual confusion) and communication expression defects (such as terminology misuse). For example, the model can determine the semantic conflict between "voltage range" and "measuring current" in the expression "measuring current with a voltage range".

[0207] 2. Representation learning at the syntactic and logical structure levels

[0208] The logical structure and rigorousness of thought are reflected in language structure. The self-attention mechanism of the Transformer architecture enables the model to automatically capture long-distance lexical dependencies and logical connections.

[0209] Core implementation path: When the model performs tasks such as extracting the "analysis process" or judging whether the logic is coherent, its internal self-attention weights dynamically model the dependency relationships and logical connection patterns between sentence components (such as "because...therefore..."), without the need for an external syntactic parser.

[0210] Related skill defect identification: The "structure awareness" ability acquired at this level is the core basis for the semantic defect scanning model to detect logical structure defects (such as logical inconsistencies and missing causal links). The model can identify missing necessary reasoning steps or erroneous causal links in a discourse.

[0211] 3. Representation learning at the deep semantic and intent levels

[0212] Higher-order thinking skills, such as problem-solving and creativity, are embedded in the deep semantics and intent of the text. The model achieves a deep understanding of the overall semantics of the text and the author's intent through end-to-end task learning.

[0213] Core implementation path: In tasks such as process specification inspection (comparing the feasibility of solutions) and defect severity assessment, the model learns to generate sentence vectors or discourse representations containing rich semantic information through comparison and reasoning, which are used to measure the relevance, innovativeness or severity of defects of the solution.

[0214] Related skill deficiency identification: Deep semantic representation at this level directly empowers the process specification inspection model to identify problem-solving deficiencies (such as infeasibility of the solution) and supports the results fusion module in quantitatively assessing the severity of the deficiencies. For example, semantic matching can be used to determine the core differences between student solutions and standard solutions.

[0215] Instead of explicitly designing and calculating shallow statistical features such as word frequency and sentence length, this system drives a powerful pre-trained model to internalize and adaptively learn a multi-layered, high-dimensional feature representation system by designing complex semantic tasks closely aligned with the assessment objectives. This system is embedded in the model parameters and directly serves downstream tasks such as information structuring, defect scanning, and severity assessment, thereby achieving end-to-end, high-precision intelligent analysis from raw text to skill defect insights.

[0216] V. Software Systems Based on Pipeline Architecture

[0217] like Figure 8 As shown, the software system of this application adopts a phased pipeline architecture, decomposing the complex defect identification task into a series of manageable and interpretable sub-tasks. The model used in the software system has undergone domain-adaptive fine-tuning, which is reflected in the following aspects.

[0218] 1. Minor adjustments to domain-specific language style

[0219] General models are usually trained on massive amounts of general data such as Wikipedia, news, and web pages. Although they understand Chinese, they are not familiar with the "jargon" of vocational education. Therefore, we further train them using sample datasets related to the field of vocational education to update their parameters and make their internal representations closer to the target field.

[0220] 2. Downstream task fine-tuning

[0221] First, a classification layer (text classifier) ​​is set at the top of the entire software system to classify the input job text, thereby triggering different rule sets to process the input job text. Then, an information extraction layer is set after the text classifier to extract text information fragments related to information slots. Finally, a defect identification layer is set to identify specific defect types.

[0222] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the foregoing embodiments have described the present invention in detail, those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and these substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A text mining-based method for personalized formative assessment of students' skills, comprising: Step 1: Obtain the original text of the formative assessment task and preprocess it to obtain the preprocessed task text; characterized in that: it further includes: Step 2: Classify the preprocessed task texts to obtain the task category to which the task text belongs; Step 3: Based on the job category to which the job text belongs, trigger the analysis engine to analyze the job text according to the corresponding set of analysis rules; Step 4: Output the evaluation report; In step 3, the analysis engine performs the following on the categorized job text: Step 31: Determine the corresponding predefined structured form based on the job category; Step 32: For each information slot in the predefined structured form, extract information, that is, identify and extract relevant text information fragments from the preprocessed job text; and establish a mapping relationship between the information slots and the text information fragments to obtain a structured data object; Step 33: Based on the type of information slot in the structured data object, input the text information fragments corresponding to the information slots into the corresponding defect recognition model to obtain a list of defects in the job text; Step 34: For each defect in the defect list, perform a severity assessment.

2. The student personalized skills formative assessment method based on text mining as described in claim 1, characterized in that: In step 2, the preprocessed assignment text is classified using a text classifier to obtain the assignment category to which the assignment text belongs; the assignment category is one of experimental report, design scheme, and case analysis.

3. The student personalized skills formative assessment method based on text mining as described in claim 2, characterized in that: For assignment texts of the type of experiment report, the information slots in the predefined structured form include: problem phenomenon, analysis process, tools used, solution, and experiment conclusion; for assignment texts of the type of design scheme, the information slots in the predefined structured form include: project requirements, technical solution, feasibility analysis, and innovation points; for assignment texts of the type of case analysis, the information slots in the predefined structured form include: case background, key issue identification, analysis process, decision and arguments.

4. The student personalized skills formative assessment method based on text mining as described in claim 3, characterized in that: In step 32, for each information slot in the structured form, firstly, based on the characteristics of the target information slot, determine whether to use sequence labeling or reading comprehension / question-answering information extraction model for that information slot. Then, for that information slot, information extraction is performed using the determined information extraction model until the information extraction for each information slot is completed. Specifically, for information slots with loose structure and discrete answers, sequence labeling is used for information extraction, while for information slots with relatively stable structure and concentrated answers, reading comprehension / question-answering is used for information extraction.

5. The student personalized skills formative assessment method based on text mining as described in claim 4, characterized in that: In step 33, based on the structured data object, a defect identification process is initiated. This process uses information slots within the structured data object as the basic processing unit and specifically includes: Step 331: Based on the predefined routing configuration table, route the text information fragment corresponding to each information slot in the structured data object to at least one of the semantic defect scanning model, process specification inspection model, and hard specification audit model; Step 332: The semantic defect scanning model, process specification inspection model, and hard specification audit model identify defects in the received text information fragments, obtain the defects existing in the input text information fragments, and establish the association relationship between the defect type and the information slots of the defect source, the defect description, and the text fragments corresponding to the defect. Step 333: The result fusion module performs multi-source evidence fusion on the mapping relationship established by the semantic defect scanning model, process specification inspection model, and hard specification audit model to generate a defect list.

6. The student personalized skills formative assessment method based on text mining as described in claim 5, characterized in that: In step 332, the process of defect identification by the semantic defect scanning model includes: Step 332A1: The BERT encoder converts the text information fragments corresponding to the information slots into context-dependent word vector sequences; Step 332A2: For the word vector sequence, the CRF layer acts as a decoder to learn the transition constraints between labels and outputs the globally optimal label sequence; Step 332A3: Scan the global optimal label sequence to find all segments with non-O labels, and establish a mapping relationship between the defect type, defect description, corresponding information slot, and corresponding text segment for segments with non-O labels.

7. The student personalized skills formative assessment method based on text mining as described in claim 6, characterized in that: In step 332, the process of defect identification by the process specification inspection model includes: Step 332B1: Retrieve the "best practice" or "standard procedure" text corresponding to the input text information fragment from the domain standard knowledge base; Step 332B2: Using the Sentence-BERT model, encode the input text information fragments and their corresponding "best practices" or "standard procedures" texts into sentence vectors respectively; Step 332B3: Calculate the cosine similarity between the corresponding sentence vectors; Step 332B4: Determine whether there is a defect based on whether the cosine similarity value is lower than the set threshold, and if there is a defect, establish a mapping relationship between defect type, defect description, and corresponding information slot, where the defect description includes student description fragment, standard description fragment, and similarity score.

8. The student personalized skills formative assessment method based on text mining as described in claim 7, characterized in that: In step 332, the process of defect identification by the rigid standard audit model includes: Step 332C1: Based on a predefined rule set, scan the input text information fragments. If the rule set requirements are not met, trigger a defect. Step 332C2: Based on the triggered defect, establish a mapping relationship between defect type, defect description, corresponding information slot, and corresponding text fragment, where the defect description includes the violated rules.

9. The student personalized skills formative assessment method based on text mining as described in claim 8, characterized in that: The results fusion module, based on a fusion algorithm combining heuristic rules and confidence level comparisons, performs multi-source evidence fusion on the mapping relationships established by the semantic defect scanning model, process specification inspection model, and hard specification audit model, generating a standardized defect list, including: Step 3331: Perform evidence aggregation, that is, merge different defective evidence pointing to the same text location; Step 3332: Conflict resolution is performed. When the rigid specification audit model determines that there is a defect but the semantic defect scanning model does not find a defect, the determination result of the rigid specification audit model shall prevail. When the defect scanning results of the semantic defect scanning model and the process specification inspection model are contradictory, the scanning result of the semantic defect scanning model shall be given higher weight. Step 3333: Generate a standardized defect list. Each record in the list includes: defect type, defect description, defect source information slot, corresponding text fragment, and initial severity score.

10. A text mining-based personalized formative assessment system for student skills, comprising a processor for running a computer program, characterized in that: To implement the text mining-based formative assessment method for personalized student skills as described in any one of claims 1-9.

Images