Generative artificial intelligence-driven teaching design automatic evaluation system

Abstract

The invention discloses a generative artificial intelligence-driven teaching design automatic evaluation system, and the system comprises an evaluation system knowledge base which is used for providing a knowledge basis and an objective judgment basis; the evaluation core processing engine is used for scoring teaching design based on an intelligent scoring mechanism of multi-source fusion; the evaluation result and feedback system is used for outputting static scores, generating targeted improvement suggestions by identifying weak links and supporting iterative optimization of teaching design; the method is mainly used for automatic evaluation, feedback and optimization of compulsory education teaching design in ordinary education.

Description

Technical Field

[0001] This invention relates to the field of intelligent teaching assessment technology, and in particular to an automatic assessment system for instructional design driven by generative artificial intelligence. Background Technology

[0002] In today's teacher education system, instructional design, as a core indicator for measuring the professional competence of teacher trainees, directly impacts the quality of future teacher training. Currently, the evaluation of instructional design courses in teacher training institutions primarily relies on the following methods:

[0003] Teacher self-review is the primary assessment method, with teachers scoring instructional designs based on their own teaching experience and subject-specific standards. However, with the expansion of university enrollment and the imbalance in the student-to-faculty ratio, teachers face an increasing workload in reviewing, often making it difficult to guarantee the timeliness and depth of their reviews. Furthermore, subjective differences in evaluation standards among different teachers lead to a lack of consistency and comparability in the scoring results, affecting the fairness of the assessment.

[0004] Peer assessment is widely used as a supplementary assessment method, allowing students to improve their critical thinking skills through mutual feedback. However, due to students' limited professional knowledge and teaching experience, peer assessment results often lack professional depth, and the quality of feedback varies, failing to provide effective guidance for improvement.

[0005] Traditional instructional management systems, while providing basic functions such as assignment submission and grade recording, remain at the level of digital management and lack the ability to intelligently analyze instructional design content. These systems cannot automatically identify the strengths and weaknesses of instructional designs, provide personalized feedback, or support the continuous optimization process of instructional design.

[0006] The common shortcomings of these existing assessment methods include: the assessment process is highly subjective, making it difficult to establish unified and objective scoring standards; assessment feedback is often delayed by several days or even weeks, failing to meet students' needs for immediate improvement; the assessment content is general and lacks specific analysis of each stage of instructional design; and the assessment-feedback-improvement cycle cannot form a closed loop, making it difficult to support the continuous optimization of instructional design and hindering the effective improvement of teacher trainees' instructional design capabilities. Summary of the Invention

[0007] To address the problems existing in the prior art, the purpose of this invention is to provide a generative artificial intelligence-driven automatic evaluation system for instructional design. This invention is mainly used for the automated evaluation, feedback, and optimization of compulsory education instructional design in teacher training.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a generative artificial intelligence-driven automatic evaluation system for instructional design, comprising:

[0009] The evaluation system's knowledge base is used to provide a knowledge foundation and objective criteria for judgment.

[0010] The evaluation core processing engine is used to score instructional designs based on a multi-source fusion-based intelligent scoring mechanism;

[0011] The assessment results and feedback system is used to output static scores and generate targeted improvement suggestions by identifying weaknesses, supporting the iterative optimization of instructional design.

[0012] As a further improvement of the present invention, the knowledge base of the assessment system specifically includes: curriculum standards, instructional design evaluation scales, and a typical case library; wherein:

[0013] The curriculum standards are teaching guidance documents used to clarify the core competency requirements, the learning task group system, and the academic quality standards; and the curriculum standards are pre-processed into editable structured documents.

[0014] The instructional design evaluation scale includes multiple levels of assessment indicators, each of which includes multi-level scoring criteria and operational descriptions, used to evaluate instructional design competence.

[0015] The typical case library is built based on multiple real teaching designs collected. Each design has expert scores and reasons for the scores in multiple dimensions, covering different levels and score ranges, to provide a reliable reference for similarity scoring.

[0016] As a further improvement of the present invention, the core evaluation processing engine specifically includes: a GENAI direct scoring module, a case similarity scoring module, and a weighted fusion scoring module; wherein:

[0017] The GENAI direct scoring module uses curriculum standards and instructional design evaluation scales as a knowledge base, adopts the retrieval-enhanced generative framework RAG, and directly scores multiple dimensions of instructional design through generative artificial intelligence GENAI.

[0018] The case similarity scoring module uses a similarity measurement method to extract core elements from different dimensions of the instructional design to be evaluated, compare them with typical cases in the typical case library, select the cases with the highest similarity, and take the average score as the scoring reference.

[0019] The weighted fusion scoring module combines the scores from the GENAI direct scoring module and the case similarity scoring module in a weighted manner to obtain the final scoring result.

[0020] As a further improvement of the present invention, by adjusting the weight parameters, the system score is made closer to the expert score, thereby improving the accuracy of the evaluation.

[0021] As a further improvement of the present invention, the evaluation results and feedback system are specifically used for dimension-specific scoring, comprehensive rating, and generation of improvement suggestions; wherein:

[0022] The scoring is dimensional and itemized, providing clear scores for each dimension of the instructional design evaluation scale, along with the reasons for the scores and suggestions for improvement.

[0023] The comprehensive rating is based on the total score, providing a multi-level evaluation.

[0024] Improvement suggestions are generated by combining instructional design content, scoring results, and similar cases, using AI technology to generate targeted improvement suggestions.

[0025] As a further improvement of the present invention, the instructional design evaluation scale includes a three-stage argumentation process, wherein:

[0026] The first stage involves literature analysis and preliminary framework construction, resulting in a preliminary evaluation framework that includes multiple multi-level evaluation indicators.

[0027] The second stage uses the Delphi method to conduct three rounds of expert consultation and indicator optimization. The first round uses an open-ended questionnaire to ask experts to provide suggestions for modifying the preliminary indicator framework. The second round uses a structured questionnaire to ask experts to score the importance of the adjusted indicators and calculate the mean, standard deviation and coefficient of variation of each indicator. The third round feeds back the statistical results of the previous round to the experts and asks them to make a final assessment based on their understanding of the group's opinions.

[0028] The third phase involved empirical testing and revision.

[0029] As a further improvement of this invention, the instructional design evaluation scale employs a dual mechanism of grade judgment and score refinement in its multi-level scoring criteria. Evaluators first determine the grade range based on overall performance, and then provide precise scores within each grade range based on specific performance. The operational description is used to establish a clear scoring rule system. First, it quantifies the scoring criteria, transforming abstract expressions into concrete score ranges. Second, it distinguishes between different scoring modes, including a score-accumulation type and a grade-judgment type. The score-accumulation type calculates the total score by accumulating scores from multiple elements, while the grade-judgment type directly determines the grade based on overall performance. The operational description also includes specific behavioral performance standards and critical judgment conditions. The behavioral performance description uses the teacher's design behavior as the basis for judgment, and the critical judgment conditions clearly define the boundaries between different grades.

[0030] As a further improvement to the present invention, the method for constructing the typical case library is as follows:

[0031] (1) Data collection: Select real teaching designs with a unified theme as raw data;

[0032] (2) Expert scoring: Each lesson plan is scored by educational experts, who give specific scores in multiple dimensions and the basis for scoring, forming a basic standard case expert scoring sample library.

[0033] (3) GENAI-generated cases: GENAI technology is used to generate standardized case descriptions for each instructional design; the generation process adopts the RAG framework, using curriculum standards and evaluation scales as the knowledge base, combined with expert scoring results; the generated case content includes three core elements: the key information that best reflects the characteristics of the indicators is extracted from typical case content; the scoring basis clearly explains the specific reasons for obtaining the corresponding score; common features summarize the common characteristics of cases of the same level; all generated cases have been reviewed by the original scoring experts to ensure the accuracy and representativeness of the content;

[0034] (4) Case Balance Optimization: Through statistical analysis and optimization algorithms, the cases are balanced to ensure that each indicator retains multiple representative cases at multiple levels, while ensuring that the actual scores are reasonably distributed across each score range.

[0035] As a further improvement to the present invention, the case similarity scoring module specifically executes the following process:

[0036] (1) Text preprocessing, including word segmentation: decomposing the case text into professional vocabulary, phrases and key expressions in the field of education; stop word removal: filtering out common words that have little semantic contribution; standardization: converting synonyms into a unified form;

[0037] (2) Feature extraction: keywords and phrases extracted directly from the text, including vocabulary related to teaching methods, design elements, and quality descriptions;

[0038] (3) Vectorization mapping: assign weights to each feature to reflect its importance in the evaluation index; comprehensively map the frequency of feature occurrence, importance and contextual relevance into vector values; construct a multi-dimensional feature space, with each dimension corresponding to an evaluation element;

[0039] (4) Customize indicator dimensions: Customize feature dimensions for each of the multiple different indicators;

[0040] (5) Case library calibration: establish a scoring reference system using the scored cases in the case library, and vectorize the typical cases of each level; form a mapping relationship between indicators, levels and feature vectors;

[0041] For the instructional design to be graded, extract the relevant design elements and content from the instructional design, and convert them into a numerical feature vector A using the same method as the semantic vectorization of the case library. Then, calculate the cosine similarity with the semantic feature vector B of the case library for the same indicators.

[0042]

[0043] The result is a real number in the range [-1, 1].

[0044] For each indicator of the instructional design to be scored, complete the above calculation process to obtain the most similar typical cases. The average score of their actual scores is the score of that indicator of the instructional design.

[0045] As a further improvement of the present invention, the final score S of the instructional design is obtained by a weighted score of GENAI direct score S1 and case similarity score S2, as shown in Equation 2:

[0046] s = a·s1 + (1-a)·s2;

[0047] Where 'a' is unknown, and it is a weighting coefficient, a floating-point number with a value ranging from 0 to 1. <a<1。

[0048] This invention combines artificial intelligence technology with theories of pedagogy and subject teaching to construct a novel intelligent evaluation system for compulsory education instructional design. This system can automatically analyze instructional design texts, identify their structured elements, including core components such as textbook and curriculum standard analysis, student learning analysis, learning objective formulation, teaching method design, teaching process design, and teaching evaluation design. Based on multi-dimensional evaluation criteria, it generates objective scores and provides detailed scoring basis and targeted modification suggestions.

[0049] This invention makes the instructional design evaluation process transparent, realizes the automation, intelligence and personalization of instructional design evaluation, and establishes a complete closed-loop mechanism of evaluation-feedback-improvement. It effectively solves the core problems of low efficiency, inconsistent standards and delayed feedback in existing technologies, and provides teacher trainees with timely, objective and professional instructional design evaluation services, significantly improving the quality and efficiency of instructional design ability training.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention can achieve automatic scoring, including (1) automatic text analysis: supports uploading teaching design text and automatically parses the content structure; (2) multi-dimensional intelligent assessment: based on educational theory, it scores from multiple dimensions such as hierarchical task analysis, teaching value positioning, accurate diagnosis of learning situation, and scientific goal setting; (3) objective score generation: adopts AI algorithm to ensure consistent scoring standards and reduce human bias; (4) detailed feedback report: provides scoring basis and gives personalized optimization suggestions for weak links.

[0052] 2. This invention can diagnose problems, identify issues such as unclear logic and mismatch between objectives and activities in instructional design, and provide a structured list of problems rather than a single score result. It provides educators with a tool for monitoring teaching quality and reduces the cost of human review. Attached Figure Description

[0053] Figure 1 This is a system overall framework diagram of an embodiment of the present invention;

[0054] Figure 2 This is a flowchart illustrating the construction process of a typical case library in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart illustrating the GENAI case generation process in an embodiment of the present invention.

[0056] Figure 4 This is a flowchart illustrating the case library balancing optimization process in an embodiment of the present invention;

[0057] Figure 5 This is a flowchart illustrating the scoring process of a teaching design in an embodiment of the present invention.

[0058] Figure 6 This is a flowchart of the case similarity scoring process in an embodiment of the present invention;

[0059] Figure 7 This is a flowchart of the similarity scoring process in an embodiment of the present invention;

[0060] Figure 8 This is a flowchart illustrating the optimization process for finding the optimal weighting coefficients in an embodiment of the present invention;

[0061] Figure 9 This is a graph showing the total mean square error of the search process as a function of the weighting coefficients in an embodiment of the present invention. Detailed Implementation

[0062] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0063] Example

[0064] Instructional design, as a core component of teachers' professional competence, exhibits specific characteristics and requirements across different subject areas. This embodiment uses the instructional design of Chinese language teaching in compulsory education as an example to construct an intelligent assessment system. As a fundamental subject, Chinese language teaching is characterized by the need for a systematic integration of language knowledge and humanistic literacy cultivation, emphasizing in-depth text interpretation and precise language expression, and focusing on the implicit permeation of emotional attitudes and values ​​while explicitly realizing the function of cultural inheritance. This complex characteristic determines that Chinese language instructional design must both adhere to general instructional design principles and possess significant subject-specific characteristics. This embodiment takes the intelligent assessment of Chinese language instructional design as its research core, constructing a theoretical and practical framework for intelligent assessment of instructional design in the field of teacher education; specifically including:

[0065] 1. System Overview:

[0066] This embodiment constructs a scientific and complete assessment benchmark system, which consists of three mutually supportive and organically integrated core components: the Chinese language curriculum standards as the theoretical basis, the instructional design evaluation scale as the assessment tool, and the typical case library as the reference benchmark. These three components work synergistically to provide a solid knowledge base and objective judgment criteria for GENAI's automated assessment, ensuring the scientific rigor, authority, and operability of the assessment results. Specifically:

[0067] This system is an intelligent evaluation platform based on the "Compulsory Education Chinese Language Curriculum Standards (2022 Edition)" (hereinafter referred to as the Curriculum Standards), the Instructional Design Evaluation Scale (hereinafter referred to as the Scale), and the Typical Case Library (hereinafter referred to as the Case Library). It aims to provide objective, comprehensive, and efficient automated evaluation of Chinese language instructional design. Through a multi-level structural design, the system integrates GENAI technology and educational expertise, achieving intelligent and standardized instructional design evaluation. The overall system framework is as follows: Figure 1 As shown.

[0068] 2. Evaluate the system's knowledge base:

[0069] The system's automatic scoring is based on three key data sources:

[0070] The Chinese Language Curriculum Standards (2022 Edition): Officially titled "Compulsory Education Chinese Language Curriculum Standards (2022 Edition)," this is a legally mandated teaching guidance document issued by the Ministry of Education of China. It clarifies the core literacy requirements, learning task cluster system, and academic quality standards for Chinese language. As the highest criterion for assessment, all evaluation dimensions are designed based on the curriculum standards. The officially provided curriculum standards are in PDF format, and most of the content is in image format. Preprocessing is required to convert them into an editable, structured document for use in building the knowledge base of an automated assessment system.

[0071] The "Evaluation Scale for Instructional Design in Compulsory Education Chinese Language Curriculum" was developed based on in-depth research into curriculum standards and the evaluation of instructional design skills. This scale, meticulously designed by this invention, is a professional assessment tool comprising 8 primary indicators and 21 secondary indicators. Developed through scientific methods such as literature analysis, expert consultation, and empirical testing, it exhibits good reliability and validity. The scale sets four levels of scoring criteria and operational descriptions for each indicator to ensure consistency and objectivity in scoring.

[0072] Typical Case Library: Built upon 102 collected authentic teaching designs, each design includes expert scores across 21 dimensions and their rationale. The library has undergone distribution optimization to ensure balanced coverage across different levels and score ranges, providing a reliable reference for similarity scoring.

[0073] 3. Evaluate the core processing engine:

[0074] The core processing mechanism of the system's automatic scoring includes three complementary scoring modules:

[0075] The GENAI direct scoring module uses curriculum standards and assessment scales as a knowledge base and adopts the RAG (Retrieval-Augmented Generation) framework to directly score the 21 dimensions of instructional design using GENAI. Preliminary research indicates that GENAI scores alone are generally higher than expert scores, necessitating the use of other scoring mechanisms to balance the results.

[0076] Case similarity scoring module: This module uses a similarity measurement method to extract core elements from different dimensions of the instructional design to be evaluated. These elements are then compared with typical cases in the case library. The three most similar cases are selected, and their average score is used as a scoring reference. This scoring result is generally slightly lower than the expert scores, providing good complementarity with the direct GENAI scoring.

[0077] Weighted Fusion Scoring Module: This module weights and fuses the direct AI score and the case similarity score to arrive at the final score. By adjusting the weight parameters, the system score is made closer to the expert score, improving the accuracy of the evaluation.

[0078] 4. Evaluation Results and Feedback System:

[0079] The system outputs three types of assessment results, providing comprehensive feedback for teachers and students:

[0080] Dimensional scoring: Clear scores are given for each of the 21 dimensions in the instructional design evaluation scale, along with the reasons for the scores and suggestions for improvement, to help users understand the specific performance of the instructional design in each aspect.

[0081] Comprehensive rating: Based on the total score, a comprehensive evaluation of four levels is given: unsatisfactory, satisfactory, good, and excellent, providing a clear positioning for teachers' professional development.

[0082] Improvement suggestion generation: Combining instructional design content, scoring results, and similar cases, AI technology generates targeted improvement suggestions to provide specific guidance for optimizing instructional design.

[0083] In summary, this embodiment:

[0084] A multi-source integrated intelligent scoring mechanism was constructed. The GENAI direct scoring module makes an initial assessment based on the curriculum standards and scales, the case similarity scoring module provides empirical reference through semantic vectorization matching, and the weighted fusion module uses an optimization algorithm (the weight coefficient a = 0.274 obtained by SciPy) to balance the two scores, thus solving the "looseness bias" problem of single AI scoring.

[0085] It achieves a structured expression of assessment knowledge. The curriculum standards in PDF format are transformed into an AI-readable knowledge base, the abstract evaluation criteria are refined into 21 operational dimensions, and expert experience is solidified into 102 standardized cases, making implicit teaching evaluation knowledge explicit and calculable.

[0086] A dynamically optimized evaluation system has been established. The system not only outputs static scores but also generates targeted improvement suggestions by identifying weaknesses, supporting iterative optimization of instructional design. The case library is continuously expanded and updated, and the scoring model can be adjusted and optimized based on new data, ensuring the long-term effectiveness of the system.

[0087] The following is a further explanation of this embodiment:

[0088] (1) Construction of Structured Chinese Language Curriculum Standards:

[0089] The "Compulsory Education Chinese Language Curriculum Standards," as the guiding document for Chinese language teaching, serves as the foundation and basis for Chinese language teaching design. The curriculum standards clearly define the core competency requirements, stage-specific goals, and content system of Chinese language education, providing directional guidance and specific norms for teachers' instructional design. In the process of designing Chinese language teaching, teachers need to deeply understand the connotations of core competencies such as "language application," "thinking ability," "aesthetic creation," and "cultural confidence" in the curriculum standards, accurately grasp the ability requirements for different stages, and design scientifically sound teaching activities based on the content framework of "basic learning task groups," "developmental learning task groups," and "extended learning task groups." High-quality Chinese language teaching design should implement the educational orientation of the curriculum standards, reflect the characteristics of the learning task groups in the curriculum standards, create authentic and meaningful learning situations, and explore changes in Chinese language teaching methods—all of which require adherence to the curriculum standards. Therefore, the "Compulsory Education Chinese Language Curriculum Standards" is not only an important benchmark for evaluating the quality of Chinese language teaching design but also the core theoretical basis for constructing the intelligent assessment system in this embodiment.

[0090] The curriculum standard document will be part of the GENAI knowledge base, assisting AI in performing structural analysis and extracting key elements from instructional designs. The Chinese language curriculum standard document downloaded from the Ministry of Education's website is a PDF document, in which the text is in image format and cannot be directly used by AI; it needs to be preprocessed into a structured document.

[0091] (2) Design of instructional design evaluation scale:

[0092] (2.1) Evaluation index design:

[0093] The construction of the indicator system underwent a three-stage scientific verification process. The first stage was literature analysis and preliminary framework construction. The research team systematically analyzed 356 articles related to instructional design evaluation published in core domestic and international journals between 2010 and 2023 using bibliometric methods, and extracted 128 evaluation elements using content analysis. Based on this, a comparative analysis was conducted on the general framework of six domains and sixteen abilities proposed in the "Teacher Professional Competency Assessment Model," as shown in Table 1. While this framework is systematic, it has limitations in its adaptability to the Chinese language subject. Therefore, the study further integrated the subject-specific requirements of the "Framework of Pre-service Chinese Language Teachers' Teaching Competency" (as shown in Table 2), particularly its detailed description of the ability to apply Chinese language knowledge. After systematic integration and theoretical analysis, a preliminary evaluation framework comprising 10 primary indicators and 35 secondary indicators was formed.

[0094] Table 1 General Model

[0095]

[0096]

[0097] Table 2 Pre-employment Framework

[0098]

[0099] The second phase employed the Delphi method for expert consultation and indicator optimization. The expert panel was selected with a strong emphasis on balancing theory and practice, comprising three professors of Chinese language curriculum and teaching methodology from comprehensive normal universities, two provincial-level outstanding Chinese language teachers, and two Chinese language teaching and research staff. These experts possessed extensive experience in teaching guidance and unique insights into evaluating instructional design. The three rounds of consultation strictly followed the standardized procedures of the Delphi method: The first round used an open-ended questionnaire, soliciting expert feedback on the preliminary indicator framework. A total of 147 suggestions were collected, leading to the addition of indicators such as "content accuracy" and "originality," while deleting some general indicators with low relevance to the Chinese language subject. The second round used a structured questionnaire, requiring experts to rate the importance of the adjusted indicators (using a 5-point Likert scale) and calculate the mean, standard deviation, and coefficient of variation for each indicator. The third round provided the experts with the statistical results from the previous rounds, allowing them to make a final assessment based on the group's opinions. The convergence of expert opinions shows that Kendall's coefficient of harmony increased from 0.62 in the first round to 0.78 in the second round, and finally reached 0.85 in the third round, while the coefficient of variation decreased from 0.32 to 0.15, indicating that the expert team reached a high degree of consensus.

[0100] The third stage involved empirical testing and revision. The research team collected 121 Chinese language teaching designs from a normal university as a validation sample, which were then scored by trained raters using a pre-determined indicator system. Psychometric methods such as item analysis and factor analysis were used to test and optimize the indicator system. Specifically, three indicators with a discrimination index below 0.3 were removed, as these indicators failed to effectively differentiate between teaching designs of different levels in actual scoring; two pairs of indicators with a correlation coefficient above 0.85 were merged to avoid duplication of evaluation dimensions; and the attribution of indicators with factor loadings below 0.4 was adjusted, making the structure of the indicator system clearer and more reasonable.

[0101] After systematic theoretical construction and empirical testing, a Chinese language teaching design evaluation system was finally formed, comprising 8 primary indicators and 21 secondary indicators, as shown in Table 3. This indicator system reflects both the general principles of instructional design, such as "Instructional Process Design Ability" which focuses on the logic of the teaching process, and the specific requirements of the Chinese language subject, such as "Hierarchical Task Analysis" which emphasizes the understanding and application of Chinese language learning task groups. Each indicator has undergone theoretical demonstration, expert approval, and practical testing, ensuring the scientific rigor, comprehensiveness, and operability of the evaluation system.

[0102] Table 3 Evaluation Scale for Chinese Language Teaching Design Ability

[0103]

[0104]

[0105] (2.2) Evaluation levels and methods:

[0106] This embodiment uses a four-level scoring system to evaluate instructional designs. The evaluation levels are based on the stages of teacher professional development and refer to the competency requirements of national teacher professional standards. The four levels are unsatisfactory, satisfactory, good, and excellent, and the corresponding score ranges were determined through statistical analysis.

[0107] A score below 0.6 for the "Unsatisfactory" grade indicates significant flaws in the instructional design, such as misinterpretations of curriculum standards, errors in teaching content, and vague statements of objectives, rendering it unable to support normal teaching activities. A score between 0.6 and 0.7 for the "Pass" grade indicates that the instructional design meets basic requirements and can complete the teaching task, but there is still room for improvement in terms of innovation and relevance. A score between 0.7 and 0.85 for the "Good" grade indicates a high-quality instructional design, demonstrating strong professional competence in goal setting, activity design, and resource utilization. A score between 0.85 and 1.0 for the "Excellent" grade indicates that the instructional design reaches expert level, serving as a model in terms of theoretical depth, practical innovation, and value guidance.

[0108] The scoring method employs a dual mechanism of "level judgment + score refinement." Raters first determine the level range based on overall performance, and then assign precise scores within that range based on specific performance. This method ensures both the standardization and consistency of the evaluation while providing sufficient differentiation. Each level is accompanied by clear judgment criteria and descriptions of typical characteristics, ensuring that different raters can accurately grasp the level boundaries. Through this tiered evaluation method, the system can provide teachers with a clear professional development positioning and improvement direction.

[0109] (2.3) Feasibility evaluation:

[0110] To ensure the objectivity and consistency of the evaluation, this invention provides detailed and actionable descriptions for each evaluation indicator. These descriptions transform abstract evaluation criteria into concrete and executable scoring rules, effectively improving the consistency of human scoring while providing clear judgment criteria for AI-automated evaluation.

[0111] The core of an operational description lies in establishing a clear scoring rule system. First, it quantifies the scoring criteria, transforming abstract expressions such as "in-depth analysis" and "reasonable design" into specific score ranges. For example, "a separate discussion of the curriculum standards and textbooks earns 3.5 points or higher," and "emotional analysis, though vaguely expressed, earns 1.6-2.3 points," enabling raters to make judgments based on clear standards. Second, it differentiates between different scoring models, including a score aggregation model and a grade-based model. The former calculates a total score by accumulating scores from multiple elements, while the latter directly determines the grade based on overall performance. The combined use of these two models ensures both flexibility and accuracy in scoring.

[0112] The operational description also includes specific behavioral performance standards and critical judgment criteria. The behavioral performance description uses the teacher's design behaviors as the basis for judgment; for example, "using both pre-test questionnaires and homework analysis as diagnostic tools" corresponds to a good level, reducing the uncertainty of subjective judgment. Critical judgment criteria clearly define the boundaries between different levels; for example, "a deviation of more than 50% from the target and curriculum standards" is considered unqualified. This clear definition avoids ambiguity in scoring. Furthermore, each level is accompanied by typical cases, providing scorers with intuitive reference standards.

[0113] By establishing a complete and operable evaluation system, this invention has concretized the evaluation criteria and standardized the scoring process. Practical verification shows that after adopting these operable descriptions, the Pearson correlation coefficient among different raters has increased to over 0.8, significantly improving the consistency and reliability of the scoring.

[0114] (3) Construction of a typical case library:

[0115] The typical case library is a key component for achieving accurate scoring in this embodiment. Practice shows that when relying solely on curriculum standards and evaluation scales for GENAI automatic scoring, system scores are generally higher than expert scores, exhibiting a "looseness bias." The main reason for this phenomenon is that the AI ​​system struggles to translate abstract evaluation criteria into concrete judgment bases, particularly lacking reference when distinguishing between adjacent levels. The typical case library, by providing real-world scoring examples, offers the AI ​​system concrete benchmarks, effectively solving the problem of overscoring.

[0116] The case library was built using a systematic process of "data collection - expert scoring - GENAI generation - expert review - balanced optimization," such as... Figure 2 As shown.

[0117] (3.1) Data Acquisition:

[0118] Through a teaching experiment in the Chinese Language and Literature major at a university, the research team collected 102 authentic lesson plans on the themes of "The Road to Shu is Hard (Part 1)" and "Replying to Bai Juyi's Gift at a Banquet in Yangzhou" as raw data. The advantage of choosing a unified theme is that it allows for control over content variables, making the quality differences between different lesson plans more apparent and facilitating the establishment of scoring criteria.

[0119] (3.2) Expert scoring:

[0120] Each lesson plan underwent meticulous evaluation by language education experts. These experts not only provided specific scores across 21 dimensions but also drafted evaluation criteria, forming a basic standard case expert evaluation sample database, as shown in Table 4.

[0121] Table 4. Sample of Expert Scoring Database for Basic Standard Cases

[0122]

[0123] (3.3) GENAI Generated Case:

[0124] Based on expert evaluations, the system uses GENAI technology to generate standardized case descriptions for each instructional design. The generation process employs the RAG framework, using curriculum standards and assessment scales as a knowledge base, combined with expert evaluation results. The generated case content includes three core elements: typical case content extracts the key information that best reflects the characteristics of the indicators; the scoring criteria clearly explain the specific reasons for obtaining the corresponding score; and common features summarize the common characteristics of cases at the same level. All generated cases are reviewed by the original scoring experts to ensure the accuracy and representativeness of the content. The GENAI case generation process is as follows: Figure 3 As shown.

[0125] (3.4) Case Balance Optimization:

[0126] A key feature of the case study library is its systematic balancing and optimization process. The initial case study library suffered from uneven distribution, with an excessive number of unqualified cases for some indicators and a relative shortage of excellent cases. Through statistical analysis and optimization algorithms, the system balanced the cases, ensuring that approximately 15 representative cases are retained for each indicator across the four levels (unqualified, qualified, good, and excellent), while also guaranteeing a reasonable distribution of actual scores across each score range. This balanced case study library not only improves the accuracy of similarity scoring but also provides teachers with reference examples at various levels for learning. The case study library balancing and optimization process is as follows: Figure 4 As shown.

[0127] In summary, the optimized case study library in this embodiment plays a crucial role in the system. During automatic scoring, the system calculates the similarity between the instructional design to be evaluated and typical cases in the library, identifying the most similar case as a scoring reference, effectively balancing the tendency of GENAI's direct scoring to be too high. Simultaneously, the case study library also provides teachers with abundant learning resources, helping them understand the specific characteristics and improvement directions of instructional designs at different levels.

[0128] (4) GENAI direct scoring module:

[0129] GENAI's direct scoring module uses the RAG framework, combining the advantages of retrieval systems and generative models. It can score instructional designs by incorporating curriculum standards and scale information from the knowledge base. Figure 5 This is the scoring process for a teaching design based on the ninth-grade classical Chinese poem "The Road to Shu is Hard".

[0130] Phase 1: Preparation before scoring:

[0131] This paper provides an in-depth analysis of the requirements for teaching classical Chinese poetry and prose in the "Compulsory Education Chinese Language Curriculum Standards (2022 Edition)," particularly the learning tasks for the ninth grade. It also systematically interprets the 21 indicators in the "Instructional Design Evaluation Scale," clarifying the evaluation focus and scoring standards for each indicator. The expert manual scoring results for the instructional designs used in the GENAI automatic scoring system are shown in the table below.

[0132] Table 5 Expert Manual Scoring

[0133]

[0134]

[0135] Phase Two: Item-by-item scoring of 21 indicators:

[0136] Let's take the first indicator, "Hierarchical Task Analysis," as an example to illustrate the scoring process:

[0137] Extract relevant design elements: Identify content from the instructional design that is relevant to the curriculum standards and textbook analysis, including statements such as "The poem 'Hard Road to Travel (Part 1)' is a poem in Unit 3 of the ninth-grade textbook published by People's Education Press."

[0138] Comparison with curriculum standards: Analysis of the accuracy of curriculum standard citations and the clarity of textbook placement in the design.

[0139] Compare with the evaluation scale indicators: Compare with the scoring criteria for the "hierarchical task analysis" indicators, such as the description of "systematically analyzing curriculum standards and textbooks and associating task groups".

[0140] Determining the grade and score: Based on the comparison results, this indicator is determined to have reached the "qualified" grade (3-3.5 points), and is awarded 3 points.

[0141] This process was followed to complete the scoring of all 21 indicators, with a focus on the quality and completeness of the content corresponding to each indicator in the instructional design.

[0142] Phase 3: Outputting Scoring Results

[0143] Output the scoring results (S1) of the 21 indicators in the required format as shown in the table below:

[0144] Table 6 GENA Direct Scoring Results

[0145]

[0146] The GENAI direct score is 61.2 points.

[0147] As shown in the table, the expert scores and AI scores generally exhibit a high degree of consistency, with 76% of the indicators differing by less than 0.5 points. The average AI score is slightly higher than the expert score (2.82 vs. 2.62), indicating a certain degree of "tolerance." The most significant differences are mainly concentrated in the scientific formulation of objectives (1.0 point difference) and the design of assignments (2.0 point difference), where AI clearly overestimates performance. In terms of scoring methods, both consistently give high scores (above 5 points) to the logical construction of activities and student participation. However, in the lower score range, expert evaluations are more stringent, with lower scores given to indicators such as practical feasibility and student autonomy.

[0148] It is evident that AI scoring has achieved a certain level of professionalism and can serve as a preliminary evaluation tool for instructional design. However, it still lags behind experts in indicators that require in-depth teaching experience. The scoring will be further optimized based on the constructed typical case library.

[0149] (5) Case similarity scoring module:

[0150] Case - based similarity scoring method. For each indicator, first extract the content elements related to this indicator in the teaching design, and then compare them with the typical cases of this indicator in the case library to identify their features and performances. By analyzing the similarity between the design content and cases of different levels in the case library, determine the 3 most similar typical cases for this indicator, and take the average of the actual scores of these cases as the score of this indicator, accurate to one decimal place. This method ensures the objectivity and consistency of scoring, and the scoring results can truly reflect the actual performance level of the teaching design on each indicator. The process of case similarity scoring (S2) is as follows Figure 6 shown.

[0151] The core link of similarity scoring is the semantic vectorization processing of each case in the typical case library. The process is as follows Figure 7 shown, where:

[0152] Text pre - processing:

[0153] ① Word segmentation: Decompose the case text into professional vocabulary, phrases and key expressions in the field of education;

[0154] ② Stop - word removal: Filter out common words with little semantic contribution (such as "of", "is", "in", etc.);

[0155] ③ Standardization: Convert synonymous expressions into a unified form (such as "independent learning" and "active learning" are regarded as the same concept).

[0156] Feature extraction:

[0157] Keywords and phrases directly extracted from the text:

[0158] ① Teaching - method vocabulary (such as "inquiry - based", "lecture method", "cooperative learning");

[0159] ② Design elements (such as "learning objectives", "classroom activities", "evaluation methods");

[0160] ③ Quality - description words (such as "reasonable", "detailed", "lacking", "innovative").

[0161] Vectorization mapping:

[0162] ① Assign weights to each feature to reflect its importance in the evaluation indicator;

[0163] ② Comprehensively map the feature occurrence frequency, importance and context relevance into vector values;

[0164] ③ Construct a multi - dimensional feature space, and each dimension corresponds to an evaluation element.

[0165] Indicator dimension customization:

[0166] ① For each of the 21 different indicators, a custom feature dimension was created; for example, the "student subjectivity" indicator focuses on extracting features related to student activity design and participation.

[0167] ② The "Objective Expression Standardization" indicator focuses on extracting features such as the use of action verbs and conditional descriptions.

[0168] Case library calibration:

[0169] ① Establish a scoring reference system using already scored cases in the case library;

[0170] ② Vectorize typical cases for each level (Excellent, Good, Satisfactory, Unsatisfactory);

[0171] ③ Form a mapping relationship between indicators, levels, and feature vectors.

[0172] Through the above operations, the textual case library can be converted into a typical case semantic feature vector library, where each typical case corresponds to a numerical feature vector.

[0173] For the instructional design to be graded, the design elements and content related to the indicators are extracted from the instructional design, and converted into numerical feature vectors (A) using the same method as the semantic vectorization of the case library. Then, the cosine similarity is calculated between the case semantic feature vectors (B) of the same indicators in the case library. The calculation formula is shown in Equation 1:

[0174]

[0175] The result is a real number in the range [-1, 1]. The larger the value, the more similar vectors A and B are.

[0176] For each indicator of the instructional design to be graded, the above calculation process is completed to obtain the three most similar typical cases. The average score of their actual scores is the score of the instructional design for that indicator. The similarity score (S2) results are shown in Table 7:

[0177] Table 7 Similarity Scores

[0178]

[0179] The total similarity score is 54 points.

[0180] (6) Weighted fusion scoring module:

[0181] The final score for the instructional design in this system is obtained by weighting the GENAI direct score (S1) and the case similarity score (S2), as shown in Equation 2:

[0182] s = a·s1 + (1-a)·s2 (Equation 2)

[0183] Where 'a' is unknown, and it is a weighting coefficient, a floating-point number with a value ranging from 0 to 1. <a<1。

[0184] Find the weighting coefficient a:

[0185] The following scores have been received for the 20 lesson plans:

[0186] 1. Expert scoring (s0): Manual scoring by experts;

[0187] 2. GENAI scoring (s1): AI-based automatic scoring based on scales and curriculum standards;

[0188] 3. Similarity score (s2): Scoring based on scales, curriculum standards, and case studies;

[0189] The weighted score s of the GENAI score (s1) and similarity score (s2) can be calculated using the following formula:

[0190] When the value of 'a' is different, different weighted scores 's' for different instructional designs can be calculated. The optimal value of 'a' can be optimized through the following process, thereby achieving:

[0191] The goal is to minimize the total mean square error (MSE) between the weighted score *s* of each instructional design and the expert score *s0* (calculated as an MSE for each design, summed for all 20 designs). The process for optimizing the solution to the optimal weighting coefficient *a* is as follows: Figure 8 As shown.

[0192] Finding the optimal weighting coefficient 'a' is essentially an optimization problem, which can be obtained using optimization algorithms from the scipy library.

[0193] ① Import necessary libraries:

[0194] Import pandas for data preprocessing

[0195] Import scipy.optimize.minimize_scalar for mathematical optimization.

[0196] ② Data preparation:

[0197] The scoring data is obtained by reading three Excel files: s0 is the expert human score, s1 is the AI ​​automatic score based on the scale and curriculum standards, and s2 is the similarity score based on the scale, curriculum standards and case library. Each dataset contains 21 indicators of scores for 20 students.

[0198] ③ Define the objective function:

[0199] Construct the function `calculate_total_mse` to calculate the total mean squared error under a given weight coefficient `a`. This function first calculates the weighted score according to Equation 2, then calculates the mean squared error for each student, and finally returns the sum of the errors for all students.

[0200] ④ Optimize using SciPy:

[0201] This is the core step of the process: calling the `minimize_scalar` function to automatically find the optimal solution. This function uses the bounded method to search within the interval (0.001, 0.999), eliminating the need for manual traversal and significantly improving efficiency and accuracy. The total mean square error of the search process as a function of the weighting coefficient 'a' is shown in the curve below. Figure 9 As shown.

[0202] ⑤ Obtain the optimization results:

[0203] Based on the best search results from scipy, such as Figure 9 As shown, the optimal weighting coefficient a = 0.274 and the minimum total mean square error min_mse = 2.65 are obtained.

[0204] Weighted fusion score:

[0205] For a new instructional design to be graded, the GENAI direct score (S1) is obtained first through the GENAI direct scoring module, and the case similarity score (S2) is obtained through the similarity scoring module. Then, the final score of the instructional design can be obtained by using Equation 2 and the optimal weighting coefficient a.

[0206] The target user in this embodiment can be:

[0207] 1. Primary and secondary school Chinese language teachers: especially inexperienced novice teachers, whose teaching reflections rely on subjective experience and have a vague direction for improvement, and who hope to predict the effectiveness of their teaching designs before public lessons and teaching competitions, need efficient and professional third-party evaluation tools to break through the limitations of personal experience;

[0208] 2. Students enrolled in teacher training colleges: especially those majoring in subjects such as Chinese language education, who need to frequently complete instructional design assignments and teaching practice training, lack instructional design experience, lack scientific assessment tools, and need efficient feedback.

[0209] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A generative artificial intelligence-driven automatic evaluation system for instructional design, characterized in that, include: The evaluation system's knowledge base is used to provide a knowledge foundation and objective criteria for judgment. The evaluation core processing engine is used to score instructional designs based on a multi-source fusion-based intelligent scoring mechanism; The assessment results and feedback system is used to output static scores and generate targeted improvement suggestions by identifying weaknesses, supporting the iterative optimization of instructional design.

2. The generative AI-driven automatic evaluation system for instructional design according to claim 1, characterized in that, The assessment system knowledge base specifically includes: curriculum standards, instructional design evaluation scales, and a database of typical cases; wherein: The curriculum standards are teaching guidance documents used to clarify the core competency requirements, the learning task group system, and the academic quality standards; and the curriculum standards are pre-processed into editable structured documents. The instructional design evaluation scale includes multiple levels of assessment indicators, each of which includes multi-level scoring criteria and operational descriptions, used to evaluate instructional design competence. The typical case library is built based on multiple real teaching designs collected. Each design has expert scores and reasons for the scores in multiple dimensions, covering different levels and score ranges, to provide a reliable reference for similarity scoring.

3. The generative AI-driven automatic evaluation system for instructional design according to claim 2, characterized in that, The core processing engine for evaluation specifically includes: a GENAI direct scoring module, a case similarity scoring module, and a weighted fusion scoring module; wherein: The GENAI direct scoring module uses curriculum standards and instructional design evaluation scales as a knowledge base, adopts the retrieval-enhanced generative framework RAG, and directly scores multiple dimensions of instructional design through generative artificial intelligence GENAI. The case similarity scoring module uses a similarity measurement method to extract core elements from different dimensions of the instructional design to be evaluated, compare them with typical cases in the typical case library, select the cases with the highest similarity, and take the average score as the scoring reference. The weighted fusion scoring module combines the scores from the GENAI direct scoring module and the case similarity scoring module in a weighted manner to obtain the final scoring result.

4. The generative AI-driven automatic evaluation system for instructional design according to claim 3, characterized in that, By adjusting the weighting parameters, the system scores can be made closer to expert scores, thus improving the accuracy of the evaluation.

5. The generative AI-driven automatic evaluation system for instructional design according to claim 3, characterized in that, The evaluation results and feedback system is specifically used for dimension-specific scoring, comprehensive rating, and generation of improvement suggestions; wherein: The scoring is dimensional and itemized, providing clear scores for each dimension of the instructional design evaluation scale, along with the reasons for the scores and suggestions for improvement. The comprehensive rating is based on the total score, providing a multi-level evaluation. Improvement suggestions are generated by combining instructional design content, scoring results, and similar cases, using AI technology to generate targeted improvement suggestions.

6. The generative AI-driven automatic evaluation system for instructional design according to claim 2, characterized in that, The instructional design evaluation scale includes a three-stage argumentation process, in which: The first stage involves literature analysis and preliminary framework construction, resulting in a preliminary evaluation framework that includes multiple multi-level evaluation indicators. The second stage uses the Delphi method to conduct three rounds of expert consultation and indicator optimization. The first round uses an open-ended questionnaire to ask experts to provide suggestions for modifying the preliminary indicator framework. The second round uses a structured questionnaire to ask experts to score the importance of the adjusted indicators and calculate the mean, standard deviation and coefficient of variation of each indicator. The third round feeds back the statistical results of the previous round to the experts and asks them to make a final assessment based on their understanding of the group's opinions. The third phase involved empirical testing and revision.

7. The generative AI-driven automatic evaluation system for instructional design according to claim 6, characterized in that, The instructional design evaluation scale employs a multi-level scoring system that combines grade judgment and detailed scoring. Raters first determine the grade range based on overall performance, then assign precise scores within each grade range based on specific performance. The operational descriptions establish a clear scoring rule system. First, they quantify the scoring standards, transforming abstract expressions into concrete score ranges. Second, they differentiate between different scoring modes, including a score-accumulation type and a grade-judgment type. The score-accumulation type calculates the total score by accumulating scores from multiple elements, while the grade-judgment type directly determines the grade based on overall performance. The operational descriptions also include specific behavioral performance standards and critical judgment conditions. The behavioral performance descriptions use the teacher's design behavior as the basis for judgment, while the critical judgment conditions clearly define the boundaries between different grades.

8. The generative AI-driven automatic evaluation system for instructional design according to claim 2, characterized in that, The specific method for constructing the typical case library is as follows: (1) Data collection: Select real teaching designs with a unified theme as raw data; (2) Expert scoring: Each lesson plan is scored by educational experts, who give specific scores in multiple dimensions and the basis for scoring, forming a basic standard case expert scoring sample library. (3) GENAI-generated cases: GENAI technology is used to generate standardized case descriptions for each instructional design; the generation process adopts the RAG framework, using curriculum standards and evaluation scales as the knowledge base, combined with expert scoring results; the generated case content includes three core elements: the key information that best reflects the characteristics of the indicators is extracted from typical case content; the scoring basis clearly explains the specific reasons for obtaining the corresponding score; common features summarize the common characteristics of cases of the same level; all generated cases have been reviewed by the original scoring experts to ensure the accuracy and representativeness of the content; (4) Case Balance Optimization: Through statistical analysis and optimization algorithms, the cases are balanced to ensure that each indicator retains multiple representative cases at multiple levels, while ensuring that the actual scores are reasonably distributed across each score range.

9. The generative AI-driven automatic evaluation system for instructional design according to claim 3, characterized in that, The case similarity scoring module executes the following process: (1) Text preprocessing, including word segmentation: decomposing the case text into professional terms, phrases and key expressions in the field of education; removing stop words: filtering out common words that have little semantic contribution; Standardization: Converting synonymous expressions into a unified form; (2) Feature extraction: keywords and phrases extracted directly from the text, including vocabulary related to teaching methods, design elements, and quality descriptions; (3) Vectorization mapping assigns weights to each feature, reflecting its importance in the evaluation index; The frequency, importance, and contextual relevance of features are comprehensively mapped into vector values; a multi-dimensional feature space is constructed, with each dimension corresponding to an evaluation element. (4) Customize indicator dimensions: Customize feature dimensions for each of the multiple different indicators; (5) Case library calibration: establish a scoring reference system using the scored cases in the case library, and vectorize the typical cases of each level; form a mapping relationship between indicators, levels and feature vectors; For the instructional design to be graded, extract the relevant design elements and content from the instructional design, and convert them into a numerical feature vector A using the same method as the semantic vectorization of the case library. Then, calculate the cosine similarity with the semantic feature vector B of the case library for the same indicators. The result is a real number in the range [-1, 1]. For each indicator of the instructional design to be scored, complete the above calculation process to obtain the most similar typical cases. The average score of their actual scores is the score of that indicator of the instructional design.

10. The generative AI-driven automatic evaluation system for instructional design according to claim 3, characterized in that, The final score S for the instructional design is obtained by weighting the GENAI direct score S1 and the case similarity score S2, as shown in Equation 2: s = a·s1 + (1-a)·s2; Where 'a' is unknown, and it is a weighting coefficient, a floating-point number with a value ranging from 0 to 1. <a<1。

Images