Big language model teaching quality evaluation method based on bloom taxonomy classification system

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a large language model teaching quality evaluation method based on Bloom target classification system, comprising the following steps: constructing a structured prompt word template according to the Bloom target classification system; driving the large language model to cooperatively generate teaching quality evaluation data sets conforming to specified formats and distribution requirements through the prompt word template; a ChatGLM3 pre-training model mainly comprises 28 GLM modules based on a Transformer network, a mixed expert network is used to replace a multi-layer perceptron of part of the GLM modules, a trainable prefix coding is added before each GLM module of the ChatGLM3 pre-training model, a Mamba state space model is added after a word embedding layer and before the GLM module, a teaching quality evaluation large language model is obtained, and the teaching quality evaluation large language model is used for teaching quality evaluation after training. The application can improve the accuracy and pertinence of teaching quality evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and constructs a large language model teaching quality evaluation method based on Bloom target classification system. BACKGROUND

[0002] In the field of higher education, with the growth of the student population, there is a certain contradiction between the limited teaching staff and the growing demand for personalized teaching. Teachers are difficult to fully and timely grasp the learning status and cognitive progress of each student, which leads to the problems encountered by students in the learning process cannot be solved in time, which not only affects the students' knowledge construction and ability improvement, but also restricts the improvement of the overall teaching quality to a certain extent.

[0003] In recent years, as an important technical breakthrough in the field of artificial intelligence, large language models have achieved good results in medical diagnosis, agricultural consultation and other professional scenarios, and have shown their natural language processing and problem diagnosis capabilities. Based on this technical trend, many universities have begun to try to introduce large language models into student learning management and teaching support platforms, and use their efficient information retrieval and generation capabilities to provide instant answers and learning resource recommendations for students, thereby making up for the lack of semantic interaction support in traditional database storage to a certain extent and promoting the improvement of education quality. However, the existing large language models applied in the education field still have the following technical limitations: (1) Single interaction mode. Most of the large language models in the education field currently use a passive response interaction mode, mainly in the form of question and answer, directly returning answers or explanations to user input. This kind of "question and answer" mechanism, although direct in response, weakens the willingness to actively explore and think deeply, which is not conducive to the cultivation of students' high-order cognitive abilities. (2) Insufficient function integration. The existing large language models still have obvious limitations in education applications. Most systems lack the ability to call and integrate external tools such as computing platforms, databases, and academic resource libraries, and do not support situational or continuous learning guidance, which limits their ability to respond to complex, dynamic, and interdisciplinary actual teaching needs. The response of the model is usually limited to its internalized knowledge structure, making it difficult to achieve adaptive teaching guidance. (3) Lack of scientific evaluation system. The existing technical solutions generally lack a multi-level evaluation mechanism for students' cognitive level, making it difficult to accurately identify students' mastery at different cognitive levels, and thus unable to provide teaching support that matches the actual cognitive development stage of students.

[0004] Therefore, it is necessary to further explore large language model recognition evaluation methods that integrate education theory, promote the transformation of large language models from single "answer providers" to "ability improvers", and establish a scientific cognitive ability evaluation system to improve the accuracy and relevance of teaching quality evaluation through more reasonable teaching design and more effective interaction strategies. SUMMARY

[0005] The purpose of the present application is to provide a large language model teaching quality evaluation method based on Bloom's target classification system, to solve the problem of insufficient automation of teaching quality evaluation and lack of multi-level cognitive ability evaluation mechanism in the prior art. The present application is based on a large language model and evaluates teaching quality according to evaluation questions and student answers. Specifically, the present application first generates student answer evaluation data sets that meet the requirements of specified cognitive level format and question type by driving a large language model with a structured question prompt word template; second, it uses parameter fine-tuning technology to further improve the balance and overall performance of the large language model in different cognitive level evaluation tasks through strategies such as separate training, instruction fine-tuning, and mixed prompt setting; at the same time, the multi-layer perceptron (MLP) of the 20-22 layer GLM module in the ChatGLM3 pre-training model is replaced by a mixed expert network (MoE), and a Mamba state space model is added after the word embedding layer and before the GLM module in the ChatGLM3 pre-training model, to reduce the consumption of computing resources, maintain the performance of the model, effectively prevent overfitting, and improve the generalization ability of the model.

[0006] The present application is realized by the following technical solutions. A large language model teaching quality evaluation method based on Bloom's target classification system, comprising the following steps:

[0007] Step one: construct a structured prompt word template according to Bloom's target classification system;

[0008] Step two: generate a teaching quality evaluation data set that meets the specified format and distribution requirements by driving a large language model with a prompt word template, the teaching quality evaluation data set containing questions that examine different Bloom's targets and corresponding student answer samples with diversity;

[0009] Step three: based on the ChatGLM3 pre-training model, optimize its architecture to obtain a teaching quality evaluation large language model, including:

[0010] Add a trainable prefix encoding before each GLM module in the ChatGLM3 pre-training model;

[0011] Add a Mamba state space model after the word embedding layer and before the GLM module;

[0012] Replace the multi-layer perceptron (MLP) of the GLM module in some layers of the ChatGLM3 pre-training model with a mixed expert network (MoE);

[0013] Step four: train the teaching quality evaluation large language model with the teaching quality evaluation data set to obtain the final teaching quality evaluation large language model for teaching quality evaluation.

[0014] Further preferably, the processing procedure of the Mamba state space model comprises:

[0015] The word embedding vector is mapped to a target dimension through linear projection to obtain a projected input vector; the projected input vector is divided into two branches: a main branch and a gating branch;

[0016] In the main branch, the projected input vector is processed through an activation function to obtain a main branch feature vector;

[0017] In the gating branch, the projected input vector is sequentially processed through a convolution layer, a selective state space model, and an activation function to obtain a gating branch feature vector;

[0018] The main branch feature vector and the gating branch feature vector are fused to obtain a fused feature vector;

[0019] The fused feature vector is output to a subsequent GLM module after linear projection and dimension compression combined with position encoding and prefix encoding.

[0020] Further preferably, in the gating branch:

[0021] The local context features of the projected input vector are extracted through the convolution layer;

[0022] The local context features are input into four different linear transformation layers to simultaneously generate four dynamic parameters required by the selective state space model, including: a dynamic state transition matrix, a dynamic input matrix, a dynamic output matrix, and a dynamic direct path vector;

[0023] The selective state space model uses the four dynamic parameters, the local context features, and the hidden state of the previous time step to perform state updating and output calculation to obtain the final output of the selective state space model, which is processed through an activation function to obtain the gating branch feature vector.

[0024] Further preferably, the state updating procedure of the selective state space model is as follows:

[0025] The state of the previous time step is attenuated according to the dynamic state transition matrix of the current time step to obtain historical retention information;

[0026] The current input features after gated nonlinear transformation are weighted according to the dynamic input matrix of the current time step to obtain new information injection;

[0027] The historical retention information and the new information injection are added to obtain the state of the current time step.

[0028] ​Further preferably, the processing procedure of the mixed expert network comprises:

[0029] inputting the input feature into the feedforward network to generate the gating weight of each expert network;

[0030] selecting the top-k expert networks according to the gating weight and distributing the input feature to the selected expert networks according to the weight;

[0031] processing the input feature independently by each expert network to obtain the expert network output;

[0032] weighting and summing the expert network outputs according to the gating weight to obtain the final output of the mixed expert network.

[0033] Further preferably, the process of generating the gating weight by the feedforward network comprises:

[0034] introducing a noise term when calculating the gating weight;

[0035] adopting a Top-K sparse gating mechanism to activate only the top-k expert networks in the gating weight ranking.

[0036] Further preferably, the training adopts a P-Tuning v2 fine-tuning technology to freeze the weights of other parts except the Mamba state space model and the mixed expert network, and inserts a trainable prefix code before each layer of the GLM module, combines a separate training, instruction fine-tuning and mixed prompt setting strategy to improve the balance and overall performance of the teaching quality evaluation large language model on different cognitive level evaluation tasks.

[0037] Further preferably, the prompt word template is constructed based on the six cognitive levels of Bloom's Taxonomy, and the six cognitive levels include memory, understanding, application, analysis, evaluation and creation.

[0038] The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the teaching quality evaluation method when executing the program.

[0039] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the teaching quality evaluation method.

[0040] The application has the following advantages:

[0041] 1. The generated teaching quality evaluation dataset covers different question types, different Bloom cognitive objectives, and different quality levels of student answer samples, and each piece of data includes "scoring reasons" and "improvement suggestions", which can simulate the evaluation thinking process of teachers, suitable for automatic scoring model training and teaching feedback generation research, and can effectively alleviate the problem of difficulty in comprehensive evaluation of student learning quality due to limited teacher resources.

[0042] 2. High efficiency in processing long text and low resource consumption. By integrating the Mamba state space model in the ChatGLM3 pre-training model, the inference efficiency is optimized, the computational resource consumption is reduced while maintaining the model performance, further improving the performance of the model in long text processing tasks, while maintaining its ability in general dialogue tasks, enhancing the ability of the model to process long context.

[0043] 3. Large model capacity and high computational efficiency. By replacing the multi-layer perceptron (MLP) in the 20-22 layer GLM module of the ChatGLM3 pre-training model with a hybrid expert network layer (MoE), the model can dynamically allocate computing resources based on input content, significantly improving model capacity and expression ability without increasing computational load. Through the multi-expert network mechanism, knowledge is divided and stored in a fine-grained manner, making the model have faster convergence speed compared to dense models with the same computational load; at the same time, larger parameter capacity helps to enhance the knowledge storage and expression ability of the model.

[0044] 4. High efficiency of fine-tuning and strong cognitive guidance ability. By using a parameter-efficient fine-tuning method, freezing the ChatGLM3 pre-training model weights and inserting trainable prefix encodings before each GLM module, combined with strategies such as separate training, instruction fine-tuning, and mixed prompt settings, the model's balance and overall performance on different cognitive level evaluation tasks are further improved; while retaining the original functions of the ChatGLM3 pre-training model, it can evaluate the learning status of students with different learning quality, overcoming the limitations of traditional dialogue models that directly provide answers without effectively guiding students to think deeply, and helping to promote the cultivation of students' high-order cognitive abilities. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a large language model teaching quality evaluation method flowchart based on the Bloom objective classification system of the present application;

[0046] Figure 2 is a teaching quality evaluation large language model architecture diagram of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below in conjunction with the drawings and examples.

[0048] Reference Figure 1 A large language model teaching quality evaluation method based on Bloom's taxonomy classification system, the steps are as follows:

[0049] Step 1: Construct a structured prompt word template according to Bloom's taxonomy classification system;

[0050] Step 2: Drive the large language model to generate teaching quality evaluation data set in accordance with the specified format and distribution requirements through the prompt word template, the teaching quality evaluation data set contains questions and corresponding student answer samples with diversity for investigating different Bloom's goals;

[0051] Step 3: Based on ChatGLM3 pre-training model, optimize its architecture to get teaching quality evaluation large language model, including:

[0052] (1) Add trainable prefix encoding before each layer GLM module of ChatGLM3 pre-training model to realize efficient parameter fine-tuning;

[0053] (2) Add Mamba state space model after the word embedding layer and before the GLM module to optimize inference efficiency, reduce computational complexity and memory consumption in long text processing, and expand the supported context length;

[0054] (3) Replace the multi-layer perceptron (MLP) in the 20-22 layer GLM module of ChatGLM3 pre-training model with a hybrid expert network (MoE) to increase model capacity without significantly increasing actual computational complexity; Through the gating network to realize the dynamic distribution of input and the competition mechanism between expert network modules, enhance the sparsity and controllability of the model, effectively prevent overfitting, and improve the generalization ability;

[0055] Step 4: Train the teaching quality evaluation large language model using the teaching quality evaluation data set to get the final teaching quality evaluation large language model for teaching quality evaluation.

[0056] In this embodiment, according to the six cognitive levels (memory, understanding, application, analysis, evaluation, and creation) of Bloom's taxonomy classification system and actual evaluation requirements, a structured prompt word template is constructed, and the data fields, question types, question frequencies, and scoring standards of each prompt word template are clearly specified.

[0057] In order to alleviate the teaching task of teachers, a structured prompt word template is input into different large language models, and the large language models are driven by the prompt word template to cooperatively generate a teaching quality evaluation data set conforming to a specified format and distribution requirement; the teaching quality evaluation data set contains questions investigating different Bloom cognitive objectives and corresponding student answer samples with diversity, and then the teaching quality evaluation data set is divided into a training set, a verification set and a test set for training of a teaching quality evaluation large language model.

[0058] Figure 2 The teaching quality evaluation large language model architecture of the application is shown, mainly composed of a trainable prompt layer, and a ChatGLM3 pre-training model with a newly added Mamba state space model and a replaced mixed expert network (MoE).

[0059] Because the calculation complexity of the self-attention layer (Self-Attention) in the ChatGLM3 pre-training model is O(n 2 ), For sequence length, this means that when the sequence length increases, the model computation and memory consumption are both quadratic growth, which will seriously limit the model's ability to handle long context, so in order to improve the original model's linear computational complexity advantage in processing long sequences, and to be able to maintain or even improve the performance of the model for a long time while significantly reducing the computational and memory overhead in the long sequence scenario, the present application introduces the Mamba state space model (State Space Model), which has a linear time complexity of sequence change, meaning that when the sequence length increases, the computation grows linearly, so it can extremely efficiently handle extremely long sequences, which is a significant advantage over traditional Transformers; Therefore, the teaching quality evaluation large language model of the present application adds a layer of Mamba state space model before the input to the GLM module based on the original ChatGLM3 pre-training model, so when faced with long sequence processing, the Mamba state space model can first efficiently encode the sequence and capture long-distance dependencies, and output a refined sequence representation, which is then input into the subsequent GLM module, so that the subsequent self-attention layer can work on a more compact and rich representation, greatly reducing the overhead spent on self-attention. Instead of replacing the self-attention layer in the ChatGLM3 pre-training model with the Mamba state space model, adding it is a more conservative and robust strategy, although module replacement has great potential, but the risk is also relatively higher, and when the Mamba state space model does not perform as expected on some tasks, the retained self-attention layer can still serve as a powerful guarantee to keep the model's performance lower limit not too bad, and for limited resources, it can achieve a reduction in computational resource consumption while maintaining or even improving model performance, combining the advantages of the two paradigms to achieve a 1+1>2 effect.

[0060] First, the input is a mixture of course materials, questions and answers, first data cleaning, then the resulting vector is input into the word embedding layer, after coming out of the word embedding layer, the word embedding vector will first go through a linear projection to map its original dimension to the target dimension of the Mamba state space model, then the amplified dimension input is divided into two branches, one is the main data channel, and the other is the gating branch, which is used to determine what the final output is. This process can be represented by formula (1):

[0061] (1);

[0062] Where, represents the word embedding vector, and represents the feature vector obtained by splicing and fusing the course materials, questions and student answers; represents the input projection weight matrix of the teaching quality evaluation task; Indicates input bias; This represents the input vector of the projected Mamba state-space model.

[0063] Then, in the gated branch, The convolutional layers extract local contextual features, and through the local connectivity and weight sharing characteristics of the convolutional layers, smooth and context-rich continuous segments are transformed, providing a more convenient input for subsequent selective state-space models. This complements the long sequence modeling capabilities of selective state-space models, enhancing the efficiency and representational ability of large language models for teaching quality assessment on sequence data. The process through the convolutional layer can be represented by formula (2):

[0064] (when (2)

[0065] if If so, then the value of this item is 0 (meaning that future time steps are not visited).

[0066] in, Indicates the kernel size, controlling the range of local dependencies. The larger the value, the more past time steps the current output can see; Indicates the current time step The local context features output by the convolutional layer, after being expanded in dimension by an expansion factor, are simultaneously fed into the state update gate and the output gate. This represents the convolution kernel weights; each row represents a convolution kernel with one channel and a length of [length value missing]. Its function is to learn how to extract the local temporal features of the channel. The overall effect of this process is to provide local information of the most recent time steps to help the gated branch decide to update the state and output, thereby ensuring that the model will not use the information of future words in future tasks. i is the index within the convolution window, used to traverse and accumulate the input in the past period that the current time step t depends on.

[0067] Next, we arrive at the core innovation of the Mamba state-space model, which distinguishes it from the traditional state-space model SSM. The key parameters of the state-space model dynamically change with the input, unlike the traditional SSM which uses fixed parameters. This invention dynamically generates the four core dynamic parameters required for the selective state-space model through a linear projection mechanism, including the dynamic state transition matrix, dynamic input matrix, dynamic output matrix, and dynamic direct path vector. This process can be expressed by formula (3):

[0068] (3);

[0069] The left four parameters come from the core matrix of the traditional state space model (SSM), but a dynamically changing output is realized in this model, denotes the dynamic state transition matrix, also called the decay gate, which controls the state of the previous time step to affect the state of the current time step Similar to the memory decay rate, it is ensured to be positive by the activation function, to highlight the current new information, and the subsequent participation in the calculation of the decay degree of the historical state, to determine how much historical information is retained; denotes the dynamic input matrix, also called the input gate, which controls how the current input features are injected into the new state Similar to the weight of the current information, it reduces the interference of other information on the state, and determines how much proportion of the current candidate state will be included in the new state; denotes the dynamic output matrix, also called the candidate state gate, which can control the historical information in the state of the current time step how to be extracted into the output Similar to reading information from memory, it is the core information source of the new state; denotes the dynamic direct path vector, also called the output gate, which can determine how much state information is finally output to the next layer; , , and are the linear transformation weight matrices of the decay gate, the input gate, the candidate state gate and the output gate, respectively, which are used to map the dimensions of to the dimensions of the gating vector, and convert the local temporal features captured by convolution into decision signals for each gate through learning, which are used to affect the output of each gate; , , and are the bias terms of the decay gate, the input gate, the candidate state gate and the output gate, respectively, which are used to fine-tune the linear transformation results; The whole process is to provide decision basis for subsequent selective scanning, and through the cooperative work of different gates, it realizes selective forgetting of historical state, selective integration of current information, and selective output of state, so that the model can accurately and efficiently process long sequence tasks.

[0070] Then the local context features extracted by convolution are projected into a high-dimensional space, and selective switches are used to remember key feature information and forget irrelevant noise, finally extracting features with long temporal dependency, which can be represented by formula (4):

[0071] ,

[0072] ,

[0073] (4);

[0074] where, is the dynamic state transition matrix generated from formula (3), which determines how fast the information at the current time is updated, represents the system evolution matrix, which is responsible for capturing long-term memory, represents the discretized state transition matrix, which represents how much historical information is retained to the present, represents the discretized input matrix, which represents how much new information at the current time is retained in the state, represents the hidden state at the current time step, which not only contains the information at the current time step t, but also compresses the previous historical information, represents the hidden state at the previous time step, the selective state space model selectively forgets old memories through and absorbs new information through to achieve the effect of long sequence modeling, ensuring the stability of training, represents the final output of the selective state space model.

[0075] Then comes the last link of the gating branch, which further introduces nonlinearity to the final output of the selective state space model through the SiLu activation function, which can be represented by formula (5):

[0076] (5);

[0077] where, and represent the two linear transformation weight matrices of the gate, can map to the "candidate feature" space to retain the potential effective information of the input; can map to the "gate signal" space, which is equivalent to a switch for filtering information; , are two bias terms of linear transformation, which are used to fine-tune the distribution of candidate features and gate signals, so as to avoid the transformation being limited in the space through the origin, increasing flexibility; represents activation function, which limits the range of output to 0~1, and its role is to convert into a gating mask, 1 means retaining the feature, and 0 means suppressing the feature; represents the feature vector of the gating branch, which only retains the local features that are more important to the current task.

[0078] In the main branch, the projected input vector is passed through The activation function processing obtains a main branch feature vector; and the main branch feature vector and the gating branch feature vector are fused in feature, and this process can be represented by formula (6):

[0079] (6);

[0080] wherein, represents In the main branch, the main branch feature vector obtained through the activation function processing, The fusion feature vector output by the Mamba state space model contains both long-term dependencies and local features. The fusion feature vector output by the Mamba state space model contains both long-term dependencies and local features. The fusion feature vector output by the Mamba state space model contains both long-term dependencies and local features.

[0081] Finally, the fusion feature vector output by the Mamba state space model is linearly projected and dimensionally compressed, so that it can be standardized and adapted to the downstream task, and this process can be represented by formula (7):

[0082] (7);

[0083] wherein, represents the feature vector after linear projection and dimensional compression; , represents two parallel linear transformation matrices, which map the fusion feature vector output by the Mamba state space model to two different subspaces, and then element-wise multiplication is performed to enhance the nonlinear expression ability, is the weight of the global gating branch, is a learnable bias vector of the global gating branch, and these are learnable parameters, is the final projection matrix, which maps the compressed feature dimension to the output dimension, adjusts the distribution of the compressed feature, and makes the output more adaptive to the subsequent GLM module. Then is spliced with the position encoding and the prefix encoding to obtain a new vector, and this new vector is input into the normalization layer of the GLM module for distribution normalization, so that the subsequent self-attention layer can more efficiently capture the correlation between features. When normalized, it enters the self-attention layer to model the global and local correlations of different position features in the sequence, and the Mamba state space model forms a complementary ability to enhance the expression ability of the features.

[0084] Another optimization point of the application is to replace the multi-layer perceptron (MLP) in the 20th to 22nd layers of the 28-layer GLM module in the ChatGLM3 pre-training model with a mixed expert network (MoE). Compared with replacing in the later or in the first few layers, this position has better stability during training, and it is a flexible replacement way to modify, and will not greatly change the original ChatGLM3 pre-training model. Because the traditional ChatGLM3 pre-training model uses all MLP modules, but the MLP module is a dense network, all parameters of the entire MLP will be activated and used for each input token, which will result in a very large model calculation, thereby affecting the efficiency of the model. The core idea of the mixed expert network (MoE) is to split a large dense network (MLP) into multiple smaller expert networks with different functions. At the same time, a gating network (Gating Network) or router (Router) is introduced. For each input token, the gating network will determine which few most relevant expert networks to process, and other expert networks will remain in a dormant state. Therefore, the replacement can achieve a reduction in calculation cost without increasing the calculation cost, but can greatly increase the model parameters and knowledge capacity, thereby achieving an optimized effect on the performance of the model. Moreover, the application adopts the mixed expert network (MoE), which aims to activate only a selected subset of expert networks at each forward pass, and achieve sparsity by computing a weighted sum of the outputs of the expert networks, rather than aggregating all expert network outputs together. When data passes through the mixed expert network (MoE), it is first preprocessed to ensure that the input shape and data type are compatible with the gating and expert networks, so that the mixed expert network (MoE) supports different formats of input. Then, the routing module calculates the gating scores and score probabilities for each input feature, dynamically selects the appropriate expert network and assigns the fusion weight, and the weight of the more important expert network is higher. This process can be represented by formulas (8), (9) and (10):

[0085] (8);

[0086] (9);

[0087] Condition: if belongs to the first k elements, (10);

[0088] wherein, represents a sparse gating function, represents the ​​Gating weights of an expert network, This represents the input data. The parameters representing the gating network, This indicates selecting the top k elements with the largest gating weights, and then... Normalize the weights into a probability distribution Representing noise terms and increasing randomness, this formula implements a sparse activation mechanism, selecting only the most important terms. A network of experts is used to process the current input; formula (9) is... The condition is defined such that when the condition satisfies formula (10), the original gating weights are retained. When the condition is met, it is set to -∞, because In the activation function, -∞ is replaced with 0 to achieve "hard" sparsity. This defines the conditions under which expert networks are activated; only those with the highest gating weights are activated. The expert network will be preserved, generally by activating only... To improve computational efficiency, one expert network, rather than all expert networks, can be used. In addition, increasing the number of expert networks can improve model scalability without significantly increasing the computational load. Different expert networks can be used to process data of different patterns, and noise can be added to avoid overloading some expert networks.

[0089] Then we move on to the expert network processing, which includes parallel computing and sparse activation. The core mechanism is that each expert network only processes the gated inputs, not all inputs, which can significantly reduce the amount of computation. At the same time, the parallel processing also means that each expert network runs independently and can be computed in parallel on the hardware. This process can be expressed by formula (11):

[0090] (11);

[0091] in, Indicates the first The network of experts has an internal three-layer structure of dimensionality increase, activation, and dimensionality reduction. Represents the input features of the expert network; Indicates the first The weights of the expert networks during the merging process are derived from the gating network. probability; This represents the final output vector after the hybrid expert network fusion, with dimensions equal to or greater than the input. Consistency facilitates subsequent residual connections. In summary, this module implements the logic of parallel expert network division of labor, dynamic selection, and weighted fusion, which can further improve the model's ability to express complex features.

[0092] In step four, most weights are frozen by using P-Tuning v2 fine-tuning technology, and then separated training, instruction fine-tuning and mixed prompt setting strategies are used. P-Tuning v2 fine-tuning technology is a parameter efficient fine-tuning method, and the core idea is to freeze all parameters of the pre-trained model, and insert some additional trainable prefix codes in each layer of the Transformer model, and train these few parameters to guide the model to adapt to the downstream task, so that its performance approaches full fine-tuning. Because all parameters are frozen, catastrophic forgetting during model training can be avoided, and the good performance of the original model is preserved. However, because the teaching quality evaluation large language model of the application adds a Mamba state space model and replaces the MLP module with a mixed expert network (MoE), these two modules will participate in training and the parameters are not frozen, so the training of these two modules can be successfully completed.

[0093] To further verify the performance of the teaching quality evaluation large language model constructed by the present application, a comparative experiment is designed to compare the large model constructed by the present application with other three large language models in five automatic evaluation indexes for evaluating text. The comparative models include Jamba1.5, which is a model based on Transformer containing a mixed expert network MoE and a Mamba state space model similar in architecture to the teaching quality evaluation large language model of the present embodiment. ChatGLM3-6b is the ChatGLM3 pre-training model used in the present embodiment, which is a model composed entirely of Transformer. The last comparative model is Llama3:8B, which is also a standard Transformer pre-training model, and its parameter quantity is very close to that of the teaching quality evaluation large language model of the present embodiment, so it is also used for comparative experiment. EduAssessLM is the teaching quality evaluation large language model obtained in the present embodiment. The format accuracy is used to measure whether the generated text meets the preset format specification. It mainly focuses on the correctness of the generated results in terms of structure, separator, field arrangement, etc. on the surface form, which is the basic requirement of structured output tasks (such as information extraction, code generation, table filling), because the evaluation result needs to be structured output, so this index is essential. Bloom scoring accuracy is usually used to evaluate the accuracy of the generated content at the cognitive level, i.e. whether it meets the requirements of a certain level in Bloom's taxonomy (such as "memory", "understanding", "application", etc.). This index focuses on whether the generated text correctly answers the question or completes the task at the expected cognitive level, and the accuracy of the score is also an index to measure the effect of the model, which plays an important role in subsequent tasks. Content correctness accuracy evaluates the correctness of the generated text in terms of factual accuracy, logical accuracy or task execution. It not only focuses on the surface form, but also on the accuracy of the content itself, which is one of the core indicators to measure the quality of generation. Field completeness measures whether the generated text covers all necessary fields or information points. It is commonly used for tasks such as filling out forms, generating reports, and completing templates, and reflects the completeness and information coverage of the generated content. The suggestion completeness rate reflects whether the generated teaching suggestions meet the requirements of length and format. The above indexes are higher, the better the performance, and the data of the present application is bolded for better comparison.

[0094] Table 1 Comparison of large model evaluation effect

[0095]

[0096] From the data in Table 1, it can be seen that the large model constructed by the present application has a relatively stable effect in the background and foreground, and in the comparison of the above five indexes, each index has outstanding performance compared with other large models.

[0097] Another embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the teaching quality evaluation method when executing the program.

[0098] Another embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the program is executed by a processor to implement the teaching quality evaluation method.

[0099] The above-mentioned embodiments of the present application only express the implementation manners of the present application, and thus cannot be understood as the limitation of the patent scope of the present application, nor any form of limitation on the structure of the embodiments of the present application. It should be pointed out that, for those skilled in the art, some changes and improvements can be made without departing from the concept of the embodiments of the present application, and these changes and improvements all belong to the protection scope of the embodiments of the present application.

Claims

1. A method for evaluating the teaching quality of a large language model based on Bloom's Taxonomy of Objectives, characterized in that, Includes the following steps: Step 1: Construct a structured prompt word template based on Bloom's Taxonomy; Step 2: Drive the large language model through prompt word templates to collaboratively generate a teaching quality assessment dataset that meets the specified format and distribution requirements. The teaching quality assessment dataset contains questions that examine different Bloom objectives and corresponding, diverse student answer samples. Step 3: Based on the ChatGLM3 pre-trained model, optimize its architecture to obtain a large language model for teaching quality assessment, including: Add trainable prefix codes before each GLM module in the ChatGLM3 pre-trained model; Add the Mamba state space model after the word embedding layer and before the GLM module; Replace the multilayer perceptron of some layers of the GLM module in the ChatGLM3 pre-trained model with a hybrid expert network; Step 4: Use the teaching quality assessment dataset to train the large language model for teaching quality assessment, and obtain the final large language model for teaching quality assessment, which will then be used for teaching quality assessment. The processing procedure for the Mamba state-space model includes: The word embedding vector is mapped to the target dimension through linear projection to obtain the projected input vector; the projected input vector is divided into two branches: the main branch and the gated branch. In the main branch, the projected input vector is processed by an activation function to obtain the main branch feature vector; In the gated branch, the projected input vector is processed sequentially through a convolutional layer, a selective state-space model, and an activation function to obtain the gated branch feature vector; The feature vectors of the main branch and the gated branch are fused to obtain the fused feature vector. The fused feature vector is subjected to linear projection and dimensionality compression, combined with positional encoding and prefix encoding, and then output to the subsequent GLM module. In the gated branch: Local contextual features of the projected input vector are extracted through convolutional layers; The local context features are input into four independent linear transformation layers, which simultaneously generate the four dynamic parameters required for the selective state-space model, including: dynamic state transition matrix, dynamic input matrix, dynamic output matrix, and dynamic direct path vector. The selective state-space model utilizes four dynamic parameters, local context features, and the hidden state from the previous time step to perform state updates and output computation, yielding the final output of the selective state-space model. The final output of the selective state-space model is then processed... The activation function is used to process the gated branch feature vectors.

2. The teaching quality assessment method according to claim 1, characterized in that, The selective state space model utilizes the dynamic state transition matrix and dynamic input matrix, combined with the hidden state of the previous time step and the local context features of the current time step, to perform selective state updates; and uses the dynamic output matrix and dynamic direct path vector to calculate the final output of the selective state space model based on the updated hidden state of the current time step.

3. The teaching quality assessment method according to claim 1, characterized in that, The processing steps of the hybrid expert network include: The input features are fed into the feedforward network to generate the gating weights for each expert network; The top k expert networks are selected based on the gating weights, and the input features are assigned to the selected expert networks according to the weights. Each expert network processes the input features independently, and the output of the expert network is obtained. The outputs of each expert network are weighted and summed according to the gating weights to obtain the final output of the hybrid expert network.

4. The teaching quality assessment method according to claim 3, characterized in that, The process of generating gated weights by the feedforward network includes: Introduce a noise term when calculating the gating weights; A Top-K sparse gating mechanism is adopted, which only activates the expert network with the top k gating weights.

5. The teaching quality assessment method according to claim 1, characterized in that, The training employs P-Tuning v2 fine-tuning technology to freeze the weights of all parts except the Mamba state-space model and the hybrid expert network, and inserts trainable prefix codes before each GLM module. This, combined with separate training, instruction fine-tuning, and hybrid prompting strategies, aims to improve the balance and overall performance of the large language model in different cognitive level assessment tasks for teaching quality evaluation.

6. The teaching quality assessment method according to claim 1, characterized in that, The prompt word template is constructed based on the six cognitive levels of Bloom's Taxonomy of Objectives, which include memory, comprehension, application, analysis, evaluation, and creation.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the teaching quality assessment method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the teaching quality assessment method as described in any one of claims 1-5.

Images