Classroom teaching quality automatic evaluation method based on behavior recognition and large language model fine tuning

Abstract

The invention provides a classroom teaching quality evaluation method based on student behavior recognition and a large language model, and aims to solve the problems that manual teaching evaluation efficiency is low and the generalization ability of an existing model in a teaching scene is insufficient. The method comprises two key technical modules: firstly, providing a behavior recognition algorithm combining multi-scale feature fusion and local feature enhancement to improve the recognition accuracy of similar behaviors (such as reading and writing), and optimizing a training process by introducing a dynamic weight mechanism; and 2, designing a knowledge oriented migration strategy LoRFA based on dynamic parameter space reconstruction, and injecting a teaching quality evaluation knowledge system into the GLM4 model to realize specialized teaching evaluation text generation under cross-modal input. Experimental results show that on a plurality of classroom behavior recognition data sets, the method is superior to an original YOLOv7 network in mAP 0.5 and mAP 0.5: 0.95 indexes, and the maximum improvement exceeds two percentage points; in a text generation task, the GLM4 subjected to LoRFA fine tuning is superior to LoRA and other mainstream fine tuning algorithms in various indexes of BLEU-4 and ROUGE, the average score is increased to 1.54 at most, and the effectiveness and advancement of the method in classroom teaching quality intelligent evaluation are verified.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and educational informatization, specifically to a classroom teaching quality evaluation method based on student behavior recognition and large language model (GLM), which falls within the research scope of intelligent evaluation systems and methods for teaching quality. Background Technology

[0002] With the release of the "Outline of the Plan for Building a Strong Education Nation (2024-2035)," the digital transformation of education evaluation has become a key direction for improving education quality. Classroom teaching quality evaluation, as an important basis for teaching improvement and educational decision-making, has traditionally relied mainly on teacher self-evaluation, student questionnaires, and expert observation and evaluation. While these methods have some guiding significance in practice, they suffer from problems such as strong subjectivity, low efficiency, and incomplete information coverage, making them difficult to adapt to the current needs of intelligent education development.

[0003] In recent years, information technology and intelligent algorithms have been gradually introduced into the evaluation of classroom teaching quality. Early methods, such as the Flanders Interaction Analysis System, improved evaluation efficiency by recording teacher-student interactions in a structured manner. Subsequent mathematical models, such as fuzzy comprehensive evaluation and multi-level analysis, introduced more quantitative mechanisms. However, these methods mostly focus on scoring structured indicators and have failed to achieve a comprehensive understanding of complex classroom scenarios and automatic text evaluation.

[0004] Advances in deep learning and artificial intelligence technologies have enabled the automated analysis of classroom speech, images, and behavioral data, resulting in greater objectivity and accuracy in teaching quality evaluation. For example, existing research has constructed indicator systems for participation and activity levels using student facial expression recognition, behavior recognition, or speech analysis, serving as supplementary evaluation tools for teaching quality. However, these studies still primarily rely on quantitative scoring and lack comprehensive text evaluation methods that can systematically analyze the teaching process and automatically generate comprehensive text evaluations with human expert linguistic styles.

[0005] Large Language Models (GLMs), with their superior natural language understanding and generation capabilities, offer a new solution for building intelligent classroom teaching evaluation systems. However, general-purpose GLMs suffer from insufficient knowledge adaptation in educational scenarios, failing to accurately output professional texts that meet teaching evaluation standards. Meanwhile, classroom behavior recognition technology exhibits low accuracy in complex teaching environments, limited by issues such as high visual similarity between behaviors and small feature regions, thus affecting the quality of the foundational data for teaching quality evaluation.

[0006] Therefore, there is an urgent need to propose a teaching quality evaluation method that integrates classroom behavior recognition optimization and evaluation knowledge transfer. This method can enhance the accuracy of behavior recognition and inject domain knowledge to adapt to large language models, thereby achieving a comprehensive evaluation of teaching quality that is intelligent, objective, and text-based. Summary of the Invention

[0007] Multi-Scale Feature Fusion (MSFF) networks first extract and reassemble features using Conv3×3 and Conv1×1 convolutions. Conv3×3 is used to extract local contextual information, while Conv1×1 is used for dimensionality reduction and integration, effectively fusing multi-dimensional features, reducing computational complexity, and preserving key features. Secondly, features from different resolutions are concatenated along the feature dimension to achieve feature cascading, forming enhanced feature representations while maintaining the integrity of their respective characteristics. This aligns with the CORE module's idea of ​​unifying features from different layers into a shared feature space, promoting the organic integration and interaction of multi-scale features. Finally, deconvolution (ConvTranspose2d) is used to adjust the spatial resolution of the feature maps, aligning features across layers and meeting the requirement of resolution consistency between feature layers. By amplifying low-resolution features and downsampling high-resolution features, feature maps can interact across scales, achieving feature alignment in feature fusion. The network structures of YOLOv7 and YOLOv7-MSFF are as follows: Figure 1 As shown, YOLOv7-MSFF first applies 1×1 convolutional dimensionality reduction to the feature maps from the middle layers of the backbone network, reducing computation and simplifying feature complexity. For the feature maps from the shallow layers of the backbone network, after 1×1 convolutional dimensionality reduction, further downsampling is performed using 3×3 convolutional operations with a stride of 2, extracting deep features while retaining some feature information from the original resolution. After performing the same operation on the deep feature maps, transposed convolutional upsampling is used to increase the level of detail while maintaining the feature size. Based on this, the resulting feature maps at different scales are concatenated, thus realizing the neck network structure YOLOv7-MSFF based on bidirectional feature fusion for multi-scale feature fusion, which alleviates the feature loss problem during downsampling.

[0008] Local feature enhancement modules based on the aggregation-excitement mechanism (Gather-Excite Enhancement, GEE) such as Figure 2 As shown, the GEE module is divided into two paths: the spatial enhancement path (green background in the figure) and the aggregation-excitement path (blue background in the figure). The aggregation-excitement path structure borrows the idea of ​​Gather-Excite (GE) (Hu et al., 2018), and both paths work together to improve feature representation capabilities. The two paths will be described in detail below. The spatial feature enhancement path first processes the input feature map... The system performs channel grouping, dividing the feature map into multiple sub-feature groups. Each sub-feature group is independently enhanced in subsequent processing steps. Next, global average pooling is applied to each sub-feature group to obtain a vector representing the overall information of that group. This vector is then multiplied by the original feature map, applying weights to each position of each channel in the original feature map. Larger weights enhance the features of their respective channels, while smaller weights suppress less important features, allowing the network to better focus on the importance of different feature groups. These weights are then converted into weights through normalization and activation functions, which are used to weight and adjust each sub-feature, ultimately generating the enhanced feature map. In this paper, these weights are reweighted with the feature map enhanced using the GE module, thereby improving the model's attention to regions of interest, suppressing irrelevant regions, and enhancing the model's key region features in similar behavior recognition.

[0009] The aggregation-excitation path involves two operators: Gather and Excite. The Gather operator is used to collect local features from specific regions within the entire feature map. The selection operator... A region is defined, which is the output location. Centered on, its size is determined by parameters (Expansion ratio) determines:

[0010]

[0011] in:

[0012]

[0013] Gather operator Aggregating spatial information from the input feature map, each output location The information collected comes only from the characteristic responses of a specific set of locations on the same channel:

[0014]

[0015] in, It is an indicator tensor that only emphasizes the selected location. and channels region, and It is the Hadamard product. The receptive field within the input feature map is defined by... definition.

[0016] The Excite operator modulates the original input feature map based on this aggregated contextual information, using a scaling function. This allows us to adjust the values ​​of the input feature map, thereby enhancing the expressive power of the features.

[0017]

[0018] Rescaled function Use the context information obtained through the Gather operator This process generates scaling factors that map to the range [0,1], adaptively modulating the input feature map.

[0019]

[0020] in:

[0021]

[0022] The Excite operator redistributes aggregated information to local features, using a gating mechanism to rescale the original input features based on the aggregation context. By adjusting the input features to be modulated by the aggregation context, the expressive power of feature selection is enhanced. Through the combination of the Gather and Excite operators, Gather-Excite integrates broad-domain contextual information into local features, achieving the goal of enhancing local features.

[0023] The pseudocode for GEE is shown in Algorithm 1, given the input feature map. By group number Will Reorganized into Through global average pooling The original features are then weighted to calculate the channel-normalized features. Then, feature map standardization and activation are performed, and finally... Gated calibration. Meanwhile, the GE path is based on range parameters. Dynamically adjust pooling kernel size The features obtained after pooling The features are then subjected to a nonlinear transformation using an MLP. Finally, the two features are passed through a gating function. Fusion, output enhanced features .

[0024]

[0025] A GEE module, named ELAN-GEE, was inserted after the ELAN module in the YOLOv7 backbone network to enhance the output features of the ELAN module. ELAN-GEE can enhance subtle detail features, improving the network's feature learning ability in student behavior recognition tasks. Experimental results show that, compared to ELAN, ELAN-GEE can improve student behavior recognition capabilities in classroom behavior recognition scenarios.

[0026] III. Propose LoRFA algorithm for fine-tuning ChatGLM

[0027] The LoRFA algorithm fine-tunes large language models through singular value decomposition, parameter matrix importance measurement, and dynamic rank assignment. A diagram illustrating LoRFA fine-tuning of GLM4 is shown below. Figure 3 As shown, the parameter matrix in the GLM block is first frozen. And for each Initialize a bypass increment matrix Then, SVD decomposition is performed on it. Next, during training, the importance of each increment matrix is ​​measured, and an importance score is assigned. The parameter matrix is ​​weighted by the gradient norm and gradient effect coefficients. Finally, low-importance singular values ​​are pruned based on matrix importance, and the rank of the increment matrix is ​​adjusted. This achieves a reasonable allocation of computational resources during fine-tuning while improving the text generation performance of GLM4. For each GLM block in GLM4, the fine-tunable parameter matrices include the QKV projection matrix, the Dense layer matrix, the hidden layer expansion matrix Dense (h to 4h), and the hidden layer compression matrix Dense (4h to h). Low-rank adaptation of these four matrices using the LoRFA method can efficiently adapt to classroom teaching quality evaluation tasks while preserving pre-training knowledge. The following sections will detail the three core steps of the LoRFA algorithm: singular value decomposition, importance measurement, and dynamic rank allocation.

[0028] As shown in Algorithm 3, LoRFA is an algorithm optimized based on LoRA, fine-tuning the model by introducing a low-rank increment matrix into the original weight matrix. LoRFA first initializes an increment matrix as the product of two matrices, and in each training iteration, recalculates these two matrices using singular value decomposition. During this process, LoRFA combines the gradient norm and gradient effect coefficients of the increment matrix to calculate a combined importance score, thereby determining the relative importance of each increment matrix. For matrices with higher importance, their rank is dynamically increased to further enhance their feature extraction capabilities; for matrices with lower importance, their rank is decreased to reduce redundancy.

[0029] Attached Figure Description

[0031] Figure 1 Comparison of YOLOv7 and YOLOv7-MSFF backbone networks with some neck networks;

[0032] Figure 2 Schematic diagram of GEE module;

[0033] Figure 3 LoRFA fine-tuning GLM diagram. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0036] The classroom teaching quality evaluation method based on behavior recognition and GLM of the present invention operates as follows:

[0037] 1. Use fixed video recording equipment to record the entire classroom teaching process. The recorded content should cover most of the student activity areas to ensure the comprehensiveness of subsequent behavior recognition.

[0038] 2. Input the acquired video frames into the behavior recognition system. The behavior recognition system includes two improved versions of YOLOv7: Multi Scale Feature Fusion (MSFF) and Gather-Excite Enhancement (GEE). This yields the distribution data of student behavior throughout the classroom.

[0039] 3. Prompt Text Generation. Populate the data table section of the dynamic prompt template with the behavior distribution data to complete the creation of the dynamic prompt text.

[0040] 4. Input the dynamic prompt word template into the LoRFA-tuned GLM4 model, and use the resulting output text as the final evaluation content for classroom teaching quality.

[0041] 5. Comparative experiments were conducted using the method of this invention on a behavior recognition dataset and a large model fine-tuning dataset, and the mAP and Rouge metrics were recorded respectively;

[0042] The impact of MSFF and GEE on the accuracy of behavior recognition was tested separately, and the experimental results are shown in Tables 1 and 2. As shown in Table 1, the neck network based on multi-scale feature fusion using the method of this invention achieved an improvement in mAP on the dataset. Specifically, both outcome metrics for the "raising hand" behavior showed some improvement, but the improvement in mAP@0.5:0.95 was greater, indicating that the model can better improve the behavior recognition results at medium to high IoU thresholds. Figure 2This demonstrates that MSFF can alleviate the feature loss problem during network downsampling, thereby improving recognition accuracy. Meanwhile, the mAP@0.5 for "writing" and "reading" shows a significant improvement, indicating that MSFF also performs well in preserving fine-grained features of similar behaviors, improving mAP@0.5 by about two percentage points. Overall, the improved multi-scale feature fusion network improves the overall performance of the model under various IoU thresholds through a more effective feature fusion strategy. In Table 2, the introduction of the GEE module improves the detection performance of the "reading" and "writing" actions. mAP@0.5 increases to 70.7 and 73.4 respectively, while the mAP@0.5:0.95 for "writing" also improves, indicating that the GEE module enables the model to better improve the recognition ability of these two similar behaviors. The mAP@0.5:0.95 for "reading" does not show a significant improvement, possibly because GEE's advantage is not obvious in high-threshold detection, but the improvement in mAP@0.5 already demonstrates the effectiveness of GEE. In these two types of highly complex actions, the module improvement effectively enhanced the model's ability to distinguish and recognize actions, verifying the important role of the local feature enhancement module in classroom behavior recognition tasks.

[0043] The impact of LoRFA fine-tuning on the classroom teaching quality evaluation generated by the GLM4 model was tested. The experimental results are shown in Table 3. Both LoRFA and LoRFA were applied to the QKV projection layer of the GLM block, and the weights of the gradient norm and gradient effect coefficient on the importance score were adjusted. , Set them all to 0.5. The initial rank was set to twice the normal value. The experimental results are shown in Table 4-7. With initial ranks of 8 and 16, LoRFA achieved average scores of 31.13 and 31.81, respectively, both higher than LoRA's 30.34 and 31.48. Furthermore, LoRFA also outperformed LoRA in all Rouge metrics and Bleu4. This indicates that LoRFA enhances the model's semantic analysis capabilities and generation quality. (The initial rank in the comparison experiment is not included in the provided text.) When the rank is 8, the computational cost of LoRA is 827,552,081 GF, while that of LoRARF is 827,683,362 GF, which is not significantly different. Meanwhile, LoRFA provides superior text generation performance with similar computational cost. This indicates that LoRFA improves generation quality while maintaining a controllable model size, making it a better choice for classroom teaching quality evaluation tasks. Furthermore, the experimental results in Table 4 demonstrate the advantages of the proposed LoRFA method. LoRFA outperforms some common large-model fine-tuning methods in various metrics. For example, compared to the fixed-rank rsLoRA, LoRFA improves the Rouge-L by 0.61 points, indicating that dynamic rank allocation can enhance the semantic coherence of long texts. Compared to the parameter search method PiSSA, LoRFA does not require iterative optimization of the sub-network structure, resulting in better training stability and generalization. These results validate the effectiveness of the dynamic low-rank allocation strategy in fine-tuning, providing a new technical path for balancing generation quality and computational cost.

[0044] Table 1. Comparison of the impact of the MSFF network of the present invention on mAP with that of the original YOLOv7.

[0045] Table 2. Comparison of the impact of the GEE network of the present invention on mAP with that of the original YOLOv7.

[0046] Table 3 Comparison of the fine-tuning effects of LoRFA and LoRA using the method of the present invention

[0047]

[0048] Table 4 Comparison of the fine-tuning effects of the LoRFA method of the present invention with other LoRA fine-tuning methods

[0049]

Claims

1. An automatic evaluation method for classroom teaching quality based on behavior recognition and fine-tuning of a large language model, characterized in that, Includes the following steps: (1) Obtain image frame sequences from classroom teaching scenarios through video capture, input them into a student behavior recognition model based on multi-scale feature fusion and local feature enhancement, extract student behavior features and identify behavior categories; (2) Construct a multi-scale feature extraction network that integrates shallow spatial features and deep semantic features to alleviate the feature decay problem, and design a local feature enhancement module to strengthen the feature expression of key behavioral regions, thereby improving the recognition accuracy of similar behaviors; (3) An optimized loss function with dynamic weighting mechanism and adaptive scale penalty factor is adopted to improve the overall accuracy of student behavior recognition; (4) Based on the behavior recognition results, construct teaching quality evaluation prompt words and input them into the large language model GLM4 after LoRFA fine-tuning to generate classroom teaching quality evaluation text; (5) In the process of constructing the prompt words, the behavioral category data is quantified and constructed into a dynamic text prompt template according to the ISTE teaching evaluation standard, so as to realize cross-modal mapping from behavioral features to text semantics; (6) During the LoRFA fine-tuning process, the parameter space of GLM4 is adaptively reconstructed by dynamically evaluating the importance score of the incremental matrix in order to achieve the directional transfer of teaching evaluation knowledge. (7) Integrate the student behavior recognition module and the teaching evaluation generation module into an integrated system. After the user inputs the classroom video, the system automatically outputs the teaching quality evaluation text.

2. The automatic evaluation method for classroom teaching quality as described in claim 1, characterized in that: The multi-scale feature extraction network integrates spatial and semantic features at different scales, using shallow convolution to extract spatial information and deep convolution to extract behavioral semantic information.

3. The automatic evaluation method for classroom teaching quality as described in claim 1, characterized in that: The local feature enhancement module focuses on key discrimination regions of similar behaviors and enhances fine-grained behavioral features through an attention mechanism.

4. The automatic evaluation method for classroom teaching quality as described in claim 1, characterized in that: The optimized loss function includes the D-ECIoU loss function, which combines a dynamic weighting mechanism and a target scale adaptive penalty factor to improve the ability to recognize small-scale behaviors.

5. The automatic evaluation method for classroom teaching quality as described in claim 1, characterized in that: The LoRFA fine-tuning algorithm uses a rank dynamic control method to optimize the low-rank structure of the fine-tuning parameters and can be adapted to the multi-layer structure of GLM4.

6. The automatic evaluation method for classroom teaching quality as described in claim 1, characterized in that: The integrated system uses PyQt5 to build a user interface that displays identified student behaviors, behavior time-series charts, and teaching quality evaluation text, supporting visualized and intelligent analysis of the teaching process.

Images