Classroom teaching quality auxiliary evaluation method, equipment and system based on cloud-edge collaboration

Abstract

The present invention discloses a classroom teaching quality auxiliary evaluation method, device and system based on cloud-edge collaboration, the method steps include: acquiring video image data of students in the classroom collected by a camera in real time, and transmitting it to an edge device; in the edge device, using a pre-trained student classroom behavior recognition model to identify and track the received video image data, using the recognition results of multiple adjacent frames and the recognition results of the current frame to fuse to obtain the final recognition result of the current frame during the tracking process, evaluating the classroom teaching quality according to the student classroom behavior recognition results, the student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and using a knowledge distillation structure; sending the student classroom behavior recognition results and evaluation results to a cloud video server for storage. The present invention has the advantages of simple implementation method, low cost, real-time performance, high accuracy, and strong robustness.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment technology for classroom teaching quality, and in particular to a cloud-edge collaborative auxiliary assessment method, device and system for classroom teaching quality. Background Technology

[0002] Currently, classroom teaching quality is primarily assessed through manual evaluation methods such as classroom observation and subject tests. While these methods can reflect educational quality to some extent, they suffer from high implementation costs, low frequency, and difficulty in effectively promoting improvements. Intelligent classroom teaching quality assessment utilizes artificial intelligence technologies such as machine learning, natural language processing, and computer vision. By extracting features from teachers' or students' speech and images, it automatically assesses and analyzes classroom teaching quality, significantly improving assessment efficiency, reducing manual costs, and enhancing accuracy.

[0003] For intelligent assessment of classroom teaching quality, existing technologies typically employ a server-based model deployment approach. A single processor runs a machine learning model, receiving real-time audio or image data collected from teachers or students. The machine learning model then identifies the teacher's or student's state to assess teaching quality. However, this method suffers from several drawbacks. Since all data processing is concentrated on a single processor, the server needs to handle a large amount of image data, increasing computational and storage pressure. This places high demands on the processor's storage capacity and data processing capabilities, leading to high overall hardware deployment costs. Furthermore, the need to transmit video data captured by cameras to the server in real-time via the network makes the system highly dependent on network stability, reducing its resistance to interference. Summary of the Invention

[0004] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a cloud-edge collaborative auxiliary evaluation method, device and system for classroom teaching quality that is simple to implement, low in cost, real-time and highly accurate, and robust.

[0005] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0006] A cloud-edge collaborative method for assisting in the evaluation of classroom teaching quality includes the following steps:

[0007] Edge devices receive video image data of students in the classroom in real time from cameras;

[0008] In the edge device, a pre-trained student classroom behavior recognition model is used to recognize student classroom behavior on the real-time received video image data, and each detected detection box is tracked. During the tracking process, the recognition results of neighboring frames are fused with the recognition results of the current frame to obtain the final recognition result of the current frame. Evaluation indicators are calculated based on the student classroom behavior recognition results to evaluate the quality of classroom teaching. The student classroom behavior recognition results and evaluation results are sent to the cloud video server for storage.

[0009] The student classroom behavior recognition model was pre-trained based on a multi-label target detection algorithm and a knowledge distillation structure.

[0010] Furthermore, the student classroom behavior recognition model is built based on YOLOv (You Only Look Once) 8, and the convolution operation in the YOLOv8 Backbone uses the RFAConv (Receptive Field Attention Convolution) operator. The expression of the RFAConv operator is as follows:

[0011]

[0012] in, Represents the RFAConv operator. This represents a grouped convolution of size 1×1. This represents a k×k grouped dilated convolution, where k represents the kernel size and Norm represents normalization. X Represents the input feature map, This represents the activation function. This represents the Softmax function. This represents soft pooling, and the calculation formula is:

[0013]

[0014] in, R The selected area This represents the result after soft pooling. , , Representing regions R The Middle k, j The feature values ​​at each position.

[0015] Furthermore, in using the student classroom behavior recognition model to identify student classroom behavior, the classification function adopts the following class weight focus loss function:

[0016]

[0017] in, Let be the weighting factor for the classroom behavior of student type i. , m represents the category of student classroom behavior, Let be the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset. Let be the predicted value of the classroom behavior of student type i. For the Sigmoid function, Let be the true label value of the classroom behavior of student type i. As a balance factor, This is the weighting factor.

[0018] Furthermore, the steps for training the student classroom behavior recognition model using a knowledge distillation structure include:

[0019] Configure the first pre-trained weights;

[0020] The student classroom behavior recognition model is trained using the first pre-trained weights to obtain the second pre-trained weights.

[0021] The second pre-trained weights are used to fine-tune the student classroom behavior recognition model using a knowledge distillation structure to obtain the third pre-trained weights.

[0022] The third pre-trained weights are used to perform quantitative training on the fine-tuned student classroom behavior recognition model using a knowledge distillation-based quantitative perception training method, resulting in the final quantitative weights of the student classroom behavior recognition model. In the knowledge distillation-based quantitative perception training method, the student model uses a quantitative perception model for parameter updates.

[0023] Furthermore, when fine-tuning the student classroom behavior recognition model using the knowledge distillation structure, the knowledge distillation model update adopts the following loss function:

[0024]

[0025]

[0026] in, The loss function represents the difference between the teacher model's predicted values ​​and the student's predicted values ​​in classroom behavior recognition. Let represent the loss function between the student model's predicted values ​​and the actual labels of student classroom behavior recognition, and let x represent the loss value between the teacher model's predicted values ​​and the actual labels of student classroom behavior recognition.

[0027] Furthermore, the process of using the third pre-training weights to perform quantitative training on the fine-tuned student classroom behavior recognition model using a knowledge distillation-based quantitative perception training method also includes, based on the... , The weighting of the model is dynamically adjusted, with the proportion being increased when the teacher model's prediction accuracy for student classroom behavior recognition is higher than that of the student model. The proportion of, and vice versa The proportion, increase The proportion of.

[0028] Furthermore, the step of fusing the recognition results of neighboring multiple frames with the recognition result of the current frame during the tracking process to obtain the final recognition result of the current frame includes:

[0029] Obtain a detection box in the current frame as the current detection box, and obtain the current detection box and the corresponding behavior recognition result;

[0030] Determine whether the current detection box has been successfully tracked;

[0031] If the current tracking fails, the current recognition result will be used as the final recognition result of the current detection box.

[0032] If the current tracking is successful, the recognition results of each action in the current detection box and the neighboring multi-frame data are statistically analyzed. The statistical results are then used to fine-tune the recognition results of each action in the current detection box to obtain the final recognition result of the current detection box.

[0033] Furthermore, the evaluation indicators include classroom interactivity indicators and student focus indicators. The classroom interactivity indicators are calculated using statistical results of student hand-raising behavior recognition. The student focus indicators are positive indicators calculated using statistical results of student hand-raising behavior recognition, and negative indicators calculated using statistical results of head-up rate recognition.

[0034] The formula for calculating the classroom interactivity index is as follows:

[0035]

[0036] The formula for calculating the student focus index is as follows:

[0037]

[0038]

[0039] in, Indicators representing classroom interactivity This indicates a positive indicator in the student attention index. This represents a negative indicator in the student attention index.N This represents the total number of visual checks conducted in the classroom. This represents the total number of pedestrians detected in the i-th detection; This represents the total number of times the hand-raising action was detected in the i-th detection; The total number of times the head-up behavior was detected in the i-th detection; , Let represent the total number of times playing on a mobile phone and lying on a table were detected during the i-th test, respectively.

[0040] An edge device includes a processor and a memory for storing a computer program, and the processor for executing the computer program to perform the method described above.

[0041] A cloud-edge collaborative classroom teaching quality auxiliary assessment system includes:

[0042] Video image acquisition equipment is used to acquire video image data of students in the classroom in real time from the camera and transmit it to the edge device;

[0043] The aforementioned edge device is used to receive video image data transmitted in real time by the video image acquisition device, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to the cloud video server.

[0044] A cloud-based video server is used to receive data transmitted from edge devices in real time, save it, and send it to mobile terminals.

[0045] Compared with existing technologies, the advantages of this invention are as follows: This invention adopts a cloud-edge collaborative architecture to achieve auxiliary assessment of classroom teaching quality. Edge devices identify and assess student classroom behavior based on real-time acquired video image data, while the server is only responsible for saving and pushing the calculation results at the edge. This reduces the computational pressure on the server and lowers the overall hardware deployment cost. At the same time, it enhances the robustness and real-time performance of the system, enabling low-cost, high-frequency auxiliary assessment of teaching quality. Furthermore, a single model simultaneously identifies multiple related behaviors of each student during the classroom. During the tracking process, the student behavior identification results of several neighboring frames are fused with those of the current frame to obtain the final identification result of the student behavior in the current frame. This allows for the fusion of multi-label target detection and target tracking algorithms to achieve multi-frame result fusion, which not only reduces the overall recognition complexity and improves the real-time performance of the algorithm but also effectively improves the accuracy and reliability of the recognition. In addition, the combination of knowledge distillation structure enables the model to be lightweight, further reducing model complexity and computational resource requirements while ensuring model recognition performance, thus lowering the overall hardware deployment cost. Attached Figure Description

[0046] Figure 1This is a schematic diagram illustrating the architecture of the classroom teaching quality auxiliary evaluation system in this embodiment.

[0047] Figure 2 This is a schematic diagram illustrating the implementation process of the cloud-edge collaborative classroom teaching quality auxiliary evaluation method at the edge device in this embodiment.

[0048] Figure 3 This is a schematic diagram illustrating the structural principle of the student classroom behavior recognition model built based on YOLOv8 in this embodiment.

[0049] Figure 4 This is a schematic diagram illustrating the process of student behavior recognition model in this embodiment.

[0050] Figure 5 This is a schematic diagram illustrating the implementation process of fusing neighboring frame detection results to obtain the current frame detection result in a specific application embodiment.

[0051] Figure 6 This is a schematic diagram illustrating the process of training a student classroom behavior recognition model in a specific application embodiment.

[0052] Figure 7 This is a schematic diagram illustrating the principle of the knowledge distillation structure of the student classroom behavior model used in this embodiment.

[0053] Figure 8 This is a schematic diagram illustrating the training principle of the quantitative perception student classroom behavior model based on knowledge distillation used in this embodiment. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0055] As disclosed in this invention, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "first," "second," and similar words used in this invention disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components.

[0056] To assist in the assessment of classroom teaching quality, this embodiment can employ methods such as... Figure 1The architecture shown deploys a classroom teaching quality auxiliary assessment system, employing a cloud-edge collaborative architecture to achieve classroom teaching quality auxiliary assessment. The system includes network cameras, switches / routers, AI edge devices, and a video server. Network cameras are deployed in classrooms to capture real-time video images of students. These images are transmitted to the AI ​​edge devices via switches and routers. The edge devices then analyze the captured video data to identify student behavior in the classroom. Based on the identified behavior, they calculate evaluation indicators to assist in assessing teaching quality. The results are then transmitted to a cloud video server for storage. In essence, the AI ​​edge devices perform AI algorithm model inference and processing, pushing the results to the cloud video server, which then pushes the video and results to client terminals (e.g., mobile devices). Users can view the video and results on their terminals. This improves the system's real-time performance, reduces its dependence on the network, and enhances its robustness. The server is only responsible for storing and pushing the edge-end calculation results, reducing the server's computational load and overall hardware deployment costs. This, combined with enhanced system robustness and real-time performance, enables low-cost, high-frequency auxiliary assessment of teaching quality, facilitating improvements and providing further impetus for further teaching quality enhancement.

[0057] like Figure 2 As shown, the steps for implementing cloud-edge collaborative auxiliary assessment of classroom teaching quality on the edge device in this embodiment include:

[0058] Step S01. The edge device receives video image data of students in the classroom in real time from the camera;

[0059] Step S02. In the edge device, the pre-trained student classroom behavior recognition model is used to recognize the real-time received video image data, and each detected detection box is tracked. During the tracking process, the recognition results of neighboring frames are fused with the recognition results of the current frame to obtain the final recognition result of the current frame. The evaluation index is calculated based on the student classroom behavior recognition results to evaluate the quality of classroom teaching. The student classroom behavior recognition results and the evaluation results are sent to the cloud video server for storage. The student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and a knowledge distillation structure.

[0060] A single behavior is insufficient for accurately evaluating classroom teaching quality. To achieve a more precise and objective assessment, this embodiment employs a multi-label target detection algorithm to train a student classroom behavior recognition model. This model can simultaneously identify multiple related behaviors of students during class, such as raising hands, looking up, using a mobile phone, or resting on the desk. Simultaneously, it tracks the detected bounding boxes, fusing the student behavior recognition results from several neighboring frames with the current frame to obtain the final recognition result for that frame. This integration of multi-label target detection and target tracking algorithms to achieve multi-frame result fusion not only reduces overall recognition complexity and improves algorithm real-time performance but also effectively enhances accuracy and reliability. Furthermore, by using edge devices for model recognition algorithm inference and employing a knowledge distillation structure to achieve model lightweighting, the model's complexity is further reduced, computational resource requirements are decreased, and overall hardware deployment costs are lowered while maintaining recognition performance. This results in an efficient, objective, and accurate assessment of classroom teaching quality.

[0061] Since the student behavior recognition model needs to run at the AI ​​edge, considering the real-time performance, accuracy, and ease of engineering deployment of the algorithm, a multi-label YOLOV8 + DeepSort target tracking approach can be used to achieve student classroom behavior recognition. Specifically, the student classroom behavior recognition model uses YOLOV8, while the DeepSort algorithm is used for target tracking. YOLOV8 improves feature extraction capabilities and target detection performance by introducing a new Backbone and Neck architecture, maintaining an optimal balance between accuracy and speed. DeepSort is a deep learning-based target tracking algorithm that improves the accuracy and robustness of target tracking by combining deep learning and tracking techniques. This embodiment, by combining multi-label YOLOV8 and DeepSort algorithms to construct a student classroom behavior recognition model, can fully leverage the tracking performance of both YOLOV8 and DeepSort to ensure the recognition performance of different student behaviors.

[0062] Traditional YOLOv8 uses ordinary convolution operators in its backbone, which presents several problems: First, for tasks like classification, object detection, and semantic segmentation, the shape, size, color, and distribution of objects at different locations in an image are variable. However, in ordinary convolution operations, the kernel uses the same parameters to extract information across all receptive domains, neglecting the differences between locations, thus limiting network performance. Second, ordinary convolution doesn't consider the importance of each feature, further reducing feature extraction efficiency and ultimately limiting model performance. Furthermore, while attention mechanisms allow models to focus on salient features, enhancing feature extraction efficiency and the ability of convolutional neural networks to capture detailed feature information, traditional spatial attention mechanisms cannot fully address the parameter sharing problem of large convolutional kernels. Additionally, traditional attention mechanisms cannot emphasize the importance of each feature in the receptive domain. To address the aforementioned issues, this embodiment replaces the traditional convolution operator with the RFAConv operator in the convolution operation of the YOLOV8 backbone network. The RFAConv operator not only emphasizes the importance of different features within the receptive field slider but also prioritizes spatial features in the receptive field. Leveraging these advantages of RFAConv convolution, within the receptive field (receptive field of the convolution kernel), the convolution can focus more on learning student behavior regions, thereby giving the feature map a stronger representation ability for student behavior and improving the model's generalization ability. Furthermore, its computational cost and parameter increment are almost negligible, significantly improving network performance.

[0063] Optionally, the computation expression for the RFAConv operator can be:

[0064] (1)

[0065] in, Represents the RFAConv operator. This represents a grouped convolution of size 1×1. This represents a k×k grouped dilated convolution, where k represents the kernel size and Norm represents normalization. X Represents the input feature map, This represents the activation function. This represents the Softmax function. This represents soft pooling, and its calculation formula is shown in Equation 2.

[0066] (2)

[0067] In equation (2), R The selected area Indicates the area R The Middlek Feature values ​​at each position, j represents the region R The Middle j One location, Indicates the region R The Middle j The feature values ​​at each position.

[0068] Compared to traditional convolution operations, this embodiment uses the aforementioned RFAConv operator for convolution operations. Dilated convolution expands the receptive field of the convolution kernel without increasing computational cost, enhancing semantic connections between pixels and improving the model network's performance. Furthermore, the RFAConv operator uses GELU as the activation function, which, compared to the traditional ReLU function, better handles gradient vanishing and differentiability, thus accelerating model convergence and improving generalization ability. Simultaneously, the use of soft pooling minimizes information loss during pooling while maintaining the pooling layer's functionality, better preserving information features and thereby improving the model's ability to extract student classroom behavior features.

[0069] Imbalanced label data may exist in the training data for student classroom behavior recognition, thus affecting the detection accuracy of the student classroom behavior recognition model. To mitigate the impact of sample imbalance on the detection accuracy of the student classroom behavior recognition model, such as... Figure 3 As shown, in this embodiment, the image to be detected is input into the backbone network. After passing through the bottleneck layer, it is provided to the detection head of the student classroom behavior recognition model for student classroom behavior detection. In the detection head of the student classroom behavior recognition model, the classification function of the classification branch (Cls) adopts an improved focal loss function. This improved focal loss function is constructed by introducing class weight factors and using the class weight focal loss function of the focal function. The loss function of the detection branch (Bbox) adopts a loss function that combines the CIOU (Complete Intersection over Union) loss function and the DFL (Distribution Focal Loss) loss function.

[0070] Alternatively, the classification function (Cls) can be a class weight focus loss function as follows:

[0071] (3)

[0072] in, Let be the weighting factor for the classroom behavior of student type i. , m represents the category of student classroom behavior, Let be the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset. Let be the predicted value of the classroom behavior of student type i. For the Sigmoid function, Let be the true label value of the classroom behavior of student type i. As a balance factor, This is the weighting factor.

[0073] This embodiment introduces a class weight factor. This approach can reduce the impact of imbalanced, multi-labeled, long-tailed data on student classroom behavior on model performance. At the same time, the focus function allows the network to pay more attention to difficult samples, reducing the impact of imbalance between easy and difficult samples on model performance, thereby further improving the detection accuracy of the student classroom behavior recognition model.

[0074] Since the student behavior recognition model needs to complete recognition and inference at the edge, considering the real-time performance, accuracy, and ease of engineering deployment of student behavior recognition, this embodiment, based on multi-label YOLOV8+DeepSort target tracking, uses the recognition results of student classroom behavior in several neighboring frames to fine-tune the detection results of the current frame, so as to fuse the recognition results of the current frame with those of several neighboring frames as the final result of student classroom behavior recognition in the current frame, making full use of the fusion of neighboring frame data to improve recognition accuracy. Figure 4 As shown, in the process of student behavior recognition model, for real-time acquired student video image data frames, the student behavior recognition model based on multi-label YOLOV8 is first used for recognition to obtain pedestrian detection boxes, pedestrian recognition scores, classroom behavior recognition scores, and detection box identifier numbers. Then, the original image and the detected pedestrian detection boxes, pedestrian recognition scores, and detection box identifier numbers are input into the DeepSort target tracking algorithm for tracking to obtain the tracking target identifier ID and the detection box identifier number after tracking. The recognition results (detection boxes, classroom behavior recognition scores, etc.) and tracking results (tracking identifier numbers, classroom behavior recognition scores, etc.) of several adjacent frames are obtained to fine-tune the detection results of the current frame and obtain the final detection result of the current frame.

[0075] In this embodiment, the steps for fusing the recognition results of neighboring multiple frames with the recognition result of the current frame during the tracking process to obtain the final recognition result of the current frame include:

[0076] Obtain a detection box in the current frame as the current detection box, and obtain the current detection box and the corresponding behavior recognition result;

[0077] Determine whether the current detection box is successfully tracked. If the current tracking is unsuccessful, the current recognition result is directly used as the final recognition result of the current detection box. If the current tracking is successful, the recognition results of each action in the current detection box and the neighboring multi-frame data are statistically analyzed. The statistical results are used to fine-tune the recognition results of each action in the current detection box to obtain the final recognition result of the current detection box.

[0078] In specific application embodiments, such as Figure 5 As shown, the detailed process of obtaining the current frame detection result using the above method in this embodiment is as follows:

[0079] Step S201. After the tracking model and neighboring frame results are initialized, obtain a detection box and its behavior recognition result in the current frame;

[0080] Step S202. Determine whether the detection box in step S201 was successfully tracked. If not, take the current detection result as the final detection result of the current box; if the tracking is successful, proceed to step S203.

[0081] Step S203. Fine-tune the current detection frame result using the results of several neighboring frames, and count the number of neighboring target identifiers (ID) (cnt) to determine whether the number of times the current detection box is tracked in neighboring frames exceeds a certain threshold (which can be specifically configured). If yes, otherwise use the current detection result as the final detection result of the current box. If yes, proceed to step S204.

[0082] Step S204. Calculate the average action scores of the current detection box and its neighboring detection boxes;

[0083] Step S205. Take the average value of each action in step S204 as the fine-tuning result of each action of the current detection box.

[0084] Considering that the student classroom behavior recognition model is deployed on edge devices, and the computing power of edge devices is limited, in order to ensure the real-time inference of the student classroom behavior recognition model, the student classroom behavior recognition model is quantized. However, quantization of the student classroom behavior recognition model may cause a certain loss of accuracy. In order to ensure that the student classroom behavior recognition algorithm model can achieve real-time operation on edge devices while ensuring recognition accuracy, this embodiment, based on the knowledge distillation structure, further adopts the knowledge distillation quantization perception training method to train the student classroom behavior recognition model, thereby achieving lightweighting of the student classroom behavior recognition model. This fully combines the advantages of knowledge distillation and quantization perception in maintaining the accuracy of the original student classroom behavior recognition model, effectively reducing the accuracy loss of the student classroom behavior recognition model during the quantization process, and achieving model lightweighting.

[0085] like Figure 6As shown, the steps for training the student classroom behavior recognition model using a knowledge distillation structure in this embodiment include:

[0086] Configure the first pre-trained weight as 1;

[0087] The student classroom behavior recognition model is trained using the first pre-training weight 1 to obtain the second pre-training weight 2;

[0088] The second pre-training weight 2 is used to fine-tune the student classroom behavior recognition model based on the knowledge distillation structure to obtain the third pre-training weight 3;

[0089] Using the third pre-training weight 3, a knowledge distillation-based quantitative perception training method is adopted to quantitatively train the fine-tuned student classroom behavior recognition model, and the final quantitative weights of the student classroom behavior recognition model are obtained. In the knowledge distillation-based quantitative perception training method, the student model uses the quantitative perception model for parameter update.

[0090] This embodiment trains the model using a knowledge distillation-based quantization perception training method. This method reduces the accuracy loss during quantization, while the introduction of knowledge distillation further guides the training of the quantization perception model using a large model, thereby improving the training accuracy and ensuring the real-time performance and accuracy of the student behavior recognition model in classroom teaching on edge devices.

[0091] Specifically, in this embodiment, the student model in the knowledge distillation-based quantization-aware training method adopts a quantization-aware model. When calculating the loss, the model parameters are of type int, and when updating the parameters, they are of type float32. This type of training method can reduce the accuracy loss during the quantization process. Traditional quantization-aware training uses the real labels to calculate the loss and update the model parameters. This embodiment introduces a knowledge distillation method on top of the quantization-aware model training, using a larger model to further guide the quantization-aware model training, thereby improving the training accuracy of the quantization-aware model.

[0092] In a specific application example, the weights trained on an open-source dataset can first be used as the first pre-training weight 1 to directly train the student classroom behavior recognition model, resulting in the pre-training weight 2. Then, using the second pre-training weight 2, the student classroom behavior recognition model is further fine-tuned using the knowledge distillation method, resulting in the third pre-training weight 3. Finally, using the third pre-training weight 3, the student classroom behavior recognition model is quantitatively trained using the knowledge distillation-based quantitative perception method, resulting in the final quantitative weights of the student classroom behavior recognition model.

[0093] To mitigate the impact of prediction errors in the Teacher model on the performance of the Student model during knowledge distillation, this embodiment employs the following knowledge distillation process: Figure 7 As shown, the loss value Hard_Loss_T of the soft label (predicted value) and the true label of student classroom behavior recognition teacher model is normalized using the arctangent function Atan. This Atan is then used to dynamically adjust the weighting of the distillation loss and the loss value Hard_Loss_S between the prediction result of the student model and the true label. Finally, the two are added together to obtain the final loss function value updated by the knowledge distillation model for student classroom behavior recognition. To ensure the performance of the student classroom behavior recognition model.

[0094] Optionally, when fine-tuning the student classroom behavior recognition model using the knowledge distillation structure, the knowledge distillation model update can use the following loss function:

[0095] (4)

[0096] (5)

[0097] in, This represents the loss function between the predicted values ​​of the teacher model and the predicted values ​​of the Student model for recognizing student classroom behavior. Let represent the loss function between the student model's predicted values ​​and the actual labels of student classroom behavior recognition, and let x represent the loss value between the teacher model's predicted values ​​and the actual labels of student classroom behavior recognition.

[0098] In this embodiment, the process of quantitatively training the fine-tuned student classroom behavior recognition model also includes, based on the... , The proportion of [something] is dynamically adjusted, whereby the prediction accuracy of the Teacher model for recognizing student classroom behavior is increased. The proportion of, and vice versa The proportion, increase The weighting of the Distillation Loss is adjusted. Specifically, when the Teacher model's prediction accuracy is higher than the Student model's, its loss value relative to the true label is smaller, so the weighting of Distillation Loss is increased. Conversely, the weighting of Hard_Loss_S is increased instead of Distillation Loss. Through this dynamic adjustment, when the Teacher model makes a prediction error, its impact on the optimization of the Student model's parameters can be reduced, thereby effectively improving the performance of the Student model.

[0099] This embodiment uses the knowledge distillation-based quantitative perception training method to train the student classroom behavior recognition model. The process is as follows: Figure 8 As shown, the student model is trained using a quantized perception model. When calculating the loss, the model parameters are of type int, and when updating the parameters, they are of type float32, which can reduce the accuracy loss in the quantization process. The training loss function is similar to formula (4), that is, the loss value Hard_Loss_T of the teacher model's soft label (predicted value) and the student's classroom behavior real label is normalized using the arctangent function Atan, and the ratio between the distillation loss and the student model's prediction result and the real label's loss value Hard_Loss_S is dynamically adjusted, and then the two are added together to obtain the final loss function value of the student classroom behavior recognition knowledge distillation model update, thereby reducing the impact of the teacher model's prediction error on the quantization model performance during the knowledge distillation process.

[0100] Student focus is evaluated from two perspectives: the positive aspect is calculated by counting the students' head-up rate, as shown in Equation 7; the negative aspect is calculated by counting students' behaviors such as playing on their phones and lying on their desks, as shown in Equation 8.

[0101] This embodiment specifically uses classroom interactivity and student focus to assist in evaluating classroom teaching quality. Classroom interactivity reflects whether the teaching is lively, whether teachers and students are deeply involved, and whether the teacher's teaching methods can stimulate students' desire for knowledge. Student focus more directly reflects students' interest in the class and classroom discipline, providing a direct reference for improving classroom teaching quality. This embodiment further sets corresponding quantitative indicators for classroom interactivity and student focus. Classroom interactivity can be evaluated using statistics such as hand-raising behavior, while student focus can be evaluated from both positive and negative perspectives. Positive aspects can be evaluated using positive indicators such as student head-up rate, while negative aspects can be evaluated using statistics such as students playing with their phones or lying on their desks. In other words, the evaluation indicators include classroom interactivity indicators and student focus indicators. The classroom interactivity indicator is calculated using the statistical results of student hand-raising behavior recognition, while the student focus indicator is calculated using the statistical results of student hand-raising behavior recognition (positive indicator) and the statistical results of head-up rate recognition (negative indicator).

[0102] Optionally, the classroom interactivity index can be calculated using the following expression:

[0103] (6)

[0104] The student focus index can be calculated using the following formula:

[0105] (7)

[0106] (8)

[0107] in, Indicators representing classroom interactivity This indicates a positive indicator in the student attention index. This represents a negative indicator in the student attention index. N This represents the total number of visual checks conducted in the classroom. This represents the total number of pedestrians detected in the i-th detection; This represents the total number of times the hand-raising action was detected in the i-th detection; The total number of times the head-up behavior was detected in the i-th detection; , Let represent the total number of times playing on a mobile phone and lying on a table were detected during the i-th test, respectively.

[0108] Among the above indicators, , , The value ranges from [0,1], with larger values ​​reflecting better classroom teaching quality. It's understandable that other indicators and calculation methods can also be used to quantify classroom teaching quality.

[0109] This embodiment also provides an edge device, including a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method described above.

[0110] like Figure 2 As shown, the cloud-edge collaborative classroom teaching quality auxiliary evaluation system in this embodiment includes:

[0111] Video image acquisition equipment is used to acquire video image data of students in the classroom in real time from the camera and transmit it to the edge device;

[0112] The edge device mentioned above is used to receive video image data transmitted in real time by the video image acquisition device, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to the cloud video server.

[0113] A cloud-based video server is used to receive data transmitted from edge devices in real time, save it, and send it to mobile terminals.

[0114] Understandably, the above system can be further equipped with other auxiliary devices to meet different needs.

[0115] It is understood that the method described in this embodiment can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration, characterized in that the steps include: The edge device receives the video image data of students in the classroom collected by the camera in real time; In the edge device, the video image data received in real time is used to identify the student classroom behavior using a pre-trained student classroom behavior recognition model, and each detected detection frame is tracked. During the tracking process, the recognition results of multiple adjacent frames are fused with the recognition results of the current frame to obtain the final recognition result of the current frame. The evaluation index is calculated based on the student classroom behavior recognition results to evaluate the classroom teaching quality, and the student classroom behavior recognition results and evaluation results are sent to the cloud video server for storage; The student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and a knowledge distillation structure; the evaluation indicators include a classroom interactivity indicator and a student concentration indicator, the classroom interactivity indicator is calculated using the statistical results of the student hand-raising behavior recognition results, the student concentration indicator is a positive indicator calculated using the statistical results of the student hand-raising behavior recognition results, and a negative indicator calculated using the statistical results of the head-up rate recognition results; The calculation expression of the classroom interactivity index is: The calculation expression of the student concentration index is: in, represents the classroom interactivity index, Indicates the positive indicators of student concentration. It represents the negative indicator of student concentration. N represents the total number of visual inspections in the classroom; represents the total number of pedestrians detected during the i-th detection; represents the total number of times the hand-raising behavior is detected during the i-th detection; The total number of times the head-up behavior was detected during the i-th detection; , They represent the total number of times playing with mobile phone and lying on the table were detected during the i-th detection.

2. According to claim 1, the classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration is characterized in that: The student classroom behavior recognition model is constructed based on YOLOV8, and the convolution operation in the BackBone part of YOLOV8 uses the RFAConv operator. The expression of the RFAConv operator is: in, represents the RFAConv operator, represents a grouped convolution of size 1×1, represents a grouped hole convolution of size k×k, k represents the size of the convolution kernel, and Norm represents normalization. X represents the input feature map, represents the activation function, represents the Softmax function, It represents soft pooling and is calculated as: in, R is the local area selected by the box. Indicates the result after soft pooling. , , Respectively represent the area R Middle k, j The feature value at each position.

3. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 1 is characterized in that: When using the student classroom behavior recognition model to identify student classroom behavior, the classification function adopts the following class weighted focal loss function: in, is the weight factor of the classroom behavior of the i-th student, , m is the category of students’ classroom behavior, is the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset, is the predicted value of the classroom behavior of the i-th class of students, is the Sigmoid function, is the true label value of the classroom behavior of the i-th student, is the balance factor, is the weight factor.

4. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 1 is characterized in that: The steps of training the student classroom behavior recognition model using the knowledge distillation structure include: Configure the first pre-training weight; Using the first pre-training weights to train the student classroom behavior recognition model to obtain second pre-training weights; The second pre-training weight is used to fine-tune the student classroom behavior recognition model based on the knowledge distillation structure to obtain the third pre-training weight; The third pre-trained weight is used to adopt a quantitative perceptual training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model to obtain the final quantitative weight of the student classroom behavior recognition model. In the quantitative perceptual training method based on knowledge distillation, the student model adopts a quantitative perceptual model to update parameters.

5. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 4 is characterized in that: When fine-tuning the student classroom behavior recognition model using the knowledge distillation structure, the knowledge distillation model is updated using the following loss function: in, represents the loss function between the predicted value of the teacher model for student classroom behavior recognition and the predicted value of the Student model for student classroom behavior recognition, represents the loss function between the predicted value of the student model for identifying student classroom behavior and the true label of the student classroom behavior, and x represents the loss value between the predicted value of the teacher model for identifying student classroom behavior and the true label of the student classroom behavior.

6. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 5 is characterized in that: The process of using the third pre-training weights to adopt a quantitative perception training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model also includes: , The proportion of the teacher model is dynamically adjusted. When the prediction accuracy of the student classroom behavior recognition teacher model is higher than that of the student model, the The proportion of The proportion of 's proportion.

7. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to any one of claims 1 to 6 is characterized in that: The step of fusing the recognition results of the adjacent multiple frames with the recognition result of the current frame to obtain the final recognition result of the current frame during the tracking process includes: Get a detection frame of the current frame as the current detection frame, and get the current detection frame and the corresponding behavior recognition result; Determine whether the current detection frame is tracked successfully; If the current tracking is unsuccessful, the current recognition result is directly used as the final recognition result of the current detection frame; If the current tracking is successful, the action recognition results of the current detection frame and the adjacent multi-frame data are counted, and the action recognition results of the current detection frame are fine-tuned using the statistical results to obtain the final recognition result of the current detection frame.

8. An edge device, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 7.

9. A classroom teaching quality auxiliary evaluation system based on cloud-edge collaboration, characterized in that: include: Video image acquisition equipment, used to obtain video image data of students in the classroom captured by the camera in real time and transmit it to the edge device; The edge device as claimed in claim 8 is used to receive video image data transmitted in real time by a video image acquisition device, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to a cloud video server; The cloud video server is used to receive data transmitted by edge devices in real time, save it, and send it to mobile terminals.

Images