Classroom teaching quality auxiliary evaluation method, equipment and system based on cloud edge collaboration

Abstract

The invention discloses a classroom teaching quality auxiliary evaluation method, device and system based on cloud edge collaboration, and the method comprises the steps: obtaining video image data, collected by a camera, of students in a classroom in real time, and transmitting the video image data to an edge device; and in the edge device, performing student classroom behavior identification and tracking on the received video image data by using a pre-trained student classroom behavior identification model, and fusing the identification results of adjacent multiple frames with the identification result of the current frame in the tracking process to obtain the final identification result of the current frame. Evaluating the classroom teaching quality according to the student classroom behavior recognition result, wherein the student classroom behavior recognition model is obtained by training based on a multi-label target detection algorithm and by adopting a knowledge distillation structure in advance; and sending a student classroom behavior identification result and an evaluation result to a cloud video server for storage. The method has the advantages of being simple in implementation method, low in cost, high in real-time performance and precision, high in robustness and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent evaluation of classroom teaching quality, and in particular to a classroom teaching quality auxiliary evaluation method, device and system based on cloud-edge collaboration. Background Art

[0002] At present, the evaluation of classroom teaching quality is mainly done through manual evaluation methods such as classroom lectures and subject tests. Although it can reflect the quality of education to a certain extent, it has the problems of high implementation cost, low frequency, and difficulty in effectively supervising the improvement of teaching quality. Intelligent evaluation of classroom teaching quality uses artificial intelligence technologies such as machine learning, natural language processing, and computer vision to automatically evaluate and analyze the quality of classroom teaching by extracting features such as teachers' or students' voices and images. It can greatly improve the efficiency of evaluation, reduce labor costs, and improve the accuracy of evaluation.

[0003] For intelligent evaluation of classroom teaching quality, the existing technology usually uses a server to deploy the model. A processor runs a machine learning model. After receiving the voice or image data collected from teachers or students in real time, the machine learning model is used to identify the status of teachers or students to evaluate the teaching quality. However, in the above method, since all data processing is concentrated in one processor, the server running model needs to process a large amount of image data, which will increase the server's computing and storage pressure, and thus has high requirements on the processor's storage capacity and data processing capabilities, resulting in a high overall hardware deployment cost; on the other hand, since the video data collected by the camera needs to be transmitted to the server for processing in real time through the network, the overall system is highly dependent on the stability of the network, and the system's anti-interference ability is reduced. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a classroom teaching quality auxiliary evaluation method, equipment and system based on cloud-edge collaboration, which has a simple implementation method, low cost, real-time performance, high accuracy and strong robustness.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is: A classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration, the steps include: The edge device receives the video image data of students in the classroom collected by the camera in real time; In the edge device, the video image data received in real time is used to identify the student classroom behavior using a pre-trained student classroom behavior recognition model, and each detected detection frame is tracked. During the tracking process, the recognition results of multiple adjacent frames are fused with the recognition results of the current frame to obtain the final recognition result of the current frame. The evaluation index is calculated based on the student classroom behavior recognition results to evaluate the classroom teaching quality, and the student classroom behavior recognition results and evaluation results are sent to the cloud video server for storage; The student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and adopts a knowledge distillation structure.

[0006] Furthermore, the student classroom behavior recognition model is constructed based on YOLOV (You Only Look Once) 8, and the convolution operation in the BackBone (backbone network) part of YOLOV8 uses the RFAConv (Receptive Field Attention Convolution) operator, and the expression of the RFAConv operator is:

[0007] in, represents the RFAConv operator, represents a grouped convolution of size 1×1, represents a grouped atrous convolution of size k×k, k represents the size of the convolution kernel, and Norm represents normalization. X represents the input feature map, represents the activation function, represents the Softmax function, It represents soft pooling and is calculated as:

[0008] in, R is the local area selected by the box. Indicates the result after soft pooling. , , Respectively represent the area R Middle k, j The feature value at each position.

[0009] Furthermore, when using the student classroom behavior recognition model to identify student classroom behavior, the classification function adopts the following class weighted focal loss function:

[0010] in, is the weight factor of the classroom behavior of the i-th student, , m is the category of students’ classroom behavior, is the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset, is the predicted value of the classroom behavior of the i-th class of students, is the Sigmoid function, is the true label value of the classroom behavior of the i-th student, is the balance factor, is the weight factor.

[0011] Furthermore, the steps of training the student classroom behavior recognition model using the knowledge distillation structure include: Configure the first pre-training weight; Using the first pre-training weights to train the student classroom behavior recognition model to obtain second pre-training weights; The second pre-training weight is used to fine-tune the student classroom behavior recognition model based on the knowledge distillation structure to obtain the third pre-training weight; The third pre-trained weight is used to adopt a quantitative perceptual training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model to obtain the final quantitative weight of the student classroom behavior recognition model. In the quantitative perceptual training method based on knowledge distillation, the student model adopts a quantitative perceptual model to update parameters.

[0012] Furthermore, when the knowledge distillation structure is used to fine-tune the student classroom behavior recognition model, the knowledge distillation model is updated using the following loss function:

[0013]

[0014] in, represents the loss function between the predicted value of the teacher model for student classroom behavior recognition and the predicted value of the Student model for student classroom behavior recognition, represents the loss function between the predicted value of the student model for identifying student classroom behavior and the true label of the student classroom behavior, and x represents the loss value between the predicted value of the teacher model for identifying student classroom behavior and the true label of the student classroom behavior.

[0015] Furthermore, the process of using the third pre-training weights to adopt a quantitative perception training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model also includes: , The proportion of the teacher model is dynamically adjusted. When the prediction accuracy of the student classroom behavior recognition teacher model is higher than that of the student model, the The proportion of The proportion of 's proportion.

[0016] Furthermore, the step of fusing the recognition results of the adjacent multiple frames with the recognition result of the current frame to obtain the final recognition result of the current frame during the tracking process includes: Get a detection frame of the current frame as the current detection frame, and get the current detection frame and the corresponding behavior recognition result; Determine whether the current detection frame is tracked successfully; If the current tracking is unsuccessful, the current recognition result is directly used as the final recognition result of the current detection frame; If the current tracking is successful, the action recognition results of the current detection frame and the adjacent multi-frame data are counted, and the action recognition results of the current detection frame are fine-tuned using the statistical results to obtain the final recognition result of the current detection frame.

[0017] Furthermore, the evaluation index includes a classroom interactivity index and a student concentration index, wherein the classroom interactivity index is calculated using the statistical results of the student hand-raising behavior recognition results, and the student concentration index is a positive index calculated using the statistical results of the student hand-raising behavior recognition results, and a negative index calculated using the statistical results of the head-up rate recognition results; The calculation expression of the classroom interactivity index is:

[0018] The calculation expression of the student concentration index is:

[0019]

[0020] in, represents the classroom interactivity index, Indicates the positive indicators of student concentration. It represents the negative indicator of student concentration. N represents the total number of visual inspections in the classroom; represents the total number of pedestrians detected during the i-th detection; represents the total number of times the hand-raising behavior is detected during the i-th detection; The total number of times the head-up behavior was detected during the i-th detection; , They represent the total number of times playing with mobile phone and lying on the table were detected during the i-th detection.

[0021] An edge device includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0022] A classroom teaching quality auxiliary evaluation system based on cloud-edge collaboration, including: Video image acquisition equipment, used to obtain video image data of students in the classroom captured by the camera in real time and transmit it to the edge device; Such edge devices as those mentioned above are used to receive video image data transmitted in real time by video image acquisition devices, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to the cloud video server; The cloud video server is used to receive data transmitted by edge devices in real time, save it, and send it to mobile terminals.

[0023] Compared with the prior art, the advantages of the present invention are: the present invention realizes auxiliary evaluation of classroom teaching quality by adopting a cloud-edge collaborative architecture, and the edge device recognizes and evaluates the student classroom behavior of the real-time collected video image data. The server is only responsible for the storage of edge calculation results and the push of results, etc., which can reduce the computing pressure of the server and the overall hardware deployment cost, while enhancing the robustness and real-time performance of the system operation, and realizing low-cost and high-frequency auxiliary evaluation of teaching quality. At the same time, a single model is used to simultaneously identify multiple related behavioral performances of each student during class, and in the tracking process, the student behavior recognition results of several adjacent frames and the current frame are fused as the final recognition result of the student behavior in the current frame. It can integrate multi-label target detection and target tracking algorithms to realize multi-frame result fusion, which can not only reduce the overall recognition complexity and improve the real-time operation of the algorithm, but also effectively improve the recognition accuracy and reliability. At the same time, the knowledge distillation structure is combined to realize the lightweight of the model, and on the basis of ensuring the model recognition performance, it can further reduce the model complexity, reduce the computing resource requirements, and reduce the overall hardware deployment cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the architecture principle of the classroom teaching quality auxiliary evaluation system of this embodiment.

[0025] Figure 2 It is a schematic diagram of the implementation process of the classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration on the edge device side in this embodiment.

[0026] Figure 3 It is a schematic diagram of the structural principle of the student classroom behavior recognition model constructed based on YOLOV8 in this embodiment.

[0027] Figure 4It is a flow chart of the student behavior recognition model in this embodiment to realize student behavior recognition.

[0028] Figure 5 It is a schematic diagram of the implementation process of fusing adjacent frame detection results to obtain the current frame detection result in a specific application embodiment.

[0029] Figure 6 It is a flowchart for implementing the training of the student classroom behavior recognition model in a specific application embodiment.

[0030] Figure 7 It is a schematic diagram of the principle of the knowledge distillation structure of the student classroom behavior model adopted in this embodiment.

[0031] Figure 8 It is a schematic diagram of the training principle of the quantitative perception student classroom behavior model based on knowledge distillation adopted in this embodiment. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0033] As shown in the disclosure of the present invention, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. The words "first", "second" and similar words used in the disclosure of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components.

[0034] In order to realize the auxiliary evaluation of classroom teaching quality, this embodiment can adopt the following method: Figure 1The architectural model shown deploys a classroom teaching quality assisted evaluation system to implement classroom teaching quality assisted evaluation using a cloud-edge collaborative architecture. The system includes a network camera, a switch / router, an AI edge device and a video server. The network camera is arranged in the classroom to collect video images of students in the classroom in real time, and transmits them to the AI ​​edge device through the switch and router. The edge device recognizes the students' classroom behavior based on the real-time collected video image data, and then calculates the evaluation index based on the recognition result of the students' classroom behavior to assist in the evaluation of the teaching quality. The detection result is then transmitted to the cloud video server for storage, that is, the AI ​​algorithm model reasoning and processing result push are completed by the AI ​​edge device. The cloud video server then pushes the video, detection results, etc. to the client terminal (such as a mobile device, etc.). The user can view the video, detection results, etc. through the terminal, which can improve the real-time operation of the system, reduce the system's dependence on the network, and improve the robustness of the system operation. The server is only responsible for the storage of the edge computing results and the push of the results, which can reduce the computing pressure of the server and the overall hardware deployment cost. At the same time, it enhances the robustness and real-time operation of the system, and can achieve low-cost and high-frequency auxiliary evaluation of teaching quality, which is conducive to improving the teaching quality and further urging the improvement of teaching quality.

[0035] like Figure 2 As shown, the steps of implementing the classroom teaching quality auxiliary evaluation based on cloud-edge collaboration on the edge device side in this embodiment include: Step S01. The edge device receives video image data of students in the classroom collected by the camera in real time; Step S02. In the edge device, the video image data received in real time is used to identify the student classroom behavior using a pre-trained student classroom behavior recognition model, and each detected detection frame is tracked. During the tracking process, the recognition results of multiple adjacent frames are fused with the recognition result of the current frame to obtain the final recognition result of the current frame. The evaluation index is calculated based on the student classroom behavior recognition result to evaluate the classroom teaching quality, and the student classroom behavior recognition result and the evaluation result are sent to the cloud video server for storage; the student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and a knowledge distillation structure.

[0036] A single behavior is difficult to accurately evaluate the quality of classroom teaching. In order to evaluate the quality of classroom teaching more accurately and objectively, this embodiment uses a multi-label target detection algorithm to train a student classroom behavior recognition model. It can use a single model to simultaneously recognize multiple related behaviors of each student during class, such as raising hands, looking up, playing with mobile phones, lying on the table, etc., and track the detection frame at the same time. During the tracking process, the student behavior recognition results of several adjacent frames and the current frame are fused as the final recognition result of the student behavior in the current frame. It can integrate multi-label target detection and target tracking algorithms to achieve multi-frame result fusion, which can not only reduce the overall recognition complexity and improve the real-time operation of the algorithm, but also effectively improve the accuracy and reliability of recognition. At the same time, on the basis of using edge devices to perform model recognition algorithm inference, the model is lightweighted by adopting a knowledge distillation structure. On the basis of ensuring the model recognition performance, the model complexity can be further reduced, the computing resource requirements can be reduced, and the overall hardware deployment cost can be reduced, thereby efficiently achieving objective and accurate classroom teaching quality evaluation.

[0037] Since the student behavior recognition model needs to run on the AI ​​edge, considering the real-time, accuracy and engineering deployment of the algorithm, multi-label YOLOV8+DeepSort target tracking can be used to realize student classroom behavior recognition, that is, the student classroom behavior recognition model adopts YOLOV8, and the DeepSort algorithm is used to realize target tracking. YOLOV8 improves the feature extraction capability and target detection performance by introducing the new Backbone and Neck architectures, and can maintain the best balance between accuracy and speed; DeepSort is a target tracking algorithm based on deep learning, which improves the accuracy and robustness of target tracking by combining deep learning and tracking technology. This embodiment combines multi-label YOLOV8 and DeepSort algorithms to construct a student classroom behavior recognition model, which can give full play to the tracking performance of YOLOV8 and DeepSort algorithms to ensure the recognition performance of different student behaviors.

[0038] The BackBone part of the traditional YOLOV8 uses ordinary convolution operators, which will have the following problems: on the one hand, for classification, target detection and semantic segmentation tasks, the shape, size, color and distribution of the target at different positions in the image are variable, and in the ordinary convolution operation process, the convolution kernel uses the same parameters to extract information in each receiving domain, without considering the difference information from different positions, so the performance of the network is limited; on the other hand, the ordinary convolution process does not take into account the importance of each feature, further reducing the efficiency of feature extraction, which ultimately limits the performance of the model. In addition, although the attention mechanism enables the model to focus on significant features, enhance the benefits of feature extraction and the ability of convolutional neural networks to capture detailed feature information, the traditional spatial attention mechanism cannot completely solve the parameter sharing problem of large convolution kernels. In addition, the traditional attention mechanism cannot emphasize the importance of each feature in the receiving domain. In view of the above problems, this embodiment uses the RFAConv operator to replace the traditional convolution operator for the convolution operation in the backbone network (BackBone) of YOLOV8. The RFAConv operator not only emphasizes the importance of different features in the receiving domain slider, but also prioritizes the spatial features of the receiving domain. By utilizing the above advantages of RFAConv convolution, the convolution can focus more on learning the student behavior area within the receiving domain (convolution kernel receptive field), so that the feature map has a stronger representation ability for the student behavior, thereby improving the generalization ability of the model. In addition, its computational cost and parameter increment are almost negligible, which can significantly improve the network performance.

[0039] Optionally, the calculation expression of the RFAConv operator can be: (1) in, represents the RFAConv operator, represents a grouped convolution of size 1×1, represents a grouped atrous convolution of size k×k, k represents the size of the convolution kernel, and Norm represents normalization. X represents the input feature map, represents the activation function, represents the Softmax function, It represents soft pooling, and its calculation formula is shown in Formula 2.

[0040] (2) In formula (2), R is the local area selected by the box. Indicates area R Middle k The eigenvalues ​​at the positions, , j represents the regionR Middle j Locations, Indicates area R Middle j The feature value at each position.

[0041] Compared with traditional convolution operations, this embodiment uses the above-mentioned RFAConv operator to perform convolution operations, in which the use of dilated convolution can expand the receptive field of the convolution kernel without increasing the amount of calculation, enhance the semantic connection between pixels, and improve the performance of the model network. In addition, the RFAConv operator uses GELU as the activation function, which has better processing of gradient vanishing and differentiability than the traditional Relu function, thereby accelerating model convergence and improving model generalization ability. At the same time, combined with the soft pooling SoftPool operation, it can minimize the information loss caused by the pooling process while maintaining the function of the pooling layer, better retain information features, and thus improve the model's ability to extract student classroom behavior features.

[0042] There may be an imbalance in the label data in the student classroom behavior recognition training data, which will affect the detection accuracy of the student classroom behavior recognition model. In order to reduce the impact of sample imbalance on the detection accuracy of the student classroom behavior recognition model, Figure 3 As shown, in this embodiment, the image to be detected is input into the backbone network, and after passing through the bottleneck layer, it is provided to the detection head (Head) of the student classroom behavior recognition model for student classroom behavior detection. In the detection head (Head) of the student classroom behavior recognition model, the classification function of the classification branch (Cls) adopts an improved focal loss function (Focal Loss), which is constructed by introducing a class weight factor and adopting a class weight focal loss function of the focal function. The loss function of the detection branch (Bbox) adopts a loss function that combines the CIOU (Complete Intersection over Union) loss function and the DFL (Distribution Focal Loss) loss function.

[0043] Optionally, the classification function (Cls) can adopt the following class-weighted focal loss function: (3) in, is the weight factor of the classroom behavior of the i-th student, , m is the category of students’ classroom behavior, is the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset, is the predicted value of the classroom behavior of the i-th class of students, is the Sigmoid function, is the true label value of the classroom behavior of the i-th student, is the balance factor, is the weight factor.

[0044] In this embodiment, a class weight factor is introduced , which can reduce the impact of the imbalance of multi-label long-tail data of students' classroom behavior on model performance. At the same time, the focus function is used to make the network pay more attention to difficult samples, reducing the impact of the imbalance of difficult and easy samples on model performance, thereby further improving the detection accuracy of the student classroom behavior recognition model.

[0045] Since the student behavior recognition model needs to complete recognition reasoning at the edge, considering the real-time, accuracy and engineering deployment of student behavior recognition, this embodiment uses the recognition results of student classroom behavior in several adjacent frames to fine-tune the current frame detection results based on multi-label YOLOV8+DeepSort target tracking, so as to fuse the current frame and the recognition results of several adjacent frames as the final result of student classroom behavior recognition in the current frame, and make full use of the fusion of adjacent frame data to improve recognition accuracy. Figure 4 As shown in the figure, in the process of student behavior recognition by the student behavior recognition model, for the student video image data frame collected in real time, the student behavior recognition model based on multi-label YOLOV8 is first used for recognition to obtain the pedestrian detection frame, pedestrian recognition score, classroom behavior recognition score, and detection frame identification number. Then, the original image and the detected pedestrian detection frame, pedestrian recognition score, and detection frame identification number are input into the DeepSort target tracking algorithm for tracking to obtain the tracking target identification number ID and the detection frame identification number after tracking, and the recognition results (detection frame, classroom behavior recognition score, etc.) and tracking results (tracking identification number, classroom behavior recognition score, etc.) of several adjacent frames are obtained to fine-tune the detection result of the current frame to obtain the final detection result of the current frame.

[0046] In this embodiment, the steps of fusing the recognition results of multiple adjacent frames with the recognition result of the current frame to obtain the final recognition result of the current frame during the tracking process include: Get a detection frame of the current frame as the current detection frame, and get the current detection frame and the corresponding behavior recognition result; Determine whether the current detection frame is tracked successfully. If the current tracking is unsuccessful, directly use the current recognition result as the final recognition result of the current detection frame. If the current tracking is successful, count the recognition results of each action in the current detection frame and the adjacent multi-frame data, and use the statistical results to fine-tune the recognition results of each action of the current detection frame to obtain the final recognition result of the current detection frame.

[0047] In a specific application embodiment, Figure 5 As shown, the detailed process of obtaining the current frame detection result by the above method in this embodiment is as follows: Step S201. After the tracking model and the adjacent frame results are initialized, a detection frame of the current frame and its behavior recognition result are obtained; Step S202: Determine whether the detection frame in step S201 is tracked successfully. If not, use the current detection result as the final detection result of the current frame. If the tracking is successful, proceed to step S203. Step S203. Use the results of several neighboring frames to fine-tune the current detection frame result, and count the number of neighboring tracking target identification numbers cnt to determine whether the number of tracking times of the current detection frame in the neighboring frame tracking exceeds a certain threshold (which can be specifically configured). If yes, take the current detection result as the final detection result of the current frame. If yes, proceed to step S204; Step S204: Calculate the average scores of each action of the current detection frame and the adjacent detection frames; Step S205: taking the mean of each action in step S204 as the fine-tuning result of each action in the current detection frame.

[0048] Considering that the student classroom behavior recognition model is deployed on the edge device, and the computing power of the edge device is also limited, in order to ensure the real-time reasoning of the student classroom behavior recognition model, the student classroom behavior recognition model is quantized, and the quantization of the student classroom behavior recognition model may cause a certain loss of accuracy. In order to ensure that the student classroom behavior recognition algorithm model can achieve real-time operation on the edge device while ensuring recognition accuracy, this embodiment, based on the knowledge distillation structure, further adopts the quantitative perception training method of knowledge distillation to train the student classroom behavior recognition model, so as to achieve the lightweight of the student classroom behavior recognition model. It can fully combine the advantages of knowledge distillation and quantitative perception in maintaining the accuracy of the original student classroom behavior recognition model, effectively reduce the accuracy loss of the student classroom behavior recognition model in the quantization process, and achieve model lightweight.

[0049] like Figure 6 As shown, the steps of training the student classroom behavior recognition model using the knowledge distillation structure in this embodiment include: Configure the first pre-training weight 1; Use the first pre-training weight 1 to train the student classroom behavior recognition model to obtain the second pre-training weight 2; The second pre-training weight 2 is used to fine-tune the student classroom behavior recognition model based on the knowledge distillation structure to obtain the third pre-training weight 3; The third pre-trained weight 3 is used to adopt the quantitative perceptual training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model to obtain the final quantitative weight of the student classroom behavior recognition model. In the quantitative perceptual training method based on knowledge distillation, the student model adopts the quantitative perceptual model to update parameters.

[0050] This embodiment trains the model by adopting a quantization perception training method based on knowledge distillation. The quantization perception training method can reduce the accuracy loss in the quantization process. At the same time, the knowledge distillation method is introduced to further guide the quantization perception model training using a large model, and the training accuracy of the quantization perception model can be improved, thereby further ensuring the real-time and accuracy of the classroom teaching student behavior recognition model running on the edge device.

[0051] Specifically, in the quantization perception training method based on knowledge distillation in this embodiment, the student model adopts a quantization perception model. When calculating the loss, the model parameters use the int type, and when updating the parameters, the float32 type is used. This type of training method can reduce the precision loss in the quantization process. In the traditional quantization perception training process, the real label used by the label calculates the loss and updates the model parameters. This embodiment introduces the knowledge distillation method based on the quantization perception model training, and uses the large model to further guide the quantization perception model training, so that the training accuracy of the quantization perception model can be improved.

[0052] In a specific application embodiment, first, the weight trained by the open source dataset can be used as the first pre-training weight 1, and the student classroom behavior recognition model can be directly trained to obtain the pre-training weight 2; then, the second pre-training weight 2 is used to further fine-tune the student classroom behavior recognition model using the knowledge distillation method to obtain the third pre-training weight 3; finally, the third pre-training weight 3 is used to quantitatively train the student classroom behavior recognition model using the quantitative perception method based on knowledge distillation to obtain the final quantitative weight of the student classroom behavior recognition model.

[0053] In order to reduce the impact of the prediction errors of the Teacher model on the performance of the Student model in the knowledge distillation process, the knowledge distillation process used in this embodiment is as follows: Figure 7As shown in the figure, the soft label (prediction value) of the teacher model for student classroom behavior recognition and the loss value Hard_Loss_T of the real label of the student classroom behavior are normalized using the inverse tangent function Atan, and the ratio between the distillation loss (DistillationLoss) and the prediction result of the student model and the loss value Hard_Loss_S of the real label is dynamically adjusted, and then the two are added together to obtain the final loss function value of the student classroom behavior recognition knowledge distillation model to update , ensuring the performance of the student classroom behavior recognition model.

[0054] Optionally, when fine-tuning the student classroom behavior recognition model using the knowledge distillation structure, the knowledge distillation model update can use the following loss function: (4) (5) in, Represents the loss function between the predicted value of the teacher model for student classroom behavior recognition and the predicted value of the Student model for student classroom behavior recognition, represents the loss function between the predicted value of the student model for identifying student classroom behavior and the true label of the student classroom behavior, and x represents the loss value between the predicted value of the teacher model for identifying student classroom behavior and the true label of the student classroom behavior.

[0055] In this embodiment, the process of quantitatively training the fine-tuned student classroom behavior recognition model also includes: , The proportion of the teacher model is dynamically adjusted, and the prediction accuracy of the student classroom behavior recognition teacher model is expanded. The proportion of The proportion of Specifically, when the prediction accuracy of the Teacher model for student classroom behavior recognition is higher than that of the Student model, the loss value between the model and the true label is small, and the proportion of Distillation Loss is increased. Conversely, the proportion of Distillation Loss is reduced and the proportion of Hard_Loss_S is increased. Through the above dynamic adjustment, when the Teacher model for student classroom behavior recognition makes a prediction error, its influence on the parameter optimization of the Student model for student classroom behavior recognition can be reduced, thereby effectively improving the performance of the Student model for student classroom behavior recognition.

[0056] This embodiment uses the quantitative perception training method of knowledge distillation to train the student classroom behavior recognition model. Figure 8As shown in the figure, the student model is trained using a quantization-aware model. Based on this model, when calculating the loss, the model parameters are of type int, and float32 is used when updating the parameters, so that the accuracy loss in the quantization process can be reduced. Its training loss function is similar to formula (4), that is, the soft label (prediction value) of the teacher model and the loss value Hard_Loss_T of the real label of the student's classroom behavior are normalized using the inverse tangent function Atan, and the ratio between the distillation loss (Distillation Loss) and the prediction result of the student model and the loss value Hard_Loss_S of the real label is dynamically adjusted, and then the two are added to obtain the final student classroom behavior recognition knowledge distillation model to update the final loss function value, thereby reducing the impact of the prediction error of the student classroom behavior recognition teacher model on the performance of the quantization model during the knowledge distillation process.

[0057] Students’ concentration is evaluated from both positive and negative aspects: positively, it is calculated by counting the rate at which students raise their heads, and its calculation formula is shown in Formula 7; negatively, it is calculated by counting the number of students playing with their mobile phones or lying on the desks, and its calculation formula is shown in Formula 8.

[0058] This embodiment specifically assists in evaluating the quality of classroom teaching from two aspects: classroom interactivity and student concentration. Classroom interactivity can reflect whether classroom teaching is lively, whether teachers and students are deeply involved in classroom teaching, whether the teacher's teaching method can stimulate students' curiosity, etc. Student concentration can more intuitively reflect the degree of students' interest in the classroom and reflect classroom discipline, which can provide a direct reference for improving the quality of classroom teaching. This embodiment further sets corresponding quantitative indicators from two aspects: classroom interactivity and student concentration. Classroom interactivity can be evaluated by statistics such as hand-raising behavior, and student concentration can be evaluated from both positive and negative sides. The positive side can be evaluated by counting positive indicators such as student head-up rate, and the negative side can be evaluated by counting negative behaviors such as students playing with mobile phones and lying on the table. That is, the evaluation indicators include classroom interactivity indicators and student concentration indicators. The classroom interactivity indicator is calculated using the statistical results of the student hand-raising behavior recognition results, and the student concentration indicator is a positive indicator calculated using the statistical results of the student hand-raising behavior recognition results, and a negative indicator calculated using the statistical results of the head-up rate recognition results.

[0059] Optionally, the classroom interactivity index may be calculated using the following expression: (6) The student concentration index can be calculated using the following expression: (7) (8) in, Indicates the classroom interactivity index. Indicates the positive indicators of student concentration. It represents the negative indicator of student concentration. N represents the total number of visual inspections in the classroom; represents the total number of pedestrians detected during the i-th detection; represents the total number of times the hand-raising behavior is detected during the i-th detection; The total number of times the head-up behavior was detected during the i-th detection; , They represent the total number of times playing with mobile phone and lying on the table were detected during the i-th detection.

[0060] Among the above indicators, , , The value range is [0,1]. The larger the value, the better the classroom teaching quality. It is understandable that other indicators and calculation methods can also be used to quantify classroom teaching quality.

[0061] This embodiment also provides an edge device, including a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0062] like Figure 2 As shown, the classroom teaching quality auxiliary evaluation system based on cloud-edge collaboration in this embodiment includes: Video image acquisition equipment, used to obtain video image data of students in the classroom captured by the camera in real time and transmit it to the edge device; The edge device, such as the one mentioned above, is used to receive the video image data transmitted in real time by the video image acquisition device, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to the cloud video server; The cloud video server is used to receive data transmitted by edge devices in real time, save it, and send it to mobile terminals.

[0063] It is understandable that the above system may be further provided with other auxiliary devices to meet different requirements.

[0064] It is understandable that the above method of this embodiment can be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0065] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration, characterized in that the steps include: The edge device receives the video image data of students in the classroom collected by the camera in real time; In the edge device, the video image data received in real time is used to identify the student classroom behavior using a pre-trained student classroom behavior recognition model, and each detected detection frame is tracked. During the tracking process, the recognition results of multiple adjacent frames are fused with the recognition results of the current frame to obtain the final recognition result of the current frame. The evaluation index is calculated based on the student classroom behavior recognition results to evaluate the classroom teaching quality, and the student classroom behavior recognition results and evaluation results are sent to the cloud video server for storage; The student classroom behavior recognition model is pre-trained based on a multi-label target detection algorithm and adopts a knowledge distillation structure.

2. According to claim 1, the classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration is characterized in that: The student classroom behavior recognition model is constructed based on YOLOV8, and the convolution operation in the BackBone part of YOLOV8 uses the RFAConv operator. The expression of the RFAConv operator is: in, represents the RFAConv operator, represents a grouped convolution of size 1×1, represents a grouped atrous convolution of size k×k, k represents the size of the convolution kernel, and Norm represents normalization. X represents the input feature map, represents the activation function, represents the Softmax function, It represents soft pooling and is calculated as: in, R is the local area selected by the box. Indicates the result after soft pooling. , , Respectively represent the area R Middle k, j The feature value at each position.

3. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 1 is characterized in that: When using the student classroom behavior recognition model to identify student classroom behavior, the classification function adopts the following class weighted focal loss function: in, is the weight factor of the classroom behavior of the i-th student, , m is the category of students’ classroom behavior, is the number of positive samples of the i-th type of student classroom behavior in the training set of the student classroom behavior dataset, is the predicted value of the classroom behavior of the i-th class of students, is the Sigmoid function, is the true label value of the classroom behavior of the i-th student, is the balance factor, is the weight factor.

4. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 1 is characterized in that: The steps of training the student classroom behavior recognition model using the knowledge distillation structure include: Configure the first pre-training weight; Using the first pre-training weights to train the student classroom behavior recognition model to obtain second pre-training weights; The second pre-training weight is used to fine-tune the student classroom behavior recognition model based on the knowledge distillation structure to obtain the third pre-training weight; The third pre-trained weight is used to adopt a quantitative perceptual training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model to obtain the final quantitative weight of the student classroom behavior recognition model. In the quantitative perceptual training method based on knowledge distillation, the student model adopts a quantitative perceptual model to update parameters.

5. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 4 is characterized in that: When fine-tuning the student classroom behavior recognition model using the knowledge distillation structure, the knowledge distillation model is updated using the following loss function: in, represents the loss function between the predicted value of the teacher model for student classroom behavior recognition and the predicted value of the Student model for student classroom behavior recognition, represents the loss function between the predicted value of the student model for identifying student classroom behavior and the true label of the student classroom behavior, and x represents the loss value between the predicted value of the teacher model for identifying student classroom behavior and the true label of the student classroom behavior.

6. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to claim 5 is characterized in that: The process of using the third pre-training weights to adopt a quantitative perception training method based on knowledge distillation to perform quantitative training on the fine-tuned student classroom behavior recognition model also includes: , The proportion of the teacher model is dynamically adjusted. When the prediction accuracy of the student classroom behavior recognition teacher model is higher than that of the student model, the The proportion of The proportion of 's proportion.

7. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to any one of claims 1 to 6 is characterized in that: The step of fusing the recognition results of the adjacent multiple frames with the recognition result of the current frame to obtain the final recognition result of the current frame during the tracking process includes: Get a detection frame of the current frame as the current detection frame, and get the current detection frame and the corresponding behavior recognition result; Determine whether the current detection frame is tracked successfully; If the current tracking is unsuccessful, the current recognition result is directly used as the final recognition result of the current detection frame; If the current tracking is successful, the action recognition results of the current detection frame and the adjacent multi-frame data are counted, and the action recognition results of the current detection frame are fine-tuned using the statistical results to obtain the final recognition result of the current detection frame.

8. The classroom teaching quality auxiliary evaluation method based on cloud-edge collaboration according to any one of claims 1 to 6 is characterized in that: The evaluation index includes a classroom interactivity index and a student concentration index. The classroom interactivity index is calculated using the statistical results of the student hand-raising behavior recognition results. The student concentration index is a positive index calculated using the statistical results of the student hand-raising behavior recognition results, and a negative index calculated using the statistical results of the head-up rate recognition results. The calculation expression of the classroom interactivity index is: The calculation expression of the student concentration index is: in, Indicates the classroom interactivity index. Indicates the positive indicators of student concentration. It represents the negative indicator of student concentration. N represents the total number of visual inspections in the classroom; represents the total number of pedestrians detected during the i-th detection; represents the total number of times the hand-raising behavior is detected during the i-th detection; The total number of times the head-up behavior was detected during the i-th detection; , They represent the total number of times playing with mobile phone and lying on the table were detected during the i-th detection.

9. An edge device, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

10. A classroom teaching quality auxiliary evaluation system based on cloud-edge collaboration, characterized in that: include: Video image acquisition equipment, used to obtain video image data of students in the classroom captured by the camera in real time and transmit it to the edge device; The edge device as claimed in claim 9 is used to receive video image data transmitted in real time by a video image acquisition device, evaluate the quality of classroom teaching, and transmit the student classroom behavior recognition results and evaluation results to a cloud video server; The cloud video server is used to receive data transmitted by edge devices in real time, save it, and send it to mobile terminals.

Images