A fine-grained teaching behavior recognition method for smart classrooms

Abstract

This invention discloses a fine-grained teaching behavior recognition method for smart classrooms. Based on YOLOv8, this method proposes a lightweight teaching behavior detection model that integrates contextual fine-grained features. To enhance the model's detection capability in the specific and complex application scenario of smart classrooms, this invention designs a fine-grained feature extraction module, combining deep convolution with an efficient multi-scale attention mechanism to effectively extract fine-grained features while optimizing the utilization efficiency of computational resources. Addressing the issue of large target scale spans due to the depth of the classroom environment, a multi-scale feature fusion module is designed, effectively improving the model's ability to detect actions of different scales, especially demonstrating outstanding performance in dense scenes. This invention, through the design of a fine-grained feature extraction module, a multi-scale feature fusion module, and a fine-grained classroom teaching behavior detection module specifically designed for small targets, improves detection accuracy while maintaining detection speed.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology and relates to a fine-grained teaching behavior recognition method for smart classrooms. Background Technology

[0002] In smart classroom teaching environments, real-time monitoring of student behavior is crucial for establishing a positive learning relationship. However, due to the complexity of the classroom environment, the large number of students, and the excessive depth of the classroom, current behavior detection algorithms suffer from shortcomings in accuracy and real-time performance. Existing classroom behavior detection methods primarily rely on computer vision and machine learning techniques, analyzing video streams or image data to identify student actions. Common detection algorithms include pose estimation and keypoint detection, action recognition based on convolutional and temporal models, and ensemble learning and multi-model fusion.

[0003] While these methods often perform well in laboratory environments or specific scenarios, they are difficult to directly apply to teaching behavior recognition in smart classrooms due to challenges such as multi-target detection, frequent changes in actions, and the preservation of fine-grained features. With advancements in artificial intelligence, utilizing convolutional neural networks (CNNs) and temporal models for student behavior detection has become a trend. These methods demonstrate great potential in improving detection accuracy and adapting to complex environments. Current target detection methods are mainly divided into two categories: single-stage detection methods and multi-stage detection methods.

[0004] Single-stage detection methods directly extract features from images and predict the location and category of targets, eliminating the need for candidate region generation. Another innovative single-stage method is DETR (Detection Transformer), proposed by Nicolas Carion et al. of Facebook AIREsearch. This method utilizes the Transformer architecture to achieve end-to-end target detection without additional region proposal or post-processing steps. However, these methods perform poorly in detecting small targets, especially in complex backgrounds or scenes with dense objects. Accuracy can easily decrease due to indistinct object features, small target size, or overlapping adjacent objects, leading to missed detections and false detections, making it difficult to meet the accuracy requirements of classroom detection.

[0005] Multi-stage detection methods typically include two stages: candidate region generation and subsequent fine-tuning. While these methods offer improved accuracy compared to single-stage methods, they suffer from slower detection speeds and higher complexity, making them unsuitable for the real-time requirements of classroom action detection. Object detection faces numerous challenges in classroom behavior analysis, particularly when dealing with small objects and complex backgrounds. Traditional object detection algorithms often perform poorly in these situations, leading to significant missed and false detections.

[0006] To address these issues, the YOLO series, as a highly efficient object detection algorithm, has gradually attracted researchers' attention, especially YOLOv8, which achieves a good balance between detection accuracy and speed. As a crucial node in the YOLO series of object detection algorithms, YOLOv8 significantly improves accuracy and speed while maintaining high detection efficiency. It performs excellently in handling targets of various scales and possesses strong real-time performance, making it suitable for applications with high speed requirements. However, YOLOv8 still has some shortcomings in practical applications, especially when dealing with small targets and complex backgrounds, where accuracy often falls short of expectations. In the classroom teaching scenario addressed in this invention, this is specifically manifested in its inability to detect small targets in the back row, resulting in frequent missed and false detections, ultimately leading to poor small target detection performance, with missed and false detection problems being particularly prominent. Summary of the Invention

[0007] The purpose of this invention is to propose a fine-grained teaching behavior recognition method for smart classrooms, which can better adapt to complex scenarios in the classroom environment while maintaining high efficiency and providing more accurate behavior detection results.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A fine-grained method for recognizing teaching behaviors in smart classrooms includes the following steps:

[0010] Step 1. Obtain the classroom teaching video data stream and construct the dataset for training the following model;

[0011] Step 2. Build a teaching behavior detection model based on the improved YOLOv8 model. The model includes a feature extraction layer, a feature fusion layer, and a detection head, which has four detection heads.

[0012] First, video frames are input to the feature extraction layer for feature extraction. The processing procedure is as follows:

[0013] In the feature extraction layer, the video frame first goes through a convolution module, then through multiple sets of convolution and fine-grained feature extraction modules for full feature extraction, and finally through fast pyramid pooling for feature enhancement to complete feature extraction.

[0014] Then, the feature fusion layer combines multiple multi-scale feature fusion modules, multiple upsampling modules, multiple connection modules, and multiple convolutional modules to perform feature fusion at different depths;

[0015] Finally, features at different depths are processed by four detection heads to obtain detection results.

[0016] Step 3. Train the teaching behavior detection model based on the dataset from Step 1, and then use the trained model to detect video images in real classroom scenarios to obtain student behavior detection results.

[0017] Furthermore, based on the aforementioned fine-grained teaching behavior recognition method for smart classrooms, this invention also proposes a computer device comprising a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the aforementioned fine-grained teaching behavior recognition method for smart classrooms.

[0018] Furthermore, based on the aforementioned fine-grained teaching behavior recognition method for smart classrooms, this invention also proposes a computer-readable storage medium on which a program is stored.

[0019] When executed by the processor, this program is used to implement a fine-grained method for recognizing teaching behaviors in smart classrooms.

[0020] The present invention has the following advantages:

[0021] As described above, this invention discloses a fine-grained teaching behavior recognition method for smart classrooms. Based on YOLOv8, this method proposes a lightweight teaching behavior detection model, the EDU-YOLO model, that integrates fine-grained contextual features to improve the model's detection accuracy and real-time performance in complex classroom scenarios. To enhance the model's detection capabilities in the specific and complex application scenario of smart classrooms, this invention designs a fine-grained feature extraction module, combining deep convolution and EMA attention mechanisms to effectively extract fine-grained features while optimizing the utilization efficiency of computational resources. Furthermore, addressing the issue of large target scale spans due to the depth of the classroom environment, this invention designs a multi-scale feature fusion module, effectively improving the model's ability to detect actions of different scales, especially excelling in dense scenes. The method of this invention outperforms existing detection algorithms in terms of accuracy and real-time performance, particularly in multi-target detection and small target recognition tasks. By designing a fine-grained feature extraction module, a multi-scale feature fusion module, and a fine-grained classroom teaching behavior detection module specifically for small targets, this invention improves detection accuracy while maintaining detection speed. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the overall processing flow of the fine-grained teaching behavior recognition method for smart classrooms in this embodiment of the invention.

[0023] Figure 2 This is a structural diagram of the EDU-YOLO teaching behavior detection model built in this embodiment of the invention;

[0024] Figure 3 This is a structural diagram of the fine-grained feature extraction (FGE) module in an embodiment of the present invention;

[0025] Figure 4 This is a structural diagram of the Bottleneck module in an embodiment of the present invention;

[0026] Figure 5 This is a structural diagram of the multi-scale feature fusion (MSE) module in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the fine-grained classroom teaching behavior detection module (detection head) in an embodiment of the present invention;

[0028] Figure 7 This is a data distribution diagram in an embodiment of the present invention;

[0029] Figure 8 This is a comparison chart of the experimental results of the model on POCO in the embodiments of the present invention;

[0030] Figure 9 This is a comparison diagram of the confusion matrices of EDU-YOLO and YOLOv8 on UOCO in an embodiment of the present invention;

[0031] Figure 10 This is a comparison chart showing the detection results of EDU-YOLO on the dataset in an embodiment of the present invention.

[0032] Figure 11 This is a comparison diagram of the detection results of EDU-YOLO in a real scene in an embodiment of the present invention. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] With the deepening integration of educational informatization and artificial intelligence technologies, real-time monitoring and evaluation of student classroom behavior has become an important direction for the development of intelligent education. However, due to the spaciousness and depth of university classrooms, the detection accuracy of traditional target detection algorithms has decreased, especially since small targets are easily overlooked, while large targets lead to redundant calculation problems.

[0035] like Figure 1 As shown, in response to the technical challenges of student classroom behavior detection in smart teaching environments, this invention proposes a fine-grained teaching behavior recognition method for smart classrooms. The method proposes an improved target detection model, EDU-YOLO, which introduces a fine-grained feature extraction module (FGE), a multi-scale feature fusion module (MSE), and a small target detection layer, thereby improving the detection accuracy and robustness of the model in cross-scale and fine-grained environments.

[0036] First, the FGE module combines a deep convolutional network with an efficient multi-scale attention module (EMA) to effectively extract fine-grained details of student behavior, enhancing the ability to recognize diverse actions. Then, multi-scale feature fusion technology improves the fusion performance of features at different scales, particularly in dense classroom scenarios, enhancing the accurate recognition of fine-grained target behaviors. Finally, the small target detection layer optimizes the capture of long-distance and minute movements, effectively improving overall detection accuracy.

[0037] Compared to traditional algorithms such as Faster R-CNN and SSD, this invention employs multi-level feature fusion to fully utilize feature information at different scales, enhancing the detection capability for complex situations and small targets. Structurally, by introducing a fine-grained feature extraction module (FGE) and a multi-scale feature fusion module (MSE), the network's receptive field and feature representation capability are improved while conserving computational resources, enabling the model to accurately and quickly detect targets even in complex backgrounds. Furthermore, the introduction of a small target detection layer significantly reduces the false negative rate and improves detection performance in dense scenes.

[0038] like Figure 1 As shown, the fine-grained teaching behavior recognition method for smart classrooms in this embodiment includes the following steps:

[0039] Step 1. Acquire the classroom teaching video data stream and segment it into frames. Then, use an annotation tool (such as MakeSense) to build a dataset for training the model described below. Each image in the dataset is carefully annotated to ensure the accuracy of the action categories.

[0040] The image labels comprise ten categories: raisehead, bowhead, turnhead, phone, playphone, book, lookbook, sleep, raise hand, and bend. The entire dataset is divided into three parts: training set, test set, and validation set.

[0041] Step 2. Build the EDU-YOLO teaching behavior detection model based on the improved YOLOv8 model. Its overall structure is as follows: Figure 1 As shown, the model consists of four parts: an input layer, a feature extraction layer, a detection layer, and an output layer.

[0042] The EDU-YOLO model significantly improves detection accuracy by designing a more efficient feature extraction and fusion mechanism, particularly enhancing small targets and subtle movements in classroom behavior detection.

[0043] Specifically, in classroom behavior detection, due to the considerable depth of the classroom, students at the front of the frame (closest to the camera) occupy a larger pixel area in the captured image, forming larger target objects. Conversely, students at the back of the frame (farthest from the camera) occupy a smaller pixel area due to perspective distortion and increased distance, becoming smaller target objects. This significant difference in target scale caused by spatial location greatly increases the difficulty and complexity of target detection algorithms, ultimately leading to the model's inability to effectively detect students at the back of the frame.

[0044] By introducing the FGE module, MSE module, and small target layer of this invention, EDU-YOLO can better adapt to complex scenarios in the classroom environment while maintaining high efficiency, and provide more accurate behavior detection results.

[0045] The network structure of the EDU-YOLO model is as follows: Figure 2 As shown in the diagram, to address the issue of low accuracy in classroom behavior detection, this invention designs an FGE module while considering real-time requirements. This module effectively extracts cross-scale behavioral features by combining a cross-stage partial fusion module (C2f) with an efficient multi-scale attention module (EMA). Then, an MSE module enhances the ability to fuse and utilize key behavioral features at different scales, improving the model's detection performance in complex classroom scenarios. Furthermore, to address the ineffective detection of small-scale targets, a fine-grained classroom behavior detection module is specifically designed to optimize the recognition of behaviors of distant and small targets, effectively improving overall detection accuracy and meeting the practical needs of classroom behavior detection.

[0046] like Figure 2 As shown, the EDU-YOLO model built in this embodiment includes a feature extraction layer, a feature fusion layer, and four detection heads. The overall processing flow of the EDU-YOLO model is as follows:

[0047] First, the input video frames are fed into the feature extraction layer for feature extraction. In this layer, the video frame image first passes through a convolutional module, then through multiple convolutional and fine-grained feature extraction modules for thorough feature extraction, and finally undergoes fast pyramid pooling for feature enhancement. Next, the feature fusion layer combines multiple multi-scale feature fusion modules with upsampling modules, and multiple multi-scale feature fusion modules with convolutional modules, to perform feature fusion at different depths. Finally, the features at different depths are processed by four detection heads to obtain the detection results.

[0048] The feature extraction layer includes a convolutional module, four sets of convolutional and fine-grained feature extraction modules, and a fast pyramid pooling (SPPF) module. The four sets of convolutional and fine-grained feature extraction modules are defined as the first, second, third, and fourth sets of convolutional and fine-grained feature extraction modules, respectively. Each set contains one convolutional module and one fine-grained feature extraction module (FGE).

[0049] The video frame first passes through a convolutional module for feature extraction, then passes through the first, second, third, and fourth convolutional modules and the fine-grained feature extraction module FGE for feature extraction in sequence, and finally passes through the fast pyramid pooling SPPF module for feature enhancement to complete the feature extraction.

[0050] There are seven multi-scale feature fusion modules (MSE), which are defined as the first, second, third, fourth, fifth, sixth, and seventh multi-scale feature fusion modules; there are three upsampling modules, which are defined as the first, second, and third upsampling modules.

[0051] There are three convolutional modules, which are defined as the first, second, and third convolutional modules; there are seven connection modules, which are defined as the first, second, third, fourth, fifth, sixth, and seventh connection modules, respectively.

[0052] The output features of the SPPF module are first connected to the output features of the fourth convolutional and fine-grained feature extraction (FGE) modules via a first connection module. The output features of the first connection module are then fed into the first upsampling module for upsampling.

[0053] The output features of the first upsampling module are connected to the output features of the FGE module in the third set of convolution and fine-grained feature extraction modules via the second connection module. The output features of the second connection module are then fed into the second upsampling module for upsampling.

[0054] The output features of the second upsampling module are connected to the output features of the FGE module in the second set of convolution and fine-grained feature extraction modules via the third connection module. The output features of the third connection module are then fed into the third upsampling module for upsampling.

[0055] The output features of the third upsampling module are connected to the output features of the FGE module in the first group of convolution and fine-grained feature extraction modules via the fourth connection module. The output features of the fourth connection module are then connected to the first multi-scale feature fusion module and the first convolution module in sequence.

[0056] The output features of the first convolutional module enter the fifth connection module; the output features of the third connection module enter the fifth connection module through a second multi-scale feature fusion module, and are then connected with the output features of the first convolutional module.

[0057] The output features of the fifth connection module are sequentially passed through the third multi-scale feature fusion module and the second convolution module.

[0058] The output features of the second convolutional module enter the sixth connection module; the output features of the second connection module pass through a fourth multi-scale feature fusion module and enter the sixth connection module, where they are connected with the output features of the second convolutional module.

[0059] The output features of the sixth connection module are sequentially passed through the fifth multi-scale feature fusion module and the third convolution module.

[0060] The output features of the third convolutional module enter the seventh connection module; the output features of the first connection module enter the seventh connection module through a sixth multi-scale feature fusion module, and are then connected with the output features of the third convolutional module.

[0061] The output features of the seventh connection module are processed by the seventh multi-scale feature fusion module.

[0062] The first, third, fifth, and seventh multi-scale feature fusion modules are each connected to a detection head.

[0063] EMA preserves information for each channel and improves pixel-level feature representation without significantly increasing computational burden. This module divides the channel dimension of the input feature map into multiple sub-feature groups and evenly distributes spatial semantic features within each group. The EMA module features two parallel branches: one uses 1×1 convolutions to encode channel attention, and the other uses 3×3 convolutions to capture multi-scale feature representations. The outputs of these two branches are further aggregated through a cross-dimensional interaction method to enhance feature interaction and information fusion. Experiments on widely used benchmark datasets such as CIFAR100, MSCOCO, and VisDrone2019 demonstrate that the efficient multi-scale attention module EMA exhibits superior performance in both image classification and object detection tasks. EMA not only significantly improves model accuracy while maintaining computational efficiency but also adapts well to network architectures of varying depths.

[0064] Based on the design concept of EMA, this invention designs a fine-grained feature extraction module (FGE), which combines the lightweight advantages of the C2f module with the refined feature extraction capability of the EMA attention mechanism to enhance the feature representation capability of the network. The C2f module has strong feature reuse capability, which improves the generalization ability of the model while maintaining computational efficiency. Meanwhile, the multi-scale feature capture capability of EMA enables it to more accurately identify diverse behaviors, especially performing well in detecting small targets in complex scenes.

[0065] like Figure 3 As shown, the processing flow of the fine-grained feature extraction (FGE) module is as follows:

[0066] The fine-grained feature extraction module (FGE) first compresses the input features through convolutional (Conv) operations and then divides the features into two branches using the split function. This branching design enhances the network's nonlinearity and expressive power, thereby improving its ability to model complex data. One branch remains unprocessed, while the other branch contains multiple Bottleneck modules (Bneck modules) for deep feature extraction, enhancing the network's overall feature extraction capability. In the diagram, 'n' represents the number of repetitions of the Bneck module. These Bneck modules transmit feature information through parallel paths and concatenate the outputs of the two branches through a connection operation. Next, a portion of the resulting feature group is fused with the original features after passing through the EMA attention mechanism, resulting in a new feature group. Finally, the features are fused with the convolutional (Concat) structure and output as features. This structure incorporates an efficient multi-scale attention mechanism to dynamically adjust the weights between channels, highlighting key features while saving computational resources. The efficient multi-scale attention module (EMA) helps the network focus on important channels, thereby enhancing the model's detection capability in complex scenes. Furthermore, this module preserves gradient information through residual connections, ensuring that important features are effectively transmitted in deep networks. This design ensures that rich gradient flow information can still be obtained after multiple cross-layer connections, enhancing the ability to capture small targets and fine-grained features.

[0067] like Figure 4 The structure of each Bottleneck module is shown. Features input to the Bottleneck module are passed through parallel paths, with the main branch path passing through two convolutional modules. The output features of the main branch path are residually connected to the original input features via a connection module. The specific computation process of the fine-grained feature extraction (FGE) module is as follows:

[0068] First, the features of the input fine-grained feature extraction (FGE) module are processed. The input features are denoted as X∈RC×H×W, where C, H, and W represent the number of channels, height, and width, respectively.

[0069] First, the input features are processed through a single convolution operation to obtain the features. Figure X Features processed by convolution Figure X Then, through the channel splitting operation, it is divided into two parts, namely:

[0070] X′=Conv1(X);

[0071] X1,X2=Split(X′).

[0072] Conv1 is a standard convolutional layer used for initial feature extraction. X1 and X2 represent the two branches of the input feature X after segmentation, which are then processed by different paths. Next, the segmented X2 is fed into multiple Bottleneck layers to further extract deeper features, resulting in X2′. After feature extraction in the Bottleneck layers, X1 and X′2 are concatenated using a Concat operation to generate a new feature map Y1, Y2, ..., Y2′. m The formula is as follows:

[0073] X′2=Bottleneck n (X2);

[0074] Y1,Y2,…,Y m =Concat(X1, X′2);

[0075] Where n is the number of times the Bneck module is repeated, and in this embodiment, n is 2.

[0076] Next, feature Y m The efficient multi-scale attention module (EMA) is input and its results are added to the original feature set. The EMA dynamically weights features across channels, highlighting channel features crucial for the detection task. Finally, all features in the resulting feature set are concatenated to obtain a processed feature map, which is then passed through a convolutional network to obtain the final feature map Y. Through this series of processing steps, the module's output and input sizes remain consistent.

[0077] Y′=Concat(Y1,Y2,…,EMA(Y m ));

[0078] Y = Conv2(Y′).

[0079] EMA is the attention mechanism function, which is responsible for assigning different weights to channels to emphasize important features.

[0080] The Multi-scale Feature Fusion (MSE) module is used to enhance and fuse features at different scales. It combines a multilayer perceptron, a random dropout strategy, and an efficient multi-scale attention mechanism to fuse and enhance input features. This design makes the connections between features closer, enhancing the network's ability to fuse and utilize behavioral detail features at different scales. Especially in complex multi-object environments, it can improve the accuracy of object detection.

[0081] like Figure 5 The structure of the multi-scale feature fusion module (MSE) is shown. The processing flow of the MSE module is as follows:

[0082] First, the input features are dynamically adjusted by the AD module to meet the model's computational requirements. Then, they are compressed by a partial convolutional network P_Conv, producing compressed feature maps. Next, these features undergo a linear transformation using a multilayer perceptron (MLP) with two convolutional layers to further extract features. Subsequently, a DropPath strategy is introduced to enhance the model's robustness and reduce overfitting. The efficient multi-scale attention module EMA in this module further highlights important features by dynamically adjusting the correlation between channels. The partial convolutional networks in this module use a specified block-based spatial mixing method to achieve adaptive learning of local features. With the help of residual connections, the output of the data after passing through the MLP module and the efficient multi-scale attention module EMA is fused with the input features, ensuring efficient information transfer. In summary, this structure, through the combination of residual connections, efficient multi-scale attention mechanisms, the dropPath strategy, and the multilayer perceptron, effectively improves the model's ability to focus on important features while enhancing its generalization ability. The specific calculation process of the multi-scale feature fusion module MSE is as follows:

[0083]

[0084] If the input channels do not match the desired output channels, use a 1×1 convolution to adjust them.

[0085] X min =P_Conv(X adj );

[0086] Y mlp =Conv(ReLU(Conv(X) min )));

[0087] Y attn =EMA(DropPath(Y mlp ));

[0088] Y final =X adj +Y attn .

[0089] Where X is the input feature map, X adj This is the feature map after channel adjustment. When the number of input channels (inc) does not match the number of target channels (dim), a 1×1 convolutional layer is used to adjust the number of channels. minIt is a feature map obtained after spatial mixing, processed by a custom P_Conv layer. ReLU is the activation function, and DropPath is a function that discards a portion of the path in the feature map with a certain probability (set to 0.1) to prevent overfitting. Efficient multi-scale attention (EMA) performs attention mechanism operations to enhance important parts of the feature map and suppress unimportant parts. mlp Y attn Y final These are the Multilayer Perceptron (MLP), the Efficient Multiscale Attention Module (EMA), and the Fusion Feature.

[0090] Furthermore, to address the challenge of small target detection in smart teaching scenarios, this invention designs a fine-grained classroom behavior detection module, namely the small target layer. This layer effectively improves the model's accuracy in identifying small targets by combining the proposed FGE and MSE modules for efficient feature extraction and fusion. In complex backgrounds and dense scenes, small targets are often overlooked due to their indistinct features. This layer optimizes the network's receptive field while enhancing the model's ability to detect the behavior of students at a distance, thereby reducing missed detections. Its representation is shown in the figure below. Figure 6 As shown.

[0091] This fine-grained classroom behavior detection module is particularly crucial in the network structure, and its specific structure is as follows: Figure 2 This invention employs a fine-grained feature extraction module, a multi-scale feature fusion module, and a small target detection head to fully extract, fuse, and detect fine-grained features of small-scale targets. This design ensures that the feature map retains more fine-grained details after passing through the fine-grained feature extraction module. Simultaneously, the multi-scale feature fusion module enhances and fuses these fine-grained features, thereby improving the detectability of small targets. Subsequently, the fused fine-grained features are detected by the small target detection head, yielding the detection result. In dense classroom scenarios, distant small targets are often difficult to detect due to feature ambiguity. This small target detection layer, through fine-grained feature extraction and multi-scale feature enhancement and fusion, significantly reduces missed detections of small targets, thus improving the overall performance of the model in classroom behavior detection tasks. The EDU-YOLO model built in this invention not only improves the accuracy of classroom behavior detection but also provides teachers with a more effective teaching management tool. This tool helps teachers understand students' classroom performance in a timely manner and develop more targeted teaching strategies and feedback mechanisms based on data analysis results.

[0092] Step 3. Based on the dataset from Step 1, train the lightweight teaching behavior detection model that integrates fine-grained contextual features. Then, use the trained lightweight teaching behavior detection model to detect video images in real classroom scenarios to obtain classroom behavior detection results in the form of images with confidence scores.

[0093] Furthermore, to verify the effectiveness of the proposed method, comparative experiments were conducted with several mainstream target detection algorithms to comprehensively evaluate its performance across different metrics. Analysis of the results from the comparative and ablation experiments will further validate the model's effectiveness and the improvement measures' effectiveness in enhancing detection accuracy.

[0094] To ensure the fairness of the experiment, all uses in this invention were performed on a Linux system equipped with a 13th Gen Intel® Core™ i9-13900K CPU and dual NVIDIA GeForce RTX 4090 GPUs.

[0095] The parameters were set as follows: Intersection over Union (IoU) of 0.7, initial learning rate (lr0) of 0.01, learning rate decay rate (lrf) of 0.01, optimizer weight decay rate (weightdecay) of 0.0005, batch size of 16, number of training epochs of 500, and image input size (imgsz) of 640. The original size of the input image was 1920*1080.

[0096] Evaluation metrics include mean average precision (mAP), precision (P), recall (R), F1 score, and intersection over union (IoU).

[0097] Precision represents the proportion of targets that are actually predicted to be positive out of all targets predicted as positive. The formula is:

[0098]

[0099] TP (TruePositive) represents true positives, indicating the number of targets correctly detected by the model, i.e., the detected target boxes and ground truth boxes meet the IoU threshold requirement (usually 0.5).

[0100] FP (False Positive) represents the number of targets that the model incorrectly detects, i.e., the IoU between the detected target box and the ground truth box is less than a set threshold, or the corresponding target does not exist in the real scene.

[0101] Recall rate represents the proportion of all positive samples that are correctly detected. The formula is:

[0102]

[0103] FN (False Negative) represents the number of real targets that the model failed to detect, meaning that the targets actually exist, but the model failed to identify them.

[0104] mAP is the most commonly used evaluation metric in object detection. It is a weighted sum of the average accuracy of various target types and can be expressed as a formula. mAP is usually divided into mAP 0。5 and mAP 0。5:0。95 :

[0105]

[0106] Where AP(j) represents the AP of the j-th category. Specifically, mAP 0。5 mAP refers to the average accuracy when the IoU (Intersection over Union) is 0.5, with particular attention to the model's performance when the predicted bounding box has a high overlap with the ground truth bounding box. 0。5:0。95 It is a more stringent evaluation standard, representing the average accuracy calculated with IoU ranging from 0.5 to 0.95 and a step size of 0.05, which better reflects the model's overall performance in terms of detection box localization accuracy.

[0107] The F1 score is the harmonic mean of precision and recall, used to balance the evaluation criteria of precision and recall.

[0108]

[0109] Intersection over Union (IoU) is a metric commonly used in object detection to measure the degree of overlap between predicted bounding boxes and actual labeled bounding boxes. The calculation formula is:

[0110]

[0111] Where Aoverlap represents the area of ​​the overlapping region between the predicted bounding box and the ground truth bounding box. Aunion represents the area of ​​the joint region between the predicted bounding box and the ground truth bounding box, which is the total area of ​​the two boxes minus the area of ​​the overlapping portion.

[0112] The comparative experiment is as follows: The POCO dataset collects real-life postures of university students in the classroom under different environments, classrooms, subjects, number of students, and camera angles. It includes very complex teaching scenarios, providing strong support for research on student states in the classroom. It contains ten categories: raisehead, bowhead, turnhead, phone, playphone, book, lookbook, sleep, raise hand, and bend. The entire dataset is divided into three parts: training set, test set, and validation set.

[0113] The training set contains 1,146 RGB images and 83,566 labels, the test set contains 378 RGB images and 26,909 labels, and the validation set contains 379 RGB images and 27,485 labels.

[0114] Figure 7 (a) shows the data distribution on the POCO dataset. To verify the effectiveness of the method, Faster R-CNN, Single-Stage Detector (SSD), Residual Network (Res2Net), Transformer-based Object Detection (DETR), YOLOv8n, YOLOv10n, YOLO11n, and RE-DETR were selected as comparison models and validated on the POCO dataset. The experimental results are shown in Table 1.

[0115] Table 1 Results of POCO comparative test

[0116]

[0117] The experimental results show that the method of this invention exhibits significant improvements in all metrics. Specifically, the EDU-YOLO model in this invention improves accuracy, recall, and mAP. 0。5 and mAP 0。5:0。95 The performance of EDU-YOLO surpasses existing YOLO models in all aspects. Firstly, in terms of precision, EDU-YOLO achieves 0.856, significantly higher than other models, especially compared to YOLOv8 and YOLOv10, representing improvements of 2.3% and 5.0% respectively. It also improves by 1.7% compared to the state-of-the-art YOLOv11 model. This improvement indicates that the method of this invention performs exceptionally well in reducing false positives. Secondly, in terms of recall, EDU-YOLO achieves a high score of 0.797, exceeding other models, especially compared to YOLOv8's 0.721 and YOLOv10's 0.730, showing a significant improvement. These results demonstrate that the method of this invention has a low false negative rate and strong target detection capability. In the most important area, mAP... 0。5 and mAP 0。5:0。95 On the above, EDU-YOLO achieved scores of 0.849 and 0.603 respectively, especially in mAP. 0。5:0。95 The above is a significant improvement compared to YOLOv8's 0.537 and YOLO11's 0.540, indicating that the method of the present invention has good generalization ability under different IoU thresholds.

[0118] Overall, the method of this invention effectively improves detection accuracy and recall by introducing the FGE module, MSE and small target detection layer, and further enhances the model's performance in complex scenarios.

[0119] Figure 8 This demonstrates the changing trends in evaluation metrics between the comparison model and the EDU-YOLO model of this invention during a 500-epoch training process. Figure 8 As shown in (a) and 8(b), in comparison, EDU-YOLO performs better in terms of accuracy (Precision) and mAP (mAP). 0。5 It performs better in this aspect, and its performance tends to stabilize after 100 epochs.

[0120] To further validate the performance of the proposed EDU-YOLO model, a new dataset, UOCO, was created, containing 300 images captured in real classroom environments. This dataset covers six specific action categories: playing phone, turning head, sleeping, standing, looking down, and raising head. The selection of these action categories aims to encompass common student behavior patterns in the classroom, providing rich labeled data to support model training and evaluation. Each image in this dataset has been carefully labeled to ensure the accuracy of the action categories. The dataset construction process followed strict standards to ensure that each action performs well under different environments and angles, reflecting the diversity of real classrooms. This diversity not only improves the model's adaptability to complex scenarios but also helps evaluate the model's generalization performance in different teaching contexts. Figure 7 (b) shows the label distribution in the UOCO dataset. The comparative experimental results of different models on UOCO are shown in Table 2.

[0121] Table 2 Results of UOCO comparative test

[0122]

[0123] According to the experimental results, EDU-YOLO significantly outperformed the YOLO series comparison models YOLOv8n, YOLOv10n, and YOLO11n across all evaluation metrics. Firstly, in terms of precision, EDU-YOLO achieved a score of 0.632, significantly higher than YOLOv8's 0.554, YOLOv10's 0.543, and YOLO11's 0.528. This improvement indicates that EDU-YOLO performs exceptionally well in reducing false positives and can more accurately identify targets. Secondly, in terms of recall, EDU-YOLO also leads, achieving a high score of 0.562, compared to other models (YOLOv8's 0.528, YOLOv10's 0.486, and YOLO11's 0.511), demonstrating stronger target detection capabilities. These results indicate that the EDU-YOLO model can more effectively capture various student behaviors in complex classroom scenarios. (mAP) 0。5 and mAP 0。5:0。95 In terms of metrics, EDU-YOLO achieved scores of 0.552 and 0.256, respectively, both higher than the performance of the comparison models. This indicates that EDU-YOLO exhibits good generalization ability at different IoU thresholds and can adapt to various detection tasks. Furthermore, this invention introduces a confusion matrix between YOLOv8 and EDU-YOLO, such as... Figure 9 As shown in (a) and (b), the classification performance of the two methods on the UOCO dataset is compared more intuitively. Figure 9 It can be seen that EDU-YOLO is better at detecting small movements.

[0124] Furthermore, to verify the impact of each improved module of this invention on model performance, ablation experiments were conducted based on the baseline model. The Fine-Grained Feature Extraction (FGE) module, the Multi-Scale Feature Fusion (MSE) module, and the small target detection layer were gradually removed, and the changes in model performance were observed. Table 3 shows the performance of the model under different module combinations in terms of Precision, Recall, and mAP. 0。5 and mAP 0。5:0。95 Performance on indicators such as [list of indicators].

[0125] Table 3 Ablation Experiment Results

[0126]

[0127] The results in Table 3 show that each module in this invention contributes to improving model performance to varying degrees:

[0128] (1) Fine-grained feature extraction module (FGE): When the FGE module is removed, the precision of the model is 0.846 and the recall is 0.792, indicating that the false positive rate and the false negative rate have both increased. This is because this module is involved in the feature extraction process. Removing this module leads to insufficient feature extraction by the model, resulting in a lack of fine-grained features in the feature fusion and detection processes, which in turn leads to a decrease in model performance.

[0129] (2) Multi-scale Feature Fusion (MSE) Module: When the MSE module was removed, the model's accuracy was 0.845, and the other three metrics also decreased. This is because the features extracted during the feature extraction process were not effectively utilized, resulting in the inability to effectively detect targets containing fine-grained features and targets with complex features, ultimately leading to a decrease in accuracy. This indicates that the Multi-scale Feature Fusion (MSE) module plays a positive role in the feature fusion and utilization process.

[0130] (3) Fine-grained classroom behavior detection module (i.e., small target layer): When the small target detection layer is removed, all indicators of the model decrease significantly compared to the complete model, indicating that the model's detection performance for fine-grained small targets drops substantially. This proves that the module effectively improves the model's ability to detect small targets, especially with a significant improvement in detection accuracy in complex backgrounds.

[0131] In summary, the experimental results show that the FGE, MSE modules, and fine-grained classroom behavior detection module (small target layer) proposed in this invention all play a positive role in improving the model's performance, significantly enhancing the model's detection accuracy, recall, and generalization ability, especially in complex backgrounds and fine-grained target detection scenarios.

[0132] Furthermore, to more intuitively verify the detection performance of the EDU-YOLO model, this invention demonstrates the detection results in both datasets and real-world scenarios, and showcases a classroom teaching behavior analysis system. The detection results are presented in three parts: image detection results in the dataset, image detection results in real-world classroom scenarios, and more, as detailed below. Figure 10 As shown. In Figure 10 The images, from left to right, represent teaching scenarios in different classrooms and lessons. The first image shows the detection results of real-world student behavior collected from the POCO dataset. It can be seen that YOLOv8 missed detecting small, distant targets in this scenario and its detection of student behavior close to the camera was blurry, while EDU-YOLO accurately detected the actions of all students. The second image shows the accurate detection of multiple student behaviors by EDU-YOLO in complex scenes. Compared with other models, EDU-YOLO significantly reduced false positives and false negatives. Figure 11The image shows the detection results in a real classroom scenario. The top left, top right, bottom left, and bottom right corners correspond to the detection results of YOLOv8, YOLOv10, YOLO11, and EDU-YOLO, respectively. YOLOv8 and YOLOv10 performed poorly in small object detection, with some missed detections. EDU-YOLO, on the other hand, accurately captured the behavior of students at a distance, and was particularly good at distinguishing the different behaviors of multiple students in dense scenes.

[0133] This invention verifies the effectiveness and practicality of its method through experiments on the POCO dataset and a self-made UOCO dataset, as well as its application in real-world classroom scenarios. Experimental results show that EDU-YOLO achieves better performance in terms of precision, recall, and mAP. 0。5 and mAP 0。5:0。95 In key metrics, it outperforms existing YOLO models, especially in multi-object detection and small object recognition tasks, demonstrating significantly improved accuracy in detecting both large and small objects, particularly in complex classroom environments. Compared to the state-of-the-art YOLO11 model, it achieves improvements of 1.7%, 7.1%, 6.7%, and 6.3%, respectively. The EDU-YOLO algorithm significantly enhances the accuracy and real-time performance of student behavior detection by introducing innovative feature extraction and fusion mechanisms.

[0134] Example 2

[0135] This embodiment 2 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the fine-grained teaching behavior recognition method for smart classrooms described in embodiment 1 above.

[0136] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0137] Example 3

[0138] This embodiment 3 describes a computer-readable storage medium storing a program that, when executed by a processor, is used to implement the steps of the fine-grained teaching behavior recognition method for smart classrooms described in embodiment 1 above.

[0139] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0140] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A fine-grained teaching behavior recognition method for smart classrooms, characterized in that, Includes the following steps: Step 1. Obtain the classroom teaching video data stream and construct the dataset for training the following model; Step 2. Build a teaching behavior detection model based on the improved YOLOv8 model. The teaching behavior detection model includes a feature extraction layer, a feature fusion layer, and detection heads of different scales, of which there are four detection heads. First, video frames are input to the feature extraction module for feature extraction, as follows: In the feature extraction layer, the video frame image first passes through a convolution module, then through multiple sets of convolution and fine-grained feature extraction modules for full feature extraction, and finally through fast pyramid pooling for feature enhancement to complete feature extraction; Then, the feature fusion layer combines multiple multi-scale feature fusion modules, multiple upsampling modules, multiple connection modules, and multiple convolutional modules to perform feature fusion at different depths; Finally, features at different depths are processed by four detection heads to obtain detection results. The multi-scale feature fusion module includes a dynamic adjustment module (AD), a partially convolutional network (P_Conv), a multilayer perceptron (MLP), and an efficient multi-scale attention module (EMA). The processing flow of the multi-scale feature fusion module is as follows: First, the input features are dynamically adjusted by the AD module to meet the model's size calculation requirements. Then, they are processed by the partial convolutional network P_Conv to compress the features and generate compressed feature maps. Then, the features are linearly transformed by a multilayer perceptron (MLP) containing two convolutional layers to further extract features. Subsequently, the robustness of the model was enhanced and overfitting was reduced by using the DropPath strategy, which randomly discards paths. The efficient multi-scale attention module (EMA) is used to highlight important features by dynamically adjusting the correlation between channels; With the help of residual connections, the output features after passing through the MLP module and the efficient multi-scale attention module EMA are fused with the input features of the multi-scale feature fusion module to ensure efficient information transfer. Step 3. Train the teaching behavior detection model based on the dataset from Step 1, and then use the trained model to detect video images in real classroom scenarios to obtain the teaching behavior detection results.

2. The fine-grained teaching behavior recognition method for smart classrooms according to claim 1, characterized in that, The feature extraction layer includes a convolution module, four sets of convolution and fine-grained feature extraction modules, and a fast pyramid pooling module; the four sets of convolution and fine-grained feature extraction modules are defined as the first, second, third, and fourth sets of convolution and fine-grained feature extraction modules, and each set contains one convolution module and one fine-grained feature extraction module; The video frames first pass through a convolutional module to extract features, then pass through the first, second, third, and fourth sets of convolutional and fine-grained feature extraction modules in sequence to extract features, and finally pass through a fast pyramid pooling module to enhance features and complete feature extraction.

3. The fine-grained teaching behavior recognition method for smart classrooms according to claim 2, characterized in that, There are seven multi-scale feature fusion modules, which are defined as the first, second, third, fourth, fifth, sixth, and seventh multi-scale feature fusion modules. There are three upsampling modules, which are defined as the first, second, and third upsampling modules; There are three convolutional modules, which are defined as the first, second, and third convolutional modules; there are seven connection modules, which are defined as the first, second, third, fourth, fifth, sixth, and seventh connection modules, respectively. The output features of the fast pyramid pooling module are first connected with the output features of the fine-grained feature extraction module in the fourth group of convolution and fine-grained feature extraction modules through the first connection module; The output features of the first connection module are fed into the first upsampling module for upsampling. The output features of the first upsampling module and the output features of the fine-grained feature extraction module in the third group of convolution and fine-grained feature extraction modules are connected through the second connection module; The output features of the second connection module are fed into the second upsampling module for upsampling. The output features of the second upsampling module and the output features of the fine-grained feature extraction module in the second set of convolution and fine-grained feature extraction modules are connected through the third connection module; The output features of the third connection module are fed into the third upsampling module for upsampling. The output features of the third upsampling module and the output features of the fine-grained feature extraction module in the first group of convolution and fine-grained feature extraction modules are connected through the fourth connection module. The output features of the fourth connection module are sequentially passed through the first multi-scale feature fusion module and the first convolution module; The output features of the first convolutional module enter the fifth connection module; the output features of the third connection module pass through a second multi-scale feature fusion module and enter the fifth connection module, where they are connected with the output features of the first convolutional module. The output features of the fifth connection module are sequentially passed through the third multi-scale feature fusion module and the second convolution module; The output features of the second convolutional module enter the sixth connection module; the output features of the second connection module pass through a fourth multi-scale feature fusion module and enter the sixth connection module, where they are connected with the output features of the second convolutional module. The output features of the sixth connection module are sequentially passed through the fifth multi-scale feature fusion module and the third convolution module; The output features of the third convolutional module enter the seventh connection module; the output features of the first connection module pass through a sixth multi-scale feature fusion module and enter the seventh connection module, where they are connected with the output features of the third convolutional module. The output features of the seventh connection module are fed into the seventh multi-scale feature fusion module for processing. The first, third, fifth, and seventh multi-scale feature fusion modules are each connected to one of the detection heads.

4. The fine-grained teaching behavior recognition method for smart classrooms according to claim 1, characterized in that, The fine-grained feature extraction module combines the lightweight advantages of the C2f module with the refined feature extraction capability of the attention mechanism in the efficient multi-scale attention module EMA, thereby enhancing the feature representation capability of the network.

5. The fine-grained teaching behavior recognition method for smart classrooms according to claim 4, characterized in that, The processing flow of the fine-grained feature extraction module is as follows: The fine-grained feature extraction module first compresses the input features through convolution operations and then divides the features into two branches using a segmentation function, which helps to enhance the nonlinearity and expressive power of the network. One of the branches contains multiple Bottleneck modules, or Bneck modules, which are used to deeply extract input features to enhance the overall feature extraction capability of the network. The Bneck modules pass feature information through parallel paths. The other branch is left unprocessed; the outputs of the two branches are concatenated together using a join operation. Next, some features of the obtained feature group are fused with the original features after passing through the EMA attention mechanism to obtain a new feature group. Finally, the features are fused with the convolutional structure through feature concatenation and output as features.

6. The fine-grained teaching behavior recognition method for smart classrooms according to claim 5, characterized in that, The specific calculation process of the fine-grained feature extraction module is as follows: First, the features input to the fine-grained feature extraction module are processed. The input features are denoted as X∈RC×H×W, where C, H, and W represent the number of channels, height, and width, respectively. First, the input features are processed through a single convolution operation to obtain the feature map. Feature maps after convolution Then, through the channel splitting operation, it is divided into two parts, namely: ； ； in It is a standard convolutional layer used for initial feature extraction. and Representing the input features respectively The two branches after splitting are used for processing different paths later; Next is the segmented The data is fed into multiple Bottleneck layers to further extract deeper features. After extracting features in the Bneck layer, the Concat operation is used to... and By splicing them together, a new feature map is generated. The formula is as follows: ； ； Where n is the number of times the Bneck module is repeated; Next, the features The input is an efficient multi-scale attention module (EMA), which dynamically weights the features between channels to highlight the channel features that are critical to the detection task. The results obtained through the EMA module Add to the original feature group; Finally, all features of the obtained feature group are concatenated to obtain a processed feature map, which is then passed through a convolutional network to obtain the final feature map Y. The output and input sizes of the module are kept consistent. ； 。 7. The fine-grained teaching behavior recognition method for smart classrooms according to claim 1, characterized in that, The specific calculation process of the multi-scale feature fusion module is as follows: ； in It is the input feature map. This is the feature map after channel adjustment, when the number of input channels... With the target number of channels When there is a mismatch, through a The number of channels in the convolutional layer is adjusted. If the input channel does not match the expected output channel, use The convolution process is adjusted, and the formula is expressed as follows: ； ； ； ； in, It is a feature map output by a partially convolutional network P_Conv, through the defined... Layers After processing, we get ReLU as the activation function, DropPath as the drop-out function, and EMA as the attention mechanism operation. , , These are the Multilayer Perceptron (MLP), the Efficient Multiscale Attention Module (EMA), and the Fusion Feature.

8. A computer device comprising a memory and one or more processors; executable code stored in the memory; characterized in that, When the processor executes the executable code, it is used to implement the fine-grained teaching behavior recognition method for smart classrooms as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon; characterized in that, When executed by the processor, the program is used to implement the fine-grained teaching behavior recognition method for smart classrooms as described in any one of claims 1 to 7.

Images