Double-attention student behavior detection and recognition model and method for dynamic panoramic classroom scene

Abstract

The invention discloses a double-attention student behavior detection and recognition model and method for a dynamic panoramic classroom scene, and belongs to the technical field of behavior detection and recognition. In a panoramic classroom, accurate detection of classroom behaviors of students is of great significance to improvement of education quality and personalized teaching. According to the method, an EDA-SBRNet model is constructed by using YOLOv10n as a basic model, an HGNet network is used as a feature extractor, ParNet is introduced to enhance the model training speed and enrich the diversity of feature space, and finally, a C2fCID block is added to the tail end of a detection layer to improve the target detection and positioning precision of the model in a complex scene. Compared with other mainstream classroom behavior detection models, the EDA-SBRNet has excellent performance in the aspects of detection performance, model parameter quantity, calculation quantity, reasoning time and the like, and shows the advantages of the EDA-SBRNet in practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of behavior detection and recognition, and in particular relates to a dual-attention student behavior detection and recognition model and method for a dynamic panoramic classroom scene. Background Art

[0002] Human behavior recognition is a challenging and attractive research direction in the field of computer vision. It is widely used in many fields such as video understanding, intelligent monitoring, security, robotics, human-computer interaction, and health care. With the development of big data smart classrooms, behavior detection and recognition are gradually expanding to the field of education and becoming an important tool for improving teaching quality. Using behavior detection and recognition technology to conduct accurate student behavior analysis can bring more comprehensive and accurate evaluation feedback to the education process. In addition, the results of behavior detection and recognition also provide multi-dimensional data support for teachers, managers, students, and parents, helping them to better evaluate the learning process and make timely adjustments. Through this precise data feedback, not only the quality of education is improved, but also the personalized and refined development of teaching is promoted.

[0003] In recent years, a variety of student classroom behavior detection models have emerged, such as: Lin et al. [2] proposed a student behavior recognition system in a classroom environment based on skeleton pose estimation and human body detection. They improved the recognition accuracy of behavior by combining skeleton pose detection with YOLO. As the research deepened, the types of behaviors detected gradually increased, and the research problems became more detailed. LIU et al. [3] Based on YOLOv8n, BRA (Vision Transformer with Bi-Level Routing Attention) and TODL (Tiny Object Detection Layer) are introduced to solve the problems of small targets and occlusion. The system can detect three student behaviors: raising hands, standing, and reading. Tang et al. [4] The problem of classroom behavior detection is transformed into a fine-grained classification problem of behavior images. By introducing the weighted bidirectional feature pyramid network (BiFPN), the multi-scale features of the target are effectively expressed, the fine-grained features of the target behavior are mined, and the number of missed detections of small target students in the back row of the classroom is reduced. The system can detect four types of student behaviors: listening to the class, looking down, sleeping, and standing. YANG et al. [5] A method for detecting and analyzing student classroom behaviors based on spatiotemporal networks and multi-model fusion is proposed. This method not only explores the advantages and disadvantages of human body detection and spatiotemporal behavior detection in student classroom behavior scenarios, but also expands the detected behaviors to five types: listening, fatigue, raising hands, learning, and reading and writing. Jiang et al. [6]A CBAM (Convolutional Block Attention Module) was inserted into the shortcut structure of the original YOLOv3 to identify six behaviors in the classroom. Liao et al. [7] Using the VGG (Visual Geometry Group) model [8] Detecting students’ abnormal behaviors of sleeping and playing with mobile phones in class. Huang et al. [9] developed a classroom behavior recognition method based on deep convolutional neural network (D-CNN) to distinguish students’ behaviors such as looking up, raising hands, standing, and looking around. Wu et al.

[10] Based on region of interest (ROI) and face tracking, two algorithms for identifying students’ standing behavior in class were proposed. Zheng et al.

[11] An intelligent student behavior analysis system based on the improved Faster R-CNN (Region-CNN) was designed to record classrooms and automatically detect students' hand-raising, standing, and sleeping behaviors. However, the above studies did not fully consider the complexity of classroom behavior, the irregularity of student behavior, and the amount of model calculation, which resulted in low model accuracy and high computational complexity.

[0004] Therefore, although the current behavior recognition and detection algorithms are quite mature, behavior recognition and detection in panoramic classrooms still face many challenges. First, the existing student classroom behavior detection model can recognize few behavior categories. Second, the existing student classroom behavior detection model has poor recognition and detection effects in the panoramic classroom environment, and the reasoning time is long. Finally, there is the occlusion problem of classroom learning behavior, which makes recognition more difficult. These factors make it difficult for the model to learn effectively, which affects the precision and limits the comprehensiveness and accuracy of the model. Summary of the invention

[0005] In a panoramic classroom, accurate detection of students' classroom behavior is of great significance for improving the quality of education and personalized teaching. However, due to the large number of students, significant scale changes, and the occlusion of classroom learning behavior, classroom behavior detection often faces missed detection and false detection. To address the above problems, the present invention proposes a dual-attention student behavior detection and recognition model (EDA-SBRNet model) and method for dynamic panoramic classroom scenes.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A dual-attention student behavior detection and recognition model for dynamic panoramic classroom scenes is proposed. The EDA-SBRNet model is constructed with YOLOv10n as the basic model, the HGNet network is used as the feature extractor, and ParNet is introduced to enhance the model training speed while enriching the diversity of the feature space. Finally, the C2fCID block is added at the end of the detection layer to improve the model's detection and positioning accuracy of targets in complex scenes.

[0008] As the preferred technical solution of the present invention, the HGNet module efficiently integrates multi-layer features through HG Block and Stage modules, and adopts Stem and a downsampling strategy with a stride of 2 to reduce the spatial dimension of the feature map, thereby reducing the amount of calculation and memory access; input feature map X∈R C×H×W After being processed by the Stem module, Stem extracts features from the original input through the convolution layer to reduce the computational complexity; then, the feature map enters the HG Block, processes the data through a multi-scale convolution strategy, captures features of different scales, and adjusts the number of channels through 1x1 convolution; the residual connection promotes the effective transmission of gradients and alleviates the gradient vanishing problem; the feature map is reduced in spatial dimensions through the downsampling layer, which reduces the computational burden and expands the receptive field; the Tiny model contains only three downsampling layers to optimize GPU utilization; finally, the feature map spatial dimensions are compressed through the global average pooling (GAP) layer to improve the network's robustness to spatial transformations; one-dimensional global pooling in the horizontal and vertical directions further captures long-range dependencies and position information, and the program calculation is as follows:

[0009]

[0010] The last part of the network includes a series of classification layers, usually with a 1x1 convolution layer as the core, aggregating features and performing one-dimensional global average pooling in the horizontal and vertical directions; in this way, the feature map is processed separately in the spatial dimensions (H and W) to capture features in different spatial directions; the 1x1 convolution kernel helps to establish the relationship between channel and spatial position information, which facilitates the decomposition and representation of features in different directions; finally, the features are mapped to the desired output category through a fully connected layer; the program is calculated as follows:

[0011]

[0012] As the preferred technical solution of the present invention, the ParNet is a neural network composed of parallel substructures, including downsampling and fusion modules: the downsampling module realizes multi-scale processing by reducing the resolution and increasing the width; the fusion module integrates information of different resolutions to reduce the inference delay; the downsampling block introduces the SE module parallel to the convolution layer, and uses 2D average pooling in the 1×1 convolution branch; the fusion block enhances the number of input channels through the series layer. At the same time, ParNet introduces the SSE layer, which combines the skip connection and the Squeeze-and-Excitation mechanism to enhance the feature expression through global average pooling, 1x1 convolution and Sigmoid activation.

[0013] As a preferred technical solution of the present invention, the C2fCID module enhances the feature capabilities of the model in the channel dimension and the spatial dimension through its spatial and channel dual-branch attention mechanism. The model first enhances the channel information of the feature map through a 3x3 deep convolution layer, and then uses the channel attention and spatial attention mechanisms to strengthen the important feature channels and spatial positions respectively; the outputs of the channel attention and spatial attention are combined with the original feature map through element-by-element multiplication to enhance important features and suppress unimportant features; finally, after further integrating the features through a 1x1 convolution layer, the model uses a residual connection to add the processed feature map to the original input to retain the original information and enhance the learning ability of the model.

[0014] The dual-attention student behavior detection and recognition method based on the above model first collects the image and video data of students in the classroom through the camera and stores them on the server or cloud platform; secondly, the model input is a video frame or an image frame, and the output is a behavior category. The collected real-time video stream is input into the trained model for inference, and the behavior category of each student in each frame of the image is output; then, the behavior is analyzed and the students' behavior performance is counted; finally, the data is stored and displayed.

[0015] Compared with the existing solutions, the beneficial effects of the present invention are:

[0016] 1. Aiming at the problem of occlusion in classroom learning behavior, the present invention designs a dual-branch attention mechanism module C2fCID to improve the capture performance of target feature information. By introducing the ParNet module, a model EDA-SBRNet with shallow depth but excellent performance is successfully constructed. The computational load of the model is reduced while maintaining accuracy. The EDA-SBRNet model proposed in the present invention uses the HGNet network as the backbone to achieve more efficient feature extraction and fusion, providing a robust basic model for the recognition and detection tasks of student actions in dynamic panoramic classrooms.

[0017] 2. The present invention constructs an EDA-SBRNet model with HGNet as the backbone network, and adopts a multi-scale feature fusion strategy to improve feature expression capabilities and reduce inference time while maintaining high detection performance. Combined with the ParNet module to build a parallel sub-network, this shallow but efficient structure significantly reduces the computational complexity and effectively maintains the detection accuracy of the model. The C2fCID module proposed in the present invention uses a dual-branch attention module internally, aiming to enhance the feature representation capabilities of feature maps in the channel dimension and spatial dimension, thereby improving the accuracy and efficiency of target recognition and detection.

[0018] 3. In order to verify the effectiveness of the algorithm, a classroom behavior dataset was constructed, covering eight typical behaviors, including lowering the head to write, lowering the head to read, raising the head to listen to the class, turning the head, raising the hand, standing, group discussion and teacher guidance. Compared with other mainstream classroom behavior detection models, EDA-SBRNet performs well in detection performance, model parameter quantity, calculation amount and reasoning time, showing its advantages in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the EDA-SBRNet model structure.

[0020] Figure 2 This is the HGNet model structure.

[0021] Figure 3 It is the ParNet model.

[0022] Figure 4 It is the SSE model structure.

[0023] Figure 5 It is the C2fCID model structure and the CID model structure.

[0024] Figure 6 Flowchart for model training.

[0025] Figure 7 It is the PR curve of EDA-SBRNet model.

[0026] Figure 8 This is the ParNet location organization diagram.

[0027] Fig. 9 Visualization results for the EDA-SBRNet model. DETAILED DESCRIPTION

[0028] Example 1

[0029] The EDA-SBRNet model proposed in this embodiment includes an HGNet network, a ParNet module, and a C2fCID module.

[0030] In view of the poor recognition effect, long reasoning time and occlusion problems of existing models in panoramic classroom environments, this paper takes YOLOv10n as the basic model and proposes a new classroom behavior recognition and detection model EDA-SBRNet to meet the challenges of behavior detection and recognition in panoramic classroom environments. While maintaining high performance, the EDA-SBRNet model has significantly reduced the number of parameters and calculations compared to mainstream detection models, thereby improving the efficiency and practicality of behavior recognition and detection tasks.

[0031] The network structure of EDA-SBRNet is as follows Figure 1 As shown. The EDA-SBRNet network consists of a backbone network, a feature fusion network and a prediction head. In view of the characteristics of the student classroom data set, the network is improved in three key aspects to improve the detection accuracy and efficiency of the network. First, in the backbone part of the model, the present invention adopts the design principles of the efficient neural network HGNet. The enhanced model captures multi-scale features, so that the model can reduce the inference time while maintaining high accuracy. Secondly, the present invention introduces the ParNet attention mechanism in the neck of the model. The introduction of the ParNet module further improves the accuracy of the EDA-SBRNet model and ensures accurate identification of student behaviors in cluttered environments. Finally, in view of the characteristics of the detection layer, the present invention designs a C2fCID module. To improve the detection effect of the back row targets. The following is a detailed description of each component:

[0032] 1HGNet Network

[0033] In the classroom behavior detection and recognition tasks in the panoramic classroom, in addition to fully capturing the global context information, it is also necessary to pay careful attention to the local detail features. This requires the model to be able to analyze and process the input data in multiple dimensions, not only to capture the macro behavior patterns in the classroom scene, but also to identify and parse the micro details related to individual behavior. By integrating global information and local features, the model can more effectively identify and distinguish complex classroom behaviors, thereby improving the model accuracy. Therefore, the present invention constructs the EDA-SBRNet model based on the HGNet network. The HGNet network can extract features at different resolutions at the same time by adopting a multi-scale pyramid structure, thereby effectively capturing the detail information and global context in the image; at the same time, by using parallel processing of different types of feature streams, focusing on the local details and global structure of the image, and finally realizing the fusion of information, significantly improving the computational efficiency and overall performance.

[0034] like Figure 2 As shown in Figure 1, the HGNet module efficiently integrates multi-layer features through the HG Block and Stage modules, and uses the Stem and stride 2 downsampling strategy to reduce the spatial dimension of the feature map, thereby reducing the amount of computation and memory access. Input feature map X∈RC×H×W After being processed by the Stem module, Stem extracts features from the original input through the convolution layer to reduce the computational complexity. Next, the feature map enters the HG Block, which is the core of the network. It processes data through a multi-scale convolution strategy (3x3, 5x5, 7x7, 9x9 convolutions), captures features of different scales, and adjusts the number of channels through 1x1 convolution. The residual connection promotes the effective transmission of gradients and alleviates the problem of gradient disappearance. The feature map is reduced in spatial dimensions through the downsampling layer, which reduces the computational burden and expands the receptive field. The Tiny model contains only three downsampling layers to optimize GPU utilization. Finally, the feature map spatial dimension is compressed through the global average pooling (GAP) layer to improve the network's robustness to spatial transformations. One-dimensional global pooling in the horizontal and vertical directions further captures long-range dependencies and position information. The above program is calculated as follows:

[0035]

[0036] The last part of the network includes a series of classification layers, usually with a 1x1 convolution layer as the core, which aggregates features and performs one-dimensional global average pooling in the horizontal and vertical directions. In this way, the feature map is processed separately in the spatial dimensions (H and W) to capture features in different spatial directions. The 1x1 convolution kernel helps to establish the relationship between channel and spatial position information, which facilitates the decomposition and representation of features in different directions. Finally, the features are mapped to the desired output category through a fully connected layer. The above program is calculated as follows:

[0037]

[0038] The network structure and computing strategy designed by the HGNet model accelerate the feature fusion process, reduce computational complexity, reduce memory usage, and improve the richness of feature representation.

[0039] 2ParNet Attention

[0040] In classroom behavior detection and recognition tasks, attention mechanisms are often used to focus on the target area and suppress interference from non-target areas such as the background, thereby improving the accuracy of behavior detection and recognition. Especially in classroom behavior recognition and detection in panoramic classroom scenes, the interference of background factors is particularly obvious. In order to solve this problem, the present invention introduces ParNet

[13] Attention mechanism, ParNet can adaptively select features in different input data, dynamically weight important areas and suppress redundant or irrelevant information, and effectively integrate features of different levels in multi-scale feature processing, and optimize the representation of local and global information by adjusting the weights of features at each scale, thereby improving the effectiveness of feature expression and the understanding of multi-scale information. A higher model performance is achieved with fewer network layers.

[0041] ParNet is a neural network composed of parallel substructures (called streams), such as Figure 3 As shown in , it is designed to process inputs of different resolutions and fuse features in the later stage to support downstream tasks. Its design improves the traditional 3x3 convolution, expands the receptive field, and enhances the feature extraction capability. In addition, ParNet contains downsampling and fusion modules: the downsampling module achieves multi-scale processing by reducing the resolution and increasing the width; the fusion module integrates information of different resolutions to reduce inference latency. To accelerate inference, ParNet expands its structure by increasing the network width, resolution, and number, and adopts parallel computing. The downsampling block introduces an SE module parallel to the convolution layer, and uses 2D average pooling in the 1×1 convolution branch; the fusion block enhances the number of input channels through a series layer. In order to overcome the problem of limited receptive field of non-deep networks, ParNet introduces the SSE (Skip-Squeeze-Excitation) layer, which combines skip connections and the Squeeze-and-Excitation mechanism to enhance feature expression through global average pooling, 1x1 convolution, and Sigmoid activation. As shown in Figure 4 As shown, the SSE module improves the feature representation capability without increasing the network depth through a parallel structure and an attention mechanism, thereby improving the accuracy of target detection or classification tasks. When dealing with complex tasks, the model often has the problem of insufficient feature expression, especially in image classification tasks, the model may ignore key local information. Therefore, the present invention places the ParNet module below the neck C2fCID and strengthens contextual dependencies by dynamically adjusting the attention mechanism. Experiments show that this configuration effectively enhances the model's ability to capture long-distance dependencies, especially in complex classroom scenarios.

[0042] 3C2fCID module

[0043] In a natural classroom environment, there are many objects such as students, desks and chairs, and books. These objects may block some of the students' movements or behaviors, making it difficult to clearly identify learning behaviors. Such occlusion will interfere with the feature extraction process of the model, thereby affecting the accuracy and robustness of the model. Therefore, the C2fCID module is designed to solve the occlusion problem of classroom learning behaviors by enhancing the model's feature capabilities in the channel dimension and spatial dimension through its spatial and channel dual-branch attention mechanism. Figure 5As shown in the figure, the C2fCID module on the left replaces the original Bottleneck part with CID. By stacking multiple CID modules (controlled by parameter n), multiple optimizations of the bottleneck layer are achieved, which can better capture fine-grained, global and rich contextual information. The CID model on the right is a neural network structure that combines deep convolution and attention mechanisms, aiming to improve the efficiency and accuracy of feature extraction. The model first enhances the channel information of the feature map through a 3x3 deep convolution layer, and secondly, uses channel attention and spatial attention mechanisms.

[14] The important feature channels and spatial positions are strengthened respectively. The outputs of channel attention and spatial attention are combined with the original feature map through element-wise multiplication to enhance important features and suppress unimportant features. Finally, after further integrating the features through the 1x1 convolution layer, the model uses residual connection to add the processed feature map to the original input to retain the original information and enhance the learning ability of the model. This design enables the network to more effectively capture the details and background information of the target in a complex environment, significantly improving the feature expression ability, thereby improving the detection accuracy of the target object.

[0044] Example 2

[0045] Experimental results and analysis, covering the data set, experimental environment and parameter settings, evaluation indicators, experimental results and analysis of the EDA-SBRNet model, compared with other models, and conducted ablation experiments on the EDA-SBRNet model.

[0046] 1 Dataset

[0047] In deep learning, the quality of dataset construction directly affects the effect of model recognition. The selection criteria of datasets include: ① The samples should cover a variety of scenarios, conditions and categories to improve the adaptability of the model; ② The dataset should be closely related to the research objectives or tasks; ③ The dataset should be large enough to ensure the training and generalization capabilities of the model; ④ Prioritize datasets with a large number of people in the group, and exclude scenarios with no students or too few students in the classroom.

[0048] The present invention constructs a new dataset called classaction, which contains classroom teaching monitoring videos and filmed videos publicly available on the Internet. The image acquisition of the self-built dataset is completed by fixing the camera in the front center of the classroom with a tripod. For example, a classroom may have more than 40 students present at the same time. In addition to each person sitting in a different position, the images in the dataset constructed by the present invention are taken from different shooting angles, including front view, side view and back view. Finally, the present invention obtains four videos with a duration of 45 minutes in MP4 format. In order to ensure the training effectiveness of the model and reduce the risk of overfitting, the original data retains images covering student behaviors in different scenarios, while eliminating redundant images. Subsequently, the present invention adopts data enhancement techniques such as flipping, cropping, scaling, translation and color brightness transformation to further enhance the diversity of the data. These rich data transformations are designed to enable the model to better adapt to various complex scenarios, thereby significantly improving the robustness and generalization ability of the detection model.

[0049] After data selection and expansion, the present invention uses the LabelImg tool for detailed annotation. In the end, the data set contains 6198 images, and eight common behaviors are used to evaluate students' participation, including: writing with head down, reading with head down, listening to the class with head up, turning head, raising hands, standing, group discussion and teacher guidance. In order to effectively carry out model training and performance evaluation, the present invention divides these images into training set and test set in a ratio of 8:2. The training set contains 4959 images and the test set contains 1240 images. In order to better display the prediction results, the labels of each category are encoded accordingly to ensure that the results can be presented intuitively. After data cleaning and annotation, the number of labels in each category is: raising hands (1050), standing up (442), turning heads (298), writing with head down (973), reading with head down (756), listening to the class with head up (1749), group discussion (793), and teacher guidance (137). The detailed introduction of the data set is shown in Table 1.

[0050] Table 1: Standard and label coding of behaviors

[0051]

[0052] The entire training process, such as Figure 6 As shown, it aims to fully tap the potential of the model through scientific training methods and evaluate its performance.

[0053] 2 Training environment and parameter settings

[0054] ①Hardware environment configuration: The main equipment configuration for this experiment is GeForce RTX 3090 (GPU) 8G video memory. ②Software environment configuration: Use Python 3.7.12 for environment deployment, and select PyTorch 2.0.1 as the deep learning framework. ③The image input size is set to 640*640, the batch size is set to 8, the initial learning rate is set to 0.01, and a total of 100 rounds of iterations.

[0055] 3 Evaluation indicators

[0056] In order to evaluate the performance of the model, the present invention selects Precision (P) and Recall (R) as evaluation indicators. Because Precision measures the proportion of correctly identified positive samples to the total number of predicted positive samples, reflecting the accuracy of the model's prediction; Recall measures the proportion of correctly predicted positive samples to the total number of actual positive samples, reflecting the model's coverage of positive samples. In order to more comprehensively evaluate the performance of the model, the mean average precision (mAP) is also used. mAP represents the average detection accuracy of all categories, providing a global perspective for the overall performance of the model. Through these indicators, the performance of the model in different aspects can be fully reflected.

[0057] Precision: Precision measures the proportion of correctly identified positive samples among all positive predictions made by the model. This metric is chosen because it can effectively evaluate the model's ability to minimize false positives. The precision calculation formula is formula (4):

[0058]

[0059] Where TP refers to True Positives, which indicates the number of targets correctly detected by the model.

[0060] FP stands for False Positives, which indicates the number of negative examples that the model incorrectly predicts as positive examples.

[0061] Recall (R): Recall measures the proportion of true positive samples correctly identified by the model, reflecting the sensitivity of the model. This indicator is an important indicator to measure the ability of the model to detect all relevant targets and is the key to comprehensively evaluating the performance of the model. The formula for calculating the recall is formula (5):

[0062]

[0063] Where FN refers to False Negatives, which indicates the number of real positive examples that the model failed to detect.

[0064] F1-Score: F1-Score is the harmonic mean of precision and recall, balancing precision and sensitivity. It is particularly valuable for evaluating the performance of models on imbalanced data. The calculation formula is formula (6):

[0065]

[0066] mAP: mAP measures the average detection accuracy of the model on multiple categories. This indicator is chosen because it can comprehensively evaluate the overall performance of the model in multi-category tasks. mAP50-95 is a more comprehensive evaluation indicator that calculates the mAP value of the 50% to 95% confidence threshold. mAP50-95 can better evaluate the robustness and performance of the model. The calculation formula is formula (7):

[0067]

[0068] Where APi refers to the AP of the i-th target class. N is the number of detected images.

[0069] 4 Experimental results and analysis

[0070] In order to evaluate the performance of the proposed overall framework, this paper uses the annotated ClassAction dataset for testing. The experimental results are shown in Figure 7 As shown. From the performance of the PR curve, the curve of the proposed model is close to the upper right corner, which shows that the model performs well in both recall and precision. In addition, the area under the PR curve is large, which further proves the high performance of the model. It is worth noting that the smoothness of the curve shows that the model proposed in the present invention maintains a stable balance between recall and precision, ensuring that the relationship between the two is relatively consistent. Therefore, these results show that the framework proposed in the present invention has a strong overall effect and stability.

[0071] 5. Comparison of the model with other detection models

[0072] In order to verify the superiority of the proposed EDA-SBRNet model over the classic target detection model, a comparative experiment was conducted. The ClassAction classroom behavior dataset built by the present invention was used, and popular behavior detection models were selected as benchmarks, including YOLOv10m

[15] 、YOLOv11

[16] 、YOLOv9

[17] 、YOLOv7+BRA

[18] ET-YOLOv 5s

[12] and RT-DETR [1] , ensuring the comprehensiveness and fairness of the comparison.

[0073] Throughout the experiment, strict control measures were adopted to ensure that all models were trained under the same hyperparameter configuration, thereby ensuring the reliability and comparability of the experimental results. The experimental results are summarized in Table 3. By comparing mAP50, mAP50-95, the number of model parameters, and the inference time, the performance differences between different models can be intuitively evaluated. From the data in Table 3, it can be seen that compared with the latest version of YOLOv11, the proposed YOLOv10n_HP has improved the accuracy by 0.2%, while the inference time has been reduced by 50% and the number of parameters has been reduced by 49%. In addition, compared with other mainstream models YOLOv9 and YOLOv7+BRA, YOLOv10n_HP has reduced the inference time by 65.9% and 23.8%, respectively. In terms of the number of parameters, YOLOv10n_HP also shows significant advantages, with a reduction of 73.1% and 54.6%, respectively. At the same time, the RT-DETR model has an accuracy improvement of 1.2% compared to YOLOv10n_HP, but also shows significant optimization in terms of parameter quantity and inference time, with a 58% reduction in parameter quantity and a 70% reduction in inference time. These results show that the proposed model has achieved significant improvements in accuracy, computational efficiency, and model complexity, reflecting its potential and advantages in practical applications.

[0074] The EDA-SBRNet model effectively reduces the complexity of the model by adopting a scale pyramid structure backbone network, and reduces the computational overhead by combining efficient modules. These measures make the deployment of the model on edge computing devices an efficient and feasible option. Therefore, EDA-SBRNet demonstrates superior performance in target detection tasks, with the characteristics of low parameter count, high accuracy, and fast detection, making it an ideal model for use in panoramic classrooms.

[0075] Table 2: Comparison experiments with other models

[0076]

[0077] 6 Ablation Experiment

[0078] In order to comprehensively verify the impact of each improved module on the performance of the EDA-SBRNet model and evaluate the effectiveness and reliability of the EDA-SBRNet model, an ablation experiment was carried out.

[0079] Experiment 1 (YOLOv10n) is used as the baseline model of the present invention to explore the effect of performance improvement.

[0080] In Experiment 2, the HGNet backbone network is used. The HGNet backbone network can capture multi-scale features at a finer granularity, thereby enhancing the feature perception ability of underrepresented categories. The results show that mAP50 and mAP50-95 are improved by 1.4% and 1.6% respectively compared with the original YOLOv10n.

[0081] In Experiment 3, the ParNet module was added. This module performs well in processing complex relationships, long-distance dependencies, and capturing multi-level relationships, which enhances the model's ability to understand contextual information. Compared with the baseline model, mAP50 and mAP50-95 increased by 0.7% and 0.6% respectively.

[0082] In Experiment 4, the C2fCID module is added. C2fCID effectively captures multi-level feature information by fusing feature maps of different scales, thereby improving the accuracy of object detection. The experimental results show that mAP50 and mAP50-95 are improved by 0.4% and 0.3% respectively compared with the baseline model.

[0083] In Experiment 5, HGNet, ParNet, and C2fCID are introduced into YOLOv10n (baseline) at the same time. The experimental results (see Table 3) show that the overall performance is significantly improved, with mAP50 and mAP50-95 increased by 2.1% and 2.4% respectively. These experimental results verify that the improvements to the YOLOv10n model do have a positive impact on detection accuracy.

[0084] Table 3: EDA-SBRNet ablation experiment

[0085]

[0086] Ablation experiment of attention mechanism position:

[0087] The present invention integrates the ParNet attention mechanism into different levels of the backbone and neck modules of the EDA-SBRNet model. Through a series of position ablation experiments, the impact of the ParNet integration position on the performance of EDA-SBRNet is evaluated. Figure 8 The figure shows the addition of ParNet at different locations, where the red module represents the added ParNet attention mechanism. The Concat layer is responsible for concatenating the output features of the previous layer with the features of other layers to achieve feature fusion.

[0088] Experimental results show that the integration of ParNet in different locations significantly affects the performance of the model compared to the EDA-SBRNet baseline model, and even leads to performance degradation in some locations. In particular, the performance of the model is most significantly improved after ParNet is integrated into the last two layers of C2fCID of the EDA-SBRNet model. Figure 8 As shown in Figure 5, under this configuration, mAP50 increased by 0.8% and mAP50-95 increased by 0.5%. Therefore, based on these experimental results, the present invention finally chose to integrate ParNet into the C2fCID of EDA-SBRNet to optimize the detection performance of the model. This result further verifies the importance of optimizing the module position.

[0089] Table 4: Ablation experiment of ParNet position

[0090]

[0091] 7 Comparison of different datasets and experimental results

[0092] In order to evaluate the generalization ability of the model, the present invention uses another dataset SCB-Dataset3

[19] A comparative experiment was conducted. This dataset is another dataset for classroom behavior recognition. The dataset includes 5686 images with 45578 labels, focusing on six behaviors: raising hands, reading, writing, using mobile phones, lowering heads, and lying on the table. The improved EDA-SBRNet model proposed in this paper was trained and tested. The results are shown in Table 5.

[0093] Table 5: Comparison of training results on the SCB-Dataset3 dataset

[0094]

[0095] From the results in Table 5, we can see that the improved EDA-SBRNet model of the present invention has achieved good results on the SCB-Dataset3 dataset. Compared with YOLOv10n, its mAP50 increased by 1.5% and mAP50-95 increased by 0.4%. Compared with other improved behavior recognition models, it is also highly competitive.

[0096] The visualization results of the EDA-SBRNet model are as follows: Fig. 9 As shown, this indicates that the improved model in the present invention has strong generalization ability and can be effectively applied to dense, multi-scale panoramic classroom scenes.

[0097] Example 3

[0098] A dual-attention student behavior detection and recognition method for dynamic panoramic classroom scenes, the steps are as follows:

[0099] 1 Data Collection

[0100] Video data collection: Collect images and video data of students in the classroom through cameras. Configure multiple cameras (such as front and back, left and right, etc.) to ensure coverage of all angles of the classroom. Video data should be collected and stored on a server or cloud platform, and the video must be smooth.

[0101] 2 Behavior Recognition and Classification

[0102] The model input is a video frame or an image frame, and the output is a behavior category. The collected real-time video stream is input into the trained model for inference, and the behavior category of each student in each frame of the image is output.

[0103] 3. Behavior Analysis and Feedback

[0104] Behavior analysis: Analyze the recognition results and count the students' behavior performance, such as:

[0105] each student’s focus time;

[0106] Do you talk to others frequently?

[0107] Is there any hand-raising behavior?

[0108] Whether you frequently leave your seat;

[0109] Comprehensive scoring is given to provide a report on students' classroom behavior.

[0110] 4Data storage and display

[0111] Data storage: Store the identified behavior data, video records, analysis reports, etc. in a database or cloud platform to ensure long-term storage and query of data.

[0112] Data display: Provides a visual data display interface to show students' behavioral change trends, overall classroom attention and other information, making it easier for teachers to manage the classroom.

[0113] 5System upgrade and optimization

[0114] Performance optimization: Optimize models and algorithms based on recognition accuracy and processing speed to improve the real-time performance and accuracy of the system.

[0115] Behavior recognition expansion: Based on the actual situation in the classroom, gradually increase the types of behaviors that can be recognized, such as whether to answer questions, whether to participate in discussions, etc.

[0116] System iterative updates: Regularly update training models, introduce more diverse data, and improve the system's recognition capabilities and adaptability.

[0117] In summary, in view of the problems of missed detection, false detection and deployment difficulties of learning behaviors in existing models when dealing with the situation of dense number of students, significant scale changes and occlusion problems in panoramic classroom scenes, the behavior recognition and detection model EDA-SBRNet proposed in this invention aims to achieve high-precision classroom behavior recognition and detection in a more efficient way, and meet the dual requirements of real-time and accuracy in panoramic classroom environments. EDA-SBRNet uses HGNet as the backbone feature extraction network, and integrates ParNet attention module and C2fCID module, which effectively improves the accuracy, speed and computational efficiency of the model. First, HGNet provides a solid feature extraction foundation. Secondly, ParNet optimizes computational complexity and performance through parallel sub-networks. Finally, the C2fCID module further enhances the recognition ability of the target. Experimental results show that EDA-SBRNet can achieve fast and accurate behavior detection in complex panoramic classroom scenes, meet real-time requirements, and provide efficient data support for classroom evaluation.

[0118] Note: The models, methods or data sets mentioned in this invention come from:

[0119] [1]ZHAO Y,LV W,XU S,et al. Detrs beat yolos on real-time object detection; proceedings of the Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, F, 2024[C].

[0121] [2]LIN FC,NGO HH,DOW CR,et al.Student behavior recognition system for the classroom environment based on skeleton pose estimation and persondetection[J].2021,21(16):5314.

[0123] [3]LIU Q,JIANG R,XU Q,et al.YOLOv8n_BT:Research on Classroom LearningBehavior Recognition Algorithm Based on Improved YOLOv8n%J IEEE Access[J].2024,Vol.12:36391-403.

[0125] [4]TANG.Classroom Behavior Detection Based on ImprovedYOLOv5Algorithm Combining Multi-Scale Feature Fusion and AttentionMechanism%J APPLIED SCIENCES-BASEL[J].2022,12(13).

[0127] [5]WANG X.A spatio-temporal attention fusion model for studentsbehaviour recognition%J EAI EndorsedTransactions on Scalable InformationSystems[J].2022,Vol.9(No.34).

[0128] [6]JIANG Q,ZHANG Y,TAN S,et al.Students’classroom behaviorrecognition based on residual network [J].2019,20:23-7.

[0130] [7]LIAO P,LIU C,SU H,et al.Detection and analysis system of students’abnormal behavior in classroom based on deep learning[J].2018,8:97-8.

[0132] [8]SIMONYAN K,ZISSERMAN A J A P A.Very deep convolutional networksfor large-scale image recognition[J].2014.

[0134] [9]HUANG W,LI N,QIU Z,et al.An Automatic Recognition Method forStudents'Classroom Behaviors Based on Image Processing[J].2020,37(3).

[0136]

[10] WU B,WANG C,HUANG W,et al.Recognition of Student ClassroomBehaviors Based on Moving Target Detection[J].2021,38(1).

[0138]

[11] ZHENG R,JIANG F,SHEN R.Intelligent student behavior analysissystem for real classrooms; proceedings of the ICASSP 2020-2020IEEEInternational Conference on Acoustics,Speech and Signal Processing(ICASSP),F,2020[C].IEEE.

[0141]

[12] LI L,LIU M,SUN L,et al.ET-YOLOv5s:toward deep identification ofstudents’in-class behaviors[J].2022,10:44200-11.

[0143]

[13] GOYAL A,BOCHKOVSKIY A,DENG J,et al.Non-deep networks[J].2022,35:6789-801.

[0144]

[14] BUI N-T,HOANG D-H,NGUYEN Q-T,et al.Meganet:Multi-scale edge-guided attention network for weak boundary polyp segmentation;proceedings ofthe Proceedings of the IEEE / CVF Winter Conferenceon Applications of ComputerVision,F,2024[C].

[0146]

[15] WANG A,CHEN H,LIU L,et al.YOLOv10:Real-Time End-to-End ObjectDetection[J].2024.

[0147]

[16] KHANAM R,HUSSAIN M J A P A.YOLOv11:An Overview of the KeyArchitectural Enhancements [J].2024.

[0149]

[17] WANG C-Y,YEH I-H,LIAO H-Y M J A P A.Yolov9:Learning what you wantto learn using programmable gradient information[J].2024.

[0151]

[18] YANG F J A P A.Student Classroom Behavior Detection based onImproved YOLOv7[J].2023.

[0152]

[19] YANG F,WANG T.SCB-Dataset3:ABenchmark for Detecting StudentClassroom Behavior[J].2023。

Claims

1. A dual-attention student behavior detection and recognition model for dynamic panoramic classroom scenes, characterized in that: The EDA-SBRNet model is built with YOLOv10n as the basic model, the HGNet network is used as the feature extractor, and ParNet is introduced to enhance the model training speed while enriching the diversity of the feature space. Finally, the C2fCID block is added at the end of the detection layer to improve the model's detection and positioning accuracy in complex scenes.

2. The dual-attention student behavior detection and recognition model for dynamic panoramic classroom scenes as claimed in claim 1, characterized in that: The HGNet module efficiently integrates multi-layer features through HG Block and Stage modules, and uses Stem and a downsampling strategy with a stride of 2 to reduce the spatial dimension of the feature map, thereby reducing the amount of computation and memory access; the input feature map X∈R C×H×W After being processed by the Stem module, Stem extracts features from the original input through the convolution layer to reduce the computational complexity; then, the feature map enters the HG Block, processes the data through a multi-scale convolution strategy, captures features of different scales, and adjusts the number of channels through 1x1 convolution; the residual connection promotes the effective transmission of gradients and alleviates the gradient vanishing problem; the feature map is reduced in spatial dimensions through the downsampling layer, which reduces the computational burden and expands the receptive field; the Tiny model contains only three downsampling layers to optimize GPU utilization; finally, the feature map spatial dimensions are compressed through the global average pooling (GAP) layer to improve the network's robustness to spatial transformations; one-dimensional global pooling in the horizontal and vertical directions further captures long-range dependencies and position information, and the program calculation is as follows: The last part of the network includes a series of classification layers, usually with a 1x1 convolution layer as the core, aggregating features and performing one-dimensional global average pooling in the horizontal and vertical directions; in this way, the feature map is processed separately in the spatial dimensions (H and W) to capture features in different spatial directions; the 1x1 convolution kernel helps to establish the relationship between channel and spatial position information, which facilitates the decomposition and representation of features in different directions; finally, the features are mapped to the desired output category through a fully connected layer; the program is calculated as follows:

3. The dual-attention student behavior detection and recognition model for dynamic panoramic classroom scenes as claimed in claim 1, characterized in that: The ParNet is a neural network composed of parallel substructures, including downsampling and fusion modules: the downsampling module achieves multi-scale processing by reducing the resolution and increasing the width; the fusion module integrates information of different resolutions to reduce inference latency; the downsampling block introduces an SE module parallel to the convolution layer, and uses 2D average pooling in the 1×1 convolution branch; the fusion block enhances the number of input channels through a series layer. At the same time, ParNet introduces an SSE layer, which combines skip connections and the Squeeze-and-Excitation mechanism to enhance feature expressiveness through global average pooling, 1x1 convolution, and Sigmoid activation.

4. The dual-attention student behavior detection and recognition model for dynamic panoramic classroom scenes as claimed in claim 1, characterized in that: The C2fCID module enhances the model's feature capabilities in the channel and spatial dimensions through its spatial and channel dual-branch attention mechanism. The model first enhances the channel information of the feature map through a 3x3 deep convolutional layer. Secondly, the channel attention and spatial attention mechanisms are used to strengthen the important feature channels and spatial positions respectively. The outputs of the channel attention and spatial attention are combined with the original feature map through element-by-element multiplication to enhance important features and suppress unimportant features. Finally, after further integrating the features through a 1x1 convolutional layer, the model uses a residual connection to add the processed feature map to the original input to retain the original information and enhance the model's learning ability.

5. A method for detecting and identifying dual attention student behavior based on the model as described in any one of claims 1 to 4, characterized in that: First, the image and video data of students in the classroom are collected through the camera and stored on the server or cloud platform; secondly, the model input is video frames or image frames, and the output is behavior categories. The collected real-time video stream is input into the trained model for inference, and the behavior category of each student in each frame of the image is output; then, the behavior is analyzed and the students' behavioral performance is counted; finally, the data is stored and displayed.

Images