Classroom behavior identification method based on deformable multi-scale adaptive detection network

Abstract

The invention discloses a classroom behavior identification method based on a deformable multi-scale adaptive detection network. The classroom behavior identification method comprises the following steps: building the deformable multi-scale adaptive detection network; dividing the classroom behavior sample data set into a training set sample data set, a verification set sample data set and a test set sample data set, extracting student behavior characteristics from pictures in the training set through a deformable multi-scale adaptive detection network, and performing model training; verifying the verification set by using a model obtained from the training set; obtaining the optimal model parameters of the verification set, inputting the photos in the test set for testing to generate a prediction image, and completing the detection of the classroom behaviors of the students. According to the invention, high-accuracy classroom behavior recognition is realized by using the feature extraction module, the multi-scale feature fusion module and the regression detection module. By dividing the sample data set into the training set, the verification set and the test set, efficient and scientific model training is ensured, and the over-fitting risk is reduced.

Description

Technical Field

[0001] The invention belongs to the field of behavior recognition and relates to a classroom behavior recognition method based on a deformable multi-scale adaptive detection network. Background Art

[0002] In recent years, the application of classroom teaching videos in student behavior analysis has shown broad prospects. Students' classroom behavior is an important indicator of learning status. Therefore, by analyzing and evaluating these behaviors, teaching improvement can be effectively promoted. However, with the rapid development of educational informatization, how to accurately and quickly identify students' key behaviors in a changing classroom environment has become a technical problem that needs to be solved in the field of education.

[0003] Classroom behavior analysis aims to explore the internal mechanisms of teachers' teaching activities and students' academic development, help teachers and students reflect on their own classroom performance, and thus improve the quality of classroom teaching. Traditional classroom behavior analysis methods mainly rely on self-evaluation, manual supervision, and manual coding. These methods often have shortcomings such as strong coding subjectivity, small sample size, and time-consuming and labor-intensive, resulting in low interpretability and scalability. Therefore, using intelligent technology to collect and analyze data can identify classroom behaviors more timely and comprehensively, gain insight into teachers' teaching and students' learning status, and provide strong support for improving teaching quality. Deep learning, with its powerful feature extraction and autonomous learning capabilities, has significantly improved the accuracy and efficiency of behavior recognition in classroom scenarios.

[0004] The current object detection framework is mainly divided into single-stage and two-stage detectors. The two-stage method first extracts RoI features, and then uses the detection head to classify and regress the RoI features. Faster R-CNN, Mask R-CNN, and Cascade R-CNN are typical representatives. The single-stage detector omits the foreground screening process and directly classifies and regresses the feature map generated by the backbone network. Typical single-stage detectors include SSD, RetinaNet, and YOLO series. Generally, the two-stage detector has better accuracy, while the single-stage detector is slightly inferior to the former in performance, but has a faster processing speed. In recent years, new methods such as DETR and Mamba based on visual Transformer have also emerged, and a series of new results have been achieved in different fields.

[0005] At present, classroom behavior recognition work is still mainly carried out around deep convolutional neural networks, and its core design concept lies in the combination of feature extraction and bounding box regression. In the actual classroom environment, occlusion and scale differences make feature extraction, feature fusion, regression and classification face huge challenges. The YOLO algorithm is a single-stage detection algorithm based on deep learning, which continuously adjusts the position and category of the bounding box in the output layer through regression. After the image is input into YOLO, the backbone feature extraction network extracts features layer by layer through convolution operations, and uses the obtained features for prediction, and finally calculates the final prediction box by combining multiple prediction results.

[0006] Although the YOLO network has achieved certain results in classroom behavior recognition, in actual classroom teaching environments, images are usually collected at wide angles or in full view. Students in the back row are far away from the monitoring equipment, and subtle body movements, such as slightly opening and closing the mouth to determine whether they are talking, and fine finger operations to distinguish between flipping through books or using mobile phones, are difficult to effectively extract sufficient distinguishing features. This leads to confusion of target features during the detection process, which directly affects the effects of feature extraction, fusion, regression and classification, reduces the accuracy and reliability of recognition, and increases the risk of missed detection and false detection. In addition, in the process of classroom behavior recognition, the occlusion of the front and back row actions and the differences in the expression of classroom behavior features at different scales bring additional challenges to the recognition system. In particular, due to the differences in detail richness and semantic levels, the use of feature superposition or splicing fusion will ignore the uniqueness and mutual promotion between features of different scales, and the information is not fully utilized.

[0007] The YOLO network relies on the loss function of bounding box regression to optimize the prediction of bounding boxes, thereby achieving higher accuracy and reliability of object detection. The loss function of bounding box regression (BBR) is crucial for object detection, and its good definition will bring significant performance improvements to the model. In the field of object detection, scale differences and mutual occlusion between objects constitute a major obstacle to improving the performance of regression models. Specifically, for small-scale objects, due to the small image area they occupy, the effective feature information extracted is relatively scarce, which greatly increases the difficulty of the regression model in accurately predicting its bounding box, resulting in a significant decrease in prediction accuracy. On the contrary, for large-scale objects, although they contain richer feature information, because they cover a wide range of image areas, the positioning algorithm is more susceptible to adverse factors such as background noise, illumination changes, and interference from neighboring objects during the processing process, thereby weakening the accuracy of positioning.

[0008] The existing bounding box regression loss function adopts the CIoU loss function, which makes the predicted box closer to the true box and improves the regression accuracy by considering the overlapping area, center point distance and aspect ratio of the true box and the predicted box. Although CIoU takes into account the aspect ratio of the predicted box and the true box, it does not take into account the problem of low-quality samples in the training data. In a real classroom environment, the diversity of student positions, scale changes and occlusion are major challenges, so it is inevitable that low-quality examples are included in the training data. These low-quality samples are prone to regression errors, which leads to imbalance of training samples, and often dominate the gradient, resulting in violent oscillation of the loss function, thereby reducing the generalization ability of the model. In addition, the CIoU loss function has a large penalty when dealing with anchor boxes and target boxes with high overlap, which may affect model training and generalization capabilities. In addition, CIoU has limitations in capturing the shape and scale information of the box, resulting in unstable bounding box regression and poor performance in complex scenes. Summary of the invention

[0009] The purpose of the present invention is to provide a classroom behavior recognition method based on a deformable multi-scale adaptive detection network, which can improve the recognition accuracy and reliability of small targets and occlusion problems in classroom behavior detection.

[0010] In order to achieve the above object, the present invention provides the following technical solutions:

[0011] A classroom behavior recognition method based on a deformable multi-scale adaptive detection network includes the following steps:

[0012] Step 1: Building a deformable multi-scale adaptive detection network, which includes a feature extraction module, a multi-scale feature fusion module and a regression detection module;

[0013] Step 2: Divide the classroom behavior sample dataset into training set, validation set and test set sample datasets, extract student behavior features from the images in the training set through a deformable multi-scale adaptive detection network, and perform model training;

[0014] Step 3: Use the model obtained from the training set to verify the validation set;

[0015] Step 4: Obtain the best model parameters for the validation set, input photos in the test set for testing to generate predicted images, and complete the detection of students' classroom behavior.

[0016] Furthermore, the specific process of classroom behavior recognition by the deformable multi-scale adaptive detection network described in step 1 is as follows:

[0017] (1) Collect classroom behavior images and input them into the feature extraction module, convert the original input images into multi-layer feature maps, and extract the feature information of the images;

[0018] (2) Inputting the feature information output from the feature extraction module at different scales into the multi-scale feature fusion module for fusion to obtain fused feature information;

[0019] (3) The fused feature information is input into the regression prediction module for behavior recognition to obtain the final recognition result.

[0020] Furthermore, the structure of the feature extraction module includes: Layer 1 that extracts underlying feature information from the input image through convolution operation, Layers 2-3 that are each composed of a Conv module and a C2f module, Layers 4-5 that are each composed of a SCDown module and a C2f module, and Layer 6 that is composed of a SPPF module and a DAttention module.

[0021] Furthermore, the process of extracting feature values ​​by the DAttention module is as follows:

[0022] (1) Feature embedding: mapping input features to a higher-dimensional space;

[0023] (2) Correlation calculation: Calculate the correlation between each pair of features;

[0024] (3) Attention weight generation: Generate attention weights using correlation as a basis;

[0025] (4) Attention feature fusion: The generated attention weights are applied to the features to enhance the model’s attention to important features.

[0026] Furthermore, the multi-scale feature fusion module includes a multi-scale attention feature pyramid structure NSAFPS module, and the NSAFPS module includes a BiFPN module that can dynamically adjust feature weights and realize bidirectional feature propagation to fuse features at different levels, a C2f module, and a multi-scale attention EMA module that can maximize the expected value of attention weights.

[0027] Furthermore, the process of the multi-scale attention EMA module is as follows: introduce the output tensors of the 1x1 and 3x3 branches, use two-dimensional global average pooling and adopt the Sofmax function to fit the linear transformation, multiply the output with the matrix dot product operation to obtain the first spatial attention map; then encode the global spatial information of the 3x3 branch and directly convert the 1x1 branch into the corresponding dimensional shape before the joint activation mechanism of the channel feature, based on which the second spatial attention map is obtained; finally, the global context is captured by the set of two spatial attention weight values, and the final output size is equal to the size of the input feature map.

[0028] Furthermore, the regression detection module includes a Wise-IoU v3 loss function that can improve the generalization ability of the network. The process of Wise-IoU v3 is as follows:

[0029] (1) Calculate the IOU value: For each true bounding box and predicted bounding box, calculate their IOU value to measure the degree of overlap between the predicted bounding box and the true bounding box;

[0030] (2) Select matching target: For each predicted bounding box, select the real bounding box with the largest IOU as its matching target;

[0031] (3) Calculate coordinate loss: Calculate the coordinate loss between the predicted bounding box and its matching target, including the square difference of the center point coordinates and the square difference of the width and height;

[0032] (4) Calculate category loss: Use the cross entropy loss function to calculate the category loss of the predicted bounding box to ensure that the category of the predicted bounding box is consistent with the true category;

[0033] (5) Merged loss: Add the coordinate loss and category loss to get the final Wise-IoU v3 loss value.

[0034] Furthermore, the process of generating the predicted image in step 4 is as follows:

[0035] (1) Input image: The image to be detected is used as the input of the model;

[0036] (2) Forward propagation: forward propagation is performed on the input image to obtain the prediction result of each position through the multi-layer structure of the neural network;

[0037] (3) Prediction box generation: Multiple bounding box predictions are generated for each position. Each bounding box corresponds to an object. These bounding boxes are screened and the bounding boxes with higher confidence are retained to determine the final prediction box.

[0038] (4) Non-maximum suppression: Non-maximum suppression is applied to the retained prediction boxes to remove boxes with more overlap and lower confidence.

[0039] (5) Obtain the final result: After non-maximum suppression, the retained box is the final predicted box, and each box contains the location information and category information of the target.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention is based on a deformable multi-scale adaptive detection network, and achieves high-accuracy classroom behavior recognition through a feature extraction module, a multi-scale feature fusion module, and a regression detection module. By dividing the sample data set into a training set, a validation set, and a test set, the model training process is ensured to be efficient and scientific, which helps to avoid overfitting.

[0042] Furthermore, the feature extraction module of the present invention introduces a DAttention module, which can dynamically adjust the sampling position and attention weight, so that the model can pay more attention to the smaller behavioral targets in the back row classroom behavior in the image, extract more refined and accurate feature representations, and improve the modeling ability of the target.

[0043] Furthermore, the present invention proposes a multi-scale attention feature pyramid structure NSAFPS, which combines the expectation-maximization deformable attention and the weighted feature pyramid network to improve the feature quality and reliability while realizing the effective transmission and fusion between features at different levels, and realizes the denoising, purification and effective fusion of features, thereby obtaining a more robust feature expression and reducing the impact of mutual occlusion and scale differences between classroom behaviors in the front and back rows.

[0044] Furthermore, the present invention introduces a bounding box localization loss function Wise-IoUv3 with a dynamic non-monotonic focusing mechanism to solve the problems that traditional IoU ignores surrounding information and is sensitive to data quality. By introducing weights and considering surrounding area information, the influence of low-quality target samples on the generalization ability of the model is reduced, thereby improving the model's detection ability for low-quality target samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the overall architecture diagram of the deformable multi-scale adaptive detection network of the present invention;

[0046] Figure 2 It is a schematic diagram of the BackBone structure of the present invention;

[0047] Figure 3(a) is a schematic diagram of the original BiFPN structure;

[0048] FIG3( b ) is a schematic diagram of the structure of the NSAFPS after the original BiFPN is modified according to the present invention;

[0049] Figure 4 It is a schematic diagram of the structural composition of the NSAFPS of the present invention. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, the classroom behavior recognition method based on a deformable multi-scale adaptive detection network described in this embodiment includes the following steps:

[0052] Step 1: Build a deformable multi-scale adaptive detection network

[0053] The deformable multi-scale adaptive detection network network includes feature extraction module, multi-scale feature fusion module and regression detection module. The feature extraction module Backbone is the main component of the network, responsible for extracting the features of the input image for subsequent processing and analysis, and has a multi-layer structure and multi-parameter settings, which can extract high-level feature representations of the image. The structure of Backbone is a six-layer feature pyramid constructed by the CNN network, such as Figure 2 As shown in the figure, the first layer L1 extracts the underlying feature information from the input image by convolution operation, the second and third layers L2 and L3 are composed of Conv modules and C2f modules, the fourth and fifth layers L4 and L5 are composed of SCDown modules and C2f modules, and the sixth layer L6 is composed of SPPF modules and DAttention modules.

[0054] The core of BackBone is the introduction of the deformable attention mechanism DAttention, which is a simple and efficient deformable self-attention module and can also be seen as a spatial adaptation mechanism that allows the model to dynamically adjust its attention focus when processing data. Traditional attention mechanisms usually pay intensive attention to the information of all grids when sampling, while the deformable self-attention mechanism DAttention enables the model to more accurately capture the key features of objects, while paying more attention to the target area of ​​classroom behavior, extracting more refined and accurate feature representations, and improving the modeling ability and feature expression ability of distant targets.

[0055] The process of extracting feature values ​​by the DAttention module is as follows:

[0056] (1) Feature embedding: mapping input features to a higher-dimensional space;

[0057] (2) Correlation calculation: Calculate the correlation between each pair of features;

[0058] (3) Attention weight generation: Generate attention weights using correlation as a basis;

[0059] (4) Attention feature fusion: The generated attention weights are applied to the features to enhance the model’s attention to important features.

[0060] The deformable self-attention mechanism DAttention can better capture the correspondence between feature points at different scales. By predicting an offset, the attention distribution is no longer fixed, but can be adaptively changed according to the characteristics of the data. This method enables the network model to fully focus on the key feature information in the classroom behavior image, especially to solve the neglect of small target behaviors in the back row, and does not simply expand the receptive field, thereby reducing unnecessary computational consumption. Deformable self-attention uses multiple sets of deformable sampling points to determine the important areas of the feature map, and uses this important area as the basis to model the relationship between different features of classroom behavior. Reduce the impact of irrelevant background, non-classroom behavior, and occlusion between behaviors, thereby improving the expressiveness of features.

[0061] The DAttention mechanism can dynamically adjust the sampling position and attention weight, allowing the model to more accurately focus on small target behaviors at long distances, enhancing the ability to model behavioral features in real classroom environments. By introducing DAttention in the Backbone layer, it is possible to effectively extract robust features that can express targets of different scales from classroom behavior images, providing strong support for subsequent target detection tasks.

[0062] The multi-scale feature fusion module Neck includes a multi-scale attention feature pyramid structure NSAFPS module. The NSAFPS module includes a BiFPN module, a C2f module, and a multi-scale attention EMA module. The BiFPN module adjusts the importance of features on different paths through dynamic feature weights, selectively fuses features, and realizes bidirectional feature propagation through multiple output connections, reducing information loss and propagation delay, so as to achieve efficient fusion between features at different levels. The EMA module maximizes the expected value of the attention weight, so that the model pays more attention to the important information in the features, and smoothes the weight distribution to reduce the interference of noise features, thereby purifying the output features. The BiFPN module can effectively fuse features at different levels and improve the accuracy and speed of target detection; the EMA module provides channel and cross-spatial information interaction, and can adjust the attention to the feature map according to the scale requirements of each detection head. The NSAFPS module combines the expected maximization attention and the weighted feature pyramid network to improve the quality and reliability of features while achieving effective transmission and fusion between features at different levels.

[0063] The main function of Neck is to reduce the dimension or adjust the features from Backbone, further extract and integrate features from multiple scales, and fuse features from different scales, thereby enhancing the model's ability to capture information of different scales in the image, effectively fusing features at different levels, and reducing the impact of mutual occlusion and scale differences between classroom behaviors in the front and back rows.

[0064] The Neck part designs a multi-scale attention feature pyramid structure NSAFPS, which is based on the original BiFPN structure, as shown in Figure 3(a). BiFPN adjusts the importance of features on different paths through dynamic feature weights, selectively fuses features, and realizes bidirectional feature propagation through multiple output connections, reducing information loss and propagation delay, so as to achieve efficient fusion between features at different levels. The original BiFPN structure adopts a bidirectional feature pyramid network structure, which effectively solves the problems of information bottleneck and feature distortion in the feature pyramid network through adaptive feature fusion and feature selection. At the same time, BiFPN also adopts cross-level feature connections and multi-scale feature fusion to further improve the performance of target detection. However, the original BiFPN structure has five output detection heads, and the feature input and detection heads in the present invention are only three. In order to introduce the advantages of BiFPN, BiFPN is transformed into three output detection heads, thereby obtaining a NSAFPS structure with three detection heads, as shown in Figure 3(b).

[0065] like Figure 4 As shown in Figure 1, the NSAFPS structure includes a BiFPN module, a C2f module, and a multi-scale attention EMA module. The BiFPN module adjusts the importance of features on different paths through dynamic feature weights, selectively fuses features, and realizes bidirectional feature propagation through multiple output connections, reducing information loss and propagation delay, so as to achieve efficient fusion between features at different levels. The EMA module is used to maximize the expected value of the attention weight, so that the model pays more attention to the important information in the features, and smoothes the weight distribution to reduce the interference of noise features, thereby purifying the output features. The EMA module focuses on retaining the information of each channel and reducing the computational overhead, reshaping some channels into batch dimensions, and grouping the channel dimensions into multiple sub-features, so that the spatial semantic features are evenly distributed within each feature group. Specifically, in addition to encoding global information into each parallel branch to recalibrate the weights on the channel dimension, the output features of the two parallel branches are further aggregated through cross-dimensional interactions to capture the relationship between pixel-level pairs.

[0066] A parallel subnetwork structure is used in EMA, and three parallel paths are used to extract the attention weight descriptor of the grouped feature map. At the same time, multi-scale features are paid attention to to better capture the overall structure and local details of the target; and cross-spatial information aggregation of different spatial dimensions is used to capture the global and local relationships between the four types of targets with position distribution patterns to obtain richer aggregate features. The process of this module is as follows:

[0067] (1) The output tensors of the 1x1 and 3x3 branches are introduced, and the two-dimensional global average pooling is used to fit the linear transformation using the Sofmax function. The output is multiplied by the matrix dot product operation to obtain the first spatial attention map.

[0068] (2) Encode the global spatial information of the 3x3 branch and directly convert the 1x1 branch into the corresponding dimensional shape before the joint activation mechanism of the channel features, based on which the second spatial attention map is obtained;

[0069] (3) The global context is captured based on the set of two spatial attention weight values, and the final output size is equal to the size of the input feature map.

[0070] The regression detection module Head maps the extracted features to the final output space and generates the final prediction results of the network. In order to better measure the similarity between the predicted bounding box and the true bounding box, the present invention adopts the Wise-IoU v3 loss function, which reduces the competitiveness of high-quality anchor boxes while also reducing the harmful gradients generated by low-quality examples. This allows Wise-IoU to focus on anchor boxes of ordinary quality and improve the overall performance of the detector. The Wise-IoU v3 loss function weakens the penalty of the set metric when the overlap between the anchor box and the target box is high, reduces the intervention in pre-training, and thus improves the generalization ability of the network. The process of Wise-IoU v3 is as follows:

[0071] (1) Calculate the IOU value: First, for each true bounding box and predicted bounding box, calculate their IOU value. IOU is used to measure the degree of overlap between the predicted bounding box and the true bounding box.

[0072] (2) Select matching target: For each predicted bounding box, select the ground-truth bounding box with the largest IOU as its matching target. This step ensures that each predicted bounding box has a corresponding ground-truth bounding box for comparison.

[0073] (3) Calculate coordinate loss: Calculate the coordinate loss between the predicted bounding box and its matching target, including the square difference of the center point coordinates and the square difference of the width and height. This step is to adjust the position and size of the predicted bounding box to make it closer to the real bounding box.

[0074] (4) Calculate the category loss: Use the cross entropy loss function to calculate the category loss of the predicted bounding box. The category loss is used to ensure that the category of the predicted bounding box is consistent with the true category.

[0075] (5) Merge loss: Add the coordinate loss and the category loss to get the final Wise-IoU v3 loss value. This step ensures that the loss function comprehensively considers the errors in position, size, and category.

[0076] The specific process of classroom behavior recognition through deformable multi-scale adaptive detection network is as follows:

[0077] (1) Collect classroom behavior images and input them into the feature extraction module, convert the original input image into a multi-layer feature map, and extract the feature information of the image;

[0078] (2) Inputting the feature information output from the feature extraction module at different scales into the multi-scale feature fusion module for fusion to obtain fused feature information;

[0079] (3) The fused feature information is input into the regression prediction module for behavior recognition to obtain the final recognition result.

[0080] Step 2: Use the open source SCB-Dataset3-S and DataFountainSCB classroom behavior sample datasets, divide the classroom behavior sample datasets into training set, validation set and test set sample datasets, extract student behavior features from the images in the training set through a deformable multi-scale adaptive detection network, and perform model training

[0081] The model training process is:

[0082] (1) Set the training parameters: training cycle is set to 500, batch size is set to 4, and number of processes is set to 8.

[0083] (2) In model training, the SGD optimizer is used for training. The model uses a weight decay strategy to prevent overfitting, and the weight decay value is set to 5×10 -4 , momentum 0.937. Early-Stopping is used for model training. When the model detects that the validation loss tends to be stable, the training is automatically terminated and the model reaches basic convergence.

[0084] (3) Input and label preparation: Provide a labeled training dataset, including images and corresponding object bounding boxes and category labels.

[0085] (4) Forward propagation: Input the image into the model, perform forward propagation calculations, and obtain the output of the model.

[0086] (5) Calculate loss: Compare the model output with the label and calculate the loss function.

[0087] (6) Back propagation: The back propagation algorithm is used to calculate the gradient of the model parameters based on the loss value.

[0088] (7) Parameter update: Use gradient descent or other optimization algorithms to update the model parameters to reduce the loss function.

[0089] (8) Iterative training: Repeat forward propagation, loss calculation, backpropagation, and parameter update until the model converges or reaches the predetermined training rounds.

[0090] Step 3: Use the model trained in step 2 to verify the validation set

[0091] After each model training cycle, the obtained model is evaluated on the validation set to determine whether the model is overfitting, underfitting, etc., and then select the best hyperparameters for the next round of training. The validation process is as follows:

[0092] (1) Data preparation: Prepare a data set for verification;

[0093] (2) Model loading: Load the pre-trained model in step 2;

[0094] (3) Start the verification process: traverse each image in the verification set and use the model to detect it;

[0095] (4) Evaluation results: The mean average precision (mAP) is used to evaluate the detection performance.

[0096] Step 4: Obtain the best model parameters for the validation set, input the photos in the test set for testing and generate prediction images to simulate the performance of the model in actual application, so as to evaluate the generalization performance of the trained model and complete the detection of students' classroom behavior

[0097] The process of generating the predicted image is:

[0098] (1) Input image: The image to be detected is used as the input of the model;

[0099] (2) Forward propagation: forward propagation is performed on the input image to obtain the prediction result of each position through the multi-layer structure of the neural network;

[0100] (3) Prediction box generation: Multiple bounding box predictions are generated for each position. Each bounding box corresponds to an object. These bounding boxes are screened and the bounding boxes with higher confidence are retained to determine the final prediction box.

[0101] (4) Non-maximum suppression: Non-maximum suppression is applied to the retained prediction boxes to remove boxes with more overlap and lower confidence.

[0102] (5) Obtain the final result: After non-maximum suppression, the retained box is the final predicted box, and each box contains the location information and category information of the target.

[0103] Experimental part:

[0104] 1) An ablation experiment is performed on the SCB-Dataset3-S dataset to verify the effectiveness of each module. Here, YOLOv10 is used as the Baseline. The results of the SCB-Dataset3-S dataset are shown in Table 1:

[0105] Table 1 Ablation experiment of SCB-Dataset3-S dataset

[0106]

[0107] The mAP50 of the Baseline is 73.1%. This is because the Baseline includes multiple convolutional layers and pooling layers in the Backbone part, which are used to extract low-level and high-level features in the image. The convolutional layer adopts the C2f structure, adds more skip-layer connections, and adds additional split operations to enrich the feature information while reducing the amount of calculation. While maintaining lightweight, it can obtain richer gradient flow information. These operations can achieve good results in natural scenes, but for images of classroom behavior recognition scenes, the background is complex, there are many small targets, and some objects are severely occluded, so there are still certain limitations.

[0108] The DAttention module is introduced on the Baseline, and the mAP50 of the model is improved by 0.4% compared with the mAP50 of the Baseline. This is because the DAttention module combines the local perception ability of CNN with the global abstraction ability of ViT, which helps the model to better focus on the area of ​​interest, effectively capture the spatial information in the image, and enhance the modeling of student behavior information in the classroom behavior image under complex environment. After the NSAFPS structure is introduced on the Baseline, the mAP50 of the model is improved by 0.7% compared with the mAP50 of the Baseline, indicating that NSAFPS combines the expected maximization attention and the weighted feature pyramid network to improve the quality and reliability of features while realizing the effective transmission and fusion between features of different levels and scales. After adding the Wise-IoU loss function on the Baseline, the mAP50 of the model is improved by 0.6% compared with the mAP50 of the Baseline, indicating that the Wise-IoU loss function can effectively measure the overlap of the target box and minimize the impact of low-quality samples.

[0109] After adding the DAttention module and the NSAFPS module to the Baseline at the same time, the mAP50 of the model increased by 1.1% compared with the mAP50 of the Baseline, and the performance was also greatly improved compared with the previous addition of the modules alone, indicating that the two modules have good independence and can play a good role in different stages of detection. After adding the three modules of DAttention, NSAFPS and Wise-IoU to the Baseline, the best effect was achieved. The mAP50 of the model increased by 1.4% compared with the mAP50 of the Baseline, indicating that these modules have a synergistic effect. By cooperating with each other and optimizing various aspects of the network, they can better adapt to the target detection tasks in complex scenes and jointly improve the performance of the target detection task in classroom behavior images.

[0110] 2) Ablation experiments are performed on the DataFountainSCB dataset. The results of the DataFountainSCB dataset are shown in Table 2:

[0111] Table 2 DataFountainSCB dataset ablation experiment

[0112]

[0113] Compared with only one module, the network with different blocks superimposed achieves better performance. The addition of three modules enables the model to capture the spatial information in the classroom behavior images, and to perceive the extracted features dimensionally, integrate them adaptively, and effectively suppress the interference of complex backgrounds to achieve better regression and classification results. At the same time, the experiments also show that there is no conflict between the proposed modules. When the three modules of DAttention, NSAFPS and Wise-IoU are used at the same time, the mAP50 of the model shows the best performance of 94.8%.

[0114] 3) A comparative experiment was conducted on the SCB-Dataset3-S dataset with other existing classic methods, as shown in Table 3:

[0115] Table 3 Comparative experiments based on SCB-Dataset3-S

[0116]

[0117] For the whole classroom behavior target detection, the mAP50:95 of the present invention is 1.2% higher than that of YOLOv10. The mAP50 of the present invention is 3.4% higher than that of YOLOv5n, 2.1% higher than that of YOLOv8n, and 1.45% higher than that of YOLOv10n, which shows that the model can still maintain good performance when the overlap between the predicted bounding box and the real box is low.

[0118] 4) On the DataFountainSCB dataset, the method of the present invention is compared with the mainstream advanced methods. It can be seen that the method of the present invention also achieves improvement on this dataset, as shown in Table 4:

[0119] Table 4 Comparative experiments of different models in DataFountainSCB

[0120]

Claims

1. A classroom behavior recognition method based on a deformable multi-scale adaptive detection network, characterized in that: The following steps are involved: Step 1: Building a deformable multi-scale adaptive detection network, which includes a feature extraction module, a multi-scale feature fusion module and a regression detection module; Step 2: Divide the classroom behavior sample dataset into training set, validation set and test set sample datasets, extract student behavior features from the images in the training set through a deformable multi-scale adaptive detection network, and perform model training; Step 3: Use the model obtained from the training set to verify the validation set; Step 4: Obtain the best model parameters for the validation set, input photos in the test set for testing to generate predicted images, and complete the detection of students' classroom behavior.

2. According to claim 1, a method for classroom behavior recognition based on a deformable multi-scale adaptive detection network is characterized in that: The specific process of classroom behavior recognition using the deformable multi-scale adaptive detection network described in step 1 is as follows: (1) Collect classroom behavior images and input them into the feature extraction module, convert the original input image into a multi-layer feature map, and extract the feature information of the image; (2) Inputting the feature information output from the feature extraction module at different scales into the multi-scale feature fusion module for fusion to obtain fused feature information; (3) The fused feature information is input into the regression prediction module for behavior recognition to obtain the final recognition result.

3. According to claim 2, a method for classroom behavior recognition based on a deformable multi-scale adaptive detection network is characterized in that: The structure of the feature extraction module includes: Layer 1 that extracts underlying feature information from an input image through a convolution operation, Layers 2-3 that are each composed of a Conv module and a C2f module, Layers 4-5 that are each composed of an SCDown module and a C2f module, and Layer 6 that is composed of an SPPF module and a DAttention module.

4. According to claim 3, a method for classroom behavior recognition based on a deformable multi-scale adaptive detection network is characterized in that: The process of extracting feature values ​​by the DAttention module is as follows: (1) Feature embedding: mapping input features to a higher-dimensional space; (2) Correlation calculation: Calculate the correlation between each pair of features; (3) Attention weight generation: Generate attention weights using correlation as a basis; (4) Attention feature fusion: The generated attention weights are applied to the features to enhance the model’s attention to important features.

5. The method for classroom behavior recognition based on a deformable multi-scale adaptive detection network according to claim 2 is characterized in that: The multi-scale feature fusion module includes a multi-scale attention feature pyramid structure NSAFPS module, and the NSAFPS module includes a BiFPN module that can dynamically adjust feature weights and realize bidirectional feature propagation to fuse features at different levels, a C2f module, and a multi-scale attention EMA module that can maximize the expected value of attention weights.

6. The method for classroom behavior recognition based on a deformable multi-scale adaptive detection network according to claim 5 is characterized in that: The process of the multi-scale attention EMA module is as follows: introduce the output tensors of the 1x1 and 3x3 branches, use two-dimensional global average pooling and adopt the Sofmax function to fit the linear transformation, multiply the output with the matrix dot product operation to obtain the first spatial attention map; then encode the global spatial information of the 3x3 branch and directly convert the 1x1 branch into the corresponding dimensional shape before the joint activation mechanism of the channel features, based on which the second spatial attention map is obtained; finally, the global context is captured by the set of two spatial attention weight values, and the final output size is equal to the size of the input feature map.

7. The method for classroom behavior recognition based on a deformable multi-scale adaptive detection network according to claim 2 is characterized in that: The regression detection module includes the Wise-IoU v3 loss function that can improve the generalization ability of the network. The process of Wise-IoU v3 is as follows: (1) Calculate the IOU value: For each true bounding box and predicted bounding box, calculate their IOU value to measure the degree of overlap between the predicted bounding box and the true bounding box; (2) Select matching target: For each predicted bounding box, select the real bounding box with the largest IOU as its matching target; (3) Calculate coordinate loss: Calculate the coordinate loss between the predicted bounding box and its matching target, including the square difference of the center point coordinates and the square difference of the width and height; (4) Calculate category loss: Use the cross entropy loss function to calculate the category loss of the predicted bounding box to ensure that the category of the predicted bounding box is consistent with the true category; (5) Merged loss: Add the coordinate loss and category loss to get the final Wise-IoU v3 loss value.

8. The method for classroom behavior recognition based on a deformable multi-scale adaptive detection network according to claim 1 is characterized in that: The process of generating the predicted image in step 4 is as follows: (1) Input image: The image to be detected is used as the input of the model; (2) Forward propagation: forward propagation is performed on the input image to obtain the prediction result of each position through the multi-layer structure of the neural network; (3) Prediction box generation: Multiple bounding box predictions are generated for each position. Each bounding box corresponds to an object. These bounding boxes are screened and the bounding boxes with higher confidence are retained to determine the final prediction box. (4) Non-maximum suppression: Non-maximum suppression is applied to the retained prediction boxes to remove boxes with more overlap and lower confidence. (5) Obtain the final result: After non-maximum suppression, the retained box is the final predicted box, and each box contains the location information and category information of the target.

Images