A deep learning-based student classroom behavior detection method

Abstract

This invention belongs to the field of object detection technology, specifically relating to a deep learning-based method for detecting student classroom behavior. The method includes: training an improved YOLOv11 model; pruning the trained improved YOLOv11 model to obtain a student classroom detection model; acquiring classroom videos of students to be detected and inputting them into the student classroom detection model for processing to obtain student classroom behavior detection results; and using an improved DeepSORT algorithm to process the student classroom behavior detection results to achieve tracking of student classroom behavior. This invention enhances the network's ability to capture global information, improves the network's performance in occlusion and multi-scale change scenarios, thereby improving the network's robustness and detection accuracy, and has good application prospects.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a method for detecting student classroom behavior based on deep learning. Background Technology

[0002] With the widespread use of cloud and online classrooms in recent years, and the development of smart classrooms as a key national project, the demand for recording and recognizing student classroom performance has increased significantly. Many studies in education use student actions and behaviors to predict and assess future classroom performance and achievement. Traditionally, teachers maintain classroom discipline and manually record student behavior. This method has two problems: first, it distracts the teacher, reducing teaching efficiency; second, it's impossible to comprehensively and accurately record the learning behavior of all students. Today, using computer vision technology to identify student states in the classroom has become a crucial technological foundation for building smart classrooms. Deep learning technology can provide data support for the construction of smart classrooms, statistically analyzing student behavior in the background to better assess student learning.

[0003] Currently, algorithms for student behavior detection fall into three categories: those based on video action recognition, those based on gesture estimation, and those based on object detection. Video action recognition is used to identify persistent behaviors in student classrooms; however, this requires labeling a large number of samples. Pose estimation algorithms describe human behavior by acquiring the position and movement information of various joints, but they are not suitable for behavior detection in crowded classrooms. Object detection algorithms are a promising solution, as they can identify specific student behaviors during learning, such as focusing, asking questions, and answering questions. By understanding each student's behavioral patterns, educators can provide more personalized and targeted educational support to meet the diverse learning needs of students. By analyzing student behavior, educators can more accurately assess students' comprehension of the teaching content and their learning outcomes. This helps adjust teaching strategies to improve teaching effectiveness and student academic performance.

[0004] With the widespread application of deep learning, object detection algorithms have developed rapidly. Object detection methods can be divided into two categories: traditional object detection and deep learning-based object detection. Traditional object detection algorithms generally consist of three steps: first, generating candidate boxes; then, extracting features from the candidate boxes; and finally, classifying based on the features. When generating candidate boxes, exhaustive search methods are typically used, but this method is computationally intensive and suffers from repetitive computation, resulting in low efficiency. Furthermore, traditional algorithms rely on manually designed features, making it difficult to guarantee feature stability. Therefore, traditional methods have limitations in both accuracy and speed. With the development of deep learning, deep learning-based object detection algorithms have achieved significant improvements in prediction speed and accuracy. These methods have gradually become mainstream, mainly divided into two categories: one-stage object detection algorithms and two-stage object detection algorithms. Compared to two-stage algorithms, one-stage algorithms may have slightly lower accuracy, but their detection process only needs to be completed once, with less computation, making them suitable for real-time applications. In one-stage algorithms, the YOLO series achieves a good balance between accuracy and speed; therefore, this invention selects the YOLO series as the basic model for student object detection. Based on this, and considering the characteristics of actual teaching scenarios and research objectives, this invention proposes an improved YOLOv11 algorithm for classroom behavior detection. This algorithm aims to enhance the network's ability to capture global information and improve its performance in occlusion and multi-scale changing scenarios, thereby increasing the network's robustness and detection accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a deep learning-based method for detecting student classroom behavior, which includes:

[0006] S1: Train the improved YOLOv11 model, prune the trained improved YOLOv11 model, and obtain the student classroom assessment model;

[0007] S2: Obtain classroom videos of the students to be detected and input them into the student classroom detection model for processing to obtain the student classroom behavior detection results;

[0008] S3: Use the improved DeepSORT algorithm to process the results of student classroom behavior detection to achieve tracking of student classroom behavior.

[0009] Preferably, the improved YOLOv11 model includes an improved Backbone network, an improved Neck network, and an improved Head network. The improved Backbone network replaces the C3K2 module in the original YOLOv11 model's Backbone network with the Star_caa module. Both the improved Backbone network and the improved Neck network replace the Conv downsampling module in the original YOLOv11 model with the LD_downsample downsampling module. The improved Head network replaces the detector head in the original YOLOv11 model's Head network with the Dyhead_DCNv4 detector head.

[0010] Furthermore, the Star_caa module's processing of input features includes: processing the input features using a first DWConv layer to obtain a first intermediate feature; processing the first intermediate feature using two parallel 1×1 convolutional layers and performing a star operation on the two convolutional outputs to obtain a second intermediate feature; processing the second intermediate feature using the CAA attention mechanism and then inputting it into a 1×1 convolutional layer to obtain a third intermediate feature; inputting the third intermediate feature into a second DWConv layer for processing to obtain a fourth intermediate feature; and performing a residual connection between the fourth intermediate feature and the input features and then performing regularization to obtain the output feature of the Star_caa module.

[0011] Furthermore, the LD_downsample downsampling module includes two branches: one branch connects the max pooling layer to the CBS module, and the other branch connects the CBS module to the LDConv layer; the outputs of the two branches are concatenated as the output features of the LD_downsample downsampling module.

[0012] Furthermore, the CBS module consists of a standard convolutional layer, a batch normalization layer, and an activation function Silu.

[0013] Furthermore, the Dyhead_DCNv4 detection header is represented as follows:

[0014] W(F) = π c (π s (π L (F)·F)·F)·F

[0015] Where W(F) represents the output features of the Dyhead_DCNv4 detector, F represents the input features of the Dyhead_DCNv4 detector, and πL(·), πS(·), and πC(·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

[0016] Preferably, the LAMP pruning algorithm is used to prune the trained improved YOLOv11 model.

[0017] Preferably, the improved DeepSORT algorithm replaces the state prediction results obtained by the Kalman filter in the original DeepSORT algorithm with a weighted sum of the state prediction results obtained by the LSTM network and the state prediction results obtained by the Kalman filter, and replaces the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network.

[0018] The beneficial effects of this invention are as follows: The student classroom behavior detection method proposed in this invention enhances the extraction of global image information and improves the model's detection performance by replacing the original C3K2 module in the backbone network with the Star_caa module. The DYHead_DCNV4 detection head module, through multi-scale feature fusion attention mechanism and variable convolution, strengthens the model's understanding of global and local information, resulting in more stable and accurate performance when dealing with occlusion and complex backgrounds. Furthermore, this invention optimizes the downsampling strategy, using the LD_downsample downsampling module to reduce the loss of detailed information, improve feature extraction performance, and significantly reduce computational load. While meeting the requirements of real-time performance and mobile deployment, this invention combines LAMP pruning technology to achieve model lightweighting, reduce redundant weights and computational resource consumption, and improve mobile performance. This invention also combines LSTM network processing of time-series data to capture long-term motion patterns of targets, improving the prediction effect of nonlinear trajectories. To further reduce computational complexity and maintain strong feature extraction capabilities, MobileNetV4 is used as the feature extractor for DeepSORT, achieving efficient tracking of student behavior. Attached Figure Description

[0019] Figure 1 This is a flowchart of the process for detecting classroom behavior of middle school students in this invention;

[0020] Figure 2 This is a schematic diagram of the improved YOLOv11 model structure in this invention;

[0021] Figure 3 This is a schematic diagram of the Sta_caa module structure in this invention;

[0022] Figure 4 This is a schematic diagram of the depth-separable convolutional structure in this invention.

[0023] Figure 5 This is a schematic diagram of the CAA attention mechanism structure in this invention;

[0024] Figure 6 This is a schematic diagram of the LD_downsample downsampling module structure in this invention;

[0025] Figure 7 This is a schematic diagram of the LDConv structure in this invention;

[0026] Figure 8 This is a schematic diagram of the Dyhead_DCNv4 detection head structure in this invention;

[0027] Figure 9 This is a schematic diagram of the LSTM network structure in this invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This invention proposes a deep learning-based method for detecting student classroom behavior, such as... Figure 1 As shown, the method includes the following:

[0030] S1: Train the improved YOLOv11 model, prune the trained improved YOLOv11 model, and obtain the student classroom assessment model.

[0031] This invention designs and improves the YOLOv11 model based on the original model to enhance its ability to extract global image information and address multi-scale issues in student classrooms. Specifically, the improved YOLOv11 model integrates several new modules and optimization strategies, including the backbone feature extraction module Star_caa, the super downsampling MP module (LD_downsample), and the multi-scale dynamic attention detection head Dyhead_DCNv4. The improved Backbone network replaces the C3K2 module in the original YOLOv11 model's Backbone network with the Star_caa module; both the improved Backbone and Neck networks replace the Conv downsampling module in the original YOLOv11 model with the LD_downsample downsampling module; and the improved Head network replaces the detection head of the original YOLOv11 model's Head network with the Dyhead_DCNv4 detection head.

[0032] like Figure 2As shown, the Backbone network extracts initial features through multi-layer convolutions and the Star_caa module, while using Spatial Pyramid Pooling Fast (SPPF) to aggregate multi-scale contextual information. The Neck network enhances the model's feature representation capability by combining C3K2 with MP downsampling and global context fusion. Upsampling and feature concatenation further ensure the preservation of details in high-resolution feature maps and improve detection accuracy. The Dyhead_DCNv4 detection head processes the feature maps and generates object detection results, including predicted bounding boxes, class confidence, and location information. The loss function and batch normalization work together through different mechanisms to optimize model performance. The loss function provides a clear optimization objective, while batch normalization (BN) accelerates the training process and improves the stability and efficiency of model training by stabilizing the feature distribution. This improved YOLOv11 model aims to reduce computational complexity while maintaining high detection accuracy and speed through these optimizations.

[0033] Star_caa module:

[0034] In real-world applications, student behavior detection involves real-time processing of multiple targets, making detection accuracy particularly crucial. The convolutional and information fusion operations in each Bottleneck layer of the C3K2 module neglect the multi-scale issues that arise in real-world student scenarios. Furthermore, on memory-constrained devices and when processing tasks with relatively simple features, multi-layered Bottlenecks reduce inference efficiency and negatively impact the model's real-time detection performance. To enhance the model's ability to detect the details and features of student behavior and improve the student detection rate, this invention constructs a Star_caa module to address the low detection rate caused by differences in the receptive fields of students before and after detection.

[0035] The Star_caa module optimizes overall model performance by integrating two key technologies from StarNet: the Starblok module and the CAA attention mechanism. For example... Figure 3As shown, the Star_caa module's processing of input features includes: using the first DWConv (depth-separable convolution) layer to process the input features; the input tensor is convolved through a depth-separable convolution layer with a kernel of 7; each channel is convolved individually to obtain the first intermediate feature. Feature extraction, channel dimension adjustment, and channel information fusion are performed using two parallel 1×1 convolutional layers FC1 and FC2. A star operation is performed on the outputs of the two convolutions: FC1 is activated using the ReLU6 activation function, and then star-operated with FC2 to obtain the second intermediate feature. This operation creates a large number of interaction terms, significantly increasing the dimensionality of the feature space. Compared to simple linear transformations, the star operation can capture more complex feature relationships, thus improving the flexibility and accuracy of feature extraction. The second intermediate feature is then reorganized using the CAA attention mechanism to obtain contextual information. The tensor output by the CAA attention mechanism is compressed using a 1×1 convolutional layer G to restore the number of channels to the level before the parallel 1×1 convolutional layers, resulting in the third intermediate feature. The third intermediate feature is then input into the second DWConv layer for further feature extraction, yielding the fourth intermediate feature. The fourth intermediate feature is then residually connected to the input features and regularized to alleviate the gradient vanishing problem in deep networks, thereby improving the ability to extract information from distant objects, resulting in the output features of the Star_caa module.

[0036] The calculation process of the Star_caa network is shown in the following formula:

[0037] W fc1 =FC1(DWConv 1×1 (x))+b1

[0038] W fc =FC2(DWConv 1×1 (x))+b2

[0039] W g =CAA(Relu6(W fc1 )*W fc)

[0040] W star =x+DropPath(DWConv 1×1 (W g (x)+b g ))

[0041] In the above formula, x represents the input of the Star_caa module, and W fc1 W fc2 This indicates that the channels are expanded using two 1×1 convolutions, where FC1 and FC2 represent the weights of these two convolutional layers, b1 and b2 are bias terms, and W... gAfter channel expansion, it is activated into a non-linear value using an activation function and then compared with W. fc2 The Star_caa network structure employs star operations and introduces a context anchor attention mechanism. Finally, it adds the input x to the feature map processed by depthwise separable convolutions via residual connections. This enhances the detection capability for targets at different scales through CAA attention and star operations. By combining local and global information from different levels of the feature map, it improves the detection accuracy for small and distant targets.

[0042] Depthwise separable convolutions, such as Figure 4 As shown, by dividing standard convolution into two steps, the first operation uses depthwise convolution, which applies a k×k convolution kernel to each input channel separately, instead of operating across channels as in standard convolution. That is, if the input has C... in For each channel, use C. in Each channel is processed by a k×k convolutional kernel. The output remains H×W×C. in The feature maps are calculated because each channel operates only with its corresponding convolutional kernel; the calculation formula for this process is expressed as:

[0043]

[0044] In the above formula, W depthwise,k,l,i It is the K×K convolution kernel of the i-th channel, y depthwise (p,i) is the output value at position p and channel i.

[0045] The second step is pointwise convolution, which uses a 1×1 convolution kernel to linearly combine information from different channels to generate C. out A new channel. Pointwise convolution is equivalent to performing C on each spatial location. in ×C out A linear transformation of dimensionality, thus compressing the result of depthwise convolution to H×W×C. in ×C out Using depthwise separable convolution splits the computation of standard convolution into two smaller operations, reducing the computational cost by approximately [amount missing]. The effect is particularly significant when the convolution kernel is large. Therefore, the number of network parameters can be reduced, thus reducing the computational cost of the model. The formula for pointwise convolution is shown below:

[0046]

[0047] In the above formula, W pointwise,i,j It uses 1×1 convolutional kernel weights to map the i-th channel to the j-th output channel, y pointwise(p,j) represents the output value at position p and channel j. CAA attention mechanism is an attention mechanism that combines the advantages of standard convolution and depthwise separable convolution, and uses average pooling to enhance global information and reduce model overfitting. The structure diagram of CAA attention mechanism is shown below. Figure 5 As shown, the input tensor is pooled through an average pooling layer to reduce the number of parameters and computational space. The output tensor after the AvgPool layer is then convolved in a standard 1×1 convolutional layer to obtain local region features. Finally, the output tensor after the standard convolution is processed by two convolutional layers with a depth and width of 1×k. b and height k b A depthwise separable convolution of ×1 is used for depthwise convolution to capture long-range contextual information. The Sigmoid function restricts the model output value to between 0 and 1, ensuring that the predicted bounding box does not exceed the specified grid cell. The computation process of this network is shown below:

[0048]

[0049] In the above formula, P avg This indicates that the average pooling operation is subsequently used as an approximation of the standard large kernel depthwise convolution, which uses two depthwise separable convolutions. and These represent widths of 1×k. b and height k b A depthwise separable convolution of ×1 is used to capture long-range contextual information, as shown in the figure. These convolutional operations generate feature maps containing contextual information, which are then fused with the original features to capture contextual information at different scales. In this way, the CAA module enhances the model's ability to model long-range dependencies, thereby improving the performance of multi-scale object detection.

[0050] LD_downsample downsampling module:

[0051] Because students sit crowded together in real classrooms, and details are easily overlooked, using the original downsampling method might result in the loss of detailed information. Furthermore, in real classroom assessments, students' reading and writing behaviors are quite similar, and this loss of detail could increase the false positive rate. Therefore, this invention uses the MP module of v7 to improve the original downsampling method. The LD_downsample downsampling module is as follows: Figure 6As shown, the first branch first goes through a max pooling process, which is essentially downsampling. Then, it goes through a CBS (Continuous Broadband) to change the number of channels. The second branch first goes through a CBS to change the number of channels, then through a linear deformable convolution, enabling irregular convolution kernels to extract features. This provides convolution kernels of arbitrary sampling shape and size for various targets, overcoming the shortcomings of conventional convolution kernels. Finally, the concat function is used to combine the results of the first and second branches, resulting in the super-downsampling result. By using two convolutions with different dilation rates, the module can extract features at different scales, effectively capturing information of different sizes and contexts in the image. The CBS module consists of a standard convolutional layer, a batch normalization layer (BN), and an activation function, Silu.

[0052] Linear Deformable Convolution (LDConv) is an improved convolutional method that enhances the flexibility of convolutional operations by introducing learnable offsets. Traditional convolution uses fixed grid sampling, while LDConv learns displacements on the input feature map, dynamically adjusting the sampling position of the convolutional kernel. This allows it to more flexibly adapt to changes in the shape of the target object, making it particularly suitable for handling complex situations such as geometric deformation and occlusion. The LDConv structure is as follows: Figure 7 As shown.

[0053] By using learnable offsets, LDConv is no longer limited to fixed kernel sampling positions during convolution operations. Instead, it dynamically adjusts sampling points based on the input feature map, allowing the kernel to flexibly adjust according to object deformation. The offsets are learned through the convolutional layers; specifically, each position in the input feature map is used to calculate a new sampling position before the convolution operation. LDConv can then dynamically adjust sampling points based on these offsets, thus exhibiting better adaptability to positional changes in object detection. The LDConv calculation formula is shown below:

[0054] y(p)=Σ k w(k)·x(p+k+Δp k )

[0055] In the above formula, p is the output position, k is the index of the convolution kernel, w(k) is the convolution weight, and Δp k It is a learnable displacement offset.

[0056] Due to p+k+Δp kThe value may not be an integer, so interpolation sampling is needed to measure the position of the input feature map. Common interpolation methods include bilinear interpolation, which calculates the value of the new sample point by weighting the values ​​of the four surrounding integer pixels.

[0057] Let p' = P + k + Δp k The value of the sampled point is then calculated through interpolation as follows:

[0058]

[0059] Where q is the pixel point near p′, and is the interpolation weight.

[0060] Dyhead_DCNv4 detection head:

[0061] In real-world classroom assessments, occlusion and complex backgrounds are unavoidable issues. Students at different distances have varying perceptual abilities; students in the back are often obscured by those in front, resulting in different receptive fields. Because students' behaviors are similar, it becomes more difficult to distinguish between them, reducing the model's detection accuracy. Therefore, a dynamic attention mechanism object detection head, the Dyhead_DCNv4 detection head, is used to address this problem. This module combines multi-scale feature fusion from DCNv4 (Deformable Convolutional Network v4) deformable convolution with a dynamic attention mechanism. The Dyhead_DCNv4 detection head structure is as follows: Figure 8 As shown. Specifically:

[0062] The core idea of ​​Deformable Convolutional Networks (DCN) is to adjust the sampling positions of the convolution kernel by adding offsets, thus more flexibly adapting to geometric changes in images. Traditional convolution uses a fixed sampling point layout (such as a 3×3 square grid), which has limitations when dealing with complex shapes. DCN enhances the model's ability to capture shape changes by shifting the sampling points. DCNv4 first proposes a linearly learnable offset mechanism. The offset is not only dynamically generated based on the input features but also influenced by multiple elements within the convolution kernel. Secondly, the calculation of the offset not only generates new sampling positions but also provides adaptive weights for each position, allowing the convolution to assign different weights at different positions, thereby better adapting to the local features of the input image. Finally, DCNv4 captures multi-scale details by adding offsets of different scales, enabling the convolution kernel to more accurately locate the edges and details of the target object. Assuming that in a standard convolution operation, the sampling point coordinates corresponding to each convolution position are p... k In DCNv4, each sampling point is dynamically adjusted to the new sampling point position p. k +Δpk And accompanied by weight Δm k Therefore, for an input feature map x, the formula for calculating the output feature map y can be expressed as:

[0063]

[0064] In the above formula, p0 represents the learning point position in the convolution, p k W represents the fixed sampling points of the convolution kernel (e.g., a 3×3 square). k It is the weight of the convolution kernel at position k, Δp k This is the offset of the convolution kernel position k. This offset is given by the offset layer calculated from the feature map, dynamically adjusting the position of the sampling point, Δm. k These are the weights for adaptive learning.

[0065] The dynamic attention mechanism combines scale-aware, spatial-aware, and task-aware attention modules.

[0066] Scale-aware attention: Scale-aware attention assigns different weights to feature layers at different scales, allowing the model to adaptively fuse features based on the importance of each scale. Its calculation formula is as follows:

[0067]

[0068] in, This is essentially an average pooling operation in the HW dimension, where f is a fully connected layer operation and σ is the hard-sigmoid function. The hard-sigmoid function is an activation function that approximates the standard sigmoid function, primarily used to reduce computational complexity. Compared to the standard sigmoid function, the hard-sigmoid approximates it using a piecewise linear function, max(0, min(1, 0.2x + 0.5)), which restricts the result of 0.2x + 0.5 to the interval [0, 1], pruning any values ​​exceeding this range.

[0069] Spatial awareness: Spatial awareness is implemented in two steps. The first is sparse sampling based on deformable convolution, and the second is aggregating features from different levels at the same spatial location. It adds a bias at the location to obtain deformation representation capabilities, and uses an importance factor m to adaptively weight the deformation sampling locations. Its structure is the middle part of dyhead_DCNv4:

[0070]

[0071] In the above formula, K is the number of sparse sampling locations, p k +Δp kIt is the spatial offset Δp learned through self-learning. k It refers to the position shift, Δp k This is the offset of the convolution kernel position k. This offset is given by the offset layer calculated from the feature map, dynamically adjusting the position of the sampling point, Δm. k These are the weights for adaptive learning.

[0072] Task awareness: The task awareness of the detector is improved by modifying the expressive power of different channels for different detection heads, as different channels are associated with different tasks. Channels are adaptively activated, which is essentially a dyrelu operation. Ordinary ReLU only restricts the input to be greater than 0; if it is, the output equals the input. dyrelu learns four values ​​for each channel in the feature layer to control the activation intensity of that channel. In the implementation structure, global average pooling is first performed in the L×S dimension to reduce dimensionality. Then, two fully connected layers and a normalization layer are used. Finally, an offset sigmoid function is applied to normalize the output to the range [-1, 1].

[0073] π c (F)·F=max(α 1 (F)·F c +β 1 (F),α 2 (F)·F c +β 2 (F))

[0074] In the above formula, F c It is a feature slice of the c-th channel, [α] 1 ,α 2 ,β 1 ,β 2 ] T =θ(·) is a superfunction used to learn and control the activation threshold.

[0075] As can be seen, the dynamic detection head includes scale-aware, spatial-aware, and task-aware attention modules, which adjust the input to the same scale to form a three-dimensional tensor F∈R. L×H×W×C The 3D tensor is used as input to the dynamic head. After passing through the scale-aware module, the feature map becomes more sensitive to scale differences of foreground objects. After passing through the spatial awareness module, the feature map becomes sparser and concentrated on the spatial location of foreground objects. Finally, after passing through the task-aware module, the feature map is reformulated into different activation features according to the requirements of different downstream tasks. Reformulating different activation features can alleviate information loss caused by occlusion and allow small-scale targets to be noticed again, which plays a role in solving the occlusion problem in student behavior detection in real classrooms. Therefore, the Dyhead_DCNv4 detection head is represented as follows:

[0076] W(F) = π c (π s (π L (F)·F)·F)·F

[0077] Where W(F) represents the output features of the Dyhead_DCNv4 detector, F represents the input features of the Dyhead_DCNv4 detector, and πL(·), πS(·), and πC(·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

[0078] After training the improved YOLOv11 model, pruning was performed on the model.

[0079] With the development of deep learning technology, Convolutional Neural Networks (CNNs) have achieved remarkable success in computer vision, especially in object detection tasks. The YOLO (You Only Look Once) series, an end-to-end object detection model based on CNNs, is widely used in real-time detection scenarios due to its balance between accuracy and speed. However, with each model update, the structure of the YOLO model has become increasingly complex, with a significant increase in the number of parameters and computational cost. Especially in resource-constrained scenarios (such as mobile and embedded devices), the complexity of the model can lead to excessively long inference times, making it difficult to meet real-time requirements. Therefore, model compression has become an important step in the practical application of deep learning models. Common model compression methods include parameter quantization, knowledge distillation, and pruning. Among these, pruning is a technique that reduces model size by removing unimportant parameters. Weight-based pruning methods are widely used in deep learning model compression tasks due to their simplicity and effectiveness. However, traditional pruning methods often fail to fully consider the characteristics of different layers in the network, leading to over- or under-pruning of some layers, affecting the overall performance of the model. LAMP is a layer-adaptive magnitude pruning algorithm that adaptively prunes the weight magnitude distribution at each layer, effectively preserving important features while reducing redundant weights. This invention proposes a compression method for the YOLOv11 model that applies the LAMP pruning algorithm, maintaining high object detection accuracy and meeting real-time requirements while significantly reducing model parameters. This addresses the problem of rapid deployment of existing YOLOv11 models in resource-constrained scenarios. By applying the LAMP pruning algorithm, the model can adaptively select the pruning ratio at different levels based on the importance of the weights, reducing redundant parameters while preserving key features, ensuring that the pruned model still maintains high accuracy in object detection tasks.

[0080] First, a pre-trained YOLOv11 model is used as the initial model. This model has been trained on a large-scale dataset and exhibits good detection performance. Subsequently, based on the LAMP pruning algorithm, the weight magnitude distribution of each convolutional layer in the YOLOv11 model is calculated. LAMP adaptively calculates the L1 norm of the weights in each layer and determines the pruning threshold based on the weight distribution of each layer. Different layers have different weight distributions, and LAMP can adaptively adjust the pruning ratio of each layer according to these differences. The pruning formula is as follows:

[0081] W′=W·1 {|w|＞λ}

[0082] Where W is the weight matrix of the convolutional layer, λ is the pruning threshold, and 1 is an indicator function that indicates that the weights are retained when they are greater than a certain threshold.

[0083] Based on the pruning threshold, weights in the YOLOv11 model with magnitudes below the threshold are reset to zero, reducing unnecessary parameters. The LAMP pruning strategy ensures that weights at different levels are adaptively pruned according to importance, preserving important feature information and avoiding model accuracy degradation due to over-pruning. Pruned models typically experience some performance degradation, therefore fine-tuning is necessary. Several rounds of fine-tuning are performed using the original training dataset to restore the model's performance in object detection tasks. This step ensures that the model maintains high detection accuracy despite the reduced number of parameters.

[0084] The LAMP pruning algorithm adaptively adjusts the pruning ratio according to the importance of the weights in each layer, avoiding the over- or under-pruning issues of traditional pruning methods. This pruning method is suitable for various application scenarios of the YOLOv11 model, especially for object detection tasks that need to be deployed on resource-constrained devices (such as mobile or embedded devices).

[0085] S2: Obtain the classroom video of the student to be tested and input it into the student classroom detection model for processing to obtain the student classroom behavior detection results.

[0086] The pruned and improved YOLOv11 model is used as the student classroom detection model. The student classroom video to be detected is input into the student classroom detection model frame by frame, and the detection results of student classroom behavior in each frame can be obtained.

[0087] S3: Use the improved DeepSORT algorithm to process the results of student classroom behavior detection to achieve tracking of student classroom behavior.

[0088] This invention improves the original DeepSORT algorithm in two ways. First, it combines LSTM with a Kalman filter, replacing the state prediction result obtained by the Kalman filter in the original DeepSORT algorithm with a weighted sum of the state prediction results obtained by the LSTM network and the Kalman filter. Specifically:

[0089] The target tracking component relies on Kalman filtering to predict the target's state in the next frame, including its position and velocity. When no target is detected (e.g., due to occlusion or brief loss), the Kalman filter estimates the target's approximate position in the next frame using trajectory information from previous frames. This process mainly involves two parts: predicting the target position and updating the target detection box position. While Kalman filtering excels at handling linear predictions of system states and effectively copes with noise and short-term motion changes, it has poor predictive ability for complex nonlinear motion patterns (such as sudden acceleration, deceleration, and curvilinear motion). LSTM, on the other hand, excels at processing time-series data, capturing target motion patterns over long periods, and exhibits better predictive performance for complex nonlinear trajectories.

[0090] During the prediction phase, the Kalman filter predicts the target's state in the next frame based on the target's state (position, velocity, etc.) from the previous frame. This process is based on the assumption of a motion model where the target's motion between frames is linear, and the target's position and motion information are described by a state vector, represented as follows: Where x and y are the coordinates of the target center point, a is the aspect ratio of the bounding box, and h is the height of the bounding box. It is the velocity component. The prediction equation for the Kalman filter is as follows:

[0091]

[0092] in It is the predicted state at time t. F is the estimated state from the previous frame, and F is the state transition matrix. As the uncertainty in the state estimation increases, the covariance matrix is ​​also updated.

[0093] P t|t-1 =F·P t-1|t-1 ·F T +Q

[0094] Where P t|t-1 Q is the covariance matrix during prediction, and Q is the process noise covariance matrix, representing the noise caused by inaccurate modeling.

[0095] The structure of the LSTM (Long Short-Term Memory) model is as follows: Figure 9As shown in the figure, a time series input is required after the input data, using the target's state in the previous few frames (e.g., (x, y, a, h)) as input. A certain time window (e.g., the past 5 or 10 frames) is selected to provide sufficient historical information for the LSTM. The output of the LSTM can then be the predicted state for the next frame or several frames. Specifically, the output dimensions are consistent with the Kalman filter, i.e., the predicted target position and size (x, y, a, h).

[0096] The second step is the update phase. When new detection results appear, the results of the Kalman filter and LSTM are combined to update the state using data by assigning different weights to the prediction results of the Kalman filter and LSTM.

[0097] Kalman filtering maps the state to the observation space using an observation model. Then, the Kalman gain K determines how to perform a weighted average between the predicted and observed values ​​to generate an updated state estimate. The formula for calculating the Kalman gain is:

[0098] K t =P t|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1

[0099] In the above equation, H represents the observation, and R is the observation noise covariance matrix. The Kalman gain is then used to combine the predicted state with the observed values ​​to obtain the updated state:

[0100]

[0101] Where z t The bounding box of the target is the actual observation data. The Kalman filter is used to predict the target's position and velocity between consecutive frames. By modeling physical quantities such as position and velocity as state vectors and combining them with the observed bounding box information, the Kalman filter can continuously adjust the target's state estimate. It effectively smooths the target's trajectory, reduces the impact of noise on tracking, and maintains the continuity of the tracking task even in the event of brief target loss.

[0102] The output of an LSTM is determined by its output gate. First, the value of the output gate is calculated, and then the final hidden state h is generated based on the updated cell state. tA sigmoid layer is run to determine which part of the cell state will be output. The cell state is then processed using tanh (resulting in a value between -1 and 1) and multiplied by the output of the sigmoid gate, ultimately outputting only the part determined by the user. Finally, different weights are assigned to the prediction results of the Kalman filter and LSTM, allowing the decision to rely more on the LSTM or Kalman filter results based on the complexity of the motion or historical data.

[0103] The second improvement of this invention to the original DeepSORT algorithm is to replace the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network. Specifically:

[0104] In DeepSORT, the feature extractor is used to extract visual features of targets from images for similarity matching during tracking. However, while existing feature extractors (such as ResNet or earlier versions of CNNs) offer high feature representation capabilities, they are computationally expensive, especially in resource-constrained real-time systems, where efficiency becomes a problem. To further reduce computational complexity while maintaining high feature extraction capabilities, this invention proposes using MobileNetV4 as the feature extractor for DeepSORT. MobileNetV4 is a lightweight convolutional neural network designed for mobile and embedded devices, enabling efficient inference with low computational resources. MobileNetV4 employs depthwise separable convolutions, a Squeeze-and-Excitation module, and improved network architecture search techniques, achieving a better balance between accuracy and speed. Its advantage lies in its lightweight feature extraction capabilities, providing fast and accurate feature representations, making it particularly suitable for real-time applications.

[0105] In DeepSORT, the original feature extractor (such as ResNet) is replaced with MobileNetV4. MobileNetV4, through its efficient network architecture, significantly reduces the computational cost per frame while still extracting sufficiently discriminative features. This invention uses a MobileNetV4 model pre-trained on ImageNet and fine-tunes it using a specific multi-object tracking dataset. In this way, the model can adapt to changes in the appearance features of targets in multi-object tracking scenarios. To ensure good performance of the extracted features in similarity calculations, L2 regularization is applied to the output features, maintaining a unit length for the feature vector, thereby enhancing the stability of tracking and matching.

[0106] The MobileNetV4 model is used to extract features of the detection boxes. Then, the Hungarian algorithm is used to match the features of the detection results with the features of the existing tracked targets. Specifically:

[0107] After the Kalman filter predicts the position of the target in the current frame, the Hungarian algorithm is used to associate the target detection boxes with the tracking trajectory across consecutive frames. Specifically, the Hungarian algorithm solves the assignment problem in tracking, i.e., how to match the detection results (such as detected people or objects) in the current frame with the existing tracked targets in the previous frame. This algorithm is a type of algorithm used to solve the optimal matching problem in bipartite graphs. In the target tracking task, it is used to minimize the matching cost between the detection boxes and the tracked targets, thereby achieving optimal assignment. The student behavior detection model generates multiple target detection boxes in each frame, while the tracker maintains some existing trajectories. Each frame needs to assign the detected targets to existing tracking trajectories in order to continuously track these targets. The Hungarian algorithm is used to solve the problem of how to match these detected targets with existing trajectories.

[0108] The Hungarian algorithm's tracking process begins by calculating a matching cost matrix. This matrix reflects the "distance" between a detected target in the current frame and a tracked target in the previous frame, which can be represented using various metrics. The matching cost matrix is ​​primarily based on positional and appearance features: Intersection over Union (IOU), Mahalanobis distance, and the cosine similarity of ReID feature vectors. The IoU is calculated based on the overlapping region of the target detection boxes, representing the spatial distance between the detection result and the tracking trajectory. Then, the matching cost is measured by the distance between the predicted state of the Kalman filter and the detection boxes. Finally, based on the ReID feature vectors (target re-identification), the cosine similarity between the feature vectors of the detection result and the existing tracked target is calculated. This similarity allows the system to distinguish between targets with similar or overlapping appearances. The cost between each detected target and each tracked trajectory predicted by the Kalman filter is then encoded into a two-dimensional matrix. Each element of the matrix represents the matching cost between the detection box and the trajectory; smaller values ​​indicate easier matching.

[0109] After obtaining the cost matrix, the Hungarian algorithm is used to solve the problem of minimizing the matching cost. The Hungarian algorithm finds an optimal set of matches in the cost matrix that minimizes the total cost. It finds the minimum cost combination from the cost matrix, ensuring that each bounding box matches each tracking trajectory at most once. If a bounding box finds a matching tracking trajectory, and the cost of this match is within a set threshold (such as an IoU or ReID feature similarity threshold), then the match is confirmed. This establishes the association between the bounding box and the tracking target. For bounding boxes that do not match, it may be a new target, so a new tracking trajectory is created; while for tracking trajectories that do not match, it is assumed that the target may be lost or temporarily occluded.

[0110] After a successful match, the improved DeepSORT algorithm updates the matched trajectory based on the results of the Hungarian algorithm, updating the status of the matched trajectory, including position, velocity, and other information. It then manages unmatched trajectories; if a tracked target fails to find a matching bounding box across multiple frames, the DeepSORT algorithm may mark it as lost and remove it. For subsequent trajectory creation, if a bounding box does not match any existing trajectory, the DeepSORT algorithm considers it a new target and creates a new tracking trajectory. This process enables the tracking of students' classroom behavior.

[0111] In summary, this invention first replaces the C3K2 module in the original backbone network with the Star_caa module, enhancing the extraction of global image information and improving the model's detection performance. Due to the complexity of real-world classroom scenarios, the original detection head alone cannot guarantee accuracy and recognition rate. Therefore, this invention employs the DYHead_DCNV4 detection head module, which strengthens the model's understanding of global and local information through multi-scale feature fusion attention mechanisms and variable convolution, resulting in more stable and accurate performance when dealing with occlusion and complex backgrounds. Furthermore, to address the issue of crowded classrooms where details are easily overlooked, the downsampling strategy is optimized. The LD_downsample downsampling module reduces the loss of detail information, improves feature extraction, and significantly reduces computational load. While meeting the requirements of real-time performance and mobile deployment, LAMP pruning technology is used to achieve model lightweighting, reducing redundant weights and computational resource consumption, and improving mobile performance. Regarding target tracking, traditional algorithms are insufficient in efficiency and accuracy, especially prone to ID switching when handling occlusion. Therefore, this invention combines LSTM networks to process time-series data, capturing long-term motion patterns of targets and improving the prediction effect of nonlinear trajectories. In addition, to further reduce computational complexity while maintaining strong feature extraction capabilities, MobileNetV4 is used as the feature extractor for DeepSORT to achieve efficient tracking of student behavior.

[0112] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting student classroom behavior based on deep learning, characterized in that, include: S1: Train the improved YOLOv11 model, and prune the trained improved YOLOv11 model to obtain the student classroom detection model; the improved YOLOv11 model includes an improved Backbone network, an improved Neck network, and an improved Head network; the improved Backbone network replaces the C3K2 module in the original YOLOv11 model's Backbone network with the Star_caa module; both the improved Backbone network and the improved Neck network replace the Conv downsampling module in the original YOLOv11 model with the LD_downsample downsampling module; the improved Head network replaces the detection head of the original YOLOv11 model's Head network with the Dyhead_DCNv4 detection head; The Star_caa module processes the input features as follows: First, the input features are processed using a first DWConv layer to obtain a first intermediate feature; then, the first intermediate feature is processed using two parallel 1×1 convolutional layers, and a star operation is performed on the two convolutional outputs to obtain a second intermediate feature; the second intermediate feature is processed using the CAA attention mechanism and then input into a 1×1 convolutional layer to obtain a third intermediate feature; the third intermediate feature is input into a second DWConv layer for processing to obtain a fourth intermediate feature; finally, the fourth intermediate feature is residually concatenated with the input features and regularized to obtain the output features of the Star_caa module. S2: Obtain classroom videos of the students to be detected and input them into the student classroom detection model for processing to obtain the student classroom behavior detection results; S3: Use the improved DeepSORT algorithm to process the student classroom behavior detection results to achieve tracking of student classroom behavior; the improved DeepSORT algorithm replaces the state prediction results obtained by the Kalman filter in the original DeepSORT algorithm with the weighted sum of the state prediction results obtained by the LSTM network and the state prediction results obtained by the Kalman filter, and replaces the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network.

2. The student classroom behavior detection method based on deep learning according to claim 1, characterized in that, The LD_downsample downsampling module consists of two branches: one branch connects the max pooling layer to the CBS module, and the other branch connects the CBS module to the LDConv layer. The outputs of the two branches are concatenated to form the output features of the LD_downsample downsampling module.

3. The student classroom behavior detection method based on deep learning according to claim 2, characterized in that, The CBS module consists of a standard convolutional layer, a batch normalization layer, and an activation function, Silu.

4. The student classroom behavior detection method based on deep learning according to claim 1, characterized in that, The Dyhead_DCNv4 detection header is represented as: W(F)=π c (p s (p L (F)·F)·F)·F Where W(F) represents the output features of the Dyhead_DCNv4 detector, F represents the input features of the Dyhead_DCNv4 detector, and π L (·), π s (·), and π c (·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

5. The student classroom behavior detection method based on deep learning according to claim 1, characterized in that, The LAMP pruning algorithm was used to prune the trained improved YOLOv11 model.

Images