Student classroom behavior detection method based on deep learning

Abstract

The invention belongs to the technical field of target detection, and particularly relates to a student classroom behavior detection method based on deep learning. The method comprises the following steps: training an improved YOLOv11 model, and pruning the trained improved YOLOv11 model to obtain a student classroom detection model; acquiring a to-be-detected student classroom video and inputting the to-be-detected student classroom video into the student classroom detection model for processing to obtain a student classroom behavior detection result; the improved DeepSORT algorithm is used for processing the student classroom behavior detection result, tracking of the student classroom behaviors is achieved, the global information capturing capacity of the network is enhanced, the performance of the network under the shielding and multi-scale change scene is improved, and therefore the robustness and detection precision of the network are improved, and the method has good application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and specifically relates to a method for detecting student classroom behavior based on deep learning. Background Art

[0002] With the extensive use of cloud classrooms and online classrooms in recent years, and the smart classroom as a national key development project, the demand for recording and identifying students' classroom performance is increasing. There are many studies in the field of education to predict and judge students' future performance and achievements in the classroom through their actions and behavioral performance. Traditionally, teachers maintain discipline in the classroom and manually record students' performance in the classroom. There are two problems with this method: first, the lecturer is distracted, which reduces the efficiency of the lecture; second, it is impossible to fully and accurately record the learning behavior of all students. Nowadays, using computer vision technology to identify the status of students in the classroom has become an important technical basis for building smart classrooms. Deep learning technology can provide data support for the construction of smart classrooms, make statistical judgments on student behavior in the background, and better evaluate students' learning.

[0003] There are three types of algorithms currently used for student behavior detection: video action recognition-based, gesture estimation-based, and object detection-based. The recognizable continuous behavior of student classroom behavior detection is based on video action recognition, however, this requires a large number of samples to be labeled. The algorithm based on gesture estimation describes human behavior by obtaining the position and motion information of each joint of the human body, but it is not suitable for behavior detection in crowded classrooms. The object detection algorithm is a promising solution. The student behavior object detection algorithm can identify students' specific behaviors in learning, such as concentration, asking questions, answering questions, etc. By understanding the behavior patterns of each student, educators can provide more personalized and targeted educational support to meet the learning needs of different students. By analyzing students' behavior, educators can more accurately assess students' understanding of teaching content and learning effects. This helps to adjust teaching strategies to improve teaching effectiveness and students' academic performance.

[0004] With the widespread application of deep learning, target detection algorithms have developed rapidly. Target detection methods can be divided into two categories: traditional target detection and target detection based on deep learning. Traditional target detection algorithms are generally divided into three steps: first, generating candidate boxes, then extracting features in the candidate boxes, and finally classifying according to the features. When generating candidate boxes, an exhaustive search method is usually used, but this method has a large amount of calculation and has the problem of repeated calculation, and is inefficient. In addition, traditional algorithms rely on artificially designed features, and the stability of features is difficult to guarantee. Therefore, traditional methods have limitations in accuracy and speed. With the development of deep learning, target detection algorithms based on deep learning have achieved significant improvements in prediction speed and accuracy. This type of method has gradually become mainstream and is mainly divided into two categories: one-stage target detection algorithm and two-stage target detection algorithm. Compared with the two-stage algorithm, the accuracy of the one-stage algorithm may be slightly lower, but its detection process only needs to be completed once, the amount of calculation is small, and it is suitable for real-time application scenarios. In the one-stage algorithm, the YOLO series achieves a good balance between accuracy and speed, so the present invention selects the YOLO series as the basic model for student target detection. Based on this, combined with the characteristics of actual teaching scenarios and research objectives, this paper proposes an improved YOLOv11 algorithm for classroom behavior detection. The algorithm aims to enhance the network's ability to capture global information and improve its performance in occlusion and multi-scale change scenarios, thereby improving the network's robustness and detection accuracy. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a method for detecting student classroom behavior based on deep learning, which includes:

[0006] S1: Train the improved YOLOv11 model, prune the trained improved YOLOv11 model, and obtain the student classroom detection model;

[0007] S2: Obtain the classroom video of the student to be detected and input it into the student classroom detection model for processing to obtain the student classroom behavior detection result;

[0008] S3: Use the improved DeepSORT algorithm to process the student classroom behavior detection results to track the student classroom behavior.

[0009] Preferably, the improved YOLOv11 model includes an improved Backbone network, an improved Neck network and an improved Head network; the improved Backbone network replaces the C3K2 module in the Backbone network of the original YOLOv11 model with a Star_caa module; the improved Backbone network and the improved Neck network both replace the Conv downsampling module in the original YOLOv11 model with a LD_downsample downsampling module; the improved Head network replaces the detection head of the Head network in the original YOLOv11 model with a Dyhead_DCNv4 detection head.

[0010] Furthermore, the Star_caa module processes the input features by: processing the input features by the first DWConv layer to obtain the first intermediate features; processing the first intermediate features by two parallel 1×1 convolutional layers, and performing a star operation on the two convolutional outputs to obtain the second intermediate features; processing the second intermediate features by the CAA attention mechanism and inputting them into the 1×1 convolutional layer to obtain the third intermediate features; inputting the third intermediate features into the second DWConv layer for processing to obtain the fourth intermediate features; performing a residual connection between the fourth intermediate features and the input features and performing regularization to obtain the output features of the Star_caa module.

[0011] Furthermore, the LD_downsample downsampling module includes two branches, one branch is the maximum pooling layer connected to the CBS module, and the other branch is the CBS module connected to the LDConv layer; the outputs of the two branches are concatenated as the output features of the LD_downsample downsampling module.

[0012] Furthermore, the CBS module consists of a standard convolutional layer, a batch normalization layer, and an activation function Silu.

[0013] Furthermore, the Dyhead_DCNv4 detection head is represented as:

[0014] W(F)=π c (π s (π L (F)·F)·F)·F

[0015] Among them, W(F) represents the output features of the Dyhead_DCNv4 detection head, F represents the input features of the Dyhead_DCNv4 detection head, and πL(·), πS(·), and πC(·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

[0016] Preferably, the LAMP pruning algorithm is used to prune the trained improved YOLOv11 model.

[0017] Preferably, the improved DeepSORT algorithm is to replace the state prediction result obtained by the Kalman filter in the original DeepSORT algorithm with the weighted sum of the state prediction result obtained by the LSTM network and the state prediction result obtained by the Kalman filter, and replace the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network.

[0018] The beneficial effects of the present invention are as follows: the student classroom behavior detection method proposed in the present invention replaces the C3K2 module in the original backbone network by designing the Star_caa module, thereby enhancing the extraction of global image information and improving the detection performance of the model. The DYHead_DCNV4 detection head module is adopted, and the model's understanding of global and local information is enhanced through multi-scale feature fusion attention mechanism and variable convolution, and the performance is more stable and accurate when dealing with occlusion and complex backgrounds. In addition, the present invention optimizes the downsampling strategy, uses the LD_downsample downsampling module to reduce the loss of detail information, improves the feature extraction effect, and significantly reduces the amount of calculation. Under the requirements of real-time and mobile terminal deployment, the present invention combines the LAMP pruning technology to realize model lightweight, reduce redundant weights and computing resource consumption, and improve mobile terminal performance; the present invention combines the LSTM network to process time series data, capture the long-term motion pattern of the target, and improve the prediction effect of nonlinear trajectory. In order to further reduce the computational complexity and maintain a strong feature extraction capability, MobileNetV4 is used as the feature extractor of DeepSORT to achieve efficient tracking of student behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of the classroom behavior detection process of middle school students in the present invention;

[0020] Figure 2 This is a schematic diagram of the improved YOLOv11 model structure in the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of the Sta_caa module in the present invention;

[0022] Figure 4 Schematic diagram of the depth-separable convolution structure in the present invention

[0023] Figure 5 Schematic diagram of the CAA attention mechanism structure in the present invention;

[0024] Figure 6 Schematic diagram of the structure of the LD_downsample downsampling module in the present invention;

[0025] Figure 7 Schematic diagram of the LDConv structure in the present invention;

[0026] Figure 8 It is a schematic diagram of the structure of the Dyhead_DCNv4 detection head in the present invention;

[0027] Fig. 9 This is a schematic diagram of the LSTM network structure in the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention proposes a method for detecting student classroom behavior based on deep learning. Figure 1 As shown, the method includes the following contents:

[0030] S1: Train the improved YOLOv11 model, prune the trained improved YOLOv11 model, and obtain the student classroom detection model.

[0031] The present invention designs and improves the YOLOv11 model based on the original YOLOv11 model to enhance the model's ability to extract global image information and solve the multi-scale problem of students in the classroom; specifically: the improved YOLOv11 model integrates multiple new modules and optimization strategies, including a backbone feature extraction module Star_caa, a super downsampling MP module (LD_downsample), and a multi-scale dynamic attention detection head Dyhead_DCNv4. The improved Backbone network replaces the C3K2 module in the Backbone network of the original YOLOv11 model with the Star_caa module; the improved Backbone network and the improved Neck network both replace the Conv downsampling module in the original YOLOv11 model with the LD_downsample downsampling module; the improved Head network replaces the detection head of the Head network in the original YOLOv11 model with the Dyhead_DCNv4 detection head.

[0032] like Figure 2As shown in the figure, the Backbone network uses multi-layer convolution and Star_caa modules to extract initial features, and uses spatial pyramid pooling fast (SPPF) to aggregate multi-scale context information. The Neck network combines MP downsampling with C3K2 and global context fusion to improve the model's feature representation capabilities. Upsampling and feature cascading further ensure the detail retention and detection accuracy of high-resolution feature maps. The Dyhead_DCNv4 detection head is responsible for processing feature maps and generating target detection results, including prediction boxes, category confidence, and location information. The loss function and batch normalization work together through different mechanisms to optimize model performance. The loss function provides a clear optimization goal, and batch normalization (BN) stabilizes the feature distribution, accelerates the training process, and improves the stability and efficiency of model training. Through these optimizations, the improved YOLOv11 model aims to reduce computational complexity while maintaining high detection accuracy and speed.

[0033] Star_caa module:

[0034] In actual application scenarios, student behavior detection involves real-time processing of multiple targets, so detection accuracy is particularly important in actual tasks. The convolution and information fusion operations of each Bottleneck layer of the C3K2 module ignore the multi-scale problems that occur in real student scenarios. And on devices with limited video memory and when processing tasks with relatively simple features, multi-level Bottleneck will reduce reasoning efficiency and affect the real-time detection performance of the model. In order to enhance the model's ability to detect details and features of student behavior and improve the student detection rate, the present invention constructs a Star_caa module to solve the problem of low detection rate caused by different front and back receptive fields of students.

[0035] The Star_caa module optimizes the overall performance of the model by integrating the two key technologies of the starblok module and the CAA attention mechanism in starNet. Figure 3As shown in the figure, the processing process of the input features by the Star_caa module includes: using the first DWConv (depth-wise separable convolution) layer to process the input features, the input tensor is convolved with the depth-wise separable convolution layer kernel of 7, and each channel is convolved separately to obtain the first intermediate feature. Two parallel 1×1 convolutional layers FC1 and FC2 are used to extract features, adjust channel dimensions, and fuse channel information. The two convolution outputs are star-operated, that is, FC1 is activated through the relu6 activation function, and then star-operated with FC2 to obtain the second intermediate feature. This operation can create a large number of interaction terms, which significantly increases the dimension of the feature space. Compared with simple linear transformations, star operations can capture more complex feature relationships, thereby improving the flexibility and accuracy of feature extraction; the second intermediate feature is reorganized through the CAA attention mechanism to obtain contextual information; the tensor output by the CAA attention mechanism is compressed using a 1×1 convolutional layer G to restore the number of channels before the parallel 1×1 convolutional layer to obtain the third intermediate feature; the third intermediate feature is input into the second DWConv layer to further extract feature values ​​to obtain the fourth intermediate feature; the fourth intermediate feature is residually connected to the input feature and regularized to alleviate the problem of gradient disappearance in the deep network, thereby improving the ability to extract information from distant objects and obtaining the output feature of the Star_caa module.

[0036] The calculation process of the Star_caa network is shown in the following formula:

[0037] W fc1 =FC1(DWConv 1×1 (x))+b1

[0038] W fc =FC2(DWConv 1×1 (x))+b2

[0039] W g =CAA(Relu6(W fc1 )*W fc)

[0040] W star =x+DropPath(DWConv 1×1 (W g (x)+b g ))

[0041] In the above formula, x represents the input of the Star_caa module, W fc1 , W fc2 It means that the channel is expanded after two 1×1 convolutions. FC1 and FC2 represent the weights of the two convolutional layers, b1 and b2 are bias terms, and W gAfter the channel is expanded, it is activated as a nonlinear value using an activation function and then combined with W fc2 The star operation is performed and the context anchor attention mechanism is introduced. Finally, the input x is added to the feature map after the depth-separable convolution through the residual connection. In the Star_caa network structure, the CAA attention mechanism and star operation are used to enhance the detection ability of objects of different scales. The local and global information are combined from different levels of the feature map to improve the detection accuracy of small and distant objects.

[0042] Depthwise separable convolution Figure 4 As shown in Figure 1, by dividing the standard convolution into two steps, the first operation uses a depthwise convolution to apply a k×k convolution kernel to each input channel separately, rather than operating across channels like a standard convolution. That is, if the input has C in channels, use C in There are k×k convolution kernels, processing each channel separately. The output is still H×W×C in feature maps, because each channel only operates with its own corresponding convolution kernel; the calculation formula of this process is expressed as:

[0043]

[0044] In the above formula, W depthwise,k,l,i is the K×K convolution kernel of the ith channel, y depthwise (p,i) is the value output at position p and channel i.

[0045] The second step is point-by-point convolution, which uses a 1×1 convolution kernel to linearly combine different channel information to generate C out Point-wise convolution is equivalent to performing C on each spatial position. in ×C out dimensional linear transformation, so it compresses the result of the depthwise convolution to H×W×C in ×C out Using depthwise separable convolutions splits the computation of standard convolution into two smaller operations, reducing the computation by about Especially when the convolution kernel is large, the effect is significant. Therefore, the number of network parameters can be reduced, which reduces the amount of model calculation. The calculation formula of point-by-point convolution is shown as follows:

[0046]

[0047] In the above formula, W pointwise,i,j is a 1×1 convolution kernel weight that maps the i-th channel to the j-th output channel, y pointwise(p,j) is the value output at position p and channel j. The CAA attention mechanism is an attention mechanism that combines the advantages of standard convolution and depthwise separable convolution, and uses average pooling to enhance global information and reduce overfitting of the model. The CAA attention mechanism structure is shown in the figure below: Figure 5 As shown in Figure 1, the input tensor is pooled through the average pooling layer to reduce the number of parameters and computational space. The output tensor after the AvgPool layer is convolved in a standard 1×1 convolution layer to obtain local area features. The output tensor after the standard convolution is then convolved using two layers with a depth of 1×k. b and height k b ×1 depthwise separable convolution is used to capture long-distance contextual information. Through the Sigmoid function, the value of the model output can be limited between 0 and 1, thereby ensuring that the predicted bounding box does not exceed the grid unit. The calculation process of the network is as follows:

[0048]

[0049] In the above formula, P avg It represents the average pooling operation followed by the use of two depth-wise separable convolutions as an approximation of the standard large kernel depth-wise convolution. and Respectively represent the width of 1×k b and height k b ×1 depthwise separable convolutions are used to capture long-range contextual information, as shown in the figure. These convolution operations can generate feature maps containing contextual information, and then we fuse these feature maps with the original features to capture contextual information at different scales. In this way, the CAA module can enhance the model's ability to model long-range dependencies, thereby improving the performance of multi-scale object detection.

[0050] LD_downsample downsampling module:

[0051] In a real classroom, students sit together in a crowded place, and details are easily overlooked. If the original downsampling is used, the detail information may be lost. In real classroom detection, students' reading and writing behaviors are similar. The loss of these detail information may increase the false detection rate. Therefore, the present invention uses the MP module of v7 to improve the original downsampling. The LD_downsample downsampling module is as follows: Figure 6As shown in the figure, the first branch first passes through a max pool, which is also called maximum pooling. The function of the maximum value is downsampling, and then it passes through a CBS to change the number of channels. The second branch first passes through a CBS to change the number of channels, and then passes through a linear deformable convolution to realize the function of extracting features with irregular convolution kernels, providing convolution kernels of arbitrary sampling shapes and sizes for various targets, making up for the shortcomings of conventional convolution kernels. Finally, the concat function is used to add the results of the first branch and the second branch together to obtain the result of super downsampling. By using two convolutions with different expansion rates, the module can extract features of different scales, which is also effective for capturing information of different sizes and different contexts in the image. Among them, the CBS module consists of a standard convolution layer, a batch normalization layer (BN layer / batch normalization layer) and an activation function Silu.

[0052] Linear Deformable Convolution (LDConv) is an improved convolution method that enhances the flexibility of convolution operations by introducing a learnable offset. Traditional convolution uses fixed grid sampling, while LDConv dynamically adjusts the sampling position of the convolution kernel by learning displacement on the input feature map, making it more flexible to adapt to the shape changes of the target object, especially suitable for dealing with complex situations such as geometric deformation and occlusion. The LDConv structure is as follows: Figure 7 shown.

[0053] Through the learnable displacement, LDConv is no longer limited to fixed convolution kernel sampling positions in convolution operations, but dynamically adjusts the sampling points according to the input feature map, allowing the convolution kernel to be flexibly adjusted according to the deformation of the object. The offset is learned through the convolution layer, which is specifically manifested as calculating the new sampling position for each position of the input feature map according to the offset, and then performing the convolution operation. Subsequently, LDConv can dynamically adjust the sampling points according to the offset, so it has better adaptability to the position changes in target detection. The LDConv calculation formula is as follows:

[0054] y(p)=Σ k w(k)·x(p+k+Δp k )

[0055] In the above formula, p is the output position, k is the index of the convolution kernel, w(k) is the convolution weight, Δp k is a learnable displacement offset.

[0056] Since p+k+Δp kIt may not be an integer, so it is necessary to interpolate the position of the input feature map. Common interpolation methods include bilinear interpolation, which is to calculate the value of the new sampling point based on the weighted average of the four surrounding integer pixels.

[0057] Let p'=P+k+Δp k , then the value of the sampling point is calculated by interpolation as:

[0058]

[0059] Where q is the pixel near p′ and is the interpolation weight.

[0060] Dyhead_DCNv4 detection head:

[0061] In real student classroom detection, occlusion and complex background are inevitable. Students at different distances have different perception abilities. Students in the back will be blocked by students in the front row, which will result in different receptive fields. Due to similar behaviors, it is more difficult to distinguish student behaviors, which will reduce the accuracy of model detection. Therefore, a dynamic attention mechanism target detection head, namely the Dyhead_DCNv4 detection head, is used to solve this problem. This module combines the multi-scale feature fusion of DCNv4 (Deformable Convolutional Network v4) deformable convolution and the dynamic attention mechanism. The structure of the Dyhead_DCNv4 detection head is as follows: Figure 8 As shown. Specifically:

[0062] The core idea of ​​DCN (Deformable Convolutional Networks) deformable convolution is to adjust the sampling position of the convolution kernel by adding offsets to adapt to geometric changes in the image more flexibly. Traditional convolution uses a fixed sampling point layout (such as a 3×3 square grid), but it has limitations when dealing with complex shapes. DCN enhances the model's ability to capture shape changes by offsetting the sampling points. DCNv4 first proposed a linear learnable offset mechanism. The offset is not only dynamically generated according to the input features, but also affected by multiple elements in the convolution kernel; secondly, the calculation of its offset not only generates new sampling positions, but also provides adaptive weights for each position, so that the convolution can assign different weights at different positions, thereby better adapting to the local features of the input image. Finally, DCNv4 captures multi-scale detail information by adding offsets of different scales, so that the convolution kernel can more accurately locate the edges and details of the target object. Assume that in a standard convolution operation, the sampling point coordinates corresponding to each convolution position are p k In DCNv4, each sampling point will be dynamically adjusted to the new sampling point position p k +Δpk With the weight Δm k Then, for the input feature map x, the calculation formula for the output feature map y can be expressed as:

[0063]

[0064] In the above formula, p0 represents the point position in the convolution, p k represents the fixed sampling point of the convolution kernel (e.g., a 3×3 square position), W k is the weight of the convolution kernel at position k, Δp k is the offset of the convolution kernel position k. This offset is given by the offset layer calculated by the feature map, dynamically adjusting the position of the sampling point, Δm k is the weight of adaptive learning.

[0065] The dynamic attention mechanism is superimposed through scale-aware, space-aware, and task-aware attention modules.

[0066] Scale-aware: Scale-aware attention is actually giving different weights to feature layers of different scales, so that the model can adaptively fuse features based on the importance of the scale. Its calculation formula is:

[0067]

[0068] in, It is an avg pool operation in the HW dimension, f is a fully connected layer operation, and σ is the hard-sigmoid function. The hard-sigmoid function is an activation function that approximates the standard sigmoid function, mainly used to reduce computational complexity. Compared with the standard sigmoid function, the hard-sigmoid function uses a piecewise linear function for approximation, which is max(0,min(1,0.2x+0.5)), limiting the results of 0.2x+0.50.2x+0.50.2x+0.5 to the interval [0,1], and clipping the excess.

[0069] Spatial perception: Spatial perception is implemented in two steps. The first is to perform sparse sampling based on deformable convolution, and the second is to aggregate features of each level at the same spatial position. It adds a bias to the position to obtain deformation representation capabilities, and uses an importance factor m to adaptively weight the deformation sampling position. Its structure is the middle part of dyhead_DCNv4:

[0070]

[0071] In the above formula, K is the number of sparse sampling positions, p k +Δp kis the spatial offset through self-learning, Δp k is the displacement position, Δp k is the offset of the convolution kernel position k. This offset is given by the offset layer calculated by the feature map, dynamically adjusting the position of the sampling point, Δm k is the weight of adaptive learning.

[0072] Task perception: The task perception ability of the detector is improved by changing the expressive power of different channels corresponding to different detection heads; different channels are related to different tasks. Adaptively activating the channel is actually a dyrelu operation. Ordinary relu only limits the input to be greater than 0. If it is greater than 0, the output is equal to the input. Dyrelu learns four values ​​for each channel of the feature layer to control the activation strength of the channel. In the implementation structure, global average pooling is first performed on the L×S dimension to reduce the dimension, then two fully connected layers and a normalization layer are used, and finally an offset Sigmoid function is applied to normalize the output to the range of [-1,1].

[0073] π c (F)·F=max(α 1 (F)·F c +β 1 (F),α 2 (F)·F c +β 2 (F)

[0074] In the above formula, F c is the feature slice of the cth channel, [α 1 ,α 2 ,β 1 ,β 2 ] T =θ(·) is a hyperfunction used to learn and control the activation threshold.

[0075] It can be seen that the dynamic detection head includes scale-aware, space-aware, and task-aware attention modules, which adjust the input to the same scale to form a three-dimensional tensor F∈R L×H×W×C , the three-dimensional tensor is used as the input of the dynamic head. After passing through the scale perception module, the feature map becomes more sensitive to the scale difference of the foreground object; after passing through the spatial perception module, the feature map becomes sparser and focuses on the discriminative spatial position of the foreground object. Finally, after passing through the task perception module, the feature map is reformed into different activation features according to the requirements of different downstream tasks. Reformulating different activation features can alleviate the information loss caused by occlusion problems, and small-scale targets can be noticed again, which plays a certain role in solving the occlusion problem of student behavior detection in real classrooms; therefore, the Dyhead_DCNv4 detection head is expressed as:

[0076] W(F)=π c (π s (π L (F)·F)·F)·F

[0077] Among them, W(F) represents the output features of the Dyhead_DCNv4 detection head, F represents the input features of the Dyhead_DCNv4 detection head, and πL(·), πS(·), and πC(·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

[0078] Improve the YOLOv11 model to perform model pruning after training:

[0079] With the development of deep learning technology, convolutional neural networks (CNNs) have achieved remarkable success in the field of computer vision, especially in object detection tasks. The YOLO (You Only Look Once) series is an end-to-end object detection model based on CNN. It is widely used in real-time detection scenarios because of its balance between accuracy and speed. However, with the update of the model version, the structure of the YOLO model has become more and more complex, and the number of parameters and the amount of calculation have increased significantly. Especially in resource-constrained scenarios (such as mobile terminals and embedded devices), the complexity of the model may lead to too long inference time and it is difficult to meet the real-time requirements. Therefore, model compression has become an important step in the practical application of deep learning models. Common model compression methods include parameter quantization, knowledge distillation, and pruning. Among them, pruning is a technology that reduces the size of the model by removing unimportant parameters. The weight amplitude-based pruning method is widely used in the compression task of deep learning models because of its simplicity and effectiveness. However, traditional pruning methods often fail to fully consider the characteristics of different layers in the network, resulting in excessive or insufficient pruning of some layers, affecting the overall performance of the model. LAMP is a layer-adaptive amplitude pruning algorithm, which can effectively retain important features and reduce redundant weights by adaptively pruning the weight amplitude distribution of each layer. The present invention proposes a compression method that applies the LAMP pruning algorithm to the YOLOv11 model, which can maintain high target detection accuracy and meet real-time requirements while significantly reducing model parameters, so as to solve the problem that the existing YOLOv11 model is difficult to deploy quickly in resource-constrained scenarios. By applying the LAMP pruning algorithm, the model can adaptively select the pruning ratio at different levels according to the importance of weights, which can not only reduce redundant parameters but also retain key features, ensuring that the pruned model can still maintain high accuracy in target detection tasks.

[0080] First, use the pre-trained YOLOv11 model as the initial model. This model has been trained on a large-scale dataset and has good detection performance. Subsequently, based on the LAMP pruning algorithm, the weight amplitude distribution of each convolutional layer in the YOLOv11 model is calculated. LAMP adaptively calculates the L1 norm of the weight of each layer and determines the pruning threshold based on the weight distribution of each layer. The weight distribution of different layers has different characteristics. LAMP can adaptively adjust the pruning ratio of each layer based on these differences. The pruning formula is as follows:

[0081] W′=W·1 {|w|＞λ}

[0082] Among them, W is the weight matrix of the convolutional layer, λ is the pruning threshold, and 1 is the indicator function, which means that the weight is retained when the weight is greater than a certain threshold.

[0083] According to the pruning threshold, the weights in the YOLOv11 model whose weight amplitude is lower than the threshold are reset to zero to reduce unnecessary parameters. The LAMP pruning strategy ensures that the weights at different levels are adaptively pruned according to their importance, which can retain important feature information and avoid the decrease in model accuracy due to excessive pruning. The pruned model usually has a certain degree of performance degradation, so it is necessary to fine-tune the pruned YOLOv11 model. By using the original training data set, several rounds of fine-tuning are performed to restore the performance of the model in the target detection task. This step can ensure that the model still maintains a high detection accuracy when the number of parameters is reduced.

[0084] The LAMP pruning algorithm can adaptively adjust the pruning ratio according to the importance of each layer's weights, avoiding the over- or under-pruning of traditional pruning methods. This pruning method is suitable for a variety of application scenarios of the YOLOv11 model, especially for object detection tasks that need to be deployed on resource-constrained devices (such as mobile or embedded devices).

[0085] S2: Obtain the classroom video of the student to be detected and input it into the student classroom detection model for processing to obtain the student classroom behavior detection result.

[0086] The pruned improved YOLOv11 model is used as the student classroom detection model, and the student classroom video to be detected is input into the student classroom detection model frame by frame, and the student classroom behavior detection results in each frame image can be obtained.

[0087] S3: Use the improved DeepSORT algorithm to process the student classroom behavior detection results to track the student classroom behavior.

[0088] The present invention makes two improvements to the original DeepSORT algorithm. First, the LSTM is combined with the Kalman filter, that is, the state prediction result obtained by the Kalman filter in the original DeepSORT algorithm is replaced by the weighted sum of the state prediction result obtained by the LSTM network and the state prediction result obtained by the Kalman filter; specifically:

[0089] The target tracking part relies on the Kalman filter to predict the state of the target in the next frame, that is, the position, speed, etc. of the target. When the target is not detected (for example, the target is occluded or temporarily lost), the Kalman filter estimates the approximate position of the target in the next frame through the trajectory information of the previous frames. This process mainly consists of two parts: one is to predict the target position, and the other is to update the position of the target detection box. Although the Kalman filter is good at processing linear predictions of system states and can cope well with noise and short-term motion changes, it has poor prediction capabilities for complex nonlinear motion patterns (such as sudden acceleration, deceleration, and curved motion). LSTM is good at processing time series data, can capture the motion pattern of the target over a long period of time, and has a good prediction effect on complex nonlinear trajectories.

[0090] In the prediction stage, the Kalman filter predicts the state of the target in the next frame based on the state of the target in the previous frame (position, speed, etc.). This process is based on the assumption of the motion model that the motion of the target between frames is linear, and the state vector is used to describe the position and motion information of the target, which is expressed as Where x and y are the coordinates of the center point of the object, a is the aspect ratio of the bounding box, and h is the height of the bounding box. is the velocity component. The prediction equation of Kalman filter is as follows:

[0091]

[0092] in is the predicted state at time t, is the estimated state of the previous frame, and F is the state transfer matrix. The covariance matrix is ​​also updated along with the uncertainty describing the state estimate.

[0093] P t|t-1 =F·P t-1|t-1 ·F T +Q

[0094] Where P t|t-1 is the covariance matrix during prediction, and Q is the process noise covariance matrix, which represents the noise generated by inaccurate modeling.

[0095] The LSTM (Long Short-Term Memory Network) model structure is as follows Fig. 9As shown in the figure, after the data is input, a time series input is required, and the state of the target in the previous frames (such as (x, y, a, h)) is used as input. Sufficient historical information is provided to LSTM by selecting a certain time window (such as the past 5 or 10 frames). The output of LSTM can then be the predicted state of one or several frames in the future. The specific output dimension is consistent with the Kalman filter, that is, the predicted target position and size (x, y, a, h).

[0096] The second step is the update phase. When new detection results appear, the results of the Kalman filter and LSTM are combined to update the state through data, by giving different weights to the prediction results of the Kalman filter and LSTM:

[0097] The Kalman filter maps the state to the observation space through the observation model, and then determines how to perform weighted averaging between the predicted value and the observed value through the Kalman gain K to generate an updated state estimate. The calculation formula of the Kalman gain is:

[0098] K t =P t|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1

[0099] In the above formula, H is the observation, R is the observation noise covariance matrix, and the Kalman gain is used to combine the predicted state and the observed value to obtain the updated state:

[0100]

[0101] where z t is the actual observed target bounding box, and the Kalman filter is used to predict the position and velocity of the target between consecutive frames. By modeling physical quantities such as position and velocity as state vectors and combining them with the observed bounding box information, the Kalman filter can continuously adjust the state estimate of the target. It effectively smoothes the trajectory of the target, reduces the impact of noise on tracking, and maintains the continuity of the tracking task in the event of a temporary loss of the target.

[0102] The output gate of LSTM determines the output at the current moment. First, the value of the output gate is calculated, and then the final hidden state h is generated based on the updated cell state. t, run a sigmoid layer to determine which part of the cell state will be output. Then process the cell state through tanh (get a value between -1 and 1) and multiply it with the output of the sigmoid gate, and finally only the part that the user determines to output will be output. Finally, different weights are given to the prediction results of the Kalman filter and LSTM. You can decide whether to rely more on the results of the LSTM or Kalman filter based on the complexity of the movement or historical data.

[0103] The second improvement of the present invention to the original DeepSORT algorithm is to replace the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network. Specifically:

[0104] In DeepSORT, the feature extractor is used to extract the visual features of the target from the image so as to perform similarity matching during the tracking process. However, although the original feature extractors (such as ResNet or earlier versions of CNN) can provide higher feature expression capabilities, the computational overhead is large, especially in resource-constrained real-time systems, efficiency becomes a problem. In order to further reduce the computational complexity while maintaining a high feature extraction capability, the present invention proposes to use MobileNetV4 as the feature extractor of DeepSORT. MobileNetV4 is a lightweight convolutional neural network designed for mobile devices and embedded devices, which can perform efficient reasoning under low computing resources. MobileNetV4 uses deep separable convolution, Squeeze-and-Excitation modules and better network structure search technology to achieve a better balance between accuracy and speed. Its advantage lies in its lightweight feature extraction capability, which can provide fast and accurate feature representation, and is particularly suitable for real-time application scenarios.

[0105] In DeepSORT, the original feature extractor (such as ResNet) is replaced with MobileNetV4. MobileNetV4 significantly reduces the amount of computational processing per frame through its efficient network architecture, while still being able to extract sufficiently discriminative features. The present invention uses a MobileNetV4 model pre-trained on ImageNet and fine-tunes it through a specific multi-target tracking dataset. In this way, the model can adapt to changes in target appearance features in multi-target tracking scenarios. In order to ensure that the extracted features have good performance in similarity calculations, the output features are L2 regularized to maintain the unit length of the feature vector, thereby enhancing the stability of tracking matching.

[0106] Use the MobileNetV4 model to extract the features of the detection frame, and then use the Hungarian algorithm to match the features of the detection results with the features of the existing tracking targets. Specifically:

[0107] After the Kalman filter predicts the position of the tracked target in the current frame, the Hungarian algorithm is used to associate the target detection box with the tracking trajectory between consecutive frames. Specifically, the Hungarian algorithm solves the allocation problem in tracking, that is, how to match the detection results in the current frame (such as detected people or objects) with the existing tracking targets in the previous frame. The algorithm is an algorithm for solving the optimal matching problem of bipartite graphs. In the target tracking task, it is used to minimize the matching cost between the detection box and the tracking target to achieve the best allocation. The student behavior detection model generates multiple target detection boxes in each frame, and the tracker maintains some existing trajectories. Each frame needs to assign the detected targets to the existing tracking trajectories in order to continuously track these targets. The Hungarian algorithm is used to solve how to match these detected targets with existing trajectories.

[0108] The Hungarian algorithm first calculates the matching cost matrix in the tracking process. The matching cost matrix reflects the "distance" between the target detected in the current frame and the tracked target in the previous frame. This distance can be expressed in a variety of metrics. The main basis of the matching cost matrix is ​​the position and appearance features: IoU, Mahalanobis distance and cosine similarity of ReID feature vectors. The IoU is calculated based on the overlapping area of ​​the target detection box as the spatial distance between the detection result and the tracking trajectory. Then the matching cost is measured by the distance between the predicted state of the Kalman filter and the detection box; finally, the cosine similarity between the detection result and the feature vector of the existing tracking target is calculated based on the ReID feature vector (target re-identification) of the appearance. Through this similarity, the system can distinguish between targets with similar or overlapping appearances. Then, the cost between each detected target and each tracking trajectory predicted by the Kalman filter is encoded to construct a two-dimensional matrix. Each element of the matrix represents the matching cost between the detection box and the trajectory. The smaller the value, the easier it is to match.

[0109] After obtaining the cost matrix, the Hungarian algorithm is used to solve the problem of minimizing the matching cost. The Hungarian algorithm will find a set of optimal matches in the cost matrix to minimize the total cost. Find the minimum cost combination from the cost matrix to ensure that each detection box and each tracking track can only match once at most; if a detection box finds a matching tracking track, and the cost of this match is within the set threshold (such as IoU or ReID feature similarity threshold), then confirm this match. In this way, the association between the detection box and the tracking target is achieved; for the detection box that is not matched, it may be a new target, so a new tracking track is created; and for the tracking track that is not matched, it is considered that the target may be lost or temporarily occluded.

[0110] After the match is successful, the improved DeepSORT algorithm will update the matching trajectory according to the results of the Hungarian algorithm, and update the status of the matched trajectory, including position, speed and other information; then manage the unmatched trajectory. If a tracking target does not find a matching detection box in multiple frames, the DeepSORT algorithm may mark it as lost and remove it; for subsequent new trajectories, if a detection box does not match any existing trajectory, the DeepSORT algorithm will consider it a new target and create a new tracking trajectory. The above process is used to track students' classroom behavior.

[0111] In summary, the present invention first replaces the C3K2 module in the original backbone network by designing the Star_caa module, enhances the extraction of global image information, and improves the detection performance of the model. Due to the complex scenes in the real classroom, it is difficult to guarantee the accuracy and recognition rate by relying on the original detection head alone. Therefore, the present invention adopts the DYHead_DCNV4 detection head module, and strengthens the model's understanding of global and local information through multi-scale feature fusion attention mechanism and variability convolution, and performs more stable and accurate in dealing with occlusion and complex backgrounds. In addition, for the situation where students are crowded and details are easy to ignore in the classroom, the downsampling strategy is optimized, and the LD_downsample downsampling module is used to reduce the loss of detail information, improve the feature extraction effect, and significantly reduce the amount of calculation. Under the requirements of meeting real-time and mobile terminal deployment, the model is lightweighted in combination with LAMP pruning technology, reducing redundant weights and computing resource consumption, and improving mobile terminal performance. In terms of target tracking, traditional algorithms are insufficient in efficiency and accuracy, especially when dealing with occlusion, ID switching is prone to occur. To this end, the present invention combines LSTM network to process time series data, captures the long-term motion pattern of the target, and improves the prediction effect of nonlinear trajectory. In addition, in order to further reduce the computational complexity and maintain strong feature extraction capabilities, MobileNetV4 is used as the feature extractor of DeepSORT to achieve efficient tracking of student behaviors.

[0112] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting student classroom behavior based on deep learning, characterized in that: include: S1: Train the improved YOLOv11 model, prune the trained improved YOLOv11 model, and obtain the student classroom detection model; S2: Obtain the classroom video of the student to be detected and input it into the student classroom detection model for processing to obtain the student classroom behavior detection results; S3: Use the improved DeepSORT algorithm to process the student classroom behavior detection results to track the student classroom behavior.

2. According to the method for detecting student classroom behavior based on deep learning in claim 1, it is characterized in that: The improved YOLOv11 model includes an improved Backbone network, an improved Neck network and an improved Head network; the improved Backbone network replaces the C3K2 module in the Backbone network of the original YOLOv11 model with a Star_caa module; the improved Backbone network and the improved Neck network both replace the Conv downsampling module in the original YOLOv11 model with a LD_downsample downsampling module; the improved Head network replaces the detection head of the Head network in the original YOLOv11 model with a Dyhead_DCNv4 detection head.

3. According to a method for detecting student classroom behavior based on deep learning in claim 2, it is characterized in that: The Star_caa module processes input features by: using the first DWConv layer to process the input features to obtain the first intermediate feature; processing the first intermediate feature through two parallel 1×1 convolutional layers, and performing a star operation on the two convolutional outputs to obtain the second intermediate feature; processing the second intermediate feature with the CAA attention mechanism and inputting it into the 1×1 convolutional layer to obtain the third intermediate feature; inputting the third intermediate feature into the second DWConv layer for processing to obtain the fourth intermediate feature; performing a residual connection between the fourth intermediate feature and the input feature and performing regularization to obtain the output feature of the Star_caa module.

4. According to a method for detecting student classroom behavior based on deep learning in claim 2, it is characterized in that: The LD_downsample downsampling module includes two branches, one branch is the maximum pooling layer connected to the CBS module, and the other branch is the CBS module connected to the LDConv layer; the outputs of the two branches are concatenated as the output features of the LD_downsample downsampling module.

5. A method for detecting student classroom behavior based on deep learning according to claim 4, characterized in that: The CBS module consists of a standard convolutional layer, a batch normalization layer, and an activation function Silu.

6. A method for detecting student classroom behavior based on deep learning according to claim 2, characterized in that: The Dyhead_DCNv4 detection head is represented as: W(F)=π c (p s (p L (F)·F)·F)·F Among them, W(F) represents the output features of the Dyhead_DCNv4 detection head, F represents the input features of the Dyhead_DCNv4 detection head, and πL(·), πS(·), and πC(·) represent three different attention functions applicable to dimensions L, S, and C, respectively.

7. The method for detecting student classroom behavior based on deep learning according to claim 1 is characterized in that: The LAMP pruning algorithm is used to prune the trained improved YOLOv11 model.

8. The method for detecting student classroom behavior based on deep learning according to claim 1, characterized in that: The improved DeepSORT algorithm replaces the state prediction result obtained by the Kalman filter in the original DeepSORT algorithm with the weighted sum of the state prediction result obtained by the LSTM network and the state prediction result obtained by the Kalman filter, and replaces the feature extractor in the original DeepSORT algorithm with the MobileNetV4 network.

Images