Light-weight small-target detection model and method for student classroom behavior recognition based on task alignment and heavy parameters

By adopting a lightweight detection model based on task alignment and heavy parameters in student classroom behavior detection, the problems of low detection accuracy of small objects and difficult model deployment are solved, efficient classroom behavior recognition is achieved, and detection accuracy is improved and reduced. Computing resource requirements.

CN119068541APending Publication Date: 2024-12-03HARBIN NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410936551.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing student classroom behavior detection methods have low accuracy in small object detection in complex classroom scenarios, and due to the high amount of model parameters and calculations, it is difficult to deploy on devices with limited performance.

Method used

Using a lightweight small object detection model based on task alignment and heavy parameters, the R-DWR module and Ghost Module are introduced, combined with FPN and PANet, the task interaction prediction head TAPH is designed, and the model parameter quantity is reduced by using Group Normalization and dynamic convolution. The calculation amount is increased while improving detection accuracy.

Benefits of technology

Improve the ability to detect small objects in severe occlusion images, reduce the calculation amount and parameter amount of the model, improve the detection accuracy, and realize effective deployment on performance-constrained devices. The detection accuracy is 1.5% higher than that of the benchmark model, and the calculation amount is and parameter volume decreased by 14.8% and 43.8% respectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068541A_ABST
    Figure CN119068541A_ABST
Patent Text Reader

Abstract

The invention discloses a light-weight small target detection model and method for student classroom behavior recognition based on task alignment and heavy parameters, the model comprises a backbone network, a neck and a task interaction prediction head based on a task alignment idea, the backbone network extracts features from an image, downsampling processing is performed, feature fusion is performed on the neck, and a task interaction prediction result is obtained. And the task interaction prediction head further extracts task interaction features and outputs a prediction result. The detection precision of the student classroom behavior detection model provided by the invention is higher than that of a reference model YOLOV8n, on an STCB data set, mAP reaches 79.2% and is 1.5% higher than that of the YOLOV8n, and the detection precision of small targets in the data set is 11.2% and is 1.7% higher than that of the YOLOV8n. The calculation amount of the model is reduced from 8.1 FLOPs to 6.9 GFLOPs, and is reduced by 14.8%. And the parameter quantity is reduced from 3.2 M to 1.8 M, and is reduced by 43.8%.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a target detection model, and in particular to a lightweight small target detection model for identifying students' classroom behaviors. Background Art

[0002] Existing classroom behavior detection methods are divided into detection methods based on human posture estimation and direct detection methods. The detection method based on human posture estimation first uses the OpenPose framework as the human posture estimation framework to generate vectors representing human postures from classroom images, including joint positions, distances, and bone angles, and then uses CNN to learn the characteristics of students' classroom behaviors from the bone information and classify students' classroom behaviors. This method is not suitable for complex classroom scenes. Pose estimation requires clear skeleton information, but there are often serious occlusions in the classroom. Pose estimation also requires more computing resources and complex models to achieve. Compared with traditional target detection methods, the algorithm is more complex, which makes it difficult to deploy in classrooms with limited teaching resources. The direct detection method uses single-stage detectors such as YOLO, SSD, and DETR to locate and classify all objects in one stage of reasoning, with faster detection speed and less computing resources required. The single-stage detector usually consists of three components: the backbone, the neck, and the detection head. The backbone extracts features from the input image, performs feature fusion and enhancement in the neck, and finally the detection head outputs the detection results. Existing detection heads usually use independent localization and classification branches, ignoring the differences in learning strategies between localization and classification tasks, which may lead to a certain degree of spatial misalignment in the predictions between the two tasks, affecting the accuracy of the predictions. Because there are seats in classroom images and they are arranged closely, the scales of the student targets to be detected are inconsistent, and there are many small targets. Direct detection methods perform poorly in detecting small targets because they use a single-stage detector and do not pre-detect candidate areas. Although Transformer-based target detection models such as DETR have high detection accuracy, they have high computational complexity, so the model has an extremely high number of parameters, making it difficult to deploy on performance-constrained devices. Summary of the invention

[0003] In order to solve the problem of low detection accuracy of existing student classroom behavior detection methods for small targets caused by widespread occlusion in the classroom and the high difficulty of deploying student classroom behavior detection models on performance-constrained devices due to excessively high number of parameters and computational complexity, the present invention provides a lightweight small target detection model and method for student classroom behavior recognition based on task alignment and heavy parameters.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A lightweight small target detection model for student classroom behavior recognition based on task alignment and heavy parameters includes a backbone network Backbone, a neck Neck, and a task interaction prediction head TAPH based on the idea of ​​task alignment. The backbone network Backbone extracts features from the image and performs downsampling processing, the neck Neck performs feature fusion, and the task interaction prediction head TAPH further extracts task interaction features and outputs prediction results, where:

[0006] The backbone network Backbone introduces the R-DWR module in DarkNet53. The R-DWR module divides the process of extracting multi-scale features into two steps: regional residualization and semantic residualization. In the regional residualization part, a 3×3 convolution combined with BN and ReLU functions is used to extract preliminary features; in the semantic residualization part, multiple dilated convolutions with different expansion rates are used to extract multi-scale features; after extracting multi-scale features, all feature maps are fused and BN normalized for the fused feature maps, and point-by-point convolution is used to merge features to form the final residual. Finally, the input feature map is added to the final residual to build a stronger and more comprehensive feature representation;

[0007] The Neck is composed of FPN and PANet. GDC is combined in the C2f layer. The feature map input by the backbone network is first adjusted by a 1×1 CBS to adjust the number of channels and extract features. Then, the split operation is used to divide the output feature map into two parts. One part has a channel number of C in / 2 feature maps are directly connected to the output through the residual, and the number of channels in the other part is C in The feature map of / 2 is passed through multiple parallel GDCs to extract more features, and then the outputs of the two parts are fused to generate a channel number of (n+2)C in / 2, n represents the number of GDCs used, and finally further processed by a CBS module to adjust the number of channels to C in , to generate the final output feature map;

[0008] The TAPH consists of four parts, namely, a feature extractor, a task decomposition module, a boundary prediction branch and a category prediction branch. The feature extractor uses two Conv_GNs to learn task interaction features from the input of Neck. Conv_GN uses GN (Group Normalization) instead of BN (Batch Normalization) to enhance the performance of detection head positioning and classification. The feature extractor inputs the learned task interaction features into the task decomposition module. The task decomposition module decomposes the task interaction features into classification prediction features and boundary prediction features through TDM. The classification prediction features and boundary prediction features are respectively used for the classification prediction branch and the boundary prediction branch to calculate the classification score and the positioning area. The output feature map uses the Scale layer to adjust the scale of the features after the convolution layer to adapt to the use of shared convolution, and finally obtains the prediction result.

[0009] A method for detecting student classroom behavior based on task alignment and re-parameterization includes the following steps:

[0010] Step 1: Input the image to be detected

[0011] Input the images containing students’ classroom behaviors into the pre-trained lightweight small object detection model for students’ classroom behavior recognition;

[0012] Step 2: Feature extraction

[0013] Use backbone network for feature extraction;

[0014] Step 3: Feature Fusion

[0015] In the neck, the features extracted by the backbone network are fused and enhanced to obtain a more powerful feature representation;

[0016] Step 4: Task interaction feature extraction

[0017] The task interaction features are calculated from the fused features output by the neck using a feature extractor;

[0018] Step 5: Task Decomposition

[0019] The task decomposition module is used to decompose the task interaction features into classification prediction features and boundary prediction features.

[0020] Calculate classification scores and regional probabilities;

[0021] Step 6: Positioning and classification

[0022] The classification prediction features and boundary prediction features are calculated by the classification prediction branch and the boundary prediction branch respectively to obtain the regional probability and classification score;

[0023] Step 7: Decoding

[0024] Decode the region probability and classification score and convert them into the final positioning result and classification result;

[0025] Step 8: Non-maximum suppression

[0026] The final prediction results are adjusted using non-maximum suppression technology, by retaining the student classroom behavior detection box (Bounding Boxes) with the highest score and suppressing other student classroom behavior detection boxes that highly overlap with it, thus eliminating redundant student classroom behavior detection boxes;

[0027] Step 9: Output test results

[0028] The lightweight small target detection model for student classroom behavior recognition outputs the final detection results. The inspection results include detection boxes, category labels, and confidence scores. The confidence score of each detection box indicates whether the box contains the target and the accuracy of the target category.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. This paper constructs a dataset for detecting and analyzing students' classroom behaviors, focusing on seven typical student behaviors and three interactive behaviors between teachers and students in the classroom.

[0031] 2. Aiming at the problem of small target recognition, the present invention designs a detection head TAPH, and designs a task alignment structure on the detection head. This structure enhances the interactivity of the traditional single-stage detector positioning task and the classification task, and learns the task interaction features from multiple convolutional layers through the feature extractor. In addition, shared convolution is used in the detection head to reduce the number of model parameters. While using shared convolution, in order to cope with the problem of inconsistent scales of the targets detected by each detection head, the Scale layer is used to scale the features. This method effectively improves the model's ability to extract small target feature information from severely occluded images.

[0032] 3. In order to adapt to the performance-limited devices in the classroom, the present invention introduces the Ghost Module in the C2f of the YOLOV8 model Neck. By replacing the ordinary convolution in the Ghost Module with dynamic convolution, the calculation amount of the model is reduced and the accuracy is improved.

[0033] 4. The present invention introduces R-DWR into the Backbone of the YOLOV8 model, extracts multi-scale context features through dilated convolutions with different dilation rates, improves the model's ability to recognize small targets, and uses a heavy parameter method to reduce the amount of model calculation.

[0034] 5. The student classroom behavior detection model proposed in this invention has higher detection accuracy than the baseline model YOLOV8n. On the STCB dataset, the mAP reaches 79.2%, which is 1.5% higher than YOLOV8n. The detection accuracy of small targets in the dataset is 11.2%, which is 1.7% higher than YOLOV8n. The computational complexity of the model is reduced from 8.1FLOPs to 6.9GFLOPs, a decrease of 14.8%. The number of parameters is reduced from 3.2M to 1.8M, a decrease of 43.8%. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the overall structure diagram of the YOLO-TGD model;

[0036] Figure 2 A flowchart of the method for detecting students' classroom behaviors based on task alignment and re-parameterization;

[0037] Figure 3 Provide specific details of the method for detecting student classroom behavior based on task alignment and re-parameterization;

[0038] Figure 4 It is a TDM structure;

[0039] Figure 5 is the C2f_GDC structure;

[0040] Figure 6 Comparison between GDC and traditional convolution;

[0041] Figure 7 It is an R-DWR structure, RR stands for regional residualization, and SR stands for semantic residualization;

[0042] Figure 8 The sample images of the STCB dataset. In each sub-image, a corresponding behavior target is marked with a red box: (a) listening, (b) reading, (c) writing, (d) standing, (e) raising hands, (f) group cooperation, (g) dozing off, (h) supervising, (i) lecturing, and (j) guiding.

[0043] Fig. 9 To compare the average accuracy of the model in each classroom behavior detection;

[0044] Fig.10 Visual comparison of YOLO-TGD and YOLOV8n results on the STCB dataset. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0046] Based on YOLOV8, the present invention provides a lightweight small target detection model YOLO-TGD for student classroom behavior recognition based on task alignment and heavy parameters, such as Figure 1 As shown, the YOLO-TGD includes a backbone network Backbone, a neck Neck and a task interaction prediction head TAPH based on the idea of ​​task alignment. The backbone network Backbone extracts features from the image and performs downsampling, the neck Neck performs feature fusion, and the task interaction prediction head TAPH further extracts task interaction features and outputs prediction results.

[0047] The present invention designs a TAPH with a novel structure, which consists of four parts, namely a feature extractor, a task decomposition module, a boundary prediction branch and a category prediction branch. The feature extractor uses two Conv_GNs to learn task interaction features from the input of Neck. Conv_GN uses Group Normalization (GN) instead of BatchNormalization (BN) to enhance the performance of detection head positioning and classification. Unlike the detection head of the YOLOV8 model, these two Conv_GNs are shared by detection tasks of different scales. The task interaction features learned by the feature extractor are then input into the task decomposition module to align classification and positioning tasks. The present invention uses TDM (Task Decomposition Module) to decompose tasks, and promotes task decomposition by dynamically calculating such task-specific features at the layer level. Its structure is as follows Figure 4 As shown. Let X inter represents the task interaction features extracted by the feature extractor from Neck, X k task From X inter The task-specific features are calculated separately for each classification or positioning task, as shown in Equation 1.

[0048]

[0049] ω=σ(fc2(δ(fc1(x inter )))) (2)

[0050] where ω k represents the kth element learned from the Layer attention layer, represents the task interaction feature learned by the kth Conv_GN from the Neck input. ω is calculated by the cross-layer task interaction feature, which can capture the dependency between layers. Its calculation formula is shown in Equation 2. σ and δ represent the sigmoid function and ReLU function respectively. fc1 and fc2 are two fully connected layers. inter By Xinter Perform global average pooling to obtain each Combine joint features X task , X task Used to calculate the positioning or classification result Z task , as shown in Equation 3. Then the sigmoid function is used to convert Z task Converted into classification prediction feature P or bounding box prediction feature B.

[0051] Z task =conv2(δ(conv1(X task ))) (3)

[0052] P and B are used for the classification prediction branch and the boundary prediction branch to calculate the final prediction result. The classification prediction branch uses the spatial probability map C to learn the consistency between the two tasks at each spatial position, as shown in Equation 4. C is derived from X inter The calculation formula is shown in Formula 5. The element product of P and B is then passed through a convolutional layer to obtain the final classification result P align .

[0053]

[0054] C = σ(conv3(δ(conv2(X inter )))) (5)

[0055] The boundary prediction branch passes through a convolutional layer from X inter The mask M and spatial offset O are calculated as shown in Equation 6 and Equation 7.

[0056] O = δ(conv1(X inter )) (6)

[0057] M=σ(δ(conv1(X inter ))) (7)

[0058] M and O are used in deformable convolution to adjust the predicted bounding box at each position. The calculation process is shown in Equation 8.

[0059]

[0060]

[0061] B align =Scale(δ(conv bbox (y))) (10)

[0062] x and y represent the input and output feature maps respectively, and y(p) is the value of the output feature map at position p. k and O kRespectively represent the weight of the kth position and the offset of the convolution kernel relative to the center point, where w k Obtained using the GN normalization method. ΔO k and ΔM k denote the learnable offset and modulation scalar at the kth position, respectively, both of which are obtained by applying a convolutional layer to the feature map x, with the input feature map at the offset position p+O. k +ΔO k The value of is calculated using bilinear interpolation, and the calculation formula is shown in Equation 9, where G is the bilinear interpolation weight, and q represents the difference between p+O and p on the feature map. k +ΔO k The output feature map y first passes through a convolution layer as in the classification prediction branch, and then uses the Scale layer to adjust the scale of the features after the convolution layer to adapt to the use of shared convolution, and finally obtains the bounding box prediction result B align , and its calculation process is shown in formula 10.

[0063] The present invention combines GDC in the C2f layer of Neck to reduce the number of parameters and calculations of the model, such as Figure 5 As shown in the figure, the input feature map first passes through a 1×1 CBS (CBS, Conv, BatchNorm, SiLU) to adjust the number of channels and extract features (C in Represents the number of channels), and then uses the split operation to divide the output feature map into two parts, one part (the number of channels is C in / 2) is directly connected to the output through the residual, and the other part (the number of channels is C in / 2) After multiple parallel GDCs are used to extract more features, the outputs of the two parts are then fused to generate a fused feature map (the number of channels is (n+2)C in / 2, n represents the number of GDCs used), and finally further processed by a CBS module to adjust the number of channels to C in , to generate the final output feature map.

[0064] GDC consists of GhostModule and dynamic convolution. The basic idea of ​​GhostModule is to generate redundant feature maps in the convolutional neural network model through a "cheap operation" called linear transformation. Figure 6The design details of GDC and the difference between GDC and ordinary convolution in generating feature maps are shown. Ghost Module divides the generation of feature maps into two parts. The original feature map is generated by a standard convolution operation to generate a part of the feature map. The convolution kernel used is smaller than the convolution used in the generation of ordinary feature maps. Then, the original feature maps are linearly transformed using deep separable convolution in Ghost feature generation to generate more Ghost feature maps. Finally, the original feature map and the Ghost feature map are combined as the final output feature map. The number of convolution kernels used in the original feature map generation in the Ghost module is smaller, and the operation of obtaining the Ghost feature map by combining linear transformation achieves the effect of reducing the amount of calculation and the amount of parameters. In order to balance the parameter amount and feature extraction capability of the model, the present invention uses dynamic convolution to replace the original ordinary convolution in the Ghost Module. Dynamic convolution is implemented using CondConv. A 1×1 CondConv is used to generate the original feature map, and then a 1×1 deep separable CondConv is used to generate the Ghost feature map. CondConv realizes the dynamic nature of convolution by inputting the parameters of the calculated convolution kernel. In CondConv, the convolution kernel parameters are parameterized as a linear combination of multiple expert knowledge, and the calculation formula is shown in Equation 11.

[0065] Output(x)=σ(α1·W1+...+α n ·W n )*x) (11)

[0066] Where x is the output of the previous layer, n represents the number of experts on this CondConv layer, α1,…,α n are the weight coefficients learned by gradient descent, W1,…,W n It is a convolution kernel that represents expert knowledge. The use of CondConv makes the model achieve a balance between accuracy improvement and inference time. Compared with ordinary convolution, the accuracy is improved but the speed is the same.

[0067] The present invention introduces the R-DWR (Reparam Dilation-wise Residual Module) module in the backbone network Backbone. The R-DWR module is an efficient multi-scale feature extraction module, which adopts the design structure of DWR. Figure 7As shown in Figure 1. The R-DWR module is designed in the style of residual, decomposing the traditional single-step method into a two-step method to reduce the difficulty of extracting multi-scale features. The process of extracting multi-scale features is divided into two steps: regional residualization and semantic residualization. In the regional residualization part, a 3×3 convolution combining BN and ReLU functions is used for preliminary feature extraction. Subsequently, in the semantic residual part, multiple dilated convolutions with different dilation rates are used to extract multi-scale features. The two-step multi-scale feature extraction method enables the model to avoid obtaining complex contextual information from feature maps that express duplication, and only needs to adjust the dilation rate of the convolution and perform simple morphological filtering on the feature maps that express conciseness, thereby making the feature learning process more orderly. In semantic residualization, the original dilated convolution is replaced by DRB (Dilated Reparam Block). The purpose of using DRB is to enhance the ability of large kernels to capture sparse patterns. It converts multiple dilated convolutions into non-dilated convolutions with larger kernels through a transposed convolution, thereby eliminating the inference cost of the additional dilated convolution layer. The calculation formula for equivalent conversion is shown in Equation 12.

[0068] W'=conv_transpose2d(W,I,stride=r) (12)

[0069] W and W′ represent the convolution kernels before and after the conversion, respectively, and r represents the stride of the convolution. Through equivalent conversion, the dilated convolution with kernel k×k can be converted to the non-dilated convolution with kernel ((k-1)r+1). Given a set of dilated convolutions with k=(5,3,3) and r=(1,2,3), this equivalent conversion can convert this set of dilated convolutions into non-dilated convolutions with kernel 7, but the number of parameters is less than that of directly using non-dilated convolutions with kernel 7. DRB implements the use of non-dilated small kernels and multiple dilated small kernel layers to enhance the non-dilated large kernel convolution layer. Compared with using large kernel convolution alone, this method allows large kernel convolution to be used together with parallel small kernel convolution, which is beneficial for the model to extract features of small objects during training. After extracting multi-scale features, all feature maps are fused and the fused feature maps are BN normalized. Subsequently, point-by-point convolution is used to merge the features to form the final residual. Finally, the input feature map is added to the final residual to build a stronger and more comprehensive feature representation.

[0070] The present invention provides a method for detecting student classroom behavior based on task alignment and re-parameterization. Figure 2 and Figure 3 ,The deduction process of the student classroom behavior detection model includes the following steps:

[0071] 301. Input the image to be detected

[0072] The images containing students’ classroom behaviors are input into the pre-trained lightweight small target detection model for students’ classroom behavior recognition. Figure 3 As shown in the figure, the backbone network of the lightweight small target detection model for student classroom behavior recognition is incorporated into DarkNet53.

[0073] Step 3: Feature Fusion

[0074] 302. Feature Extraction

[0075] The backbone network DarkNet53 of the student classroom behavior detection model is used for feature extraction to extract low-level to high-level features. These features have different spatial resolutions and semantic information at different levels. DarkNet53 is combined with the R-DWR module to enhance its multi-scale feature extraction capability.

[0076] 303. Feature Fusion

[0077] The extracted features are fused using FPN (Feature Pyramid Network) and PANet (Path Aggregation Network). FPN fuses the features from different levels of the main network to generate multi-scale feature maps, which can handle objects of different sizes. PANet further enhances feature fusion based on FPN to make the feature map more expressive. FPN and PANet form the model neck, and combined with the GDC module, the model parameters and calculation amount are reduced.

[0078] 304. Task Interaction Feature Extraction

[0079] The task interaction features are calculated from the fusion features output by the network neck. Figure 3 As shown, a feature extractor consisting of two shared convolutions is used to calculate task interaction features.

[0080] 305. Task Decomposition

[0081] The task decomposition module TDM is used to decompose the task interaction features into classification prediction features and boundary prediction features.

[0082] 306. Positioning and Classification

[0083] Use classification prediction features and boundary prediction features for positioning and classification. Figure 3 As shown, the classification prediction features and the boundary prediction features are respectively calculated by the classification prediction branch and the boundary prediction branch to obtain the regional probability and the classification score.

[0084] 307. Decoding

[0085] The region probability and classification score are decoded and converted into the final positioning and classification results.

[0086] 308. Non-maximum suppression

[0087] The final prediction results are adjusted using non-maximum suppression technology. The non-maximum suppression technology eliminates redundant student classroom behavior detection boxes by retaining the student classroom behavior detection boxes (Bounding Boxes) with the highest score and suppressing other student classroom behavior detection boxes that highly overlap with them.

[0088] 309. Output test results

[0089] The lightweight small target detection model for student classroom behavior recognition outputs the final detection results. The inspection results include detection boxes, category labels, and confidence scores. The confidence score of each detection box indicates whether the box contains the target and the accuracy of the target category. This score is the product of the probability of the target existing and the category probability.

[0090] Example:

[0091] Deploy cameras in front of or to the side of the classroom to obtain video data of students' classroom behavior and connect to the server terminal through a network cable to save the video data. Use automatic frame cutting tools such as Ffmpeg to automatically cut the video data, thereby converting video data into image data. The model code for student classroom behavior detection is written in Pytorch as the framework. The image data is imported into the pre-trained model to output the prediction results, the prediction results are used to draw standard images, and the annotated images are used to regenerate the video for easy visual analysis. Scientific data analysis libraries such as Pandas and Numpy are used to assist in the analysis of detection results.

[0092] 1. Dataset

[0093] Currently, there are few public datasets available for student classroom behavior. Therefore, this embodiment uses 161 primary school classroom videos to construct the dataset STCB to verify the performance of the model. These original video data were downloaded from the National Public Resource Platform. Before data collection, teachers and students agreed to take videos to ensure that the dataset was only used for research purposes. After collecting and organizing the video data, Ffmpeg was used to automatically cut the video data, and images that were highly related to student classroom behavior were screened out for annotation. The image annotation was carried out under the guidance of teachers with rich teaching experience and completed with the help of Labelimg tools. The student classroom behaviors in the image were labeled into 7 categories, namely listening, reading, writing, standing, raising hands, group cooperation and sleepiness. Taking into account the needs of teaching scenarios, 3 types of teacher classroom behaviors were also labeled to better study the interactive behaviors between students and teachers, namely supervision, lecturing and guidance. The classification criteria for these behaviors are shown in Table 1. After annotation, a dataset consisting of 4242 images and 74,571 annotations for detecting 10 typical student and teacher classroom behaviors was obtained. Figure 8 Some sample images in the dataset are shown.

[0094] Table 1. Guidelines for labeling students’ and teachers’ classroom behaviors

[0095]

[0096] In order to further verify the effectiveness and generalization ability of the proposed model, this embodiment also conducts experiments on the public dataset PASCAL VOC. The PASCAL VOC dataset contains images of 20 common objects in daily life, such as people, birds, airplanes, chairs, televisions, etc. It is a widely used target detection benchmark dataset. The PASCAL VOC dataset has multiple versions, each corresponding to a year. The commonly used versions are VOC2007 and 2007. In this embodiment, the training set and validation set of VOC2007 and VOC2012 are used as the training set and validation set, and the trained model is tested on the test set of VOC2007.

[0097] 2. Experimental Setup

[0098] All experiments were conducted on a server equipped with an Intel Xeon Gold 6230R CPU and an NVIDIA GeForce2080Ti (11G) GPU, using pytorch1.13.0 and CUDA 11.7 as the framework for deep learning training, and Python version 3.8.

[0099] Before model training, the training set, validation set, and test set were divided into 8:1:1 ratios. In order to fully train the model, the epoch was set to 500, and the early stopping method was used to avoid overfitting that may occur during training and the corresponding patience was set to 75, which means that the training will be stopped when the model's indicators on the validation set have not been improved for 75 consecutive epochs. The momentum stochastic gradient descent optimizer was used for training, with the initial learning rate set to 0.01 and the momentum set to 0.937. The batch size in training and other experiments was set to 32.

[0100] In the field of object detection, accuracy, recall and mAP are usually used as important indicators for evaluating models.

[0101] The calculation of precision and recall is shown in Equation 13 and Equation 14.

[0102]

[0103] TP represents the true positive rate, FP represents the false positive rate, and FN represents the false negative rate. With recall as the x-axis and precision as the y-axis, we can draw a precision-recall PR curve. AP is obtained by calculating the area under the PR curve, which can effectively reflect the performance of the model on an unbalanced dataset. mAP is the average of AP for all categories. Their calculations are shown in Equations 15 and 16.

[0104]

[0105] Where AP(i) represents the AP corresponding to the i-th category, and N is the total number of categories. mAP50 and mAP50:95 are two forms of mAP. mAP50 is the mAP when the IoU (Intersection over Union) threshold is fixed at 0.5, and mAP50:95 is the average value of mAP calculated when the IoU threshold is between 0.5 and 0.95. In addition to mAP, FLOPs and Params are also used to measure the lightweight performance of the model. FLOPs represents the number of floating-point operations, which is used to measure the computational complexity of the model. Params refers to the number of parameters of the model, which determines the storage size of the model.

[0106] 3. Experimental results on STCB dataset

[0107] The YOLO-TGD model proposed in this paper is compared with some advanced lightweight models. GhostNet, GhostNetV2, MobileNetV2, MobileNetV3, ShuttleNetV2 and FasterNet are used to replace the backbone of the YOLOV8n model and combined with the inspection head of YOLOV8n to form a complete detector. In addition, YOLOV5n is also compared. Table 2 lists the experimental results of the above models. The best results of each indicator used to evaluate the model are indicated in bold and underlined.

[0108] The results in Table 2 show that the comprehensive performance of the YOLO-TGD model is stronger than other models. The mAP50 and mAP50:95 of YOLO-TGD reached 79.2% and 54.6% respectively, which is 1.5% and 1.1% higher than that of YOLOv8n. The design of TAPH and R-DWR also makes the model better than other models in detecting small targets. In terms of lightweight performance, YOLO-TGD has the least number of parameters, only about 1.8M parameters. Although YOLO-TGD is not the least in terms of computational complexity, it reduces the amount of computation by 1.2G compared to YOLOV8n, and has better comprehensive performance than models with less computational complexity. It is worth noting that MobileNetV2, MobileV3-Small and MobileV3-Large use deep separable convolution and use a large number of 1×1 convolution kernels in their network structures. Although this greatly reduces its parameter volume, it also reduces the model's feature expression ability. The highly complex network design makes deployment more difficult.

[0109] Table 2 Performance of the comparison model on the STCB dataset

[0110]

[0111] Fig. 9 The AP performance of each classroom behavior category of the comparison model is shown, and the value range in the radar chart is set to 0.1 to 0.95 for easier visualization. It can be seen that the AP of the model of the present invention exceeds that of other models in the behavior categories of listening (91.9%), standing (90.2%), group cooperation (77.9%), and dozing off (33.9%), and it is also ranked at the top in other behavior categories. Different models perform poorly in detecting drowsy behavior. This may be because sleepy students are usually blocked by their own bodies, and their curled-up bodies account for a small proportion of the image. These factors prevent the model from extracting effective feature information, and thus the detection effect of sleepy behavior is poor. The model of the present invention far exceeds other models in detecting drowsy behavior, which shows that the model of the present invention has a better detection effect on small targets.

[0112] In order to further verify the proposed TAPH in improving the model's ability to detect small targets, the AP of the model for identifying small, medium and large targets was compared. As shown in Table 3, the addition of TAPH significantly improves the model's detection accuracy for small targets, which is higher than the detection accuracy of large targets, and the detection accuracy of slightly smaller and medium targets is improved, which shows that the model of the present invention can extract multi-scale features well.

[0113] Table 3 TAPH improves the model's small target detection capability

[0114]

[0115] This example also compares the visualization performance of YOLO-TGD and YOLOV8. Fig.10 As shown in the figure, the original version, YOLOV8n detection version, YOLO-TGD detection version and true annotation of 4 test set images are given. Fig.10 As can be seen from the figure, the model of the present invention has advantages over YOLOV8n in both classification and positioning. Fig.10 As shown in the first three rows, the model of the present invention correctly classifies the students who raise their hands, the students who fall asleep, and the teachers who are in class, which is the same as Ground Truth. However, the YOLOV8n model classifies them as listening and standing, respectively, which shows that the model is more accurate in predicting the categories. Secondly, Fig.10 The 4th row in the figure shows that the model of the present invention successfully locates and classifies the student who is reading on the left side of the image, but YOLOV8n does not detect this smaller target, which indicates that the model of the present invention is superior to YOLOV8n in detecting small targets. In addition, compared with GroundTruth, both the model of the present invention and YOLOV8n give redundant annotations, detecting students who are working in groups as reading, listening to lectures and other classroom behaviors, so there is still room for improvement in positioning and classification. Overall, the above analysis of the results shows that the model of the present invention can overcome the difficulty of strong occlusion in classroom scenes and is competent for the detection of dense small targets, and performs better than the baseline model YOLOV8n in the task of classroom behavior detection.

[0116] 4. Experimental results on VOC dataset

[0117] In order to verify the generalization ability of the YOLO-TGD model proposed in this invention in different scenarios, experiments were conducted on the VOC dataset. The comparison model used is the same as the model used in STCB, the training cycle is set to 300 epochs, and other settings remain unchanged. As shown in Table 4, the YOLO-TGD model still performs better than the baseline model in mAP, and the mAP50 is the highest among all models, reaching 64.4%, 1.8% higher than YOLOV8n, and mAP50:95 is only 0.4% lower than the highest MobileNetV2 model. In the detection of small targets, the mAP of YOLO-TGD exceeds all other models and is 1.4% higher than the second highest FasterNet.

[0118] The above experimental results show that YOLO-TGD is not only capable of classroom behavior detection tasks, but also performs well in other scenarios, especially for the detection of dense small objects. YOLO-TGD has very few calculations and parameters, which makes it better deployed in performance-constrained life scenarios.

[0119] Table 4 Performance of the comparison model on the VOC dataset

[0120]

[0121] 5. Ablation experiment

[0122] In order to verify the contribution of the key modules proposed in this paper to improving the model performance, TAPH, GDC, and R-DWR are integrated into the baseline YOLOv8n model in turn to study the enhancement of the model's small object detection capability and lightweight performance by these improvement strategies. The experiment was conducted on STCB, and the comparison before and after adding different modules is shown in Table 5.

[0123] Table 5 Comparison before and after adding different modules

[0124]

[0125] TAPH uses shared convolution and designs a task alignment structure to improve the interactivity of classification and positioning tasks. Compared with the detection head used by YOLOV8n, TAPH extracts common features of objects of different scales, thus significantly improving the detection accuracy of small objects. sCompared with the baseline model, it is improved by 2.3%. At the same time, the use of shared convolution reduces the number of parameters by 1.0M, but the more complex network result design increases the model's computational complexity by 0.6G. The combination of C2f and GDC greatly reduces the model's computational complexity. The use of dynamic convolution increases the number of parameters compared to the original Ghost Module, balancing the model's performance in terms of accuracy and lightweight. By adding GDC to TAPH, the model's computational complexity and parameter complexity are reduced by 1.5G and 0.2M respectively, while avoiding the loss of accuracy as much as possible. Subsequently, R-DWR was added. The addition of R-DWR also improves the model's ability to extract multi-scale features, AP s The improvement is 0.7%, which is due to the use of dilated convolutions with multiple dilation rates in this module. At the same time, the use of reparameterization technology further reduces the model's computational complexity by 0.3G and the number of parameters by 0.2. Finally, the model of the present invention outperforms the YOLOV8n benchmark model in both accuracy and lightweight performance, which proves the effectiveness of the optimization strategy of the present invention.

Claims

1. A lightweight small target detection model for student classroom behavior recognition based on task alignment and heavy parameters, Features The model includes a backbone network Backbone, a neck Neck, and a task interaction prediction head TAPH based on the idea of ​​task alignment. The backbone network Backbone extracts features from the image and performs downsampling processing, the neck Neck performs feature fusion, and the task interaction prediction head TAPH further extracts task interaction features and outputs prediction results, wherein: The backbone network Backbone introduces the R-DWR module in DarkNet53. The R-DWR module divides the process of extracting multi-scale features into two steps: regional residualization and semantic residualization. In the regional residualization part, a 3×3 convolution combined with BN and ReLU functions is used to extract preliminary features; in the semantic residualization part, multiple dilated convolutions with different expansion rates are used to extract multi-scale features; after extracting multi-scale features, all feature maps are fused and BN normalized for the fused feature maps, and point-by-point convolution is used to merge features to form the final residual. Finally, the input feature map is added to the final residual to build a stronger and more comprehensive feature representation; The Neck is composed of FPN and PANet. GDC is combined in the C2f layer. The feature map input by the backbone network is first adjusted by a 1×1 CBS to adjust the number of channels and extract features. Then, the split operation is used to divide the output feature map into two parts. One part has a channel number of C in / 2 feature maps are directly connected to the output through the residual, and the number of channels in the other part is C in The feature map of / 2 is passed through multiple parallel GDCs to extract more features, and then the outputs of the two parts are fused to generate a channel number of (n+2)C in / 2, n represents the number of GDCs used, and finally further processed by a CBS module to adjust the number of channels to C in , to generate the final output feature map; The TAPH consists of four parts, namely, a feature extractor, a task decomposition module, a boundary prediction branch and a category prediction branch. The feature extractor uses two Conv_GNs to learn task interaction features from the input of Neck. The feature extractor inputs the learned task interaction features into the task decomposition module. The task decomposition module decomposes the task interaction features into classification prediction features and boundary prediction features through TDM. The classification prediction features and boundary prediction features are used by the classification prediction branch and the boundary prediction branch to calculate the classification score and locate the area respectively. The output feature map uses the Scale layer to adjust the scale of the features after the convolution layer to adapt to the use of shared convolution, and finally obtains the prediction result.

2. According to the lightweight small target detection model for student classroom behavior recognition based on task alignment and heavy parameters in claim 1, Features The TDM facilitates task decomposition by dynamically computing task-specific features at the layer level. Let X inter represents the task interaction features extracted by the feature extractor from Neck, From X inter The features of specific tasks are calculated separately for each classification or positioning task, as shown in Equation 1: ω=σ(fc 2 (δ(fc 1 (x inter )))) (2) where ω k represents the kth element learned from the Layer attention layer, represents the task interaction feature learned by the kth Conv_GN from the Neck input; ω is calculated by the cross-layer task interaction feature, which can capture the dependency between layers. Its calculation formula is shown in Equation 2; σ and δ represent the sigmoid function and the ReLU function respectively, and fc 1 and fc 2 are two fully connected layers, x inter By X inter Perform global average pooling to obtain; each Combine joint features X task , X task Used to calculate the positioning or classification result Z task , as shown in Equation 3, and then the sigmoid function is used to convert Z task Convert to classification prediction feature P or bounding box prediction feature B: Z task =conv 2 (δ(conv 1 (X task ))) (3) P and B are used for classification prediction branch and boundary prediction branch calculation respectively to obtain the final prediction result.

3. The lightweight small target detection model for student classroom behavior recognition based on task alignment and heavy parameters according to claim 2, Features The classification prediction branch uses the spatial probability map C to learn the consistency between the two tasks at each spatial position, as shown in Equation 4, where C is derived from X inter The calculation formula is shown in Formula 5. The element product of P and B is then passed through a convolutional layer to obtain the final classification result P align : C=σ(conv 3 (δ(conv 2 (X inter )))) (5) The boundary prediction branch passes through a convolutional layer from X inter The mask M and spatial offset O are calculated as shown in Equation 6 and Equation 7. O=δ(conv 1 (X inter )) (6) M=σ(δ(conv 1 (X inter ))) (7) M and O are used in deformable convolution to adjust the predicted bounding box at each position. The calculation process is shown in Equation 8: B align =Scale(δ(conv bbox (y))) (10) x and y represent the input and output feature maps respectively, y(p) is the value of the output feature map at position p; w k and O k Respectively represent the weight of the kth position and the offset of the convolution kernel relative to the center point; ΔO k and ΔM k denote the learnable offset and modulation scalar at the kth position, respectively, both of which are obtained by applying a convolutional layer to the feature map x, with the input feature map at the offset position p+O. k +ΔO k The value of is calculated using bilinear interpolation, and the calculation formula is shown in Equation 9, where G is the bilinear interpolation weight, and q represents the difference between p+O and p on the feature map. k +ΔO k The four closest points; the output feature map y first passes through a convolution layer, and then the Scale layer is used to adjust the scale of the features after the convolution layer to adapt to the use of shared convolution, and finally the border prediction result B is obtained align , and its calculation process is shown in formula 10.

4. According to claim 1, the lightweight small target detection model for student classroom behavior recognition based on task alignment and heavy parameters, Features The GDC consists of Ghost Module and dynamic convolution, where: The Ghost Module divides the generation of feature maps into two parts. The original feature map is generated by a standard convolution operation to generate a part of the feature map. In the Ghost feature generation, the depth-separable convolution is used to perform linear transformation on these original feature maps to generate more Ghost feature maps. Finally, the original feature map and the Ghost feature map are combined as the final output feature map. The dynamic convolution is implemented using CondConv. A 1×1 CondConv is used to generate the original feature map, and then a 1×1 depth-separable CondConv is used to generate the Ghost feature map. In CondConv, the convolution kernel parameters are parameterized as a linear combination of multiple expert knowledge, and the calculation formula is shown in Equation 11: Output(x)=σ(α 1 ·W 1 +...+a n ·W n )*x) (11) Where x is the output of the previous layer, n represents the number of experts on this CondConv layer, and α 1 ,…,α n is the weight coefficient learned by gradient descent, W 1 ,…,W n is the convolution kernel representing expert knowledge.

5. A method for detecting student classroom behavior based on task alignment and re-parameterization using the model described in any one of claims 1 to 4, Features The method comprises the following steps: Step 1: Input the image to be detected Input the images containing students’ classroom behaviors into the pre-trained lightweight small object detection model for students’ classroom behavior recognition; Step 2: Feature extraction Use backbone network for feature extraction; Step 3: Feature Fusion In the neck, the features extracted by the backbone network are fused and enhanced to obtain a more powerful feature representation; Step 4: Task interaction feature extraction The task interaction features are calculated from the fused features output by the neck using a feature extractor; Step 5: Task Decomposition The task decomposition module is used to decompose the task interaction features into classification prediction features and boundary prediction features. Calculate classification scores and regional probabilities; Step 6: Positioning and classification The classification prediction features and boundary prediction features are calculated by the classification prediction branch and the boundary prediction branch respectively to obtain the regional probability and classification score; Step 7: Decoding Decode the region probability and classification score and convert them into the final positioning result and classification result; Step 8: Non-maximum suppression The final prediction results are adjusted using non-maximum suppression technology, by retaining the student classroom behavior detection frame with the highest score and suppressing other student classroom behavior detection frames that highly overlap with it, thus eliminating redundant student classroom behavior detection frames; Step 9: Output test results The lightweight small target detection model for student classroom behavior recognition outputs the final detection results. The inspection results include detection boxes, category labels, and confidence scores. The confidence score of each detection box indicates whether the box contains the target and the accuracy of the target category.