A student behavior detection method based on an improved RT-DETR target detection model

Abstract

The application discloses a student behavior detection method based on an improved RT-DETR target detection model, relates to the technical field of computer vision and target detection, and solves the problems that the existing RT-DETR model has low precision in real-time detection in a complex scene and has too large calculation amount and parameter amount of the model, and the like. The application collects student behavior images of a teaching scene, prepares a student behavior data set, constructs an improved RT-DETR model, initializes the improved RT-DETR model for parameter setting and training, obtains an optimal weight file of the model, and detects student behavior of the teaching scene by using the optimal weight file. The application takes YOLOv8 as a backbone network, introduces an additive attention mechanism, constructs a convolutional additive attention module, greatly reduces the calculation complexity of the RT-DETR target detection model on the basis of keeping the original precision, and improves the performance of the target detection network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and object detection, and particularly relates to an improved RT-DETR algorithm design method for recognizing classroom student behavior recognition BACKGROUND

[0002] The learning state of a student can be reflected by the student's classroom behavior performance. Traditional classroom behavior analysis mainly relies on manual observation, and student behavior is analyzed by playing back classroom video records. This method has problems such as low efficiency, strong subjectivity, and large workload, and it is difficult to provide reasonable and effective feedback for teachers. Classroom behavior detection technology based on deep learning has become the mainstream due to its automation, real-time performance, and efficiency.

[0003] Traditional classroom object detection tasks often use traditional computer vision algorithms and machine learning techniques to process and analyze image data. However, these methods have limited adaptability and accuracy in complex classroom scenes with many student objects, and the model performance may decrease significantly due to the increase in the number of objects.

[0004] Real-Time Detection Transformer (RT-DETR) was released in 2023 by Baidu and is a real-time end-to-end object detector based on the DETR architecture. It can provide excellent performance while maintaining high accuracy. It is based on the idea of DETR and introduces a conv-based backbone and an efficient hybrid encoder to achieve faster real-time speed. RT-DETR efficiently handles multi-scale features by decoupling intra-scale interaction and cross-scale fusion. This model has strong adaptability and supports the use of different decoder layers to flexibly adjust the inference speed without retraining. RT-DETR performs well on the accelerated backend and outperforms many other real-time object detectors.

[0005] Although RT-DETR removes the traditional non-maximum suppression step and reduces the computational complexity, its Transformer-based architecture still requires high computational resources. In the classroom scene, due to the large number of students and the large size difference of the objects, the real-time performance of RT-DETR may be limited. In addition, its large parameter quantity also puts high requirements on hardware devices, limiting its application in resource-constrained environments. How to maintain accuracy while reducing the computational burden of RT-DETR is an important technical challenge in the field.

[0006] Therefore, in order to improve the accuracy of the model in the complex classroom scene, an improved RT-DETR object detection model is constructed in the method of the present application, which not only improves the accuracy of the model in the complex scene, but also reduces the computational complexity and parameter quantity of the model. SUMMARY

[0007] The application provides a student behavior detection method based on an improved RT-DETR target detection model to solve the problems of low accuracy, excessive calculation and parameter amount of the model in real-time detection in a complex scene.

[0008] A student behavior detection method based on an improved RT-DETR target detection model, which is realized by the following steps:

[0009] Step one, collect student behavior images in a teaching scene and make a student behavior dataset;

[0010] Step two, build an improved RT-DETR model;

[0011] The improved RT-DETR model comprises a backbone network of YOLOv8, a hybrid encoder and a decoder;

[0012] The backbone network based on YOLOv8 is composed of multiple convolution layers and multiple convolution additive attention modules, and three convolution additive attention modules are used as output channels;

[0013] The hybrid encoder is composed of a dynamic hybrid self-attention mechanism module and a cross-scale feature fusion module;

[0014] The input image data is subjected to local feature extraction by the convolution layer of the backbone network of YOLOv8 and subjected to feature splicing and fusion operation by the convolution additive attention module, and a feature map is output;

[0015] The output channels output feature layer P3, feature layer P4 and feature layer P5 respectively; feature layer P3 and feature layer P4 are input into the cross-scale feature fusion module in the hybrid encoder;

[0016] Feature layer P5 is input into the dynamic hybrid self-attention mechanism module in the hybrid encoder for further feature processing and then input into the cross-scale feature fusion module;

[0017] The cross-scale feature fusion module fuses feature layer P3, feature layer P4 and processed feature layer P5, generates a target bounding box and a confidence score through the decoder, and is used for a target detection task to complete the positioning and classification of the target in the input image;

[0018] Step three, initialize the improved RT-DETR model to set parameters and train the model to obtain an optimal weight file of the model;

[0019] Step four, use the optimal weight file to detect student behavior in a teaching scene.

[0020] The beneficial effects of the present application are:

[0021] The detection method described in the present application greatly reduces the computational complexity of the RT-DETR target detection model while maintaining the original accuracy.

[0022] The detection method described in the present application reduces the algorithm running calculation amount and parameter amount, while effectively improving the recognition accuracy, and realizes the effect of recognizing multiple complex targets in a limited hardware scenario.

[0023] The present application takes YOLOv8 as the backbone network, introduces additive attention mechanism, constructs convolutional additive attention module, and further constructs a new backbone network. In the encoder aspect, the scale-in feature interaction module based on attention in the hybrid encoder of the RT-DETR model is replaced by a dynamic hybrid self-attention module, and Inner-MPDIoU is used as the loss function to improve the performance of the target detection network. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The flowchart of the student behavior detection method based on the improved RT-DETR target detection model described in the present application is shown in the figure.

[0025] Figure 2 The structure diagram of the convolutional additive attention module (CAB) is shown in the figure.

[0026] Figure 3 The structure diagram of the improved YOLOv8 backbone network is shown in the figure.

[0027] Figure 4 The structure diagram of the improved RT-DETR model is shown in the figure.

[0028] Figure 5 The model detection effect diagram is shown in the figure. (a) is the effect diagram of the improved RT-DETR model on the accuracy Precision; (b) is the effect diagram on the recall rate Recall; (c) is the effect diagram on the average precision mean mAP50; (d) is the effect diagram on the average precision mean mAP50-95 in the intersection over union threshold range from 0.5 to 0.95.

[0029] Figure 6 The PR curve diagram comparison effect diagram of the original RT-DETR model and the improved RT-DETR model of the present application is shown in the figure. (a) is the PR curve diagram of the original RT-DETR model; (b) is the PR curve diagram of the improved RT-DETR model of the present application. DETAILED DESCRIPTION

[0030] In combination Figures 1 to 5To illustrate the present embodiment, a student behavior detection method based on an improved RT-DETR target detection model is provided, which is realized by the following steps:

[0031] Step one, collect the data set;

[0032] Step 1-1. Image collection: collect the monitoring overhead view of the student behavior in the teaching scene, and make the student behavior data set;

[0033] Step 1-2. Image annotation: use the image annotation tool (LableMe software) to perform category annotation on the student target behavior of the data set, including eight categories of low head writing, low head reading, high head listening, turning head, raising hand, standing, group discussion, and teacher guidance;

[0034] Step 1-3. File organization: create two folders, one for storing the annotated image files, and the image file format is jpg format, and the other for storing the annotated text files, i.e. the label file; the label file format is txt format. The naming of the image file and the label file both adopts the serial number form, and they are one-to-one corresponding;

[0035] Step 1-4. Format conversion: run the script file to convert the annotated.txt format label file to.xml format for subsequent processing and application;

[0036] Step 1-5. Constructing the training data set: the label file after format conversion and the image file constitute the training data set, which is used for subsequent model training.

[0037] Step two, network structure design; construct an improved RT-DETR target detection model;

[0038] The improved RT-DETR target detection model is composed of three parts of backbone network (BackBone), hybrid encoder (Hybrid Encoder) and decoder (Decoder). The hybrid encoder is composed of dynamic hybrid self-attention module (Dynamic Hybrid Self-Attention Mechanism, DHSA) and cross-scale feature fusion module (CNN-based Cross-scale Feature Fusion Module, CCFM).

[0039] In this embodiment, first, the backbone network in the original model RT-DETR is improved. The improved network structure adopts the backbone network of YOLOv8 to replace the backbone network of the original RT-DETR network. First, the SPPF module of the YOLOv8 backbone network is removed, secondly, the bottleneck layer (Bottleneck) of the C2f module in the YOLOv8 backbone network module is replaced with an additive attention module (AdditiveBlock), and a residual addition operation (R) is used to replace channel concatenation (Concat) to form a convolutional additive attention module (CAB), which is used to replace the three C2f modules in YOLOv8 respectively.

[0040] As shown in Figure 3 In this embodiment, the YOLOv8 backbone network includes nine layers:

[0041] The first layer of convolutional layer (Conv) is composed of a standard convolutional module, which directly receives the input image. The input is an image with a size of 640x640x3, and the output channel number is 64. The convolution kernel size is 3x3, and the stride is 2. Through the convolution operation, the spatial size of the image is reduced by half, and the preliminary local features are extracted.

[0042] The second layer continues to use the convolutional layer, and the output channel number increases to 128. The convolution kernel size is still 3x3, and the stride is 2. This layer again reduces the resolution of the feature map by half, while further extracting deep features.

[0043] The third layer adopts a convolutional additive attention module (CAB);

[0044] The convolutional additive attention module CAB adds an additive attention module (AdditiveBlock) to replace the bottleneck layer (Bottleneck) of the original C2f module in the YOLOv8 backbone network, and modifies the channel concatenation (Concat) operation in the original C2f module to a residual addition operation (R). The structure of the convolutional additive attention module CAB is as shown in Figure 2 As shown in Figure 1Part of the direct transmission to the final residual addition operation block (R), using the residual addition connection to alleviate the gradient vanishing problem, the other part is further processed by the stacked three additive attention modules. The feature map processed by the bottleneck block is spliced with the part of the feature map directly transmitted through the residual addition operation block (R) to form the fused feature map, which is input to the second CBS module, and the feature map is output after 1x1 convolution operation for the next step processing.

[0045] The fourth layer uses a convolutional layer configured to increase the number of output channels to 256, with a convolution kernel size of 3x3 and a stride of 2.

[0046] The fifth layer applies a convolutional additive attention module with an input channel number of 256. Similar to the third layer, it processes the feature map through multiple convolution branches and additive fusion mechanisms.

[0047] The sixth layer of convolutional layer applies a channel number increase to 384, with a convolution kernel size of 3x3 and a stride of 2. This layer continues to downsample the feature map, reducing the resolution from 80x80 to 40x40.

[0048] The seventh layer is a convolutional additive attention module with an input channel number of 384.

[0049] The eighth layer adds a convolutional layer with an applied channel number increase to 384, with a convolution kernel size of 3x3 and a stride of 2.

[0050] The ninth layer is the last layer, which adds a convolutional additive attention module with an input channel number of 384.

[0051] The feature map extracted from the input image data by the backbone network is output to the next network structure through three channels. The first channel is the fifth layer of the backbone network, which is a convolutional additive attention module, outputting a feature map with a size of 80x80x256. The feature map output by this layer is set as feature layer P3. The second channel is the seventh layer of the backbone network, which is a convolutional additive attention module, outputting a feature map with a size of 40x40x384. The feature map output by this layer is set as feature layer P4. The third channel is the ninth layer of the backbone network, which is a convolutional additive attention module, outputting a feature map with a size of 20x20x384. The feature map output by this layer is set as feature layer P5.

[0052] The feature layer P5 is input to the dynamic hybrid self-attention mechanism module in the hybrid encoder for further feature processing and then input to the cross-scale feature fusion module.

[0053] The cross-scale feature fusion module fuses feature maps of the feature layer P3, the feature layer P4 and the processed feature layer P5, and generates a bounding box and a confidence score of a target through a decoder, which is used for a target detection task to complete positioning and classification of the target in the input image.

[0054] Step three, using the script file to divide the data set into training set, validation set and test set in the proportion of 8:1:1, initializing the improved RT-DETR model for parameter setting and training, setting the input image size to 640x640, the training round to 100, the input batch size to 4 each time, using the training set to train the improved RT-DETR model, using the validation set to evaluate the performance of the model in the training process after each training iteration, helping to adjust the hyperparameters of the model and preventing overfitting, and finally obtaining the optimal weight file best.pt of the improved RT-DETR model after training.

[0055] As shown in Figure 4 During the training process, the image with a size of 640x640 is input into the backbone network of the improved RT-DETR model, that is, the backbone network of YOLOv8. First, the first convolutional layer outputs a feature with a size of 320x320x64, which is input into the second convolutional layer. This layer again reduces the resolution of the feature map by half, outputs a feature with a size of 160x160x128, and is input into the next layer. The feature input into the third layer is processed by the CAB module for feature fusion, and the dimension remains unchanged, which is input into the fourth layer. After the convolutional layer processing, the output dimension is 80x80x256, which is output to the fifth layer. After the fifth layer CAB module feature fusion operation, the processed feature dimension size remains unchanged, which is 80x80x256, which is output to the sixth layer. After the convolutional operation, the output dimension size is 40x40x384, which is output to the seventh layer. After the CAB module feature fusion operation, the dimension size remains unchanged, which is still 40x40x384, which is output to the eighth layer. After the convolutional operation, the feature with a dimension size of 20x20x384 is obtained, which is output to the ninth layer. After the CAB module feature fusion operation, the output dimension size remains unchanged, which is 20x20x384. The feature maps output by the fifth layer CAB module, the seventh layer CAB module and the ninth layer CAB module are extracted and labeled as feature layer P3, feature layer P4 and feature layer P5 respectively, and the features output by the three channels are input into the mixed encoder. The feature layer P3 and the feature layer P4 are directly input into the cross-scale feature fusion module (CCFM) in the mixed encoder through two channels, and the feature layer P5 is input into the DHSA module of the model.

[0056] The DHSA module in this embodiment is an improved attention mechanism that replaces the original attention-based intra-scale feature interaction module (AIFI) in the hybrid encoder. In the dynamic hybrid self-attention mechanism module, first, the input feature map is split into two parts: one part is directly passed to the R block to preserve the original feature information; the other part enters the additive attention module to extract important features. In the additive attention module, the feature map is further processed, and each attention head is treated as an expert, with the Top-K heads being selected for activation through a dynamic routing algorithm. Assuming that the number of selected heads is K, the output of each head can be represented as:

[0057]

[0058] where Q, K, and V are the query, key, and value matrices extracted from the input features, d k is the dimension of each head. The output F add of the additive attention module is subjected to a residual addition operation with the directly passed feature F res to enhance the robustness of the feature:

[0059] F out =F add +F res

[0060] The size of the final feature map F out is still 20x20x384, maintaining the same number of channels as the input. The feature map F out after the residual addition operation is further subjected to convolution operations to adjust the number of channels and output for further processing. Through this dynamic hybrid self-attention mechanism, the model can efficiently process features while enhancing its adaptability to different input data.

[0061] The processed feature layer P5 is input into the CCFM module of the hybrid encoder along with the feature layers P3 and P4. This module uses a designed feature pyramid network architecture to fuse feature maps from different scales to enhance the model's detection ability for multi-scale targets. The CCFM module adopts a bidirectional fusion strategy from top to bottom and bottom to top. The final fused feature map not only retains the spatial details of the low-level feature map but also fuses the semantic information of the high-level feature map. This effectively integrates feature information from different scales and enhances the model's detection ability for multi-scale targets.

[0062] The feature map processed by the CCFM module enters the decoder. In the embodiment, the decoder selects the top K features from the feature sequence output by the decoder as the initial object query through the IoU-aware query selection mechanism. The IoU-aware query selection mechanism imposes constraints on the model during training, so that the model produces high classification scores for features with high IoU scores and low classification scores for features with low IoU scores. The decoder optimizes the object query through multiple rounds of iteration to generate the bounding box and confidence score of the target, which is used for the target detection task to complete the positioning and classification of the target in the input image.

[0063] In the embodiment, the improved RT-DETR network can significantly improve the performance of the model. The YOLOv8 backbone network enhances the detection capability of multi-scale targets by optimizing feature extraction, especially in small target detection. Its efficient network design reduces the computational complexity and improves the real-time performance. The convolutional additive attention module composed of the C2f module and the additive attention module further strengthens the integration of context information and improves the understanding ability of the model in complex scenes. In addition, this combination simplifies the training process and reduces the complexity of hyperparameter adjustment, making the RT-DETR more suitable for real-time applications while maintaining high performance.

[0064] In the embodiment, the AIFI module in the hybrid encoder in the original RT-DETR model is replaced by the DHSA module, and the parameters are set as follows: the input channel number is set to 384 to match the channel number of the input feature map; the output channel number is set to 256 to ensure consistency with the channel number of the subsequent modules of the model; the hidden dimension is also set to 256 for internal feature processing; the number of attention heads is set to 8 to capture features in different subspaces using the multi-head self-attention mechanism; the activation function uses GELU to provide non-linear activation ability for the model; the Dropout rate is set to 0.0, which can be adjusted according to the regularization needs of the model.

[0065] In the training process, the original RT-DETR model loss function GIoU is replaced by the Inner-MPDIoU loss function.

[0066] Inner-MPDIoU is a loss function that considers both the intersection over union (IoU) and the multi-point distance of the bounding box. It optimizes the loss calculation of the bounding box regression by introducing auxiliary bounding boxes and multi-point distances. To better capture the internal structure of the target, auxiliary bounding boxes are introduced.

[0067] The auxiliary bounding box is obtained by scaling the real bounding box, and the width and height of the auxiliary bounding box are the width and height of the real box multiplied by a scaling ratio:

[0068] w inner =w gt ×ratio,h inner =h gt ×ratio

[0069] where w inner and h inner are the width and height of the auxiliary bounding box, w gt and h gt are the width and height of the real bounding box, and ratio is the scaling ratio, usually in the range (0, 1); the center point of the auxiliary bounding box is the same as that of the real bounding box, so its coordinates can be represented as: (x inner ,y inner ) = (x gt ,y gt ) ;

[0070] where (x inner ,y inner ) is the center point coordinate of the auxiliary bounding box, and (x gt ,y gt ) is the center point coordinate of the real bounding box.

[0071] In this embodiment, in order to further optimize the regression of the bounding box, the concept of multi-point distance is introduced. Specifically, the distance between the vertices of the predicted box and the real box is calculated. Assuming that the vertex coordinates of the predicted box B pred : top left corner (x1 pred , y1 pred ), bottom right corner (x2 pred , y2 pred ); real box B gt : top left corner (x1 gt , y1 gt ), bottom right corner (x2 gt , y2 gt ); the distance between the two vertices is defined as:

[0072] d1 2 =(x1 pred -x1 gt ) 2 +(y1 pred -y1 gt ) 2

[0073] d2 2 =(x2 pred -x2gt ) 2 +(y2 pred -y2 gt ) 2

[0074] where d1 and d2 represent the distances between the top-left corners and the bottom-right corners of the predicted and real bounding boxes, respectively.

[0075] Combining IoU and multi-point distance, the Inner-MPD IoU loss function is defined as follows:

[0076]

[0077] where W and H are the width and height of the input image, respectively, and α is a weight parameter to balance the contributions of IoU and multi-point distance, usually taking a value in the range (0, 1). IoU is the traditional intersection over union, and the calculation formula is:

[0078]

[0079] where the Area() function specifically calculates the area of the bounding box or.

[0080] In this embodiment, Inner-MPDIoU optimizes the bounding box regression accuracy by introducing auxiliary bounding boxes and vertex distance calculation, especially in complex target detection. This method can quickly converge, adapt to different target sizes, and is easy to implement. Experiments show that the model using Inner-MPDIoU is significantly better than traditional methods in bounding box regression accuracy, effectively improving the target detection performance.

[0081] Step four, using the test set to evaluate the training results of the improved RT-DETR model; load the weight file best.pt as the new weight file of the improved RT-DETR model, and detect new images based on the weight file for student behavior recognition.

[0082] As shown in Table 1, Table 1 is the performance comparison of the improved RT-DETR model and the original model of the application.

[0083] Table 1

[0084]

[0085] In Table 1, 1. Precision refers to the proportion of correct positive samples identified by the model to the total number of samples identified as positive samples. Precision focuses on the proportion of true positives in the predicted positive class. The formula can be expressed as:

[0086]

[0087] Where TP represents the number of correctly identified positive samples, and FP represents the number of incorrectly identified positive samples.

[0088] 2. Recall refers to the proportion of true positive samples identified by the model to the total number of true positive samples. It is often used in conjunction with Precision. Recall focuses on the proportion of all actual positive classes that are correctly predicted. The formula can be expressed as:

[0089]

[0090] Where FN represents the number of incorrectly identified negative samples.

[0091] 3. Mean Average Precision (mAP50, mAP@0.5, Mean Average Precision) is used to evaluate the overall accuracy of multi-class detection tasks. The average precision (AP) of each class is determined by drawing its precision-recall curve, calculating the area under the curve, and representing the AP of the class. Finally, mAP is obtained by averaging the AP of all classes, which is a comprehensive indicator of model detection performance. The formula can be expressed as:

[0092]

[0093] Where N represents the total number of classes, and AP_i represents the average precision of the i-th class.

[0094] 4. Mean Average Precision (mAP50-95) of the model in the Intersection over Union (IoU) threshold range from 0.5 to 0.95: It is an indicator for evaluating the performance of target detection models, representing the average value of the average precision calculated in the IoU range from 0.5 to 0.95. Unlike mAP50, mAP50-95 considers a wider range of IoU, which allows the evaluation of model performance at different overlap levels.

[0095] 5. Parameter amount: used to evaluate the size and complexity of the model, obtained by summing the number of weight parameters in each layer, is an important indicator to measure the complexity of the model, and low parameters are better.

[0096] 6. Computational amount: an indicator to measure the computational complexity, representing the number of floating-point operations. It reflects the computational amount of the model during runtime, and is an important indicator to evaluate the operation speed of the model. Lower GFLOPs value means that the model can complete the inference task faster in practical application, improving real-time performance.

[0097] Combining Table 1 and Figure 5As shown, by comparing the performance of the original RT-DETR, the improved RT-DETR model, YOLOv8m, YOLOv5m, and Faster-RCNN, it can be clearly seen that the improved model has significant advantages. Figure 5 In (a), the improved RT-DETR model achieved a Precision of 0.8809, which is the highest among all models, indicating that it has the highest accuracy in predicting positive samples and can more effectively reduce false positives. Figure 5 In (b), its Recall is 0.86999, second only to YOLOv8m, indicating that it performs well in identifying actual positive samples. More importantly, in (c), Figure 5 In (c), the improved RT-DETR model achieved a mAP50 of 0.91687, which is the highest among all models, indicating that it has the best overall performance in the target detection task. In (d), Figure 5 In (d), mAP50-95 reached 0.77325, which is better than the original RT-DETR model, indicating that it has higher comprehensive detection accuracy within the stricter IoU threshold range (0.5-0.95), especially in positioning accuracy (such as bounding box fitting).

[0098] In terms of computational efficiency, the improved model also performs well. Its computational complexity is only 48.9 GFLOPs, which is the lowest among all models, indicating that it requires the least computational resources during runtime and runs faster. In addition, the parameter amount of the improved model is 19347120, which is also the lowest among all models, indicating that its model structure is more lightweight and more suitable for deployment on devices with limited resources. In contrast, the computational complexity and parameter amount of YOLOv8m and Faster-RCNN are significantly higher than those of the improved model, while YOLOv5m has slightly lower accuracy than the improved model.

[0099] In summary, the improved RT-DETR model described in this embodiment significantly reduces the computational complexity and parameter amount while maintaining high accuracy, achieving a perfect balance between performance and efficiency. It not only outperforms other models in key performance indicators such as precision, recall, and mAP50, but also performs well in computational efficiency and model lightweight, making it the best choice for target detection tasks.

[0100] As shown in Figure 6 When comparing the PR curve of the improved RT-DETR model with the original RT-DETR model, the improved RT-DETR model achieved an average precision mean (mAP@0.5) of 0.916, which is 0.001 higher than the original model.

[0101] Specifically, the PR curve chart shows the precision performance of the model at different recall rates. The curve of the improved model is significantly higher than that of the original model in the high recall rate area, which indicates that the improved model can effectively maintain a high precision rate while striving to recall more positive samples. The improved model performs better in reducing false positives, thereby improving the comprehensive indicator mAP@0.5.

[0102] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0103] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application patent should be subject to the appended claims.

Claims

1. A student behavior detection method based on an improved RT-DETR target detection model, characterized by: The method is realized by the following steps: Step one, collect student behavior images in teaching scenes, and make a student behavior dataset; Step two, build an improved RT-DETR model; The improved RT-DETR model includes a YOLOv8-based backbone network, a hybrid encoder, and a decoder; The YOLOv8-based backbone network is composed of multiple convolution layers and multiple convolution additive attention modules, and three of them are used as output channels; The hybrid encoder is composed of a dynamic hybrid self-attention mechanism module and a cross-scale feature fusion module; The input image data is subjected to local feature extraction by the convolution layer of the YOLOv8 backbone network, and feature splicing and fusion operations by the convolution additive attention module, outputting a feature map; The output channel outputs feature layer P3, feature layer P4, and feature layer P5 respectively; Feature layer P3 and feature layer P4 are input into the cross-scale feature fusion module in the hybrid encoder; Feature layer P5 is input into the dynamic hybrid self-attention mechanism module in the hybrid encoder for further feature processing and then input into the cross-scale feature fusion module; The cross-scale feature fusion module fuses feature layer P3, feature layer P4, and processed feature layer P5, and generates the bounding box and confidence score of the target through the decoder, which is used for target detection tasks to complete the positioning and classification of the target in the input image; Step three, initialize the improved RT-DETR model for parameter setting and training to obtain the optimal weight file of the model; Step four, use the optimal weight file to detect student behavior in teaching scenes.

2. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: The specific process of step one is as follows: Step one, collect teaching scene monitoring overhead view as dataset; Step two, use image annotation tools to label student target behavior categories for the dataset; Step three, store the labeled image files and label files in two folders respectively, and use one-to-one serial numbers for folder naming; Step four, convert the format of the label file to.xml format; Step five, use the image file and the converted label file as the dataset.

3. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: In step two, the YOLOv8-based backbone network includes nine layers; the first and second layers are convolution layers for extracting local features of the input image; the third layer is a convolution additive attention module for fusing the local features extracted by the convolution layer to obtain a fused feature map; the fourth to ninth layers are arranged with convolution layers and convolution additive attention modules, and the fifth, seventh, and ninth layers are used as output channels to output feature layer P3, feature layer P4, and feature layer P5 respectively.

4. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: Each convolution additive attention module is composed of two CBS modules and three additive attention modules; The feature map is subjected to 1x1 convolution processing through the first CBS module, and then subjected to feature map splitting operation, part of the feature map after splitting is transmitted to the residual addition operation block, and the other part of the feature map is further processed through the stacked three additive attention modules; the processed feature map is spliced with the feature map output by the residual addition operation block to form a fused feature map, and the fused feature map is input into the 1x1 convolution operation of the second CBS module and then output.

5. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: In step two, the feature layer P5 is input into the dynamic mixed self-attention mechanism module, and the input feature map is split into two parts: one part of the feature map is transmitted to the residual addition operation block, and the other part of the feature map is input into the additive attention module; The feature map is processed through the additive attention module, and the output feature map is subjected to residual addition operation with the feature map output by the residual addition operation block to output the final feature map; and the feature map is subjected to further convolution operation and then output.

6. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: The decoder selects the top K features in the feature sequence output by the decoder as the initial object query through the IoU-aware query selection mechanism; the IoU-aware query selection mechanism applies constraints to the model during training, so that the model produces high classification scores for features with high IoU scores and low classification scores for features with low IoU scores; the decoder optimizes the object query through multiple rounds of iteration to generate the bounding box and confidence score of the target, which are used for the target detection task.

7. The student behavior detection method based on the improved RT-DETR target detection model according to claim 1, characterized in that: In step three, the improved RT-DETR target detection model is trained, and an Inner-MPD IoU loss function is set to introduce auxiliary bounding boxes and multi-point distances to optimize the loss calculation of bounding box regression.

8. The student behavior detection method based on the improved RT-DETR target detection model according to claim 7, characterized in that: The auxiliary bounding box is obtained by scaling the real bounding box, and the width and height of the auxiliary bounding box are respectively the width and height of the real bounding box multiplied by a scaling ratio, as follows: ， ， wherein, and are the width and height of the auxiliary bounding box, respectively, and are the width and height of the real bounding box, respectively, is the scaling ratio; The center point of the auxiliary bounding box is the same as that of the real bounding box, and the coordinates are represented as: ; wherein, is the center point coordinate of the auxiliary bounding box, is the center point coordinate of the real bounding box; The distance between the vertices of the predicted bounding box and the real bounding box is calculated by the multi-point distance calculation. The vertex coordinates of the predicted bounding box and the real bounding box are set as follows: the vertex coordinates of the predicted bounding box : upper left corner coordinate , right lower corner coordinate ; the vertex coordinates of the real bounding box : upper left corner coordinate , right lower corner coordinate ; The distance between the two vertices is defined as: ， ， In the formula, is the distance between the left upper corner of the predicted bounding box and the left upper corner of the real bounding box; is the distance between the right lower corner of the predicted bounding box and the right lower corner of the real bounding box. The Inner-MPD IoU loss function is defined as follows: ， wherein: and are the width and height of the input image, respectively, is a weight parameter balancing the contribution of and the multi-point distance, is the intersection over union.

Images