A student classroom behavior detection method based on transformer and task dynamic alignment

Abstract

The application provides a student classroom behavior detection method based on a Transformer and task dynamic alignment. On the one hand, the method adds GLTB to the backbone network to enhance feature extraction capability. Among them, CGLU is used to enhance the nonlinear feature expression capability, and the global feature extraction capability of MHSA is combined, so that the method has stronger feature extraction capability. On the other hand, a neck structure network improved by GLSA is adopted, which can not only effectively extract global features, but also capture fine-grained local features, and can promote the interaction between global and local information. In this way, when dealing with complex tasks, the method can more accurately identify and understand the key information in the image. In addition, a new detection head ETADH is adopted, which learns the interaction features of the classification and positioning tasks through a feature extractor and performs task decomposition through a task decomposition module. In this way, the interaction between the two tasks is enhanced, and the related information between them is effectively utilized to improve the detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and computer vision, and relates to the technical field of classroom behavior detection, in particular to deep learning, single-stage target detection and the like, and more particularly to a student classroom behavior detection method based on Transformer and task dynamic alignment. BACKGROUND

[0002] In recent years, with the development of behavior detection technology, it has become possible to analyze the behavior of students in the classroom video and obtain information about the students' classroom state and learning performance. This technology is very important for teachers, administrators, students and parents in schools. However, in the traditional teaching mode, it is difficult for teachers to pay attention to the learning situation of each student, and they can only observe a few students to understand the effectiveness of their teaching methods. School administrators rely on on-site observation and student performance reports to find problems in education and teaching. Parents can only understand their children's learning situation through communication with teachers and students. Therefore, using behavior detection technology to accurately detect students' behavior and analyze their learning situation and performance can provide more comprehensive and accurate feedback for education and teaching.

[0003] Algorithms based on pose estimation represent human behavior by obtaining the position and motion information of each joint of the human body, but are not suitable for behavior detection in crowded classrooms. Considering the challenges faced at present, algorithms based on target detection provide a promising solution. In fact, great breakthroughs have been made in algorithms based on target detection in recent years. Therefore, in this paper, algorithms based on deep learning target detection are used to analyze the behavior of students.

[0004] In terms of target detection, two-stage and one-stage target detection frameworks have received more attention due to their impressive detection results on public datasets. However, the dataset of real classrooms is very different from that of public classrooms, and the performance of classic methods in real classrooms is not good. One of the representative problems is the large-scale variation between different positions, for example, students in the front row of the classroom are about 40x40 pixels, while students in the back row are about 200x200 pixels, which results in a high scale variation of nearly 25 times. Worse still, compared with the most popular target detection dataset MSCOCO, the occlusion between students is very serious. Moreover, the behavior of raising hands varies greatly in different environments, different people and different angles.

[0005] Therefore, there is a need for a new method of student classroom behavior target detection that can solve the above problems. SUMMARY

[0006] The purpose of the present application is to solve the problem of low accuracy and poor detection effect of small targets caused by image target occlusion and inconsistent object size of existing methods. In view of the above shortcomings, a student classroom behavior detection method based on Transformer and task dynamic alignment is proposed. Experimental results prove that the method of the present application is more superior than the prior art, and meets the current needs of primary and secondary school student classroom behavior detection.

[0007] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0008] 1) Obtain monitoring videos of students in different grades in classroom scenes, including classroom pictures of primary and secondary school students of the same age, frame the obtained videos, and obtain student classroom behavior pictures.

[0009] 2) Label all student classroom behavior pictures using the labelImg image labeling tool to obtain a student classroom behavior dataset. The dataset contains three categories: "reading", "hand-raising", and "writing". In subsequent experiments, it will be found that "reading" and "writing" have the problem of overlapping bounding boxes, and "reading" and "writing" behaviors are similar in vision.

[0010] 3) Data preprocessing. All available labeled data is divided into two parts, 80% of the images and label files are taken as the training set, and the other 20% is taken as the test set.

[0011] 4) Model establishment. Refer to the YOLOv8 backbone network, the model is mainly divided into three parts: the main network adds GLTB (Global-to-Local Transformer Block) to enhance the feature extraction capability, the feature fusion network is the improved neck network of GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Align Detection Head) is used in the final prediction head part. The main network is the basic part of the model, responsible for extracting key local and global features from the input image. After obtaining the preliminary features, the neck structure of the model further processes these features to enrich the multi-scale feature representation. The neck fuses information from different scales to generate more detailed and context-aware feature maps. Finally, the features extracted and fused by the main network and the neck are passed to the head structure, which is responsible for specific prediction tasks, including classification tasks and regression tasks.

[0012] 5) Data augmentation. A variety of data augmentation techniques are applied, including mosaic augmentation, mixup augmentation, random perturbation, and color perturbation. Data augmentation helps improve the generalization ability and robustness of the model.

[0013] 6) Input the training set images and label data in the dataset into the established model, set fixed parameters, determine the optimizer and loss function, and obtain the trained student classroom behavior detection model. Through AMP (Automatic Mixed Precision) training technology, different precision floating-point numbers are mixed to optimize the training process of the deep learning model, greatly improving the training efficiency and resource utilization, while ensuring the stability of the model performance.

[0014] 7) Use the backbone network with GLTB to extract features from the input image, thereby obtaining key local and global features.

[0015] 8) The neck network improved by GLSA further processes the features after obtaining the preliminary features, and performs feature fusion to enrich the multi-scale feature representation.

[0016] 9) Use the new detection head ETADH module to receive the features extracted and fused by the backbone and neck, and perform the prediction task.

[0017] 10) Input the images and label data in the test set into the trained model to classify and locate the student classroom behavior in each image, and obtain the corresponding prediction label and position information. Then, compare the prediction results with the actual label to obtain the detection results, and calculate the mean average precision (mAP) and other indicators.

[0018] 11) Input the test image into the trained model to output the student classroom behavior detection results in the test image, and highlight the detection results with colored frames and labels.

[0019] Specifically, the step 1) comprises:

[0020] The images come from different shooting angles, including front, side and back. These angles significantly affect the visual features of the student's hand-raising behavior, increasing the difficulty of the detection task.

[0021] In addition, the classroom environment and seat arrangement may vary from course to course, adding more complexity to the detection and recognition of hand-raising behavior. At the same time, there are obvious differences in hand-raising behavior among students at different learning stages, further increasing the challenge of detection.

[0022] Further, the step 2) comprises:

[0023] Specifically, labelImg is an open-source image annotation tool. During the annotation process, the student's classroom behavior area in each image is manually framed, and each behavior area is assigned a corresponding label, such as "reading", "hand-raising", and "writing".

[0024] After annotation, labelImg automatically generates a label information file corresponding to each image and saves these label information in text format (txt file). Each txt file is named consistently with the image file, facilitating subsequent reading and processing.

[0025] In this way, all student classroom behavior images are systematically annotated, ultimately forming a complete student classroom behavior dataset.

[0026] Further, the step 4) comprises:

[0027] The student classroom behavior detection deep learning network mainly consists of three parts: a backbone network improved by GLTB (Global-to-Local Transformer Block), a feature fusion network improved by GLSA (Global-to-Local Spatial Aggregation), and a new detection head ETADH (Efficient Task Align Detection Head).

[0028] In order to improve the detection speed and recognition accuracy, the original YOLOv8 backbone network is modified. Since the original network has limited effect on global feature extraction, the more advanced GLTB based on Transformer is used to replace the CNN based feature extraction module in the 7th and 9th layers of the original backbone. Specifically, GLTB is added to the backbone network as a feature extraction structure, thereby enhancing the global and local feature extraction capabilities. Due to the high computational complexity of the Transformer structure, applying it directly to all channels will result in high computational cost. To solve this problem and reduce the computational burden while maintaining efficient feature extraction, a hybrid structure is designed. This structure divides the input feature map into two parts, which are processed by convolutional neural network (CNN) and Transformer respectively. By combining the local feature extraction capability of CNN and the global feature extraction capability of Transformer, this design enhances the feature extraction effect. Specifically, the input channels are partially divided and processed by CNN and Transformer respectively to achieve more efficient feature extraction. In the Transformer module, EFEB (Efficient Feature Extraction Block) is used, which contains MHSA (Multi-Head Self-Attention) and CGLU (Convolutional Gated Linear Unit). CGLU is used to enhance the nonlinear feature expression capability, and combined with the global feature extraction capability of MHSA, it has stronger feature extraction capability.

[0029] At the same time, in order to avoid a significant increase in computational complexity when introducing the Transformer structure to extract global features, the Transformer Block is only applied to part of the channels.

[0030] The function of the neck network is to integrate features of different scales extracted from the backbone network. Specifically, a GLSA improved neck structure network is used to fuse semantic and positional features.

[0031] The original YOLOv8 neck network enhances information related to the optimization target while suppressing irrelevant information. To capture global and local spatial features, the GLSA module is proposed, which fuses the results of independent local and global attention units. This dual-stream design of GLSA effectively preserves local and global modeling capabilities. In addition, channel separation is used to balance accuracy and computational resources. Specifically, the feature map Γ i |i∈(2,3,4), is evenly divided into two groups of feature maps are input into the global spatial attention (GSA) module and the local spatial attention (LSA) module, respectively. Finally, the outputs of the two attention units are fused through a 1x1 convolutional layer. This process is represented as:

[0032]

[0033] where G sa is the global spatial attention, L sa is the local spatial attention. is the output feature.

[0034] In ETADH, a task-aligned structure is adopted to address the complex recognition tasks in a classroom environment. In such an environment, students in the back rows often exhibit blurry visual features, making these areas in the image difficult to detect. Traditional detectors typically use independent classification and localization branches when dealing with such issues, which results in insufficient interaction between the two tasks and fails to effectively utilize the associated information to improve detection accuracy.

[0035] To address this issue, ETADH designs a task-aligned custom structure. Specifically, the feature extractor learns the interactive features between tasks from multiple convolutional layers to generate joint features. These joint features contain mutual influence information between the classification and localization tasks, allowing the two tasks to work together to improve detection performance.

[0036] In the localization branch, ETADH introduces DCNv2 and interactive features to generate the offset and mask of DCNv2. DCNv2 is adaptive and can dynamically adjust the position of the convolution kernel based on the input features, making the network more flexible when dealing with complex scenes, especially when dealing with different student poses and positions, allowing for more accurate localization.

[0037] Meanwhile, in the classification branch, ETADH uses interactive features for dynamic feature selection. In this way, the classification task can optimize feature selection and processing based on localization information, improving classification accuracy. This inter-task collaboration not only improves the overall performance of the detection head but also enhances the model's robustness in complex classroom environments.

[0038] By introducing shared convolution and GroupNorm, the number of parameters of the model is significantly reduced, making the overall model more lightweight. Such design is particularly suitable for running on resource-constrained devices, such as mobile devices, edge computing devices, or embedded systems, etc. In these environments, computing resources and memory are usually very limited, so it is crucial to reduce the parameter quantity and computational demand of the model. Shared convolution avoids repeated computation by sharing convolution operations between different detection heads, not only reducing the number of parameters, but also reducing the computational cost. This simplified structure improves the inference speed of the model, enabling efficient real-time detection on low-power devices.

[0039] However, while using shared convolution, one challenge is that each detection head needs to handle targets of different scales. This inconsistency in target scale can affect detection performance, as shared convolution may not be able to adapt to features of different scales. Therefore, to address this issue, the Scale layer is introduced in the design. The Scale layer is used to scale the features, ensuring that each detection head can obtain appropriate feature representations when handling targets of different scales. In this way, the model can more flexibly cope with various target scales, thereby improving detection accuracy.

[0040] Further, the step 5) comprises:

[0041] The main steps of data augmentation include:

[0042] 1. Randomly read four pictures from the dataset each time.

[0043] 2. Flip the four pictures respectively, for example, flip the original picture left and right, scale, for example, scale the size of the original picture, color gamut change, for example, change the brightness, saturation, and hue of the original picture. After the operation is completed, then place the original picture in the upper left, the second picture in the lower left, the third picture in the lower right, and the fourth picture in the upper right.

[0044] 3. Perform image combination and frame combination. After completing the placement of the four pictures, use the matrix method to cut the fixed area of the four pictures, and then splice them together to form a new picture containing a series of frames and other content. The data augmentation method refers to the CutMix data augmentation method, which is an improved version of the CutMix data augmentation method. CutMix algorithm uses two pictures for splicing, while here four pictures are used for splicing. This method greatly enriches the background of the detected objects, and when calculating the standard BN, the data of four pictures is calculated at once. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 The method flowchart of the present application.

[0047] Figure 2 The model framework diagram of the present application.

[0048] Figure 3 The Global-to-Local Transformer Block (GLTB) structure diagram in the backbone network in the experiment of the present application.

[0049] Figure 4 The Efficient Task Align Detection Head (ETADH) structure diagram used in the experiment of the present application

[0050] Figure 5 The performance comparison diagram of the present application and other methods on the student classroom behavior detection data set.

[0051] Figure 6 The performance comparison diagram of different variants of the present application on student classroom behavior detection.

[0052] Figure 7 The part of the prediction result diagram obtained by the present application for predicting the student classroom behavior detection picture DETAILED DESCRIPTION

[0053] The purpose of the present application is to solve the problem of low accuracy and poor small target detection effect caused by image target occlusion and object scale inconsistency in the prior art. In view of the above problems, a student classroom behavior detection method based on Transformer and task dynamic alignment is proposed. The experimental results prove that the method of the present application is more superior than the prior art, and meets the needs of current primary and secondary school student classroom behavior detection.

[0054] To achieve the above purpose, the technical solutions adopted by the present application are as follows, the main process of the method is as shown in Figure 1

[0055] 1) Obtain monitoring videos of students in different grades in classroom scenes, including classroom pictures of primary and secondary school students of different ages, frame the obtained videos to obtain student classroom behavior pictures.

[0056] ​2) Use labelImg image labeling tool to label all student classroom behavior pictures to obtain student classroom behavior dataset. It contains three categories: "reading", "hand-raising", and "writing". In the subsequent experiment, it will be found that "reading" and "writing" have the problem of overlapping bounding boxes, and "reading" and "writing" behaviors are similar in vision.

[0057] 3) Data preprocessing. Divide all available labeled data into two parts, take 80% of the images and label files as the training set, and the other 20% as the test set.

[0058] 4) Model building. Refer to YOLOv8 backbone network, which has three scales of feature maps: 80x80x256, 40x40x512, and 20x20x1024. Take the feature map with a feature scale of 80x80x256 as input. The model mainly consists of three parts: the main network adds GLTB (Global-to-Local Transformer Block) to enhance feature extraction capability, the feature fusion network is a neck network improved by GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Align Detection Head) is used in the final prediction head. The backbone network is the basic part of the model, responsible for extracting key local and global features from the input image. After obtaining the preliminary features, the neck structure of the model further processes these features to enrich the multi-scale feature representation. The neck fuses information from different scales to generate more detailed and context-aware feature maps. Finally, the features extracted and fused by the backbone and neck are passed to the head structure, which is responsible for specific prediction tasks, including classification and regression tasks.

[0059] 5) Data augmentation. Apply various data augmentation techniques, including mosaic augmentation, mixed augmentation, random disturbance, and color disturbance. Data augmentation helps improve the generalization ability and robustness of the model.

[0060] 6) Input the training set images and label data in the dataset into the established model, set fixed parameters, determine the optimizer and loss function, and obtain the trained student classroom behavior detection model. Through AMP (Automatic Mixed Precision) training technology, mixed use of different precision floating-point numbers is used to optimize the training process of deep learning models, greatly improving the training efficiency and resource utilization, while ensuring the stability of the model performance.

[0061] 7) Feature extraction from the input images using the backbone network with GLTB added, resulting in key local and global features.

[0062] 8) A neck network improved by GLSA (Global-to-Local Spatial Aggregation), which further processes the features obtained after the preliminary feature extraction, and fuses them to enrich the multi-scale feature representation.

[0063] 9) Use of the new detection head ETADH module to receive the features extracted and fused by the backbone and neck, and perform the prediction task.

[0064] 10) Input the image and label data in the test set into the trained model to classify and locate the student classroom behavior in each picture, and obtain the corresponding prediction label and position information. Then, compare the prediction result with the actual label to obtain the detection result, and calculate the average precision mean (mAP) and other indicators.

[0065] 11) Input the test image into the trained model to output the student classroom behavior detection result in the test image, and highlight the detection result with colored frames and labels.

[0066] Specifically, the step 1) comprises:

[0067] The images come from different shooting angles, including front, side and back. These angles significantly affect the visual features of the student's hand-raising behavior, increasing the difficulty of the detection task.

[0068] In addition, the classroom environment and seating arrangement may vary from course to course, adding more complexity to the detection and recognition of hand-raising behavior. At the same time, there are obvious differences in hand-raising behavior among students at different learning stages, further increasing the challenge of detection.

[0069] Further, the step 2) comprises:

[0070] Specifically, labelImg is an open-source image annotation tool. During the annotation process, the student's classroom behavior area in each image is manually framed, and each behavior area is assigned a corresponding label, such as "reading", "hand-raising", "writing".

[0071] After annotation, labelImg will automatically generate a label information file corresponding to each image, and save these label information in text format (txt file). Each txt file is named consistently with the image file, facilitating subsequent reading and processing.

[0072] In this way, all student classroom behavior images are systematically labeled, and finally a complete student classroom behavior dataset is formed.

[0073] Further, the step 4) comprises:

[0074] The student classroom behavior detection deep learning network mainly consists of three parts: a backbone network improved by GLTB (Global-to-Local Transformer Block), a neck network improved by GLSA (Global-to-Local Spatial Aggregation) for feature fusion, and a new detection head ETADH (Efficient Task Align Detection Head), as shown in Figure 2 .

[0075] In order to improve the detection speed and recognition accuracy, the original YOLOv8 backbone network is modified. Since the original network structure has limited effect on global feature extraction, a more advanced GLTB based on Transformer is used to replace the feature extraction modules in the 7th and 9th layers of the original backbone. Specifically, GLTB is added to the backbone network as a feature extraction structure to enhance global and local feature extraction capabilities. Due to the high computational complexity of the Transformer structure, directly applying it to all channels will result in high computational cost. To solve this problem and reduce the computational burden while maintaining efficient feature extraction, a hybrid structure is designed. This structure divides the input feature map into two parts, which are processed by convolutional neural networks and Transformers respectively. By combining the local feature extraction ability of CNN and the global modeling ability of Transformer, this design enhances the feature extraction effect. Specifically, the input channels are partially divided and processed by CNN and Transformer respectively to achieve more efficient feature extraction. The TransformerBlock uses EFEB (Efficient Feature Extraction Block), which contains MHSA (Multi-Head Self-Attention) and CGLU (Convolutional Gated Linear Unit) to enhance the nonlinear feature expression ability of CGLU and the global feature extraction ability of MHSA, making it have stronger feature extraction ability. The module framework of the application is shown in Figure 3 .

[0076] At the same time, in order to avoid a substantial increase in computational complexity when introducing the Transformer structure to extract global features, the Transformer Block is only applied to part of the channels.

[0077] MHSA (Multi-Head Self-Attention) is a key component of the Transformer model, used to capture complex relationships in sequence data. It uses a self-attention mechanism to compute the relationships between elements in a sequence in parallel, and improves the model's expressive power by setting multiple attention heads.

[0078] The core idea of the Self-Attention mechanism is to calculate the correlation between each element in the sequence and other elements. For each element in a sequence, the model will reweight and combine all other elements based on their relationship with the element.

[0079] Specifically, for an input sequence X, three matrices are first calculated: query matrix Q, key matrix K, and value matrix V, which are obtained by different linear transformations of the input. When calculating attention scores, the dot product of Q and K is used, and then the result is normalized using the softmax function to get the weight of each element for other elements. The final result is to apply these weights to the value matrix V to produce a new weighted representation. The calculation output matrix is:

[0080]

[0081] where d k is the dimension of the key vector, Q is the query matrix, K is the key matrix, and V is the value matrix.

[0082] When d k is small, the performance of dot-product attention and additive attention is similar, but when d k is large, the performance of additive attention is usually better than that of dot-product attention. This is because when d k is large, the result of the dot product will become large, causing the output of the softmax function to concentrate in the area of very small gradients. To solve this problem, the dot product is scaled by

[0083] The core idea of the multi-head attention mechanism is to capture diverse relationships from different subspaces by parallel computing multiple self-attention. In a single self-attention mechanism, the model can only identify one type of relationship, while multi-head attention allows the model to focus on multiple relationships at the same time. In terms of specific implementation, the input data is first divided into multiple subspaces, and each subspace independently performs self-attention calculation. The calculation result of each subspace corresponds to a head. The results of these different heads are spliced together, and after linear transformation, the final output is generated. Specifically as follows:

[0084] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0085] Wherein, head i =Attention(QW i Q ,KW i K ,VW i V ), each W i Q , W i K , W i V is a different linear projection matrix, and W O is an output linear transformation matrix.

[0086] The gated linear unit (GLU) is a channel mixer composed of two linear projections, which are element-wise multiplied, and one of the projections is activated by a gating function. The gating signal of each token of it comes from the token itself, and there is no acceptance domain larger than the value branch. Only by adding a minimum form of 3x3 deep convolution in front of the activation function of the GLU gating branch, its structure can meet the design concept of gated channel attention, and it can be converted into a gated channel attention mechanism based on nearest neighbor features, which is CGLU (Convolutional Gated Linear Unit).

[0087] Each token in CGLU has a unique gating signal based on its recent fine-grained features. This addresses the drawback of the global average pooling being too coarse-grained in the SE (Squeeze-and-Excitation) mechanism. CGLU generates two feature maps, A and B, through two convolution operations, where A serves as a linear transformation and B as a gating signal. The gating signal B is activated by a Sigmoid function to generate a gating factor σ(B) that controls the information flow. The final output Y is the product of A and σ(B), in this way, the network selectively passes or suppresses information flow under the control of B, thus preserving important nonlinear features.

[0088] The function of the neck network is to integrate features of different scales extracted from the backbone network to enhance the performance of the overall model. Specifically, the neck structure network used in this paper introduces the GLSA module, which not only effectively extracts global features but also captures fine-grained local features. The core advantage of this design is to promote the interaction between global and local information, ensuring that the model can better balance semantic information and location information when performing feature fusion. By introducing the GLSA structure, the model can more accurately identify and understand key information in images when processing complex tasks, thereby improving overall performance.

[0089] The function of the neck network is to integrate features of different scales extracted from the backbone network. Specifically, a GLSA improved neck structure network is used to fuse semantic and location features.

[0090] The original YOLOv8 neck network enhances information related to the optimization target while suppressing irrelevant information. To capture global and local spatial features, the GLSA module is proposed, which fuses the results of independent local attention units and global attention units. This dual-stream design of GLSA effectively preserves local and global modeling capabilities. In addition, channel separation is used to balance accuracy and computational resources. Specifically, the feature map Γ i |i∈(2,3,4), is evenly divided into two groups of feature maps are input into the global spatial attention (GSA) module and the local spatial attention (LSA) module, respectively. Finally, the outputs of these two attention units are fused through a 1x1 convolution layer. This process can be represented as:

[0091]

[0092] where G sa is the global spatial attention, L sa is the local spatial attention. is the output feature.

[0093] The design of the GSA (Global Spatial Aggregation) module focuses on capturing the long-range relationships of each pixel in space, which enables it to effectively complement the local spatial attention mechanism. By considering the long-distance pixel relationships, GSA can capture complex dependencies in a broader spatial range, enhancing the overall performance of the model. Numerous studies have shown that long-range interactions play an important role in improving model functionality, providing important theoretical support for this finding.

[0094] Therefore, a simple and effective method is adopted to generate a global spatial attention map Together as input, it can better capture global features and improve the model's understanding of complex scenes. The specific generation method and input process can be represented as follows:

[0095]

[0096] where A G Convolution. represents matrix multiplication. MLP(·) consists of two fully connected layers and a ReLU nonlinearity and normalization layer. The first layer of MLP converts its input to a high-dimensional space with an expansion ratio of 2, and the second layer restores it to the same dimension as the input.

[0097] The LSA (Local Spatial Aggregation) module has excellent feature extraction capabilities in the spatial dimension of the feature map, and it can effectively capture local features from the region of interest, which is particularly helpful for handling details such as small objects. In classroom behavior detection, there are many cases where the pixel value is small, and the LSA module can provide great help for such cases.

[0098] Through this method, the LSA module can more accurately locate and describe the detailed information of the target, thereby improving the model's recognition and processing ability for small targets. Local spatial attention response is introduced i 2 and combined as input for processing. Specifically, the LSA module calculates the attention weights within the local space, highlighting the most representative areas in the feature map, and combines them with the original features Figure 1 as input for subsequent network layers. This can be represented as follows:

[0099]

[0100] where Γ c (·) represents concatenating three 1x1 convolution layers and 3x3 deep convolution layers. Γ c The channel number is adjusted to 32. AL (·) is a local attention operation, is a sigmoid function, and is a point-wise multiplication. This structure design can effectively aggregate local spatial information with fewer parameters.

[0101] In the complex recognition task in the classroom environment, especially the students in the back row usually show a more blurred visual feature, which makes these areas in the image become the difficulty of detection. When dealing with such problems, the traditional detector usually adopts independent classification and positioning branches, which leads to insufficient interaction between the two tasks and cannot effectively utilize the correlation information between each other to improve the detection accuracy.

[0102] To solve this problem, a task-aligned prediction head ETADH is designed. Specifically, the interactive features between tasks are learned from multiple convolutional layers through a feature extractor to generate joint features. These joint features contain the mutual influence information between classification and positioning tasks, so that the two tasks can work together to improve the detection performance. First, the features after the feature fusion network are input into two shared 3x3 convolutions to obtain the interactive features, and the joint features are obtained using the task decomposition module TD (Task Decomposition). At the same time, the offset and mask are generated using the interactive features, which are input into the DCNv2, and the classification branch uses the interactive features for dynamic feature selection. The specific module structure is shown in Figure 4 .

[0103] Among them, object classification and positioning are performed on the calculated task interaction features, and the two tasks can well perceive each other's state. However, due to the single branch design, the task interaction features will inevitably introduce a certain degree of feature conflict between the two different tasks. Intuitively, the tasks of object classification and positioning have different goals, so they focus on different types of features. Therefore, a layer attention mechanism is used to encourage task decomposition by dynamically calculating these task-specific features at the layer level, and the task features of each classification or positioning task are calculated respectively:

[0104]

[0105] where w k is the k-th element of the learned layer attention w is calculated from the cross-layer task interaction features and can capture the dependency between layers:

[0106] w∈σ(fc2(δ(fc1(x inter ))))

[0107] where fc1 and fc2 represent two fully connected layers. σ is a sigmoid function on X inter is obtained by average pooling x inter , which is the concatenation feature of . Finally, the results of classification or localization are predicted from each X task :

[0108] Z task = conv2(δ(conv1(X task )))

[0109] where X task is the concatenation feature of , and conv1 is a 1x1 transformation layer for dimension reduction. Then, Z task is converted to dense classification scores

[0110] In the localization branch, ETADH introduces DCNv2 and interaction features, which are used to generate the offset and mask of DCNv2. DCNv2 has adaptability, which can dynamically adjust the position of the convolution kernel according to the input features, making the network more flexible when dealing with complex scenes, especially when dealing with different students' postures and positions, it can more accurately locate.

[0111] At the same time, in the classification branch, ETADH uses interaction features for dynamic feature selection. In this way, the classification task can optimize feature selection and processing according to the localization information, thereby improving classification accuracy. This inter-task cooperation not only improves the overall performance of the detection head, but also enhances the robustness of the model in complex classroom environments.

[0112] By introducing shared convolution and GroupNorm, the number of model parameters is significantly reduced, making the overall model more lightweight. Such design is particularly suitable for running on resource-constrained devices, such as mobile devices, edge computing devices, or embedded systems, etc. In these environments, computing resources and memory are usually very limited, so it is crucial to reduce the number of model parameters and computational requirements. Shared convolution avoids repeated computation by sharing convolution operations between different detection heads, not only reducing the number of parameters, but also reducing the computational cost. This simplified structure improves the inference speed of the model, enabling efficient real-time detection on low-power devices.

[0113] However, while using shared convolution, one challenge is that each detection head needs to handle targets of different scales. This inconsistency in target scale can affect detection performance, as shared convolution may not be able to adapt to features of different scales. Therefore, to solve this problem, a Scale layer is introduced in the design. The Scale layer is used to scale the features, ensuring that each detection head can obtain appropriate feature representations when handling targets of different scales. In this way, the model can more flexibly cope with various target scales, thereby improving detection accuracy.

[0114] Further, the step 5) comprises:

[0115] The main steps of data augmentation include:

[0116] 1. Randomly read four pictures from the data set each time.

[0117] 2. Flip the four pictures respectively, for example, flip the original picture left and right, scale, for example, scale the size of the original picture, color gamut change, for example, change the brightness, saturation and hue of the original picture. After the operation is completed, then the original picture is placed in the upper left, the second picture is placed in the lower left, the third picture is placed in the lower right, and the fourth picture is placed in the upper right.

[0118] 3. Perform picture combination and frame combination. After the four pictures are placed, the fixed area of the four pictures is cut off using a matrix method, and then they are spliced together to form a new picture containing a series of frame contents. The data augmentation method refers to the CutMix data augmentation method, which is an improved version of the CutMix data augmentation method. The CutMix algorithm uses two pictures for splicing, while four pictures are used for splicing here. This method greatly enriches the background of the detected object, and when standardizing the calculation, the data of four pictures is calculated at once.

[0119] Experimental verification

[0120] 1. Implementation details

[0121] The present application divides the data set into a training set and a test set using a split ratio of 8:2.

[0122] For model performance evaluation, the present application uses mAP50, mAP50:95, AP s , AP m and AP lThe value is taken as the evaluation index. The initial learning rate of the experiment is 0.01, 100 epochs are adopted, the batchsize is set to 32, and the SGD optimizer is used to optimize the parameters of the model. The experiment is carried out on a server with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz * 7 cores, NVIDIA GeForce GTX3090.

[0123] 2. Comparative experiment

[0124] The proposed method is compared with some representative models, and the specific results are shown in Figure 5 .

[0125] (1) RTDETR-r18: RTDETR is a target detection algorithm based on Transformer, which combines the advantages of traditional detection methods and Transformer architecture, aiming to realize real-time target detection. RTDETR-r18 is a version in this series, which uses lightweight ResNet-18 as the backbone network.

[0126] (2) RTDETR-l: is a larger version of the RTDETR series, compared with RTDETR-r18, it uses a more complex backbone network to improve the expression ability and detection accuracy of the model.

[0127] (3) YOLOv6s: is a target detection algorithm, the main goal is to realize efficient and accurate target detection. YOLOv6s is a lightweight version in this series, focusing on the balance between inference speed and accuracy, commonly used in real-time target detection tasks.

[0128] (4) YOLOv8s: continues the consistent style of the YOLO series, maintaining the characteristics of high efficiency and lightness. Compared with previous versions, it has further improved in accuracy and speed, especially in small target detection and real-time applications.

[0129] (5) YOLOv9s: is a further improved version of the YOLO series. While retaining high efficiency and speed, it optimizes the network structure at a deeper level, enhancing the detection ability in complex scenes.

[0130] (6) YOLOv10s: is the latest generation of the YOLO series, emphasizing further improving accuracy while maintaining high efficiency. Compared with YOLOv9s, it has more innovative improvements in network structure and training methods.

[0131] 3. Ablation experiment

[0132] Since the target detection network model proposed in the present application contains multiple key components, in this section, the present application will compare each module from the following aspects, and use mAP50 and mAP50:95 as evaluation indexes to prove the effectiveness of the overall network model and each module, Figure 6 The effects of each module on the data set are shown. From Figure 6 It can be seen from the above table that the performance of GLTB+GLSA+ETADH is significantly better than that of GLTB, GLSA, ETADH and GLTB+ETADH, which shows the effectiveness of each sub-module to the overall model.

Claims

1. A student classroom behavior detection method based on Transformer and task dynamic alignment, characterized in that, The method comprises the following steps: 1) Obtain monitoring videos of students in different grades in a classroom scene, including classroom pictures of primary and secondary school students, frame the obtained videos to obtain student classroom behavior pictures; 2) Label all student classroom behavior pictures using the labellmg image labeling tool to obtain a student classroom behavior dataset, which contains three categories: "reading", "hand-raising", and "writing"; 3) Data preprocessing, divide all available labeled data into two parts, take 80% of the images and label files as the training set, and the other 20% as the test set; 4) Model establishment, refer to the YOLOv8 backbone network, the model is divided into three parts: the main network adds GLTB to enhance feature extraction capability, the feature fusion network is the neck network improved by GLSA, and the new detection head ETADH is used in the final prediction head; 5) Data enhancement, apply multiple data enhancement techniques, including mosaic enhancement, mixed enhancement, random disturbance, and color disturbance, which helps to improve the generalization ability and robustness of the model through data enhancement; 6) Input the training set images and label data in the dataset into the established model, set fixed parameters, determine the optimizer and loss function, and obtain the trained student classroom behavior detection model; 7) Use the main network with GLTB to extract features from the input images to obtain key local and global features; 8) The neck network improved by GLSA further processes the features after obtaining the preliminary features, and performs feature fusion to enrich the multi-scale feature representation; 9) Use the new detection head ETADH module to receive the features extracted and fused by the backbone and neck, and perform the prediction task; 10) Input the images and label data in the test set into the trained model to classify and locate the student classroom behavior in each picture, obtain the corresponding prediction label and position information, compare the prediction result with the actual label to obtain the detection result, and calculate the average precision mean, small target average precision, medium target average precision, and large target average precision; 11) Input the test image into the trained model to output the student classroom behavior detection result in the test image, and highlight the detection result with colored frames and labels.

2. The student classroom behavior detection method based on Transformer and task dynamic alignment according to claim 1, characterized in that, To enhance the extraction effect of global features, GLTB is added to the backbone network as a feature extraction structure to enhance the feature extraction capability of global and local features. Since the computational complexity of the Transformer structure is significantly higher than that of the convolutional neural network, if it is directly applied to all channels, the computational cost will increase significantly. Therefore, the input feature map is divided into two parts, which are processed by convolutional neural network and Transformer respectively. By combining the local feature extraction capability of CNN and the global modeling capability of Transformer, this design also enhances the feature extraction effect.

3. The student classroom behavior detection method based on Transformer and task dynamic alignment according to claim 1, characterized in that, The GLTB uses EFEB, which contains MHSA and CGLU, and uses CGLU to enhance the nonlinear feature expression capability, combines the global feature extraction capability of MHSA, so that it has stronger feature extraction capability.

4. The student classroom behavior detection method based on Transformer and task dynamic alignment according to claim 1, characterized in that, The GLSA module is introduced, which can not only effectively extract global features, but also capture fine-grained local features. This design can promote the interaction between global and local information, ensure that the model can better balance semantic information and location information when performing feature fusion, and can more accurately identify and understand key information in the image when processing complex tasks, thereby improving overall performance.

5. The student classroom behavior detection method based on Transformer and task dynamic alignment according to claim 1, characterized in that, A task-aligned prediction head ETADH learns the interaction features between tasks from multiple convolutional layers through a feature extractor to generate joint features, which contain the mutual influence information between classification and positioning tasks, so that the two tasks can work together to improve detection performance. First, the features after the feature fusion network are input into two shared 3x3 convolutions to obtain interaction features, and the joint features are obtained using the TaskDecomposition module. At the same time, the displacement and mask are generated using the interaction features, which are input into the DCNv2, and the classification branch uses the interaction features for dynamic feature selection.

6. The student classroom behavior detection method based on Transformer and task dynamic alignment according to claim 1, characterized in that, By introducing shared convolution and GroupNorm, the number of model parameters is significantly reduced, making the overall model more lightweight. Shared convolution shares convolution operations between different detection heads, avoiding repeated calculations, reducing the number of parameters, and reducing the computational cost.

Images