Student classroom behavior detection method based on Transform and task dynamic alignment

Abstract

The invention provides a student classroom behavior detection method based on Transform and task dynamic alignment. On one hand, the GLTB is added into the backbone network to enhance the feature extraction capability. Wherein the CGLU is used for enhancing the non-linear feature expression capability, and the global feature extraction capability of the MHSA is combined, so that the method has higher feature extraction capability. And on the other hand, a neck structure network improved by GLSA is adopted, global features can be effectively extracted, fine-grained local features can be captured, and interaction between global information and local information can be promoted. According to the method, when a complex task is processed, key information in the image can be identified and understood more accurately. In addition, a novel detection head ETADH is adopted, interactive features of classification and positioning tasks are learned through a feature extractor, and task decomposition is performed through a task decomposition module. In this way, the interactivity between the two tasks is enhanced, and the detection precision is improved by effectively utilizing the associated information between the two tasks.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision technology, and relates to the field of classroom behavior detection technology, specifically to technologies such as deep learning and single-stage target detection, and in particular to a student classroom behavior detection method based on Transformer and task dynamic alignment. Background Art

[0002] In recent years, with the development of behavior detection technology, it has become possible to analyze student behavior in classroom videos and obtain information about students' classroom status and learning performance. This technology is very important for school teachers, administrators, students and parents. However, in the traditional teaching model, it is difficult for teachers to pay attention to the learning situation of each student, and they can only understand the effectiveness of their teaching methods by observing a few students. School administrators rely on on-site observations and student performance reports to discover problems in education and teaching. Parents can only understand their children's learning situation through communication with teachers and students. Therefore, using behavior detection technology to accurately detect students' behavior and analyze their learning status and performance can provide more comprehensive and accurate feedback for education and teaching.

[0003] Algorithms based on posture estimation characterize human behavior by obtaining the position and motion information of each joint of the human body, but they are not suitable for behavior detection in crowded classrooms. Considering the current challenges, algorithms based on object detection provide a promising solution. In fact, algorithms based on object detection have made great breakthroughs in recent years. Therefore, in this paper, an algorithm based on deep learning object detection is used to analyze students' behavior.

[0004] In terms of object detection, two-stage and one-stage object detection frameworks have received more attention due to their impressive detection results on public datasets. However, datasets of real classrooms are very different from those of public classrooms, and classic methods perform poorly in real classrooms. One of the representative problems is the large scale variation between different positions, for example, students in the front row of the classroom are about 40×40 pixels, while students in the back row are about 200×200 pixels, which leads to nearly 25 times high scale variation. What's worse, compared with the most popular object detection dataset MSCOCO, the occlusion between students is very serious. Moreover, the behavior of raising hands varies greatly in different environments, different people, and different angles.

[0005] Therefore, a new method for detecting students' classroom behavior goals that can solve the above problems is needed. Summary of the invention

[0006] The purpose of this invention is to solve the problems of low accuracy and poor small target detection caused by image target occlusion and inconsistent object scale in existing methods. In view of the above shortcomings, a student classroom behavior detection method based on Transformer and task dynamic alignment is proposed. The experimental results show that the method of this invention is superior to the existing technology and meets the current needs of classroom behavior detection of primary and secondary school students.

[0007] To achieve the above object, the technical solution adopted in the present invention is as follows:

[0008] 1) Obtain surveillance videos of classroom scenes of students of different grades, including classroom scenes of primary and secondary school students, deframe the obtained videos, and obtain pictures of students' classroom behavior.

[0009] 2) Use the labelImg image annotation tool to annotate all the pictures of students’ classroom behaviors and obtain a dataset of students’ classroom behaviors. It contains three categories: “reading”, “hand-raising”, and “writing”. In subsequent experiments, it will be found that the bounding boxes of “reading” and “writing” overlap, and the behaviors of “reading” and “writing” are visually similar.

[0010] 3) Data preprocessing: All available labeled data are divided into two parts, 80% of the images and label files are used as training sets, and the other 20% are used as test sets.

[0011] 4) Model establishment. Referring to the YOLOv8 backbone network, the model is mainly divided into three parts: the backbone network adds GLTB (Global-to-Local Transformer Block) to enhance the feature extraction capability, the feature fusion network is a neck network improved by GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Alignment Detection Head) is used in the final prediction head. The backbone network is the basic part of the model, responsible for extracting key local and global features from the input image. After obtaining the preliminary features, the neck structure of the model further processes these features to enrich the multi-scale feature representation. The neck generates a more detailed feature map containing contextual information by fusing information from different scales. Finally, the features extracted and fused by the backbone and neck will be passed to the head structure, which is responsible for specific prediction tasks, including classification tasks and regression tasks.

[0012] 5) Data enhancement. Applying a variety of data enhancement techniques, including mosaic enhancement, mixed enhancement, random perturbation, and color perturbation, helps improve the generalization ability and robustness of the model through data enhancement.

[0013] 6) Input the training set images and label data in the data set into the established model, set fixed parameters, determine the optimizer and loss function, and obtain the trained student classroom behavior detection model. Through AMP (Automatic Mixed Precision) training technology, mixed floating point numbers of different precisions are used to optimize the training process of the deep learning model, which greatly improves the training efficiency and resource utilization, while ensuring the stability of the model performance.

[0014] 7) Use the backbone network with GLTB added to extract features from the input image to obtain key local and global features.

[0015] 8) After obtaining the preliminary features, the neck network improved by GLSA further processes these features and fuses them to enrich the multi-scale feature representation.

[0016] 9) Use the new detection head ETADH module to receive the features extracted and fused from the trunk and neck to perform the prediction task.

[0017] 10) Input the images and label data in the test set into the trained model, classify and locate the students' classroom behaviors in each picture, and obtain the corresponding predicted labels and location information. Then, compare the predicted results with the actual labels to obtain the test results, and calculate indicators such as mean average precision (mAP).

[0018] 11) Input the test image into the trained model, output the detection results of students’ classroom behavior in the test image, and highlight the detection results with colored boxes and marks.

[0019] Specifically, the step 1) includes:

[0020] The images are taken from different shooting angles, including front, side, and back. These angles significantly affect the visual characteristics of students’ hand-raising behavior and increase the difficulty of the detection task.

[0021] In addition, classroom environments and seating arrangements may vary from course to course, adding more complexity to the detection and recognition of hand-raising behaviors. At the same time, students’ hand-raising behaviors also vary significantly at different stages of learning, further increasing the challenge of detection.

[0022] Further, the step 2) comprises:

[0023] Specifically, labelImg is an open source image annotation tool. During the annotation process, the classroom behavior areas of students in each image are manually selected and corresponding labels are assigned to each behavior area, such as "reading", "hand-raising", and "writing".

[0024] After the labeling is completed, labelImg will automatically generate a label information file corresponding to each image and save the label information in text format (txt file). The name of each txt file is consistent with the image file to facilitate subsequent reading and processing.

[0025] In this way, all students’ classroom behavior images are systematically annotated, eventually forming a complete student classroom behavior dataset.

[0026] Further, the step 4) comprises:

[0027] This deep learning network for student classroom behavior detection is mainly composed of three parts: a backbone network improved by GLTB (Global-to-Local Transformer Block), a feature fusion network improved by GLSA (Global-to-Local Spatial Aggregation), and a new detection head ETADH (Efficient Task Align Detection Head).

[0028] The original YOLOv8 backbone network was modified to improve the detection speed and recognition accuracy. Since the original network has limited effect on global feature extraction, the CNN-based feature extraction modules in the 7th and 9th layers of the original backbone are replaced with the more advanced Transformer-based GLTB in the backbone network. Specifically, GLTB is added to the backbone network as a feature extraction structure to enhance the global and local feature extraction capabilities. Due to the high computational complexity of the Transformer structure, if it is directly applied to all channels, it will incur a high computational cost. To solve this problem and reduce the computational burden while maintaining efficient feature extraction, a hybrid structure is designed. This structure divides the input feature map into two parts, which are processed by a convolutional neural network (CNN) and a Transformer respectively. This design enhances the effect of feature extraction by combining the local feature extraction capabilities of CNN and the global feature extraction capabilities of Transformer. The specific approach is to divide the input channel into parts and process them separately by CNN and Transformer to achieve more efficient feature extraction. EFEB (Efficient Feature Extraction Block) is used in the Transformer module, which contains MHSA (Multi-Head Self-Attention) and CGLU (Convolutional Gated Linear Unit). CGLU is used to enhance the nonlinear feature expression capability, combined with the global feature extraction capability of MHSA, so that it has stronger feature extraction capability.

[0029] At the same time, in order to avoid a significant increase in computational complexity when introducing the Transformer structure to extract global features, the Transformer Block is only applied to some channels.

[0030] The function of the neck network is to integrate the features of different scales extracted from the backbone network. Specifically, the neck structure network improved by GLSA is used to fuse semantic and position features.

[0031] The original YOLOv8 neck network enhances the information related to the optimization target while suppressing irrelevant information. To capture global and local spatial features, a GLSA module is proposed, which fuses the results of independent local attention units and global attention units. This two-stream design of GLSA effectively retains local and global modeling capabilities. In addition, a balance is achieved between accuracy and computational resources by channel separation. Specifically, the feature map Γ containing 64 channels i |i∈(2,3,4), is evenly divided into two groups of feature maps They are input into the global spatial attention (GSA) module and the local spatial attention (LSA) module respectively. Finally, the outputs of these two attention units are fused through a 1×1 convolutional layer. This process can be expressed as:

[0032]

[0033] Among them, G sa is the global spatial attention, L sa Pay attention to the local space. is the output feature.

[0034] In ETADH, a task alignment structure is adopted to cope with complex recognition tasks in a classroom environment. In this environment, students in the back row in particular usually show vague visual features, which makes these areas in the image difficult to detect. Traditional detectors usually use independent classification and localization branches when dealing with such problems, which leads to insufficient interactivity between the two tasks and the inability to effectively utilize the correlation information between them to improve detection accuracy.

[0035] To solve this problem, ETADH designed a customized structure for task alignment. Specifically, the feature extractor learns the interactive features between tasks from multiple convolutional layers to generate joint features. These joint features contain the mutual influence information between classification and localization tasks, allowing the two tasks to work together to improve detection performance.

[0036] In the localization branch, ETADH introduces DCNv2 and interactive features, and uses these features to generate the offset and mask of DCNv2. DCNv2 is adaptive and can dynamically adjust the position of the convolution kernel according to the input features, making the network more flexible in processing complex scenes, especially when dealing with different postures and positions of students, and can perform more accurate positioning.

[0037] Meanwhile, in the classification branch, ETADH uses interactive features for dynamic feature selection. In this way, the classification task can optimize the selection and processing of features based on the positioning information, thereby improving the classification accuracy. This synergy between tasks not only improves the overall performance of the detection head, but also enhances the robustness of the model in complex classroom environments.

[0038] By introducing shared convolution and GroupNorm, the number of model parameters has been significantly reduced, making the overall model more lightweight. Such a design is particularly suitable for running on resource-constrained devices, such as mobile devices, edge computing devices, or embedded systems. In these environments, computing resources and memory are usually very limited, so reducing the number of model parameters and computational requirements is crucial. Shared convolution avoids repeated calculations by sharing convolution operations between different detection heads, which not only reduces the number of parameters but also reduces computational costs. This simplified structure improves the inference speed of the model, enabling efficient real-time detection on low-power devices.

[0039] However, one challenge when using shared convolution is that each detection head needs to process objects of different scales. This inconsistency in target scale will affect the detection effect because the shared convolution may not be able to adapt to features of different scales. Therefore, in order to solve this problem, the Scale layer is introduced in the design. The Scale layer is used to scale the features to ensure that each detection head can get the appropriate feature representation when processing objects of different scales. In this way, the model can cope with various target scales more flexibly, thereby improving the accuracy of detection.

[0040] Further, the step 5) comprises:

[0041] The main steps of data augmentation include:

[0042] 1. Randomly read four pictures from the dataset each time.

[0043] 2. Flip the four pictures respectively, such as flipping the original picture left and right, scaling, such as scaling the original picture, and changing the color gamut, such as changing the brightness, saturation, and hue of the original picture. After the operation is completed, place the original pictures in four directions, with the first picture on the upper left, the second picture on the lower left, the third picture on the lower right, and the fourth picture on the upper right.

[0044] 3. Combine pictures and frames. After the four pictures are arranged, the fixed areas of the four pictures are cut out by matrix, and then they are spliced ​​together to form a new picture, which contains a series of contents such as frames. The data enhancement method refers to the CutMix data enhancement method, which is an improved version of the CutMix data enhancement method. The CutMix algorithm uses two pictures for splicing, while four pictures are used here. This method greatly enriches the background of the detected object, and the data of four pictures will be calculated at once during the standardized BN calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0047] Figure 2 It is a schematic diagram of the model framework of the present invention.

[0048] Figure 3 This is the structure diagram of the Global-to-Local Transformer Block (GLTB) in the backbone network in the experiment of the present invention.

[0049] Figure 4 This is the structure diagram of the new detection head EfficientTaskAlignDetectionHead (ETADH) used in the experiment of this invention

[0050] Figure 5 This is a performance comparison chart of the present invention and other methods on a student classroom behavior detection dataset.

[0051] Figure 6 The figure is a performance comparison chart of different variants of the present invention on detecting students' classroom behavior.

[0052] Figure 7 This is a partial prediction result diagram obtained by predicting the student classroom behavior detection pictures in the present invention DETAILED DESCRIPTION

[0053] The purpose of this invention is to solve the problems of low accuracy and poor small target detection caused by image target occlusion and inconsistent object scale in existing methods. In view of the above shortcomings, a student classroom behavior detection method based on Transformer and task dynamic alignment is proposed. The experimental results show that the method of this invention is superior to the existing technology and meets the current needs of classroom behavior detection of primary and secondary school students.

[0054] To achieve the above object, the technical solution adopted by the present invention is as follows. The main process of the method is as follows: Figure 1 As shown:

[0055] 1) Obtain surveillance videos of classroom scenes of students of different grades, including classroom scenes of primary and secondary school students, deframe the obtained videos, and obtain pictures of students' classroom behavior.

[0056] 2) Use the labelImg image annotation tool to annotate all the pictures of students’ classroom behaviors and obtain a dataset of students’ classroom behaviors. It contains three categories: “reading”, “hand-raising”, and “writing”. In subsequent experiments, it will be found that the bounding boxes of “reading” and “writing” overlap, and the behaviors of “reading” and “writing” are visually similar.

[0057] 3) Data preprocessing: All available labeled data are divided into two parts, 80% of the images and label files are used as training sets, and the other 20% are used as test sets.

[0058] 4) Model establishment. Referring to the YOLOv8 backbone network, there are feature maps of three scales: 80×80×256, 40×40×512, and 20×20×1024. The feature map of the feature scale of 80×80×256 is used as input. The model is mainly divided into three parts: the backbone network adds GLTB (Global-to-Local Transformer Block) to enhance the feature extraction capability, the feature fusion network is a neck network improved by GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Alignment Detection Head) is used in the final prediction head. The backbone network is the basic part of the model, responsible for extracting key local and global features from the input image. After obtaining the preliminary features, the neck structure of the model further processes these features to enrich the multi-scale feature representation. The neck generates a more detailed feature map containing contextual information by fusing information from different scales. Finally, the features extracted and fused by the trunk and neck are passed to the head structure, which is responsible for specific prediction tasks, including classification and regression tasks.

[0059] 5) Data enhancement. Applying a variety of data enhancement techniques, including mosaic enhancement, mixed enhancement, random perturbation, and color perturbation, helps improve the generalization ability and robustness of the model through data enhancement.

[0060] 6) Input the training set images and label data in the data set into the established model, set fixed parameters, determine the optimizer and loss function, and obtain the trained student classroom behavior detection model. Through AMP (Automatic Mixed Precision) training technology, mixed floating point numbers of different precisions are used to optimize the training process of the deep learning model, which greatly improves the training efficiency and resource utilization, while ensuring the stability of the model performance.

[0061] 7) Use the backbone network with GLTB added to extract features from the input image to obtain key local and global features.

[0062] 8) The neck network improved by GLSA (Global-to-Local Spatial Aggregation), after obtaining the preliminary features, the neck structure of the model further processes these features and fuses them to enrich the multi-scale feature representation.

[0063] 9) Use the new detection head ETADH module to receive the features extracted and fused from the trunk and neck to perform the prediction task.

[0064] 10) Input the images and label data in the test set into the trained model, classify and locate the students' classroom behaviors in each picture, and obtain the corresponding predicted labels and location information. Then, compare the predicted results with the actual labels to obtain the test results, and calculate indicators such as mean average precision (mAP).

[0065] 11) Input the test image into the trained model, output the detection results of students’ classroom behavior in the test image, and highlight the detection results with colored boxes and marks.

[0066] Specifically, the step 1) includes:

[0067] The images are taken from different shooting angles, including front, side, and back. These angles significantly affect the visual characteristics of students’ hand-raising behavior and increase the difficulty of the detection task.

[0068] In addition, classroom environments and seating arrangements may vary from course to course, adding more complexity to the detection and recognition of hand-raising behaviors. At the same time, students’ hand-raising behaviors also vary significantly at different stages of learning, further increasing the challenge of detection.

[0069] Further, the step 2) comprises:

[0070] Specifically, labelImg is an open source image annotation tool. During the annotation process, the classroom behavior areas of students in each image are manually selected and corresponding labels are assigned to each behavior area, such as "reading", "hand-raising", and "writing".

[0071] After the labeling is completed, labelImg will automatically generate a label information file corresponding to each image and save the label information in text format (txt file). The name of each txt file is consistent with the image file to facilitate subsequent reading and processing.

[0072] In this way, all students’ classroom behavior images are systematically annotated, eventually forming a complete student classroom behavior dataset.

[0073] Further, the step 4) comprises:

[0074] The deep learning network for student classroom behavior detection is mainly composed of three parts: the backbone network improved by GLTB (Global-to-Local Transformer Block), the feature fusion network is the neck network improved by GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Align Detection Head). Figure 2 shown.

[0075] The original YOLOv8 backbone network was modified to improve the detection speed and recognition accuracy. Since the original network structure has limited effect on global feature extraction, the feature extraction modules of the 7th and 9th layers in the original backbone are replaced with the more advanced Transformer-based GLTB in the backbone network. Specifically, GLTB is added to the backbone network as a feature extraction structure to enhance the global and local feature extraction capabilities. Since the Transformer structure has a large computational complexity, if it is directly applied to all channels, it will incur a high computational cost. In order to solve this problem and reduce the computational burden while maintaining efficient feature extraction, a hybrid structure is designed. This structure divides the input feature map into two parts, which are processed by a convolutional neural network and a Transformer respectively. This design enhances the effect of feature extraction by combining the local feature extraction capability of CNN and the global modeling capability of Transformer. Specifically, the input channel is partially divided and processed by CNN and Transformer respectively, so as to achieve more efficient feature extraction. EFEB (Efficient Feature Extraction Block) is used in TransformerBlock, which includes MHSA (Multi-Head Self-Attention) and CGLU (Convolutional Gated Linear Unit). CGLU is used to enhance the nonlinear feature expression ability, combined with the global feature extraction ability of MHSA, so that it has stronger feature extraction ability. The module framework of the present invention is as follows: Figure 3 shown.

[0076] At the same time, in order to avoid a significant increase in computational complexity when introducing the Transformer structure to extract global features, the Transformer Block is only applied to some channels.

[0077] MHSA (Multi-Head Self-Attention) is a key component of the Transformer model, which is used to capture complex relationships in sequence data. Through the self-attention mechanism, it can parallelly calculate the relationship between elements in the sequence, and improve the model's expressiveness by setting multiple attention heads.

[0078] Self-Attention mechanism, the core idea of ​​the self-attention mechanism is to calculate the correlation between each element in the sequence and other elements. For each element in a sequence, the model reweights and combines them according to its relationship with all other elements in the sequence.

[0079] Specifically, for the input sequence X, three matrices are first calculated: the query matrix Q, the key matrix K, and the value matrix V, which are obtained by different linear transformations of the input. When calculating the attention score, the dot product of Q and K is used, and then the result is normalized and the softmax function is used to obtain the weight of each element for other elements. The final result is to apply these weights to the value matrix V to produce a new weighted representation. The calculated output matrix is:

[0080]

[0081] Among them, d k is the dimension of the key vector, Q is the query matrix, K is the key matrix, and V is the value matrix.

[0082] When k When the value is small, the performance of dot product attention and additive attention is similar, but when d k When the value is large, additive attention usually performs better than dot-product attention. This is because when d k When the value is large, the dot product result will become larger, causing the output of the softmax function to be concentrated in the area with very small gradient. To solve this problem, the dot product is scaled and multiplied by

[0083] The core idea of ​​the Multi-Head Attention mechanism is to capture diverse relationships from different subspaces by calculating multiple self-attentions in parallel. In a single self-attention mechanism, the model can only recognize one type of relationship, while multi-head attention allows the model to pay attention to multiple relationships at the same time. In specific implementation, the input data is first divided into multiple subspaces, and each subspace performs self-attention calculations independently. The calculation results of each subspace correspond to a head. The results of these different heads are spliced ​​together and, after linear transformation, generate the final output. The details are as follows:

[0084] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0085] Among them, head i =Attention(QW i Q ,KW i K ,VW i V ), each W i Q , W i K , W i V are different linear projection matrices, W O is the output linear transformation matrix.

[0086] The Gated Linear Unit (GLU) is a channel mixer consisting of two linear projections that are element-wise multiplied, one of which is activated by a gating function. Its gating signal for each token comes from the token itself, and it has no larger receptive field than the value branch. Simply adding a minimal 3×3 depth convolution before the activation function of the GLU gating branch can make its structure conform to the design concept of gated channel attention and transform it into a gated channel attention mechanism based on nearest neighbor features, which is CGLU (Convolutional Gated Linear Unit).

[0087] Based on its nearest fine-grained features, each token in CGLU has a unique gating signal. This solves the shortcoming of the global average pooling in the SE (Squeeze-and-Excitation) mechanism being too coarse-grained. CGLU generates two feature maps A and B through two convolution operations, where A acts as a linear transformation and B acts as a gating signal. The gating signal B is activated by the Sigmoid function to generate a gating factor σ(B) for controlling the information flow. The final output Y is the product of A and σ(B). In this way, the network selectively passes or inhibits the flow of information under the control of B, thereby retaining important nonlinear features.

[0088] The function of the neck network is to integrate features of different scales extracted from the backbone network to enhance the performance of the overall model. Specifically, the neck structure network used in this paper introduces the GLSA module, which can not only effectively extract global features, but also capture fine-grained local features. The core advantage of this design is that it can promote the interaction between global and local information, ensuring that the model can better balance semantic information and location information when performing feature fusion. By introducing the GLSA structure, the model can more accurately identify and understand the key information in the image when processing complex tasks, thereby improving the overall performance.

[0089] The function of the neck network is to integrate the features of different scales extracted from the backbone network. Specifically, the neck structure network improved by GLSA is used to fuse semantic and position features.

[0090] The original YOLOv8 neck network enhances the information related to the optimization target while suppressing irrelevant information. To capture global and local spatial features, a GLSA module is proposed, which fuses the results of independent local attention units and global attention units. This two-stream design of GLSA effectively retains local and global modeling capabilities. In addition, a balance is achieved between accuracy and computational resources by channel separation. Specifically, the feature map Γ containing 64 channels i |i∈(2,3,4), is evenly divided into two groups of feature maps They are input into the global spatial attention (GSA) module and the local spatial attention (LSA) module respectively. Finally, the outputs of these two attention units are fused through a 1×1 convolutional layer. This process can be expressed as:

[0091]

[0092] Among them, G sa is the global spatial attention, L sa Pay attention to the local space. is the output feature.

[0093] The design of the GSA (Global Spatial Aggregation) module focuses on capturing the long-range relationship of each pixel in space, which enables it to effectively complement the local spatial attention mechanism. By considering long-distance pixel relationships, GSA can capture complex dependencies in a wider spatial range, thereby enhancing the overall performance of the model. Many studies have shown that long-range interactions play an important role in improving model functions, and this finding provides important theoretical support for this.

[0094] Therefore, a simple yet effective method is adopted to generate a global spatial attention map Taking them together as input can better capture global features and improve the model's ability to understand complex scenes. The following is the specific generation method and input process:

[0095]

[0096] Among them, A G convolution. Represents matrix multiplication. MLP(·) consists of two fully connected layers and a ReLU nonlinearity and normalization layer. The first layer of MLP transforms its input into a high-dimensional space with an expansion ratio of 2, and the second layer restores it to the same dimension as the input.

[0097] The LSA (Local Spatial Aggregation) module has excellent feature extraction capabilities in the spatial dimension of the feature map. It can effectively capture local features from the region of interest, which is especially helpful for processing details such as small objects. There are many cases where the pixel values ​​are very small in classroom behavior detection, and the LSA module will provide great help in this case.

[0098] In this way, the LSA module can more accurately locate and describe the details of the target, thereby improving the model's ability to recognize and process small targets. And compare it with the feature map Γ i 2 Specifically, the LSA module calculates the attention weights in the local space, highlights the most representative area in the feature map, and combines it with the original feature map. Figure 1 It serves as the input of the subsequent network layer. As shown below:

[0099]

[0100] where Γ c (·) represents a cascade of three 1×1 convolutional layers and a 3×3 depthwise convolutional layer. Γ c The number of channels is adjusted to 32.L (·) is the local attention operation, is the sigmoid function, and ⊙ is the point-by-point multiplication. This structural design can effectively aggregate local spatial information with fewer parameters.

[0101] In complex recognition tasks in classroom environments, students in the back row, in particular, often exhibit vague visual features, which makes these areas in the image difficult to detect. Traditional detectors usually use independent classification and localization branches to deal with such problems, which results in insufficient interactivity between the two tasks and inability to effectively utilize the correlation information between them to improve detection accuracy.

[0102] To solve this problem, a task-aligned prediction head ETADH is designed. Specifically, the feature extractor learns the interactive features between tasks from multiple convolutional layers to generate joint features. These joint features contain the mutual influence information between classification and positioning tasks, so that the two tasks can work together to improve detection performance. First, the features after the feature fusion network are input into two shared 3×3 convolutions to obtain interactive features. The task decomposition module TD (Task Decomposition) is used to obtain joint features. At the same time, the interactive features are used to generate offsets and masks, which are input into DCNv2. At the same time, the classification branch uses interactive features for dynamic feature selection. The specific module structure is as follows: Figure 4 shown.

[0103] Among them, object classification and localization are performed on the computed task interaction features, and the two tasks can perceive each other's status well. However, due to the single branch design, the task interaction features inevitably introduce a certain degree of feature conflict between the two different tasks. Intuitively, the tasks of object classification and localization have different goals and therefore focus on different types of features. Therefore, a layer attention mechanism is used to encourage task decomposition by dynamically computing these task-specific features at the layer level. The task features of each classification or localization task are calculated separately:

[0104]

[0105] Among them, w k is the learning layer attention The kth element of . w is calculated from the cross-layer task interaction features and can capture the dependencies between layers:

[0106] w∈σ(fc2(δ(fc1(x inter ))))

[0107] Where fc1 and fc2 represent two fully connected layers. σ is a sigmoid function that inter Perform average pooling to get x inter ,This is Finally, from each X task The results of predicting classification or positioning:

[0108] Z task =conv2(δ(conv1(X task )))

[0109] Among them, X task yes The connection characteristics of conv1 are 1×1 conversion layers for dimensionality reduction. Then the sigmoid function is used to convert Z task Convert to dense classification scores

[0110] In the localization branch, ETADH introduces DCNv2 and interactive features, and uses these features to generate the offset and mask of DCNv2. DCNv2 is adaptive and can dynamically adjust the position of the convolution kernel according to the input features, making the network more flexible in processing complex scenes, especially when dealing with different postures and positions of students, and can perform more accurate positioning.

[0111] Meanwhile, in the classification branch, ETADH uses interactive features for dynamic feature selection. In this way, the classification task can optimize the selection and processing of features based on the positioning information, thereby improving the classification accuracy. This synergy between tasks not only improves the overall performance of the detection head, but also enhances the robustness of the model in complex classroom environments.

[0112] By introducing shared convolution and GroupNorm, the number of model parameters has been significantly reduced, making the overall model more lightweight. Such a design is particularly suitable for running on resource-constrained devices, such as mobile devices, edge computing devices, or embedded systems. In these environments, computing resources and memory are usually very limited, so reducing the number of model parameters and computational requirements is crucial. Shared convolution avoids repeated calculations by sharing convolution operations between different detection heads, which not only reduces the number of parameters but also reduces computational costs. This simplified structure improves the inference speed of the model, enabling efficient real-time detection on low-power devices.

[0113] However, one challenge when using shared convolution is that each detection head needs to process objects of different scales. This inconsistency in target scale will affect the detection effect because the shared convolution may not be able to adapt to features of different scales. Therefore, in order to solve this problem, the Scale layer is introduced in the design. The Scale layer is used to scale the features to ensure that each detection head can get the appropriate feature representation when processing objects of different scales. In this way, the model can cope with various target scales more flexibly, thereby improving the accuracy of detection.

[0114] Further, the step 5) comprises:

[0115] The main steps of data augmentation include:

[0116] 1. Randomly read four pictures from the dataset each time.

[0117] 2. Flip the four pictures respectively, such as flipping the original picture left and right, scaling, such as scaling the original picture, and changing the color gamut, such as changing the brightness, saturation, and hue of the original picture. After the operation is completed, place the original pictures in four directions, with the first picture on the upper left, the second picture on the lower left, the third picture on the lower right, and the fourth picture on the upper right.

[0118] 3. Combine pictures and frames. After the four pictures are arranged, the fixed areas of the four pictures are cut out by matrix, and then they are spliced ​​together to form a new picture, which contains a series of contents such as frames. The data enhancement method refers to the CutMix data enhancement method and is an improved version of the CutMix data enhancement method. The CutMix algorithm uses two pictures for splicing, while four pictures are used here. This method greatly enriches the background of the detected object, and the data of four pictures will be calculated at once during the standardization calculation.

[0119] Experimental verification

[0120] 1. Implementation details

[0121] The present invention uses a split ratio of 8:2 to divide the data set into a training set and a test set.

[0122] For the evaluation of model performance, the present invention uses mAP50, mAP50:95, AP s ,AP m and AP lThe value is used as the evaluation index. The initial learning rate of the experiment is 0.01, 100 epochs are used, the batch size is set to 32, and the SGD optimizer is used to optimize the parameters of the model. The experiment is conducted on a server with Intel(R)Xeon(R)CPU E5-2680 v4@2.40GHz*7 cores and NVIDIAGeForceGTX3090.

[0123] 2. Comparative experiment

[0124] The proposed method is compared with some representative models. Figure 5 shown.

[0125] (1) RTDETR-r18: RTDETR is a Transformer-based object detection algorithm that combines the advantages of traditional detection methods and Transformer architecture to achieve real-time object detection. RTDETR-r18 is a version in this series that uses a lightweight ResNet-18 as the backbone network.

[0126] (2) RTDETR-l: It is a larger-scale version of the RTDETR series. Compared with RTDETR-r18, it uses a more complex backbone network to improve the model's expressiveness and detection accuracy.

[0127] (3) YOLOv6s: is an object detection algorithm whose main goal is to achieve efficient and accurate object detection. YOLOv6s is a lightweight version in the series, focusing on the balance between inference speed and accuracy, and is often used in real-time object detection tasks.

[0128] (4) YOLOv8s: It continues the consistent style of the YOLO series and maintains the characteristics of high efficiency and light weight. Compared with previous versions, it has further improved in accuracy and speed, especially in small object detection and real-time applications.

[0129] (5) YOLOv9s: It is a further improved version of the YOLO series. While maintaining high efficiency and speed, it optimizes the network structure at a deeper level and enhances the detection capability in complex scenarios.

[0130] (6) YOLOv10s: It is the latest generation of the YOLO series, emphasizing further improving accuracy while maintaining high efficiency. Compared with YOLOv9s, it has made more innovative improvements in network structure and training methods.

[0131] 3. Ablation experiment

[0132] Since the target detection network model proposed in this paper contains multiple key components, in this section, the present invention will compare each module from the following aspects, using mAP50 and mAP50:95 as evaluation indicators to prove the effectiveness of the overall network model and each module. Figure 6 The effect of each module on the dataset is shown. Figure 6 It is not difficult to see that the performance of GLTB+GLSA+ETADH is significantly better than GLTB, GLSA, ETADH and GLTB+ETADH, which illustrates the effectiveness of each sub-block to the overall model.

Claims

1. A method for detecting student classroom behavior based on Transformer and task dynamic alignment, characterized in that: The following steps are involved: 1) Obtain surveillance videos of classroom scenes of students of different grades, including classroom scenes of primary and secondary school students, and deframe the obtained videos to obtain images of students' classroom behaviors; 2) Use labelImg image annotation tool to annotate all the pictures of students’ classroom behaviors and obtain the students’ classroom behavior dataset, which contains three categories: "reading", "hand-raising", and "writing"; 3) Data preprocessing: All available labeled data are divided into two parts, 80% of the images and label files are used as training sets, and the remaining 20% ​​are used as test sets; 4) Model establishment. Referring to the YOLOv8 backbone network, the model is mainly divided into three parts: the backbone network adds GLTB (Global-to-Local Transformer Block) to enhance the feature extraction capability, the feature fusion network is a neck network improved by GLSA (Global-to-Local Spatial Aggregation), and the new detection head ETADH (Efficient Task Align Detection Head) is used in the final prediction of the head; 5) Data enhancement. Apply a variety of data enhancement techniques, including mosaic enhancement, mixed enhancement, random perturbation, and color perturbation. Data enhancement helps improve the generalization ability and robustness of the model; 6) Input the training set images and label data in the data set into the established model, set fixed parameters, determine the optimizer and loss function, and obtain a trained student classroom behavior detection model; 7) Use the backbone network with GLTB added to extract features from the input image to obtain key local and global features; 8) After obtaining the preliminary features, the neck network improved by GLSA further processes these features and fuses them to enrich the multi-scale feature representation; 9) Use the new detection head ETADH module to receive the features extracted and fused from the trunk and neck to perform prediction tasks; 10) Input the images and label data in the test set into the trained model, classify and locate the students' classroom behaviors in each picture, and obtain the corresponding predicted labels and location information. Then, compare the predicted results with the actual labels to obtain the test results, and calculate indicators such as mean average precision (mAP); 11) Input the test image into the trained model, output the detection results of students’ classroom behavior in the test image, and highlight the detection results with colored boxes and marks.

2. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1 is characterized in that The backbone network adds GLTB (Global-to-Local Transformer Block) to enhance the feature extraction capability. The feature fusion network is a neck network improved by GLSA (Global-to-Local Spatial Aggregation), and a new detection head ETADH (Efficient Task Align Detection Head) is used in the final prediction head.

3. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1 is characterized in that: As the original network structure has limited effect on global feature extraction, GLTB is added to the backbone network as the feature extraction structure to enhance the global and local feature extraction capabilities. Since the Transformer structure has a large computational complexity, if it is directly applied to all channels, it will incur a high computational cost. Therefore, the input feature map is divided into two parts, which are processed by the convolutional neural network and the Transformer respectively. By combining the local feature extraction capability of CNN and the global modeling capability of Transformer, this design also enhances the effect of feature extraction.

4. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1 is characterized in that: The GLTB uses EFEB (Efficient Feature Extraction Block), which includes MHSA (Multi-Head Self-Attention) and CGLU (Convolutional Gated Linear Unit). CGLU is used to enhance the nonlinear feature expression capability, and combined with the global feature extraction capability of MHSA, it has stronger feature extraction capability.

5. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1, characterized in that: The GLSA module introduced in the neck structure network can not only effectively extract global features, but also capture fine-grained local features. This design can promote the interaction between global and local information, ensuring that the model can better balance semantic information and position information when performing feature fusion. When processing complex tasks, it can more accurately identify and understand key information in the image, thereby improving overall performance.

6. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1, characterized in that: The task-aligned prediction head ETADH learns the interactive features between tasks from multiple convolutional layers through a feature extractor to generate joint features. These joint features contain the mutual influence information between classification and positioning tasks, so that the two tasks can work together to improve detection performance. First, the features after the feature fusion network are input into two shared 3×3 convolutions to obtain interactive features, and the task decomposition module Task Decomposition is used to obtain joint features. At the same time, the interactive features are used to generate offsets and masks, which are input into DCNv2, and the classification branch uses the interactive features for dynamic feature selection.

7. The method for detecting student classroom behavior based on Transformer and task dynamic alignment according to claim 1, characterized in that: By introducing shared convolution and GroupNorm, the number of model parameters has been significantly reduced, making the overall model more lightweight. Shared convolution avoids repeated calculations by sharing convolution operations between different detection heads, which not only reduces the number of parameters but also reduces computational costs.

Images