Detection method of students' classroom posture behavior based on smart classroom scenario

Abstract

The present invention discloses a method for detecting students' classroom posture behaviors in a smart classroom scenario, specifically comprising the following steps: Step 1, creating an SCB_dataset dataset; Step 2, constructing a Backbone module to extract the characteristics of students' head-raising and head-lowering behaviors when listening to a class; Step 3, building a feature perception pyramid module in an FPA Transformer to perform feature fusion on the feature map extracted in Step 2; Step 4, designing a Prediction Head module to locate students' positions on the feature map obtained in Step 3 and classify their head-raising and head-lowering behaviors; Step 5, saving the weight file generated by the training; Step 6, inputting classroom images into the trained FPA Transformer to obtain the final network detection results. This method can accurately locate students' positions and classify their head-raising and head-lowering behaviors when listening to a class.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing methods, and specifically relates to a method for detecting students' classroom posture behaviors in a smart classroom scenario. Background Art

[0002] Smart education has brought about significant changes in educational formats and learning methods. In smart classroom scenarios, intelligent detection of student posture is a crucial component of building an intelligent classroom environment. Despite the proliferation of target detection methods, existing methods cannot accurately detect the positions of students in a classroom, and are even less capable of recognizing the postures of densely packed students. The emergence of SwinTransformer has improved the accuracy of target detection in densely packed environments. However, in smart classroom scenarios, students are typically represented as small, densely packed objects in images. The SwinTransformer network does not adequately extract features for these small objects, resulting in high false detection and missed detection rates for student posture in smart classrooms. Summary of the Invention

[0003] The purpose of this invention is to provide a method for detecting students' classroom posture behaviors in a smart classroom scenario, which can accurately locate students and accurately classify their head-raising and head-lowering behaviors in complex backgrounds and classroom environments with dense student populations.

[0004] The technical solution adopted by this paper is a method for detecting student posture behavior in a smart classroom scenario. This method uses a Transformer (FPATransformer) with Feature Perception Attention (FPA) to detect and classify students' head-raising and head-lowering behaviors in the classroom. Specifically, the method includes the following steps: Step 1: Create the StudentClassroomBehaviordataset (SCB_dataset);

[0005] Step 2: Construct the Backbone module in the FPA Transformer. The Backbone module uses four Feature Perception Blocks (FPBs) to extract the characteristics of students’ head-up and head-down behaviors during lectures and obtain the corresponding feature maps.

[0006] Step 3: Build the Feature Perception Pyramid (FPP) module in the FPA Transformer to perform feature fusion on the feature map extracted in step 2.

[0007] Step 4: Design the Prediction Head module in the FPA Transformer to locate the student's position on the feature map obtained in step 3 and classify the student's head-up and head-down behaviors;

[0008] Step 5: Use the SCB_dateset training set from step 1 to train the FPA Transformer consisting of the Backbone module, FPP module, and PredictionHead module, and save the weight file generated by the training;

[0009] Step 6: Load the weight file trained in step 5 and input the classroom image into the trained FPATransformer to obtain the final network detection results.

[0010] The present invention is also characterized in that

[0011] Step 1 is to create the SCB_dataset training dataset, specifically: create 6,000 classroom scene images of 640×640×3 size as training samples, mark the students' head-up and head-down behaviors in each image, and create the SCB_dataset training dataset in VOC format after the marked images.

[0012] Step 2 is implemented as follows:

[0013] Step 2.1: Build the Backbone module. The Backbone module consists of a Patch Partition layer, a Linear Embedding layer, three Patch Merging layers, and four FPB blocks. The Patch Partition layer in the Backbone module divides the input image into blocks. The Linear Embedding layer in the Backbone module then performs a linear transformation on each pixel's channel. Four identical FPB blocks are then used to extract student classroom behavior features. Except for the first FPB block, the remaining three FPB blocks are downsampled through the Patch Merging layer. Finally, the Backbone module outputs feature maps of student behavior information at four different scales.

[0014] Step 2.2, in the PatchPartition layer of the Backbone module, the image X in step 1 is divided into blocks, and every 4 adjacent pixels are a patch (X i ), and then flatten the block-wise input image in the channel direction to obtain image L1;

[0015] Step 2.3: Apply linear transformation to each pixel channel of the image L1 obtained in step 2.2 through the LinearEmbedding layer of the FPA Transformer to obtain the feature map L2.

[0016] In step 2.4, construct the FPB. Each FPB consists of two consecutive Swin Transformer blocks, but replace the Windowed Multi-Head Self-Attention (W-MSA) in the Swin Transformer block with Local Feature Perception Attention (LFPA). The feature map L2 obtained in step 2.3 is input into the first FPB block to extract the features of students' head-up and head-down behaviors while listening to the lecture, resulting in a feature map feat1 of size 160×160×128.

[0017] Step 2.5: Input the feature map feat1 obtained in step 2.4 into the first PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is input into the second FPB block. The features of the students looking up and looking down while listening to the class are extracted to obtain the feature map feat2 of size 80×80×256.

[0018] Step 2.6: Input the feature map feat2 obtained in step 2.5 into the second PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is then input into the third FPB block. Here, it needs to be stacked three times to extract the features of students' head-up and head-down behaviors during class, resulting in a feature map feat3 of size 40×40×512.

[0019] In step 2.7, the feature map feat3 obtained in step 2.6 is input into the third PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is then input into the fourth FPB block to extract the features of students' head-up and head-down behaviors during class, resulting in a feature map feat4 of size 20 × 20 × 1024.

[0020] In step 2.4, the method for calculating the local feature perception multi-head attention (LFPA) is as follows: First, the feature map L2 is input into the perception network (PerceptionNetwork) to obtain the feature map pass Get vector and The vector q is obtained from the feature map L2. Finally, q, and Input into the multi-head self-attention (Multi-head Attention) for calculation, and finally get the output feature map Z. The local feature perception multi-head attention (LFPA) process is as follows Figure 4 shown.

[0021] Among them, the perception network structure is as follows Figure 5 As shown in the figure, it consists of two convolutional layers, each of which has H×W×C convolution kernels of size 3×3, where H, W, and C are the height, width, and number of channels of the feature map, respectively. The first convolutional layer is used to obtain the coordinate position offset of each element in the feature map L2. Then, the new pixel value of the element is calculated by bilinear interpolation based on the position offset. Finally, the feature map is obtained through the second convolutional layer.

[0022] Step 3 is implemented as follows:

[0023] Step 3.1, build the FPP module, perform feature fusion on the feature maps extracted in step 2, and transmit the feature maps feat1, feat2, feat3, and feat4 output in steps 2.4-2.7 to the FPP module. The FPP module structure is as follows: Figure 2 As shown in the second column, the FPP module includes three Conv2D convolutional layers, three UpSampling2D upsampling layers, six Concat layers, six CSPLayer (Cross Stage Partial Layer) in YOLOv5, and three feature-aware downsampling layers; the FPP module ultimately outputs four student behavior feature maps with different resolutions;

[0024] Step 3.2: Perform a 1×1 convolution Conv2D on the student listening state feature map feat4 generated in step 2.7 to adjust the channel and obtain the feature map P4. After upsampling UpSampling2D, the feature map P4 is fused with the feature map feat3 obtained in step 2.6 by Concat. Then, CSPLayer is used for feature extraction to obtain a feature map P4_upsample of size 40×40×512.

[0025] In step 3.3, the feature map P4_upsample obtained in step 3.2 is passed through a 1×1 convolution Conv2D to adjust the channel, obtaining feature map P3. Feature map P3 is then upsampled UpSampling2D and concatenated with the feature map feat2 obtained in step 2.5. CSPLayer is then used for feature extraction, resulting in a feature map P3_upsample of size 80×80×256.

[0026] In step 3.4, the feature map P3_upsample obtained in step 3.3 is passed through a 1×1 convolution Conv2D to adjust the channels and obtain the feature map P2. After the feature map P2 is upsampled UpSampling2D, it is concatenated with the feature map feat1 obtained in step 2.4. Then, feature extraction is performed using CSPLayer to obtain the feature map P1_out with a size of 160×160×128.

[0027] In step 3.5, the feature map P1_out obtained in step 3.4 is subjected to feature perception down-sampling (Feature Perception Down-Sampling). Feature perception down-sampling is achieved through a perception network with a convolutional layer sampling step of 2. The output of the perception network is then concatenated with the feature map P2 obtained in step 3.4. Feature extraction is then performed using CSPLayer to obtain a feature map P2_out of size 80×80×256.

[0028] In step 3.6, the feature map P2_out obtained in step 3.5 is downsampled by feature perception, and then concat-fused with the feature map P3 obtained in step 3.3. Then, CSPLayer is used for feature extraction to obtain a feature map P3_out with a size of 40×40×512.

[0029] In step 3.7, the feature map P3_out obtained in step 3.6 is downsampled by feature perception, and then concat-fused with the feature map P4 obtained in step 3.2. Then, CSPLayer is used for feature extraction to obtain a feature map P4_out with a size of 20×20×1024.

[0030] Step 4 is implemented as follows:

[0031] Step 4.1: Design the Prediction Head module in the FPA Transformer. The Prediction Head module includes four detection heads. The feature maps P1_out, P2_out, P3_out, and P4_out output from steps 3.4-3.7 are respectively fed into the four detection heads of the Prediction Head module. Each head detection head includes a regression branch, a threshold branch, and a classification branch. Each branch uses four convolutional layers for feature enhancement. Finally, each branch uses a different convolutional layer to output the convolution result. The regression branch locates the student's position at the resolution, the threshold branch calculates the score for the predicted box obtained at the resolution, and the classification branch calculates the student behavior category at the resolution, that is, the probability of looking up and looking down. Finally, non-maximum suppression (NMS) is used to obtain the final detection results of students' head-raising and head-looking behaviors in class.

[0032] In step 4.2, the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 are respectively input into the four detection heads of PredictionHead. The four convolutional layers of its regression branch perform feature enhancement. Then, a convolutional layer is used to calculate the prediction box of each student position and output the distance from each pixel in the prediction box to the left, top, right, and bottom four boundaries of GroundTruth, i.e., l, t, r, and b.

[0033] In step 4.3, the l, t, r, and b values obtained in step 4.2 are input to the threshold branch of the detection head, and the score of each pixel in the prediction box is calculated using formula (1). The score value can be used to determine the distance between the pixel and the center of the GroundTruth coordinate. The closer the pixel is to the center of the GroundTruth coordinate, the higher the score value. The score expression is as follows:

[0034]

[0035] Step 4.4, calculate the final confidence s: First, the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 are respectively input into the classification branches of the four detection heads of PredictionHead. The classification branches perform feature enhancement through four convolutional layers, and then use a convolutional layer to calculate the category corresponding to each point, that is, the probability p of looking up and looking down; then the score of each point obtained in step 4.3 and the category probability p of each point are combined through formula (2) to obtain the confidence s.

[0036]

[0037] In step 4.5, the confidence score s obtained in step 4.4 is used to remove multiple duplicate detection frames using non-maximum suppression (NMS). The detection frame with the highest confidence score is used as the final detection result to obtain the classification of students' classroom posture behaviors, i.e., looking up and looking down.

[0038] The beneficial effects of the present invention are:

[0039] (1) The method of the present invention detects and classifies students' head-raising and head-lowering behaviors in the classroom by using a Transformer (FPA Transformer) with Feature Perception Attention (FPA). The FPA Transformer structure consists of three modules: Backbone module, FPP module (Feature Perception Pyramid, FPP), and Prediction Head module: Among them, the Backbone module adds local feature perception multi-head attention (LFPA) to the FPB, so that the Backbone module pays attention to more features of students' head-raising and head-lowering behaviors in the classroom, thereby improving the ability to extract features of students' head-raising and head-lowering behaviors in the classroom; the FPP module selectively fuses the extracted student classroom behaviors by introducing Feature Perception Down-Sampling (Feature Perception Down-Sampling), thereby further enhancing its ability to obtain more detailed features of student classroom behaviors; the Prediction Head module filters low-quality detection frames by adding a threshold branch, further improving the accuracy of the FPA Transformer network in detecting and classifying students' head-raising and head-lowering behaviors in the classroom.

[0040] (2) Compared with some deep learning-based target detection networks, the method of the present invention can extract more characteristic information of students' classroom behaviors. It can not only identify small targets well, but also locate students and correctly identify their behaviors in scenarios with high density and serious overlap.

[0041] (3) Compared with general target detection algorithms, the method of the present invention designs an FPP module so that the downsampling process will not lose too much student behavior information. This not only preserves important information about student behavior, but also provides guarantees for subsequent detection, thereby improving detection performance. This method can accurately locate students and accurately classify students' listening behaviors in a classroom with dense student environments.

[0042] (4) The method of the present invention sets the classification and regression branches as independent branches in the detection stage, and shares a branch between the regression branch and the threshold branch to filter low-quality anchor frames, which can effectively solve the problem of smaller targets of students in the back row. It not only improves the positioning accuracy of smaller target students, but also improves the behavior detection accuracy of all students. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a method for detecting student posture behaviors in a smart classroom scenario according to the present invention;

[0044] Figure 2 This is the FPA Transformer structure diagram in the method for detecting student posture behaviors in a smart classroom scenario according to the present invention;

[0045] Figure 3 This is the FPB structure diagram of the Backbone module in the FPA Transformer of the present invention;

[0046] Figure 4 This is the local feature perception multi-head attention LFPA flow chart of the Backbone module in the FPA Transformer of the present invention;

[0047] Figure 5 This is the structure diagram of the Perception Network of the Backbone module in the FPA Transformer of the present invention;

[0048] Figure 6 This is the structure diagram of the Prediction Head module in the FPA Transformer of the present invention;

[0049] Figure 7 Is the original image to be detected and classified;

[0050] Figure 8 yes Figure 7 Experimental results obtained after FasterR-CNN network detection;

[0051] Figure 9 yes Figure 7 Experimental results obtained after testing with the YOLOx network;

[0052] Figure 10 yes Figure 7 Experimental results obtained after FPA Transformer detection. DETAILED DESCRIPTION

[0053] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention provides a detection method for students' classroom posture behavior in a smart classroom scenario, and inputs the prepared data set into the FPA Transformer for learning. Since the Backbone module in the FPA Transformer adopts local feature perception attention (LFPA) to form a feature perception block (FPB), the FPB can make the Backbone module pay attention to more features of the students' head-up and head-down behaviors in the classroom; then, the output of the Backbone module is input into the feature perception pyramid FPP module for feature fusion. Through feature perception downsampling, the FPP module can obtain more detailed feature information of the students' behaviors in the classroom. Finally, the feature map output from the FPP module is input into the PredictionHead module, and the low-quality detection frame is filtered through the threshold branch of the PredictionHead module, thereby improving the positioning accuracy of the FPA Transformer for the students' positions. It is implemented specifically in the following steps, such as Figure 1 shown.

[0055] Step 1: Create the SCB_dataset training dataset. Specifically, create 6,000 classroom scene images of 640×640×3 size as training samples, mark the students' head-up and head-down behaviors in each image, and create the SCB_dataset training dataset in VOC format from the marked images.

[0056] Step 2: Construct the Backbone module in the FPA Transformer. The Backbone module uses four feature perception blocks to extract the characteristics of students' head-up and head-down behaviors when listening to the class and obtain the corresponding feature maps.

[0057] Step 2 is implemented as follows:

[0058] Step 2.1, build the Backbone module. The Backbone module structure is as follows Figure 2As shown in the first column, the Backbone module includes a Patch Partition layer, a Linear Embedding layer, three Patch Merging layers, and four FPB blocks. The Patch Partition layer in the Backbone module divides the input image into blocks. The Linear Embedding layer in the Backbone module then performs a linear transformation on each pixel's channel. Four identical FPB blocks are then used to extract student classroom behavior features. Except for the first FPB block, the remaining three FPB blocks are downsampled through the Patch Merging layer. Finally, the Backbone module outputs feature maps of student behavior at four different scales.

[0059] Step 2.2, in the PatchPartition layer of the Backbone module, the image X in step 1 is divided into blocks, and every 4 adjacent pixels are a patch (X i ), and then flatten the block-wise input image in the channel direction to obtain image L1;

[0060] Step 2.3: Apply linear transformation to each pixel channel of the image L1 obtained in step 2.2 through the LinearEmbedding layer of the FPA Transformer to obtain the feature map L2.

[0061] Step 2.4, construct FPB, such as Figure 3 As shown, each FPB block structure consists of two consecutive SwinTransformer blocks, but the windowed multi-head self-attention (W-MSA) in the Swin Transformer block is replaced with local feature perception attention (LFPA). The feature map L2 obtained in step 2.3 is input into the first FPB block to extract the characteristics of students' head-up and head-down behaviors during class, resulting in a feature map feat1 of size 160×160×128.

[0062] In step 2.4, the method for calculating the local feature perception multi-head attention (LFPA) is as follows: First, the feature map L2 is input into the perception network (PerceptionNetwork) to obtain the feature map pass Get vector and The vector q is obtained from the feature map L2. Finally, q, and Input into the multi-head self-attention (Multi-head Attention) for calculation, and finally get the output feature map Z. The local feature perception multi-head attention (LFPA) process is as follows Figure 4 shown.

[0063] Among them, the perception network structure is as follows Figure 5 As shown in the figure, it consists of two convolutional layers, each of which has H×W×C (H, W, C are the height, width and number of channels of the feature map respectively) convolution kernels, and the convolution kernel size is 3×3. The first convolutional layer is used to obtain the coordinate position offset of each element in the feature map L2, and then the new pixel value of the element is calculated by bilinear interpolation based on the position offset. Finally, the feature map is obtained through the second convolutional layer.

[0064] Step 2.5: Input the feature map feat1 obtained in step 2.4 into the first PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is input into the second FPB block. The features of the students looking up and looking down while listening to the class are extracted to obtain the feature map feat2 of size 80×80×256.

[0065] Step 2.6: Input the feature map feat2 obtained in step 2.5 into the second PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is then input into the third FPB block. Here, it needs to be stacked three times to extract the features of students' head-up and head-down behaviors during class, resulting in a feature map feat3 of size 40×40×512.

[0066] In step 2.7, the feature map feat3 obtained in step 2.6 is input into the third PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is then input into the fourth FPB block to extract the features of students' head-up and head-down behaviors during class, resulting in a feature map feat4 of size 20 × 20 × 1024.

[0067] Step 3: Build the Feature Perception Pyramid (FPP) module in the FPA Transformer to perform feature fusion on the feature map extracted in step 2.

[0068] Step 3 is implemented as follows:

[0069] Step 3.1, build the FPP module, perform feature fusion on the feature maps extracted in step 2, and transmit the feature maps feat1, feat2, feat3, and feat4 output in steps 2.4-2.7 to the FPP module. The FPP module structure is as follows: Figure 2As shown in the second column, the FPP module includes three Conv2D convolutional layers, three UpSampling2D upsampling layers, six Concat layers, six CSPLayer (Cross Stage Partial Layer) in YOLOv5, and three feature-aware downsampling layers; the FPP module ultimately outputs four student behavior feature maps with different resolutions;

[0070] Step 3.2: Perform a 1×1 convolution Conv2D on the student listening state feature map feat4 generated in step 2.7 to adjust the channel and obtain the feature map P4. After upsampling UpSampling2D, the feature map P4 is fused with the feature map feat3 obtained in step 2.6 by Concat. Then, CSPLayer is used for feature extraction to obtain a feature map P4_upsample of size 40×40×512.

[0071] In step 3.3, the feature map P4_upsample obtained in step 3.2 is passed through a 1×1 convolution Conv2D to adjust the channel, obtaining feature map P3. Feature map P3 is then upsampled UpSampling2D and concatenated with the feature map feat2 obtained in step 2.5. CSPLayer is then used for feature extraction, resulting in a feature map P3_upsample of size 80×80×256.

[0072] In step 3.4, the feature map P3_upsample obtained in step 3.3 is passed through a 1×1 convolution Conv2D to adjust the channels and obtain the feature map P2. After the feature map P2 is upsampled UpSampling2D, it is concatenated with the feature map feat1 obtained in step 2.4. Then, feature extraction is performed using CSPLayer to obtain the feature map P1_out with a size of 160×160×128.

[0073] In step 3.5, the feature map P1_out obtained in step 3.4 is subjected to feature perception down-sampling (Feature Perception Down-Sampling). Feature perception down-sampling is implemented through a perception network with a convolutional layer sampling step of 2. The output of the perception network is then concatenated with the feature map P2 obtained in step 3.4. Feature extraction is then performed using CSPLayer to obtain a feature map P2_out of size 80×80×256.

[0074] In step 3.6, the feature map P2_out obtained in step 3.5 is downsampled by feature perception, and then concat-fused with the feature map P3 obtained in step 3.3. Then, CSPLayer is used for feature extraction to obtain a feature map P3_out with a size of 40×40×512.

[0075] In step 3.7, the feature map P3_out obtained in step 3.6 is downsampled by feature perception, and then concat-fused with the feature map P4 obtained in step 3.2. Then, CSPLayer is used for feature extraction to obtain a feature map P4_out with a size of 20×20×1024.

[0076] Step 4: Design the Prediction Head module in the FPA Transformer to locate the student's position on the feature map obtained in step 3 and classify the student's head-up and head-down behaviors;

[0077] Step 4 is implemented as follows:

[0078] Step 4.1, design the Prediction Head module in FPA Transformer. The Prediction Head module includes four detection heads. The Prediction Head module structure is as follows: Figure 6 As shown, the feature maps P1_out, P2_out, P3_out, and P4_out output in steps 3.4-3.7 are respectively fed into the four detection heads of the Prediction Head module. Each head detection head includes a regression branch, a threshold branch, and a classification branch. Each branch uses four convolutional layers for feature enhancement. Finally, each branch uses a different convolutional layer to output the convolution result. The regression branch locates the student's position at the resolution, the threshold branch calculates the score for the predicted box obtained at the resolution, and the classification branch calculates the probability of the student's behavior category (looking up and looking down) at the resolution. Finally, non-maximum suppression (NMS) is used to obtain the final detection results of students' looking up and looking down in class.

[0079] In step 4.2, the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 are respectively input into the four detection heads of PredictionHead. The four convolutional layers of its regression branch perform feature enhancement. Then, a convolutional layer is used to calculate the prediction box of each student position and output the distance from each pixel in the prediction box to the left, top, right, and bottom four boundaries of GroundTruth, i.e., l, t, r, and b.

[0080] In step 4.3, the l, t, r, and b values obtained in step 4.2 are input to the threshold branch of the detection head, and the score of each pixel in the prediction box is calculated using formula (1). The score value can be used to determine the distance between the pixel and the center of the GroundTruth coordinate. The closer the pixel is to the center of the GroundTruth coordinate, the higher the score value. The score expression is as follows:

[0081]

[0082] Step 4.4: Calculate the final confidence score s. First, input the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 into the classification branches of the four detection heads of PredictionHead. The classification branches perform feature enhancement through four convolutional layers, and then use a convolutional layer to calculate the category probability p corresponding to each point (head up and head down). Then, the score of each point obtained in step 4.3 and the category probability p of each point are combined to obtain the confidence score s using formula (2).

[0083]

[0084] In step 4.5, the confidence score s obtained in step 4.4 is used to remove multiple duplicate detection frames using non-maximum suppression (NMS). The detection frame with the highest confidence score is used as the final detection result to obtain the classification of students' classroom posture behaviors (raising and lowering their heads).

[0085] Step 5: Use the SCB_dateset training set from step 1 to train the FPA Transformer consisting of the Backbone module, FPP module, and PredictionHead module, and save the weight file generated by the training;

[0086] Step 6: Load the weight file trained in step 5, input the classroom image into the trained FPATransformer, and obtain the final network detection result.

[0087] like Figure 4 The Local Feature Perception Attention (LFPA) shown enables the Backbone module to perceive the characteristics of students' head-up and head-down behaviors in a local area; the LFPA in the four FPBs can perceive and extract student behavior information at all scales, while at the same time weakening the mutual influence between adjacent student behaviors and capturing more characteristic information of students' head-up and head-down behaviors during class. This makes it easier for the subsequent FPP module to efficiently extract and fuse student behavior features at different scales.

[0088] like Figure 6 As shown in the Prediction Head module structure, the present invention places classification and regression in two different branches, allowing each to perform convolution operations to extract features, and adds a threshold branch to the regression branch to filter out a large number of low-quality detection frames, thereby improving the detection accuracy of the FPA Transformer.

[0089] Table 1 Comparison of experimental results in the FPA Transformer of the present invention

[0090] network mAP50 (%) mAP60 (%) mAP70 (%) YOLOx 84.71 75.83 48.20 Faster R-CNN 74.1 62.4 36.6 FPA Transformer 87.68 82.98 63.43

[0091] As can be seen from Table 1, the accuracy of FPA Transformer is higher than that of other networks in each threshold range. FasterR-CNN is an anchor-frame-based target detection algorithm. Since FasterR-CNN requires pre-setting the anchor frame ratio, its detection effect on multi-scale, small-scale students looking up and down in the classroom is poor. YOLOx is an anchor-free detection algorithm. It can be seen that the detection accuracy has been significantly improved, but the detection accuracy generated when the threshold is 0.7 is not high, indicating that there are deficiencies in generating high-quality detection frames. FPA Transformer can achieve the highest accuracy under all threshold conditions, and when generating high-quality detection frames (when the threshold is 0.7), the accuracy is much higher than the other two networks.

[0092] Depend on Figure 7-10 As can be seen, FasterR-CNN cannot effectively detect all students in the classroom, even missing and misdetecting the first two rows, let alone detecting student behavior. YOLOx can clearly detect more students, but there are significant misdetections of behavior in the denser areas in the middle. In contrast, FPA Transformer can detect all students and accurately classify students' head-up and head-down behaviors, without errors caused by scale or crowd density, making it more suitable for detecting students' head-up and head-down behaviors in classrooms.

Claims

1. A method for detecting students' posture and behavior in a smart classroom scenario, characterized by: Specifically: Step 1, create the SCB_dataset training data set; Step 2: Construct the Backbone module in the FPA Transformer. The Backbone module uses four feature perception blocks to extract the characteristics of students' head-up and head-down behaviors when listening to the class and obtain the corresponding feature maps. Step 3: Build the feature perception pyramid module in the FPA Transformer and perform feature fusion on the feature map extracted in step 2; Step 4: Design the Prediction Head module in the FPA Transformer to locate the student's position on the feature map obtained in step 3 and classify the student's head-up and head-down behaviors; Step 5: Use the SCB_dateset training set from step 1 to train the FPA Transformer consisting of the Backbone module, FPP module, and PredictionHead module, and save the weight file generated by the training; Step 6: Load the weight file trained in step 5, input the classroom image into the trained FPATransformer, and obtain the final network detection result; Among them, the Backbone module adds local feature-aware multi-head attention to the feature-aware block FPB, so that the Backbone module pays attention to more features of students' head-raising and head-lowering behaviors in the classroom, thereby improving the ability to extract features of students' head-raising and head-lowering behaviors in the classroom; the FPP module selectively fuses the extracted features of students' classroom behaviors by introducing feature-aware downsampling, thereby further enhancing its ability to obtain more detailed features of students' classroom behaviors; the PredictionHead module filters low-quality detection frames by adding a threshold branch, further improving the accuracy of the FPA Transformer network in detecting and classifying students' head-raising and head-lowering behaviors in the classroom.

2. The method for detecting students' posture and behavior in a smart classroom scenario according to claim 1 is characterized in that: Step 1 is to create the SCB_dataset training dataset, specifically: create 6000 images We use classroom scene images of different sizes as training samples, mark the students’ head-raising and head-lowering behaviors in each image, and make the marked images into the SCB_dataset training dataset in VOC format.

3. The method for detecting students' posture and behavior in a smart classroom scenario according to claim 2 is characterized in that: Step 2 is implemented as follows: Step 2.1: Construct the Backbone module. The Backbone module includes a PatchPartition layer, a LinearEmbedding layer, three PatchMerging layers, and four FPB blocks. The PatchPartition layer in the Backbone module divides the input image into blocks, and then the LinearEmbedding layer in the Backbone module performs a linear transformation on the channel of each pixel. After that, four identical FPB blocks are used to extract student classroom behavior features. Except for the first FPB block, the remaining three FPB blocks need to be downsampled through the PatchMerging layer. Finally, the Backbone module outputs feature maps of student behavior information at four different scales. Step 2.2, in the PatchPartition layer of the Backbone module, for the image in step 1 X Divide into blocks, with every 4 adjacent pixels as a patch ( X i ), and then flatten the block-wise input image in the channel direction to obtain image L1; Step 2.3: Apply linear transformation to each pixel channel of the image L1 obtained in step 2.2 through the LinearEmbedding layer of the FPA Transformer to obtain the feature map L2. Step 2.4, construct FPB. Each FPB block structure contains two consecutive Swin Transformer Blocks, but replace the window multi-head self-attention in the Swin Transformer Block with the local feature-aware multi-head attention. Input the feature map L2 obtained in step 2.3 into the first FPB block to extract the features of students’ head-up and head-down behaviors when listening to the class, and obtain a size of Feature map feat1; Step 2.5: Input the feature map feat1 obtained in step 2.4 into the first PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is input into the second FPB block. The features of the students looking up and looking down when listening to the class are extracted to obtain a size of Feature map feat2; Step 2.6: Input the feature map feat2 obtained in step 2.5 into the second PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is input into the third FPB block. Here, it is necessary to stack 3 times to extract the features of students' head-up and head-down behaviors during class. The size is obtained. Feature map feat3; Step 2.7: Input the feature map feat3 obtained in step 2.6 into the third PatchMerging layer for downsampling. After the width and height of the feature map are halved and the depth is doubled, it is input into the fourth FPB block to extract the features of students' head-up and head-down behaviors during class. The size is obtained. Feature map feat4.

4. The method for detecting student posture behavior in a smart classroom scenario according to claim 3 is characterized in that: In step 2.4, the method for calculating the local feature perception multi-head attention is as follows: First, the feature map L2 is input into the perception network to obtain the feature map ,pass Get vector and ; and the vector q is obtained from the feature map L2; finally, q, and Input into the multi-head self-attention for calculation, and finally obtain the output feature map Z; The perception network structure consists of two convolutional layers, each of which has H×W×C convolution kernels of size 3×3, where H, W, and C are the height, width, and number of channels of the feature map, respectively. The first convolutional layer is used to obtain the coordinate position offset of each element in the feature map L2, and then the new pixel value of the element is calculated by bilinear interpolation based on the position offset. Finally, the feature map is obtained through the second convolutional layer. .

5. The method for detecting students' posture and behavior in a smart classroom scenario according to claim 4 is characterized in that: Step 3 is implemented as follows: Step 3.1: Build the FPP module and perform feature fusion on the feature maps extracted in step 2. The feature maps feat1, feat2, feat3, and feat4 output in steps 2.4-2.7 are fed into the FPP module. The FPP module includes three Conv2D convolutional layers, three UpSampling2D upsampling layers, six Concat layers, six CSPLayer layers in YOLOv5, and three feature-aware downsampling layers. The FPP module finally outputs four student behavior feature maps with different resolutions. Step 3.2: Perform a 1×1 convolution Conv2D on the student listening state feature map feat4 generated in step 2.7 to adjust the channel and obtain the feature map P4. After upsampling UpSampling2D, the feature map P4 is fused with the feature map feat3 obtained in step 2.6 by Concat. Then, CSPLayer is used for feature extraction to obtain a feature map P4_upsample of size 40×40×512. In step 3.3, the feature map P4_upsample obtained in step 3.2 is adjusted through one 1×1 convolution Conv2D to obtain the feature map P3; the feature map P3 is then upsampled UpSampling2D and fused with the feature map feat2 obtained in step 2.5 by Concat, and then feature extraction is performed using CSPLayer to obtain a feature map P3_upsample of size 80×80×256; In step 3.4, the feature map P3_upsample obtained in step 3.3 is adjusted through one 1×1 convolution Conv2D to obtain the feature map P2; the feature map P2 is then upsampled UpSampling2D and fused with the feature map feat1 obtained in step 2.4 by Concat, and then feature extraction is performed using CSPLayer to obtain a feature map P1_out of size 160×160×128; In step 3.5, the feature map P1_out obtained in step 3.4 is subjected to feature-aware downsampling. Feature-aware downsampling is achieved through a perceptual network with a convolutional layer sampling step of 2. The output of the perceptual network is then concatenated with the feature map P2 obtained in step 3.4, and then feature extraction is performed using CSPLayer to obtain a feature map P2_out of size 80×80×256. In step 3.6, the feature map P2_out obtained in step 3.5 is downsampled by feature perception, and then concat-fused with the feature map P3 obtained in step 3.

3. Then, CSPLayer is used for feature extraction to obtain a feature map P3_out with a size of 40×40×512. In step 3.7, the feature map P3_out obtained in step 3.6 is downsampled by feature perception, and then concat-fused with the feature map P4 obtained in step 3.

2. Then, CSPLayer is used for feature extraction to obtain a feature map P4_out with a size of 20×20×1024.

6. The method for detecting student posture behavior in a smart classroom scenario according to claim 5 is characterized in that: Step 4 is implemented as follows: Step 4.1: Design the Prediction Head module in the FPA Transformer. The Prediction Head module includes four detection heads. The feature maps P1_out, P2_out, P3_out, and P4_out output in steps 3.4-3.7 are respectively transmitted to the four detection heads of the Prediction Head module. Each Head detection head includes a regression branch, a threshold branch, and a classification branch. Each branch uses four convolutional layers for feature enhancement. Finally, each branch uses a different convolutional layer to output the convolution result. The regression branch locates the student position at the resolution, the threshold branch calculates the score of the prediction box obtained at the resolution, and the classification branch calculates the student behavior category at the resolution, that is, the probability of looking up and looking down. Finally, the final detection results of students' head-raising and head-looking behaviors in class are obtained by non-maximum suppression. In step 4.2, the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 are respectively input into the four detection heads of the Prediction Head, and the four convolutional layers of the regression branch are used for feature enhancement. Then, a convolutional layer is used to calculate the prediction box of each student position, and the distance from each pixel in the prediction box to the left, top, right, and bottom four boundaries of GroundTruth is output, i.e., l, t, r, b; Step 4.3: Input the l, t, r, and b values obtained in step 4.2 into the threshold branch of the detection head, and calculate the coordinate score of each pixel in the prediction box using formula (1); The score value is used to determine the distance between the pixel point and the center of the GroundTruth coordinate. The closer the pixel point is to the center of the GroundTruth coordinate, the higher the score value. The score expression is as follows: Step 4.4, calculate the final confidence score s: First, the feature maps P1_out, P2_out, P3_out, and P4_out obtained in steps 3.4-3.7 are respectively input into the classification branches of the four detection heads of the Prediction Head. The classification branch performs feature enhancement through four convolutional layers, and then uses a convolutional layer to calculate the probability p of each point corresponding to the category, that is, looking up and looking down; Then, the score of each point obtained in step 4.3 and the category probability p of each point are combined to obtain the confidence s through formula (2): In step 4.5, the confidence score s obtained in step 4.4 is used to remove multiple duplicate detection frames using non-maximum suppression. The detection frame with the highest confidence score is used as the final detection result to obtain the classification of students' classroom posture behaviors, i.e., looking up and looking down.

Images