Real-time classroom behavior recognition method based on self-attention and multi-scale feature aggregation

By introducing self-attention and multi-scale aggregation technology in classroom behavior recognition, students' classroom behavior recognition model is optimized, and the accuracy and speed of deep learning in classroom behavior recognition is solved, and more efficient behavior recognition and feedback is achieved.

CN118675226BActive Publication Date: 2024-11-15WUHAN TEXTILE UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410676531.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-28
Publication Date
2024-11-15
Estimated Expiration
2044-05-28

AI Technical Summary

Technical Problem

Deep learning has accuracy problems in classroom behavior recognition, especially when distinguishing between appearance-like behaviors and processing continuous dynamic behaviors, and the model processing speed is not enough to achieve real-time feedback.

Method used

Using a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, a specific loss function is used by designing a student classroom behavior recognition model, including data preprocessing, feature extraction, channel space enhancement, multi-scale feature aggregation and behavior prediction modules. and structures such as channel attention, spatial attention and multi-scale feature aggregation to optimize model performance.

Benefits of technology

It improves the accuracy and speed of classroom behavior recognition, can more accurately identify and locate student behavior, reduce missed detection, and enhances the robustness of the model and the ability to process timing information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675226B_ABST
    Figure CN118675226B_ABST
Patent Text Reader

Abstract

The present invention relates to a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, and the real-time classroom behavior recognition method specifically comprises the following steps: S1. Designing a student classroom behavior recognition model suitable for real-time classroom behavior recognition; the student classroom behavior recognition model sequentially comprises a data preprocessing module, a feature extraction module, a channel space enhancement module, a multi-scale feature aggregation module and a behavior prediction module; S2. Designing a loss function used in the process of training the student classroom behavior recognition model, and training the student classroom behavior recognition model designed in step S1; S3. Using the student classroom behavior recognition model trained in step S2 to parse real-time video and generate behavior prediction results. By optimizing the model and the loss function, the accuracy and speed of classroom behavior recognition are significantly improved, and a method for efficiently parsing classroom behavior is provided, which greatly saves labor costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of classroom behavior recognition, and in particular to a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation. Background Art

[0002] Classroom behavior recognition methods are an important application in the field of educational technology. For example, in scenarios such as smart classrooms and remote education monitoring, students' behaviors need to be recognized in real time to evaluate their engagement and learning outcomes. In scenarios such as smart classrooms, real-time performance and accuracy are critical to improving teaching quality and student experience.

[0003] With the rapid development of computer vision and artificial intelligence, classroom behavior recognition technology based on deep learning has become the forefront of educational technology innovation. This method uses a complex algorithm network to automatically identify and extract students' behavior patterns from classroom videos or images, such as key indicators such as concentration, participation, and interactivity. These models are trained with a large amount of data and can accurately identify students' behaviors such as raising their hands, writing, reading, and even distraction, thereby providing real-time feedback to teachers.

[0004] However, deep learning still has some shortcomings in classroom behavior recognition. For example, the model may have accuracy issues in distinguishing different types of behaviors, because some behaviors may be very similar in appearance, making it difficult for the model to accurately distinguish them. Secondly, deep learning models are more challenging to recognize continuous dynamic behaviors, because students' behaviors are continuous and dynamic, and traditional models have difficulty effectively processing this timing information. In addition, deep learning models still have limitations in processing timing information and long-term dependencies, which may affect the model's understanding and accurate recognition of dynamic changes in behavior. Finally, for classroom environments that require real-time feedback, the processing speed of the model needs to be improved to achieve timely behavior recognition and feedback.

[0005] The Chinese patent with publication number CN113158880A discloses "a method for identifying student classroom behaviors based on deep learning", which uses YOLOv3 target detection and EDSR super-resolution methods to extract and enhance student images in classroom environments. This method can help us better analyze and understand students' behaviors, while providing clearer and more detailed image data, providing a more reliable basis for subsequent classroom behavior identification and analysis. Although this method has made some progress in identifying students' classroom behaviors, it has certain limitations when processing time series information and long-term dependencies, making it difficult to fully capture the dynamic changes of behaviors, and has low recognition accuracy for some behaviors that look similar. This may result in inaccurate recognition results for classroom behaviors or the inability to provide timely feedback. Summary of the invention

[0006] In view of this, the present invention provides a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, the purpose of which is to effectively identify behavioral information in real-time classroom videos through self-attention and multi-scale aggregation, improve the accuracy and speed of classroom behavior recognition, and provide a method for efficiently parsing classroom behaviors, which greatly saves labor costs.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, the real-time classroom behavior recognition method specifically comprises the following steps:

[0009] S1. Design a student classroom behavior recognition model suitable for real-time classroom behavior recognition;

[0010] The student classroom behavior recognition model includes a data preprocessing module, a feature extraction module, a channel space enhancement module, a multi-scale feature aggregation module and a behavior prediction module in sequence;

[0011] S2. Design a loss function used in the process of training the student classroom behavior recognition model, and train the student classroom behavior recognition model designed in step S1;

[0012] The loss function is composed of the feature extraction module loss function And the behavior prediction module loss function composition;

[0013] The feature extraction module loss function The details are as follows:

[0014] ;

[0015] in, is the positioning loss function, which is used to enable the model to accurately locate the target bounding box; is the classification loss function, which measures the difference between the model prediction value and the actual label; and is a hyperparameter used to control the weights of localization loss and classification loss;

[0016] The behavior prediction module loss function The details are as follows:

[0017] ;

[0018] in, represents the number of samples; i Representative i samples; Representative i The true labels of samples; Representative i The predicted probability of samples; Represents the regularization term, which is used to control the complexity of the model and prevent overfitting; is the regularization term The weight parameter of represents the bounding box regression loss function, which is used to optimize the bounding box position prediction in the object detection task; is the bounding box regression loss function The weight parameter of .

[0019] S3. Use the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results.

[0020] Furthermore, the step S1 designs a student classroom behavior recognition model suitable for real-time classroom behavior recognition as follows:

[0021] S1-1. Design a data preprocessing module, obtain the original image frames from the video stream through the data preprocessing module, and preprocess the obtained original image frames to form a data set, and divide the data set into a training set and a validation set according to an 8:2 ratio;

[0022] S1-2. Design a feature extraction module, through which the image frames in the training set and the validation set are sent to the cross-stage fusion network for feature extraction, and high-level features and low-level features of the image frame data in the training set and the validation set are extracted;

[0023] S1-3. Design a channel space enhancement module, through the channel space enhancement module to reduce the weight of the background information extracted in step S1-2 features, so that the weight of the behavior information extracted in step S1-2 features is enhanced;

[0024] S1-4. Design a multi-scale feature aggregation module to fuse features of different modalities;

[0025] S1-5. Design a behavior prediction module, input the features fused in step S1-4 into the behavior prediction module, and obtain the final prediction result.

[0026] Furthermore, the feature extraction module includes 1 convolution layer and 3 residual structures. The specific structure of the feature extraction module is as follows:

[0027] The image frame output by the data preprocessing module is input into a convolution layer for convolution operation, wherein the convolution kernel size of the convolution layer is 3×3, the stride of the convolution operation is 2, and the padding is 1;

[0028] Then enter the first residual structure, which includes two convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0029] Then enter the second residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0030] Finally, we enter the third residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0031] Furthermore, the channel space enhancement module is composed of a channel attention module and a spatial attention module. The specific structure of the channel space enhancement module is as follows:

[0032] On the one hand, the channel attention module processes the feature map output by the feature extraction module in the channel domain. The specific steps are as follows:

[0033] The channel attention module first performs maximum pooling and average pooling on the feature maps output by the feature extraction module, compresses the feature maps based on two dimensions, and obtains two feature descriptions of different dimensions;

[0034] The feature map after maximum pooling and average pooling is input into a shared full connection. The shared full connection first reduces the number of channels through the first fully connected layer, and then restores the number of channels through the second fully connected layer. The feature map after shared full connection is output at the maximum and average respectively, and then the two feature maps are stacked in the channel dimension. The weight of each channel of the feature map is normalized to between 0 and 1 through the sigmoid activation function;

[0035] On the other hand, the spatial attention module also processes the feature map output by the feature extraction module in the spatial domain. The specific steps are as follows:

[0036] First, the feature map output by the feature extraction module is subjected to maximum pooling and average pooling in the spatial dimension, and the two feature maps after maximum pooling and average pooling are spliced ​​in the spatial dimension;

[0037] Next, a 3×3 convolution kernel is used to fuse the spatial information. The stride of the convolution kernel is 2 and the padding is 1, so that the size of the feature map changes.

[0038] Then the convolution result is normalized to the range of 0-1 by the sigmoid activation function.

[0039] Finally, the feature maps processed by the channel attention module and the spatial attention module are concatenated.

[0040] Furthermore, the specific structure of the multi-scale feature aggregation module is as follows:

[0041] S1-4-1. First, the feature map output by the channel space enhancement module is convolved through two convolutional layers, with the convolution kernel size of 3×3 and the step size of 2 to extract key features;

[0042] S1-4-2. Next, a convolution operation is performed through a convolution layer with a convolution kernel size of 3×3 and a step size of 4, and the output result is divided into two branches;

[0043] S1-4-3. Process one branch in step S1-4-2 through the first 1×1 convolutional layer to retain fine-grained information; at the same time, pass the other branch in step S1-4-2 through a convolutional layer with a kernel size of 3×3 and a stride of 8 to capture larger-scale features, and divide its output into two branches;

[0044] S1-4-4. Process one branch in step S1-4-3 through a second 1×1 convolutional layer to further refine features; at the same time, pass the other branch in step S1-4-3 through a convolutional layer with a kernel size of 3×3 and a step size of 16 to capture broader context information, and divide its output into two branches;

[0045] S1-4-5. Process one branch in step S1-4-4 through the third 1×1 convolution layer to enhance feature expression; at the same time, pass the other branch in step S1-4-4 through a convolution layer with a convolution kernel size of 3×3 and a step size of 32 to enhance the receptive field, and process its output result through the fourth 1×1 convolution layer;

[0046] S1-4-6. Finally, concatenate the results after the first 1×1 convolution layer and the second 1×1 convolution layer and input them into a 3×3 convolution layer; concatenate the results after the third 1×1 convolution layer and the fourth 1×1 convolution layer and input them into another 3×3 convolution layer; and concatenate the results after the two 3×3 convolution layers to achieve the final aggregation of features.

[0047] Furthermore, the behavior prediction module includes a convolutional layer and a recurrent neural network. The specific structure of the behavior prediction module is as follows:

[0048] The feature map output by the multi-scale feature aggregation module is input into a convolutional layer with a convolution kernel size of 3×3, a stride of 2, and a padding of 1.

[0049] The output of the convolutional layer is processed through a recurrent neural network to obtain the final prediction result.

[0050] Furthermore, the positioning loss function The details are as follows:

[0051] ;

[0052] in, represents the predicted bounding box; represents the true bounding box; Represents the degree of overlap between the predicted bounding box and the true bounding box.

[0053] Furthermore, the classification loss function The details are as follows:

[0054] ;

[0055] in, represents the number of samples; i Representative i samples; Representative i The true labels of samples; Representative i The probability of predicting a sample.

[0056] Furthermore, the bounding box regression loss function loses The details are as follows:

[0057] ;

[0058] in, Represents the horizontal coordinate of the center point of the predicted bounding box; Represents the ordinate of the center point of the predicted bounding box; Indicates the width of the predicted bounding box; Represents the height of the predicted bounding box; Represents the horizontal coordinate of the center point of the real bounding box; Represents the ordinate of the center point of the true bounding box, represents the width of the true bounding box, Indicates the height of the ground-truth bounding box.

[0059] Furthermore, S3 uses the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results, as follows:

[0060] By loading the pre-trained student classroom behavior recognition model, the classroom behavior video to be analyzed is preprocessed and model inference is performed to generate behavior prediction results.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] (1) By designing a channel-space enhancement module, we realize a sequential attention structure from channel to space from two analysis dimensions: spatial attention and channel attention, thereby eliminating some unnecessary information, such as background information, and optimizing the performance of the model, thereby effectively solving the problem of inaccurate recognition of continuous dynamic behavior.

[0063] (2) The features of different modalities are fused through the multi-scale feature aggregation module, and a feature pyramid structure of different sizes is constructed, thereby improving the accuracy and performance of the model, enabling the model to comprehensively utilize multiple types of information, and helping the model to process temporal information and long-term dependencies.

[0064] (3) By designing specific loss functions, including the feature extraction module loss function and the behavior prediction module loss function, the feature extraction module loss function includes the positioning loss function and the classification loss function. The positioning loss function enables the model to accurately locate the student's behavior by locating the target bounding box, reducing the number of missed detections and improving the accuracy and robustness of the model. The classification loss function can help the model more accurately distinguish different classroom behaviors and improve the model's ability to recognize targets of different categories. The behavior prediction module loss function includes the bounding box regression loss function. The bounding box loss function adjusts the predicted bounding box to enable the model to more accurately locate the area where the student's behavior occurs, thereby improving the model's accuracy in identifying classroom behaviors.

[0065] Other features and advantages of the present invention will be described in the following description, and partly become obvious from the description, or be understood by implementing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A schematic diagram of the classroom behavior recognition model structure of a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation provided by an embodiment of the present invention is shown;

[0068] Figure 2A schematic diagram of a channel space enhancement module of a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation provided by an embodiment of the present invention is shown;

[0069] Figure 3 A schematic diagram of a multi-scale feature aggregation module of a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The present invention provides a real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, and the real-time classroom behavior recognition method specifically comprises the following steps:

[0072] S1. Design a student classroom behavior recognition model suitable for real-time classroom behavior recognition;

[0073] As attached Figure 1 As shown, the student classroom behavior recognition model in step S1 includes a data preprocessing module, a feature extraction module, a channel space enhancement module, a multi-scale feature aggregation module and a behavior prediction module in sequence;

[0074] S2. Design a loss function used in the process of training the student classroom behavior recognition model, and train the student classroom behavior recognition model designed in step S1;

[0075] S3. Use the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results.

[0076] Step S1 designs a student classroom behavior recognition model suitable for real-time classroom behavior recognition as follows:

[0077] S1-1. Design a data preprocessing module, obtain the original image frames from the video stream through the data preprocessing module, and preprocess the obtained original image frames to form a data set, and divide the data set into a training set and a validation set according to an 8:2 ratio;

[0078] S1-2. Design a feature extraction module, through which the image frames in the training set and the validation set are fed into the cross-stage fusion network for feature extraction, and high-level and low-level features of the image frame data in the training set and the validation set are extracted, which helps to distinguish the appearance of similar behaviors;

[0079] S1-3. Design a channel space enhancement module, which reduces the weight of the background information in the features extracted in step S1-2 through the channel space enhancement module, so that the weight of the behavior information in the features extracted in step S1-2 is enhanced, which effectively solves the problem of inaccurate recognition of continuous dynamic behavior;

[0080] S1-4. Design a multi-scale feature aggregation module to fuse features of different modalities, so that the model can comprehensively utilize multiple information, which helps the model process temporal information and long-term dependencies;

[0081] S1-5. Design a behavior prediction module, input the features fused in step S1-4 into the behavior prediction module, and obtain the final prediction result.

[0082] The feature extraction module includes 1 convolution layer and 3 residual structures. The specific structure of the feature extraction module is as follows:

[0083] The image frame output by the data preprocessing module is input into a convolution layer for convolution operation, wherein the convolution kernel size of the convolution layer is 3×3, the stride of the convolution operation is 2, and the padding is 1;

[0084] Then enter the first residual structure, which includes two convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0085] Then enter the second residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0086] Finally, we enter the third residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

[0087] The channel space enhancement module is composed of a channel attention module and a space attention module. By introducing two analysis dimensions, channel attention and space attention, a sequential attention structure from channel to space is realized. Figure 2 As shown, the specific structure of the channel space enhancement module is as follows:

[0088] On the one hand, the channel attention module processes the feature map output by the feature extraction module in the channel domain. The specific steps are as follows:

[0089] The channel attention module first performs maximum pooling and average pooling on the feature maps output by the feature extraction module, compresses the feature maps based on two dimensions, and obtains two feature descriptions of different dimensions;

[0090] The feature map after maximum pooling and average pooling is input into a shared full connection. The shared full connection first reduces the number of channels through the first fully connected layer, and then restores the number of channels through the second fully connected layer. The feature map after shared full connection is output at the maximum and average respectively, and then the two feature maps are stacked in the channel dimension. The weight of each channel of the feature map is normalized to between 0 and 1 through the sigmoid activation function;

[0091] On the other hand, the spatial attention module also processes the feature map output by the feature extraction module in the spatial domain. The specific steps are as follows:

[0092] First, the feature map output by the feature extraction module is subjected to maximum pooling and average pooling in the spatial dimension, and the two feature maps after maximum pooling and average pooling are spliced ​​in the spatial dimension;

[0093] Next, a 3×3 convolution kernel is used to fuse the spatial information. The stride of the convolution kernel is 2 and the padding is 1, so that the size of the feature map changes. Specifically, the height and width of the feature map are halved. For example, the size of the feature map changes from 256×256 to 128×128.

[0094] The convolution result is then normalized to a value between 0 and 1 using a sigmoid activation function.

[0095] Finally, the feature maps processed by the channel attention module and the spatial attention module are concatenated.

[0096] By designing a channel space enhancement module, a sequential attention structure from channel to space is realized from the two analysis dimensions of spatial attention and channel attention, thereby excluding some unnecessary information, such as background information, and optimizing the performance of the model, thereby effectively solving the problem of inaccurate recognition of continuous dynamic behavior.

[0097] The multi-scale feature aggregation module introduces a path aggregation mechanism to better capture and integrate multi-scale information. Figure 3 As shown, the specific structure of the multi-scale feature aggregation module is as follows:

[0098] The feature map output by the channel space enhancement module is used as input, and a series of convolution operations are used to process the features.

[0099] S1-4-1. First, the feature map output by the channel space enhancement module is convolved through two convolutional layers, with the convolution kernel size of 3×3 and the step size of 2 to extract key features;

[0100] S1-4-2. Next, a convolution operation is performed through a convolution layer with a convolution kernel size of 3×3 and a step size of 4, and the output result is divided into two branches;

[0101] S1-4-3. Process one branch in step S1-4-2 through the first 1×1 convolutional layer to retain fine-grained information; at the same time, pass the other branch in step S1-4-2 through a convolutional layer with a kernel size of 3×3 and a stride of 8 to capture larger-scale features, and divide its output into two branches;

[0102] S1-4-4. Process one branch in step S1-4-3 through a second 1×1 convolutional layer to further refine features; at the same time, pass the other branch in step S1-4-3 through a convolutional layer with a kernel size of 3×3 and a step size of 16 to capture broader context information, and divide its output into two branches;

[0103] S1-4-5. Process one branch in step S1-4-4 through the third 1×1 convolution layer to enhance feature expression; at the same time, pass the other branch in step S1-4-4 through a convolution layer with a convolution kernel size of 3×3 and a step size of 32 to enhance the receptive field, and process its output result through the fourth 1×1 convolution layer;

[0104] S1-4-6. Finally, concatenate the results after the first 1×1 convolution layer and the second 1×1 convolution layer and input them into a 3×3 convolution layer; concatenate the results after the third 1×1 convolution layer and the fourth 1×1 convolution layer and input them into another 3×3 convolution layer; and concatenate the results after the two 3×3 convolution layers to achieve the final aggregation of features.

[0105] Through the multi-scale feature aggregation module, features of different modalities are fused and feature pyramid structures of different sizes are constructed, thereby improving the accuracy and performance of the model, enabling the model to comprehensively utilize multiple information, and helping the model to process temporal information and long-term dependencies.

[0106] The behavior prediction module includes a convolutional layer and a recurrent neural network. The specific structure of the behavior prediction module is as follows:

[0107] The feature map output by the multi-scale feature aggregation module is input into a convolutional layer with a convolution kernel size of 3×3, a stride of 2, and a padding of 1.

[0108] The output of the convolutional layer is processed through a recurrent neural network to obtain the final prediction result.

[0109] The loss function in step S2 is composed of the feature extraction module loss function And the behavior prediction module loss function composition.

[0110] The feature extraction module loss function The details are as follows:

[0111] ;

[0112] in, It is a positioning loss function that allows the model to more accurately locate the target bounding box, thereby improving the performance of target detection; is the classification loss function, which measures the difference between the model prediction value and the actual label; and is a hyperparameter used to control the weights of localization loss and classification loss.

[0113] The positioning loss function The details are as follows:

[0114] ;

[0115] in, represents the predicted bounding box; represents the true bounding box; Represents the degree of overlap between the predicted bounding box and the true bounding box.

[0116] The classification loss function The details are as follows:

[0117] ;

[0118] in, represents the number of samples; i Representative i samples; Representative i The true labels of samples; Representative i The probability of predicting a sample.

[0119] The behavior prediction module loss function The details are as follows:

[0120] ;

[0121] in, represents the number of samples; iRepresentative i samples; Representative i The true labels of samples; Representative i The predicted probability of samples; Represents the regularization term, which is used to control the complexity of the model and prevent overfitting; is the regularization term The weight parameter of represents the bounding box regression loss function, which is used to optimize the bounding box position prediction in the object detection task; is the bounding box regression loss function The weight parameter of .

[0122] Bounding box regression loss The details are as follows:

[0123] ;

[0124] in, Represents the horizontal coordinate of the center point of the predicted bounding box; Represents the ordinate of the center point of the predicted bounding box; Indicates the width of the predicted bounding box; Represents the height of the predicted bounding box; Represents the horizontal coordinate of the center point of the real bounding box; Represents the ordinate of the center point of the true bounding box, represents the width of the true bounding box, Indicates the height of the ground-truth bounding box.

[0125] By designing specific loss functions, including the feature extraction module loss function and the behavior prediction module loss function, the feature extraction module loss function includes the positioning loss function and the classification loss function. The positioning loss function enables the model to accurately locate the student's behavior by locating the target bounding box, reduces the possibility of missed detection, and improves the accuracy and robustness of the model. The classification loss function can help the model more accurately distinguish different classroom behaviors and improve the model's ability to recognize targets of different categories. The behavior prediction module loss function includes the bounding box regression loss function. The bounding box loss function adjusts the predicted bounding box to enable the model to more accurately locate the area where the student's behavior occurs, thereby improving the model's accuracy in identifying classroom behaviors.

[0126] S3 uses the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results, as follows:

[0127] By loading the pre-trained student classroom behavior recognition model, the classroom behavior video to be analyzed is preprocessed and model inference is performed to generate behavior prediction results.

[0128] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time classroom behavior recognition method based on self-attention and multi-scale aggregation, characterized in that: The real-time classroom behavior recognition method specifically comprises the following steps: S1. Design a student classroom behavior recognition model suitable for real-time classroom behavior recognition; The student classroom behavior recognition model includes a data preprocessing module, a feature extraction module, a channel space enhancement module, a multi-scale feature aggregation module and a behavior prediction module in sequence; S2. Design a loss function used in the process of training the student classroom behavior recognition model, and train the student classroom behavior recognition model designed in step S1; The loss function is composed of the feature extraction module loss function And the behavior prediction module loss function composition; The feature extraction module loss function The details are as follows: ; in, is the positioning loss function, which is used to enable the model to accurately locate the target bounding box; is the classification loss function, which measures the difference between the model prediction value and the actual label; and is a hyperparameter used to control the weights of localization loss and classification loss; The behavior prediction module loss function The details are as follows: ; in, represents the number of samples; i Representative i samples; Representative i The true labels of samples; Representative i The predicted probability of samples; Represents the regularization term, which is used to control the complexity of the model and prevent overfitting; is the regularization term The weight parameter of represents the bounding box regression loss function, which is used to optimize the bounding box position prediction in the object detection task; is the bounding box regression loss function The weight parameter of The positioning loss function The details are as follows: ; in, represents the predicted bounding box; represents the true bounding box; Represents the degree of overlap between the predicted bounding box and the true bounding box; The classification loss function The details are as follows: ; in, represents the number of samples; i Representative i samples; Representative i The true labels of samples; Representative i The probability of sample prediction; The bounding box regression loss function The details are as follows: ; in, Represents the horizontal coordinate of the center point of the predicted bounding box; Represents the ordinate of the center point of the predicted bounding box; Indicates the width of the predicted bounding box; Represents the height of the predicted bounding box; Represents the horizontal coordinate of the center point of the real bounding box; Represents the ordinate of the center point of the true bounding box, represents the width of the true bounding box, Indicates the height of the true bounding box; S3. Use the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results.

2. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 1, characterized in that: The step S1 designs a student classroom behavior recognition model suitable for real-time classroom behavior recognition as follows: S1-1. Design a data preprocessing module, obtain the original image frames from the video stream through the data preprocessing module, and preprocess the obtained original image frames to form a data set, and divide the data set into a training set and a validation set according to an 8:2 ratio; S1-2. Design a feature extraction module, through which the image frames in the training set and the validation set are sent to the cross-stage fusion network for feature extraction, and high-level features and low-level features of the image frame data in the training set and the validation set are extracted; S1-3. Design a channel space enhancement module, through the channel space enhancement module to reduce the weight of the background information extracted in step S1-2 features, so that the weight of the behavior information extracted in step S1-2 features is enhanced; S1-4. Design a multi-scale feature aggregation module to fuse features of different modalities; S1-5. Design a behavior prediction module, input the features fused in step S1-4 into the behavior prediction module, and obtain the final prediction result.

3. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 2, characterized in that: The feature extraction module includes 1 convolution layer and 3 residual structures. The specific structure of the feature extraction module is as follows: The image frame output by the data preprocessing module is input into a convolution layer for convolution operation, wherein the convolution kernel size of the convolution layer is 3×3, the stride of the convolution operation is 2, and the padding is 1; Then enter the first residual structure, which includes two convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1. Then enter the second residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1. Finally, we enter the third residual structure, which includes three convolution kernels and a batch normalization. The convolution kernel size is 3×3, the stride of the convolution operation is 2, and the padding is 1.

4. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 2, characterized in that: The channel space enhancement module is composed of a channel attention module and a spatial attention module. The specific structure of the channel space enhancement module is as follows: On the one hand, the channel attention module processes the feature map output by the feature extraction module in the channel domain. The specific steps are as follows: The channel attention module first performs maximum pooling and average pooling on the feature maps output by the feature extraction module, compresses the feature maps based on two dimensions, and obtains two feature descriptions of different dimensions; The feature map after maximum pooling and average pooling is input into a shared full connection. The shared full connection first reduces the number of channels through the first fully connected layer, and then restores the number of channels through the second fully connected layer. The feature map after shared full connection is output at the maximum and average respectively, and then the two feature maps are stacked in the channel dimension. The weight of each channel of the feature map is normalized to between 0 and 1 through the sigmoid activation function; On the other hand, the spatial attention module also processes the feature map output by the feature extraction module in the spatial domain. The specific steps are as follows: First, the feature map output by the feature extraction module is subjected to maximum pooling and average pooling in the spatial dimension, and the two feature maps after maximum pooling and average pooling are spliced ​​in the spatial dimension; Next, a 3×3 convolution kernel is used to fuse the spatial information. The stride of the convolution kernel is 2 and the padding is 1, so that the size of the feature map changes. Then the convolution result is normalized to the range of 0-1 by the sigmoid activation function. Finally, the feature maps processed by the channel attention module and the spatial attention module are concatenated.

5. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 2, characterized in that: The specific structure of the multi-scale feature aggregation module is as follows: S1-4-1. First, the feature map output by the channel space enhancement module is convolved through two convolutional layers, with the convolution kernel size of 3×3 and the step size of 2 to extract key features; S1-4-2. Next, a convolution operation is performed through a convolution layer, the convolution kernel size is 3×3, the step length is 4, and the output result is divided into two branches; S1-4-3. Process one branch in step S1-4-2 through the first 1×1 convolutional layer to retain fine-grained information; at the same time, pass the other branch in step S1-4-2 through a convolutional layer with a kernel size of 3×3 and a stride of 8 to capture larger-scale features, and divide its output into two branches; S1-4-4. Process one branch in step S1-4-3 through a second 1×1 convolutional layer to further refine features; at the same time, pass the other branch in step S1-4-3 through a convolutional layer with a kernel size of 3×3 and a step size of 16 to capture broader context information, and divide its output into two branches; S1-4-5. Process one branch in step S1-4-4 through the third 1×1 convolution layer to enhance feature expression; at the same time, pass the other branch in step S1-4-4 through a convolution layer with a convolution kernel size of 3×3 and a step size of 32 to enhance the receptive field, and process its output result through the fourth 1×1 convolution layer; S1-4-6. Finally, concatenate the results after the first 1×1 convolution layer and the second 1×1 convolution layer and input them into a 3×3 convolution layer; concatenate the results after the third 1×1 convolution layer and the fourth 1×1 convolution layer and input them into another 3×3 convolution layer; and concatenate the results after the two 3×3 convolution layers to achieve the final aggregation of features.

6. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 2, characterized in that: The behavior prediction module includes a convolutional layer and a recurrent neural network. The specific structure of the behavior prediction module is as follows: The feature map output by the multi-scale feature aggregation module is input into a convolutional layer with a convolution kernel size of 3×3, a stride of 2, and a padding of 1. The output of the convolutional layer is processed through a recurrent neural network to obtain the final prediction result.

7. The real-time classroom behavior recognition method based on self-attention and multi-scale aggregation as claimed in claim 1, characterized in that: S3 uses the student classroom behavior recognition model trained in step S2 to parse the real-time video and generate behavior prediction results, as follows: By loading the pre-trained student classroom behavior recognition model, the classroom behavior video to be analyzed is preprocessed and model inference is performed to generate behavior prediction results.