An end-to-end student behavior detection method and system based on a large convolution kernel

Abstract

The present application belongs to but is not limited to the technical field of computer vision, and particularly relates to an end-to-end student behavior detection method and system based on a large convolution kernel, comprising: preprocessing of classroom video data to obtain a preliminary feature representation of student behavior; designing a large convolution kernel backbone network to perform deep convolution operation on the extracted image features to obtain an upstream feature representation with a larger receptive field, and combining a multi-scale feature fusion module to fuse behavior features of different scales; introducing a coordinate attention mechanism to filter invalid information in the feature map and enhance the attention of the model to key behaviors; inputting the fused features into a Transformer decoder for global behavior prediction; optimizing model parameters by constructing a joint loss function to output frame-level prediction results of the final classroom behavior and accurately identify the type of student behavior in the classroom.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of computer vision technology, and particularly relates to an end-to-end student behavior detection method and system based on large convolutional kernels. Background Technology

[0002] With the development of educational informatization, modern teaching models are gradually shifting from traditional face-to-face teaching to digitalization and intelligence. The widespread adoption of smart classrooms and online education platforms has enabled classroom behavior analysis to move from manual observation to in-depth mining of large-scale video data. Student behavior monitoring occupies an important position in modern educational research and practice. In-depth exploration in this field can provide a new perspective for understanding and optimizing the teaching process, and has profound significance for improving teaching quality and promoting students' all-round development.

[0003] Precise student behavior monitoring can reveal various behavioral patterns in the classroom, quantify students' focus, participation, and interaction with teachers, thus providing educators with objective data to guide the development and adjustment of teaching strategies, achieve individualized instruction, and improve teaching effectiveness. Furthermore, behavioral monitoring can help education administrators understand the overall learning atmosphere of the classroom, identify potential teaching problems and students' psychological states, which is crucial for building a positive learning environment and enhancing students' learning motivation.

[0004] However, facing the transformation of education towards digitalization, relying solely on manual methods to analyze and evaluate these complex and ever-changing classroom behavior data proves inadequate. This not only requires a significant investment of time and effort but is also inefficient and susceptible to subjective judgment. Therefore, developing an efficient, accurate, and automated student behavior detection method is crucial. Currently, automated student behavior detection methods mainly fall into two categories: detection methods based on traditional machine learning and detection methods based on deep learning. Traditional machine learning methods typically include steps such as feature extraction, classifier training, and behavior recognition. While these methods can achieve good results in specific scenarios, they often fall short when dealing with complex classroom environments, especially when faced with occlusion, ambiguity, and diverse behavioral features, where detection accuracy and stability are problematic. Furthermore, traditional one-stage and two-stage object detection algorithms often involve complex computational operations in post-processing steps. These complex computational operations include Non-Maximum Suppression (NMS), used to filter redundant detection boxes and accurately locate targets. However, these steps typically require substantial computational resources, increasing the time complexity and computational cost of the detection process. Furthermore, these algorithms often require additional adjustments and optimizations when handling multi-scale, multi-class targets, further increasing the system's complexity and computational burden. This not only affects detection speed and limits real-time applications, but also reduces the algorithm's versatility and robustness to some extent, especially when facing dynamically changing and complex environments, where the post-processing steps of traditional algorithms can become a performance bottleneck.

[0005] Extracting valuable student behavior information from a large amount of classroom video data is no easy task. Raw classroom video data often contains various occlusion phenomena and complex interaction patterns between students and teachers, all of which pose significant challenges to effectively analyzing and understanding student behavior.

[0006] Based on the above analysis, the urgent technical problems that need to be solved in the existing technology are:

[0007] (1) The lack of automation technology to assist teachers in analysis resulted in low efficiency.

[0008] (2) Existing detection technologies do not take into account the mutual influence between students' behaviors and the phenomenon of occlusion in the complex classroom environment;

[0009] (3) The post-processing stage of existing detection technologies is too complex and requires a lot of computing resources, which increases the time complexity and computing cost of the detection process. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides an end-to-end student behavior detection method and system based on large convolutional kernels. It extracts features through a large convolutional kernel backbone network and combines a hybrid feature fusion architecture (HFFA) to achieve cross-level feature fusion before inputting the refined Transformer encoder-decoder structure for prediction, effectively improving the detection capability of student behavior in complex classroom environments.

[0011] This invention is implemented as follows: an end-to-end student behavior detection method based on large convolutional kernels, comprising:

[0012] Step 1: Input classroom environment student behavior image processing, scale the image to a fixed size to match the input size of the model, adjust the width and height of the image while keeping the aspect ratio unchanged to avoid image distortion, and then perform data augmentation operations such as random rotation, translation, flipping, and brightness adjustment.

[0013] Step 2: Design a large convolutional kernel backbone network to extract upstream features, deeply extract student behavior features in the classroom environment, and obtain preliminary features by combining the behavior with the surrounding environment through two types of large convolutional kernels.

[0014] Step 3: Introduce the Hybrid Feature Fusion Architecture (HFFA) to improve the model's feature extraction and fusion capabilities in complex and multi-scale scenarios. It combines the advantages of feature pyramids and coordinate attention mechanisms, stacks multiple Transformer encoders, captures global contextual information of the input feature map, generates high-dimensional feature representations, and adds positional encoding to the feature map.

[0015] Step 4: The fused student behavior features extracted by the hybrid feature fusion architecture are fed into the decoder. Multiple decoders are stacked using a fixed set of target queries. These queries are learnable embedding vectors representing the targets the model wants to detect. Finally, the decoder output predicts the target category through a linear layer and predicts the bounding box location of the target through a regression head.

[0016] Step 5: Combine the encoder output with the labels of student behavior in the classroom to predict the behavior of each student, construct the corresponding loss function, and train the system by combining the three loss functions.

[0017] Step Six: Apply the trained parameters and weights to predict student behavior in the classroom environment of the test set and obtain the corresponding results. Then, analyze the predicted student behavior results, analyze the student behavior in the classroom environment, and visualize the analysis results.

[0018] Furthermore, step one specifically includes:

[0019] (1.1) Data preprocessing: including the acquisition of student behavior datasets in a classroom environment.

[0020] (1.2) Data augmentation operations such as random rotation, translation, flipping, brightness change, mosaic, etc., as well as the division of training and test sets.

[0021] Furthermore, step two specifically includes:

[0022] (2.1) For the classroom student behavior images input in step one, design a large convolutional kernel backbone network to extract upstream student behavior features;

[0023] First, the input image X is fed into the ConvNeXt large convolution module:

[0024] X′=PWConv(GELU(PWConv(LN(DWConv(X)))))

[0025] Here, PWConv represents pointwise convolution, GELU represents Gaussian error linear unit, LN represents layer normalization, and DWConv represents depthwise separable convolution. The extracted X' features are then input into the next large convolutional module LSKBlock, and this process is repeated to extract upstream features.

[0026]

[0027] Furthermore, step three specifically includes:

[0028] (3.1) Use the multi-scale feature pyramid network structure (NASFCOSFPN) searched by the spatial search mechanism to perform preliminary feature fusion and feature extraction.

[0029] (3.2) Combining the coordinate attention mechanism, the input feature map is first subjected to global pooling in both the horizontal and vertical directions to generate one-dimensional feature codes. Then, these codes are combined through convolutional operations to form an attention feature map containing positional information. Finally, a weight matrix is ​​generated using two 1×1 convolutional layers and a sigmoid function, and applied to the input feature map to enhance the model's attention to specific locations.

[0030]

[0031] Generate the final weight matrix and apply it to the input feature map:

[0032]

[0033] (3.3) Integrate multiple Transformer encoders to capture global and contextual information and generate high-dimensional feature representations. At the same time, introduce a deformable attention mechanism into the encoder to reduce computation by focusing on only a few key points (instead of all locations) in the feature map. The multi-head attention mechanism performs global feature modeling, and its specific formula is as follows:

[0034]

[0035] Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

[0036] Furthermore, step four specifically includes:

[0037] (4.1) After completing the extraction of upstream features and multi-scale feature fusion, the fused features are fed into the Transformer decoder for further processing. The self-attention module calculates the relationships between different positions in the feature map to capture global contextual information. Next, the cross-attention module calculates the relationship between the query and the encoder output features, mapping the features to target categories and bounding box predictions. To improve computational efficiency, a deformable attention mechanism, DeformableAttention, is introduced to optimize computation by focusing on only a few key points, specifically:

[0038]

[0039] The decoder receives the output feature sequence from the encoder and performs cross-attention computation with a set of learnable queries. Each query represents a potential object detection result. Through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder to compute features related to each object.

[0040] Furthermore, step five specifically includes:

[0041] (5.1) Match the feature sequence output by the encoder with the pre-labeled student behavior labels. Each feature vector represents a potential student behavior. The decoder compares these features with the student behavior labels to predict the specific behavior of each student (such as raising hands, reading, discussing, etc.).

[0042] (5.2) To optimize the model's prediction performance, several loss functions were constructed to measure the gap between the model's prediction and the true label: classification loss measures the difference between the model's predicted behavior category and the true label; bounding box regression loss measures the distance between the model's predicted behavior location and the actual location; and IoU loss measures the overlap between the predicted bounding box and the true bounding box. By weighted summing these loss functions, a joint loss function is obtained to guide the model's training, enabling it to optimize the accuracy of behavior prediction across multiple dimensions.

[0043] Furthermore, step six specifically includes...

[0044] (6.1) After the model training is completed, the trained parameters and weights are applied to the test set. The data in the test set is new to the model. By predicting the behavior of students in the test set, the model's performance in real-world scenarios is evaluated. Based on the previously learned features and labels, the model predicts the behavior of each student in the test set and generates the corresponding bounding box and behavior category.

[0045] (6.2) Conduct in-depth analysis of the predicted student behavior outcomes and evaluate the model's performance metrics (such as precision, recall, F1 score, etc.) to understand the model's performance across different behavior categories. Furthermore, use visualization techniques such as confusion matrices, PR curves, and behavior detection boxes to display the distribution and trends of student behavior in different classroom environments. This visualization analysis helps teachers or researchers intuitively understand student behavior in the classroom and provides support for the development of educational strategies.

[0046] Another objective of this invention is to provide an end-to-end student behavior detection system based on large convolutional kernels, which implements the aforementioned end-to-end student behavior detection method based on large convolutional kernels, comprising:

[0047] Data preprocessing module: Used to acquire student behavior datasets in a classroom environment, and to ensure data diversity and generalization ability through data augmentation operations such as random rotation, translation, flipping, brightness adjustment, and mosaic, as well as the division of training and test sets.

[0048] Upstream Feature Extraction Module: By designing a large convolutional kernel backbone network, the input student behavior image is used to extract features. First, the image is input into the ConvNeXt large convolutional module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features.

[0049] Multi-scale feature fusion module: The multi-scale feature pyramid network (NASFCOSFPN) constructed using the spatial search mechanism performs preliminary feature fusion, and combined with the coordinate attention mechanism, it generates attention feature maps containing location information through global pooling and convolution operations, thereby enhancing the model's attention to specific locations.

[0050] Behavior decoding module: This module feeds the fused features into the Transformer decoder for further processing. The decoder calculates the relationships between features using self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism.

[0051] Loss calculation and model training module: Matches the feature sequence output by the decoder with the pre-labeled student behavior labels, and calculates the joint loss function by constructing classification loss, bounding box regression loss and IoU loss to optimize the behavior prediction accuracy of the model and guide the joint training of the model.

[0052] The results generation and analysis module applies the trained model to the test set to predict student behavior, generating bounding boxes and behavior categories. Subsequently, it conducts in-depth analysis of the prediction results, evaluates model performance metrics, and uses visualization techniques to display the distribution and trends of student behavior in different classroom environments, providing support for educational strategies.

[0053] Another object of the present invention is to provide a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the end-to-end student behavior detection method based on large convolution kernels.

[0054] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the end-to-end student behavior detection method based on large convolutional kernels.

[0055] Another objective of this invention is to provide an information data processing terminal, which includes the aforementioned end-to-end student behavior detection system based on large convolutional kernels.

[0056] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0057] First, the end-to-end student behavior detection method based on large convolutional kernels provided in this invention aims to solve the aforementioned problems. This method fully utilizes deep learning techniques, enhancing upstream feature extraction capabilities through advanced large convolutional kernels to ensure accurate capture of each student's behavior. Combined with a Hybrid Feature Fusion Architecture (HFFA) design, it extracts multi-scale features from images through feature extraction layers of different scales, while also incorporating spatial location information to enhance the perception of target spatial distribution. Finally, a Transformer encoder-decoder structure further processes and refines features, capturing global contextual information and generating high-level feature representations. Ultimately, it generates behavior classification and bounding box predictions, accurately locating and classifying targets through an attention mechanism. This provides educational researchers with more refined and comprehensive insights into classroom behavior, propelling educational practice and research into a deeper data-driven stage.

[0058] (1) This invention constructs an automated classroom student behavior detection and analysis system, which can effectively improve the efficiency of manual analysis and provide teachers, students and related researchers with more accurate behavior analysis references.

[0059] (2) The present invention adopts a feature extraction method with large convolution kernels, which can better capture complex behavioral features in the classroom environment and improve the accuracy and robustness of detection.

[0060] (3) This invention constructs an end-to-end behavior detection system based on multi-scale feature fusion and Transformer decoding. By combining global and local information, the recognition accuracy is higher and it can cope with diverse classroom scenarios.

[0061] (4) This invention solves the problem of detecting multiple types of behavior in real classroom scenarios, and has stronger universality and adaptability, and is applicable to various teaching environments.

[0062] Secondly, the end-to-end student behavior detection and analysis system based on large convolutional kernels provided by this invention offers an automated tool for analyzing classroom student behavior. This system effectively extracts deep features of student behavior through large convolutional kernel networks and combines multi-scale feature fusion with a Transformer decoder to achieve accurate detection of student behavior in complex classroom environments. To address diverse classroom scenarios, the system also introduces a coordinate attention mechanism to enhance the model's focus on key behaviors. Compared with traditional student behavior analysis methods, this invention significantly improves the accuracy and efficiency of behavior recognition, overcoming the problem of insufficient detection accuracy in complex environments using traditional methods.

[0063] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0064] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0065] This invention proposes an end-to-end student behavior detection and analysis method based on large convolutional kernels. By combining advanced feature extraction and multi-scale fusion techniques, it improves upon traditional behavior detection methods. This method significantly enhances the efficiency of classroom behavior analysis while improving the accuracy of behavior recognition. For educators, this technology helps them better understand student behavior in the classroom, thereby optimizing teaching strategies. For students and researchers, the data analysis provided by the system can help them reflect on and study classroom behavior, demonstrating significant commercial value and application potential.

[0066] (2) The technical solution of this invention fills a technical gap in the industry both domestically and internationally:

[0067] Existing student behavior detection methods often suffer from low accuracy in complex classroom environments and fail to fully utilize multi-scale features and spatial information. This invention, by introducing large convolutional kernels and multi-scale feature fusion technology, effectively improves the detection capability of student behavior and achieves more accurate behavior recognition in complex scenarios, filling a technological gap in this field.

[0068] (3) The technical solution of the present invention solves a technical problem that people have long wanted to solve but have never been able to solve successfully:

[0069] Currently, the market lacks tools capable of automating the analysis of classroom student behavior, relying primarily on manual analysis, which is inefficient. This invention introduces end-to-end student behavior detection technology into classroom behavior analysis. Addressing the challenge of behavior detection in complex scenarios, it combines large convolutional kernels and multi-scale feature fusion techniques to solve the problem of inaccurate behavior recognition in complex environments found in existing technologies. This technology can help educational institutions conduct their work more systematically, improving the efficiency and accuracy of classroom behavior analysis.

[0070] Fourth, the end-to-end student behavior detection method based on large convolution kernel proposed in this invention mainly solves the problem of insufficient accuracy and robustness of student behavior detection in complex scenarios in the existing technology, and achieves significant technological progress.

[0071] 1. Improved Accuracy: Traditional student behavior detection methods often struggle to accurately identify student behavior in complex and ever-changing classroom environments, especially under conditions of varying lighting, occlusion, and diverse behaviors. By designing a backbone network with large convolutional kernels, this invention can better capture key features in the classroom environment, effectively enhancing feature extraction capabilities and significantly improving the model's behavior detection accuracy in complex environments.

[0072] 2. Multi-scale Feature Fusion: Traditional methods struggle to simultaneously capture global information and local details when processing multi-scale features, resulting in poor performance when handling behaviors at different scales. This invention introduces the Hybrid Feature Fusion Architecture (HFFA), combining feature pyramids and coordinate attention mechanisms to successfully achieve effective fusion and extraction of features at different scales. By stacking multiple Transformer encoders, this invention can capture global contextual information and generate high-dimensional feature representations, thereby accurately detecting student behavior even in complex scenarios.

[0073] 3. Optimized Computational Efficiency: Existing methods for detecting complex behaviors often involve high computational costs, making them difficult to process in real-time in practical applications. This invention introduces a deformable attention mechanism into the Transformer decoder, which significantly reduces computational costs and improves the model's computational efficiency by focusing on only a few key points in the feature map, enabling the method to run efficiently in real-time application scenarios.

[0074] 4. Enhanced Robustness: By combining various data augmentation techniques and feature extraction strategies, this invention significantly enhances the model's robustness in different classroom environments, enabling it to cope with various complex situations, including changes in lighting, student posture, and occlusion. Ultimately, this invention achieves more stable and reliable student behavior detection results, with broad prospects for industrial applications.

[0075] In summary, this invention not only achieves significant technological advancements in behavior detection accuracy, multi-scale feature processing, and computational efficiency, but also addresses the problem of traditional technologies performing poorly in complex classroom environments by enhancing model robustness, providing a more reliable and efficient solution for intelligent classroom monitoring and analysis systems. Attached Figure Description

[0076] Figure 1 This is a flowchart of the end-to-end student behavior detection and analysis method based on large convolutional kernels provided in this embodiment of the invention;

[0077] Figure 2 This is a schematic diagram of the structure of the end-to-end student behavior detection and analysis method based on large convolutional kernels provided in this invention.

[0078] Figure 3 This is a model structure diagram of the transformer encoder used in the end-to-end student behavior detection and analysis system based on large convolutional kernels provided in this invention example;

[0079] Figure 4 This is a module structure diagram of the end-to-end student behavior detection and analysis method based on large convolutional kernels provided in this invention example;

[0080] Figure 5This is a schematic diagram illustrating the comparison of experimental results on a simulated classroom dataset provided by an example of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0082] like Figure 1 As shown, the end-to-end student behavior detection and analysis method based on large convolutional kernels provided in this embodiment of the invention includes the following steps:

[0083] S01. Obtain a dataset of student behavior in a classroom environment, and ensure the diversity and generalization ability of the data through data augmentation operations such as random rotation, translation, flipping, changing brightness, and mosaic, as well as the division of training and test sets.

[0084] S02, firstly, the image is input into the ConvNeXt large convolutional module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features;

[0085] S03 utilizes a multi-scale feature pyramid network constructed using a spatial search mechanism, combined with a coordinate attention mechanism, to generate an attention feature map containing location information through global pooling and convolution operations.

[0086] S04: The fused features are fed into the Transformer decoder for further processing. The decoder calculates the relationships between features through self-attention and cross-attention mechanisms to generate target category and bounding box predictions;

[0087] S05: The feature sequence output by the decoder is matched with the pre-labeled student behavior labels. The joint loss function is calculated by constructing classification loss, bounding box regression loss and IoU loss.

[0088] S06. The trained model is applied to the test set to predict student behavior, generating bounding boxes and behavior categories. Subsequently, the prediction results are analyzed in depth to evaluate the model's performance metrics.

[0089] like Figure 4 As shown, another objective of this invention is to provide an end-to-end student behavior detection and analysis system based on large convolutional kernels, comprising:

[0090] Data preprocessing module: Used to acquire student behavior datasets in a classroom environment, and to ensure data diversity and generalization ability through data augmentation operations such as random rotation, translation, flipping, brightness adjustment, and mosaic, as well as the division of training and test sets.

[0091] Upstream Feature Extraction Module: By designing a large convolutional kernel backbone network, the input student behavior image is used to extract features. First, the image is input into the ConvNeXt large convolutional module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features.

[0092] Multi-scale feature fusion module: The multi-scale feature pyramid network (NASFCOSFPN) constructed using the spatial search mechanism performs preliminary feature fusion, and combined with the coordinate attention mechanism, it generates attention feature maps containing location information through global pooling and convolution operations, thereby enhancing the model's attention to specific locations.

[0093] Behavior decoding module: This module feeds the fused features into the Transformer decoder for further processing. The decoder calculates the relationships between features using self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism.

[0094] Loss calculation and model training module: Matches the feature sequence output by the decoder with the pre-labeled student behavior labels, and calculates the joint loss function by constructing classification loss, bounding box regression loss and IoU loss to optimize the behavior prediction accuracy of the model and guide the joint training of the model.

[0095] The results generation and analysis module applies the trained model to the test set to predict student behavior, generating bounding boxes and behavior categories. Subsequently, it conducts in-depth analysis of the prediction results, evaluates model performance metrics, and uses visualization techniques to display the distribution and trends of student behavior in different classroom environments, providing support for educational strategies.

[0096] The schematic diagram of the end-to-end student behavior detection and analysis system based on large convolutional kernels provided in this embodiment of the invention is as follows: Figure 2 As shown in Table 1. The symbols used in the embodiments of this invention are also shown in Table 1.

[0097] Table 1. Symbols involved in the embodiments of the present invention

[0098]

[0099] Dataset preprocessing: This includes acquiring student behavior datasets in a classroom setting.

[0100] (1.1) Data augmentation operations such as random rotation, translation, flipping, brightness change, mosaic, etc., as well as the division of training and test sets.

[0101] Step two specifically includes:

[0102] (2.1) For the classroom student behavior images input in step one, design a large convolutional kernel backbone network to extract upstream student behavior features;

[0103] First, the input image X is fed into the ConvNeXt large convolution module:

[0104] X′=PWConv(GELU(PWConv(LN(DWConv(X)))))

[0105] Here, PWConv represents pointwise convolution, GELU represents Gaussian error linear unit, LN represents layer normalization, and DWConv represents depthwise separable convolution. The extracted X' features are then input into the next large convolutional module LSKBlock, and this process is repeated to extract upstream features.

[0106]

[0107] Step three specifically involves:

[0108] (3.1) Use the multi-scale feature pyramid network structure (NASFCOSFPN) searched by the spatial search mechanism to perform preliminary feature fusion and feature extraction.

[0109] (3.2) Combining the coordinate attention mechanism, the input feature map is first subjected to global pooling in both the horizontal and vertical directions to generate one-dimensional feature codes. Then, these codes are combined through convolutional operations to form an attention feature map containing positional information. Finally, a weight matrix is ​​generated using two 1×1 convolutional layers and a sigmoid function, and applied to the input feature map to enhance the model's attention to specific locations.

[0110]

[0111] Generate the final weight matrix and apply it to the input feature map:

[0112]

[0113] (3.3) Integrate multiple Transformer encoders to capture global and contextual information and generate high-dimensional feature representations. At the same time, introduce a deformable attention mechanism into the encoder to reduce computation by focusing on only a few key points (instead of all locations) in the feature map. The multi-head attention mechanism performs global feature modeling, and its specific formula is as follows:

[0114]

[0115] Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

[0116] Step four specifically includes:

[0117] (4.1) After completing the extraction of upstream features and multi-scale feature fusion, the fused features are fed into the Transformer decoder for further processing. The self-attention module calculates the relationships between different positions in the feature map to capture global contextual information. Next, the cross-attention module calculates the relationship between the query and the encoder output features, mapping the features to target categories and bounding box predictions. To improve computational efficiency, a deformable attention mechanism, DeformableAttention, is introduced to optimize computation by focusing on only a few key points, specifically:

[0118]

[0119] The decoder receives the output feature sequence from the encoder and performs cross-attention computation with a set of learnable queries. Each query represents a potential object detection result. Through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder to compute features related to each object.

[0120] Step five specifically includes:

[0121] (5.1) Match the feature sequence output by the encoder with the pre-labeled student behavior labels. Each feature vector represents a potential student behavior. The decoder compares these features with the student behavior labels to predict the specific behavior of each student (such as raising hands, reading, discussing, etc.).

[0122] (5.2) To optimize the model's prediction performance, several loss functions were constructed to measure the gap between the model's prediction and the true label: classification loss measures the difference between the model's predicted behavior category and the true label; bounding box regression loss measures the distance between the model's predicted behavior location and the actual location; and IoU loss measures the overlap between the predicted bounding box and the true bounding box. By weighted summing these loss functions, a joint loss function is obtained to guide the model's training, enabling it to optimize the accuracy of behavior prediction across multiple dimensions.

[0123] Step six specifically includes

[0124] (6.1) After the model training is completed, the trained parameters and weights are applied to the test set. The data in the test set is new to the model. By predicting the behavior of students in the test set, the model's performance in real-world scenarios is evaluated. Based on the previously learned features and labels, the model predicts the behavior of each student in the test set and generates the corresponding bounding box and behavior category.

[0125] (6.2) Conduct in-depth analysis of the predicted student behavior outcomes and evaluate the model's performance metrics (such as precision, recall, F1 score, etc.) to understand the model's performance across different behavior categories. Furthermore, use visualization techniques such as confusion matrices, PR curves, and behavior detection boxes to display the distribution and trends of student behavior in different classroom environments. This visualization analysis helps teachers or researchers intuitively understand student behavior in the classroom and provides support for the development of educational strategies.

[0126] The main development environment used in this invention experiment includes: Ubuntu 20.04, GTX 4090Ti, Chainer 5.4.0, and Python 3.7. The specific hyperparameter settings of the model are shown in Table 2 below.

[0127] Table 2. Hyperparameter settings of the experimental model

[0128] hyperparameters numerical values batch_size 64 epoch 20 dropout 0.1 learning_rate 0.001 transformer_encoder_n_layers 6 transformer_encoder_n_heads 6

[0129] The student behavior detection and analysis system based on large convolutional kernels and multi-scale feature fusion provided in the application embodiment of the present invention is applied to a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it causes the processor to perform the steps of the student behavior detection and analysis system based on large convolutional kernels and multi-scale feature fusion.

[0130] The student behavior detection and analysis method based on large convolutional kernels provided in the application embodiment of the present invention is applied to an information data processing terminal, which is used to implement the student behavior detection and analysis system based on large convolutional kernels.

[0131] This invention compares a student behavior detection and analysis method based on large convolutional kernels with existing end-to-end behavior detection methods. The model's performance is evaluated by calculating its precision, recall, and F1 score.

[0132] • Recall represents the proportion of samples that correctly identify a particular behavior category out of all samples that actually belong to that category. The formula is:

[0133]

[0134] Precision represents the proportion of samples that the model predicts as belonging to a certain behavior category, but which actually belong to that behavior category. The formula is:

[0135]

[0136] ·F1 Score is the harmonic mean of precision and recall, which is used to balance the weights of the two metrics.

[0137] It can be seen from the experimental results that the student behavior detection and analysis method based on large convolution kernels and multi-scale feature fusion provided by the present invention exhibits excellent performance in multiple experiments. First, through ablation experiments, we evaluated the contribution of each module to the overall performance of the model. After removing the LKNeXt backbone network, the mAP of the model decreased by 1.2%, and the recall rate decreased by 1.8%, which indicates that the large convolution kernel plays a key role in extracting complex behavior features. Similarly, when the HFFA hybrid feature fusion architecture is removed, the mAP and recall rate of the model decrease by 1.5% and 1.7% respectively, which further verifies the importance of HFFA in multi-scale feature fusion and global information capture. The ablation experiments show that each module in the present invention makes a significant contribution to improving the overall performance of the model.

[0138]

[0139]

[0140] Secondly, through comparative experiments, we conducted a detailed comparison between the model of the present invention and the traditional DeformableDETR model. On the SCB1 classroom behavior dataset, the mAP of the model of the present invention is improved by 1.5%, and the recall rate is improved by 2%, which is significantly better than the traditional model. This is mainly attributed to the design of the LKNeXt backbone network, which effectively captures the detailed information of student behaviors through convolution kernels with a larger receptive field. In addition, the HFFA hybrid feature fusion architecture further enhances the model's ability to fuse and extract global and local information in complex scenes by introducing an attention mechanism. The results of the comparative experiments prove that the present invention not only surpasses traditional methods in detection accuracy, but also exhibits more stable performance when processing multi-scale targets and complex scenes.

[0141] Next, the classroom interaction network graph formed for this class is taken as an example to analyze the classroom interaction situation of this class.

[0142] First, an overall behavior analysis is performed. We calculate the student behavior frequency density ρ=m / (n×t) detected by the model, where m represents the total number of detected behaviors, n represents the number of students participating in the class, and t represents the total duration of the class. We consider that when ρ>Threshold, it indicates that students are active in the class with high participation; while when ρ<Threshold, it indicates that the classroom participation is relatively low. In this way, teachers can preliminarily judge the activity level of the classroom and the overall participation situation of students.

[0143] Next, individual behavior analysis is conducted. Individual behavior analysis primarily uses two indicators to measure students' classroom performance:

[0144] • Participation Index: Measured by the frequency of each student's behavior detected by the model. A higher participation index indicates a more active and proactive student in the classroom. The formula is: Each student's participation index equals the total number of times their behavior is detected throughout the entire class.

[0145] • Focus Index: This measures the duration of time students maintain focus in the classroom. A higher focus index indicates a higher level of concentration among students in the classroom.

[0146] Behavioral engagement helps teachers understand how actively each student participates in the classroom; while behavioral focus reflects how attentive students are during learning, which is crucial for assessing student learning outcomes.

[0147] We believe that in an active classroom, if the teacher's behavioral participation and focus are the highest, it indicates that the teacher is in a dominant and controlling position, and this can be identified as a dominant classroom. Conversely, if students' behavioral participation and focus are high, it can be identified as a discussion-based classroom. In a classroom with low participation, if the teacher's behavioral participation is the highest, it can be identified as a lecture-based classroom. If a particular student's behavioral participation and focus are high, it can be identified as a presentation-based classroom.

[0148] Taking a specific lesson as an example, with a total of 120 behavioral tests and a total class duration of 60 minutes, the calculated behavioral density ρ indicates a moderate level of classroom participation. Furthermore, in this lesson, some students exhibited high behavioral participation, while others showed lower participation and focus. This provides teachers with a basis for adjusting their teaching strategies accordingly. Students can also make timely adjustments based on their individual classroom performance to better participate in learning.

[0149] The system ultimately generates a detailed student behavior analysis report, recording the types of classroom participation, behavioral data for each student, and overall classroom performance for reference. In summary, the end-to-end student behavior detection system based on large convolutional kernels proposed in this invention has played a significant role in classroom education scenarios. It not only effectively solves the efficiency and accuracy problems of student behavior analysis in actual classrooms but also automates the processing of student behavior detection. By quickly identifying and accurately labeling student behavior, this invention can fully utilize classroom video resources, greatly reducing the workload of teachers, and possesses the ability to monitor and analyze student behavior in real time. This helps teachers to promptly grasp the interaction situation in the classroom, identify potential teaching problems and make improvements, thereby enhancing teaching effectiveness, increasing students' learning enthusiasm and participation, and promoting the efficient conduct of classroom education activities.

[0150] Example 1: Intelligent Classroom Behavior Analysis System

[0151] In modern educational environments, teachers need to understand and analyze students' behavior in the classroom in real time in order to adjust teaching strategies and improve teaching effectiveness. However, traditional behavioral analysis mainly relies on teachers' subjective observation, which suffers from problems such as insufficient accuracy and poor real-time performance.

[0152] Implementation process:

[0153] 1. System Deployment: High-resolution cameras are installed in the classroom to capture real-time images of students' behavior. The image data is transmitted to a server for processing via a network.

[0154] 2. Behavior Detection and Analysis: The end-to-end student behavior detection method based on large convolutional kernels of this invention is used to process real-time captured images. The system classifies each student's behavior, such as focusing, raising hands, and daydreaming, and generates behavior records and statistical data for each student.

[0155] 3. Teacher Feedback and Strategy Adjustment: The behavioral data generated by the system is presented to teachers through visualization tools, such as real-time behavior monitoring charts and behavior trend analysis charts. Teachers adjust their teaching strategies based on students' behavioral status, such as adjusting questioning and interaction, and changing the way content is explained.

[0156] This invention enables automated, real-time analysis of student classroom behavior, improving the accuracy of classroom management. It also provides data-driven teaching feedback tools to help teachers understand student status promptly and optimize the teaching process.

[0157] Example 2: Monitoring Learning Behavior on Online Education Platforms

[0158] In online education platforms, teachers and educational institutions often struggle to monitor students' learning behaviors in real time, making it difficult to guarantee learning outcomes. Introducing intelligent behavior detection systems can improve the quality of online education by enabling real-time monitoring of students' learning status.

[0159] 1. Integration with Online Education Platforms: The student behavior detection system based on this invention is integrated into an online education platform. When students participate in online courses via camera, the system captures images of their learning behavior in real time.

[0160] 2. Behavioral Detection and Feedback: The system monitors students' learning behaviors in real time, such as focusing, inattentiveness, and leaving their seats. Based on the detection results, the platform provides immediate feedback, such as reminding students of their lack of concentration, recording study time, and analyzing learning behavior patterns.

[0161] 3. Learning Outcome Assessment and Personalized Learning Plans: The system generates learning reports based on students' learning behavior data and provides them to teachers or educational institutions. These reports include data such as students' focus levels and behavioral trends. Based on the reports, the platform can recommend personalized learning content and strategies to students to improve learning outcomes.

[0162] This invention enhances the intelligence of online education platforms, enabling real-time monitoring and feedback of student learning behavior. It provides personalized learning solutions based on behavioral data, improving student learning outcomes and engagement.

[0163] It should be noted that embodiments of the present invention can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.

[0164] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An end-to-end student behavior detection and analysis method based on large convolutional kernels, characterized in that, The end-to-end student behavior detection and analysis method based on large convolutional kernels includes: Step 1: Input classroom environment student behavior image processing, scale the image to a fixed size to match the input size of the model, adjust the width and height of the image while keeping the aspect ratio unchanged to avoid image distortion, and then perform data augmentation operations such as random rotation, translation, flipping, and brightness adjustment. Step 2: Design a large convolutional kernel backbone network to extract upstream features, deeply extract student behavior features in the classroom environment, and obtain preliminary features by combining the behavior with the surrounding environment through two types of large convolutional kernels. Step 3: Introduce the Hybrid Feature Fusion Architecture (HFFA) to improve the model's feature extraction and fusion capabilities in complex and multi-scale scenarios. It combines the advantages of feature pyramids and coordinate attention mechanisms, stacks multiple Transformer encoders, captures global contextual information of the input feature map, generates high-dimensional feature representations, and thus adds positional encoding to the feature map. Step 4: Extract the fused student behavior features from the hybrid feature fusion architecture and feed them into the decoder. Multiple decoders are stacked using a fixed set of target queries. These queries are learnable embedding vectors that represent the targets that the model wants to detect. Finally, the decoder output predicts the target category through a linear layer and predicts the bounding box location of the target through a regression head. Step 5: Combine the encoder output with the labels of student behavior in the classroom to predict the behavior of each student, construct the corresponding loss function, and train the three loss functions together. Step Six: Apply the trained parameters and weights to predict student behavior in the classroom environment of the test set and obtain the corresponding results. Then, analyze the predicted student behavior results, analyze the student behavior in the classroom environment, and visualize the analysis results. Step one specifically includes: Dataset preprocessing: including the acquisition of student behavior datasets in a classroom setting; Random rotation, translation, flipping, brightness adjustment, mosaic data augmentation operations, and the division of training and test sets; Step two specifically includes: For the classroom student behavior images input in step one, a large convolutional kernel backbone network is designed to extract upstream student behavior features; First, the input image X is fed into the ConvNeXt large convolution module: ； Where PWConv represents pointwise convolution, GELU represents Gaussian error linear unit, LN represents layer normalization, and DWConv represents depthwise separable convolution; then the extracted X' features are input into the next large convolution module LSKBlock, and this process is repeated to extract upstream features; ； Step three specifically involves: The multi-scale feature pyramid network structure (NASFCOS-FPN) searched using a spatial search mechanism is used for preliminary feature fusion and feature extraction; By incorporating a coordinate attention mechanism, the input feature map is first subjected to global pooling in both the horizontal and vertical directions to generate one-dimensional feature codes. These codes are then combined through convolutional operations to form an attention feature map containing positional information. Finally, a weight matrix is ​​generated using two 1×1 convolutional layers and a sigmoid function, and applied to the input feature map, thereby enhancing the model's ability to perceive the spatial distribution of the target. ； ； Generate the final weight matrix and apply it to the input feature map: ； Multiple Transformer encoders are integrated to capture global, contextual information and generate high-dimensional feature representations. A deformable attention mechanism is introduced into the encoder to reduce computation by focusing on only a few key points in the feature map instead of all locations. The multi-head attention mechanism performs global feature modeling, and its specific formula is as follows: ； Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

2. The end-to-end student behavior detection and analysis method based on large convolutional kernels according to claim 1, characterized in that... Step four: After extracting upstream features and fusing multi-scale features, the fused features are fed into the Transformer decoder for further processing. The self-attention module calculates the relationships between different locations in the feature map to capture global contextual information. Next, the cross-attention module calculates the relationship between the query and the encoder output features, mapping the features to target categories and bounding box predictions. To improve computational efficiency, a deformable attention mechanism is introduced, optimizing computation by focusing on only a few key points. Specifically: ； The decoder receives the output feature sequence from the encoder and performs cross-attention calculation with a set of learnable queries; each query represents a potential target detection result; through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder to calculate the features related to each target.

3. The end-to-end student behavior detection and analysis method based on large convolutional kernels according to claim 1, characterized in that... Step five specifically includes: The feature sequence output by the encoder is matched with pre-labeled student behavior tags; each feature vector represents a potential student behavior. The decoder compares these features with the student behavior tags to predict the specific behavior of each student, including raising their hand, reading, and discussing. To optimize the model's prediction performance, several loss functions were constructed to measure the gap between the model's prediction and the true label: classification loss measures the difference between the model's predicted behavior category and the true label; bounding box regression loss measures the distance between the model's predicted behavior location and the actual location; and IoU loss measures the degree of overlap between the predicted bounding box and the true bounding box. By weighted summing these loss functions, a joint loss function is obtained to guide the model's training, enabling it to optimize the accuracy of behavior prediction across multiple dimensions.

4. The end-to-end student behavior detection and analysis method based on large convolutional kernels according to claim 1, characterized in that... Step six specifically includes: After the model is trained, the trained parameters and weights are applied to the test set. The data in the test set is data that the model has never seen before. By predicting the behavior of students in the test set, the model's performance in real-world scenarios is evaluated. Based on the features and labels learned previously, the model predicts the behavior of each student in the test set and generates the corresponding bounding boxes and behavior categories. The predicted student behavior outcomes are analyzed in depth to evaluate the model's performance metrics, including accuracy, recall, and F1 score, to understand the model's performance across different behavior categories. Visualization is achieved using confusion matrices, PR curves, and behavior detection boxes to show the distribution and trends of student behavior in different classroom environments. This visualization analysis helps teachers or researchers intuitively understand student behavior in the classroom and provides support for the development of educational strategies.

5. A system for implementing the end-to-end student behavior detection and analysis method based on large convolutional kernels as described in any one of claims 1 to 4, characterized in that, The end-to-end student behavior detection and analysis system based on large convolutional kernels includes: Data preprocessing module: used to acquire student behavior datasets in a classroom environment, and to ensure data diversity and generalization ability through random rotation, translation, flipping, brightness adjustment, mosaic data augmentation operations, as well as the division of training and test sets; Upstream feature extraction module: By designing a large convolutional kernel backbone network, the input student behavior image is used to extract features. First, the image is input into the ConvNeXt large convolutional module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features. Multi-scale feature fusion module: The multi-scale feature pyramid network (NASFCOS-FPN) constructed using the spatial search mechanism performs preliminary feature fusion, and combines the coordinate attention mechanism to generate attention feature maps containing location information through global pooling and convolution operations, thereby enhancing the model's ability to perceive the spatial distribution of the target. Behavior decoding module: The fused features are fed into the Transformer decoder for further processing; The decoder calculates the relationship between features through self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism; Loss calculation and model training module: Matches the feature sequence output by the decoder with the pre-labeled student behavior labels, and calculates the joint loss function by constructing classification loss, bounding box regression loss and IoU loss to optimize the behavior prediction accuracy of the model and guide the joint training of the model; The results generation and analysis module applies the trained model to the test set to predict student behavior, generating bounding boxes and behavior categories. Subsequently, it conducts in-depth analysis of the prediction results, evaluates the model's performance metrics, and uses visualization to display the distribution and changing trends of student behavior in different classroom environments, providing support for educational strategies.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the end-to-end student behavior detection and analysis method based on large convolutional kernels as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the end-to-end student behavior detection and analysis method based on large convolutional kernels as described in any one of claims 1 to 4.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the end-to-end student behavior detection and analysis system based on large convolutional kernels as described in claim 5.

Images