End-to-end student behavior detection method and system based on large convolution kernel

Through the end-to-end detection method of large convolution kernel and hybrid feature fusion architecture, the problems of low efficiency of student behavior detection and insufficient accuracy in complex environments in the existing technology are solved, and efficient and accurate classroom behavior analysis is achieved.

CN119068548AActive Publication Date: 2024-12-03HUAZHONG NORMAL UNIV

Patent Information

Application Number
CN202411177520.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-03
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

The existing student behavior detection technology is inefficient and fails to effectively deal with occlusion and diversified behavioral characteristics in complex classroom environments. The post-processing calculation is complex and resource-consuming, affecting the detection speed and accuracy.

Method used

The end-to-end detection method based on large convolution kernels is adopted to extract features through the large convolution kernel backbone network, and combine the hybrid feature fusion architecture and Transformer encoder to perform cross-level fusion and prediction of features to improve detection capabilities.

Benefits of technology

It improves the accuracy and robustness of student behavior detection, reduces the computational complexity, is suitable for diverse classroom scenarios, and achieves more efficient behavioral analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068548A_ABST
    Figure CN119068548A_ABST
Patent Text Reader

Abstract

The invention belongs to but not limited to the technical field of computer vision, and particularly relates to an end-to-end student behavior detection method and system based on a large convolution kernel, and the method comprises the steps: carrying out the preprocessing of classroom video data, and obtaining the preliminary feature representation of student behaviors; a large convolution kernel backbone network is designed to carry out deep convolution operation on the extracted image features, upstream feature representation of a larger receptive field is obtained, and behavior features of different scales are fused in combination with a multi-scale feature fusion module; a coordinate attention mechanism is introduced, invalid information in the feature map is filtered, and the attention degree of the model on key behaviors is enhanced; the fused features are input into a Transform decoder, and global behavior prediction is carried out; and through constructing a joint loss function optimization model parameter, outputting a final frame-level prediction result of the classroom behavior, and accurately identifying the student behavior type in the classroom.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of computer vision technology, and in particular relates to an end-to-end student behavior detection method and system based on a large convolution kernel. Background Art

[0002] With the development of educational informatization, modern teaching models have gradually shifted from traditional face-to-face teaching to digital and intelligent ones. The popularity of smart classrooms and online education platforms has gradually shifted classroom behavior analysis from manual observation to deep mining of large-scale video data. Student behavior detection occupies an important position in modern educational research and practice. In-depth exploration in this field can understand and optimize the teaching process from a new perspective, which has far-reaching significance for improving teaching quality and promoting the all-round development of students.

[0003] Through accurate student behavior detection, it is possible to reveal students' various behavior patterns in class, quantify students' concentration, participation, and interaction with teachers, and thus provide educators with objective data basis to guide the formulation and adjustment of teaching strategies, achieve teaching in accordance with students' aptitude, and improve teaching effectiveness. In addition, behavior detection can also help education managers understand the overall learning atmosphere of the class, discover potential teaching problems and students' psychological state, which is crucial to building a positive learning environment and enhancing students' learning motivation.

[0004] However, in the face of the digital transformation of education, it is difficult to analyze and evaluate these complex and changeable classroom behavior data manually. This not only requires a lot of time and effort, but is also inefficient and easily affected by subjective judgment. Therefore, it is crucial to develop an efficient, accurate and automated student behavior detection method. At present, there are two main types of automated student behavior detection methods: detection methods based on traditional machine learning and detection methods based on deep learning. Traditional machine learning methods usually include steps such as feature extraction, classifier training and behavior recognition. Although these methods can achieve good results in specific scenarios, they often perform poorly when dealing with complex classroom environments, especially when faced with occlusion, blur and diverse behavioral features. There are certain problems with detection accuracy and stability. At the same time, traditional one-stage and two-stage target detection algorithms often have complex calculation operations in the post-processing steps. These complex calculation operations include non-maximum suppression (NMS), which is used to filter redundant detection boxes and accurately locate targets. However, these steps usually require a lot of computing resources, which increases the time complexity and computational cost of the detection process. In addition, these algorithms often require additional adjustments and optimizations when dealing with multi-scale and multi-category targets, further increasing the complexity and computational burden of the system. This not only affects the detection speed and limits the real-time application, but also reduces the versatility and robustness of the algorithm to a certain extent. Especially in the face of dynamically changing complex environments, the post-processing steps of traditional algorithms will become a performance bottleneck.

[0005] It is not easy to extract valuable student behavior information from a large amount of classroom video data. The original classroom video data often contains various occlusion phenomena and complex interactive behavior patterns between students and teachers, which poses a huge challenge to effectively analyze and understand student behavior.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0007] (1) Automation technology was not used to assist teachers in analysis, resulting in low efficiency.

[0008] (2) Existing detection technologies do not consider the mutual influence and occlusion of student behaviors in complex classroom environments;

[0009] (3) The post-processing stage of existing detection technology is too complex and requires a lot of computing resources, which increases the time complexity and computational cost of the detection process. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides an end-to-end student behavior detection method and system based on large convolution kernels. Feature extraction is performed through a large convolution kernel backbone network, and the hybrid feature fusion architecture (HFFA) is combined to realize cross-level fusion of features, which are then input into the Transformer encoder-decoder structure for refinement and prediction, effectively improving the ability to detect student behavior in complex classroom environments.

[0011] The present invention is implemented as follows: an end-to-end student behavior detection method based on a large convolution kernel, comprising:

[0012] Step 1: Input the classroom environment student behavior image processing, scale the image to a fixed size to match the input size of the model, adjust the width and height of the image while keeping the aspect ratio of the image unchanged to avoid image deformation, and then perform data augmentation operations such as random rotation, translation, flipping, and brightness change.

[0013] Step 2: Design a large convolution kernel backbone network for upstream feature extraction, deeply extract the student behavior characteristics in the classroom environment, and obtain preliminary features through two types of large convolution kernels and the surrounding environment of the behavior.

[0014] Step 3: Introduce the hybrid feature fusion architecture HFFA to improve the model's feature extraction and fusion capabilities in complex and multi-scale scenarios. It combines the advantages of feature pyramid and coordinate attention mechanism, stacks multiple Transformer encoders, captures the global context information of the input feature map, and generates a high-dimensional feature representation, thereby adding position encoding to the feature map.

[0015] Step 4: The hybrid feature fusion architecture extracts the fused student behavior features and passes them into the decoder. Multiple decoders are stacked and use a fixed number of target queries. These queries are learnable embedding vectors that represent the targets that the model wants to detect. Finally, the output of the decoder predicts the target category through a linear layer and the bounding box position of the target through a regression head.

[0016] Step 5: Combine the output results of the encoder and the labels of the students’ behaviors in the classroom to predict the behavior of each student, construct the corresponding loss function, and combine the three losses for joint training.

[0017] Step 6: Use the trained parameters and weights to predict the student behavior in the classroom environment in the test set and obtain the corresponding results. Then analyze the predicted student behavior results, analyze the student behavior status in the classroom environment and visualize the analysis results.

[0018] Further, the step 1 specifically includes:

[0019] (1.1) Dataset preprocessing: including the acquisition of student behavior datasets in a classroom environment.

[0020] (1.2) Data augmentation operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training and test sets.

[0021] Further, the step 2 specifically includes:

[0022] (2.1) For the classroom student behavior image input in step 1, a large convolution kernel backbone network is designed to extract upstream student behavior features;

[0023] First, input the input image X into the ConvNeXt large convolution module:

[0024] X′=PWConv(GELU(PWConv(LN(DWConv(X)))))

[0025] Among them, PWConv stands for point-by-point convolution, GELU stands for Gaussian error linear unit, LN stands for layer normalization, and DWConv stands for depthwise separable convolution. The extracted X' feature is then input into the next large convolution module LSKBlock, and the upstream feature is extracted repeatedly.

[0026]

[0027] Further, the step three is specifically as follows:

[0028] (3.1) Use the multi-scale feature pyramid network structure (NASFCOSFPN) searched out by the spatial search mechanism to perform preliminary feature fusion and feature extraction.

[0029] (3.2) Combined with the coordinate attention mechanism, the input feature map is first globally pooled in the horizontal and vertical directions to generate a one-dimensional feature encoding. Then, these encodings are combined through convolution operations to form an attention feature map containing position information. Finally, a weight matrix is ​​generated through two 1×1 convolutional layers and a sigmoid function and applied to the input feature map to enhance the model's attention to specific locations:

[0030]

[0031] Generate the final weight matrix and apply it to the input feature map:

[0032]

[0033] (3.3) Integrate multiple Transformer encoders to capture global, contextual information and generate high-dimensional feature representations. At the same time, introduce a deformable attention mechanism in the encoder to reduce the amount of computation by focusing on only a few key points (rather than all positions) in the feature map. The multi-head attention mechanism performs global feature modeling. The specific formula is as follows:

[0034]

[0035] Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

[0036] Further, the step 4 specifically includes:

[0037] (4.1) After completing the extraction of upstream features and multi-scale feature fusion, the fused features are passed to the Transformer decoder for further processing. The self-attention module calculates the relationship between different positions in the feature map to capture global context information. Next, the cross-attention module calculates the relationship between the query and the encoder output features, and maps the features into target categories and bounding box predictions. In order to improve computational efficiency, the deformable attention mechanism DeformableAttention is introduced to optimize the calculation by focusing on only a few key point positions, specifically:

[0038]

[0039] The decoder receives the output feature sequence of the encoder and performs cross-attention calculations with a set of learnable queries. Each query represents a potential target detection result. Through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder and calculates the features related to each target.

[0040] Further, the step five specifically includes:

[0041] (5.1) Match the feature sequence output by the encoder with the pre-labeled student behavior labels. Each feature vector represents a potential student behavior. The decoder compares these features with the student behavior labels to predict the specific behavior of each student (such as raising hands, reading, discussing, etc.).

[0042] (5.2) In order to optimize the prediction effect of the model, multiple loss functions are constructed to measure the gap between the model prediction and the true label: the classification loss is used to measure the difference between the behavior category predicted by the model and the true label, and the bounding box regression loss is used to measure the distance between the behavior location predicted by the model and the actual location. The IoU loss is used to measure the overlap between the predicted bounding box and the true bounding box. By weighted summing these loss functions, a joint loss function is obtained to guide the training of the model so that it can optimize the accuracy of behavior prediction in multiple dimensions.

[0043] Further, the step six specifically includes

[0044] (6.1) After the model training is completed, the trained parameters and weights are applied to the test set. The data in the test set is new to the model. By predicting the behavior of the students in the test set, the model's performance in the actual scenario is evaluated. Based on the previously learned features and labels, the model predicts the behavior of each student in the test set and generates the corresponding bounding box and behavior category.

[0045] (6.2) Conduct in-depth analysis of the predicted student behavior results and evaluate the model's performance indicators (such as accuracy, recall, F1 score, etc.) to understand the model's performance on different behavior categories. Furthermore, through visualization methods such as confusion matrix, PR curve, and behavior detection box, the distribution and change trend of student behavior in different classroom environments are displayed. This kind of visualization analysis helps teachers or researchers to intuitively understand student behavior in the classroom and provide support for the formulation of educational strategies.

[0046] Another object of the present invention is to provide an end-to-end student behavior detection system based on a large convolution kernel for implementing the end-to-end student behavior detection method based on a large convolution kernel, comprising:

[0047] Data preprocessing module: used to obtain student behavior datasets in a classroom environment, and ensure data diversity and generalization capabilities through data enhancement operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training sets and test sets.

[0048] Upstream feature extraction module: By designing a large convolution kernel backbone network, the input student behavior image is subjected to feature extraction. The image is first input into the ConvNeXt large convolution module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features.

[0049] Multi-scale feature fusion module: A multi-scale feature pyramid network (NASFCOSFPN) constructed using a spatial search mechanism is used for preliminary feature fusion, and combined with the coordinate attention mechanism, an attention feature map containing position information is generated through global pooling and convolution operations, thereby enhancing the model's attention to specific locations.

[0050] Behavior decoding module: The fused features are passed to the Transformer decoder for further processing. The decoder calculates the relationship between features through self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism.

[0051] Loss calculation and model training module: Match the feature sequence output by the decoder with the pre-labeled student behavior labels, and calculate the joint loss function by constructing classification loss, bounding box regression loss and IoU loss to optimize the behavior prediction accuracy of the model and guide the joint training of the model.

[0052] Result generation and analysis module: Apply the trained model to the test set, predict student behavior, generate bounding boxes and behavior categories. Then, conduct in-depth analysis of the prediction results, evaluate model performance indicators, and use visualization to display the distribution and change trends of student behavior in different classroom environments to provide support for educational strategies.

[0053] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the end-to-end student behavior detection method based on a large convolution kernel.

[0054] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the end-to-end student behavior detection method based on a large convolution kernel.

[0055] Another object of the present invention is to provide an information data processing terminal, which includes the end-to-end student behavior detection system based on a large convolution kernel.

[0056] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0057] First, the end-to-end student behavior detection method based on large convolution kernels provided by the present invention aims to solve the above problems. This method makes full use of deep learning technology, improves the upstream feature extraction capability through advanced large convolution kernel technology, and ensures the accurate capture of each student's behavior; combined with the hybrid feature fusion architecture (HFFA) design, it extracts multi-scale features of the image through feature extraction layers of different scales, and introduces spatial position information at the same time, which enhances the perception of the target spatial distribution. Finally, the Transformer encoder-decoder structure is used to further process and refine the features, capture global context information, generate high-level feature representations, and finally generate behavior classification and bounding box predictions, and accurately locate and classify targets through attention mechanisms. This provides educational researchers with more sophisticated and comprehensive insights into classroom behavior, and promotes educational practice and research into a deeper data-driven stage.

[0058] (1) The present invention constructs an automated classroom student behavior detection and analysis system, which can effectively improve the efficiency of manual analysis and provide more accurate behavior analysis references for teachers, students and related researchers.

[0059] (2) The present invention adopts a feature extraction method with a large convolution kernel, which can better capture the complex behavioral characteristics in the classroom environment and improve the accuracy and robustness of detection.

[0060] (3) The present invention constructs an end-to-end behavior detection system based on multi-scale feature fusion and Transformer decoding, which combines global and local information, has higher recognition accuracy and can cope with diverse classroom scenarios.

[0061] (4) The present invention solves the problem of detecting various types of behaviors in real classroom scenarios, has stronger universality and adaptability, and is suitable for various different teaching environments.

[0062] Second, the end-to-end student behavior detection and analysis system based on large convolution kernels provided by the present invention provides an automated classroom student behavior analysis tool. The system effectively extracts the deep features of student behavior through a large convolution kernel network, and combines multi-scale feature fusion with the Transformer decoder to achieve accurate detection of student behavior in complex classroom environments. In order to cope with diverse scenarios in the classroom, the system also introduces a coordinate attention mechanism to enhance the model's attention to key behaviors. Compared with traditional student behavior analysis methods, the present invention has significantly improved the accuracy and efficiency of behavior recognition, overcoming the problem of insufficient detection accuracy of traditional methods in complex environments.

[0063] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0064] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0065] The end-to-end student behavior detection and analysis method based on large convolution kernels proposed in this invention improves the traditional behavior detection method by combining advanced feature extraction and multi-scale fusion technology. This method significantly improves the efficiency of classroom behavior analysis while improving the accuracy of behavior recognition. For educators, this technology can help them better understand students' behavior in the classroom and optimize teaching strategies. For students and researchers, the data analysis provided by the system can help them reflect on and study classroom behavior, which has great commercial value and application potential.

[0066] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0067] Existing student behavior detection methods often have low recognition accuracy when facing complex classroom environments, and fail to make full use of multi-scale features and spatial information. This invention effectively improves the detection ability of student behavior by introducing large convolution kernels and multi-scale feature fusion technology, and achieves more accurate behavior recognition in complex scenarios, filling the technical gap in this field in the industry.

[0068] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0069] At present, there is a lack of tools on the market that can automatically analyze classroom student behavior, and it mainly relies on manual analysis, which is inefficient. The present invention introduces end-to-end student behavior detection technology into classroom behavior analysis. Aiming at the problem of behavior detection in complex scenarios, the present invention solves the problem of inaccurate behavior recognition in complex environments in the prior art by combining large convolution kernels and multi-scale feature fusion technology. This technology can help education-related units to carry out their work in a more orderly manner and improve the efficiency and accuracy of classroom behavior analysis.

[0070] Fourth, the end-to-end student behavior detection method based on large convolution kernels proposed in the present invention mainly solves the problems of insufficient accuracy and robustness of student behavior detection in complex scenarios in the prior art, and achieves significant technological progress.

[0071] 1. Improved accuracy: Traditional student behavior detection methods often have difficulty accurately identifying student behavior in complex and changing classroom environments, especially when there are large lighting changes, occlusions, and behavioral diversity. By designing a large convolutional kernel backbone network, this invention can better capture key features in the classroom environment, effectively enhance the ability of feature extraction, and significantly improve the model's behavior detection accuracy in complex environments.

[0072] 2. Multi-scale feature fusion: In the processing of multi-scale features, traditional methods find it difficult to take into account both global information and local details at the same time, resulting in poor results when processing behaviors of different scales. The present invention successfully achieves effective fusion and extraction of features of different scales by introducing the hybrid feature fusion architecture HFFA, combined with the feature pyramid and coordinate attention mechanism. By stacking multiple Transformer encoders, the present invention can capture global contextual information and generate high-dimensional feature representations, so that students' behaviors can be accurately detected even in complex scenarios.

[0073] 3. Computational efficiency optimization: Existing complex behavior detection methods often have large computational workloads and are difficult to process in real time in practical applications. This paper introduces a deformable attention mechanism in the Transformer decoder, which greatly reduces the amount of computation by focusing only on a few key points in the feature map, improves the computational efficiency of the model, and enables the method to run efficiently in real-time application scenarios.

[0074] 4. Enhanced robustness: Through the combination of multiple data enhancement methods and feature extraction strategies, the present invention significantly enhances the robustness of the model in different classroom environments, and can cope with various complex situations, including lighting changes, student posture changes, and occlusion. Ultimately, the present invention achieves more stable and reliable student behavior detection results, and has broad industrial application prospects.

[0075] In summary, the present invention not only achieves significant technological progress in behavior detection accuracy, multi-scale feature processing, and computational efficiency, but also solves the problem of poor performance of traditional technologies in complex classroom environments by enhancing the robustness of the model, providing a more reliable and efficient solution for smart classroom monitoring and analysis systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of an end-to-end student behavior detection and analysis method based on a large convolution kernel provided by an embodiment of the present invention;

[0077] Figure 2 It is a structural schematic diagram of an end-to-end student behavior detection and analysis method based on a large convolution kernel provided by an example of the present invention;

[0078] Figure 3 It is a model structure diagram of a transformer encoder used in an end-to-end student behavior detection and analysis system based on a large convolution kernel provided by an example of the present invention;

[0079] Figure 4 It is a module structure diagram used in the end-to-end student behavior detection and analysis method based on a large convolution kernel provided by an example of the present invention;

[0080] Figure 5It is a schematic diagram for comparing experimental results provided by the examples of the present invention on a simulated classroom data set. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0082] like Figure 1 As shown, the end-to-end student behavior detection and analysis method based on a large convolution kernel provided by an embodiment of the present invention includes the following steps:

[0083] S01, obtain the student behavior dataset in the classroom environment, and ensure the diversity and generalization ability of the data through data enhancement operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training and test sets;

[0084] S02, first input the image into the ConvNeXt large convolution module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features;

[0085] S03, a multi-scale feature pyramid network built using a spatial search mechanism combined with a coordinate attention mechanism, generates an attention feature map containing position information through global pooling and convolution operations;

[0086] S04: The fused features are passed to the Transformer decoder for further processing. The decoder calculates the relationship between features through self-attention and cross-attention mechanisms to generate target category and bounding box predictions;

[0087] S05: The feature sequence output by the decoder is matched with the pre-labeled student behavior labels, and the joint loss function is calculated by constructing the classification loss, bounding box regression loss and IoU loss;

[0088] S06, the trained model is applied to the test set to predict student behaviors and generate bounding boxes and behavior categories. Subsequently, the prediction results are analyzed in depth to evaluate the model performance indicators;

[0089] like Figure 4 As shown, another object of the present invention is to provide an end-to-end student behavior detection and analysis system based on a large convolution kernel, comprising:

[0090] Data preprocessing module: used to obtain student behavior datasets in a classroom environment, and ensure data diversity and generalization capabilities through data enhancement operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training sets and test sets.

[0091] Upstream feature extraction module: By designing a large convolution kernel backbone network, the input student behavior image is subjected to feature extraction. The image is first input into the ConvNeXt large convolution module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features.

[0092] Multi-scale feature fusion module: A multi-scale feature pyramid network (NASFCOSFPN) constructed using a spatial search mechanism is used for preliminary feature fusion, and combined with the coordinate attention mechanism, an attention feature map containing position information is generated through global pooling and convolution operations, thereby enhancing the model's attention to specific locations.

[0093] Behavior decoding module: The fused features are passed to the Transformer decoder for further processing. The decoder calculates the relationship between features through self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism.

[0094] Loss calculation and model training module: Match the feature sequence output by the decoder with the pre-labeled student behavior labels, and calculate the joint loss function by constructing classification loss, bounding box regression loss and IoU loss to optimize the behavior prediction accuracy of the model and guide the joint training of the model.

[0095] Result generation and analysis module: Apply the trained model to the test set, predict student behavior, generate bounding boxes and behavior categories. Then, conduct in-depth analysis of the prediction results, evaluate model performance indicators, and use visualization to display the distribution and change trends of student behavior in different classroom environments to provide support for educational strategies.

[0096] The schematic diagram of the end-to-end student behavior detection and analysis system based on a large convolution kernel provided by an embodiment of the present invention is as follows: Figure 2 The symbols involved in the embodiment of the present invention are shown in Table 1.

[0097] Table 1 Symbols involved in the embodiments of the present invention

[0098]

[0099] Preprocessing of the data set: including the acquisition of student behavior data sets in a classroom environment.

[0100] (1.1) Data augmentation operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training and test sets.

[0101] The step 2 specifically includes:

[0102] (2.1) For the classroom student behavior image input in step 1, a large convolution kernel backbone network is designed to extract upstream student behavior features;

[0103] First, input the input image X into the ConvNeXt large convolution module:

[0104] X′=PWConv(GELU(PWConv(LN(DWConv(X)))))

[0105] Among them, PWConv stands for point-by-point convolution, GELU stands for Gaussian error linear unit, LN stands for layer normalization, and DWConv stands for depthwise separable convolution. The extracted X' feature is then input into the next large convolution module LSKBlock, and the upstream feature is extracted repeatedly.

[0106]

[0107] The step three is specifically as follows:

[0108] (3.1) Use the multi-scale feature pyramid network structure (NASFCOSFPN) searched out by the spatial search mechanism to perform preliminary feature fusion and feature extraction.

[0109] (3.2) Combined with the coordinate attention mechanism, the input feature map is first globally pooled in the horizontal and vertical directions to generate a one-dimensional feature encoding. Then, these encodings are combined through convolution operations to form an attention feature map containing position information. Finally, a weight matrix is ​​generated through two 1×1 convolutional layers and a sigmoid function and applied to the input feature map to enhance the model's attention to specific locations:

[0110]

[0111] Generate the final weight matrix and apply it to the input feature map:

[0112]

[0113] (3.3) Integrate multiple Transformer encoders to capture global, contextual information and generate high-dimensional feature representations. At the same time, introduce a deformable attention mechanism in the encoder to reduce the amount of computation by focusing on only a few key points (rather than all positions) in the feature map. The multi-head attention mechanism performs global feature modeling. The specific formula is as follows:

[0114]

[0115] Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

[0116] The step 4 specifically includes:

[0117] (4.1) After completing the extraction of upstream features and multi-scale feature fusion, the fused features are passed to the Transformer decoder for further processing. The self-attention module calculates the relationship between different positions in the feature map to capture global context information. Next, the cross-attention module calculates the relationship between the query and the encoder output features, and maps the features into target categories and bounding box predictions. In order to improve computational efficiency, the deformable attention mechanism DeformableAttention is introduced to optimize the calculation by focusing on only a few key point positions, specifically:

[0118]

[0119] The decoder receives the output feature sequence of the encoder and performs cross-attention calculations with a set of learnable queries. Each query represents a potential target detection result. Through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder and calculates the features related to each target.

[0120] The step five specifically includes:

[0121] (5.1) Match the feature sequence output by the encoder with the pre-labeled student behavior labels. Each feature vector represents a potential student behavior. The decoder compares these features with the student behavior labels to predict the specific behavior of each student (such as raising hands, reading, discussing, etc.).

[0122] (5.2) In order to optimize the prediction effect of the model, multiple loss functions are constructed to measure the gap between the model prediction and the true label: the classification loss is used to measure the difference between the behavior category predicted by the model and the true label, and the bounding box regression loss is used to measure the distance between the behavior location predicted by the model and the actual location. The IoU loss is used to measure the overlap between the predicted bounding box and the true bounding box. By weighted summing these loss functions, a joint loss function is obtained to guide the training of the model so that it can optimize the accuracy of behavior prediction in multiple dimensions.

[0123] The step six specifically includes

[0124] (6.1) After the model training is completed, the trained parameters and weights are applied to the test set. The data in the test set is new to the model. By predicting the behavior of the students in the test set, the model's performance in the actual scenario is evaluated. Based on the previously learned features and labels, the model predicts the behavior of each student in the test set and generates the corresponding bounding box and behavior category.

[0125] (6.2) Conduct in-depth analysis of the predicted student behavior results and evaluate the model's performance indicators (such as accuracy, recall, F1 score, etc.) to understand the model's performance on different behavior categories. Furthermore, through visualization methods such as confusion matrix, PR curve, and behavior detection box, the distribution and change trend of student behavior in different classroom environments are displayed. This kind of visualization analysis helps teachers or researchers to intuitively understand student behavior in the classroom and provide support for the formulation of educational strategies.

[0126] In the experiment of the present invention, the main development environment includes: Ubuntu 20.04, GTX 4090Ti, Chainer5.4.0, Python3.7, and the specific hyperparameter settings of the model are shown in Table 2.

[0127] Table 2 Experimental model hyperparameter settings

[0128] Hyperparameters Numeric batch_size 64 epoch 20 dropout 0.1 learning_rate 0.001 transformer_encoder_n_layers 6 transformer_encoder_n_heads 6

[0129] The student behavior detection and analysis system based on large convolution kernel and multi-scale feature fusion provided by the application embodiment of the present invention is applied to a computer device, and the computer device includes a memory and a processor, and a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the student behavior detection and analysis system based on large convolution kernel and multi-scale feature fusion.

[0130] The student behavior detection and analysis method based on a large convolution kernel provided in an application embodiment of the present invention is applied to an information data processing terminal, and the information data processing terminal is used to implement the student behavior detection and analysis system based on a large convolution kernel.

[0131] This paper compares the student behavior detection and analysis method based on large convolution kernels with the existing end-to-end behavior detection method. The evaluation of the model effect is judged by calculating the model's accuracy (Precision), recall rate (Recall) and F1 score:

[0132] Recall refers to the proportion of samples that the model correctly identifies among all samples that actually belong to a certain behavior category. The formula is:

[0133]

[0134] Precision refers to the proportion of samples that actually belong to a certain behavior category among those predicted by the model. The formula is:

[0135]

[0136] · The F1 Score is the harmonic mean of precision and recall, used to balance the weights of the two.

[0137] From the experimental results, it can be seen that the student behavior detection and analysis method based on large convolutional kernels and multi-scale feature fusion provided by the present invention has demonstrated excellent performance in multiple experiments. First, through ablation experiments, we evaluated the contribution of each module to the overall performance of the model. After removing the LKNeXt backbone network, the mAP of the model decreased by 1.2% and the recall rate decreased by 1.8%, indicating that the large convolutional kernel plays a key role in extracting complex behavior features. Similarly, when the HFFA hybrid feature fusion architecture was removed, the mAP and recall rate of the model decreased by 1.5% and 1.7% respectively, further verifying the importance of HFFA in multi-scale feature fusion and global information capture. The ablation experiments show that each module in the present invention has made a significant contribution to improving the overall performance of the model.

[0138]

[0139]

[0140] Secondly, through comparative experiments, we compared the model of the present invention in detail with the traditional Deformable DETR model. On the SCB1 classroom behavior dataset, the mAP of the model of the present invention increased by 1.5% and the recall rate increased by 2%, significantly outperforming the traditional model. This is mainly due to the design of the LKNeXt backbone network, which effectively captures the detailed information of student behavior through convolutional kernels with larger receptive fields. In addition, the HFFA hybrid feature fusion architecture further enhances the model's ability to fuse and extract global and local information in complex scenarios by introducing an attention mechanism. The results of the comparative experiments prove that the present invention not only surpasses the traditional method in detection accuracy, but also performs more stably when dealing with multi-scale targets and complex scenarios.

[0141] Next, taking the classroom interaction network graph formed in this class as an example, we analyze the classroom interaction situation of this class.

[0142] First, conduct an overall behavior analysis. We will calculate the frequency density of student behaviors detected by the model ρ = m / (n × t), where m represents the total number of detected behaviors, n represents the number of students participating in the class, and t represents the total duration of the class. We believe that when ρ > Threshold, it indicates that students are active in class and have a high participation rate; while when ρ < Threshold, it indicates that the classroom participation rate is relatively low. In this way, teachers can initially judge the activity of the classroom and the overall participation of students.

[0143] Next, conduct individual behavior analysis. Individual behavior analysis mainly measures students' classroom performance through two indicators:

[0144] Participation Index: It is measured by the frequency of each student's behavior detected by the model. The higher the participation index, the more active the student is in class and the more active the student is. The calculation formula is: The participation index of each student is equal to the total number of behavior detections of the student in the entire class.

[0145] Focus Index: This is measured by analyzing how long students stay focused in class. The higher the focus index, the more focused the student is in class.

[0146] Behavioral engagement can help teachers understand each student's active participation in class; while behavioral concentration reflects students' concentration in learning, which is crucial for evaluating students' learning outcomes.

[0147] We believe that if in an active class, the teacher's behavioral participation is the highest and the behavioral concentration is high, it means that the teacher is in a dominant and controlling position in the class, and it can be identified as a dominant class; if the students' behavioral participation and concentration are high, it can be identified as a discussion class. If in a class with low participation, the teacher's behavioral participation is the highest, it can be identified as a lecture class; if a student's behavioral participation and concentration are high, it can be identified as a personal display class.

[0148] Taking a class as an example, the total number of behavioral tests is 120 times, the total class time is 60 minutes, and the calculated behavior density ρ indicates that the class participation is medium. At the same time, in this class, some students have a high degree of behavioral participation, while some students show low behavioral participation and concentration, which provides a basis for teachers to adjust teaching strategies in a targeted manner. Students can also make timely adjustments based on their personal performance in class to better participate in classroom learning.

[0149] Finally, the system will generate a detailed student behavior analysis report, recording the type of class participation, each student's behavior data, and the overall class performance for reference. In summary, the end-to-end student behavior detection system based on large convolution kernels proposed in the present invention has played a significant role in classroom education scenarios. It not only effectively solves the efficiency and accuracy problems of student behavior analysis in actual classrooms, but also realizes the automated processing of student behavior detection in the classroom. By quickly identifying and accurately marking students' behaviors, the present invention can fully utilize classroom video resources, greatly reducing the workload of teachers, and has the ability to monitor and analyze student behavior in real time, which helps teachers to grasp the interaction in the classroom in a timely manner, discover potential teaching problems and improve them, thereby improving teaching effectiveness, enhancing students' learning enthusiasm and participation, and promoting the efficient conduct of classroom education activities.

[0150] Example 1: Intelligent classroom behavior analysis system

[0151] In the modern education environment, teachers need to understand and analyze students' behaviors in real time in order to adjust teaching strategies and improve teaching effectiveness. However, traditional behavior analysis mainly relies on teachers' subjective observations, which has problems such as insufficient accuracy and poor real-time performance.

[0152] Implementation process:

[0153] 1. System deployment: Install high-resolution cameras in the classroom to capture students’ behavior images in real time. The image data is transmitted to the server via the network for processing.

[0154] 2. Behavior detection and analysis: The end-to-end student behavior detection method based on large convolution kernels of the present invention is used to process the real-time captured images. The system classifies each student's behavior, such as concentration, raising hands, distraction, etc., and generates each student's behavior record and behavior statistics.

[0155] 3. Teacher feedback and strategy adjustment: The behavioral data generated by the system is presented to teachers through visualization tools, such as real-time behavior monitoring charts, behavior trend analysis charts, etc. Teachers adjust teaching strategies according to students' behavioral status, such as interactive questions and content explanation methods.

[0156] The embodiment of the present invention realizes the automatic and real-time analysis of students' classroom behavior, and improves the accuracy of classroom management. The embodiment of the present invention provides a data-driven teaching feedback tool to help teachers understand students' status in time and optimize the teaching process.

[0157] Example 2: Learning behavior monitoring on online education platform

[0158] In online education platforms, it is difficult for teachers and educational institutions to monitor students' learning behavior in real time, which makes it difficult to ensure students' learning results. By introducing an intelligent behavior detection system, students' learning status can be monitored in real time, improving the teaching quality of online education.

[0159] 1. Integration with online education platform: The student behavior detection system based on the present invention is integrated into the online education platform. When students participate in online courses through cameras, the system captures the students' learning behavior images in real time.

[0160] 2. Behavior detection and feedback: The system detects students’ learning behaviors in real time, such as concentration, distraction, and leaving seats. Based on the detection results, the platform will provide instant feedback, such as reminding students of inattention, recording study time, and analyzing learning behavior patterns.

[0161] 3. Learning effect evaluation and personalized learning plan: The system generates a learning report based on the student's learning behavior data and provides it to teachers or educational institutions. The report contains data such as the student's learning concentration and behavior change trends. Based on the report, the platform can recommend personalized learning content and strategies for students to improve learning effects.

[0162] The embodiments of the present invention enhance the intelligence level of the online education platform, realize real-time monitoring and feedback of students' learning behaviors, provide personalized learning solutions based on behavioral data, and improve students' learning effects and participation.

[0163] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVDROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0164] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An end-to-end student behavior detection method based on large convolution kernels, characterized in that: Firstly, the images of student behaviors in classroom environments are preprocessed, and the integrity and diversity of the images are maintained through data augmentation operations. Secondly, a large convolution kernel backbone network is designed to deeply extract student behavior features through large convolution kernels, and combined with the hybrid feature fusion architecture HFFA, feature pyramid and coordinate attention mechanism are used to fuse multi-scale features and capture global context information. Then, the fused features are further processed through the Transformer decoder to predict the category and bounding box of student behaviors. Finally, the prediction accuracy of the model is optimized by constructing a joint training model with multiple loss functions, and the model is verified and visualized on the test set to evaluate its performance in actual scenarios.

2. The end-to-end student behavior detection method based on large convolution kernel as claimed in claim 1, characterized in that: include: Step 1: Input the classroom environment student behavior image processing, scale the image to a fixed size to match the model's input size, adjust the image width and height while keeping the image aspect ratio unchanged to avoid image deformation, and then perform random rotation, translation, flipping, and brightness change data augmentation operations; Step 2: Design a large convolution kernel backbone network for upstream feature extraction, deeply extract the student behavior characteristics in the classroom environment, and obtain preliminary features through two types of large convolution kernels and the surrounding environment of the behavior; Step 3: Introduce the hybrid feature fusion architecture HFFA to improve the model's feature extraction and fusion capabilities in complex and multi-scale scenarios. It combines the advantages of feature pyramid and coordinate attention mechanism, stacks multiple Transformer encoders, captures the global context information of the input feature map, generates high-dimensional feature representation, and thus adds position encoding to the feature map; Step 4: The hybrid feature fusion architecture extracts the fused student behavior features and passes them into the decoder. Multiple decoders are stacked and use a fixed number of target queries. These queries are learnable embedding vectors that represent the objects that the model wants to detect. Finally, the output of the decoder is passed through a linear layer to predict the object category and a regression head to predict the bounding box location of the object. Step 5: Combine the output of the encoder and the labels of the students’ behaviors in the classroom to predict the behavior of each student, construct the corresponding loss function, and conduct joint training with the three losses; Step 6: Use the trained parameters and weights to predict the student behavior in the classroom environment in the test set and obtain the corresponding results. Then analyze the predicted student behavior results, analyze the student behavior status in the classroom environment and visualize the analysis results.

3. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step 1 specifically includes: (1.3) Dataset preprocessing: including the acquisition of student behavior datasets in a classroom environment; (1.4) Data augmentation operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training and test sets.

4. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step 2 specifically includes: (2.1) For the classroom student behavior image input in step 1, a large convolution kernel backbone network is designed to extract upstream student behavior features; First, input the input image X into the ConvNeXt large convolution module: X′=PWConv(GELU(PWConv(LN(DWConv(X))))) Among them, PWConv represents point-by-point convolution, GELU represents Gaussian error linear unit, LN represents layer normalization, and DWConv represents depthwise separable convolution; then the extracted X' feature is input into the next large convolution module LSKBlock, and the upstream feature is extracted repeatedly; 5. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step three is specifically as follows: (3.1) Use the multi-scale feature pyramid network structure searched by the spatial search mechanism to perform preliminary feature fusion and feature extraction; (3.2) Combined with the coordinate attention mechanism, the input feature map is first globally pooled in the horizontal and vertical directions to generate a one-dimensional feature encoding; then, these encodings are combined through a convolution operation to form an attention feature map containing position information. Finally, a weight matrix is ​​generated through two 1×1 convolutional layers and a sigmoid function and applied to the input feature map, thereby enhancing the model's attention to specific locations: Generate the final weight matrix and apply it to the input feature map: (3.3) Integrate multiple Transformer encoders to capture global,contextual information and generate high-dimensional feature representations. At the same time, introduce a deformable attention mechanism in the encoder to reduce the amount of computation by focusing on only a few key points in the feature map. The multi-head attention mechanism performs global feature modeling. The specific formula is as follows: Where m represents the number of attention heads, Amqk is the attention weight, and W and Wm' are learnable weight matrices.

6. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step 4 specifically includes: (4.1) After completing the extraction of upstream features and multi-scale feature fusion, the fused features are passed to the Transformer decoder for further processing; the self-attention module calculates the relationship between different positions in the feature map to capture global context information; then, the cross-attention module calculates the relationship between the query and the encoder output features, and maps the features into target categories and bounding box predictions; in order to improve computational efficiency, the deformable attention mechanism DeformableAttention is introduced to optimize the computation by focusing on only a few key point positions, specifically: The decoder receives the output feature sequence of the encoder and performs cross-attention calculations with a set of learnable queries; each query represents a potential target detection result; through the cross-attention mechanism, the decoder matches each query with the global features output by the encoder and calculates features related to each target.

7. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step five specifically includes: (5.1) Match the feature sequence output by the encoder with the pre-labeled student behavior labels; each feature vector represents a potential student behavior, and the decoder compares these features with the student behavior labels to predict the specific behavior of each student; (5.2) In order to optimize the prediction effect of the model, multiple loss functions are constructed to measure the gap between the model prediction and the true label: the classification loss is used to measure the difference between the behavior category predicted by the model and the true label, the bounding box regression loss is used to measure the distance between the behavior position predicted by the model and the actual position; the IoU loss is used to measure the degree of overlap between the predicted bounding box and the true bounding box; by weighted summing these loss functions, a joint loss function is obtained to guide the training of the model so that it can optimize the accuracy of behavior prediction in multiple dimensions.

8. The end-to-end student behavior detection method based on large convolution kernels as described in claim 2, characterized in that: The step six specifically includes (6.1) After the model training is completed, the trained parameters and weights are applied to the test set; the data in the test set is new to the model, and the performance of the model in the actual scenario is evaluated by predicting the behavior of the students in the test set; The model predicts the behavior of each student in the test set based on the previously learned features and labels, and generates the corresponding bounding box and behavior category; (6.2) Conduct in-depth analysis of the predicted student behavior results and evaluate the performance indicators of the model to understand the performance of the model in different behavior categories; use visualization methods such as confusion matrix, PR curve, and behavior detection box to display the distribution and change trend of student behavior in different classroom environments; this visualization analysis helps teachers or researchers to intuitively understand student behavior in the classroom and provide support for the formulation of educational strategies.

9. An end-to-end student behavior detection system based on a large convolution kernel that implements the end-to-end student behavior detection method based on a large convolution kernel as described in any one of claims 1 to 8, characterized in that: include: Data preprocessing module: used to obtain student behavior datasets in classroom environments, and ensure data diversity and generalization capabilities through data enhancement operations such as random rotation, translation, flipping, brightness change, mosaic, and division of training and test sets; Upstream feature extraction module: By designing a large convolution kernel backbone network, the input student behavior image is subjected to feature extraction. The image is first input into the ConvNeXt large convolution module, and then the extracted features are input into LSKBlock to extract the upstream student behavior features. Multi-scale feature fusion module: A multi-scale feature pyramid network constructed using a spatial search mechanism is used for preliminary feature fusion. Combined with the coordinate attention mechanism, an attention feature map containing position information is generated through global pooling and convolution operations, thereby enhancing the model's attention to specific locations. Behavior decoding module: passes the fused features to the Transformer decoder for processing; The decoder calculates the relationship between features through self-attention and cross-attention mechanisms, generates target category and bounding box predictions, and optimizes computational efficiency by introducing a deformable attention mechanism; Loss calculation and model training module: Match the feature sequence output by the decoder with the pre-labeled student behavior labels, construct classification loss, bounding box regression loss and IoU loss, calculate the joint loss function to optimize the model's behavior prediction accuracy and guide the joint training of the model; Result generation and analysis module: Apply the trained model to the test set, predict student behavior, generate bounding boxes and behavior categories; then, conduct in-depth analysis of the prediction results, evaluate model performance indicators, and use visualization to display the distribution and change trends of student behavior in different classroom environments to provide support for educational strategies.

10. An information data processing terminal, comprising the end-to-end student behavior detection system based on a large convolution kernel as described in claim 9.

Citation Information

Patent Citations

  • Student behavior identification method, device and system and computer readable storage medium

    CN117152846A

  • Classroom behavior recognition method based on double-flow convolutional neural network

    CN118135496A

Cited By

  • Student classroom learning state evaluation method and system based on artificial intelligence

    CN120032300A

  • Student classroom behavior detection method based on Transform and improved VGG16 model

    CN120236239A

  • Multi-level facial feature fusion method and system for evaluating student participation degree, and medium

    CN120411712A

  • Method suitable for character behavior analysis based on multi-modal training model

    CN121708648A