Lightweight wisdom classroom learning behavior detection method and system

Abstract

The invention belongs to the technical field of computer vision and artificial intelligence, and discloses a lightweight wisdom classroom learning behavior detection method and system, and the method comprises the steps: extracting student behavior features in a classroom image, and obtaining preliminary multi-scale features; designing a deep convolutional neural network to perform deep spatial-temporal feature representation on the extracted features, and introducing a focusing modulation module to enhance the features; designing a dynamic detection head to fuse and process features of different scales; and calculating the difference between a prediction result and a real label by using the optimized classification loss function and bounding box loss function, optimizing model parameters, and outputting a final student behavior detection result through the model. And finally, the system performs detailed analysis on the detected student behaviors and generates a visual classroom behavior report. According to the method, more accurate student behavior detection in a classroom scene is realized, a complex classroom environment and diversified student behavior modes can be effectively processed, and the method has relatively high practicability.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and in particular, relates to a lightweight smart classroom learning behavior detection method and system. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and computer vision technology, the demand for real-time monitoring and analysis of student behavior in the field of intelligent education has increased. These technologies can not only help improve the quality of teaching, but also promote students' active participation. However, existing behavior detection methods still face many challenges in practical applications, including behavior diversity, scene complexity, sample imbalance, real-time requirements, and computing resource limitations. These challenges limit the performance of existing methods in accurate detection and real-time processing. First, students have a wide variety of behaviors in the classroom, such as raising hands, standing, walking, reading, etc. Existing methods are difficult to detect and identify these behaviors comprehensively and accurately, especially when multiple behaviors occur at the same time, the detection is more difficult. Second, the classroom environment is usually complex and changeable, with a variety of interference factors, such as lighting changes, occlusion, multi-person interaction, etc. These factors can significantly affect the accuracy of behavior detection, and existing methods often perform poorly when dealing with these complex scenarios. In addition, the number of different behavior samples in the behavior detection task is often unbalanced, and some behaviors have fewer samples, which makes the model prone to overfitting or underfitting during training, thus affecting the detection effect. Real-time performance is a key requirement for classroom behavior detection. Teachers need to understand students' status and behavior in a timely manner in order to make corresponding teaching adjustments. However, existing methods face great challenges in achieving low-latency processing while ensuring high accuracy, especially for resource-constrained educational environments, which are difficult to meet the needs of real-time processing. In addition, many existing methods usually require high computing resources to achieve high-precision detection, which is difficult to achieve in practical applications, especially in educational scenarios with limited equipment and resources.

[0003] In order to meet the above challenges, the present invention proposes a lightweight smart classroom learning behavior detection method and system. By introducing a multi-scale feature fusion module, features of different scales can be effectively extracted and fused to improve the detection accuracy and robustness of the model. Multi-scale feature fusion technology captures richer and more comprehensive behavior information by fusing features of different scales, thereby improving the accuracy and robustness of detection. This method performs well in dealing with complex scenarios and diverse behaviors.

[0004] At present, the mainstream student behavior detection methods mainly rely on deep learning technology, especially the application of convolutional neural network (CNN). CNN extracts local features of images through convolution operations and synthesizes higher-level abstract features layer by layer. Convolutional layer, pooling layer and fully connected layer are the basic components of CNN. The convolutional layer scans the image through the convolution kernel to extract local features such as edges and textures; the pooling layer reduces the size of the feature map through downsampling operations, retains the main features, and reduces the computational complexity; the fully connected layer integrates the features extracted by the convolutional layer and the pooling layer to perform the final target detection task. However, most of these methods rely on large-scale pre-trained models. Although these models perform well in classification and detection tasks, they have high computational complexity and resource requirements and are not suitable for classroom environments with limited resources. In addition, the existing methods are prone to problems such as inaccurate detection and missed detection when dealing with multiple behaviors in complex scenes. At the same time, the datasets designed for the classroom student behavior task are often unbalanced. In most cases, students' behaviors are concentrated on simple samples such as reading and looking up, resulting in a large number of samples for these two behaviors. In contrast, samples of other difficult behaviors such as standing, raising hands, and turning heads are relatively scarce, which easily blocks other behaviors and leads to poor model training results. These have brought certain challenges to classroom student behavior detection.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) The lack of lightweight and efficient detection models in the education field makes it difficult to monitor and analyze student behavior in real time.

[0007] (2) Classroom student behavior datasets usually have an imbalance in the number of samples, which results in poor detection results.

[0008] (3) In the actual classroom environment, students’ behaviors vary in size and often there is occlusion. The body shapes, postures, and positions of different students vary greatly, and the occlusion of objects such as desks and chairs poses a huge challenge to behavior detection. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a lightweight smart classroom learning behavior detection method and system.

[0010] The present invention is implemented by designing a lightweight network structure, which aims to significantly reduce the computational complexity of the model while improving the accuracy and robustness of detection. The multi-scale feature fusion module captures more detailed and comprehensive behavioral information by combining features of different scales, especially in complex scenarios, which can effectively distinguish and identify different behaviors, while combining the modification of the loss function to balance the problem of uneven distribution of data sets. Thereby promoting the development of the task of student behavior detection in the field of education.

[0011] Further, including:

[0012] Step 1: Extract frame-level features from the classroom environment image; input the classroom environment image and process it, scale it to a specific size of n×n, extract multi-scale features in the image as a feature vector representation of student behavior;

[0013] Step 2: Extract local features through convolution operation, then improve linear expression ability through batch normalization and Silu activation function, and combine cross-layer connection to extract student behavior characteristics in the classroom environment in layers to improve feature fusion and expression capabilities. In this step, shallow features P3 and middle features P4 are generated;

[0014] Step 3: Introduce a focus modulation mechanism to replace the traditional self-attention mechanism, thereby further improving the model's feature extraction capability; its specific role is to receive the processed value passed in step 2, use a deep convolutional layer stack to capture the visual context from local to global, and then use a gating mechanism to adaptively aggregate features at different levels according to the query content, and finally use a simple modulator to fuse the aggregated context features into the query, and finally generate a deeper feature map P5;

[0015] Step 4: According to the three hierarchical features P3, P4, and P5 generated in the previous step, further upsample and merge them to achieve multi-scale feature fusion and capture more detailed and close feature associations;

[0016] Step 5: Pass the multi-level features into the dynamic detection head, enhance and fuse these features through various attention mechanisms to further improve the feature expression ability;

[0017] Step 6: The model finally outputs the predicted student behavior detection results; these predicted results need to be compared with the real labels through the constructed loss function to optimize the model parameters;

[0018] Step 7: Use the trained parameters to predict the student behavior in the test set classroom and analyze the results, and provide a visual analysis result.

[0019] Further, the step 1 specifically includes:

[0020] Dataset preprocessing: including the collection of classroom student behavior datasets and the division of training sets and test sets;

[0021] The collected and divided data sets are rotated and scaled for data enhancement, and the images are made into a consistent size, and finally the processed input image X is obtained. resize =resize(X,(640,640)).

[0022] Further, the step 2 specifically includes:

[0023] Feature extraction is performed through the initial convolution layer, and then batch normalization and Silu activation function processing are performed based on the output of the initial convolution layer:

[0024] C1=SiLu(BatchNorm(Conv(X resized )))

[0025] After the shallow feature P3 is generated through the shallow convolution layer and cross-layer connection, the middle-level feature P4 is further extracted through the feature extraction module C2f module:

[0026] P3=C2f(C1)

[0027] P4=C2f(P3).

[0028] Further, the step three specifically includes:

[0029] The focus modulation mechanism module is designed at this position to extract the deepest features of P5. It first projects the input feature map X into the new feature space through a linear layer to generate the initial feature map Z_0:

[0030] Z0=f z (X)∈R H×W×C

[0031] A set of deep convolutional layer stacks are used to extract contextual information from local to global; the output of each layer is:

[0032]

[0033] Then, a linear layer is used to obtain the spatial and hierarchical gating weights G, and the feature maps at different levels are weighted summed to generate a single feature map Z_out:

[0034]

[0035] Finally, the aggregated context features are processed by a simple modulator function h and multiplied with the query projection function q(xi) to complete feature modulation:

[0036] y i =q(xi)⊙h(Z out )

[0037] Further, the step 4 specifically includes:

[0038] Upsample the three-level features P3, P4, and P5:

[0039] P3 upsampled =Upsample(P3)

[0040] P4 upsampled =Upsample(P4)

[0041] Concatenate and fuse the upsampled features with the corresponding hierarchical features:

[0042] P fused =Concat(P3 upsampled ,P4 upsampled ,P5)

[0043] Further, the step five specifically includes:

[0044] Receive the multi-level feature maps P3, P4, and P5 from the Neck layer;

[0045] Introducing scale-aware attention mechanism π L , feature enhancement based on the importance of different scales:

[0046]

[0047] Where σ(·) is a linear function using 1×1 convolution and hard sigmoid activation function;

[0048] Introduce a spatially aware attention mechanism to focus on features at specific spatial locations:

[0049] π S (F) = Conv(F)

[0050] Finally, we introduce the task-aware attention mechanism to focus on the importance of the output channel:

[0051] π C (F)=FullyConnected(ReLU(Normalize(F)))

[0052] The outputs of the three attention mechanisms are fused to generate an enhanced feature map;

[0053] The step six specifically includes:

[0054] During the training phase, the corresponding loss is first calculated using the prediction results and the speaker label vector, and then measured by combining the classification loss and the bounding box loss:

[0055] L=λ cls L cls +λ box L box

[0056] First, the simple samples and difficult samples are distinguished according to the IoU between the predicted box and the true box; the mean IoU of all bounding boxes is used as the threshold μ; samples with IoU lower than μ are regarded as negative samples, and samples higher than μ are regarded as positive samples. For simple samples and difficult samples, the sliding weighting function Slide is used to emphasize the boundary samples; the specific Slide weighting function is as follows:

[0057]

[0058] Among them, x represents the IoU between the predicted box and the true box, and μ represents the weighted threshold. In order to enhance the robustness of the parameter μ, the concept of exponential moving average (EMA) is introduced to smooth the threshold μ and reduce the significant fluctuations caused by outliers.

[0059] μ t =β×μ t -1+(1-β)×θ t

[0060] For classification loss, binary cross entropy loss (BCE) is used to calculate the classification loss;

[0061] The step seven specifically includes:

[0062] After the model is trained, it is predicted by inputting the test set data, and the model will output the classification probability and bounding box position. The prediction results and the true labels are combined to calculate the model's precision, recall rate, and mean average precision (mAP), and the results are analyzed to evaluate the performance of the model in practical applications.

[0063] Another object of the present invention is to provide a lightweight smart classroom learning behavior detection system comprising:

[0064] Classroom student behavior extraction module: This module is responsible for extracting deep student behavior features from images; through convolution operations and cross-layer connections, it extracts student behavior features in the classroom environment layer by layer, and finally generates a multi-level feature map;

[0065] Context-focused modulation feature extraction enhancement module: This module uses the context-focused modulation mechanism to capture visual context from local to global through deep convolutional layer stacks and gated aggregation methods; it adaptively fuses features at different levels according to the query content and finally generates a deeper feature map;

[0066] Multi-scale feature fusion module: This module upsamples and concatenates feature maps at different levels to achieve multi-scale feature fusion. Through multi-scale feature fusion, more detailed and close feature associations can be captured.

[0067] Multi-attention mechanism fusion module: This module uses a dynamic detection head and multiple attention mechanisms to enhance and process the fused feature map. Specifically, it includes scale-aware attention, space-aware attention, and task-aware attention to further refine and optimize the features.

[0068] Student behavior prediction module: This module uses the feature map output by the dynamic detection head for behavior prediction, calculates the classification probability and bounding box position; measures the difference between the predicted category probability and the true category label through binary cross entropy loss (BCE), and uses exponential moving average loss to measure the difference between the predicted bounding box position and the true bounding box position;

[0069] Result generation and analysis module: This module uses the prediction results to generate and analyze the detection report; distinguish the student behavior in each frame and generate a visual report to help teachers and students understand classroom behavior and improve teaching methods and learning habits.

[0070] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the lightweight smart classroom learning behavior detection method.

[0071] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the lightweight smart classroom learning behavior detection method.

[0072] Another object of the present invention is to provide an information data processing terminal, which is used to implement the lightweight smart classroom learning behavior detection system.

[0073] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0074] Construction of an automated detection system: The present invention constructs an automated classroom student behavior detection system, which can effectively improve the efficiency of manual detection and analysis, and better provide reference for teachers, students, and related researchers.

[0075] Feature extraction and enhancement: This invention introduces multi-scale feature fusion technology, which captures more detailed and comprehensive behavior information by combining features of different scales, and can effectively distinguish and identify different behaviors in complex scenes. The focus modulation mechanism further enhances the ability of feature extraction and significantly improves the accuracy and efficiency of detection.

[0076] Lightweight design: This paper designs a lightweight network structure, which significantly reduces the computational complexity of the model. It is suitable for resource-constrained educational environments and can achieve efficient and real-time student behavior detection.

[0077] Solution to the sample imbalance problem: By optimizing the loss function, the present invention effectively solves the problem of sample imbalance in the data set. The improved bounding box loss exponential moving average loss function is adopted to enhance the robustness of the model for samples of different difficulty levels and improve the overall detection performance.

[0078] Compared with traditional behavior detection methods, the present invention has significant improvements in detection accuracy, efficiency and resource utilization, and is particularly suitable for applications in educational scenarios that have high requirements for real-time performance and computing resources.

[0079] The lightweight smart classroom learning behavior detection method proposed in this invention improves the traditional behavior detection method by combining special scenarios, and can detect student behavior in the classroom more accurately, thereby more efficiently conducting classroom management and teaching research. It has huge commercial value and application prospects.

[0080] Existing student behavior detection methods often perform poorly in complex scenarios and do not fully utilize the multi-scale information in the image. The present invention uses multi-scale feature fusion and focus modulation mechanism to achieve more accurate student behavior detection, filling the technical gap in this field at home and abroad.

[0081] Currently, the education field lacks tools for automated detection of student behavior in the classroom, and manual analysis is inefficient. This invention combines deep learning technology with the actual classroom environment to solve the problem of behavior detection in complex scenarios. Through multi-scale feature fusion and loss function optimization, it significantly improves the accuracy and efficiency of detection, which can help educators carry out teaching work more orderly.

[0082] The technical solution of the present invention aims to solve the problem of insufficient efficiency and accuracy of smart classroom learning behavior detection in the prior art. Traditional detection methods usually rely on complex and computationally intensive models, which are difficult to apply in real time in actual classroom environments, and lack multi-scale and multi-level feature fusion processing, resulting in low accuracy and robustness of behavior detection.

[0083] By introducing a lightweight model architecture, the present invention proposes a smart classroom learning behavior detection method based on multi-level feature extraction and focus modulation mechanism. This method can effectively extract multi-scale features from classroom environment images, and use technical means such as convolution operations, batch normalization, Silu activation functions, etc. to extract and fuse features in layers. At the same time, the focus modulation mechanism replaces the traditional self-attention mechanism, further improving the ability and efficiency of feature extraction. In addition, the present invention introduces a variety of attention mechanisms to enhance features for different scales, spaces, and task features, greatly improving the model's behavior detection accuracy.

[0084] The technical progress of the present invention is mainly reflected in the following aspects:

[0085] 1) Lightweight model design: By reducing the computational complexity and the amount of model parameters, the detection method of the present invention can perform behavior detection faster while ensuring high accuracy, and is suitable for real-time applications.

[0086] 2) Multi-scale feature fusion: The present invention effectively captures more detailed feature associations through upsampling and multi-level feature fusion operations, thereby improving the robustness and accuracy of detection.

[0087] 3) Innovative feature extraction mechanism: By introducing focus modulation mechanism and multiple attention mechanisms, the present invention can extract and enhance features more accurately, overcoming the shortcomings of traditional methods in feature extraction.

[0088] 4) Comprehensive performance improvement: This technical solution not only improves the accuracy and recall rate of behavior detection, but also achieves significant optimization in model calculation efficiency and resource usage, making it suitable for application in actual classroom scenarios.

[0089] In summary, the present invention not only solves the problems of the prior art, but also achieves significant technical progress in the method of detecting learning behavior in smart classrooms. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a flow chart of a lightweight smart classroom learning behavior detection method provided by an embodiment of the present invention.

[0091] Figure 2 It is a schematic diagram of a lightweight smart classroom learning behavior detection system provided by an embodiment of the present invention.

[0092] Figure 3 It is a structural block diagram of a lightweight smart classroom learning behavior detection system provided by an embodiment of the present invention.

[0093] Figure 4 The embodiments of the present invention provide (a) monotone partial parse tree (SPPF), (b) focus modulation, (c) context aggregation, and (d) self-attention map.

[0094] Figure 5 This is a physical diagram of the improved model provided by an embodiment of the present invention.

[0095] Figure 6 It is a comparison chart of mAP50-95 across training cycles (Epochs) provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0097] like Figure 1 As shown, a lightweight smart classroom learning behavior detection method provided by an embodiment of the present invention includes the following steps:

[0098] S101: Extract frame-level features from the classroom environment image; input the classroom environment image and process it, scale it to a specific size of n×n, extract multi-scale features in the image as a feature vector representation of student behavior;

[0099] S102: Extract local features through convolution operation, then improve linear expression ability through batch normalization and Silu activation function, and combine cross-layer connection to extract student behavior characteristics in the classroom environment in layers, improve feature fusion and expression capabilities. In this step, shallow features P3 and middle features P4 are generated;

[0100] S103: Introduce a focus modulation mechanism to replace the traditional self-attention mechanism, thereby further improving the model's feature extraction capability; its specific role is to receive the processed value passed in step 2, use a deep convolutional layer stack to capture the visual context from local to global, and then adaptively aggregate features of different levels according to the query content through a gating mechanism, and finally fuse the aggregated context features into the query through a simple modulator, and finally generate a deeper feature map P5;

[0101] S104: According to the three hierarchical features P3, P4, and P5 generated in the previous step, further upsampling and merging operations are performed on them, so as to achieve multi-scale feature fusion and capture more detailed and tighter feature associations;

[0102] S105: Multi-level features are passed into the dynamic detection head, and these features are enhanced and integrated through multiple attention mechanisms to further improve the feature expression capability;

[0103] S106: The model finally outputs the predicted student behavior detection results; these predicted results need to be compared with the real labels through the constructed loss function to optimize the model parameters;

[0104] S107: Use the trained parameters to predict the student behavior in the test set classroom, analyze the obtained results, and provide a visual analysis result.

[0105] S101 provided in the embodiment of the present invention specifically includes:

[0106] Dataset preprocessing: including the collection of classroom student behavior datasets and the division of training sets and test sets;

[0107] The collected and divided data sets are rotated and scaled for data enhancement, and the images are made into a consistent size, and finally the processed input image X is obtained. resize =resize(X,(640,640)).

[0108] S102 provided in the embodiment of the present invention specifically includes:

[0109] Feature extraction is performed through the initial convolution layer, and then batch normalization and Silu activation function processing are performed based on the output of the initial convolution layer:

[0110] C1=SiLu(BatchNorm(Conv(X resized )))

[0111] After the shallow feature P3 is generated through the shallow convolution layer and cross-layer connection, the middle-level feature P4 is further extracted through the feature extraction module C2f module:

[0112] P3=C2f(C1)

[0113] P4=C2f(P3).

[0114] S103 provided in the embodiment of the present invention specifically includes:

[0115] The focus modulation mechanism module is designed at this position to extract the deepest features of P5. It first projects the input feature map X into the new feature space through a linear layer to generate the initial feature map Z_0:

[0116] Z0=f z (X)∈R H×W×C

[0117] A set of deep convolutional layer stacks are used to extract contextual information from local to global; the output of each layer is:

[0118]

[0119] Then, a linear layer is used to obtain the spatial and hierarchical gating weights G, and the feature maps at different levels are weighted summed to generate a single feature map Z_out:

[0120]

[0121] Finally, the aggregated context features are processed by a simple modulator function h and multiplied with the query projection function q(xi) to complete feature modulation:

[0122] y i =q(xi)⊙h(Z out )

[0123] S104 provided in the embodiment of the present invention specifically includes:

[0124] Upsample the three-level features P3, P4, and P5:

[0125] P3 upsampled =Upsample(P3)

[0126] P4 upsampled =Upsample(P4)

[0127] Concatenate and fuse the upsampled features with the corresponding hierarchical features:

[0128] P fused =Concat(P3 upsampled ,P4 upsampled ,P5).

[0129] S105 provided in the embodiment of the present invention specifically includes:

[0130] Receive the multi-level feature maps P3, P4, and P5 from the Neck layer;

[0131] Introducing scale-aware attention mechanism π L , feature enhancement based on the importance of different scales:

[0132]

[0133] Where σ(·) is a linear function using 1×1 convolution and hard sigmoid activation function;

[0134] Introduce a spatially aware attention mechanism to focus on features at specific spatial locations:

[0135] π S (F) = Conv(F)

[0136] Finally, we introduce the task-aware attention mechanism to focus on the importance of the output channel:

[0137] π C (F)=FullyConnected(ReLU(Normalize(F)))

[0138] The outputs of the three attention mechanisms are fused to generate an enhanced feature map;

[0139] S106 provided in the embodiment of the present invention specifically includes:

[0140] During the training phase, the corresponding loss is first calculated using the prediction results and the speaker label vector, and then measured by combining the classification loss and the bounding box loss:

[0141] L=λ cls L cls +λ box L box

[0142] First, the simple samples and difficult samples are distinguished according to the IoU between the predicted box and the true box; the mean IoU of all bounding boxes is used as the threshold μ; samples with IoU lower than μ are regarded as negative samples, and samples higher than μ are regarded as positive samples. For simple samples and difficult samples, the sliding weighting function Slide is used to emphasize the boundary samples; the specific Slide weighting function is as follows:

[0143]

[0144] Among them, x represents the IoU between the predicted box and the true box, and μ represents the weighted threshold. In order to enhance the robustness of the parameter μ, the concept of exponential moving average (EMA) is introduced to smooth the threshold μ and reduce the significant fluctuations caused by outliers.

[0145] μ t =β×μ t -1+(1-β)×θ t

[0146] For classification loss, binary cross entropy loss (BCE) is used to calculate the classification loss;

[0147] S107 provided in the embodiment of the present invention specifically includes:

[0148] After the model is trained, it is predicted by inputting the test set data, and the model will output the classification probability and bounding box position. The prediction results and the true labels are combined to calculate the model's precision, recall rate, and mean average precision (mAP), and the results are analyzed to evaluate the performance of the model in practical applications.

[0149] like Figure 3 As shown, a lightweight smart classroom learning behavior detection system provided by an embodiment of the present invention includes:

[0150] Classroom student behavior extraction module: This module is responsible for extracting deep student behavior features from images; through convolution operations and cross-layer connections, it extracts student behavior features in the classroom environment layer by layer, and finally generates a multi-level feature map;

[0151] Context-focused modulation feature extraction enhancement module: This module uses the context-focused modulation mechanism to capture visual context from local to global through deep convolutional layer stacks and gated aggregation methods; it adaptively fuses features at different levels according to the query content and finally generates a deeper feature map;

[0152] Multi-scale feature fusion module: This module upsamples and concatenates feature maps at different levels to achieve multi-scale feature fusion. Through multi-scale feature fusion, more detailed and close feature associations can be captured.

[0153] Multi-attention mechanism fusion module: This module uses a dynamic detection head and multiple attention mechanisms to enhance and process the fused feature map. Specifically, it includes scale-aware attention, space-aware attention, and task-aware attention to further refine and optimize the features.

[0154] Student behavior prediction module: This module uses the feature map output by the dynamic detection head for behavior prediction, calculates the classification probability and bounding box position; measures the difference between the predicted category probability and the true category label through binary cross entropy loss (BCE), and uses exponential moving average loss to measure the difference between the predicted bounding box position and the true bounding box position;

[0155] Result generation and analysis module: This module uses the prediction results to generate and analyze the detection report; distinguish the student behavior in each frame and generate a visual report to help teachers and students understand classroom behavior and improve teaching methods and learning habits.

[0156] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the lightweight smart classroom learning behavior detection method.

[0157] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the lightweight smart classroom learning behavior detection method.

[0158] Another object of the present invention is to provide an information data processing terminal, which is used to implement the lightweight smart classroom learning behavior detection system.

[0159] The present invention is specifically implemented:

[0160] The system principle diagram of the present invention is as follows Figure 2 As shown, the symbols involved in the embodiments of the present invention are shown in Table 1.

[0161] symbol illustrate FocalModulation Focus modulation mechanism <![CDATA[P z ]]> The zth feature layer X Input Image BatchNorm Batch Normalization Conv Convolution Operation <![CDATA[f z ]]> Linear layer projection function <![CDATA[q(x i )]]> Query projection function DW-Conv Depthwise convolution operation <![CDATA[G l ]]> The gating weights associated with the spatial layer, the weighting coefficients of the feature maps of each layer <![CDATA[Z out ]]> The final feature map is a combination of feature maps at all levels. μ Weighted Threshold EMA Exponential Moving Average L Loss Function GELU Gaussian Error Linear Unit activation function SiLu Sigmoid-Weighted Linear Unit activation function μ Weighted Threshold

[0162] S101 provided in the embodiment of the present invention specifically includes:

[0163] (1.1) First, collect image data related to student behavior in the classroom. This data can be obtained in a variety of ways, including but not limited to real-time camera shooting, historical classroom recordings, etc. We selected videos of primary and secondary school classroom environments and captured them as single-frame images. Subsequently, the collected image data is divided into training sets and test sets according to a certain ratio. The training set is used for model training, and the test set is used for model verification and evaluation. Ensure that the data is diverse and representative during the division process to improve the generalization ability of the model.

[0164] (1.2) Data enhancement processing:

[0165] 1. Rotation: Rotate the divided data set to increase the diversity of the data. By randomly rotating images at different angles, we can simulate various postures and perspectives that appear in a real classroom environment.

[0166] 2. Scaling: Scale the image to meet the uniform size requirements. During the scaling process, the aspect ratio of the image is maintained to avoid image distortion. Ultimately, all images are adjusted to a specific n×n standard size.

[0167] Other data augmentation operations such as flipping, translation, and cropping are then performed to further enrich the training data. Through the above detailed dataset preprocessing steps, the training effect and robustness of the model can be significantly improved, ensuring that the model can maintain good performance in different classroom environments.

[0168] (2.1) First, the input image X is subjected to feature extraction through the initial convolutional layer to obtain the convolutional feature map Conv(X resized), and then perform batch normalization on the feature map BatchNorm(Conv(X resized )) to make its mean 0 and variance 1. Then, apply the SiLU activation function SiLU(BatchNorm(Conv(X resized ))) Enhance the linear expression ability and obtain the processed feature map C1. The specific process is as follows:

[0169] C1=SiLU(BatchNorm(Conv(X resized )))

[0170] (2.2) Then, the shallow feature map P3 is generated through the shallow convolution layer and cross-layer connection, that is:

[0171] P3=C2f(C1)

[0172] The C2f module is then used to further extract features from P3 to generate the middle-level feature map P4:

[0173] P4=C2f(P3)

[0174] In this process, the C2f module effectively captures the multi-level features in the input image through multiple convolutions and cross-layer connection operations, enhancing the richness and robustness of the features. Through this layer-by-layer extraction method, the model can capture the multi-scale features of student behavior in the classroom environment, thereby providing a more accurate and comprehensive feature representation for subsequent behavior detection.

[0175] (3.1) The focus modulation module is designed at this position to extract the deepest features of P5. First, the input feature map X is projected to the new feature space through a linear layer to generate the initial feature map Z_0. The process is as follows:

[0176] Z0=f z (X)∈R H×W×C

[0177] (3.2) Next, a set of deep convolutional layer stacks are used to extract contextual information from local to global. The output of each layer is:

[0178]

[0179] Among them, GeLU represents the Gaussian error linear unit activation function, and DW-Conv represents the deep convolution operation. Then, a linear layer is used to obtain the spatial and hierarchical related gating weights G, and the feature maps of different levels are weighted summed to generate a single feature map Z_out:

[0180]

[0181] (3.3) Finally, the aggregated context features are processed by a simple modulator function h and multiplied with the query projection function q(xi) to complete feature modulation:

[0182]

[0183] In the whole process, the focus modulation mechanism module extracts context information from local to global through layer-by-layer convolution and activation operations, and then weights and integrates the features of each layer through the gating mechanism, and finally completes the feature fusion through the modulator function. This series of operations not only reduces the computational complexity, but also enhances the feature expression ability, significantly improving the detection accuracy and efficiency of the model.

[0184] (4.1) Upsample the three-level features P3, P4, and P5:

[0185] P3 upsampled =Upsample(P3)

[0186] P4 upsampled =Upsample(P4)

[0187] (4.2) The upsampled features are concatenated and fused with the corresponding hierarchical features to generate the fused feature map. The specific operations are as follows:

[0188] P fused =Concat(P3 upsampled ,P4 upsampled ,P5)

[0189] In this process, by upsampling and concatenating features at different levels, multi-scale features are fused to capture more detailed and closer feature associations, thereby improving feature richness and representation capabilities.

[0190] (5.1) Receive the multi-level feature maps P3, P4, and P5 passed in by the Neck layer.

[0191] (5.2) Introduce scale-aware attention mechanism to enhance features based on the importance of different scales. The specific operation is as follows:

[0192]

[0193] Here, σ(·) is a linear function using 1×1 convolution and hard sigmoid activation function.

[0194] (5.3) Introduce a spatially aware attention mechanism to focus on features at specific spatial locations:

[0195] π S (F) = Conv(F)

[0196] (5.4) Finally, we introduce the task-aware attention mechanism to focus on the importance of the output channel:

[0197] π C (F)=FullyConnected(ReLU(Normalize(F)))

[0198] In this process, multiple attention mechanisms are used to enhance and process features, further refine and optimize feature expression capabilities, and improve the accuracy and robustness of target detection. Through the above detailed steps, the model can more accurately capture student behavior characteristics, thereby improving detection results.

[0199] (6.1) In the training phase, the corresponding loss is first calculated by the prediction results and the label vector. We use the total loss function combining the classification loss Lcls and the bounding box loss Lbox to measure the prediction performance of the model. The specific expression is as follows:

[0200] L=λ cls L cls +λ box L box

[0201] Among them, the classification loss Lcls uses binary cross entropy loss (BCE) to calculate the difference between the predicted category probability and the true category label. The bounding box loss Lbox uses our optimized exponential moving average loss function to solve the problem of sample imbalance in the classroom student behavior dataset. Specifically, first distinguish simple samples from difficult samples based on the IoU (Intersection over Union) between the predicted box and the true box. The mean IoU of all bounding boxes is used as the threshold μ. Samples with IoU lower than μ are considered negative samples, and samples higher than μ are considered positive samples. For simple and difficult samples, we use the sliding weighting function Slide to enhance boundary samples. The specific Slide weighting function is as follows:

[0202]

[0203] Among them, x represents the IoU between the predicted box and the true box, and μ represents the weighted threshold. In order to enhance the robustness of the parameter μ, we introduce the concept of exponential moving average (EMA) to smooth the threshold μ and reduce the significant fluctuations caused by outliers:

[0204] μ t =β×μ t -1+(1-β)×θ t

[0205] Through the above optimization, the bounding box loss can better handle the problem of sample imbalance and improve the detection effect and robustness of the model. Finally, the classification loss and bounding box loss are combined to optimize the model parameters by minimizing the total loss function L. After the model training is completed, the test set data is input for prediction, and the model will output the classification probability and bounding box position. Combining the prediction results and the true labels, the model's precision, recall rate, and mean average accuracy (mAP) are calculated, and the results are analyzed to evaluate the performance of the model in practical applications. Through the above steps, the model can detect and identify classroom student behavior more accurately and improve the detection effect.

[0206] Figure 4 The embodiments of the present invention provide (a) monotone partial parse tree (SPPF), (b) focus modulation, (c) context aggregation, and (d) self-attention map.

[0207] Figure 5 This is a physical diagram of the improved model provided by an embodiment of the present invention.

[0208] Figure 6 It is a comparison chart of mAP50-95 across training cycles (Epochs) provided by an embodiment of the present invention.

[0209] In the experiment of the present invention, the main development environment includes: Ubuntu 20.04, RTX 4090 24GB, CUDA 11.3, Python 3.8, PyTorch 1.10.0, AMD EPYC 965496-Core Processor. The specific hyperparameter settings of the model are shown in Table 2 below.

[0210] Table 2 Experimental model hyperparameter settings

[0211]

[0212]

[0213] The lightweight smart classroom learning behavior detection system provided by the application embodiment of the present invention is applied to a computer device, and the computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the program, the processor is enabled to execute each step of the lightweight smart classroom learning behavior detection system.

[0214] Another object of the present invention is to apply the lightweight smart classroom learning behavior detection method to an information data processing terminal. The information data processing terminal is used to implement the lightweight smart classroom learning behavior detection system.

[0215] To comprehensively evaluate model performance, we use several key metrics, including mean average accuracy (mAP), precision, recall, and floating point operations (FLOPs). These metrics provide detailed insights into the effectiveness and efficiency of the model. First, precision measures the accuracy of the model's predictions for positive samples, which is defined as follows:

[0216]

[0217] Among them, TP (True Positive) represents the number of correctly detected target behavior instances, and FP (False Positive) represents the number of incorrectly detected instances, that is, the number of instances that are predicted to be target behaviors but are not actually.

[0218] Recall measures the ability of a model to detect all relevant instances and is defined as follows:

[0219]

[0220] Among them, FN (False Negative) represents the number of instances not detected by the model, that is, the number of target behavior instances that actually exist but the model fails to recognize. A high recall rate indicates the effectiveness of the model in finding all relevant instances, which is particularly important in applications where positive samples cannot be missed.

[0221] In order to evaluate the effectiveness of the lightweight smart classroom learning behavior detection method proposed in this paper, this paper conducted a number of experiments to compare the performance of different models under various evaluation indicators. The experimental results show that the present invention performs superiorly in identifying classroom student behaviors, with precision and recall rates reaching 0.804 and 0.763, respectively. In addition, SBD-Net achieved 0.824 and 0.619 in the mAP@0.5 and mAP@0.5-0.95 indicators, respectively, demonstrating its effectiveness in accurately detecting and classifying student behaviors. In order to more comprehensively evaluate the performance of SBD-Net, we compared it with several benchmark models, including YOLOv5, YOLOv7, YOLOv8, SSD, Faster-RCNN, and Deformable-DETR. From Tables 3 and Figure 6 It can be seen from the data that the present invention outperforms these benchmark models in multiple indicators. Specifically, the number of parameters of the present invention is 36.5M, which strikes a balance between complexity and computational efficiency and is suitable for deployment in resource-constrained environments, such as classroom cameras. In addition, the modified exponential moving average loss function improves the robustness and accuracy of the model in processing unbalanced data sets, especially in the data imbalance problem that is common in actual classroom scenarios. The specific effects are shown in Table 3 below:

[0222]

[0223] In order to verify the contribution of each module to the model performance, we also conducted an ablation experiment. The experimental results show that after adding the focus modulation mechanism module, the mAP@0.5 and mAP@0.5-0.95 of the present invention increased by 1.8% and 0.8% respectively, and after adding the dynamic detection head, mAP@0.5 further increased by 2.7%, and mAP@0.5-0.95 increased by 0.4%. Finally, after using the optimized exponential moving average classification loss function, the accuracy of the model increased by 0.5%, the recall rate increased by 3.8%, and the mAP@0.5 and mAP@0.5-0.95 increased by 4.3% and 2.4% respectively.

[0224]

[0225] In summary, the present invention performs well in many indicators, especially in dealing with complex student behavior detection tasks, demonstrating its excellent performance and computational efficiency. The invention not only has advantages in precision and recall, but also performs well in the utilization of computing resources, and is suitable for real-time application in classroom environments.

[0226] Next, the classroom behavior detection results generated by the present invention are used to analyze the classroom interaction of the class. By interpreting the data, we can deeply understand the classroom teaching effect and provide a scientific basis for teaching improvement.

[0227] (1) First, count the frequency of each student behavior. These behaviors include reading, writing, looking up, raising hands, turning heads, standing, and discussing. By calculating the distribution ratio of each behavior throughout the class time, we can understand the main activity patterns of students in class. The high frequency of reading and writing indicates that students are mainly focused on personal learning and recording. If the ratio of reading and writing is too high, it means that there is a lack of interaction in the class. The frequency of raising hands and discussing reflects the enthusiasm of students to participate in class interaction. If the ratio of such behaviors is low, teachers need to increase interactive sessions and encourage students to participate more in discussions and ask questions. Looking up and turning heads can indicate the degree of students' attention to the teacher's explanation. Frequent looking up means that students are paying attention, while frequent turning heads means that the class is distracted or students are communicating.

[0228] (2) By drawing a time series graph of behavior frequency, we can observe the changing trends of student behavior in different time periods of the class. For example, at the beginning and end of the class, students are more focused, while in the middle, there is a trend of declining attention. By analyzing these trends, teachers can arrange class content reasonably, such as explaining important knowledge during the peak period of attention and arranging interaction or rest time during the trough period of attention.

[0229] (3) Analyzing the relationship between teachers’ behaviors and students’ responses in class can help us understand the effectiveness of different teaching methods. Teachers’ behaviors include explaining, asking questions, and interacting. By correlating these behaviors with students’ response behaviors (such as raising hands, discussing, etc.), we can identify effective teaching strategies.

[0230] Based on the above analysis results, the overall effect of the class is evaluated and an evaluation report is generated.

[0231] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0232] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A lightweight smart classroom learning behavior detection method and system, characterized in that: This lightweight smart classroom learning behavior detection method detects student behavior in the classroom through multi-step image processing and feature extraction. First, multi-scale feature extraction is performed on the classroom environment image, and features are extracted and fused layer by layer through convolution operations and focus modulation mechanisms. Then, multiple attention mechanisms are used to enhance feature expression, and finally the behavior detection results are output, and the model is optimized through loss functions. The method also includes the entire process of data preprocessing, feature extraction, feature fusion, and model training and testing to improve the accuracy and adaptability of student behavior detection.

2. The lightweight smart classroom learning behavior detection method according to claim 1 is characterized in that: The following steps are involved: Step 1: Extract frame-level features from classroom environment images; Input the classroom environment image and process it, scale it to a specific size of n×n, extract multi-scale features in the image as a feature vector representation of student behavior; Step 2: Extract local features through convolution operation, then improve linear expression ability through batch normalization and Silu activation function, and combine cross-layer connection to extract student behavior characteristics in the classroom environment in layers to improve feature fusion and expression capabilities. In this step, shallow features P3 and middle features P4 are generated; Step 3: Introduce a focus modulation mechanism to replace the traditional self-attention mechanism, thereby further improving the model's feature extraction capability; its specific role is to receive the processed value passed in step 2, use a deep convolutional layer stack to capture the visual context from local to global, and then use a gating mechanism to adaptively aggregate features at different levels according to the query content, and finally use a simple modulator to fuse the aggregated context features into the query, and finally generate a deeper feature map P5; Step 4: According to the three hierarchical features P3, P4, and P5 generated in the previous step, further upsample and merge them to achieve multi-scale feature fusion and capture more detailed and close feature associations; Step 5: Pass the multi-level features into the dynamic detection head, enhance and fuse these features through various attention mechanisms to further improve the feature expression ability; Step 6: The model finally outputs the predicted student behavior detection results; these predicted results need to be compared with the real labels through the constructed loss function to optimize the model parameters; Step 7: Use the trained parameters to predict the student behavior in the test set classroom and analyze the results, and provide a visual analysis result.

3. The lightweight smart classroom learning behavior detection method according to claim 1 is characterized in that: The step 1 specifically includes: Dataset preprocessing: including the collection of classroom student behavior datasets and the division of training sets and test sets; The collected and divided data sets are rotated and scaled for data enhancement, and the images are made into a consistent size, and finally the processed input image X is obtained. resize =resize(X,(640,640)).

4. The lightweight smart classroom learning behavior detection method according to claim 2 is characterized in that: The step 2 specifically includes: Feature extraction is performed through the initial convolution layer, and then batch normalization and Silu activation function processing are performed based on the output of the initial convolution layer: C1=SiLu(BatchNorm(Conv(X resized ))) After the shallow feature P3 is generated through the shallow convolution layer and cross-layer connection, the middle-level feature P4 is further extracted through the feature extraction module C2f module: P3=C2f(C1) P4=C2f(P3).

5. The lightweight smart classroom learning behavior detection method according to claim 2 is characterized in that: The step three specifically includes: The focus modulation mechanism module is designed at this position to extract the deepest features of P5. It first projects the input feature map X into the new feature space through a linear layer to generate the initial feature map Z_0: Z0=f z (X)∈R H×W×C A set of deep convolutional layer stacks are used to extract contextual information from local to global; the output of each layer is: Then, a linear layer is used to obtain the spatial and hierarchical gating weights G, and the feature maps at different levels are weighted summed to generate a single feature map Z_out: Finally, the aggregated context features are processed by a simple modulator function h and multiplied with the query projection function q(xi) to complete feature modulation: y i =q(xi)⊙h(Z out )。 6. The lightweight smart classroom learning behavior detection method according to claim 2 is characterized in that: The step 4 specifically includes: Upsample the three-level features P3, P4, and P5: P3 upsampled =Upsample(P3) P4 upsampled =Upsample(P4) Concatenate and fuse the upsampled features with the corresponding hierarchical features: P fused =Concat(P3 upsampled ,P4 upsampled ,P5)。 7. The lightweight smart classroom learning behavior detection method according to claim 2 is characterized in that: The step five specifically includes: Receive the multi-level feature maps P3, P4, and P5 from the Neck layer; Introducing scale-aware attention mechanism π L , feature enhancement based on the importance of different scales: Where σ(·) is a linear function using 1×1 convolution and hard sigmoid activation function; Introduce a spatially aware attention mechanism to focus on features at specific spatial locations: π S (F)=Conv(F) Finally, we introduce the task-aware attention mechanism to focus on the importance of the output channel: π C (F)=FullyConnected(ReLU(Normalize(F))) The outputs of the three attention mechanisms are fused to generate an enhanced feature map; The step six specifically includes: During the training phase, the corresponding loss is first calculated using the prediction results and the speaker label vector, and then measured by combining the classification loss and the bounding box loss: L=λ cls L cls +λ box L box First, the simple samples and difficult samples are distinguished according to the IoU between the predicted box and the true box; the mean IoU of all bounding boxes is used as the threshold μ; samples with IoU lower than μ are regarded as negative samples, and samples higher than μ are regarded as positive samples. For simple samples and difficult samples, the sliding weighting function Slide is used to emphasize the boundary samples; the specific Slide weighting function is as follows: Among them, x represents the IoU between the predicted box and the true box, and μ represents the weighted threshold. In order to enhance the robustness of the parameter μ, the concept of exponential moving average (EMA) is introduced to smooth the threshold μ and reduce the significant fluctuations caused by outliers. m t =β×μ t -1+(1-β)×θ t For classification loss, binary cross entropy loss (BCE) is used to calculate the classification loss; The step seven specifically includes: After the model is trained, it is predicted by inputting the test set data, and the model will output the classification probability and bounding box position. The prediction results and the true labels are combined to calculate the model's precision, recall rate, and mean average precision (mAP), and the results are analyzed to evaluate the performance of the model in practical applications.

8. A lightweight smart classroom learning behavior detection system implementing the lightweight smart classroom learning behavior detection method as described in any one of claims 1 to 7, characterized in that: The lightweight smart classroom learning behavior detection system includes: Classroom student behavior extraction module: This module is responsible for extracting deep student behavior features from images; through convolution operations and cross-layer connections, it extracts student behavior features in the classroom environment layer by layer, and finally generates a multi-level feature map; Context-focused modulation feature extraction enhancement module: This module uses the context-focused modulation mechanism to capture visual context from local to global through deep convolutional layer stacks and gated aggregation methods; it adaptively fuses features at different levels according to the query content and finally generates a deeper feature map; Multi-scale feature fusion module: This module upsamples and concatenates feature maps at different levels to achieve multi-scale feature fusion. Through multi-scale feature fusion, more detailed and close feature associations can be captured. Multi-attention mechanism fusion module: This module uses a dynamic detection head and multiple attention mechanisms to enhance and process the fused feature map. Specifically, it includes scale-aware attention, space-aware attention, and task-aware attention to further refine and optimize the features. Student behavior prediction module: This module uses the feature map output by the dynamic detection head for behavior prediction, calculates the classification probability and bounding box position; measures the difference between the predicted category probability and the true category label through binary cross entropy loss (BCE), and uses exponential moving average loss to measure the difference between the predicted bounding box position and the true bounding box position; Result generation and analysis module: This module uses the prediction results to generate and analyze the detection report; distinguish the student behavior in each frame and generate a visual report to help teachers and students understand classroom behavior and improve teaching methods and learning habits.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the lightweight smart classroom learning behavior detection method as described in any one of claims 1-7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the lightweight smart classroom learning behavior detection system as described in claim 8.

Images