Real-time classroom behavior detection method based on edge information and decoupling shared detection head

Abstract

The invention provides a real-time classroom behavior detection method based on edge information and a decoupling shared detection head. On one hand, an efficient edge information convolution module is provided to extract edge information of classroom body behaviors, the performance of model detection is better improved, a global-to-local Transform module is added to enhance the feature extraction capability, and then context anchor attention is utilized to make up for the deficiency of global information perception when only dependence on local multi-scale convolution is performed, so that the accuracy of model detection is improved. And on the other hand, a decoupling sharing detection head with light weight and efficiency merging is adopted, all convolution layers share the same weight, and each corresponding normalization layer carries out independent calculation, so that numerical value normalization among different level features is ensured, cross contamination of statistical errors of batch normalization layers is also avoided, and the detection accuracy is improved. And meanwhile, additional group normalization calculation overhead is not introduced. According to the technical scheme, the requirements of high real-time performance and easy deployment are met, and an innovative method is provided for solving the problem.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and computer vision, relates to the technical field of classroom behavior detection, and in particular to deep learning, real-time target detection and the like, and especially to a real-time classroom behavior detection method based on edge information and decoupled shared detection heads. BACKGROUND

[0002] With the rapid development of economy, the requirements for the field of education and teaching are also increasing, and more and more people pay more and more attention to the importance of classroom teaching, therefore, it is particularly important to pay attention to the reaction and behavior change of students in the classroom. In the early years, due to the limitation of technology, it can only be judged by observing with the naked eye, in recent years, with the development of deep learning and target detection technology, it has become possible to analyze the behavior of students in the classroom video and obtain the information of the state and learning performance of students. This technology is very important for teachers, management personnel, students and parents of the school. However, in the traditional teaching mode, it is difficult for teachers to pay attention to the learning situation of each student, and they can only observe a few students to understand the effectiveness of their own teaching methods. School administrators rely on on-site observation and student performance reports to find problems in education and teaching. Parents can only understand the learning situation of their children through communication with teachers and students. Since the traditional manual observation and analysis method is time-consuming and laborious, it is challenging to evaluate a large amount of video. Therefore, the demand and interest in using artificial intelligence technology to analyze the behavior of students in the classroom video to evaluate the effectiveness of teaching has increased significantly. Therefore, using behavior detection technology to accurately detect the behavior of students and analyze the learning situation and performance of students can provide more comprehensive and accurate feedback for education and teaching.

[0003] The process of classroom behavior recognition includes feature extraction, target detection and behavior classification. Feature extraction includes traditional manual methods and deep learning methods. On the basis of traditional manual feature extraction, some researchers apply multi-class fuzzy support matrix machine (MCFSMM) to classroom behavior recognition. For example, a fuzzy generalized learning system (BLS) is proposed, which combines MCFSMM with BLS to improve the performance of the classifier. The manually crafted features are reduced in dimension and input into the fuzzy BLS for behavior classification. Compared with classic classifiers such as support vector machine (SVM) and decision tree, the fuzzy BLS achieves better results. However, traditional machine learning methods such as MCFSMM and SVM usually rely on pre-defined or manually crafted features. These methods may not have the ability to capture complex patterns and high-level representations as effectively as deep learning models. In contrast, deep learning models provide the advantage of autonomous learning and direct extraction of relevant features from input data. This feature has been proven to be advantageous when dealing with diverse classroom behaviors.

[0004] In recent years, advancements in deep learning theory have greatly promoted the development of classroom behavior recognition. Human pose estimation methods utilize convolutional neural networks (CNNs) to learn features from images or videos to estimate pose. Therefore, researchers have proposed a Large Motion Action Recognition (GM-AR) system, which uses the OpenPose framework and sensors to identify 13 typical large-amplitude movements of students. While pose estimation methods show promise for behavior recognition, the challenge of detecting the complex details of human key points in a classroom environment remains. To address this challenge, researchers have utilized CNN-based object detection models to directly identify and classify student behaviors, thus proposing an enhanced, faster region-CNN (R-CNN) model for classroom behavior recognition. This model is capable of directly detecting specific behaviors such as raising a hand, standing, and sleeping. Although the above methods have achieved some success, some issues still warrant consideration in classroom behavior recognition tasks. Firstly, in classroom behavior images, occlusion and inconsistent object scales lead to fragmented visual features, hindering the detection network's accurate identification of occluded objects. Secondly, previous research lacked optimization for small target recognition. Students in classroom corners have relatively small pixel sizes in images, making their visual features less prominent. Downsampling operations in deep neural networks may risk losing features. Simultaneously, there is a need for high real-time performance and ease of deployment. To address the challenges of data storage for recorded classroom videos, accurate and real-time student identification and analysis are essential. Therefore, a lightweight, hardware-friendly method for detecting student classroom behavior targets is needed. Summary of the Invention

[0005] The purpose of this invention is to address the slow detection speed of existing methods due to the large number of model parameters and computational load. To overcome these shortcomings, a real-time classroom behavior detection method based on edge information and decoupled shared detection heads is proposed. Experimental results demonstrate that the method of this invention is superior to existing technologies and meets the current needs of real-time classroom behavior detection.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] 1) Acquire classroom monitoring videos covering different grade levels (including primary and secondary school students) to capture students' real-life behavior in the classroom. Process the acquired videos frame by frame to extract static images of student classroom behavior, which will serve as the basis for subsequent modeling.

[0008] 2) Using image annotation tools (such as labelImg), each extracted image was manually labeled with its behavioral category to construct a student classroom behavior dataset. This dataset contains three main behavioral categories: "reading," "hand-raising," and "writing." In actual experiments, it was observed that "reading" and "writing" behaviors have high visual similarity, often leading to overlapping bounding boxes, which poses a challenge to the model's differentiation.

[0009] 3) The labeled data is cleaned and organized, and then divided into training and testing sets. 80% of the images and corresponding label files are used to train the model, and the remaining 20% ​​is used to evaluate model performance.

[0010] 4) Model Construction. Referring to the YOLOv8 backbone network, this model has the following main improvements: It proposes an efficient Edge Information Convolution Module (EICM) to extract edge information of classroom body behavior, thus improving the model's detection performance; it incorporates GLTB (Global-to-Local Transformer Block) to enhance feature extraction capabilities; it utilizes CAA (Context Anchor Attention) to compensate for the lack of global information perception when relying solely on local multi-scale convolutions; and it employs a lightweight and efficient Decoupled Shared Detection Head (DSDH) in the final prediction head. The backbone network is the foundation of the model, responsible for extracting key local and global features from the input image. After obtaining the initial features, the neck structure further processes these features to enrich the multi-scale feature representation. The neck fuses information from different scales to generate more detailed feature maps containing contextual information. Finally, the features extracted and fused by the backbone and neck are passed to the head structure, which is responsible for specific prediction tasks, including classification and regression tasks.

[0011] 5) Apply various enhancement operations to the original training images, including mosaic enhancement, mix-up enhancement, random geometric perturbation, and color space perturbation. These diverse enhancement strategies can significantly improve the model's generalization ability and robustness to noise in different scenarios.

[0012] 6) Input the processed training set images and their labels into the constructed detection model, set fixed hyperparameters, and select an appropriate optimizer and loss function for iterative training. The entire training process incorporates AMP (Automatic Mixed Precision) technology, which, while ensuring model accuracy, uses a mixture of various floating-point precision methods for computation, thereby significantly improving training speed and optimizing the utilization of GPU memory and computing resources.

[0013] 7) Input the image and label data from the test set into the trained model to classify and locate student classroom behavior in each image, obtaining the corresponding predicted labels and location information. Then, compare the prediction results with the actual labels to obtain the detection results, and calculate metrics such as mean precision (mAP), number of parameters (Params), and gigaflops per second (GFLOPs).

[0014] 8) Input the test image into the trained model, output the detection results of student classroom behavior in the test image, and highlight the detection results with colored boxes and symbols.

[0015] Specifically, step 1) includes:

[0016] Images are taken from multiple perspectives, including front, side, and back views. These angular differences significantly alter the visual characteristics of the hand-raising gesture, making detection more challenging. Further, step 2) includes:

[0017] Specifically, labelImg is an open-source image annotation tool that allows users to manually select areas of students' classroom behavior in each image and assign corresponding labels to each area, such as "reading", "hand-raising", and "writing".

[0018] After annotation is complete, labelImg will automatically generate a label information file corresponding to each image and save this label information in text format (txt file). The name of each txt file is consistent with the image file for easy reading and processing later.

[0019] In this way, all images of student classroom behavior are systematically labeled, ultimately forming a complete dataset of student classroom behavior.

[0020] Further, step 4) includes:

[0021] This deep learning network for detecting student classroom behavior mainly consists of three parts: a feature extraction module, a feature fusion module, and a detection head module. Specifically, it enhances feature extraction capabilities by combining an edge information convolutional module (EICM) with GLTB, improves efficiency while reducing the number of parameters and computational cost using CAA, and significantly reduces the number of parameters and computational cost in the final prediction head by employing a decoupled shared detection head (DSDH).

[0022] In the feature extraction module, the original YOLOv8 backbone network was modified to improve detection speed and recognition accuracy while reducing the number of parameters and computational overhead. Given the limitations of the original network in global feature extraction and its large number of parameters, an EICM module was introduced in the feature fusion module to specifically capture edge information of the image. Considering that traditional convolutional neural networks (CNNs) are good at learning spatial structure but not at extracting edge details, EICM explicitly uses the classic Sobel filter operator through the SobelConv branch to efficiently detect gray-level abrupt changes in the image, thereby obtaining key edge features.

[0023] Meanwhile, to take into account the spatial semantics of the image, EICM also sets up a regular convolutional branch (conv_branch) to directly extract rich spatial details from the original image. Finally, the edge features extracted by SobelConv are fused with the spatial features of conv_branch to form a composite representation that is both rich in edge information and takes into account the spatial layout, thus providing a more comprehensive description of the image content.

[0024] Simultaneously, a more advanced Transformer-based GLTB module was introduced, replacing the original CNN feature extraction unit after EICM in the backbone network. Specifically, GLTB is embedded in the backbone network to enhance the ability to capture global and local information. Considering the high computational cost of the Transformer, applying it to all channels would result in excessive costs. Therefore, the input feature map is first split into two parts along the channel dimension, and then processed by the CNN and Transformer branches respectively. This reduces the computational burden while maintaining efficient feature extraction.

[0025] Finally, CAA is used to capture the contextual relationships between distant pixels by combining global average pooling with one-dimensional "strip" convolution, and to enhance feature representation in concentrated regions, thereby compensating for the lack of global information perception when relying solely on local multi-scale convolution. Compared with previous methods that rely on large convolutional kernels or dilated convolutions to expand the receptive field, CAA effectively models long-range dependencies while maintaining computational efficiency, helping the feature extraction module achieve significant performance improvements on object detection benchmarks.

[0026] In DSDH, a series of lightweight and efficient designs significantly improve the overall performance of the detection head in localization and classification tasks. First, the traditional BatchNorm is replaced with GroupNorm, so that the normalization operation no longer relies on large batches of data, but instead groups the channel dimensions according to a pre-defined number of groups. This not only alleviates the instability of normalization during small-batch training, but also further improves the accuracy of the detection head in regressing fine-grained targets and bounding boxes in multi-scale feature fusion and high-resolution input scenarios.

[0027] To significantly reduce the number of model parameters and computational overhead, a shared convolutional structure is employed across all detection heads. All scale branches share the same set of convolutional kernels. This weighted reuse strategy allows the model to maintain its expressive power while drastically reducing the overall number of parameters, greatly improving deployment feasibility in resource-constrained environments such as mobile or embedded devices. However, the response requirements of detection heads at different scales are not entirely the same: low-level features primarily focus on the detailed texture of small targets, while mid-to-high-level features are more inclined towards the overall semantic information of large targets. To address the decreased scale sensitivity caused by shared convolution, a learnable scale layer is added after the convolutional output of each detection head. This layer dynamically adjusts the feature amplitude of different scale branches by scaling the channel features one by one, making them better suited to the size distribution of the targets they are responsible for.

[0028] Regarding normalization, features at different levels have different statistical distributions; for example, lower-level features are often sensitive to details, while higher-level features encode more semantic information. Without a normalization layer, the network struggles to synchronize the numerical scale of features across different layers during training, potentially leading to unstable gradients or slow convergence. Therefore, regardless of the network design, a normalization layer remains indispensable.

[0029] In object detection, it is generally desirable for detection heads to share parameters to reduce model size and accelerate training and inference. However, if batch normalization (BN) is introduced directly into the detection head with shared parameters, a moving average cumulative error will occur: the difference in feature distribution in different input batches will cause the same set of BN parameters to perform inconsistently during training and inference, thus affecting detection accuracy.

[0030] In contrast, layer normalization or group normalization (GN) does not rely on statistics of the batch dimension, which theoretically avoids the above problems. However, GN still needs to perform group calculations within each sample during the inference stage, which significantly increases the computational overhead, especially in real-time detection scenarios.

[0031] To balance accuracy and efficiency, all convolutional layers in the detection head share the same weights—thus keeping the model parameter scale constant; while each corresponding normalization layer (BN) independently calculates and maintains its moving average statistics. This ensures numerical normalization between features at different levels, avoids cross-contamination of BN statistical errors, and does not introduce additional GN computation overhead. The entire structure is lightweight and robust, significantly improving the performance and convergence of the detection network in multi-scale feature fusion.

[0032] Further, step 5) includes:

[0033] The main steps of data augmentation include:

[0034] 1. Randomly read four images from the dataset each time.

[0035] 2. Perform four processing steps on the original image, including horizontal flipping, resizing, and adjusting color gamut parameters such as brightness and saturation. After completing these operations, arrange the four processed images in order: the first image in the top left corner, the second in the bottom left corner, the third in the bottom right corner, and the fourth in the top right corner.

[0036] 3. After processing and layouting the four images, the corresponding fixed regions are matrix-wise cropped, and these regions are stitched together to form a new image containing visual elements such as borders. This method borrows from and improves upon the CutMix data augmentation strategy. Unlike CutMix, which uses two images for stitching, this method uses four images for fusion. This not only enhances the diversity of backgrounds in the object detection task but also allows for the simultaneous processing of more image information during batch normalization (BN) calculation, thereby improving the model's generalization ability. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0039] Figure 2 This is a schematic diagram of the model framework of the present invention.

[0040] Figure 3 This is a performance comparison chart of the present invention and the benchmark model method on a real-time student classroom behavior detection dataset. Detailed Implementation

[0041] The purpose of this invention is to address the slow detection speed of existing methods due to the large number of model parameters and computational load. To overcome these shortcomings, a real-time classroom behavior detection method based on edge information and decoupled shared detection heads is proposed. Experimental results demonstrate that the method of this invention is superior to existing technologies and meets the current needs of real-time classroom behavior detection.

[0042] To achieve the above objectives, the technical solution adopted by the present invention is as follows, and the main process of the method is as follows: Figure 1 As shown:

[0043] 1) Acquire classroom monitoring videos covering different grade levels (including primary and secondary school students) to capture students' real-life behavior in the classroom. Process the acquired videos frame by frame to extract static images of student classroom behavior, which will serve as the basis for subsequent modeling.

[0044] 2) Using image annotation tools (such as labelImg), each extracted image was manually labeled with its behavioral category to construct a student classroom behavior dataset. This dataset contains three main behavioral categories: "reading," "hand-raising," and "writing." In actual experiments, it was observed that "reading" and "writing" behaviors have high visual similarity, often leading to overlapping bounding boxes, which poses a challenge to the model's differentiation.

[0045] 3) The labeled data is cleaned and organized, and then divided into training and testing sets. 80% of the images and corresponding label files are used to train the model, and the remaining 20% ​​is used to evaluate model performance.

[0046] 4) Model Construction. Referring to the YOLOv8 backbone network, this model has the following main improvements: It proposes an efficient Edge Information Convolution Module (EICM) to extract edge information of classroom body behavior, thus improving the model's detection performance; it incorporates GLTB (Global-to-Local Transformer Block) to enhance feature extraction capabilities; it utilizes CAA (Context Anchor Attention) to compensate for the lack of global information perception when relying solely on local multi-scale convolutions; and it employs a lightweight and efficient Decoupled Shared Detection Head (DSDH) in the final prediction head. The backbone network is the foundation of the model, responsible for extracting key local and global features from the input image. After obtaining the initial features, the neck structure further processes these features to enrich the multi-scale feature representation. The neck fuses information from different scales to generate more detailed feature maps containing contextual information. Finally, the features extracted and fused by the backbone and neck are passed to the head structure, which is responsible for specific prediction tasks, including classification and regression tasks.

[0047] 5) Apply various enhancement operations to the original training images, including mosaic enhancement, mix-up enhancement, random geometric perturbation, and color space perturbation. These diverse enhancement strategies can significantly improve the model's generalization ability and robustness to noise in different scenarios.

[0048] 6) Input the processed training set images and their labels into the constructed detection model, set fixed hyperparameters, and select an appropriate optimizer and loss function for iterative training. The entire training process incorporates AMP (Automatic Mixed Precision) technology, which, while ensuring model accuracy, uses a mixture of various floating-point precision methods for computation, thereby significantly improving training speed and optimizing the utilization of GPU memory and computing resources.

[0049] 7) Input the image and label data from the test set into the trained model to classify and locate student classroom behavior in each image, obtaining the corresponding predicted labels and location information. Then, compare the prediction results with the actual labels to obtain the detection results, and calculate metrics such as mean precision (mAP), number of parameters (Params), and gigaflops per second (GFLOPs).

[0050] 8) Input the test image into the trained model, output the detection results of student classroom behavior in the test image, and highlight the detection results with colored boxes and symbols.

[0051] Specifically, step 1) includes:

[0052] Images are taken from multiple perspectives, including front, side, and back views. These angular differences significantly alter the visual characteristics of the hand-raising gesture, making detection more challenging. Further, step 2) includes:

[0053] Specifically, labelImg is an open-source image annotation tool that allows users to manually select areas of students' classroom behavior in each image and assign corresponding labels to each area, such as "reading", "hand-raising", and "writing".

[0054] After annotation is complete, labelImg will automatically generate a label information file corresponding to each image and save this label information in text format (txt file). The name of each txt file is consistent with the image file for easy reading and processing later.

[0055] In this way, all images of student classroom behavior are systematically labeled, ultimately forming a complete dataset of student classroom behavior.

[0056] Further, step 4) includes:

[0057] This deep learning network for detecting student classroom behavior mainly consists of three parts: a feature extraction module, a feature fusion module, and a detection head module. Specifically, it enhances feature extraction capabilities by combining an edge information convolutional module (EICM) with GLTB, improves efficiency while reducing the number of parameters and computational cost using CAA, and significantly reduces the number of parameters and computational cost in the final prediction head by employing a decoupled shared detection head (DSDH). Figure 2 As shown.

[0058] In the feature extraction module, the original YOLOv8 backbone network was modified to improve detection speed and recognition accuracy while reducing the number of parameters and computational overhead. Given the limitations of the original network in global feature extraction and its large number of parameters, an EICM module was introduced in the feature fusion module to specifically capture edge information of the image. Considering that traditional convolutional neural networks (CNNs) are good at learning spatial structure but not at extracting edge details, EICM explicitly uses the classic Sobel filter operator through the SobelConv branch to efficiently detect gray-level abrupt changes in the image, thereby obtaining key edge features.

[0059] The basic principle of the Sobel filter: Assuming the function of a continuous image is f(x,y), the gradient of the function at (x,y) is a gradient vector with direction and magnitude, that is:

[0060]

[0061] In equation (1), i and j are the unit vectors in the x and y directions, respectively. Gradient-based operators are all based on this, and most gradient operators define the magnitude and direction of the marginal gradient at (x, y) as follows:

[0062]

[0063] The calculated grad[f(x,y)] value represents the edge data of the image. The above describes the processing of continuous image functions. However, in practical applications, the acquired image data is discrete data of a certain pixel size. Therefore, in image processing, the difference between adjacent or spaced pixels is often used to represent the edge information of the image.

[0064] The Sobel operator performs weighted calculations on discrete data, using a small convolution template to sum the template and the corresponding image data with varying degrees of weight. The two directional templates are the horizontal and vertical image edge detection templates, respectively.

[0065] The basic idea of ​​the Sobel operator is that in the edge regions of an image, the brightness changes of pixels are more significant. Therefore, pixels whose gray values ​​are thresholded are considered edge points. The specific steps are as follows:

[0066] (1) The two directional templates are arranged from left to right and from top to bottom, along each pixel in the image, and the corresponding pixel corresponds to the center point of the template.

[0067] (2) Perform a convolution operation on the weights of each position in the two templates and their corresponding image pixel values.

[0068] (3) Assign the maximum value of the two convolutions to the pixel value at the center of the image of the corresponding template, as the new gray value of that pixel.

[0069] The Sobel edge detection operator is often used as a function to find the maximum value of the data. The two convolution values ​​are respectively understood as the absolute values ​​of the differences between adjacent column and row image data. Although this method can identify a large difference value in the image as a boundary point by directly performing superposition calculation in areas where noise exists, it does not perform a process of filtering and judging noise or data.

[0070] Meanwhile, to take into account the spatial semantics of the image, EICM also sets up a regular convolutional branch (conv_branch) to directly extract rich spatial details from the original image. Finally, the edge features extracted by SobelConv are fused with the spatial features of conv_branch to form a composite representation that is both rich in edge information and takes into account the spatial layout, thus providing a more comprehensive description of the image content.

[0071] Simultaneously, a more advanced Transformer-based GLTB module was introduced, replacing the original CNN feature extraction unit after EICM in the backbone network. Specifically, GLTB is embedded in the backbone network to enhance the ability to capture global and local information. Considering the high computational cost of the Transformer, applying it to all channels would result in excessive costs. Therefore, the input feature map is first split into two parts along the channel dimension, and then processed by the CNN and Transformer branches respectively. This reduces the computational burden while maintaining efficient feature extraction.

[0072] Finally, CAA is used to capture the contextual relationships between distant pixels by combining global average pooling with one-dimensional "strip" convolution, and to enhance feature representation in concentrated regions, thereby compensating for the lack of global information perception when relying solely on local multi-scale convolution. Compared with previous methods that rely on large convolutional kernels or dilated convolutions to expand the receptive field, CAA effectively models long-range dependencies while maintaining computational efficiency, helping the feature extraction module achieve significant performance improvements on object detection benchmarks.

[0073] The goal of the CAA module is to capture contextual dependencies between distant pixels while enhancing the representation of central features. It employs average pooling and 1×1 convolutions to obtain local region features.

[0074]

[0075] Where P avg This represents the average pooling operation. For n = 0, we have Then, we apply two depthwise strip convolutions as approximations of the standard large-kernel depthwise convolution:

[0076]

[0077] The choice of depthwise strip convolution was based on two main considerations. First, because strip convolution is lightweight.

[0078] Compared with traditional k b ×k bCompared to 2D depthwise convolution, we can achieve a similar effect using several 1D depthwise convolution kernels, reducing the number of parameters by k. b / 2. Secondly, strip convolution can facilitate feature recognition and extraction of slender objects (such as bridges). To increase the receptive field of the CAA module as it progresses through the PK1 block to which it belongs, we set k... b =11+2×l, which means calculating the kernel size k b The depth n is a function of the feature extraction module. This design enhances the feature extraction module's ability to establish relationships between long-distance pixels, and does not significantly increase computational cost due to the strip depth design.

[0079] Finally, the CAA module generates an attention weight. Further used to improve the output of the PKI module:

[0080]

[0081] Here, the Sigmoid function ensures attention to Figure A. l-1,n Within the range (0,1), ⊙ represents element-wise multiplication, and ⊕ represents element-wise summation. This is an enhancement feature. The output is obtained in the following way:

[0082]

[0083] The output of the last module is represented as

[0084] In DSDH, a series of lightweight and efficient designs significantly improve the overall performance of the detector head in localization and classification tasks. First, the traditional BatchNorm is replaced with GroupNorm, so that the normalization operation no longer relies on large batches of data, but instead groups the channel dimensions according to a pre-defined number of groups. Since directly introducing BatchN into the detector head with shared parameters can lead to errors in its moving average, while introducing GroupNorm increases inference overhead, the detector heads share convolutional layers, while BatchN is calculated independently. This not only alleviates the instability of normalization during small-batch training, but also further improves the accuracy of the detector head in regressing fine-grained targets and bounding boxes in multi-scale feature fusion and high-resolution input scenarios.

[0085] To significantly reduce the number of model parameters and computational overhead, a shared convolutional structure is employed across all detection heads. All scale branches share the same set of convolutional kernels. This weighted reuse strategy allows the model to maintain its expressive power while drastically reducing the overall number of parameters, greatly improving deployment feasibility in resource-constrained environments such as mobile or embedded devices. However, the response requirements of detection heads at different scales are not entirely the same: low-level features primarily focus on the detailed texture of small targets, while mid-to-high-level features are more inclined towards the overall semantic information of large targets. To address the decreased scale sensitivity caused by shared convolution, a learnable scale layer is added after the convolutional output of each detection head. This layer dynamically adjusts the feature amplitude of different scale branches by scaling the channel features one by one, making them better suited to the size distribution of the targets they are responsible for.

[0086] Regarding normalization, features at different levels have different statistical distributions; for example, lower-level features are often sensitive to details, while higher-level features encode more semantic information. Without a normalization layer, the network struggles to synchronize the numerical scale of features across different layers during training, potentially leading to unstable gradients or slow convergence. Therefore, regardless of the network design, a normalization layer remains indispensable.

[0087] In object detection, it is generally desirable for detection heads to share parameters to reduce model size and accelerate training and inference. However, if batch normalization (BN) is introduced directly into the detection head with shared parameters, a moving average cumulative error will occur: the difference in feature distribution in different input batches will cause the same set of BN parameters to perform inconsistently during training and inference, thus affecting detection accuracy.

[0088] In contrast, layer normalization or group normalization (GN) does not rely on statistics of the batch dimension, which theoretically avoids the above problems. However, GN still needs to perform group calculations within each sample during the inference stage, which significantly increases the computational overhead, especially in real-time detection scenarios.

[0089] To balance accuracy and efficiency, all convolutional layers in the detection head share the same weights, thus keeping the model parameter scale constant. Meanwhile, each normalization layer (BN) independently calculates and maintains its moving average statistics. This ensures numerical normalization between features at different levels, avoids cross-contamination of BN statistical errors, and does not introduce additional GN computation overhead. The entire structure is lightweight and robust, significantly improving the performance and convergence of the detection network in multi-scale feature fusion.

[0090] Further, step 5) includes:

[0091] The main steps of data augmentation include:

[0092] 1. Randomly read four images from the dataset each time.

[0093] 2. Perform four processing steps on the original image, including horizontal flipping, resizing, and adjusting color gamut parameters such as brightness and saturation. After completing these operations, arrange the four processed images in order: the first image in the top left corner, the second in the bottom left corner, the third in the bottom right corner, and the fourth in the top right corner.

[0094] 3. After processing and layouting the four images, the corresponding fixed regions are matrix-wise cropped, and these regions are stitched together to form a new image containing visual elements such as borders. This method borrows from and improves upon the CutMix data augmentation strategy. Unlike CutMix, which uses two images for stitching, this method uses four images for fusion. This not only enhances the diversity of backgrounds in the object detection task but also allows for the simultaneous processing of more image information during batch normalization (BN) calculation, thereby improving the model's generalization ability.

[0095] Experimental verification

[0096] 1. Implementation details

[0097] This invention uses an 8:2 split ratio to divide the dataset into a training set and a test set.

[0098] For model performance evaluation, this invention used mAP50, mAP50:95, Params, and GFLOPs as evaluation metrics. The initial learning rate was 0.01, with 100 epochs and a batch size of 32. The SGD optimizer was used to optimize the model parameters. The experiments were conducted on a server equipped with an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz * 7 cores and an NVIDIA GeForce GTX3090 graphics card.

[0099] 2. Comparative Experiment

[0100] This invention compares the proposed method with the benchmark model Yolov8n, and the specific results are as follows: Figure 3 As shown.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time classroom behavior detection method based on edge information and decoupled shared detection heads, characterized in that, Includes the following steps: 1) Collect surveillance videos covering classroom scenes of primary and secondary school students, and split these videos into frames to extract classroom behavior images for analysis; 2) Use the image annotation tool labellmg to manually annotate all the extracted student classroom behavior images to construct a student classroom behavior dataset containing three types of behaviors: "reading", "hand-raising", and "writing". 3) Preprocess the labeled data: Divide all labeled data into training set and test set, with 80% used as training data and the remaining 20% ​​used for model testing and evaluation; 4) Model establishment: Based on YOLOv8n, an efficient edge information convolution module is proposed to extract edge information of classroom body behavior, which improves the performance of model detection. A global-to-local Transformer module is added to enhance feature extraction capability. Context anchor attention is used to make up for the lack of global information perception when relying only on local multi-scale convolution. On the other hand, a lightweight and efficient decoupled shared detection head is adopted. 5) Enhance the data by using various methods, including mosaic stitching, image blending, random perturbation, and color perturbation, to improve the model's generalization ability and robustness in different scenarios. 6) Input the images and labels from the training set into the built model, set fixed training parameters, select an appropriate optimizer and loss function, train the model, and finally obtain a model for detecting students' classroom behavior. 7) Input the images and corresponding labels of the test set into the trained model, identify and locate the student behavior in each image, output the predicted category label and its location information, then compare these prediction results with the real labels, evaluate the detection effect of the model, and calculate the average precision, average precision of small targets, average precision of medium targets, and average precision of large targets. 8) Input the test image into the trained model to detect the students' classroom behavior in the image, and visualize the detection results with colored borders and annotations to intuitively show the content recognized by the model.

2. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that... We propose an efficient edge information convolution module (EICM) to extract edge information of classroom body behavior, which improves the performance of model detection. We also add GLTB to enhance feature extraction capabilities, and use CAA to compensate for the lack of global information perception when relying solely on local multi-scale convolution. Finally, we adopt a lightweight and efficient decoupled shared detection head (DSDH) in the final prediction head.

3. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that, The deep learning network for detecting student classroom behavior mainly consists of three parts: a feature extraction module, a feature fusion module, and a detection head module. The feature extraction capability is enhanced by combining the edge information convolution module EICM with GLTB, and CAA is used to increase efficiency while reducing the number of parameters and computation. Finally, the decoupled shared detection head DSDH is used in the final prediction head to significantly reduce the number of parameters and computation while ensuring accuracy.

4. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that, Given the limitations of the original network in global feature extraction and its large number of parameters, the EICM module was introduced into the feature fusion module to specifically capture the edge information of the image. Considering that although traditional convolutional neural networks can learn spatial structure well, they are not good at extracting edge details, EICM uses the Sobel filter, a classic operator, through the SobelConv branch to efficiently detect gray-level changes in the image and thus obtain key edge features.

5. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that, The use of CAA combines global average pooling with one-dimensional "strip" convolution to capture contextual relationships between distant pixels and enhance feature representation in concentrated regions, thereby compensating for the lack of global information perception when relying solely on local multi-scale convolution.

6. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that, The aforementioned DSDH significantly improves the overall performance of the detection head in localization and classification tasks by introducing a series of lightweight and efficient designs. First, the traditional BatchNorm is replaced with GroupNorm, so that the normalization operation no longer depends on a large batch of data, but instead groups and counts the channel dimensions according to a pre-set number of groups.

7. The real-time classroom behavior detection method based on edge information and decoupled shared detection head according to claim 1, characterized in that, To balance accuracy and efficiency, all convolutional layers in the detection head share the same weights, thus keeping the model parameter scale constant. Meanwhile, each normalization layer independently calculates and maintains its moving average statistics. This ensures the numerical normalization between features at different levels, avoids cross-contamination of BN statistical errors, and does not introduce additional GN computation overhead. The entire structure is lightweight and robust, significantly improving the performance and convergence of the detection network in multi-scale feature fusion.

Images