Classroom behavior identification method based on weak supervision fine-grained two-way network

Abstract

The invention relates to a class behavior identification method based on a weak supervision fine-grained two-way network. The method comprises the following steps: acquiring an original classroom video through preset fixed angle monitoring equipment, and extracting video frames to generate a standardized image set after screening; generating a student target frame detection result by using the target detection model, and finishing unique identification in combination with a preset classroom behavior standard after correction; establishing cross-frame association between the student ID and the unique identifier by using a time domain tracking algorithm, and generating a structured data set; inputting the structured data set into a weakly supervised fine-grained two-way network model, processing spatial semantics and motion information, generating a first identifier and a second identifier, and performing feature fusion to obtain a classroom behavior recognition result; an intelligent monitoring system is constructed based on an identification result, classroom behavior data are integrally processed through a bimodal data center, and meanwhile, a dynamic threshold value monitoring and increment iteration optimization module is arranged, so that optimization processing of classroom behavior identification is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent education and computer vision technology, specifically relating to a classroom behavior recognition method based on a weakly supervised fine-grained dual-path network. Background Technology

[0002] The following areas still need improvement in classroom behavior recognition: Insufficient dataset adaptability: Existing public classroom behavior datasets suffer from issues such as significant subjective disputes over category classification, limited labeling, and limited scenario coverage. Furthermore, many datasets originate from online videos rather than real classroom monitoring scenarios, failing to meet the needs of recognizing multiple individuals and multi-labeled behaviors in crowded classroom environments. Additionally, the datasets lack clear definitions and annotations for subtle classroom behaviors with similar postures but minor behavioral differences, such as reading, writing, and using mobile phones, resulting in a lack of effective data support for model training.

[0003] Limited accuracy of recognition algorithms: Traditional behavior recognition models are susceptible to interference from occlusion, truncation, and dynamic blurring in classroom monitoring scenarios, and their fusion processing of spatial semantic information and motion information is not accurate enough. Especially for classroom behaviors with small differences, existing algorithms struggle to focus on local feature differences, relying solely on overall action features, resulting in low recognition accuracy. Furthermore, they do not fully combine weakly supervised learning and fine-grained classification methods to improve intra-class discrimination capabilities.

[0004] Insufficient application of the technology: Existing classroom behavior recognition systems mostly operate independently, and the recognition results lack data correlation with school teaching quality evaluation, intelligent supervision, and other systems, making it impossible to transform them into valuable teaching decision-making references. At the same time, the models lack dynamic optimization mechanisms, making it difficult to adapt to changes in different classroom environments and teaching scenarios, and the lack of standardized data processing procedures results in insufficient reliability and practicality of the recognition results.

[0005] Low data processing efficiency: Classroom monitoring video data is large in volume and contains a lot of redundant information. Traditional data preprocessing methods lack targeted screening and frame extraction strategies, and there are problems of missed detection and false detection in the target detection process. Manual annotation is costly and inefficient, making it difficult to form high-quality structured datasets, which restricts the training effect of algorithm models and the efficiency of their practical application. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a classroom behavior recognition method based on a weakly supervised fine-grained dual-path network; The objective of this invention can be achieved through the following technical solutions: S1: Obtain raw video data of classroom behavior through monitoring equipment deployed at a fixed angle; S2: Filter the original video data, extract video frames based on the filtering results, and generate a standardized image set; use a target detection model to generate detection results of student target boxes, correct the detection results, and use preset classroom behavior standards for unique identification; use a temporal tracking algorithm to establish cross-frame association between student IDs and the unique identifiers, and generate a structured dataset. S3: Input the structured dataset into a weakly supervised fine-grained dual-path network model, and generate a first identifier and a second identifier by processing spatial semantic information and motion information; perform feature fusion based on the first identifier and the second identifier to generate classroom behavior recognition results; S4: Based on the classroom behavior recognition results, construct an intelligent monitoring system for classroom behavior recognition, and perform integrated processing of classroom behavior data through a dual-modal data hub; at the same time, establish a dynamic threshold monitoring and incremental iterative optimization module to optimize the classroom behavior recognition.

[0007] As a preferred technical solution of the present invention, the specific process of filtering the original video data is as follows: automatically identify video segments through inter-frame variance analysis, calculate the gray-level variance of consecutive frame images, and perform variance-based filtering on the original video data; embed classroom environment parameters based on the filtering results and record the environmental data of the original video data to obtain the filtered standardized video source.

[0008] Specifically, the process of generating the standardized image set includes: using a dynamic frame extraction strategy to perform adaptive deblurring on the standardized video source to obtain the processing result; performing standard format conversion based on the processing result, and establishing an image, video, and environment association index table to obtain the standardized image set.

[0009] Specifically, the process of generating the detection results of student target boxes includes: inputting the standardized image set into the retrained YOLOv7 model, scaling the input image using the multi-scale feature fusion module, extracting student target features at the corresponding scale, and generating initial detection results through model training; Based on the initial test results, an occlusion determination and compensation algorithm is introduced. The occluded area is filled with neighborhood pixel features to correct the target box coordinates of the occluded students and generate the target box results.

[0010] Specifically, the unique identifier is a three-dimensional combination identifier based on behavioral tags, student ID, and spatiotemporal stamp, and a credibility weight is added to the unique identifier to obtain the weight data of the unique identifier; the three-dimensional identifier and weight data are directly used for the label construction of the structured dataset.

[0011] Specifically, the process of generating the structured dataset is as follows: based on the target bounding box results, an intelligent annotation tool is used to automatically recommend behavioral labels that match the student target, and an appearance feature association strategy is incorporated to extract the HOG features and color histogram of the student target; cross-frame similarity is calculated through coordinate changes, and appearance features are used to complete the matching to obtain the continuity of the student ID in the corresponding scene; based on the continuity of the student ID, the dataset is divided through a stratified sampling strategy to generate a structured dataset.

[0012] Specifically, the first identifier is as follows: based on the structured dataset, the image is sampled through a slow path to obtain the basic spatial feature map of the image; the basic spatial feature map is input into a bilinear attention pooling module to generate an attention map of the feature region; the basic spatial feature map and the attention map are multiplied to obtain a local feature map of the student behavior; the feature channels of the local feature map are filtered through a channel attention mechanism to generate the first identifier of spatial semantics.

[0013] Specifically, the second identifier is as follows: based on the structured dataset, the image is sampled through a fast path, basic motion features are extracted using a channel lightweight convolution module, motion features of consecutive frames are calculated using a temporal attention mechanism based on the basic motion features, and corresponding weights are assigned to the actions; a second identifier of motion information is generated through a one-dimensional convolution kernel.

[0014] Specifically, the process of generating classroom behavior recognition results includes: using dynamic weight fusion calculation to perform feature fusion on the first identifier and the second identifier, automatically assigning weights to classroom behaviors through scene adaptation factors; obtaining initial confidence through a multi-label refined classifier, introducing a cross-frame temporal voting mechanism to count the average confidence of consecutive frames, generating classroom behavior recognition results and obtaining the classroom behavior reliability score.

[0015] Specifically, the integrated processing includes: using the dual-modal data hub to access the classroom behavior recognition results, classifying and storing unstructured video data and structured tag data through a distributed storage architecture; and using a temporal association algorithm to associate and bind the cross-frame behavior data, spatiotemporal stamp information, and credibility weight of student IDs.

[0016] Specifically, the dynamic threshold monitoring involves: pre-setting an initial confidence threshold range based on the classroom scenario type; collecting reliability scores of classroom behavior recognition results in real time; and using a sliding window algorithm to calculate the mean and variance of the scores over a continuous period. The threshold sensitivity is then adaptively adjusted based on changes in variance.

[0017] Specifically, the incremental iterative optimization module employs an incremental training strategy and uses a momentum update algorithm to iteratively optimize the parameters of a weakly supervised fine-grained dual-path network model, updating the feature weights for complex scenarios; after a preset number of iterations, the model performance is verified using an offline test set, and the model performance is optimized.

[0018] The beneficial effects of this invention are as follows: A multi-label dataset containing 14 categories of classroom behavior was constructed. Subjectively controversial categories were removed, and only behaviors that can be objectively judged by computer vision were retained, making it suitable for real classroom monitoring scenarios. It covers actual classroom environments such as crowded classrooms, occlusion, and dynamic blur. The annotation process combines object detection and temporal tracking to ensure the continuity of student IDs across frames, making the data quality more reliable. It clearly distinguishes between regular classroom behaviors and slightly different classroom behaviors, providing targeted data support for fine-grained behavior recognition and solving the problems of limited categories and poor applicability of existing datasets.

[0019] At the algorithm level: To optimize recognition accuracy and overcome the bottleneck of distinguishing small differences in behavior, the WS-FC SlowFast network is proposed. Weakly supervised fine-grained analysis is incorporated into the Slow path. Local feature extraction is enhanced through bilinear attention pooling and attention-guided pruning and discarding strategies. For similar posture behaviors such as reading, writing, using a mobile phone, and eating, the recognition accuracy is improved by 4%-11%, and the overall classroom behavior recognition accuracy is improved by 5.75%, which is significantly better than the traditional SlowFast network. A dual-path network structure is adopted to process spatial semantics and motion information separately. Combined with dynamic weight fusion and cross-frame temporal voting mechanism, the stability and reliability of behavior recognition in complex scenarios are improved.

[0020] Application Level: Empowering teaching evaluation and reducing implementation costs, it leverages existing classroom monitoring equipment to collect data without additional hardware investment. Cloud-based analysis enables low-cost deployment, adapting to classrooms of varying sizes. The identification results can be directly integrated into the teaching quality evaluation system, providing data support for teacher feedback and school management, and promoting smart education and data-driven educational reform. The constructed intelligent monitoring system supports integrated processing of dual-modal data, combining dynamic threshold monitoring and incremental iterative optimization to continuously adapt to changes in classroom scenarios and improve long-term application effectiveness. Attached Figure Description

[0021] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a classroom behavior recognition method based on a weakly supervised fine-grained dual-path network according to the present invention. Figure 2 This is a flowchart of the fine-grained behavior recognition process in this invention. Detailed Implementation

[0023] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0024] Please see Figure 1-2 A classroom behavior recognition method based on a weakly supervised fine-grained dual-path network includes: S1: Obtain raw video data of classroom behavior through monitoring equipment deployed at a fixed angle; S2: Filter the original video data, extract video frames based on the filtering results, and generate a standardized image set; use a target detection model to generate detection results of student target boxes, correct the detection results, and use preset classroom behavior standards for unique identification; use a temporal tracking algorithm to establish cross-frame association between student IDs and the unique identifiers, and generate a structured dataset. S3: Input the structured dataset into a weakly supervised fine-grained dual-path network model, and generate a first identifier and a second identifier by processing spatial semantic information and motion information; perform feature fusion based on the first identifier and the second identifier to generate classroom behavior recognition results; S4: Based on the classroom behavior recognition results, construct an intelligent monitoring system for classroom behavior recognition, and perform integrated processing of classroom behavior data through a dual-modal data hub; at the same time, establish a dynamic threshold monitoring and incremental iterative optimization module to optimize the classroom behavior recognition.

[0025] As a preferred technical solution of the present invention, the specific process of filtering the original video data is as follows: automatically identify video segments through inter-frame variance analysis, calculate the gray-level variance of consecutive frame images, and perform variance-based filtering on the original video data; embed classroom environment parameters based on the filtering results and record the environmental data of the original video data to obtain the filtered standardized video source.

[0026] In this embodiment, inter-frame variance analysis is performed on the raw classroom video data collected by the monitoring equipment. The grayscale variance of consecutive images is calculated frame by frame to identify the degree of dynamic change in the video content. When the grayscale variance is lower than a preset threshold, it is determined to be an invalid video segment (e.g., no one in the classroom, static image, video stuttering, etc.) and is automatically filtered out; valid segments with grayscale variance that meet the requirements and contain dynamic student classroom behavior are retained. Based on the filtered valid videos, classroom environment parameters (e.g., classroom number, monitoring angle, light intensity, equipment model, etc.) are embedded, and the environmental data corresponding to each video segment is recorded synchronously to ensure the binding of video source and scene information, ultimately obtaining a standardized video source, laying the foundation for subsequent image extraction and analysis.

[0027] Specifically, the process of generating the standardized image set includes: using a dynamic frame extraction strategy to perform adaptive deblurring on the standardized video source to obtain the processing result; performing standard format conversion based on the processing result, and establishing an image, video, and environment association index table to obtain the standardized image set.

[0028] In this embodiment, a dynamic frame extraction strategy is adopted for the standardized video source. The frame extraction frequency is adaptively adjusted according to the dynamic complexity of the video content. The frame extraction frequency is increased during periods of frequent changes in classroom behavior (such as group discussions and raising hands to answer questions), and decreased during periods of stable behavior (such as quiet listening and writing). At the same time, adaptive deblurring processing is performed on the extracted images to correct the blurring effect caused by motion or device jitter, resulting in a clear image processing result. The processed images are uniformly converted to a standard format (such as uniform resolution and JPG image format), and an associated index table containing image files, corresponding video segment indexes, and environmental parameters is established to clarify the source and scene background of each image, ultimately forming a standardized image set with a standardized structure and traceability.

[0029] Specifically, the process of generating the detection results of student target boxes includes: inputting the standardized image set into the retrained YOLOv7 model, scaling the input image using the multi-scale feature fusion module, extracting student target features at the corresponding scale, and generating initial detection results through model training; Based on the initial test results, an occlusion determination and compensation algorithm is introduced. The occluded area is filled with neighborhood pixel features to correct the target box coordinates of the occluded students and generate the target box results.

[0030] In this embodiment, a standardized image set is input into the YOLOv7 model retrained on the CrowdHuman dataset. The CrowdHuman dataset contains approximately 24,000 images: 15,000 for training, 4,370 for validation, and 5,000 for testing, covering various real-world crowd scenarios. The training and validation sets together contain 470,000 human instances, averaging 22-23 people per image, far exceeding the crowd density of typical pedestrian detection datasets (such as CityPersons), effectively simulating the characteristics of crowded scenes. The YOLOv7 model consists of a backbone network (ELAN): addressing issues such as dynamic blur, uneven lighting, and dense overlapping of students in classroom monitoring by integrating Mosaic v2 enhancement (stitching four images to simulate crowded scenes), adaptive denoising, and scale normalization (uniform scaling to 640×640 resolution), ensuring the consistency and diversity of the input data. Neck feature fusion (PANet+SPP) integrates multi-scale features output from the backbone network to address issues such as large front-row student targets, small back-row student targets, and partial occlusion in classroom scenarios. PANet achieves bidirectional fusion from top to bottom and bottom to top, transferring deep semantic features (e.g., student category information) to shallow layers to improve the detection rate of small back-row targets; and transferring shallow detail features (e.g., outlines of objects like pens and phones) to deep layers to assist in the initial differentiation of behaviors such as writing and using a phone. The embedded SPP module (1×1, 5×5, 9×9 pooling kernels) enhances the feature capture capability for deformed targets such as bending over to write or turning their heads to the side, reducing the impact of feature loss due to occlusion. The detection head (RepDetHead) outputs the bounding box, confidence score, and preliminary behavior category of the student target based on the fused features, supporting multi-label annotation. An anchor box mechanism is adopted, which generates anchor boxes (such as anchor boxes of different sizes for students' bodies, hands, and mobile phones) that are adapted to classroom objectives through K-means clustering; it supports multi-label classification (such as students can be labeled as looking down and using mobile phones at the same time), adapting to scenarios with multiple overlapping classroom behaviors; a multi-branch structure is used in the training phase to enhance the fitting ability, and the inference phase is fused into a single branch, without increasing the amount of computation, ensuring real-time performance.

[0031] Input data type: Image data (JPG format) extracted from classroom monitoring video, with a resolution of 1080P (original output from monitoring), and each image contains 10-120 students (covering different classroom sizes).

[0032] The input data carries information such as image pixel matrix, shooting timestamp, classroom number, etc., but no pre-labeled labels (the input during the model training phase includes manually labeled student bounding boxes and behavior category labels).

[0033] Input size: Batch input (Batch Size=8), adapted to GPU memory (16GB can support processing 8 images at the same time), matching the batch processing needs after frame extraction from classroom videos.

[0034] Data flow and processing between modules: (1) Input preprocessing module: standardization and enhancement Scale normalization: Scale 1080P images to a uniform resolution of 640×640 while maintaining the target aspect ratio; Data augmentation: Mosaic v2 stitching (randomly selecting 4 different classroom scene images for stitching), random flipping (simulating the left and right orientation changes of students), and brightness and contrast adjustment (adapting to changes in classroom brightness) were performed on the training set images. Format conversion: Normalize the image pixel values ​​to the [0,1] range, convert to Tensor format, add batch dimensions, and then pass them into the backbone network.

[0035] Output data: Standardized batch feature maps (dimensions: 8×3×640×640, i.e., Batch Size×Number of Channels×Height×Width), which are robust against blurring and illumination interference.

[0036] (2) Backbone Network (ELAN): Feature Extraction Multi-branch feature extraction: The input feature map is divided into 4 parallel branches, which extract detailed and global features through different numbers of convolutional kernels. The small branch (64 convolutional kernels) captures details such as students' hair and pens, while the large branch (256 convolutional kernels) captures body contours and behavioral patterns. Feature aggregation: The outputs of the four branches are concatenated according to the channel dimension (dimension: 8×1024×160×160), and the number of channels is compressed to 512 through 1×1 convolution to avoid parameter expansion; Residual connections: Input features and aggregated features are superimposed across layers to alleviate gradient vanishing and ensure that shallow details (such as the outline of a mobile phone in a student's hand) can be transmitted to deeper layers.

[0037] Output data: 3 feature maps at different scales (C3, C4, C5), corresponding to resolutions of 160×160, 80×80, and 40×40 respectively, covering multi-dimensional features from details to semantics, providing support for subsequent detection of small targets (such as mobile phones) and large targets (such as student bodies).

[0038] (3) Neck feature fusion (PANet+SPP): Feature enhancement Top-down fusion (FPN): Upsamples the deep C5 features (40×40) to 80×80, adds them to the C4 features, and passes semantic information to the middle layer to improve the detection capability of small targets in the back row (such as the head of a student in the distance). Bottom-up fusion (PANet): The mid-layer fusion features are downsampled to 40×40 and added to the C5 features to pass detailed information to the deep layers, helping to distinguish behaviors that rely on hand details, such as writing and using a mobile phone; SPP multi-scale pooling: performs 1×1, 5×5, and 9×9 pooling on the top-level features, and outputs them after concatenation, which enhances the feature capture of deformed targets such as those lying on the table or falling down, and reduces the feature distortion caused by classroom occlusion.

[0039] Output data: 3 fused feature maps (P3, P4, P5), with resolutions of 160×160, 80×80, and 40×40 respectively, and dimensions of 8×256× corresponding to the resolution, achieving deep fusion of details and semantic features.

[0040] (4) Detection head (RepDetHead): Target localization and classification Predictive convolution: Perform 3×3 convolution on the P3, P4, and P5 feature maps respectively, mapping them to a prediction tensor of bounding box parameters + class probability + confidence. The bounding box parameters (4 values: x, y, w, h) locate the student target, the class probability corresponds to 14 types of classroom behavior, and the confidence determines the probability of the target's existence. Anchor box matching: Call the anchor boxes pre-trained in the classroom scene (such as student body anchor boxes [60,120], mobile phone anchor boxes [10,20]), match them with the predicted bounding boxes, and filter candidate targets with confidence ≥0.3; Non-maximum suppression (NMS): Deduplicates multiple candidate boxes for the same student (IOU threshold set to 0.45), retaining the bounding box with the highest confidence to avoid duplicate detection; Multi-label output: By replacing Softmax with the Sigmoid activation function, it supports labeling multiple behaviors of the same student (e.g., looking down + using a mobile phone), adapting to classroom behavior overlay scenarios.

[0041] Output data: Batch object detection results (Tensor format). Each result includes the object bounding box coordinates (x1, y1, x2, y2), confidence score, and probabilities of 14 behavior categories. For example, [x1=120, y1=150, x2=200, y2=300, conf=0.92, class_probs=[0.01, 0.85, ..., 0.78]] corresponds to the double-label result of looking down (0.85) + using a mobile phone (0.78).

[0042] Final output and downstream connection: Output format: JSON format detection result file, including image number, timestamp, student target list (boundary box, confidence score, and behavior probability for each target), and generates a visualization annotation map (overlaying bounding boxes and behavior labels).

[0043] Downstream data flow: The detection results are directly fed into the Slow path of the WS-FC SlowFast algorithm, where bounding boxes are used to crop local areas of students (such as hands and heads), and behavior probabilities serve as weakly supervised signals to assist in fine-grained classification; the visualization is used for real-time classroom observation display in the teaching quality evaluation system.

[0044] Classroom scenario testing performance verification (data support): On a self-built classroom behavior dataset (6755 images, 45348 student targets), the detection performance of the YOLOv7 module is as follows: Precision (P): 85.2%, Recall (R): 81.7%, mAP@0.5: 87.5%; The false negative rate for densely packed students and partially occluded scenes is ≤5%, which is much lower than that of traditional models (15%-20%). The inference time for a single image is ≤20ms, and it supports real-time processing of more than 30FPS, meeting the needs of continuous analysis of classroom videos.

[0045] These data demonstrate that the YOLOv7 module, through module adaptation and process optimization, perfectly undertakes the front-end target localization task of classroom behavior detection. Its output of high-quality bounding boxes and behavior probabilities lays a solid foundation for subsequent fine-grained recognition of small differences in classroom behavior.

[0046] After optimization, the model significantly improves target detection capabilities in crowded and occluded scenarios. The model utilizes a multi-scale feature fusion module to scale the input image at different scales, extracting student target features at small scale (back row students), medium scale (middle area students), and large scale (front row students), comprehensively capturing the contour and morphological information of students in different positions. The model training outputs initial detection results containing student target coordinates and confidence scores. To address the occlusion issue in the initial detection results, an occlusion determination and compensation algorithm is introduced: first, the occlusion area and degree are determined by the overlap and edge integrity of the target bounding boxes. For partially occluded student targets, a neighborhood pixel feature completion method is used, referencing the pixel distribution and contour features around the occluded area to correct the target bounding box coordinates, ensuring the target bounding box completely encompasses the student subject. For severely occluded but identifiable targets, the target bounding box range is adjusted based on contextual information, ultimately generating accurate and complete student target bounding box results.

[0047] Specifically, the unique identifier is a three-dimensional combination identifier based on behavioral tags, student ID, and spatiotemporal stamp, and a credibility weight is added to the unique identifier to obtain the weight data of the unique identifier; the three-dimensional identifier and weight data are directly used for the label construction of the structured dataset.

[0048] In this embodiment, the unique identifier adopts a three-dimensional combination of behavior tags, student IDs, and spatiotemporal stamps: behavior tags correspond to 14 preset classroom behaviors (such as looking up, writing, using a mobile phone, etc.), student IDs are unique numerical codes assigned by the system, and spatiotemporal stamps are accurate to the millisecond level, including the video capture time and the student's spatial coordinate range in the image. Simultaneously, a confidence weight is added to the unique identifier. The weight value is calculated comprehensively based on the target detection confidence, behavior tag matching degree, and spatiotemporal stamp accuracy; the higher the detection confidence and the more accurate the tag matching, the higher the confidence weight. This three-dimensional identifier and its corresponding weight data are directly embedded into the label field of the structured dataset, enabling unique traceability of each student behavior instance and providing a core identifier basis for subsequent cross-frame association and data management.

[0049] Specifically, the process of generating the structured dataset is as follows: based on the target bounding box results, an intelligent annotation tool is used to automatically recommend behavioral labels that match the student target, and an appearance feature association strategy is incorporated to extract the HOG features and color histogram of the student target; cross-frame similarity is calculated through coordinate changes, and appearance features are used to complete the matching to obtain the continuity of the student ID in the corresponding scene; based on the continuity of the student ID, the dataset is divided through a stratified sampling strategy to generate a structured dataset.

[0050] In this embodiment, based on the student bounding box results, the intelligent annotation tool VoTT is invoked to automatically recommend the most relevant behavior labels based on the student's posture and action features (e.g., if a student is detected holding a pen and facing a book, a "writing" label is recommended). Simultaneously, an appearance feature association strategy is incorporated to extract the HOG features (edge ​​direction distribution) and color histograms (clothing and hairstyle color features) of the student targets, establishing a student appearance feature database. By calculating the coordinate changes and appearance feature similarity of student targets across frames, matching breaks caused by occlusion or slight posture changes are filled, ensuring the continuity of student IDs in consecutive frames. Based on the continuity of student IDs, a stratified sampling strategy is adopted, dividing the training, testing, and validation sets according to classroom type, class time, and behavior category ratio. Information such as 3D unique identifiers, confidence weights, and environmental parameters are synchronously associated, ultimately generating a structured dataset with standardized structure, complete labels, and direct usability for model training.

[0051] Specifically, the first identifier is as follows: based on the structured dataset, the image is sampled through a slow path to obtain the basic spatial feature map of the image; the basic spatial feature map is input into a bilinear attention pooling module to generate an attention map of the feature region; the basic spatial feature map and the attention map are multiplied to obtain a local feature map of the student behavior; the feature channels of the local feature map are filtered through a channel attention mechanism to generate the first identifier of spatial semantics.

[0052] In this embodiment, the slow path of the weakly supervised fine-grained dual-path network model samples images from the structured dataset at a low frame rate, focusing on spatial semantic information extraction to generate a basic spatial feature map containing the student's overall posture and local action details. This feature map is input into the bilinear attention pooling module, and the model automatically learns and generates attention maps for the feature regions, highlighting key areas related to classroom behavior (such as hand movements, facial orientation, and held objects). The basic spatial feature map and the attention map are multiplied element-wise to enhance key region features and suppress background interference, obtaining local feature maps of student behavior (such as hand holding features for using a mobile phone and pen-book contact features for writing). The discriminative feature channels in the local feature maps are selected through a channel attention mechanism to remove redundant information, ultimately generating a first identifier that focuses on core spatial semantics, providing a key basis for distinguishing subtle differences in classroom behavior.

[0053] Specifically, the second identifier is as follows: based on the structured dataset, the image is sampled through a fast path, basic motion features are extracted using a channel lightweight convolution module, motion features of consecutive frames are calculated using a temporal attention mechanism based on the basic motion features, and corresponding weights are assigned to the actions; a second identifier of motion information is generated through a one-dimensional convolution kernel.

[0054] In this embodiment, the fast path of the weakly supervised fine-grained dual-path network model samples images from the structured dataset at a high frame rate, focusing on motion information extraction. Lightweight convolutional modules are used to quickly extract features from consecutive frames, obtaining basic motion features of student behavior (such as head rotation trajectory, hand raising motion, and body movement trends). Based on these basic motion features, a temporal attention mechanism is used to calculate the amplitude and pattern of motion changes between consecutive frames. Motions with behavioral discriminative value (such as the upward movement of a hand when raising it, and the rotation of the head when turning the head) are assigned higher weights, while meaningless minor tremors are assigned lower weights. One-dimensional convolutional kernels are used to temporally compress and integrate the weighted motion features, capturing the temporal sequence patterns of behavior, and finally generating a second identifier representing motion information, which complements the first identifier.

[0055] Specifically, the process of generating classroom behavior recognition results includes: using dynamic weight fusion calculation to perform feature fusion on the first identifier and the second identifier, automatically assigning weights to classroom behaviors through scene adaptation factors; obtaining initial confidence through a multi-label refined classifier, introducing a cross-frame temporal voting mechanism to count the average confidence of consecutive frames, generating classroom behavior recognition results and obtaining the classroom behavior reliability score.

[0056] In this embodiment, a dynamic weighted fusion algorithm is used to fuse the features of the first identifier (spatial semantics) and the second identifier (motion information). A scene adaptation factor is introduced to automatically assign weights; for example, static behaviors such as reading and writing are assigned higher weights to the first identifier (spatial semantics), while dynamic behaviors such as walking and clapping are assigned higher weights to the second identifier (motion information). The fused features are input into a multi-label refined classifier to obtain the initial confidence scores for various classroom behaviors. To improve recognition reliability, a cross-frame temporal voting mechanism is introduced, statistically analyzing the mean confidence score of consecutive frames. If the mean score is higher than a preset threshold, the classroom behavior is confirmed as valid. Simultaneously, the classroom behavior reliability score is calculated by combining the mean confidence score and variance, ultimately generating a classroom behavior recognition result that includes the behavior category, confidence score, and reliability score.

[0057] Data Input Stage: Input data type: Structured data after YOLOv7 detection, including: Student target cropped image: 640×640 resolution, one image corresponds to one student target, including head, body, hands and other areas; Related information: target bounding box coordinates, confidence level (≥0.3), timestamp, classroom scene identifier (e.g., level flight / takeoff and landing corresponds to quiet / interactive classroom scene); Batch input: Batch Size=8, each batch contains 16 consecutive frames of images of 8 student targets (adapted to Slow path τ=16 sampling step size).

[0058] Input data scale: Based on a self-built classroom behavior dataset, 6755 labeled images (including 45348 student targets and 94674 behavior labels) were input during the training phase and divided into training set, validation set and test set in an 8:1:1 ratio.

[0059] Data flow and processing between modules: (1) Data preprocessing module: standardization and enhancement Image standardization: The cropped student target images are uniformly scaled to 640×640, and the pixel values ​​are normalized to [0,1]. Temporal alignment: Sort the image sequence by timestamp to construct a continuous frame sequence (each sequence contains 16 frames, corresponding to a Slow path with T=16 sampling times). Data augmentation: The training set is subjected to random flipping, brightness adjustment of ±10%, and partial occlusion simulation (adapted to classroom scenarios where students are occluded in front and behind).

[0060] Output data: Standardized batch image sequence (dimensions: 8×3×16×640×640, i.e., Batch Size×Number of Channels×Number of Frames×Height×Width), including temporal correlation information.

[0061] (2) Slow path: spatial semantics and fine-grained feature extraction Low frame rate sampling: Sampling is performed from a 16-frame sequence at a step size of τ=16, with 2 keyframes retained per sequence (corresponding to a sampling rate of 2 frames / second). Feature extraction: The ResNet-35 convolutional module outputs a 512-dimensional spatial feature map (H×W×C=160×160×512). Weakly supervised attention guidance: An attention map is generated based on behavior labels (e.g., writing). The feature map and the attention map are multiplied element-wise by the BAP module to generate a local feature map (focusing on the hand region). Attention cropping and discarding: Set thresholds θc=0.3 and θd=0.2, retain regions with attention values ​​higher than the thresholds, remove background interference, and obtain a local feature map of 1 / 4 size; Fine-grained classification: Subdivide the local feature map within each class and output the preliminary probabilities of four small differences in behavior (e.g., the probability of writing is 0.72, and the probability of reading is 0.15).

[0062] Output data: 512-dimensional spatial semantic feature vectors (including local feature enhancement information), with each batch outputting an 8×512-dimensional feature matrix.

[0063] (3) Fast path: motion information extraction High frame rate sampling: Sample from a 16-frame sequence at a step size of τ / α=2 (α=8), retaining 8 frames per sequence (corresponding to a sampling rate of 8 frames / second); Lightweight convolution extraction: Motion features (such as hand movement trajectory and head turning trend) are extracted through a convolution module with βC=64 channels (β=1 / 8). Temporal convolution enhancement: Employs 5×12 convolution kernels to capture motion correlations between consecutive frames (such as the rhythm of finger swiping when using a mobile phone). Feature compression: Motion features are compressed into a 256-dimensional vector through global average pooling.

[0064] Output data: 256-dimensional motion feature vectors, with each batch outputting an 8×256-dimensional feature matrix.

[0065] (4) Lateral connection fusion module: cross-path information fusion Dimension alignment: The 256-dimensional features of the Fast path are transformed into 512 dimensions through temporal convolution to match the feature dimensions of the Slow path; Dynamic weight fusion: Introduce scene adaptation factors (the weight of the flying scene is biased towards the slow path, and the weight of the interactive scene is biased towards the fast path), and automatically allocate weights (e.g., the weight of the writing behavior is 0.7 for the slow path and 0.3 for the fast path). Feature concatenation: The dual-path features are concatenated along the channel dimension to generate a 1024-dimensional fused feature vector.

[0066] Output data: 1024-dimensional fusion feature matrix (8×1024), containing collaborative features of spatial semantics and motion information.

[0067] (5) Multi-label classifier: Behavior recognition output Initial confidence calculation: The 1024-dimensional features are mapped to confidence scores for 14 classes of classroom behaviors through a fully connected layer; Cross-frame temporal voting: Calculate the average confidence score of 10 consecutive frames to reduce the impact of false detections in a single frame (such as misjudging a momentary head tilt as using a mobile phone). Reliability score: Combine the mean and variance of confidence to generate a behavior recognition reliability score (0-100 points).

[0068] Output data: Confidence scores for 14 types of behaviors for each student's goal (e.g., reading 0.82, writing 0.03, using mobile phone 0.05), reliability scores (e.g., 89 points), and final behavior labels (e.g., reading).

[0069] Output format: JSON format recognition results, including student ID, timestamp, behavior label, confidence score, reliability score, and target bounding box coordinates; Downstream flow: Output data is fed into the teaching quality evaluation system to generate student behavior analysis reports (such as head-up rate, distribution of minor behavioral differences) and real-time classroom monitoring alerts (such as reminders of abnormal mobile phone use); at the same time, it is synchronized to the data layer for incremental model training.

[0070] Overall behavior recognition accuracy: 80.55%, an improvement of 5.75% compared to the original SlowFast; Accuracy of identifying small differences in classroom behavior: average improvement of 4%-11% (writing behavior increased from 69.82% to 70.26%, and mobile phone use increased from 66.40% to 71.33%). Real-time performance: Single-frame inference time ≤30ms, supports real-time processing of 30FPS and above, meeting the needs of continuous analysis of classroom videos; Robustness: In occlusion and dynamic blurring scenarios, the recall rate is ≥75%, which is significantly better than traditional dual-path networks.

[0071] Specifically, the integrated processing includes: using the dual-modal data hub to access the classroom behavior recognition results, classifying and storing unstructured video data and structured tag data through a distributed storage architecture; and using a temporal association algorithm to associate and bind the cross-frame behavior data, spatiotemporal stamp information, and credibility weight of student IDs.

[0072] In this embodiment, after the dual-modal data hub receives the classroom behavior recognition results, it uses a distributed storage architecture for categorized storage: unstructured video data (original video, standardized image set) is stored on a high-speed file server, supporting fast retrieval and playback; structured tag data (behavior category, student ID, spatiotemporal stamp, credibility weight, reliability score, etc.) is stored in a relational database for efficient querying and statistics. A temporal association algorithm is used to concatenate cross-frame behavior data for the same student ID according to spatiotemporal stamp order, forming an individual classroom behavior temporal trajectory; simultaneously, behavior data is associated and bound with credibility weight and environmental parameters to ensure that each behavior record can be traced back to the original video clip and collection scene, achieving integrated management and full-link traceability of classroom behavior data.

[0073] Specifically, the dynamic threshold monitoring involves: pre-setting an initial confidence threshold range based on the classroom scenario type; collecting reliability scores of classroom behavior recognition results in real time; and using a sliding window algorithm to calculate the mean and variance of the scores over a continuous period. The threshold sensitivity is then adaptively adjusted based on changes in variance.

[0074] In this embodiment, an initial confidence threshold range is preset based on the type of classroom scenario (e.g., different threshold ranges are set for theoretical classes, practical classes, and group discussion classes). The system collects the reliability score of classroom behavior recognition results in real time and uses a sliding window algorithm to continuously calculate the mean and variance of the scores over consecutive time periods. When the variance is small (the classroom behavior recognition results are stable), the threshold sensitivity is appropriately increased to strictly filter low-confidence behaviors; when the variance is large (the classroom scenario is complex and behaviors are variable), the threshold sensitivity is decreased to avoid missing valid behaviors. Through adaptive adjustment, the matching degree between the threshold and the actual classroom scenario is ensured, thereby improving the accuracy and robustness of the recognition results.

[0075] Specifically, the incremental iterative optimization module employs an incremental training strategy and uses a momentum update algorithm to iteratively optimize the parameters of a weakly supervised fine-grained dual-path network model, updating the feature weights for complex scenarios; after a preset number of iterations, the model performance is verified using an offline test set, and the model performance is optimized.

[0076] In this embodiment, the incremental iterative optimization module adopts an incremental training strategy, continuously collecting complex scene data (such as severe occlusion, special behavioral postures, and newly emerging classroom behavior types) that occur during classroom behavior recognition, and adding them as incremental samples to the training set. Based on the momentum update algorithm, combined with the momentum information of historical training parameters, the parameters of the weakly supervised fine-grained dual-path network model are iteratively optimized, focusing on updating the feature weights in complex scenes (such as strengthening the recognition weights of local features in occluded scenes). After a preset number of iterations, the optimization effect is verified through model performance evaluation metrics (precision, recall, mAP). If the improvement of the metrics reaches a preset threshold, the updated model parameters are saved; if not, the learning rate is adjusted to continue iteration, continuously improving the model's adaptability to complex classroom scenes and recognition accuracy.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A classroom behavior recognition method based on a weakly supervised fine-grained dual-path network, characterized in that, include: S1: Obtain raw video data of classroom behavior through monitoring equipment deployed at a fixed angle; S2: Filter the original video data, extract video frames based on the filtering results, and generate a standardized image set; A target detection model is used to generate detection results of student target boxes. The detection results are then corrected and uniquely identified using preset classroom behavior standards. A time-domain tracking algorithm is used to establish a cross-frame association between student IDs and the unique identifiers, generating a structured dataset; S3: Input the structured dataset into a weakly supervised fine-grained dual-path network model, and generate a first identifier and a second identifier by processing spatial semantic information and motion information; Based on the feature fusion of the first identifier and the second identifier, classroom behavior recognition results are generated; S4: Based on the classroom behavior recognition results, construct an intelligent monitoring system for classroom behavior recognition, and perform integrated processing of classroom behavior data through a dual-modal data hub; Simultaneously, a dynamic threshold monitoring and incremental iterative optimization module is established to optimize the classroom behavior recognition.

2. The method according to claim 1, characterized in that, The specific process for filtering the original video data is as follows: video segments are automatically identified through inter-frame variance analysis, the grayscale variance of consecutive frame images is calculated, and the original video data is filtered by variance-based filtering. Based on the screening results, classroom environment parameters are embedded and environmental data of the original video data is recorded to obtain the screened standardized video source.

3. The method according to claim 1, characterized in that, The specific process of generating the standardized image set includes: using a dynamic frame extraction strategy to perform adaptive deblurring on the standardized video source to obtain the processing result; performing standard format conversion based on the processing result, and establishing an image, video and environment association index table to obtain the standardized image set.

4. The method according to claim 1, characterized in that, The specific process of generating the detection results of student target boxes includes: inputting the standardized image set into the retrained YOLOv7 model, using the multi-scale feature fusion module to scale the input image, extracting student target features at the corresponding scale, and generating initial detection results through model training; Based on the initial test results, an occlusion determination and compensation algorithm is introduced. The occluded area is filled with neighborhood pixel features to correct the target box coordinates of the occluded students and generate the target box results.

5. The method according to claim 1, characterized in that, The unique identifier is a three-dimensional combination identifier based on behavioral tags, student ID, and spatiotemporal stamp, and a credibility weight is added to the unique identifier to obtain the weight data of the unique identifier; the three-dimensional identifier and weight data are directly used for the label construction of the structured dataset.

6. The method according to claim 1, characterized in that, The specific process for generating the structured dataset is as follows: Based on the target bounding box results, an intelligent annotation tool is used to automatically recommend behavioral labels that match the student target, and an appearance feature association strategy is incorporated to extract the HOG features and color histogram of the student target; cross-frame similarity is calculated through coordinate changes, and appearance features are used to complete the matching to obtain the continuity of the student ID in the corresponding scene; based on the continuity of the student ID, the dataset is divided through a stratified sampling strategy to generate a structured dataset.

7. The method according to claim 1, characterized in that, The first identifier is as follows: Based on the structured dataset, the image is sampled through a slow path to obtain the basic spatial feature map of the image; the basic spatial feature map is input into a bilinear attention pooling module to generate an attention map of the feature region; the basic spatial feature map and the attention map are multiplied to obtain a local feature map of the student behavior; the feature channels of the local feature map are filtered through a channel attention mechanism to generate the first identifier of spatial semantics.

8. The method according to claim 1, characterized in that, The second identifier is as follows: Based on the structured dataset, the image is sampled through a fast path, basic motion features are extracted using a channel lightweight convolution module, and motion features of consecutive frames are calculated using a temporal attention mechanism based on the basic motion features, while assigning corresponding weights to the actions; a second identifier of motion information is generated through a one-dimensional convolution kernel.

9. The method according to claim 1, characterized in that, The specific process for generating classroom behavior recognition results includes: using dynamic weight fusion calculation to perform feature fusion on the first identifier and the second identifier, automatically assigning weights through scene adaptation factors to assign corresponding weights to classroom behaviors; obtaining initial confidence through a multi-label refined classifier, introducing a cross-frame temporal voting mechanism to count the average confidence of consecutive frames, generating classroom behavior recognition results and obtaining the classroom behavior reliability score.

10. The method according to claim 1, characterized in that, The specific process of the integrated processing includes: using the dual-modal data hub to access the classroom behavior recognition results, classifying and storing unstructured video data and structured tag data through a distributed storage architecture; and using a temporal association algorithm to associate and bind the cross-frame behavior data of student IDs, spatiotemporal stamp information, and credibility weights.

11. The method according to claim 1, characterized in that, The dynamic threshold monitoring involves: pre-setting an initial confidence threshold range based on the classroom scenario type; collecting reliability scores of classroom behavior recognition results in real time; and using a sliding window algorithm to calculate the mean and variance of the scores over a continuous period. The threshold sensitivity is then adaptively adjusted based on changes in variance.

12. The method according to claim 1, characterized in that, The incremental iterative optimization module employs an incremental training strategy and uses a momentum update algorithm to iteratively optimize the parameters of a weakly supervised fine-grained dual-path network model, updating the feature weights for complex scenarios. The model performance is verified using an offline test set after a preset number of iterations, and the model performance is then optimized.

Images