Classroom abnormal behavior recognition method and device fusing human body key points and spatial-temporal characteristics

Abstract

The invention provides a classroom abnormal behavior recognition method and device fusing human body key points and spatial-temporal characteristics, and the method is characterized in that the method comprises the steps: detecting a target student in a monitoring video stream through a target detection network, and determining a first bounding box of the target student; performing identification association on the target student based on the first bounding box by using a multi-target tracking algorithm, and generating a video sequence fragment containing the identification; inputting the video sequence segment into a human body posture estimation network, extracting human body key points of the target student in each frame of image, and generating a key point heat map sequence; the video sequence segment and the key point heat map sequence are input into a classroom behavior recognition network for behavior recognition, a classroom abnormal behavior recognition result is obtained, and the classroom behavior recognition network is constructed based on a key point guided space attention mechanism.

Description

A method and device for identifying abnormal classroom behavior by integrating key human body points and spatiotemporal features. Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and artificial intelligence, specifically to a method and device for identifying abnormal classroom behavior by integrating key points of the human body and spatiotemporal features. Background Technology

[0002] Classroom teaching quality evaluation and student behavior monitoring are important components of smart education. However, existing classroom behavior recognition technologies face significant bottlenecks in practical applications: First, the discrimination criteria are singular and lack anti-interference capabilities. Existing technologies mostly rely on single-frame two-dimensional features or simple coordinate calculations, which can easily lead to false detections in densely populated or occluded environments, such as difficulty in distinguishing between "looking down at a book" and "playing on a phone under the table." Second, temporal features are not fully utilized. Classification based solely on key point coordinates often discards crucial RGB visual information, resulting in an inability to effectively perceive environmental objects (such as mobile phones) and uncover long-term dynamic features of actions. Third, the models are bloated and difficult to deploy in real time. High-precision 3D convolutional networks or standard pose estimation models have a huge computational load, making it difficult to meet the high frame rate real-time analysis requirements of edge devices for multiple targets in the classroom. Therefore, there is an urgent need to develop a classroom behavior recognition method that can achieve lightweight operation and deeply integrate pose geometric features and visual spatiotemporal features to overcome the limitations of existing technologies. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for identifying abnormal classroom behavior by integrating human key points and spatiotemporal features. The specific technical solution adopted is as follows: In a first aspect, embodiments of this invention provide a method for identifying abnormal classroom behavior by integrating human key points and spatiotemporal features. The method includes: using a target detection network to detect a target student in a monitoring video stream and determining a first bounding box of the target student; using a multi-target tracking algorithm to identify and associate the target student based on the first bounding box, generating a video sequence segment containing the identification; inputting the video sequence segment into a human pose estimation network to extract the human key points of the target student in each frame of the image, generating a key point heatmap sequence; inputting the video sequence segment and the key point heatmap sequence into a classroom behavior recognition network for behavior recognition, obtaining a result of abnormal classroom behavior recognition, wherein the classroom behavior recognition network is constructed based on a key point-guided spatial attention mechanism.

[0004] Secondly, a classroom abnormal behavior recognition device integrating human key points and spatiotemporal features is provided. The device includes: a detection module for detecting target students in a monitoring video stream using a target detection network and determining a first bounding box of the target students; an association module for using a multi-target tracking algorithm to identify and associate the target students based on the first bounding box, generating a video sequence segment containing the identification; an extraction module for inputting the video sequence segment into a human pose estimation network to extract human key points of the target students in each frame of the image, generating a key point heatmap sequence; and a recognition module for inputting the video sequence segment and the key point heatmap sequence into a classroom behavior recognition network for behavior recognition, obtaining a classroom abnormal behavior recognition result, wherein the classroom behavior recognition network is constructed based on a key point-guided spatial attention mechanism.

[0005] Thirdly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0006] Fourthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0007] This invention offers the following advantages: It constructs a dual-stream multimodal input architecture, feeding video sequence segments into the backbone of a deep spatiotemporal residual network while simultaneously using keypoint heatmap sequences as side-guiding signals. A keypoint-guided spatial attention module (KSAM) is embedded, generating a spatial attention mask using keypoint distribution and weighting the original video features. This significantly enhances attention to behaviorally sensitive areas such as hands and faces, while preserving environmental features and improving the ability to detect subtle abnormal behaviors such as "playing on a phone" and "sleeping." It enables intelligent monitoring of student behavior, suitable for teaching environments with strict requirements for students, and effectively improves the real-time recognition efficiency of various behaviors such as playing on a phone and sleeping. The accompanying drawings are provided to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art. The drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 is a schematic diagram of an abnormal behavior recognition system provided in an embodiment of this application; Figure 2 is a schematic diagram of a lightweight attention-based module provided in an embodiment of this application; Figure 3 is a schematic diagram of a classroom behavior recognition network provided in an embodiment of this application; Figure 4 is a flowchart of a classroom abnormal behavior recognition method that integrates human body key points and spatiotemporal features provided in an embodiment of this application; Figure 5 is a schematic diagram of a classroom abnormal behavior recognition device that integrates human body key points and spatiotemporal features provided in an embodiment of this application.

[0009] Figure 6 is a schematic diagram of the structure of a computer block device provided in an embodiment of this application. Detailed Implementation

[0010] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a classroom abnormal behavior identification method integrating human key points and spatiotemporal features proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined from any suitable form.

[0011] In the description of the embodiments of the present invention, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" means two or more.

[0012] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0014] Figure 1 is a schematic diagram of an abnormal behavior recognition system provided in an embodiment of this application. As shown in Figure 1, the abnormal behavior recognition system includes at least: Module 1 (detection module), used for real-time target detection using YOLO; Module 2 (association module), used for real-time target tracking using DeepSORT and associating with student identifiers; Module 3 (extraction module), used for human key point detection using EL-HRNet and generating a key point heatmap sequence; and Module 4 (recognition module), used for classroom behavior recognition using KSAM-3D-FFC-ResNet.

[0015] During implementation, the following steps can be performed using the above modules to achieve classroom behavior recognition: Step S1: Obtain the real-time monitoring video stream of the classroom, use Module 1 (Lightweight Object Detection Network) to locate the target student in the video frame, and output the target bounding box; Here, the lightweight object detection network can be YOLOv10, which adopts a consistent dual allocation strategy to achieve end-to-end detection without nonmaximum suppression (NMS-free).

[0016] During implementation, real-time detection of student objectives can be performed on classroom teaching videos collected using edge devices.

[0017] The target student detected in the video to be identified (surveillance video stream) is the student detected according to the preset YOLOv10 target detection model, wherein the YOLOv10 target detection model is an end-to-end non-maximum suppression (NMS-free) model trained based on video frames and student detection results.

[0018] In some embodiments, preprocessing each frame of image includes: using an adaptive scaling algorithm to adjust input images of different resolutions to a uniform size required by the network while maintaining the original aspect ratio to avoid target distortion; normalizing the image pixel data by subtracting the channel mean and dividing by the standard deviation to accelerate network convergence; and finally performing color space conversion to construct a standardized data stream that meets the requirements of the network input tensor.

[0019] In some embodiments, the YOLOv10 network structure includes: a backbone network employing an enhanced CSPNet structure, introducing Compressed Index Blocks (CIBs) and Polarized Self-Attention (PSA) modules for extracting image features; a neck network using a Path Aggregation Network (PANet) structure to achieve multi-scale feature fusion; and a head network employing a decoupled head design and trained based on a consistent dual allocation strategy, enabling only one-to-one matching branches during the inference phase to directly output target bounding boxes. The Compressed Index Block (CIB) is an efficient basic network building block that improves the model's computational efficiency and performance. Depthwise separable convolutions are used for spatial mixing, and pointwise convolutions are used for channel mixing, thereby reducing computational costs while maintaining competitiveness. PSA is a self-attention module optimized for pixel-level regression tasks (such as semantic segmentation, pose estimation, and object detection). Its core design revolves around a dual-polarization filtering architecture and fine-grained nonlocal operations, aiming to address the information loss and computational efficiency issues of traditional self-attention in pixel-level tasks.

[0020] Step S2: Use Module 2 (Multi-target tracking algorithm) to perform ID association on student targets in consecutive video frames to generate video sequence segments containing the same student ID; here, the multi-target tracking algorithm can be the DeepSORT algorithm, which uses Kalman filtering to predict the target state and combines ReID features for data association.

[0021] The real-time target tracking module (Module 2) constructs and maintains a stable identity association system across video frames based on the student bounding boxes output by the target detection module for each frame. The DeepSORT algorithm is used as the core technical framework, and its workflow includes the following state prediction, feature extraction, and data association: (1) State prediction: State prediction is performed using Kalman filtering. A Kalman filter is maintained for each tracking track. The state vector is defined as 8-dimensional: , where (u, v) are the coordinates of the center point of the bounding box, a is the aspect ratio, and h is the height. This corresponds to the velocity. In each frame, the Kalman filter first predicts the bounding box (Bbox) position of the trajectory in the current frame.

[0022] (2) Appearance feature extraction: A 128-dimensional appearance feature vector is extracted from the image patch cropped from each detection box (bounding box) output in step S1 using the lightweight deep convolutional neural network MobileNetV2. This vector will be input into the cost matrix calculation formula of the data association part to calculate the appearance cost.

[0023] (3) Data association: The Hungarian algorithm is used for global optimal matching. The core is to construct a cost matrix that quantifies the degree of matching between the predicted trajectory and the current detection box. In order to keep the ID stable in the scenarios of dense occlusion and student pose changes, the cost matrix adopts a weighted fusion calculation method of motion features and appearance features. The total cost formula (1) is as follows: (1); where, For the price of sports, For the sake of appearance, These are the weighting coefficients.

[0024] During implementation, Mahalanobis distance is first used to calculate the motion cost. This distance measures the statistical deviation between the prior position predicted by the Kalman filter and the observed position detected by YOLO in real time. It is used to exclude logically impossible correlations (such as abrupt changes in position) and ensure a smooth trajectory. Appearance Cost The cosine distance calculation is then used. The system maintains a historical feature database for each trajectory, calculates the cosine distance between the Re-ID feature vector of the current detection box and all feature vectors in the historical database of that trajectory, and takes the minimum value as the final cost. The purpose of introducing the "minimum" distance mechanism is to adapt to changes in posture caused by students turning their heads or turning to the side in the classroom. As long as the current viewpoint is similar to any viewpoint that the student has ever appeared from in the past, it can be determined to be the same identifier (ID), thereby effectively preventing tracking loss due to changes in posture.

[0025] Step S3: Input the video sequence segment into module three (lightweight human pose estimation network), extract the student's human key points in each frame image, and generate a key point heatmap sequence and key point visibility confidence. Here, the lightweight human pose estimation network can be an improved EL-HRNet, which includes the lightweight attention basic block (LA-Basicblock) shown in Figure 2. As shown in Figure 2, the LA-Basicblock includes a lightweight convolutional unit and an occlusion-aware attention module (OAA) connected in sequence. The lightweight convolutional unit includes a cascaded 1x1 dimensionality reduction convolution and a 3x3 grouped convolution. The occlusion-aware attention module (OAA) is used to generate channel and spatial attention weights based on key point features and output a key point visibility mask to suppress the feature response of the occluded part.

[0026] In the implementation process, the human keypoint detection module (Module 3) is used to extract key nodes of human pose from the N consecutive frame image sequences corresponding to each tracked student ID. This embodiment uses a lightweight high-resolution network (EL-HRNet) as the backbone network to locate several human keypoints in each frame image. Specifically, this embodiment uses 17 keypoints defined in the COCO dataset as an example for illustration. As an optional solution to the above implementation, EL-HRNet includes a lightweight attention base module (L-Basicblock) as shown in Figure 2. The input video frame image is first processed by the initial convolutional layer of the EL-HRNet network front end, mapping the original RGB pixel data into a high-dimensional feature map with more channels, which serves as the input to subsequent network modules.

[0027] L-Basicblock replaces the standard 3x3 convolution with cascaded dimensionality-reducing convolutions and grouped convolutions, improving upon it as follows: First, 1x1 convolutions are used to halve the number of input channels for feature compression and dimensionality reduction; then, 3x3 grouped convolutions are used to extract spatial features from the compressed feature map. Furthermore, to address the frequent occlusion issues in classroom scenarios, and to compensate for the information loss caused by lightweight design and enhance anti-occlusion capabilities, an occlusion-aware attention module (OAA) is cascaded after L-Basicblock, forming LA-Basicblock (refer to Figure 2). The OAA module, while extracting channel and spatial features, can simultaneously evaluate the response intensity of feature points to generate visibility confidence scores for keypoints.

[0028] The aforementioned LA-Basicblock, as a basic unit, is stacked in the four feature extraction stages (Stage 1 to Stage 4) of the EL-HRNet network. While maintaining a single high-resolution main branch, the EL-HRNet network generates multiple low-resolution branches (1 / 2, 1 / 4, and 1 / 8 resolution) in parallel. Branches of different resolutions frequently interact through multi-scale fusion; low-resolution features are upsampled and fused with high-resolution features, while high-resolution features are downsampled to enhance semantic information. This ensures that at the end of the network, the highest-resolution branch (the main branch output from Stage 4) contains both accurate geometric location information and deep semantic features. After four stages of processing, the network output contains only a 32-channel spatiotemporal feature map. To obtain the final K keypoint heatmaps, the system uses a 1×1 convolutional layer as the regression head. This convolutional layer maps the number of channels in the high-dimensional feature map to K, outputting a tensor of size W×H×K. Each channel in the output tensor corresponds to a probability distribution map (i.e., a Gaussian heatmap) of a specific human keypoint (such as the left elbow or right knee). The position with the highest pixel value in the heatmap represents the predicted coordinate of that keypoint, and the magnitude of this peak directly characterizes the visibility confidence of that keypoint. Finally, the improved EL-HRNet outputs two sets of data: one set consists of K (K=17) Gaussian heatmaps, corresponding to the probability distribution of human keypoints; the other set consists of the corresponding visibility confidence scores, used to suppress weight interference from occluded keypoints in subsequent steps.

[0029] Step S4: Construct a dual-stream multimodal input module four (classroom behavior recognition network) containing RGB image data and key point heatmap sequences; input the RGB image data into the backbone of the deep spatiotemporal residual network, and input the key point heatmap sequences into the key point guided spatial attention module (KSAM) embedded in the network to perform cross-modal feature extraction and classification, and output the classroom behavior recognition results.

[0030] Here, the four processing steps of the dual-stream multimodal input module include: adjusting the RGB images in the video sequence segments to the network's preset resolution, and using them as the spatiotemporal feature extraction input for the backbone network; and performing multi-scale downsampling on the keypoint heatmap sequence to make its spatial resolution consistent with the resolution of the intermediate layer feature maps of different levels in the deep spatiotemporal residual network, and using it as the side-guided input for the KSAM module.

[0031] Figure 3 is a schematic diagram of a classroom behavior recognition network provided in an embodiment of this application. As shown in Figure 3, the classroom behavior recognition network includes: a deep spatiotemporal residual network 31 and a key point guided spatial attention module (KSAM) 32.

[0032] Among them, the deep spatiotemporal residual network 31 adopts a decomposed (2+1)D convolution structure and integrates a fast Fourier convolution (FFC) residual block; the FFC residual block contains parallel local temporal branch and global frequency domain branch, wherein the global frequency domain branch extracts long-term behavioral features through fast Fourier transform.

[0033] The processing steps of the Key Point Guided Spatial Attention Module (KSAM) 32 include: receiving a downsampled key point heatmap sequence and a network intermediate layer feature map (third feature map) as dual inputs; generating a spatial attention mask based on the spatial distribution of the key point heatmap, which has high weighted responses in key areas of the human hand and head; multiplying the spatial attention mask element-wise with the network intermediate layer feature map (third feature map) to obtain a weighted feature map (fourth feature map); adding the weighted feature map element-wise with the original network intermediate layer feature map (third feature map) to retain background context information, and then inputting the added feature (fifth feature map) into a 3D convolutional layer for channel integration, and outputting enhanced spatiotemporal features through a nonlinear activation function.

[0034] Step S4 is the core processing step of this invention. The classroom behavior recognition module (refer to Figure 3) constructs a key point-guided deep spatiotemporal residual network. By fusing the visual texture features of the RGB video stream with the geometric structure features of the human key point stream, it achieves accurate identification of abnormal behavior in complex classroom scenarios. The process is specifically divided into the following four sub-steps: Step S41, Construction and spatiotemporal alignment of multimodal dual-stream data; The system first constructs two parallel data input streams. The RGB visual stream (main data stream) extracts the N consecutive frame image sequence (N=32) of the target tracking module outputting a specific student ID, and adjusts it to a uniform resolution (224×224) as the main input for capturing appearance texture and environmental background. The key point heatmap stream (guided data stream) corresponds to the N frame human key point heatmap sequence of the RGB sequence. In order to eliminate the scale difference caused by the different positions of students in the picture, the system normalizes the key point coordinates before input. The normalization formulas (2) and (3) are as follows:

[0035]

[0036] in, Original coordinates These are normalized coordinates.

[0037] The normalized coordinates are remapped to generate a standard-sized Gaussian heatmap sequence. To adapt to feature map sizes of different depths in the deep network, the system pre-samples the keypoint heatmap sequence at multiple scales to construct a heatmap pyramid, corresponding to different resolution levels in the network, such as 1 / 4, 1 / 8, and 1 / 16, so as to achieve accurate spatial alignment in the subsequent KSAM module.

[0038] Step S42: Spatiotemporal feature extraction based on FFC decomposition convolution; the RGB visual stream is fed into a deep spatiotemporal backbone network (3D-FFC-ResNet). To address the problem of excessive computational cost and difficulty in capturing long-term temporal dependencies in traditional 3D convolution, the network structure features the following innovative design: the network adopts a decompositional (2+1)D structure, decomposing the standard 3D convolution into concatenated 2D spatial convolutions and 1D temporal convolutions. Simultaneously, the residual blocks in the network are replaced with Fast Fourier Convolution (FFC) residual blocks. The FFC residual blocks divide the input feature map into two paths in the channel: the first path is a local temporal branch, capturing local motion through standard (2+1)D convolution; the second path is a global frequency domain branch, transforming the temporal features to the frequency domain for convolution through Fast Fourier Transform, and then transforming them back to the temporal domain through inverse FFT to capture long-term global behavioral patterns.

[0039] Step S43: Keypoint-Guided Spatial Attention Mechanism (KSAM); To effectively utilize human pose information to guide visual feature extraction, a keypoint-guided spatial attention module (KSAM) is embedded in the deep spatiotemporal backbone network. In this embodiment, this module is placed after the shallow feature extraction stage of the backbone network (i.e., between Block 1 and Block 2) to suppress background noise and focus on key human regions before semantic feature abstraction. Its processing flow is as follows: The module simultaneously receives two inputs, one of which is the intermediate layer feature map output by the backbone network. The other approach is a key point heatmap at the same scale. Heatmap of key input points Convolutional encoding and sigmoid activation are performed. This mask has high response values ​​at key locations such as the hand and head, while approaching 0 in the background region. A spatial attention mask (M) is generated. This mask has high response values ​​at key anatomical locations such as the hand (features of using a mobile phone) and the head (features of sleeping), while approaching 0 in the background region.

[0040] Compare the generated mask M with the RGB feature map Perform element-wise multiplication to obtain the weighted feature map. This step aims to suppress background noise unrelated to human movement, such as desks and books, forcing the network to focus on the human limb area. Retaining only foreground features leads to the loss of environmental context information (such as whether the student is lying on the "desk") and easily causes gradient vanishing in deep networks. Therefore, this invention introduces a residual reconstruction mechanism to reconstruct the weighted feature map... Compared with the original input feature map To perform element-wise addition, that is... The fused features undergo a process... The 3D convolutional layer integrates information between channels, and after processing with BatchNorm normalization and ReLU activation function, the enhanced spatiotemporal features are passed to the next layer.

[0041] Step S44: Global Pooling and Behavior Classification; After processing through multiple layers of FFC residual blocks and KSAM modules, the network outputs a high-dimensional spatiotemporal feature tensor. The system uses 3D global average pooling to compress the feature tensor in both spatial and temporal dimensions, resulting in a one-dimensional feature vector. This vector is input into a fully connected layer, first passing through a hidden layer (FC1) containing 512 neurons, and finally being mapped to a preset behavior category (e.g., playing on a mobile phone, sleeping, raising a hand, etc.) through a classification layer (FC2). The probability distribution of each behavior category is calculated using the Softmax function, and the category with the highest probability is output as the final recognition result. During the training phase, the cross-entropy loss function is used to calculate the difference between the predicted value and the true label, and the network parameters are updated through backpropagation.

[0042] In this embodiment, YOLOv10 is used to achieve efficient detection without non-maximum suppression (NMS), combined with occlusion-aware pose extraction using an improved EL-HRNet, and finally a two-stream network is constructed using 3D-FFC-ResNet. Through the "guided-weighted-reconstruction" mechanism of the KSAM module, dynamic calibration of visual features based on prior knowledge of key points is achieved. While preserving environmental background information, this significantly improves the recognition accuracy and system robustness for subtle abnormal behaviors such as "playing on a mobile phone" and "sleeping."

[0043] This application provides a method for identifying abnormal classroom behavior by integrating human key points and spatiotemporal features, as shown in Figure 4. This method can be implemented through the following steps: Step S410: Detect target students in a monitoring video stream using an object detection network, and determine the first bounding box of the target student. Here, a real-time classroom monitoring video stream can be acquired, and the target student in the video frame can be located using the lightweight object detection network YOLOv10, outputting the first bounding box. Here, the first bounding box is the identified target bounding box that includes the target student.

[0044] The YOLOv10 object detection model is an end-to-end non-nonmaximum suppression (NMS-free) model trained based on video frames and student detection results. The structure of the YOLOv10 network includes: Backbone: adopts an enhanced CSPNet structure, introducing Compressed Index Block (CIB) and part of Polarized Self-Attention (PSA) modules for extracting image features; Neck: adopts a Path Aggregation Network (PANet) structure to achieve multi-scale feature fusion; Head: adopts a decoupled head design and is trained based on a consistent dual allocation strategy, enabling only one-to-one matching branches during the inference phase to directly output the target bounding box.

[0045] Step S420: Using a multi-target tracking algorithm, the target student is identified and associated based on the first bounding box to generate a video sequence segment containing the identified object. Here, the multi-target tracking algorithm can be the DeepSORT algorithm. The DeepSORT algorithm is a deep learning-based multi-target tracking algorithm that introduces a deep learning feature extractor on the basis of the SORT algorithm. By combining motion information and appearance features for data association, it improves the accuracy and robustness of tracking.

[0046] After object detection, DeepSORT uses deep neural networks (such as CNNs) to extract appearance features (ReID) for each detected object. These features are used to describe the appearance of the object, enabling the same object to be identified in different frames.

[0047] DeepSORT uses Kalman filters to predict the target's motion state (position and velocity) and update the target's state. The Kalman filter predicts the target state in the current frame based on the observations of the current frame and the state estimate of the previous frame, and updates these states when new observations arrive, achieving smooth tracking of moving targets.

[0048] In each frame, DeepSORT performs data association by combining motion information and target features. The Hungarian algorithm is used to solve the optimal matching problem between the target and the detection box, and the cost matrix is ​​constructed based on the similarity of distance and appearance features. DeepSORT combines two metrics, Mahalanobis distance and cosine distance, to integrate motion and appearance information. Mahalanobis distance, based on motion information, provides information about the possible location of the target, which is effective for short-term prediction; while cosine distance considers appearance information, which is effective for retrieving the ID of targets that have been occluded for a long time.

[0049] Step S430: Input the video sequence segment into the human pose estimation network, extract the human key points of the target student in each frame image, and generate a key point heatmap sequence; here, the human pose estimation network can be an improved EL-HRNet, which includes the lightweight attention basic block (LA-Basicblock) shown in Figure 2.

[0050] In the implementation process, a human pose estimation network is used to extract key human pose nodes from a continuous N-frame image sequence corresponding to each tracked student ID. This embodiment uses a lightweight high-resolution network (EL-HRNet) as the backbone network to locate several human key points in each frame. Finally, the improved EL-HRNet outputs two sets of data: one set is K (K=17) Gaussian heatmaps, corresponding to the probability distribution of human key points; the other set is the corresponding visibility confidence score, which is used to suppress weight interference from occluded key points in subsequent steps.

[0051] Step S440: Input the video sequence segment and the key point heatmap sequence into the classroom behavior recognition network for behavior recognition to obtain the classroom abnormal behavior recognition result. The classroom behavior recognition network is constructed based on the key point-guided spatial attention mechanism.

[0052] In the implementation process, a dual-stream multimodal input module four (classroom behavior recognition network) was constructed, containing RGB image data and keypoint heatmap sequences. As shown in Figure 3, RGB image data is input into the backbone of the deep spatiotemporal residual network, and the keypoint heatmap sequences are input into the Keypoint Guided Spatial Attention (KSAM) module embedded in the network for cross-modal feature extraction and classification, outputting classroom behavior recognition results. The KSAM module can fuse RGB image data and keypoint heatmap sequences, alleviating the problem of uncertain knowledge alignment and improving the accuracy and efficiency of classroom behavior recognition.

[0053] In this embodiment, a classroom behavior recognition network is constructed based on a key-point guided spatial attention mechanism. This network combines video sequence segments and key-point heatmap sequences for recognition, thereby achieving dynamic calibration of visual features by prior knowledge of key points. This significantly improves the recognition accuracy and system robustness for subtle abnormal behaviors such as "playing on a mobile phone" and "sleeping".

[0054] In some embodiments, the step S420 above, "identifying and associating the target student based on the first bounding box using a multi-object tracking algorithm," can be implemented through the following steps: Step 421, predicting the second bounding box of the target student in the image using a Kalman filter to obtain the image patch corresponding to the target student; here, the multi-object tracking algorithm can be Deep Simple Online and Realtime Tracking (DeepSORT), a classic multi-object tracking (MOT) algorithm. Its core idea is to combine Kalman filtering for target state prediction and utilize re-identification (ReID) features extracted by deep learning to achieve efficient data association. The re-identification features can solve the identity matching problem when there is long-term occlusion or changes in target appearance.

[0055] During implementation, Kalman filtering is used for state prediction. A Kalman filter is maintained for each tracking track. The state vector is defined as 8-dimensional: , where (u, v) are the coordinates of the center point of the bounding box, a is the aspect ratio, and h is the height. This corresponds to the velocity. In each frame, the Kalman filter first predicts the location of the bounding box (Bbox) of the trajectory in the current frame, which is the second bounding box.

[0056] Step 422: Perform convolution processing on the image patch of the first bounding box to obtain the appearance feature vector corresponding to the image patch. In the implementation process, a lightweight deep convolutional neural network MobileNetV2 is used to extract a 128-dimensional appearance feature vector from the image patch cropped from each detection box (first bounding box) output in step S410. This vector will be input into the cost matrix calculation formula of the data association part to calculate the appearance cost.

[0057] Step 423: Identify and associate the target student based on the first bounding box, the second bounding box, and the appearance feature vector.

[0058] During implementation, the error between the first and second bounding boxes can be statistically analyzed, and the appearance feature vector of the target can be extracted using the ReID model for identity matching in long-term tracking.

[0059] Then, based on the error and identity matching results, the target student is identified and associated.

[0060] In this embodiment, by combining motion prediction using Kalman filtering and appearance features of ReID, a good balance is achieved between the real-time performance and accuracy of identity identification, which can effectively improve the accuracy of identification.

[0061] In some embodiments, step 423 above, "identifying and associating the target student based on the first bounding box, the second bounding box, and the appearance feature vector," can be implemented through the following steps: Step 4231, using Mahalanobis distance to determine the first distance between the first bounding box and the second bounding box as the motion cost; the Hungarian algorithm can be used for global optimal matching, the core of which is to construct a cost matrix that quantifies the degree of matching between the predicted trajectory and the current detection box. In order to maintain ID stability in scenarios with dense occlusion and changes in student posture, the cost matrix adopts a weighted fusion calculation method of motion features and appearance features, and the total cost formula (1) is as above. For the price of sports, For the sake of appearance, These are the weighting coefficients.

[0062] During implementation, Mahalanobis distance is first used to calculate the motion cost. This distance measures the statistical deviation between the prior position predicted by the Kalman filter (second bounding box) and the observed position detected by YOLO in real time (first bounding box), and is used to exclude logically impossible associations (such as sudden changes in position) to ensure smooth trajectory.

[0063] Step 4232: Obtain the set of historical feature vectors stored based on the identifiers of the target students; here, the system can maintain a historical feature library of the trajectory corresponding to each student identifier, and obtain the set of historical feature vectors of the target students from the historical feature library.

[0064] Step 4233: Calculate the cosine distance between the appearance feature vector and all feature vectors in the historical feature vector set using cosine distance, and determine the minimum cosine distance as the appearance cost; in the implementation process, the appearance cost... The cosine distance is used for calculation. The system maintains a historical feature database for each trajectory, calculates the cosine distance between the Re-ID feature vector (appearance feature vector) of the current detection box and all feature vectors in the historical database of that trajectory, and takes the minimum value as the final cost. The purpose of introducing the "minimum" distance mechanism is to adapt to changes in posture caused by students turning their heads or turning to the side in the classroom. As long as the current viewpoint is similar to any viewpoint that the student has ever appeared from in the past, it can be determined to be the same identifier (ID), thereby effectively preventing tracking loss due to changes in posture.

[0065] Step 4234: The motion cost and the appearance cost are weighted and summed to obtain the total cost; during implementation, weight parameters can be set according to requirements. Using the formula (1) above, the total cost is calculated. For example, the weights are typically set to 0.4 to 0.6 to balance motion and appearance information.

[0066] Step 4235: Identify and associate the target student based on the total cost.

[0067] During implementation, the similarity between the detection box and the trajectory can be quantified by the total cost. By combining cascaded matching and IOU matching for hierarchical processing, the complementarity of motion and appearance information can be achieved.

[0068] In this embodiment, the total cost is a weighted fusion of motion cost and appearance cost, used to quantify the matching similarity between the bounding box and the trajectory. By minimizing the total cost, the algorithm achieves the optimal association between the bounding box and the trajectory. This significantly improves tracking accuracy in complex scenes while maintaining real-time performance.

[0069] In some embodiments, the human pose estimation network includes at least a first convolutional layer, an occlusion-aware attention layer, and a second convolutional layer. Step S430, "Inputting the video sequence segment into the human pose estimation network, extracting the human key points of the target student in each frame image, and generating a key point heatmap sequence," can be implemented through the following steps: Step 431, inputting the video sequence segment into the first convolutional layer to obtain a first feature map. Here, the lightweight human pose estimation network can be an improved EL-HRNet, which includes a lightweight attention basic block (LA-Basicblock) as shown in Figure 2. As shown in Figure 2, the LA-Basicblock includes a lightweight convolutional unit and an occlusion-aware attention module (OAA) connected sequentially.

[0070] During implementation, video sequence segments can be input into the first convolutional layer for convolution processing to obtain the first feature map.

[0071] Step 432: Input the first feature map into the occlusion-aware attention layer to extract the channel and spatial features of the first feature map and obtain a high-dimensional feature map; here, the occlusion-aware attention module (OAA) is used to generate channel and spatial attention weights based on key point features and output the visibility mask of key points to suppress the feature response of the occluded part.

[0072] During implementation, the first feature map can be input into the Occlusion Awareness Attention (OAA) layer to extract the channel and spatial features of the first feature map, and the original RGB pixel data can be mapped into a high-dimensional feature map with more channels, which can then be used as the input for subsequent network modules.

[0073] Step 433: Input the high-dimensional feature map into the second convolutional layer to obtain the first feature tensor, and generate the key point heatmap sequence based on the first feature tensor.

[0074] In the implementation process, to obtain the final K keypoint heatmaps, the system uses a 1×1 convolutional layer as the regression head. This convolutional layer maps the number of channels in the high-dimensional feature map to K, outputting a tensor of size W×H×K (the first feature tensor). Each channel in the output tensor corresponds to a probability distribution map (i.e., a Gaussian heatmap) of a specific human keypoint (such as the left elbow or right knee). The position with the highest pixel value in the heatmap represents the predicted coordinates of that keypoint, and the magnitude of this peak directly characterizes the visibility confidence of that keypoint.

[0075] In some embodiments, the first feature map is input into the occlusion-aware attention layer to evaluate the response intensity of each feature point in the first feature map, and the response intensity is used to generate the visibility confidence of the key point.

[0076] Here, the Occlusion Aware Attention (OAA) module can be used to generate channel and spatial attention weights based on keypoint features and output a visibility mask for keypoints to suppress the feature responses of occluded areas. While extracting channel and spatial features, the OAA module can also evaluate the response intensity of feature points in parallel to generate the visibility confidence of keypoints.

[0077] Finally, the improved EL-HRNet outputs two sets of data: one set consists of K (K=17) Gaussian heatmaps, corresponding to the probability distribution of human keypoints; the other set consists of the corresponding visibility confidence scores, which are used to suppress weight interference from occluded keypoints in subsequent steps.

[0078] In this embodiment of the application, in order to compensate for the information loss caused by lightweighting and enhance the anti-occlusion ability, an occlusion perception attention layer is connected in series after L-Basicblock to improve the accuracy of generating effective key point heatmap sequences in classroom scenarios with frequent table and chair occlusion.

[0079] In some embodiments, the first convolutional layer includes a 1×1 convolutional sub-layer and a 3×3 convolutional sub-layer connected in sequence; step 431 above, "inputting the video sequence segment into the first convolutional layer to obtain a first feature map," can be implemented through the following steps: Step 4311, inputting the video sequence segment into the 1×1 convolutional sub-layer for feature compression and dimensionality reduction processing to obtain a second feature map; here, the lightweight convolutional unit (first convolutional layer) includes 1x1 dimensionality-reduced convolution and 3x3 grouped convolution. Specifically, inputting the video sequence segment into the 1×1 convolutional sub-layer for feature compression and dimensionality reduction processing to obtain the second feature map.

[0080] Step 4312: Input the second feature map into the 3×3 convolutional sub-layer for spatial feature extraction to obtain the first feature map.

[0081] During implementation, 3x3 grouped convolutions are used to extract spatial features on the compressed second feature map.

[0082] In this embodiment, the standard 3x3 convolution is replaced with cascaded dimensionality-reducing convolution and grouped convolution, improved as follows: First, a 1x1 convolution is used to halve the number of input channels for feature compression and dimensionality reduction; then, 3x3 grouped convolution is used to extract spatial features from the compressed feature map. This achieves both real-time processing requirements and model lightweighting.

[0083] In some embodiments, the behavior recognition network includes a preprocessing module, a deep spatiotemporal backbone network, a keypoint-guided spatial attention layer, and a global pooling layer. Step S440 above, "inputting the video sequence fragments and the keypoint heatmap sequence into the classroom behavior recognition network for behavior recognition to obtain classroom abnormal behavior recognition results," can be achieved through the following steps: Step 441, adjusting the resolution of the video sequence fragments through the preprocessing module to obtain target video sequence fragments that meet a preset resolution. During implementation, the RGB images in the video sequence fragments can be adjusted to a preset resolution (network preset resolution) as the spatiotemporal feature extraction output of the backbone network. The RGB visual stream (main data stream) extracts a continuous N-frame image sequence (N=32) belonging to a specific student ID from the target tracking module output, and adjusts it to a uniform resolution (224×224) as the main input for capturing appearance texture and environmental background.

[0084] Step 442: Input the target video sequence segment into a deep spatiotemporal backbone network for convolution and Fourier transform to obtain a third feature map representing behavioral features. To address the problem of excessive computational cost and difficulty in capturing long-term temporal dependencies in traditional 3D convolution, the network structure is innovatively designed as follows: The network adopts a decomposed (2+1)D structure, decomposing the standard 3D convolution into concatenated 2D spatial convolution and 1D temporal convolution. Simultaneously, the residual blocks in the network are replaced with Fast Fourier Convolution (FFC) residual blocks. The FFC residual blocks divide the input feature map into two paths in the channel: the first path is a local temporal branch, capturing local motion through standard (2+1)D convolution; the second path is a global frequency domain branch, transforming the temporal features to the frequency domain for convolution through Fast Fourier Transform, and then converting them back to the temporal domain through inverse FFT to capture long-term global behavioral patterns.

[0085] Step 443: Downsample the key point heatmap sequence to obtain a target key point heatmap sequence spatially aligned with the target video sequence segment; during implementation, the key point heatmap sequence is downsampled at multiple scales to ensure that its spatial resolution is consistent with the resolution of intermediate layer feature maps at different levels in the deep spatiotemporal residual network, serving as the side-guided input of the KSAM module.

[0086] Step 444: Input the third feature map and the target key point heatmap sequence into the key point-guided spatial attention layer to process the third feature map based on the key points and obtain a second feature tensor with enhanced spatiotemporal features; The processing of the key point-guided spatial attention layer KSAM includes: receiving the downsampled target key point heatmap sequence and the network intermediate layer feature map (third feature map) as dual inputs; processing the third feature map based on the key points obtained from the target key point heatmap to obtain a second feature tensor with enhanced spatiotemporal features.

[0087] Step 445: Input the second feature tensor into the global pooling layer to obtain the classroom abnormal behavior recognition result.

[0088] After processing by multiple layers of FFC residual blocks and KSAM modules, the network outputs a high-dimensional spatiotemporal feature tensor (second feature tensor). The system uses 3D global average pooling to process this second feature tensor to obtain the classroom abnormal behavior recognition results.

[0089] In this embodiment, the "guided-weighted-reconstruction" mechanism of the KSAM module enables dynamic calibration of visual features based on prior knowledge of key points. While preserving environmental background information, it significantly improves the recognition accuracy and system robustness of subtle abnormal behaviors such as "playing on a mobile phone" and "sleeping".

[0090] In some embodiments, step 444 above, "inputting the third feature map and the target keypoint heatmap sequence into the keypoint-guided spatial attention layer to process the third feature map based on keypoints and obtain a second feature tensor with enhanced spatiotemporal features," can be implemented through the following steps: Step 4441, using the keypoint-guided spatial attention layer to perform convolution and activation processing on the target keypoint heatmap sequence to generate a spatial attention mask; during implementation, the input keypoint heatmap... Perform convolutional encoding and sigmoid activation. Based on the keypoint heatmap... The spatial distribution generates a spatial attention mask, which has a high weighted response in key areas of the human hand and head.

[0091] Step 4442: Multiply the spatial attention mask element-wise with the third feature map to obtain a weighted fourth feature map; in the implementation process, multiply the spatial attention mask element-wise with the network intermediate layer feature map (third feature map) to obtain a weighted feature map (fourth feature map).

[0092] Step 4443: Add the fourth feature map to the third feature map element by element to obtain the fifth feature map of feature fusion; in the process, add the weighted feature map to the original network intermediate layer feature map (third feature map) element by element to retain the background context information to obtain the fifth feature map of feature fusion.

[0093] Step 4444: Perform information integration, normalization and activation processing on the fifth feature map in sequence to obtain the second feature vector.

[0094] During implementation, the summed features (the fifth feature map) are input into a 3D convolutional layer for channel integration, and after normalization and nonlinear activation function processing, the enhanced spatiotemporal features, namely the second feature tensor, are output.

[0095] In this embodiment, the spatial attention layer receives two inputs simultaneously: one is the intermediate layer feature map output by the backbone network. The other approach is a key point heatmap at the same scale. Heatmap of key input points Convolutional encoding and sigmoid activation are performed. This mask has high response values ​​at key locations such as the hand and head, while approaching 0 in the background region. A spatial attention mask (M) is then generated. This mask has high response values ​​at key anatomical locations such as the hand (features of using a mobile phone) and the head (features of sleeping), while approaching 0 in the background region. This effectively improves the accuracy of action recognition in key regions.

[0096] In some embodiments, step 445 above, "inputting the second feature tensor into the global pooling layer to obtain the classroom abnormal behavior recognition result," can be implemented through the following steps: Step 4451, compressing the second feature tensor in the spatial and temporal dimensions using the global pooling layer to obtain a one-dimensional feature vector; after processing by multiple layers of FFC residual blocks and KSAM modules, the network outputs a high-dimensional spatiotemporal feature tensor (the second feature tensor). The system uses 3D global average pooling to compress the feature tensor in the spatial and temporal dimensions to obtain a one-dimensional feature vector.

[0097] Step 4452: Input the one-dimensional feature vector into a fully connected layer classifier for classification to obtain the classroom abnormal behavior recognition result.

[0098] In the implementation process, one-dimensional features are input into a fully connected layer, first passing through a hidden layer (FC1) containing 512 neurons, and finally being mapped to a preset behavior category (such as playing on a mobile phone, sleeping, raising a hand, etc.) through a classification layer (FC2). The probability distribution of each behavior category is calculated using the Softmax function, and the category with the highest probability is output as the final recognition result.

[0099] In this embodiment, compressing the high-dimensional spatiotemporal feature tensor into a one-dimensional vector significantly reduces the number of parameters and lowers the risk of overfitting. Mapping the compressed feature vector to a more abstract semantic space makes the feature distribution of similar behaviors (such as different postures of "playing on a mobile phone") more compact and the dissimilar behaviors (such as "playing on a mobile phone" and "sleeping") more distinct. This enhances the model's ability to distinguish complex behaviors (such as distinguishing between "looking down at a mobile phone" and "looking down at a book").

[0100] This invention provides a classroom abnormal behavior recognition device that integrates human key points and spatiotemporal features. Referring to Figure 5, the device 500 includes: a detection module 510, used to detect target students in a monitoring video stream using a target detection network and determine the first bounding box of the target student; an association module 520, used to use a multi-target tracking algorithm to identify and associate the target student based on the first bounding box, generating a video sequence segment containing the identification; an extraction module 530, used to input the video sequence segment into a human pose estimation network to extract the human key points of the target student in each frame image, generating a key point heatmap sequence; and a recognition module 540, used to input the video sequence segment and the key point heatmap sequence into a classroom behavior recognition network for behavior recognition, obtaining a classroom abnormal behavior recognition result. The classroom behavior recognition network is constructed based on a key point-guided spatial attention mechanism.

[0101] In some embodiments, the association module 520 includes a prediction submodule, a convolution submodule, and an association submodule, wherein the prediction submodule is used to predict the second bounding box of the target student in the image using a Kalman filter; the convolution submodule is used to perform convolution processing on the image patch of the first bounding box to obtain the appearance feature vector corresponding to the image patch; and the association submodule is used to identify and associate the target student based on the first bounding box, the second bounding box, and the appearance feature vector.

[0102] In some embodiments, the association submodule includes a first determining unit, an acquiring unit, a second determining unit, a weighted summation unit, and an association unit, wherein the first determining unit is used to determine the first distance between the first bounding box and the second bounding box as the motion cost using Mahalanobis distance; the acquiring unit is used to acquire a set of historical feature vectors stored based on the identifier of the target student; the second determining unit is used to calculate the cosine distance between the appearance feature vector and all feature vectors in the historical feature vector set using cosine distance, so as to determine the minimum cosine distance as the appearance cost; the weighted summation unit is used to perform a weighted summation of the motion cost and the appearance cost to obtain the total cost; and the association unit is used to perform identifier association on the target student based on the total cost.

[0103] In some embodiments, the human pose estimation network includes at least a first convolutional layer, an occlusion-aware attention layer, and a second convolutional layer; the extraction module 530 includes a first convolutional submodule, an extraction submodule, and a second convolutional submodule, wherein the first convolutional submodule is used to input the video sequence segment into the convolutional layer to obtain a first feature map; the extraction submodule is used to input the first feature map into the occlusion-aware attention layer to extract the channel and spatial features of the first feature map to obtain a high-dimensional feature map; the second convolutional submodule is used to input the high-dimensional feature map into the second convolutional layer to obtain a first feature tensor to generate the keypoint heatmap sequence based on the first feature tensor.

[0104] In some embodiments, the extraction module 530 further includes an evaluation submodule, which is used to input the first feature map into the occlusion perception attention layer and evaluate the response intensity of each feature point in the first feature map, wherein the response intensity is used to generate the visibility confidence of the key point.

[0105] In some embodiments, the first convolutional layer includes a 1×1 convolutional sub-layer and a 3×3 convolutional sub-layer connected in sequence; the first convolutional sub-module includes a first convolutional unit and a second convolutional unit, wherein the first convolutional unit is used to input the video sequence segment into the 1×1 convolutional sub-layer for feature compression and dimensionality reduction processing to obtain a second feature map; the second convolutional unit is used to input the second feature map into the 3×3 convolutional sub-layer for spatial feature extraction to obtain the first feature map.

[0106] In some embodiments, the behavior recognition network includes a preprocessing module, a deep spatiotemporal backbone network, a keypoint-guided spatial attention layer, and a global pooling layer; the recognition module 540 includes an adjustment submodule, a first processing submodule, a downsampling submodule, a second processing submodule, and a third processing submodule, wherein the adjustment submodule is used to adjust the resolution of the video sequence segment through the preprocessing module to obtain a target video sequence segment that meets a preset resolution; the first processing submodule is used to input the target video sequence segment into the deep spatiotemporal backbone network for convolution processing and Fourier transform to obtain a third feature map representing behavioral features; the downsampling submodule is used to downsample the keypoint heatmap sequence to obtain a target keypoint heatmap sequence spatially aligned with the target video sequence segment; the second processing submodule is used to input the third feature map and the target keypoint heatmap sequence into the keypoint-guided spatial attention layer to process the third feature map based on keypoints to obtain a second feature tensor that enhances spatiotemporal features; the third processing submodule is used to input the second feature tensor into the global pooling layer to obtain the classroom abnormal behavior recognition result.

[0107] In some embodiments, the second processing submodule includes a first processing unit, a multiplication unit, an addition unit, and a second processing unit. The first processing unit is configured to perform convolution and activation processing on the target keypoint heatmap sequence using the keypoint-guided spatial attention layer to generate a spatial attention mask. The multiplication unit is configured to perform element-wise multiplication of the spatial attention mask with the third feature map to obtain a weighted fourth feature map. The addition unit is configured to perform element-wise addition of the fourth feature map with the third feature map to obtain a feature-fused fifth feature map. The second processing unit is configured to sequentially perform information integration, normalization, and activation processing on the fifth feature map to obtain the second feature vector.

[0108] In some embodiments, the third processing submodule includes a compression unit and a classification unit, wherein the compression unit is used to compress the second feature tensor in the spatial and temporal dimensions using the global pooling layer to obtain a one-dimensional feature vector; the classification unit is used to input the one-dimensional feature vector into a fully connected layer classifier for classification to obtain the classroom abnormal behavior recognition result.

[0109] Figure 6 is a schematic diagram of the structure of a computer block device provided in an embodiment of the present invention. As exemplarily shown in Figure 6, the computer block device 600 includes: a memory 601, a processor 602, and a computer program 603 stored in the memory 601 and running on the processor 602. When the processor 602 executes the computer program 603, the computer block device can execute any of the aforementioned methods for identifying abnormal classroom behavior by integrating human key points and spatiotemporal features.

[0110] Furthermore, embodiments of the present invention also protect a control block device, which may include a memory and a processor. The memory stores executable program code, and the processor is used to call and execute the executable program code to perform a classroom abnormal behavior recognition method integrating human key points and spatiotemporal features provided by embodiments of the present invention. Embodiments of the present invention can divide the control block device into functional modules according to the above method examples. For example, each module may correspond to a specific function, or two or more functions may be integrated into a processing module. The integrated module can be implemented in hardware. It should be noted that the module division in embodiments of the present invention is illustrative and only represents a logical functional division; other division methods may exist in actual implementation. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced to the functional description of the corresponding functional module, and will not be repeated here. It should be understood that the control block device provided by embodiments of the present invention is used to execute the above-mentioned classroom abnormal behavior recognition method integrating human key points and spatiotemporal features, and therefore can achieve the same effect as the above-described implementation method. When using integrated units, the control block device may include a processing module and a storage module. When the control block device is applied to a block device, the processing module can be used to control and manage the actions of the block device. The storage module can be used to support the block device in executing mutual program code, etc. The processing module can be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of Digital Signal Processing (DSP) and a microprocessor, etc., and the storage module can be a memory.

[0111] Furthermore, the control block device provided in the embodiments of the present invention may specifically be a chip, component, or module. The chip may include a connected processor and a memory. The memory stores instructions, and when the processor calls and executes the instructions, the chip can execute the classroom abnormal behavior recognition method integrating human key points and spatiotemporal features provided in the above embodiments. The embodiments of the present invention also provide a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement the classroom abnormal behavior recognition method integrating human key points and spatiotemporal features provided in the above embodiments.

[0112] This invention also provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the aforementioned steps to achieve the classroom abnormal behavior recognition method that integrates human key points and spatiotemporal features provided in the above embodiments. The control block device, computer-readable storage medium, computer program product, or chip provided in this invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they achieve can be referred to in the beneficial effects of the corresponding methods provided above, and will not be repeated here. Through the description of the above embodiments, those skilled in the art can understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the control block device can be divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by this invention, it should be understood that the disclosed control block device and method can be implemented in other ways. For example, the control block device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another control block device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between control block devices or units may be electrical, mechanical, or other forms.

[0113] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multiple task processing and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. The above content is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for identifying abnormal classroom behavior by integrating key human body points and spatiotemporal features, characterized in that, The method includes: using an object detection network to detect target students in a surveillance video stream and determining a first bounding box for the target students; using a multi-object tracking algorithm to identify and associate the target students based on the first bounding box, generating a video sequence segment containing the identification; inputting the video sequence segment into a human pose estimation network to extract human key points of the target students in each frame image, generating a key point heatmap sequence; inputting the video sequence segment and the key point heatmap sequence into a classroom behavior recognition network for behavior recognition, obtaining classroom abnormal behavior recognition results, wherein the classroom behavior recognition network is constructed based on a key point-guided spatial attention mechanism.

2. The method as described in claim 1, characterized in that, The step of using a multi-target tracking algorithm to identify and associate the target student based on the first bounding box includes: using a Kalman filter to predict the second bounding box of the target student in the image; performing convolution processing on the image patch of the first bounding box to obtain the appearance feature vector corresponding to the image patch; and identifying and associating the target student based on the first bounding box, the second bounding box, and the appearance feature vector.

3. The method as described in claim 2, characterized in that, The step of identifying and associating the target student based on the first bounding box, the second bounding box, and the appearance feature vector includes: determining the first distance between the first bounding box and the second bounding box as the motion cost using Mahalanobis distance; obtaining a historical feature vector set stored based on the target student's identification; calculating the cosine distance between the appearance feature vector and all feature vectors in the historical feature vector set, and determining the minimum cosine distance as the appearance cost; performing a weighted summation of the motion cost and the appearance cost to obtain the total cost; and identifying and associating the target student based on the total cost.

4. The method as described in claim 1, characterized in that, The human pose estimation network includes at least a first convolutional layer, an occlusion-aware attention layer, and a second convolutional layer. The step of inputting the video sequence fragment into the human pose estimation network to extract the human key points of the target student in each frame and generate a key point heatmap sequence includes: inputting the video sequence fragment into the first convolutional layer to obtain a first feature map; inputting the first feature map into the occlusion-aware attention layer to extract the channel and spatial features of the first feature map to obtain a high-dimensional feature map; and inputting the high-dimensional feature map into the second convolutional layer to obtain a first feature tensor, thereby generating the key point heatmap sequence based on the first feature tensor.

5. The method as described in claim 4, characterized in that, The method further includes: inputting the first feature map into the occlusion-aware attention layer, evaluating the response intensity of each feature point in the first feature map, wherein the response intensity is used to generate the visibility confidence of the key point.

6. The method as described in claim 4, characterized in that, The first convolutional layer includes a 1×1 convolutional sub-layer and a 3×3 convolutional sub-layer connected in sequence; the step of inputting the video sequence segment into the first convolutional layer to obtain a first feature map includes: inputting the video sequence segment into the 1×1 convolutional sub-layer for feature compression and dimensionality reduction processing to obtain a second feature map; and inputting the second feature map into the 3×3 convolutional layer for spatial feature extraction to obtain the first feature map.

7. The method according to any one of claims 1 to 6, characterized in that, The behavior recognition network includes a preprocessing module, a deep spatiotemporal backbone network, a key-point guided spatial attention layer, and a global pooling layer. The step of inputting the video sequence fragments and the keypoint heatmap sequence into a classroom behavior recognition network for behavior recognition to obtain classroom abnormal behavior recognition results includes: adjusting the resolution of the video sequence fragments through a preprocessing module to obtain target video sequence fragments that meet a preset resolution; inputting the target video sequence fragments into a deep spatiotemporal backbone network for convolution processing and Fourier transform to obtain a third feature map representing behavioral features; downsampling the keypoint heatmap sequence to obtain a target keypoint heatmap sequence spatially aligned with the target video sequence fragments; inputting the third feature map and the target keypoint heatmap sequence into a keypoint-guided spatial attention layer to process the third feature map based on keypoints to obtain a second feature tensor that enhances spatiotemporal features; and inputting the second feature tensor into the global pooling layer to obtain the classroom abnormal behavior recognition results.

8. The method as described in claim 7, characterized in that, The step of inputting the third feature map and the target key point heatmap sequence into the key point-guided spatial attention layer to process the third feature map based on key points and obtain a second feature tensor with enhanced spatiotemporal features includes: using the key point-guided spatial attention layer to perform convolution and activation processing on the target key point heatmap sequence to generate a spatial attention mask; multiplying the spatial attention mask element-wise with the third feature map to obtain a weighted fourth feature map; adding the fourth feature map element-wise with the third feature map to obtain a feature-fused fifth feature map; and sequentially performing information integration, normalization, and activation processing on the fifth feature map to obtain the second feature vector.

9. The method as described in claim 7, characterized in that, The step of inputting the second feature tensor into the global pooling layer to obtain the classroom abnormal behavior recognition result includes: using the global pooling layer to compress the second feature tensor in the spatial and temporal dimensions to obtain a one-dimensional feature vector; and inputting the one-dimensional feature vector into a fully connected layer classifier for classification to obtain the classroom abnormal behavior recognition result.

10. A classroom abnormal behavior recognition device that integrates key human body points and spatiotemporal features, characterized in that, The device includes: a detection module for detecting target students in a surveillance video stream using a target detection network and determining a first bounding box of the target students; an association module for using a multi-target tracking algorithm to identify and associate the target students based on the first bounding box, generating a video sequence segment containing the identification; an extraction module for inputting the video sequence segment into a human pose estimation network to extract human key points of the target students in each frame image, generating a key point heatmap sequence; and a recognition module for inputting the video sequence segment and the key point heatmap sequence into a classroom behavior recognition network for behavior recognition, obtaining classroom abnormal behavior recognition results, wherein the classroom behavior recognition network is constructed based on a key point-guided spatial attention mechanism.

Images