Lightweight anomaly behavior detection method and system based on spatio-temporal attention distillation

Abstract

The application discloses a lightweight abnormal behavior detection method and system based on space-time attention distillation, belongs to the technical field of computer vision, deep learning and edge computing, and comprises the following steps: S1, acquiring a monitoring video stream and extracting human skeleton key points, and constructing a space-time graph matrix; S2, proportionally establishing a hidden layer mapping relationship between a deep teacher network and a lightweight student network, and obtaining a cross-layer uniform feature mapping pairing set; S3, introducing an adaptive conversion operator, and eliminating the channel dimension difference between the output feature maps of each mapping layer of the teacher and student networks; S4, extracting a space and time sequence attention graph for the dimension-aligned feature maps; S5, constructing an overall objective function, and optimizing student network parameters; and S6, deploying the trained lightweight student network to an edge computing node for real-time recognition. While maintaining the high-speed reasoning of the lightweight student network, the application can effectively retain the deep network's ability to capture local high-risk actions and key time instants.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning and edge computing, and in particular to a lightweight method and system for detecting abnormal behavior based on spatiotemporal attention distillation. Background Technology

[0002] In recent years, with the widespread adoption of security monitoring equipment and the exponential growth of video data, the focus of technology has shifted from static target detection to dynamic human behavior analysis within video sequences, i.e., abnormal behavior detection. The task of abnormal behavior detection is to analyze surveillance video frame sequences in real time to identify sets of actions with potential safety risks, such as falls or unauthorized climbing. For large-scale construction site monitoring systems, traditional deep learning techniques based on video images are often limited by the large number of model parameters and computational costs, making it difficult to achieve accurate real-time detection on front-end devices.

[0003] To reduce the impact of background interference on recognition performance, detection methods based on spatiotemporal graph convolutional networks (ST-GCN) have been introduced into the field of abnormal behavior detection. These methods extract key points of the human skeleton from video frames using pose estimation algorithms, mapping continuous human skeleton sequences into graph data that simultaneously contains spatial topological relationships and temporal evolution information. This effectively eliminates interference from complex backgrounds and improves the accuracy of behavior recognition. Skeleton-based spatiotemporal graph methods are therefore widely used for abnormal behavior recognition in video surveillance scenarios.

[0004] Based on varying network computational complexity, existing spatiotemporal graph convolutional anomaly detection methods can be broadly categorized into two types: deep, heavy-duty methods and lightweight methods. Deep, heavy-duty methods extract deep action features by stacking numerous spatiotemporal graph convolutional layers, ensuring high recognition accuracy. However, in practical deployments, such large-scale networks require significant hardware computational power and resources, making them difficult to adapt to edge devices. Lightweight methods, on the other hand, aim to reduce the number of network layers and channels to decrease the number of parameters and computational load, thus adapting to the computational capabilities of edge devices. However, excessive simplification of network parameters often leads to insufficient feature representation capabilities and a significant decrease in recognition accuracy. Against this backdrop, knowledge distillation methods have been used to address the insufficient accuracy of lightweight models. Specifically, they utilize the intermediate layer knowledge distribution of a deep teacher network with a large number of parameters to guide the training of a lightweight student network, improving the recognition accuracy of the lightweight model without increasing inference costs.

[0005] However, existing knowledge distillation methods still have the following shortcomings when applied to lightweight spatiotemporal graph convolutional networks: First, there is a significant asymmetry in the number of layers between deep teacher networks and lightweight student networks. Teacher networks typically contain dozens of spatiotemporal graph convolutional layers, while student networks retain only a few layers to meet the low computational requirements at the edge. This discrepancy in the depth and abstraction level of feature evolution leads to semantic misalignment when directly aligning intermediate feature maps, negatively impacting distillation results.

[0006] Secondly, most existing distillation methods only perform knowledge transfer from the logic output layer or a single feature layer, neglecting the spatiotemporal joint reasoning features that are crucial in abnormal behavior detection tasks. The determination of abnormal behavior depends on both the perception of spatial anomalies in specific body parts (such as limb extremities) and the capture of instantaneous temporal changes in the action. However, existing methods lack a mechanism to simultaneously extract and transmit attention information from both aspects.

[0007] Third, there is often a significant difference in the feature channel dimensions between deep teacher networks and lightweight student networks. Teacher networks have far more high-dimensional feature channels than student networks, and directly calculating the distance metric between their attention maps will fail due to the dimensionality mismatch, lacking an effective cross-dimensional alignment method. Summary of the Invention

[0008] The purpose of this invention is to provide a lightweight abnormal behavior detection method and system based on spatiotemporal attention distillation. It solves the semantic misalignment problem caused by the unequal number of layers in teacher-student networks by using a cross-layer uniform mapping mechanism. It extracts spatial node attention maps and temporal keyframe attention maps by decoupling spatiotemporal attention, thereby maintaining the high-speed inference of lightweight student networks while approximating the spatiotemporal inference accuracy of deep teacher networks.

[0009] To achieve the above objectives, in a first aspect, the present invention provides a lightweight abnormal behavior detection method based on spatiotemporal attention distillation, comprising the following steps: S1. Obtain the surveillance video stream, use the pose estimation algorithm to extract the key points of the human skeleton, connect the same point between adjacent frames to construct the temporal edge, and connect the natural limbs to construct the spatial edge, and obtain the spatiotemporal graph matrix as the original input data. S2. Based on the heterogeneity of the asymmetrical number of layers between the deep teacher network and the lightweight student network, a hidden layer mapping relationship between the deep teacher network and the lightweight student network is established proportionally to obtain a set of uniform feature mapping pairs across layers. Each pair of corresponding hidden layers in the mapping pair set constitutes a mapping layer. S3. Input the spatiotemporal graph matrix constructed in S1 into the deep teacher network and the lightweight student network respectively, and obtain the feature maps output by each mapping layer after forward propagation. Since the number of feature channels in the deep teacher network is usually much larger than that in the lightweight student network, directly extracting attention will result in dimensionality mismatch; therefore, a 1×1 convolution operator is used as an adaptive transformation operator to map the number of channels in the feature map of the lightweight student network to the same dimension as that of the deep teacher network, thus completing the physical alignment at the feature map level. S4. For the feature map after dimension alignment in step S3, the spatial node attention map and the temporal keyframe attention map are extracted respectively by the attention decoupling operator to characterize the network’s focus on high-risk limb parts and its sensitivity to the moment when abnormal actions occur. S5. Construct a joint attention distillation loss function, and combine the classification prediction loss, soft label knowledge distillation loss, joint attention alignment term and regularization term to obtain the overall objective function. Optimize the parameters of the lightweight student network through backpropagation, forcing the lightweight student network to approximate the attention path of the deep teacher network in terms of spatiotemporal feature distribution. S6. Deploy the trained lightweight student network to the edge computing node, input the newly collected monitoring video stream into the deployed lightweight student network, and obtain the real-time identification results of abnormal behavior through forward propagation calculation.

[0010] Preferably, step S1 involves extracting key points of the human skeleton using a pose estimation algorithm and constructing a spatiotemporal graph matrix, specifically including: S11. Using a skeleton extraction algorithm, each frame is converted into a frame containing... A set of coordinates for each node; S12. Connect the same point between adjacent frames to construct a temporal edge, and connect natural limbs to construct a spatial edge, obtaining decentralized spatiotemporal graph matrix data, denoted as... ,in The time frame length, This refers to the number of joints in the human skeleton. This represents the initial feature dimension.

[0011] Preferably, the step S2, which establishes a hidden layer mapping relationship between the deep teacher network and the lightweight student network proportionally, specifically includes: S21, Setting up a deep teacher network includes A hidden layer, a lightweight student network containing One hidden layer; S22, Using the proportional index function Establish a set of mapping pairs ,in, The first term represents a lightweight student network. A hidden layer, The first term representing a deep teacher network A hidden layer.

[0012] Preferably, in step S4, the spatial node attention map is extracted using the attention decoupling operator, specifically calculated according to the following formula: ; in, For the first Spatial node attention feature matrix output by each mapping layer For the first In the mapping layer, the first The spatiotemporal feature maps of each channel are input. For the first The total number of feature channels output by each mapping layer This is a learnable spatial mapping weight matrix used to adaptively adjust the weight distribution of different skeleton nodes. For the two-dimensional convolution operation operator of feature mapping, It is a non-linear activation function. This is a spatial mask matrix based on the physical prior constraints of the human skeleton, used to filter out abnormal connection interference between non-adjacent nodes. This is the Hadamard product of matrices, which is the product of corresponding elements. This is a normalized exponential function used to map the attention weights of spatial nodes to... Interval.

[0013] Preferably, in step S4, the attention map of the temporal keyframe is extracted using the attention decoupling operator, specifically calculated according to the following formula: ; in, For the first The mapping layer in the th... The temporal attention weights of the frames. This refers to the number of joints in the human skeleton. For the first In the mapping layer, the first The first channel, the first Frame, First scalar feature response values ​​at each node For the first The total number of feature channels output by each mapping layer In order to target the The temporal motion sensitivity weight of a frame is used to amplify the peak response of frames with drastic motion changes. For learnable bias terms in the time-series dimension, It is a smoothing normalization function used to constrain the calculated temporal keyframe attention weight values ​​within a stable range.

[0014] Preferably, in step S3, an adaptive transformation operator is introduced to eliminate the channel dimension difference between the deep teacher network and the lightweight student network, specifically including: A 1×1 convolution operator is used as an adaptive transformation operator to map the number of feature channels of the lightweight student network to the same dimension as the deep teacher network.

[0015] Preferably, the overall objective function in S5 is: ; in, Tag for real action Predicting probabilities with lightweight student networks Cross-entropy classification loss between them For relative entropy, The temperature coefficient for knowledge distillation is used to soften the output probability distribution of the teacher network. and These represent the original logical values ​​output by the deep teacher network and the lightweight student network at the fully connected layer, respectively. and These are hyperparameters that control the weighting of the soft-label distillation term and the joint attention alignment term, respectively. For deep teacher networks in the first Spatial node attention map output by each mapping layer For lightweight student networks in the first Spatial node attention map output by each mapping layer For deep teacher networks in the first Temporal keyframe attention maps output by each mapping layer. For lightweight student networks in the first Temporal keyframe attention maps output by each mapping layer. For a set of cross-layer feature mapping pairs, and These are adaptive transformation operators for spatial and temporal dimensions, respectively. is the square of the Frobenius norm of the matrix, used to accurately measure the topological distance between high-dimensional attention feature matrices. and These are the balancing weights for spatial node loss and temporal keyframe loss, respectively. Lightweight student network weight parameters of Regularization penalty term.

[0016] Secondly, the present invention provides a lightweight abnormal behavior detection system based on spatiotemporal attention distillation, comprising: a spatiotemporal graph construction module, a cross-layer mapping management module, a joint attention extraction module, a distillation optimization module, and a real-time monitoring module; The spatiotemporal graph construction module is used to acquire surveillance video streams and extract key points of the human skeleton to construct a spatiotemporal graph input matrix that reflects the topology and temporal relationship of limbs; The cross-layer mapping management module is connected to the joint attention extraction module and is used to establish a feature mapping pairing index between the deep teacher network and the lightweight student network according to the preset layer ratio, and to determine the correspondence of each mapping layer. The joint attention extraction module is connected to the spatiotemporal graph construction module and the cross-layer mapping management module, respectively. It is used to first perform dimensional alignment of feature channels through an adaptive transformation operator, and then decouple the spatial and temporal features of the aligned feature map to generate spatial node attention map and temporal keyframe attention map. The distillation optimization module is connected to the joint attention extraction module to construct a multi-objective loss function with attention alignment constraints, guiding the lightweight student network to deeply mimic the inference feature distribution of the deep teacher network during the training phase; The real-time monitoring module is connected to the distillation optimization module and is used to deploy the trained lightweight student network to the edge computing node. The newly acquired monitoring video stream is input into the deployed lightweight student network, and the real-time identification results of abnormal behavior are obtained through forward propagation calculation.

[0017] Therefore, compared with the prior art, the present invention has the following beneficial effects: (1) The cross-layer uniform mapping mechanism maintains the synchronization between deep and shallow semantics of the teacher-student network, effectively solving the semantic misalignment problem caused by the unequal number of layers between the deep teacher network and the lightweight student network; (2) By using the spatiotemporal attention decoupling technique, spatial node attention maps and temporal keyframe attention maps are extracted respectively, which solves the problem of loss of key action features caused by parameter reduction in lightweight networks; (3) When applied, it can simultaneously maintain spatial focus on high-risk body parts and the ability to capture the timing of violations at the moment of occurrence; (4) The lightweight student network has a small number of parameters, fast inference speed and high recognition accuracy. It is suitable for edge monitoring of construction sites with limited computing power and can significantly improve the real-time monitoring efficiency of abnormal behavior in large-scale scenarios.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the alignment of cross-layer uniform mapping and joint attention feature extraction in the teacher-student network according to an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] like Figure 1 As shown, a lightweight abnormal behavior detection method based on spatiotemporal attention distillation includes the following steps: S1. Obtain the surveillance video stream, use the pose estimation algorithm to extract the key points of the human skeleton, connect the same point between adjacent frames to construct the temporal edge, and connect the natural limbs to construct the spatial edge, and obtain the spatiotemporal graph matrix as the original input data. This step aims to transform the raw video data into structured graph data suitable for processing by spatiotemporal graph convolutional networks. Specifically, the raw data includes a set of video samples containing various abnormal behaviors, such as training and testing sets of samples from construction site scenes, where the video samples contain abnormal behaviors such as people falling and climbing illegally.

[0023] S11. Using a skeleton extraction algorithm, each frame is converted into a frame containing... A set of coordinates of nodes; skeleton extraction algorithms such as the OpenPose algorithm, which can detect multiple key points of the human body in real time, including the head, shoulders, elbows, wrists, hips, knees, ankles and other parts.

[0024] S12. After completing the single-frame skeleton extraction, connect the same point between adjacent frames to construct temporal edges, and connect natural limbs to construct spatial edges (i.e., skeletal connections, such as elbow joint connecting to wrist joint, knee joint connecting to ankle joint, etc.), to obtain decentralized spatiotemporal graph matrix data, denoted as... ,in The time frame length, This refers to the number of joints in the human skeleton. The initial feature dimension is defined as follows. After the above processing, the continuous human skeleton sequence is mapped into a graph data form containing spatial topology and temporal dimensions. This form can significantly reduce background interference in the original video and provide standardized input for subsequent processing of the teacher-student network.

[0025] S2. Based on the heterogeneity of the asymmetrical number of layers between the deep teacher network and the lightweight student network, a hidden layer mapping relationship between the deep teacher network and the lightweight student network is established proportionally to obtain a set of uniform feature mapping pairs across layers. Each pair of corresponding hidden layers in the mapping pair set constitutes a mapping layer. This step aims to address the semantic gap between deep teacher networks and lightweight student networks. In the knowledge distillation framework, deep teacher networks typically contain dozens of spatiotemporal graph convolutional layers, while lightweight student networks, in pursuit of small size and fast inference speed, retain only a few network layers. The two differ in the depth and abstraction level of feature evolution; directly aligning intermediate layer feature maps would lead to severe semantic misalignment. Therefore, this invention establishes a proportionally uniformly mapped hidden layer pairing mechanism.

[0026] S21, Setting up a deep teacher network includes A hidden layer, a lightweight student network containing There are 1 hidden layer, among which ; S22, Using the proportional index function Establish a set of mapping pairs ,in, The first term represents a lightweight student network. A hidden layer, The first term representing a deep teacher network A hidden layer, This indicates rounding down. This indexing function ensures that each layer in the student network corresponds to a layer in the teacher network with a similar semantic depth.

[0027] In one specific embodiment of the present invention, the deep teacher network employs a spatiotemporal graph convolutional network, comprising 10 spatiotemporal graph convolutional layers (…). =10), the lightweight student network employs a lightweight spatiotemporal graph convolutional network with reduced channels and layers, containing 4 spatiotemporal graph convolutional layers ( =4). Based on the above proportional index function. / =2.5, then =1 =2, =2 =5, =3 =7, =4 =10, thus establishing a set of mapping pairs. .

[0028] Through the aforementioned cross-layer uniform mapping, the student network can maintain the feature evolution logic synchronized with the teacher network in the shallow structure, laying the foundation for subsequent layer-by-layer attention alignment. Figure 3 This schematically illustrates the process of aligning the cross-layer uniform mapping and joint attention feature extraction of the aforementioned teacher-student network.

[0029] S3. Input the spatiotemporal graph matrix constructed in S1 into the deep teacher network and the lightweight student network respectively. After forward propagation, obtain the feature maps output by each mapping layer. After establishing a set of cross-layer uniform feature mapping pairings, perform dimension alignment using an adaptive transformation operator.

[0030] Specifically, since the number of feature channels in a deep teacher network is usually much larger than that in a lightweight student network, their feature maps exhibit heterogeneity. This embodiment uses a 1×1 convolution operator as an adaptive transformation operator. During the forward propagation process, the original feature map output by the lightweight student network's mapping layer is first up-mapped to force its feature channel number to align to the same dimension as the deep teacher network, thus completing the physical alignment at the feature map level.

[0031] S4. Using the attention decoupling operator, spatial node attention maps and temporal keyframe attention maps of the corresponding levels are calculated and extracted from the dimension-aligned teacher-student network feature maps to characterize the network’s focus on high-risk limb parts and its sensitivity to the moment abnormal actions occur.

[0032] Specifically, for the feature map in S3 after alignment by the 1×1 convolution dimension... ,in, Indicates the first The total number of feature channels output by each mapping layer The index of the mapping layer is used to extract attention feature maps in two dimensions using the attention decoupling operator. (a) Extraction of spatial node attention map The spatial node attention map is obtained by aggregating channel features along the time axis, which characterizes the degree of focus of the network on different body parts. Specifically, the spatial node attention map is calculated according to the following formula: ; in, For the first Spatial node attention feature matrix output by each mapping layer For the first In the mapping layer, the first The spatiotemporal feature maps of each channel are input. For the first The total number of feature channels output by each mapping layer This is a learnable spatial mapping weight matrix used to adaptively adjust the weight distribution of different skeleton nodes. For the two-dimensional convolution operation operator of feature mapping, It is a non-linear activation function. This is a spatial mask matrix based on the physical prior constraints of the human skeleton, used to filter out abnormal connection interference between non-adjacent nodes. This is the Hadamard product of matrices, which is the product of corresponding elements. This is a normalized exponential function used to map the attention weights of spatial nodes to... Interval.

[0033] (ii) Extraction of attention maps for temporal keyframes The temporal keyframe attention map is obtained by aggregating channel features along spatial nodes. This map characterizes the network's sensitivity to the instant of anomaly events. Specifically, the temporal keyframe attention map is calculated using the following formula: ; in, For the first The mapping layer in the th... The temporal attention weights of the frames. This refers to the number of joints in the human skeleton. For the first In the mapping layer, the first The first channel, the first Frame, First scalar feature response values ​​at each node For the first The total number of feature channels output by each mapping layer In order to target the The temporal motion sensitivity weight of a frame is used to amplify the peak response of frames with drastic motion changes. For learnable bias terms in the time-series dimension, It is a smoothing normalization function used to constrain the calculated temporal keyframe attention weight values ​​within a stable range.

[0034] S5. Construct a joint attention distillation loss function, and combine the classification prediction loss, soft label knowledge distillation loss, joint attention alignment term and regularization term to obtain the overall objective function. Optimize the parameters of the lightweight student network through backpropagation, forcing the lightweight student network to approximate the attention path of the deep teacher network in terms of spatiotemporal feature distribution. This step aims to achieve knowledge transfer from a deep teacher network to a lightweight student network through multi-objective joint optimization. This involves cross-layer feature mapping pairs. Under the constraints, the overall optimization objective function of the lightweight student network integrates classification prediction bias, soft label distribution approximation, and spatiotemporal joint attention distance. To improve generalization ability, this invention designs the following overall objective function: ; in, Tag for real action Predicting probabilities with lightweight student networks The cross-entropy classification loss is used to ensure the basic classification performance of the student network. Relative entropy is used to calculate the difference in the output distribution of the teacher-student network logic layer. This is the temperature coefficient for knowledge distillation, used to soften the output probability distribution of the teacher network; its value is typically between 3 and 10. and These represent the original logical values ​​output by the deep teacher network and the lightweight student network at the fully connected layer, respectively. and These are hyperparameters that control the proportions of the soft-label distillation term and the joint attention alignment term, respectively.

[0035] For joint attention alignment terms, where For a set of cross-layer feature mapping pairs, For deep teacher networks in the first Spatial node attention map output by each mapping layer For lightweight student networks in the first Spatial node attention map output by each mapping layer For deep teacher networks in the first Temporal keyframe attention maps output by each mapping layer. For lightweight student networks in the first Temporal keyframe attention maps output by each mapping layer. and These are adaptive transformation operators for spatial and temporal dimensions, used to eliminate channel dimension heterogeneity between teacher-student networks. is the square of the Frobenius norm of the matrix, used to accurately measure the topological distance between high-dimensional attention feature matrices. and These are the balancing weights for spatial node loss and temporal keyframe loss, respectively. Lightweight student network weight parameters of Regularization penalty terms are used to prevent overfitting in lightweight student networks. This is the hyperparameter for regularization intensity.

[0036] In a specific embodiment of the present invention, the range of hyperparameter values ​​is as follows: =0.5, =0.1, =1.0, =1.0, =1×10 -4 Temperature coefficient =4. By adjusting the relative sizes of the above hyperparameters, a balance can be struck between classification accuracy, knowledge distillation fidelity, and student network complexity.

[0037] Minimize the overall training loss using the backpropagation algorithm. Optimize parameters for lightweight student networks This allows it to approximate the "attention" path of deep teacher networks in terms of spatiotemporal feature distribution, enabling deep knowledge transfer from deep teacher networks to lightweight student networks.

[0038] S6. Deploy the trained lightweight student network to the edge computing node, input the newly collected monitoring video stream into the deployed lightweight student network, and obtain the real-time identification results of abnormal behavior through forward propagation calculation.

[0039] After training, the lightweight student network has transferred key spatiotemporal attention knowledge from the deep teacher network, enabling it to recognize abnormal behaviors with high accuracy. At this point, the trained lightweight student network is deployed to edge computing nodes (such as embedded AI computing modules built into construction site surveillance cameras, edge gateways, or industrial computers).

[0040] During the application phase, newly acquired surveillance video streams are input into the deployed lightweight student network. The network first uses a pose estimation algorithm to extract key points of the human skeleton and construct a spatiotemporal graph matrix. Then, through forward propagation calculations, it outputs the predicted probabilities of various abnormal behaviors. When the predicted probability exceeds a preset threshold, the system triggers an alarm, records the type of abnormal behavior and the time of occurrence in the log, and simultaneously pushes the alarm information to the monitoring center.

[0041] In one specific embodiment of the present invention, a lightweight student network is deployed on an NVIDIA Jetson XavierNX edge computing device with a video stream input frame rate of 30 FPS and a network inference speed of approximately 7.1 ms per frame, enabling real-time early warning of abnormal behaviors such as falls and unauthorized climbing.

[0042] like Figure 2 The detection system of the present invention includes: a spatiotemporal graph construction module, a cross-layer mapping management module, a joint attention extraction module, a distillation optimization module, and a real-time monitoring module; The spatiotemporal graph construction module is used to acquire surveillance video streams and extract key points of the human skeleton to construct a spatiotemporal graph input matrix that reflects the topology and temporal relationship of limbs; The cross-layer mapping management module is connected to the joint attention extraction module and is used to establish a feature mapping pairing index between the deep teacher network and the lightweight student network according to the preset layer ratio, and to determine the correspondence of each mapping layer. The joint attention extraction module is connected to the spatiotemporal graph construction module and the cross-layer mapping management module, respectively. It is used to first perform dimensional alignment of feature channels through an adaptive transformation operator, and then decouple the spatial and temporal features of the aligned feature map to generate spatial node attention map and temporal keyframe attention map. The distillation optimization module is connected to the joint attention extraction module to construct a multi-objective loss function with attention alignment constraints, guiding the lightweight student network to deeply mimic the inference feature distribution of the deep teacher network during the training phase; The real-time monitoring module is connected to the distillation optimization module and is used to deploy the trained lightweight student network to the edge computing node. The newly acquired monitoring video stream is input into the deployed lightweight student network, and the real-time identification results of abnormal behavior are obtained through forward propagation calculation.

[0043] To verify the superiority of the method in abnormal behavior detection, experiments were conducted using the NTU RGB+D public dataset and a custom construction site abnormal behavior dataset. The cross-layer alignment mechanism enabled the lightweight student network in this embodiment to improve accuracy by 5%-8% compared to conventional distillation while compressing the number of parameters by 70%.

[0044] Table 1 presents the experimental results on a custom construction site dataset. The dataset contains approximately 12,000 video samples across six categories of abnormal behaviors, including falls, unauthorized climbing, and entry into dangerous areas. The training and test sets are split in a 7:3 ratio. All methods utilize the same spatiotemporal graph convolutional network skeleton extraction and preprocessing workflow.

[0045] Table 1. Experimental results on a custom construction site dataset.

[0046] Experimental results show that the native depth ST-GCN achieves the highest accuracy (92.4%), but its large parameter count (18.5M) and slow inference speed (32FPS) make it difficult to meet the real-time detection requirements at the edge. While the traditional label distillation method compresses the parameter count to 2.1M and increases the inference speed to 145FPS, the accuracy drops significantly to 81.6%, with a precision loss exceeding 10%. This embodiment, while maintaining a parameter count (2.2M) and inference speed (140FPS) roughly equivalent to the traditional label distillation method, improves the accuracy to 88.9%, a 7.3 percentage point improvement compared to the traditional method, thus validating the effectiveness of the spatiotemporal attention distillation method proposed in this invention.

[0047] Therefore, the present invention adopts the above-mentioned lightweight abnormal behavior detection method and system based on spatiotemporal attention distillation, and uses the cross-layer uniform mapping mechanism to maintain the synchronization of deep and shallow semantics of teacher-student networks, effectively solving the semantic misalignment problem caused by the unequal number of layers between deep teacher networks and lightweight student networks. By using the spatiotemporal attention decoupling technique, spatial node attention maps and temporal keyframe attention maps are extracted separately, which solves the problem of loss of key action features caused by parameter reduction in lightweight networks. When applied, it can simultaneously maintain spatial focus on high-risk body parts and the ability to capture the timing of violations. With the number of parameters compressed to 2.2M, the inference speed reaches 140FPS and the detection accuracy reaches 88.9%, which is 7.3 percentage points higher than the traditional distillation method. It is suitable for edge monitoring of construction sites with limited computing power.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A lightweight abnormal behavior detection method based on spatiotemporal attention distillation, characterized in that, Includes the following steps: S1. Obtain the surveillance video stream, use the pose estimation algorithm to extract the key points of the human skeleton, connect the same point between adjacent frames to construct the temporal edge, and connect the natural limbs to construct the spatial edge, and obtain the spatiotemporal graph matrix as the original input data. S2. Based on the heterogeneity of the asymmetrical number of layers between the deep teacher network and the lightweight student network, a hidden layer mapping relationship between the deep teacher network and the lightweight student network is established proportionally to obtain a set of uniform feature mapping pairs across layers. Each pair of corresponding hidden layers in the mapping pair set constitutes a mapping layer. S3. Input the spatiotemporal graph matrix constructed in S1 into the deep teacher network and the lightweight student network respectively. After forward propagation, obtain the feature maps output by each mapping layer. Introduce an adaptive transformation operator to eliminate the channel dimension difference between the feature maps of the lightweight student network and the deep teacher network, so that the low-dimensional features of the student network and the high-dimensional features of the teacher network are aligned in the channel dimension. S4. For the dimensionally aligned feature map, extract the spatial node attention map and the temporal keyframe attention map respectively using the attention decoupling operator. S5. Construct a joint attention distillation loss function, and combine the classification prediction loss, soft label knowledge distillation loss, joint attention alignment term and regularization term to obtain the overall objective function. Optimize the parameters of the lightweight student network through backpropagation, forcing the lightweight student network to approximate the attention path of the deep teacher network in terms of spatiotemporal feature distribution. S6. Deploy the trained lightweight student network to the edge computing node, input the newly collected monitoring video stream into the deployed lightweight student network, and obtain the real-time identification results of abnormal behavior through forward propagation calculation.

2. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, S1 utilizes a pose estimation algorithm to extract key points of the human skeleton and construct a spatiotemporal graph matrix, specifically including: S11. Using a skeleton extraction algorithm, each frame is converted into a frame containing... A set of coordinates for each node; S12. Connect the same point between adjacent frames to construct a temporal edge, and connect natural limbs to construct a spatial edge, obtaining decentralized spatiotemporal graph matrix data, denoted as... ,in The time frame length, This refers to the number of joints in the human skeleton. This represents the initial feature dimension.

3. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, The hidden layer mapping relationship between the deep teacher network and the lightweight student network established proportionally in S2 specifically includes: S21, Setting up a deep teacher network includes A hidden layer, a lightweight student network containing One hidden layer; S22, Using the proportional index function Establish a set of mapping pairs ,in, The first term represents a lightweight student network. A hidden layer, The first term representing a deep teacher network A hidden layer.

4. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, In step S4, the spatial node attention map is extracted using the attention decoupling operator, specifically calculated according to the following formula: ； in, For the first Spatial node attention feature matrix output by each mapping layer For the first In the mapping layer, the first The spatiotemporal feature maps of each channel are input. For the first The total number of feature channels output by each mapping layer The weight matrix is ​​a learnable space mapping. For the two-dimensional convolution operation operator of feature mapping, It is a non-linear activation function. This is a spatial mask matrix based on the physical prior constraints of the human skeleton. The Hadamard product of matrices. It is a normalized exponential function.

5. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, In step S4, the attention map of the temporal keyframe is extracted using the attention decoupling operator, specifically calculated according to the following formula: ； in, For the first The mapping layer in the th... The temporal attention weights of the frames. This refers to the number of joints in the human skeleton. For the first In the mapping layer, the first The first channel, the first Frame, First scalar feature response values ​​at each node For the first The total number of feature channels output by each mapping layer In order to target the The temporal action sensitivity weight of the frame. For learnable bias terms in the time-series dimension, This is the smoothing normalization function.

6. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, The S3 section introduces an adaptive transformation operator to eliminate the channel dimension difference between the deep teacher network and the lightweight student network, specifically including: A 1×1 convolution operator is used as an adaptive transformation operator to map the number of feature channels of the lightweight student network to the same dimension as the deep teacher network.

7. The lightweight anomalous behavior detection method based on spatiotemporal attention distillation according to claim 1, characterized in that, The overall objective function in S5 is: ； in, Tag for real action Predicting probabilities with lightweight student networks Cross-entropy classification loss between them For relative entropy, The temperature coefficient for knowledge distillation is used to soften the output probability distribution of the teacher network. and These represent the original logical values ​​output by the deep teacher network and the lightweight student network at the fully connected layer, respectively. and These are hyperparameters that control the weighting of the soft-label distillation term and the joint attention alignment term, respectively. For deep teacher networks in the first Spatial node attention map output by each mapping layer For lightweight student networks in the first Spatial node attention map output by each mapping layer For deep teacher networks in the first Temporal keyframe attention maps output by each mapping layer. For lightweight student networks in the first Temporal keyframe attention maps output by each mapping layer. For a set of cross-layer feature mapping pairs, and These are adaptive transformation operators for spatial and temporal dimensions, respectively. is the square of the Frobenius norm of the matrix, used to accurately measure the topological distance between high-dimensional attention feature matrices. and These are the balancing weights for spatial node loss and temporal keyframe loss, respectively. Lightweight student network weight parameters of Regularization penalty term.

8. A lightweight abnormal behavior detection system based on spatiotemporal attention distillation, used to perform the method according to any one of claims 1-7, characterized in that, include: The module includes a spatiotemporal graph construction module, a cross-layer mapping management module, a joint attention extraction module, a distillation optimization module, and a real-time monitoring module. The spatiotemporal graph construction module is used to acquire surveillance video streams and extract key points of the human skeleton to construct a spatiotemporal graph input matrix that reflects the topology and temporal relationship of limbs; The cross-layer mapping management module is connected to the joint attention extraction module and is used to establish a feature mapping pairing index between the deep teacher network and the lightweight student network according to the preset layer ratio, and to determine the correspondence of each mapping layer. The joint attention extraction module is connected to the spatiotemporal graph construction module and the cross-layer mapping management module, respectively. It is used to first perform dimensional alignment of feature channels through an adaptive transformation operator, and then decouple the spatial and temporal features of the aligned feature map to generate spatial node attention map and temporal keyframe attention map. The distillation optimization module is connected to the joint attention extraction module to construct a multi-objective loss function with attention alignment constraints, guiding the lightweight student network to deeply mimic the inference feature distribution of the deep teacher network during the training phase; The real-time monitoring module is connected to the distillation optimization module and is used to deploy the trained lightweight student network to the edge computing node. The newly acquired monitoring video stream is input into the deployed lightweight student network, and the real-time identification results of abnormal behavior are obtained through forward propagation calculation.

Images