Behavior recognition method for improving space-time diagram convolutional network

Abstract

The invention provides a human skeleton behavior recognition method based on an improved space-time diagram convolutional network (ST-GCN). According to a traditional ST-GCN method, structural relations of joint points are modeled through space graph convolution, dynamic changes are captured in combination with time convolution, however, the problems that a topological structure is fixed, time sequence modeling is limited and channels have no difference weighting exist, and the recognition precision and adaptability of the traditional ST-GCN method are limited. According to the method, a joint channel attention module (JCA) and a multi-scale path attention residual network (MSPARN) are introduced on the basis of an ST-GCN architecture, so that the adaptive weighting capability of a feature channel and the multi-scale modeling capability of a time sequence dynamic feature are respectively improved. Specifically, the JCA module strengthens the response of the key channel by dynamically adjusting the channel weight; the MSPARN module utilizes multi-scale cavity convolution and a path attention mechanism to enhance the processing ability of the model to complex time sequence dynamics. Experiments show that the method has strong robustness in a complex background and a multi-person scene, and has wide application prospects, especially in the fields of intelligent security and protection, health monitoring, human-computer interaction and the like.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and specifically relates to a behavior recognition method based on skeleton data, in particular to an improved ST-GCN network model that combines multi-scale dilated convolution and channel attention mechanism to achieve a highly accurate and robust human skeleton behavior recognition method. Background Art

[0002] Human behavior recognition technology is a key research area in computer vision and artificial intelligence, widely used in scenarios such as intelligent surveillance, intelligent security, health monitoring, and human-computer interaction. Behavior recognition methods can be broadly categorized as those based on visual appearance features and those based on human skeleton data. Appearance-based methods rely on RGB video streams and extract static and dynamic features directly from video frames. These methods are susceptible to environmental factors such as lighting variations, background complexity, and camera angle, resulting in unstable recognition results. Recognition accuracy is particularly difficult to guarantee in complex backgrounds or scenarios involving multiple people interacting.

[0003] Compared with methods based on visual appearance features, methods based on human skeleton data have significant advantages in anti-interference ability. By capturing the spatial position information of joints through human skeleton data, the influence of background noise and illumination changes can be effectively avoided, especially when processing static or periodic movements, with higher stability and robustness. As a deep learning method based on skeleton data, the spatiotemporal graph convolutional network (ST-GCN) has become an important breakthrough in this field due to its ability to capture both spatial and temporal features. ST-GCN constructs a graph structure and uses a graph convolutional network (GCN) to model the spatial relationship between joints. It combines the temporal convolution module to capture the temporal changes of joints, effectively improving the performance of skeleton data.

[0004] Although ST-GCN has demonstrated good performance in spatiotemporal modeling, the network still faces some challenges. First, ST-GCN uses a predefined fixed topology to represent the connection relationship between joints. This fixed structure limits the network's ability to model complex interactions between non-directly connected joints and cannot effectively handle long-range dependencies in various complex actions. Second, ST-GCN's temporal modeling module relies on fixed-size temporal convolution kernels, which makes it unable to flexibly adapt to different action rhythms and time spans, resulting in insufficient extraction of action features for long and short time scales. In addition, ST-GCN lacks flexibility in its processing of feature channels. All feature channels are treated equally, making it difficult for the model to adaptively weight different feature channels, which in turn affects the ability to extract key dynamic features.

[0005] In summary, existing human action recognition methods still have much room for improvement in spatial modeling and temporal modeling. Based on this, this paper proposes a human skeleton action recognition method based on an improved ST-GCN network, aiming to enhance the model's spatiotemporal modeling capabilities, especially for complex actions in complex environments. By introducing the Joint Channel Attention (JCA) module and the Multi-Scale Path Attention Residual Network (MSPARN) module, the perception and modeling capabilities of spatiotemporal information are enhanced, thereby improving the accuracy and robustness of action recognition. Summary of the Invention

[0006] (1) Technical issues to be resolved

[0007] This paper proposes an improved ST-GCN network model that combines the joint channel attention module (JCA) and the multi-scale path attention residual network (MSPARN). It aims to address the problems of no difference in channel processing and lack of adaptability in the ST-GCN network, as well as the problem of single temporal modeling scale and difficulty in capturing long-term and short-term actions. It improves the model's ability to model key spatial structures and multi-scale temporal features in skeleton action sequences, and achieves human action recognition with higher accuracy and stronger robustness.

[0008] (2) Technical solution

[0009] The technical solution of the present invention specifically includes the following steps:

[0010] S1: Obtain human skeleton sequence data and standardize the original skeleton data;

[0011] S2: Construct a spatiotemporal graph structure, modeling the skeleton sequence as a graph structure, where skeleton key points are graph nodes, joint connections are graph edges, and cross-frame connections are established in the time dimension;

[0012] S3: Based on the ST-GCN backbone network, a human skeleton action recognition method based on an improved spatiotemporal convolutional network is constructed;

[0013] S4: Using the labeled human skeleton dataset as training samples, the improved spatiotemporal graph convolutional network is trained to obtain an action recognition network model based on the improved spatiotemporal graph convolutional network;

[0014] S5: Input the human skeleton data to be identified into the trained spatiotemporal graph convolutional network model to obtain the human behavior recognition results.

[0015] Furthermore, the improved spatiotemporal convolutional network in S3 is based on the ST-GCN backbone network structure, and introduces two modules, joint channel attention (JCA) and multi-scale path attention residual network (MSPARN), in sequence (see Appendix 1). Figure 3 shown), where:

[0016] SS1: JCA module performs weighted adjustment on feature channels and strengthens key feature channels. It is composed of channel enhancement components and two-layer fully connected networks (see attached Figure 4 As shown). First, the module input feature map X∈R is passed through the channel enhancement module N×C×T×V Compressed into channel dimension vector Z, that is:

[0017] Z=JCA(X)

[0018] Secondly, channel attention is generated and the Z vector is nonlinearly transformed through two fully connected layers;

[0019] Finally, the channel attention weight is generated by the Sigmoid function:

[0020] Att c =σ(W2·ReLU(W1·Z))

[0021] Where W1 and W2 are learnable parameters, and σ represents the Sigmoid function.

[0022] Through the JCA module, the model can adaptively allocate the contribution of each channel in behavior recognition, which can effectively improve the network's expressive ability in different joint dimensions.

[0023] SS2: The MSPARN module uses multi-scale dilated convolution to extract behavioral features of different durations, introduces a path attention mechanism to adjust the multi-scale response strength, and fuses the residual structure to enhance feature expression. It consists of a multi-scale dilated convolution module (MDC) and a path attention module (PAM) (see Appendix). Figure 5 where:

[0024] The multi-scale dilated convolution module (MDC) can effectively extract feature information at different time scales. The path attention module (PAM) dynamically calculates the importance weight of each scale path through the attention mechanism and uses it to weight the output of each path. The specific calculation process is as follows:

[0025] First, the feature map of each path is average pooled and maximum pooled in the channel dimension to obtain two representative feature vectors v avg and v max ;

[0026] Secondly, the eigenvector v avg 、v max They are transformed through two fully connected layers (FC) respectively, and the path attention weight vector z is generated by the Sigmoid function:

[0027] z=σ(W2δ(W1v max )+W4δ(W3v avg ))

[0028] Where σ is the Sigmoid function, δ is the ReLU activation function, W1, W2, W3, and W4 are network learning parameters;

[0029] Finally, based on the obtained weights, the feature map of the original path is recalibrated to enhance the path features that are meaningful for fault diagnosis.

[0030] In addition, the MSPARN network contains multiple multi-scale attention residual blocks (MARBs) to improve network stability. MARB consists of sub-modules and operators such as batch normalization (BN), ReLU activation function, multi-scale dilated convolution (MDC), path attention module (PAM) and residual connection (RC) (see attached). Figure 6 The specific workflow is as follows:

[0031] Step 1: Process input features through BN and ReLU functions;

[0032] Step 2: Use MDC to extract multi-scale features, and assign corresponding attention weights to different path features through the PAM module;

[0033] Step 3: Repeat Step 1, use a one-dimensional convolutional layer to achieve feature fusion and downsampling (stride = 2), and use residual connections to avoid the gradient vanishing problem in deep networks;

[0034] Step 4: Use the global average pooling (GAP) layer to compress the feature dimension, and output the final behavior category through the fully connected (FC) layer and the Softmax function.

[0035] Beneficial effects

[0036] This paper proposes a human skeleton action recognition method based on an improved spatiotemporal graph convolutional network (ST-GCN). Building on the traditional ST-GCN model, it introduces a joint channel attention module (JCA) and a multi-scale path attention residual network (MSPARN), effectively improving the accuracy and robustness of action recognition. This method is particularly suitable for complex scenes with occlusion, multiple people, and varying action rhythms. The JCA module weights feature channels, adaptively highlighting key features and suppressing redundant information, thereby enhancing the model's ability to perceive local dynamic features.

[0037] The MSPARN module jointly models motion features across different time spans through multi-scale atrous convolution and path attention mechanisms, significantly enhancing the understanding of both long- and short-term actions while maintaining model stability. Experimental results show that on the NTURGB+D60 dataset, this method achieved accuracies of 86.88% and 92.75% in cross-subject and cross-view evaluations, respectively. These improvements, compared to the original ST-GCN model, represent 5.4% and 4.5% improvements, respectively, demonstrating its significant advantages in spatial structure modeling and dynamic feature capture. This method boasts lightweight structure, accurate recognition, and strong adaptability, making it suitable for a variety of practical application scenarios, including smart security, public space monitoring, and human-computer interaction. It possesses excellent practicality and industrial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flow chart of the steps for implementing the present invention;

[0039] Figure 2 It is the system workflow diagram;

[0040] Figure 3 This is the improved ST-GCN network architecture diagram;

[0041] Figure 4 This is the JCA module structure diagram;

[0042] Figure 5 This is the structure diagram of the MSPARN module;

[0043] Figure 6 This is the MARB module structure diagram;

[0044] Figure 7 This is a schematic diagram of human skeleton data;

[0045] Figure 8 This is the effect diagram of human behavior detection. DETAILED DESCRIPTION

[0046] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] S1: Skeleton data acquisition and preprocessing

[0048] This step is based on the publicly available large-scale standard human skeleton dataset, NTU RGB+D 60. This dataset, collected by the KinectV2 depth camera, covers 60 types of human motion and contains 56,880 skeleton motion sequences. These sequences were generated by 40 subjects performing different motions from three camera perspectives. Each frame of skeleton data contains 25 key joint nodes, each of which represents its position in space using three-dimensional coordinates (x, y, z). These key points cover major human structures, such as the head, neck, spine, shoulders, elbows, wrists, hips, knees, and ankles, effectively capturing the spatial topology of human motion.

[0049] To improve the recognition accuracy and robustness of the model, the original skeleton data is standardized: first, the coordinates of all key points are normalized to map them to a uniform scale range to reduce data offset caused by differences in individual height or movement amplitude; second, the "spine center point" or "hip center point" is used as the reference origin for translation standardization, so that the skeleton is relatively stable in space; third, since the number of time frames of the action sequence is different, the length of all sequences is unified to a fixed frame length to facilitate subsequent network batch processing; finally, the skeleton sequence is organized into a tensor input model in the format of N×C×T×V. Where: N represents the number of people participating in the action in each frame; C represents the number of coordinate channels; T represents the length of the action sequence; and V represents the number of key points (25).

[0050] S2: Constructing a spatiotemporal graph structure

[0051] The skeleton within each time frame is modeled as an undirected graph G = (V, E), where V is the set of joints and E is the set of edges constructed based on the natural connections of the human skeleton. The temporal dimension is constructed by connecting keypoint nodes with the same number between different frames to form temporal edges, thereby constructing a complete spatiotemporal graph structure. This graph structure is input into the improved ST-GCN backbone network for joint spatial and temporal feature modeling.

[0052] S3: Building an improved spatiotemporal convolutional network behavior recognition model

[0053] Reference Figure 3 Based on the ST-GCN backbone structure, two modules, Joint Channel Attention (JCA) and Multi-Scale Path Attention Residual Network (MSPARN), are introduced in sequence, where:

[0054] (1) Joint Channel Attention Module (JCA)

[0055] Reference Figure 4 , This module performs weighted adjustment on the feature channels and strengthens the key feature channels. It consists of a channel enhancement component and a two-layer fully connected network. The module input feature map is X∈R N×C×T×V, first compressed into a channel dimension vector z through the channel enhancement module, calculated as follows:

[0056] Z=JCA(X)

[0057] Then the channel attention is generated: the Z vector is nonlinearly transformed through two layers of fully connected layers, and finally the channel attention weight is generated through the Sigmoid function:

[0058] Att c =σ(W2·ReLU(W1·Z))

[0059] Where W1 and W2 are learnable parameters, and σ represents the Sigmoid function.

[0060] Through this module, the model can adaptively allocate the contribution of each channel in behavior recognition, effectively improving the network's expressiveness in different joint dimensions. Through this module, the model can adaptively allocate the contribution of each channel in behavior recognition, effectively improving the network's expressiveness in different joint dimensions.

[0061] (2) Integrated Multi-Scale Path Attention Residual Network (MSPARN)

[0062] Reference Figure 5 The MSPARN module uses multi-scale dilated convolution to extract behavioral features of different durations, introduces a path attention mechanism to adjust the multi-scale response intensity, and fuses the residual structure to enhance feature expression. It consists of a multi-scale dilated convolution module (MDC), a path attention module (PAM), etc.

[0063] The multi-scale dilated convolution module (MDC) can effectively extract feature information at different time scales.

[0064] The Path Attention Module (PAM) dynamically calculates the importance weight of each scale path through the attention mechanism. The specific calculation process is as follows:

[0065] First, the feature map of each path is average pooled and maximum pooled in the channel dimension to obtain two representative feature vectors v avg and v max ;

[0066] Secondly, the eigenvector v avg 、v max They are transformed through two fully connected layers (FC) respectively, and the path attention weight vector z is generated by the Sigmoid function:

[0067] z=σ(W2δ(W1v max )+W4δ(W3v avg ))

[0068] Where σ is the Sigmoid function, δ is the ReLU activation function, and W1, W2, W3, and W4 are network learning parameters. Finally, based on the obtained weights, the feature map of the original path is recalibrated to enhance the path features that are meaningful for fault diagnosis.

[0069] In addition, the MSPARN network contains multiple multi-scale attention residual blocks (MARBs) to improve network stability. Figure 6 MARB consists of submodules and operators such as batch normalization (BN), ReLU activation function, multi-scale dilated convolution (MDC), path attention module (PAM) and residual connection (RC). The specific workflow is as follows:

[0070] Step 1: Process input features through BN and ReLU functions;

[0071] Step 2: Use MDC to extract multi-scale features, and assign corresponding attention weights to different path features through the PAM module;

[0072] Step 3: Repeat Step 1, use a one-dimensional convolutional layer to achieve feature fusion and downsampling (stride = 2), and use residual connections to avoid the gradient vanishing problem in deep networks;

[0073] Step 4: Use the global average pooling (GAP) layer to compress the feature dimension, and output the final behavior category through the fully connected (FC) layer and the Softmax function.

[0074] S4: Improved training and validation of the ST-GCN model

[0075] The model is trained using a labeled skeleton action dataset (NTU RGB+D 60 / 120). The training uses the Focal Loss loss function and the Cosine Annealing learning rate strategy to optimize the model's ability to discriminate difficult-to-classify samples.

[0076] S5: Human Action Recognition with Skeleton Data

[0077] Input the skeleton sequence of the action to be identified into the trained model and output the behavior category to which it belongs (such as the attached Figure 8 shown).

Claims

1. An improved behavior recognition method based on spatiotemporal graph convolutional network, characterized in that: The steps include: S1: Obtain human skeleton sequence data and standardize the original skeleton data; S2: Construct a spatiotemporal graph structure, modeling the skeleton sequence as a graph structure, where skeleton key points are graph nodes, joint connections are graph edges, and cross-frame connections are established in the time dimension; S3: Based on the ST-GCN backbone network, a human skeleton behavior recognition method based on an improved spatiotemporal convolutional network (improved ST-GCN network) is constructed; S4: Using the labeled human skeleton dataset as training samples, the improved spatiotemporal graph convolutional network is trained to obtain an action recognition network model based on the improved spatiotemporal graph convolutional network; S5: Input the human skeleton data to be identified into the trained spatiotemporal graph convolutional network model to obtain the human behavior recognition results.

2. The behavior recognition method based on an improved spatiotemporal graph convolutional network according to claim 1, characterized in that: The improved spatiotemporal convolutional network described in S3 is based on the ST-GCN backbone network structure, and introduces two modules: joint channel attention (JCA) and multi-scale path attention residual network (MSPARN), where: SS1: JCA module performs weighted adjustment on feature channels and strengthens key feature channels. It consists of a channel enhancement component and a two-layer fully connected network. First, the module input feature map X∈R is passed through the channel enhancement module. N×C×T×V Compressed into channel dimension vector Z, that is: Z=JCA(X) Secondly, channel attention is generated and the Z vector is nonlinearly transformed through two fully connected layers; Finally, the channel attention weight is generated by the Sigmoid function: Att c =σ(W2 ReLU(W1 Z)) Where W1 and W2 are learnable parameters, and σ represents the Sigmoid function. Through the JCA module, the model can adaptively allocate the contribution of each channel in behavior recognition, which can effectively improve the network's expressive ability in different joint dimensions. SS2: The MSPARN module uses multi-scale dilated convolution to extract behavioral features of different durations, introduces a path attention mechanism to adjust the multi-scale response strength, and integrates a residual structure to enhance feature expression. It consists of a multi-scale dilated convolution module (MDC) and a path attention module (PAM). The multi-scale dilated convolution module (MDC) can effectively extract feature information at different time scales. The path attention module (PAM) dynamically calculates the importance weight of each scale path through the attention mechanism and uses it to weight the output of each path. The specific calculation process is as follows: First, the feature map of each path is average pooled and maximum pooled in the channel dimension to obtain two representative feature vectors v avg and v max ; Secondly, the eigenvector v avg 、v max They are transformed through two fully connected layers (FC) respectively, and the path attention weight vector z is generated by the Sigmoid function: z=σ(W2δ(W1v max )+W4δ(W3v avg )) Where σ is the Sigmoid function, δ is the ReLU activation function, W1, W2, W3, and W4 are network learning parameters; Finally, based on the obtained weights, the feature map of the original path is recalibrated to enhance the path features that are meaningful for fault diagnosis. In addition, the MSPARN network includes multiple multi-scale residual attention blocks (MARBs) to improve network stability. MARB consists of sub-modules and operators such as batch normalization (BN), ReLU activation function, multi-scale dilated convolution (MDC), path attention module (PAM) and residual connection (RC). The specific workflow is as follows: Step 1: Process input features through BN and ReLU functions; Step 2: Use MDC to extract multi-scale features, and assign corresponding attention weights to different path features through the PAM module; Step 3: Repeat Step 1, use a one-dimensional convolutional layer to achieve feature fusion and downsampling (stride = 2), and use residual connections to avoid the gradient vanishing problem in deep networks; Step 4: Use the global average pooling (GAP) layer to compress the feature dimension, and output the final behavior category through the fully connected (FC) layer and the Softmax function.

Images