Skeleton action recognition method based on graph convolutional network

Abstract

The invention relates to the technical field of computer vision, in particular to a skeleton action recognition method based on a graph convolution network, and the method comprises the steps: respectively feeding an input feature graph into a space-time dynamic hypergraph convolution flow and a channel difference hypergraph convolution flow for feature extraction; fusing the features of the space-time dynamic hypergraph convolution flow and the channel difference hypergraph convolution to obtain a fused feature; and sending the fusion features into a classifier to obtain an action category. According to the method, space-time dynamic hypergraph convolution and channel difference hypergraph convolution are integrated. The network utilizes a hypergraph structure to handle more complex and higher order associations to enhance characterization learning. A channel refining module and a hypergraph feature interaction module are adopted, and information representation is effectively extracted from action data. Besides, by integrating a channel difference mechanism, the recognition performance of the model is remarkably improved, and therefore the problem that high-order motion information in skeleton data is not fully utilized due to the fact that a traditional GCN cannot capture potential correlation of non-adjacent joints is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a skeleton action recognition method based on a graph convolutional network. Background Art

[0002] Action recognition is a core research topic in computer vision, encompassing a variety of feature representation methods, including RGB image frames, human skeleton data, and depth maps. In recent years, human skeleton-based action recognition has garnered widespread attention due to its superiority in handling background noise and robustness to issues such as illumination variations, color distortion, and occlusion. Unlike traditional RGB image frame action recognition methods, human skeleton data preserves high-order motion information and, by combining temporal and spatial information, effectively extracts the spatiotemporal characteristics of actions. This plays a crucial role in the development of fields such as human-computer interaction, action recognition, and video content analysis.

[0003] In the early days of deep learning, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) served as mainstream methods. They treated human joint coordinates as a set of independent features, organized them into vector sequences or pseudo-images, and fed them into the network to predict action labels. However, these methods ignored the intrinsic correlation between joints and failed to fully extract features from skeleton data. To this end, Yan et al. used GCN to model the correlation between human joints and graphs, proposing a spatiotemporal graph convolutional network (ST-GCN). However, its static segmentation strategy struggled to adapt to a wide range of actions. Subsequently, Shi et al. proposed a two-stream adaptive graph convolutional network (2s-AGCN), which adaptively learns graph structures based on input data. However, using the same topology for all channels limited the flexibility of feature extraction. Although the Channel Topology Refinement Graph Convolutional Network (CTR-GCN) proposed by Chen et al. optimizes the topological graph structure and improves the flexibility of feature extraction, the GCN-based method still has two obvious shortcomings: first, the edges in the graph structure can only connect two adjacent nodes, which cannot reflect the dependency relationship of multiple joints and the indirectly related joint relationship, and omits a lot of high-order information hidden between joints; second, the sample skeleton graph structure is fixed, but the positions and angles of human joints in the dataset are different, resulting in partial missing of joint-related data. Summary of the Invention

[0004] The purpose of this invention is to provide a skeleton action recognition method based on graph convolutional network, aiming to solve the problem that traditional GCN cannot capture the potential association of non-adjacent joints, resulting in insufficient utilization of high-order motion information in skeleton data.

[0005] To achieve the above object, the present invention provides a skeleton action recognition method based on graph convolutional network, comprising the following steps:

[0006] Preprocess the human skeleton data to obtain the input feature map;

[0007] The input feature map is fed into the spatiotemporal dynamic hypergraph convolution flow and the channel differential hypergraph convolution flow for feature extraction;

[0008] The features of spatiotemporal dynamic hypergraph convolution flow and channel differential hypergraph convolution are fused to obtain fused features;

[0009] The fused features are fed into the classifier to obtain the action category.

[0010] Among them, the spatiotemporal dynamic hypergraph convolution flow includes a spatial convolutional network and an expanded time convolution module, and the spatial convolutional network includes a dynamic channel refinement module, a hypergraph feature interaction module and a feature aggregation module.

[0011] The dynamic channel refinement module includes dynamic modeling, feature transformation and channel aggregation steps to generate channel-specific topological relationship representation and aggregate channel graphs to obtain the final output features.

[0012] The hypergraph feature interaction module extracts features through linear transformation and Einstein summation convention function, generates a difference feature matrix, and obtains comprehensive interactive output through feature interaction.

[0013] The channel-differential hypergraph convolutional flow processes the input feature map through a channel-differential mechanism to learn high-level representations of differential features.

[0014] The present invention provides a skeleton action recognition method based on a graph convolutional network, comprising the following steps: preprocessing human skeleton data to obtain an input feature map; feeding the input feature map into a spatiotemporal dynamic hypergraph convolution flow and a channel differential hypergraph convolution flow for feature extraction; fusing the features of the spatiotemporal dynamic hypergraph convolution flow and the channel differential hypergraph convolution flow to obtain a fused feature; and feeding the fused feature into a classifier to obtain an action category. The present invention integrates spatiotemporal dynamic hypergraph convolution and channel differential hypergraph convolution. Compared with traditional graph structures, the network utilizes a hypergraph structure to handle more complex and high-order correlations to enhance representation learning. A channel refinement module and a hypergraph feature interaction module are used to effectively extract information representations from action data. In addition, by incorporating a channel differential mechanism, the recognition performance of the model is significantly improved, thereby solving the problem that traditional GCN cannot capture the potential correlations between non-adjacent joints, resulting in insufficient utilization of high-order motion information in skeleton data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flowchart of a skeleton action recognition method based on graph convolutional network provided by the present invention.

[0017] Figure 2 (a) is the overall framework of DBC-HCN; (b) is the spatial convolutional layer structure; (c) is the dilated temporal convolutional layer structure.

[0018] Figure 3 It is the channel difference mechanism in the channel difference hypergraph convolutional network.

[0019] Figure 4 This is the confusion matrix for the NTU120 dataset. The yellower the diagonal squares, the more accurate the recognition. (a) X-sub benchmark on the NTU120 dataset. (b) X-set benchmark on the NTU120 dataset. DETAILED DESCRIPTION

[0020] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0021] See also Figures 1 to 4 The present invention provides a skeleton action recognition method based on graph convolutional network, comprising the following steps:

[0022] S1 preprocesses the human skeleton data to obtain the input feature map;

[0023] Specifically, first, the human skeleton data is preprocessed to obtain input feature maps suitable for subsequent processing. This step ensures the standardization and consistency of the data, laying the foundation for subsequent feature extraction.

[0024] S2 sends the input feature map into the spatiotemporal dynamic hypergraph convolution flow and the channel difference hypergraph convolution flow for feature extraction;

[0025] The spatiotemporal dynamic hypergraph convolution flow includes a spatial convolutional network and a dilated temporal convolution module. The spatial convolutional network includes a dynamic channel refinement module, a hypergraph feature interaction module and a feature aggregation module.

[0026] The dynamic channel refinement module includes dynamic modeling, feature transformation and channel aggregation steps to generate channel-specific topological relationship representation and aggregate channel graphs to obtain the final output features.

[0027] The hypergraph feature interaction module extracts features through linear transformation and Einstein summation convention function, generates a difference feature matrix, and obtains comprehensive interactive output through feature interaction.

[0028] The channel-differential hypergraph convolutional flow processes the input feature map through a channel-differential mechanism to learn high-level representations of differential features.

[0029] Specifically, the feature interaction between hypergraphs aims to reveal the deep connections between nodes in different hypergraph structures, so as to understand and utilize the multi-hypergraph structure information. To this end, a novel feature interaction mechanism is proposed, such as Figure 2 (b) First, the linear transformation functions λ and φ are applied to the input features, and then the Einstein summation convention function is used Perform characterization extraction, the formula is:

[0030]

[0031] Among them H T is the adjacency matrix of the static hypergraph, H N and H M is the hypergraph adjacency matrix constructed by K-NN and K-means algorithms, H A The adjacency matrix H is N and H M The composite hypergraph adjacency matrix is ​​formed by the splicing operation. X1 and X2 are the input and hypergraph adjacency matrix H A and H T Output features after feature interaction.

[0032] Next, the γ(·) function is used to perform element-by-element subtraction of the feature vectors of each corresponding node in different hypergraphs to obtain a difference feature matrix, which is formulated as follows:

[0033] H B =γ(H A ,H T )=σ(H A -H T ) (8)

[0034] where σ(·) is a nonlinear activation function, H B is the feature interaction matrix of the hypergraph.

[0035] After that, we first use the linear function Transform the features and then feed the transformed features into the function With the interaction matrix H B After completing the feature interaction, the interactive features are finally fused with the feature X3 to obtain a more comprehensive interactive output. The formula is:

[0036]

[0037] Where α is a trainable scalar parameter and X4 is the final output feature, which contains the interaction information of all the above input features.

[0038] Dynamic Channel Refinement (DCR): The process of dynamic channel refinement is divided into three steps. First, dynamic modeling and channel refinement are completed by inferring the hypergraph structure, then the transformation function is used to transform the features, and finally the channel topology is aggregated. In this process, Q∈R N×N×C′ represents the channel correlation matrix, represents the channel aggregation matrix.

[0039] Dynamic modeling: Figure 2 As shown in (b), the input features are first mapped using linear transformation functions φ and ψ, and then these dimensionality reduction features are fed into the dynamic modeling function The dynamic modeling function is expressed as:

[0040]

[0041] where σ(·) is a nonlinear activation function, The core of this is to calculate the node features φ(x i ) and ψ(x j ) and generate channel-specific topological relationship representations.

[0042] Secondly, based on the modeling function, linear transformation ξ is used to increase the dimension of the features and learn the correlation Q∈R between channels. N×N×C′ , the formula is:

[0043]

[0044] Among them H T is the adjacency matrix of the static hypergraph, and the ξ function is a multidimensional tensor algorithm.

[0045] Finally, in order to further refine the correlation between channels, H with channel correlation Q T To refine it, the calculation formula is:

[0046]

[0047] in Represented as the channel topology after refinement, It is the Einstein summation convention.

[0048] Feature transformation: Using feature transformation aims to Convert the input into a high-level feature representation, such as Figure 2(b) is shown. To this end, a simple linear transformation is used to perform graph convolution, and the formula is as follows:

[0049]

[0050] in is the transformed feature, X represents the input feature matrix, and W represents the shared weight matrix, which is responsible for linearly combining the input features during the graph convolution process to extract feature representations rich in structural information.

[0051] Channel aggregation: In the case of channel topology and advanced features Under the premise of , the final output feature X4 can be obtained by aggregating the channel graph through the aggregation function, and its formula is:

[0052]

[0053] in and From and The cth channel, channel aggregation function According to the given channel topology and advanced features The output matrix X4 is obtained by performing channel aggregation on all connected channel graphs.

[0054] Channel-wise Differential Hypergraph Convolution (CD-HC): To achieve more accurate action recognition, we propose a channel-wise differential hypergraph convolution (CD-HC) with a structure similar to ST-HC. Unlike traditional HCN and the aforementioned ST-HCN, this model uses a channel-wise differential mechanism (CDM), which focuses more on revealing the changes in the relationship between channels and frames, showing significant differences in feature extraction and information utilization.

[0055] The core of the channel-wise differential hypergraph convolution method is to first perform differential operations on each channel of the input data to achieve refined extraction of skeleton point features, and then input the features into the model. Specifically, each skeleton point has a corresponding channel definition in the channel-wise differential HCN, and these channels represent the three-dimensional coordinate information of the skeleton point. By performing differential operations on these channels, multiple single channels containing the three-dimensional motion information of the skeleton points are obtained. The processing formula is:

[0056]

[0057] Where C is the number of input feature channels, X :,i and X :,j is the feature vector of the i-th and j-th channels, D :,i is the differential vector of the i-th channel.

[0058] S3 fuses the features of spatiotemporal dynamic hypergraph convolution flow and channel differential hypergraph convolution to obtain fused features;

[0059] Specifically, the four output features involved reveal the differentiated action characteristics of each node in the network. We aggregate them to obtain the correlation between these nodes and then construct a more comprehensive global feature representation. The corresponding calculation formula is:

[0060] X=Cat(X1,X2,X3,X4) (15)

[0061] Among them, Cat represents the splicing operation.

[0062] Finally, based on spatial convolution, a temporal convolution method for feature extraction of time series data is proposed, such as Figure 2 As shown in (c), the dilated temporal convolution (DTC) strategy is adopted to effectively improve the ability to extract features from time series data by flexibly adjusting the dilation coefficient.

[0063] S4 feeds the fused features into the classifier to obtain the action category.

[0064] Specifically, the fused features are fed into a classifier to obtain the final action category. In this way, the method of the present invention can more accurately identify skeleton actions. Experimental results on multiple datasets demonstrate the effectiveness and competitiveness of this method.

[0065] To evaluate the performance of the proposed model, we conducted extensive experiments on three datasets: NTU-RGB+D60, NTU-RGB+D120, and KineticsSkeleton400. First, we provide a detailed description of each dataset. Then, we conduct an ablation study on the NTU-RGB+D60 dataset to examine the contribution of each component to the model. Finally, we compare the proposed model with several state-of-the-art methods to verify its effectiveness.

[0066] Dataset:

[0067] NTURGB+D60: The NTURGB+D60 dataset is a large-scale dataset widely used in the field of action recognition. It contains 56,880 skeleton action sequences performed by 40 volunteers, covering 60 different action categories. Each action sample ensures that at most two subjects participate and is captured simultaneously by three cameras with different perspectives, thus providing rich three-dimensional skeleton joint information. The authors of this dataset recommend two evaluation benchmarks: (1) Cross-Subject (X-Sub) benchmark: The dataset is divided into two groups, with training data from 20 subjects and test data from the remaining 20 subjects, totaling 40,320 training samples and 26,560 test samples. (2) Cross-View (X-View) benchmark: The training samples come from camera views. Figure 2 and 3 , a total of 37,920, and the test samples come from cameras, a total of 18,960.

[0068] NTURGB+D120: The NTURGB+D120 dataset is an important extension of the NTURGB+D60 dataset, adding 57,367 new skeleton sequences and 60 new action categories, making it the largest 3D jointly annotated human action recognition dataset. The dataset consists of more than 114,000 skeleton action sequences, performed by 40 volunteers in 32 different settings, each setting representing a different position and background to enhance the generalization ability of the model in different environments. In order to evaluate the performance of the model, NTU-120 proposes two benchmark evaluation methods: (1) Cross-Subject (X-Sub) benchmark: Same as the X-Sub benchmark of NTU-60, the X-Sub evaluation divides the dataset into two groups, one for training and the other for testing. (2) Cross-Set (X-Set): The X-Set evaluation splits the training and test samples based on the name of the camera setting ID.

[0069] Kinetics-Skeleton: The Kinetics-Skeleton dataset is a large-scale human action recognition benchmark based on YouTube videos. It contains approximately 300,000 video clips covering up to 400 human behaviors, ranging from daily activities to sports scenes and complex human-computer interactions. The original Kinetics dataset only provides raw video clips without skeleton sequences. ST-GCN uses the publicly available OpenPose toolbox to obtain the positions of 18 joint points in each video frame and selects the two people with the highest average joint confidence for skeleton data extraction. The processed skeleton data is divided into 240,000 training clips and 20,000 validation clips.

[0070] Implementation details:

[0071] All experiments were conducted in the PyTorch deep learning framework, using stochastic gradient descent (SGD) with a momentum of 0.9 as the optimizer, cross entropy as the loss function for backpropagating gradients, and weight decay set to 0.0001. When training on NTURGB+D60 and NTURGB+D120, the learning rate was decayed at the 35th and 55th epochs, and training ended at the 80th epoch. For Kinetics-Skeleton, the learning rate was decayed at the 45th and 55th epochs, and training ended at the 70th epoch.

[0072] To verify the effectiveness of the proposed module and two-stream framework, the following experiments are conducted using the X-Sub benchmark on the NTURGB+D60 dataset.

[0073]

[0074] Table 1. Performance comparison of input data with different modes.

[0075] In multi-stream fusion, different output features are integrated, including J, B, and M (i.e., joints, skeletons, and motion). To evaluate the impact of each stream on model performance, three sets of control experiments were conducted, as shown in Table 1. The results show that the B (skeleton) modality has the most significant impact, with performance gains of up to 2.6%, further improving recognition performance.

[0076]

[0077] Table 2. Performance comparison when HFI, DCR, ST, or CD are gradually added or removed from DBC-HCN.

[0078] To validate the effectiveness of the dynamic channel refinement (DCR) module, hypergraph feature interaction (HFI) module, and the two-stream approach of spatiotemporal dynamic hypergraph convolution (ST) and channel-differential hypergraph convolution (CD), we conducted a component-by-component removal ablation study and analyzed the DBC-HCN model. The effects of these components were evaluated by monitoring the performance changes under B-mode, as shown in Table 2. A 0.6% performance drop was observed in the DBC-HCN w / o HFI model with HFI removed, confirming the critical role of hypergraph feature interaction between channels. Furthermore, the performance of the DBC-HCN w / o DCR model dropped by 2.0%, highlighting the importance of dynamic channel refinement in improving model performance. Meanwhile, the performance of the DBC-HCN w / o CD and DBC-HCN w / o ST models dropped by 1.4% and 1.2%, respectively, clearly demonstrating the importance of two-stream interaction in significantly enhancing model performance.

[0079] The DBC-HCN model is evaluated against state-of-the-art methods and its performance is analyzed. We test it on different benchmarks on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics-Skeleton datasets.

[0080]

[0081]

[0082] Table 3. Performance comparison of DBC-HCN with the state-of-the-art methods on the NTU RGB+D 60 dataset.

[0083]

[0084] Table 4. Performance comparison of DBC-HCN with the state-of-the-art methods on the NTU RGB+D 120 dataset.

[0085]

[0086]

[0087] Table 5. Performance comparison of DBC-HCN with the state-of-the-art methods on the Kinetics-Skeleton dataset.

[0088] Comparison data for the models is shown in Tables 3, 4, and 5. On all three datasets, our approach surpasses most existing techniques in almost all evaluation metrics. In particular, on the NTU RGB+D 120 dataset, models integrating joint and skeletal information achieve state-of-the-art performance. The DBC-HCN model outperforms the current hypergraph model, DST-HCN, by 0.6% and 0.5% on both evaluation benchmarks, respectively.

[0089] The X-Sub benchmark of the NTU RGB+D 120 dataset is used as a challenging test platform to evaluate the model performance. Figure 4 The confusion matrix of the BCD-HCN model on the NTU 120X-Sub benchmark is presented. This matrix details the model's classification performance across various action categories. The analysis results show that BCD-HCN achieves over 90% accuracy on 71 action categories, representing 59.17% of the total action categories, demonstrating that the model is capable of accurately identifying actions in most situations. Furthermore, the model achieves over 80% accuracy on 97 action categories, representing 81.83% of the total action categories. This statistic further demonstrates the high accuracy and stability of BCD-HCN in action recognition tasks.

[0090] Among the 120 action categories, the model performed particularly well in recognizing the actions "taking off a jacket," "jumping up," "nodding," "stumbling," and "cheers and drinking." The high accuracy of these actions may be attributed to their distinct visual features and the model's effective capture of these features. In contrast, the "staple" action had the lowest recognition accuracy.

[0091] The above disclosure is only a preferred embodiment of the skeleton action recognition method based on graph convolutional network of the present invention. Of course, it cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that implementing all or part of the processes of the above embodiment and making equivalent changes in accordance with the claims of the present invention still fall within the scope of the invention.

Claims

1. A skeleton action recognition method based on graph convolutional network, characterized in that: The following steps are involved: Preprocess the human skeleton data to obtain the input feature map; The input feature map is fed into the spatiotemporal dynamic hypergraph convolution flow and the channel differential hypergraph convolution flow for feature extraction; The features of spatiotemporal dynamic hypergraph convolution flow and channel differential hypergraph convolution are fused to obtain fused features; The fused features are fed into the classifier to obtain the action category.

2. The skeleton action recognition method based on graph convolutional network according to claim 1, characterized in that The spatiotemporal dynamic hypergraph convolution flow includes a spatial convolutional network and a dilated temporal convolution module. The spatial convolutional network includes a dynamic channel refinement module, a hypergraph feature interaction module and a feature aggregation module.

3. The skeleton action recognition method based on graph convolutional network according to claim 2, characterized in that The dynamic channel refinement module includes dynamic modeling, feature transformation and channel aggregation steps to generate channel-specific topological relationship representation and aggregate channel graphs to obtain the final output features.

4. The skeleton action recognition method based on graph convolutional network according to claim 2, characterized in that The hypergraph feature interaction module extracts features through linear transformation and Einstein summation convention function, generates a difference feature matrix, and obtains comprehensive interactive output through feature interaction.

5. The skeleton action recognition method based on graph convolutional network according to claim 1, characterized in that The channel-differential hypergraph convolutional flow processes the input feature map through a channel-differential mechanism to learn high-level representations of differential features.

Images