A skeleton action recognition method based on feature pyramid

Abstract

This invention discloses a method for skeleton action recognition based on a feature pyramid. The method inputs a skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolutional blocks to obtain multi-scale features of the human skeleton. The multi-scale features output by the feature extraction network are then subjected to frame fusion and individual fusion to obtain a channel-dimensional vector. Finally, the channel-dimensional vector is linearly projected and then subjected to a classification function to obtain the recognition result. By capturing the multi-scale spatial features of the skeleton sequence, the invention enhances the model's representational capabilities. Through an attention-based graph pooling module, the skeleton can be effectively compressed into multiple low-order subgraphs. Compared to existing technical solutions, the present invention achieves the highest recognition accuracy.

Description

Technical Field

[0001] The present application belongs to the field of action recognition technology, and in particular relates to a skeleton action recognition method based on feature pyramid. Background Art

[0002] Skeleton-based action recognition plays a crucial role in computer vision. It aims to identify and understand various actions based on human poses (e.g., skeleton sequences). Compared to traditional image- or video-based methods, skeleton-based methods exhibit greater robustness and can achieve reliable action recognition in complex environments such as illumination variations, occlusions, and background noise.

[0003] In recent years, there has been growing interest in using graph convolutional networks (GCNs) to process skeleton data. GCNs preserve the topological structure between nodes when processing graph data, meaning that convolution operations consider not only node features but also the connections between them. Typically, human skeleton data is converted into a structured graph, with the skeleton's joints serving as nodes and the relationships between them as edges. GCNs then leverage this structured nature to process skeleton data by propagating information between nodes, improving the capture of deep features in action sequences.

[0004] However, existing methods usually emphasize global features while neglecting local, fine-grained representations, limiting the performance of the model. Summary of the Invention

[0005] The purpose of this application is to provide a skeleton action recognition method based on feature pyramid to overcome the problems raised in the above background technology.

[0006] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0007] A skeleton action recognition method based on feature pyramid, comprising:

[0008] Step 1: Input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton;

[0009] Each pyramid graph convolution block performs the following operations:

[0010] The input features of the pyramid graph convolution block pass through three convolution layers to obtain the first features, second features, and third features of different scales;

[0011] The first feature remains unchanged, and the attention-based graph pooling operation is performed on the second and third features to obtain the corresponding first skeleton features, second skeleton features, and third skeleton features;

[0012] The first skeleton feature, the second skeleton feature, and the third skeleton feature are respectively used to extract the relationship features between joints through graph convolution to obtain the corresponding first graph feature, second graph feature, and third graph feature;

[0013] Through the graph unpooling operation, the second and third graph features are restored to the same shape as the first graph feature;

[0014] Add the first image feature to the second image feature after image depooling and the third image feature to obtain the fused feature;

[0015] Perform temporal convolution on the fused features to obtain multi-scale features;

[0016] Step 2: Perform frame fusion and individual fusion on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector;

[0017] Step 3: After linear projection of the channel number dimension vector, the recognition result is obtained through the classification function.

[0018] Furthermore, the attention-based graph pooling operation includes:

[0019] The second feature or the third feature is used as the input feature, and the input feature is linearly changed to obtain the query matrix and the key matrix respectively;

[0020] Perform a dot product operation on the query matrix and the transposed key matrix to obtain the attention matrix between joints;

[0021] Apply the ReLU activation function to the attention matrix between joints and sum it up row by row to obtain the weight score vector of each joint;

[0022] Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton;

[0023] Extract the rows and columns corresponding to the joint point numbers from the input features to obtain the pooled skeleton features.

[0024] Furthermore, the frame fusion is a summation operation performed in the time dimension.

[0025] Furthermore, the individual fusion is to perform a summation operation on the node dimension.

[0026] Furthermore, the step of restoring the second image feature and the third image feature to the same shape as the first image feature through the image unpooling operation includes:

[0027] Construct an initial matrix of all zeros with the same size as the first image feature;

[0028] The eigenvalues ​​corresponding to the joint point numbers in the second and third graph features are filled into the initial matrix to obtain the second and third graph features after graph depooling.

[0029] Furthermore, the temporal convolution includes four temporal convolution branches, wherein the convolution kernel size is Two-dimensional void convolution, and the expansion coefficients are 1, 2, 3 and 4 respectively. The results of the four temporal convolution branches are finally connected together to obtain multi-scale features.

[0030] This application proposes a skeleton action recognition method based on a feature pyramid, and innovatively proposes a feature pyramid graph convolutional network (FP-GCN). By capturing the multi-scale spatial features of the skeleton sequence, the model's representational ability is enhanced. Specifically, an attention-based graph pooling module is proposed, which can effectively compress the skeleton into multiple low-order subgraphs. These subgraphs serve as spatial representations of the skeleton at the corresponding level, reflecting local, fine-grained representations. The original skeleton and these subgraphs are combined to form a feature pyramid, in which the joints of each layer are unfolded in the same semantic space. In addition, a graph depooling module is introduced to restore the pooled subgraphs to their original topological structure. Compared with existing technical solutions, this application achieves the highest recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of the skeleton action recognition method based on feature pyramid in this application.

[0032] Figure 2 Schematic diagram of the feature pyramid graph convolutional network structure for this application.

[0033] Figure 3 Schematic diagram of the attention-based graph pooling module in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0035] In one embodiment, Figure 1 As shown in the figure, a skeleton action recognition method based on feature pyramid is proposed, including:

[0036] Step S1: Input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton.

[0037] This embodiment proposes a feature pyramid graph convolutional network, such as Figure 2As shown, it is used for skeletal action recognition. The feature extraction network includes a preset number of pyramid graph convolution blocks, the specific number of which is determined through experiments. In a specific embodiment, the feature extraction network includes ten pyramid graph convolution (GC) blocks, with the output channels of each block being 64, 64, 64, 64, 128, 128, 128, 256, 256, and 256, respectively. The features output by the previous GC block serve as the input to the next GC block.

[0038] The input features are processed as follows in each pyramid graph convolution block:

[0039] Step 1.1: The input features of the pyramid graph convolution block pass through three convolution layers to obtain the first features, second features, and third features of different scales.

[0040] The skeleton sequence to be detected , where C represents the number of channels of the joint, and The number of frames of the skeleton sequence and the number of joint points of a single frame skeleton are respectively input through three two-dimensional convolution layers with a convolution kernel of 1×1 to obtain the first feature projected into the high-dimensional space. , the second feature and the third characteristic :

[0041]

[0042] in, Represents the two-dimensional convolution of the i-th scale, where i=1,2,3.

[0043] It should be noted that the features in this embodiment correspond to the feature graph in the figure, and will not be described in detail below.

[0044] Step 1.2: The first feature remains unchanged, and attention-based graph pooling operations are performed on the second and third features to obtain the corresponding first skeleton features, second skeleton features, and third skeleton features.

[0045] Specifically, the first feature Remain unchanged and directly used as the first skeleton feature, still expressed as .

[0046] Second feature or the third characteristic As input features, it is input to the attention-based graph pooling module, such as Figure 3 As shown, perform the following operations:

[0047] right Perform linear changes to obtain query matrices and key matrix :

[0048]

[0049]

[0050] in, and are trainable weights, and are the trainable offsets, respectively.

[0051] Next, With the transposed Perform dot multiplication to obtain the attention matrix between joints :

[0052] .

[0053] Attention matrix between joints After using the ReLU activation function and summing up by row, we get the weight score vector of each joint:

[0054]

[0055] in, Represents a row-wise sum function, For the The weight score vector of the joint points under the scale.

[0056] Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton :

[0057]

[0058] in, Indicates the number of selected joint points, Express After sorting the elements in the elements.

[0059] Finally, extract the joint point number from the input feature The corresponding rows and columns are pooled to obtain the feature map, which is the skeleton feature. and the third characteristic , input to the attention-based graph pooling module, and obtain the second skeleton features respectively and the third skeleton feature .

[0060] The first skeleton feature, the second skeleton feature and the third skeleton feature describe the local topological relationship of the skeleton joints, and are stacked to form a skeleton pyramid structure [ , , ].

[0061] Step 1.3: The first skeleton feature, the second skeleton feature, and the third skeleton feature are respectively used to extract the relationship features between the joints through graph convolution to obtain the corresponding first graph feature, second graph feature, and third graph feature.

[0062] Specifically, this embodiment inputs the first skeleton feature, the second skeleton feature, and the third skeleton feature into their respective corresponding graph convolution modules to perform graph convolution operations.

[0063] Generated skeleton features in three scale spaces 、 and Represent the original skeleton topology and two local skeleton topologies respectively, forming a feature pyramid structure. Then these three data are input into three independent graph convolution modules to enhance the representation ability of skeleton features, and the relationship features between joints are extracted through graph convolution to obtain high-dimensional graph features. , the second graph feature and the third figure feature ,in The number of joints in the input skeleton is the same as the number of joints in the input skeleton.

[0064] Graph convolution operations realize the aggregation of spatial features. Graph convolution operations are relatively mature technologies in this field and will not be described in detail here.

[0065] Step 1.4: Through the image unpooling operation, the second image feature and the third image feature are restored to the same shape as the first image feature.

[0066] Specifically, the graph depooling operation of this embodiment is performed in the graph depooling module. In the graph depooling module, according to the joint point sequence number , ,Will 、 Input to the graph depooling module to restore the pooled features, ensuring that the final features are consistent with the first graph features The number of joint points is the same. That is, construct a feature with the first graph The same size of the initial matrix of all zeros, and Central joint number The corresponding eigenvalues ​​are filled into the initial matrix, and the and Recovery and The topological form with the same number of joints obtains the second graph feature after graph depooling and the third figure feature So far, 、 and It is a three-dimensional matrix with the same shape, where the dimensions represent the number of channels, frames, and joints.

[0067] The graph depooling module restores the feature map of each layer of the pyramid to the original number of nodes

[0068] Step 1.5: Add the first image feature to the second image feature after image depooling and the third image feature to obtain a fused feature.

[0069] In this step, the first image feature The second image feature after image depooling and the third figure feature , add them together to get the fused feature Y. This step fuses these recovered feature maps to construct a multi-scale spatial feature map.

[0070] Step 1.6: Perform temporal convolution on the fused features to obtain multi-scale features.

[0071] Temporal convolution (TCN) is performed on the fused features Y. This involves performing multi-scale, two-dimensional, dilated convolution operations on the fused features in the temporal dimension. The convolution kernel slides along the time axis, convolving the input data to produce skeleton data after extracting temporal features. This convolution operation captures dependencies between adjacent frames and between long-distance frames, outputting a multi-scale feature map Z. This step passes the multi-scale spatial feature map through the Temporal Convolutional Network (TCN) module to complete temporal feature aggregation.

[0072] The temporal convolution in this embodiment includes four temporal convolution branches, where the convolution kernel sizes are all Two-dimensional atrous convolution with dilation coefficients of 1, 2, 3, and 4, respectively. Convolution operations are performed on frames [t−1, t, t+1], [t−2, t, t+2], [t−3, t, t+3], and [t−4, t, t+4] of the skeleton sequence, where t is the current frame. This effectively captures features across different temporal ranges while increasing the receptive field through dilated convolution. The results of the four temporal convolution branches are ultimately concatenated to obtain multi-scale features.

[0073] It should be noted that this application will also and Independent output for independent constraints at each scale feature.

[0074] Step S2: perform frame fusion and individual fusion on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector.

[0075] Specifically, the multi-scale features output by the feature extraction network are subjected to frame fusion, i.e., summation on the temporal dimension. Individual fusion, i.e., summation on the node dimension, is performed. This step compresses the multi-scale feature matrix into a single vector u with the same number of channels, summarizing the temporal and spatial information of all joints in the entire sequence.

[0076] Step S3: After performing linear projection on the channel number dimension vector, the recognition result is obtained through the classification function.

[0077] This step performs a linear projection transformation on the vector u, mapping it to a vector v with the same dimensions as the action categories. This process is implemented through a fully connected layer, which converts the high-dimensional feature vector u into vectors v representing each action category, preparing for the classification task.

[0078] Apply the softmax function to vector v to obtain the confidence score for each action category. The softmax function converts the linearly transformed output into a probability distribution so that the sum of the confidence scores for each action category is 1, representing the normalized confidence score for each action category. The maximum confidence score is selected as the predicted action category. The action category corresponding to the maximum confidence score is the model's final prediction.

[0079] In this embodiment, after passing through all pyramid GC blocks and focusing on the multi-scale features of the human skeleton, the network uses global average pooling to compress features in the frame fusion layer and the human fusion layer. Finally, the model uses the softmax function to predict the action category.

[0080] The experimental data of this application show that the method of this application has better interaction detection accuracy than other methods in the prior art. The experimental data are shown in Tables 1 and 2 below:

[0081] Table 1

[0082]

[0083] Table 1 compares the accuracy of skeleton action recognition using the proposed method with existing methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets. The results demonstrate that the proposed method can effectively leverage fine-grained features for accurate skeletal action recognition. X-Sub and X-View represent different ways of partitioning the datasets.

[0084] Table 2 shows the accuracy of the method of the present application on the NW-UCLA dataset compared with the accuracy of other existing methods.

[0085] Table 2

[0086]

[0087] Table 2 shows that the accuracy of the technical solution of this application is significantly improved compared with the existing technology, with an improvement of at least 1.1%.

[0088] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A skeleton action recognition method based on feature pyramid, characterized in that: The skeleton action recognition method based on feature pyramid includes: Step 1: Input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton; Each pyramid graph convolution block performs the following operations: The input features of the pyramid graph convolution block pass through three convolution layers to obtain the first features, second features, and third features of different scales; The first feature remains unchanged, and the attention-based graph pooling operation is performed on the second and third features to obtain the corresponding first skeleton features, second skeleton features, and third skeleton features; The first skeleton feature, the second skeleton feature, and the third skeleton feature are respectively used to extract the relationship features between joints through graph convolution to obtain the corresponding first graph feature, second graph feature, and third graph feature; Through the graph unpooling operation, the second and third graph features are restored to the same shape as the first graph feature; Add the first image feature to the second image feature after image depooling and the third image feature to obtain the fused feature; Perform temporal convolution on the fused features to obtain multi-scale features; Step 2: Perform frame fusion and individual fusion on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector; Step 3: After linear projection of the channel number dimension vector, the recognition result is obtained through the classification function.

2. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The attention-based graph pooling operation includes: The second feature or the third feature is used as the input feature, and the input feature is linearly changed to obtain the query matrix and the key matrix respectively; Perform a dot product operation on the query matrix and the transposed key matrix to obtain the attention matrix between joints; Apply the ReLU activation function to the attention matrix between joints and sum it up row by row to obtain the weight score vector of each joint; Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton; Extract the rows and columns corresponding to the joint point numbers from the input features to obtain the pooled skeleton features.

3. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The frame fusion is a summation operation performed in the time dimension.

4. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The individual fusion is to perform a summation operation on the node dimension.

5. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The method of restoring the second image feature and the third image feature to the same shape as the first image feature through the image depooling operation includes: Construct an initial matrix of all zeros with the same size as the first image feature; The eigenvalues ​​corresponding to the joint point numbers in the second and third graph features are filled into the initial matrix to obtain the second and third graph features after graph depooling.

6. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The temporal convolution includes four temporal convolution branches, where the convolution kernel sizes are all Two-dimensional void convolution, and the expansion coefficients are 1, 2, 3 and 4 respectively. The results of the four temporal convolution branches are finally connected together to obtain multi-scale features.

Images