Skeleton action recognition method based on feature pyramid

Abstract

The invention discloses a skeleton action recognition method based on a feature pyramid, and the method comprises the steps: inputting a to-be-detected skeleton sequence into a feature extraction network composed of a preset number of pyramid graph convolution blocks, and obtaining the multi-scale features of a human skeleton; then, performing frame fusion and individual fusion on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector; and finally, performing linear projection change on the channel number dimension vector, and obtaining an identification result through a classification function. According to the method, the representation capability of the model is enhanced by capturing the multi-scale spatial features of the skeleton sequence, the skeleton can be effectively compressed into a plurality of low-order sub-images through the attention-based image pooling module, and compared with the prior art, the highest recognition accuracy is achieved.

Description

Technical Field

[0001] The present application belongs to the technical field of action recognition, and in particular, relates to a skeleton action recognition method based on feature pyramid. Background Art

[0002] Skeleton-based action recognition plays an important role in computer vision, aiming to recognize and understand various actions through human postures (such as skeleton sequences). Compared with traditional image- or video-based methods, skeleton-based methods show stronger robustness and can achieve reliable action recognition in complex environments such as illumination changes, occlusions, and background noise.

[0003] In recent years, there has been increasing interest in using graph convolutional networks (GCNs) to process skeleton data. GCNs are able to preserve the topological structural features between nodes when processing graph data, which means that the convolution operation considers not only the node features but also the connection relationships between nodes. The typical approach is to convert the human skeleton data into a structured graph, with the skeleton joints as the nodes of the graph and the relationships between the joints as the edges of the graph. Subsequently, GCNs use this structured feature to process skeleton data by propagating information between nodes, thereby improving the capture of deep features in action sequences.

[0004] However, existing methods usually emphasize global features and ignore local, fine-grained representations, limiting the performance of the model. Summary of the invention

[0005] The purpose of this application is to provide a skeleton action recognition method based on feature pyramid to overcome the problems raised in the above background technology.

[0006] In order to achieve the above purpose, the technical solution of this application is as follows: A skeleton action recognition method based on feature pyramid, comprising: Step 1: Input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton; Each pyramid graph convolution block performs the following operations: The input features of the pyramid graph convolution block are respectively passed through three convolution layers to obtain the first features, second features, and third features of different scales; The first feature remains unchanged, and the attention-based graph pooling operation is performed on the second feature and the third feature to obtain the corresponding first skeleton feature, second skeleton feature and third skeleton feature; The first skeleton feature, the second skeleton feature, and the third skeleton feature respectively extract the relationship features between the joints through graph convolution to obtain the corresponding first graph feature, the second graph feature, and the third graph feature; Through the graph unpooling operation, the second graph feature and the third graph feature are restored to the same shape as the first graph feature; Add the first image feature to the second image feature and the third image feature after image depooling to obtain a fused feature; Perform temporal convolution on the fused features to obtain multi-scale features; Step 2: Frame fusion and individual fusion are performed on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector; Step 3: After linear projection of the channel number dimension vector, the recognition result is obtained through the classification function.

[0007] Furthermore, the attention-based graph pooling operation includes: The second feature or the third feature is used as the input feature, and the input feature is linearly changed to obtain the query matrix and the key matrix respectively; Perform a dot multiplication operation on the query matrix and the transposed key matrix to obtain the attention matrix between joints; Use the ReLU activation function on the attention matrix between joints and sum them row by row to get the weight score vector of each joint; Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton; Extract the rows and columns corresponding to the joint point numbers from the input features to obtain the pooled skeleton features.

[0008] Furthermore, the frame fusion is a summation operation performed in the time dimension.

[0009] Furthermore, the individual fusion is to perform a summation operation on the node dimension.

[0010] Furthermore, the step of restoring the second image feature and the third image feature to the same shape as the first image feature through the image unpooling operation includes: Construct an initial matrix of all zeros with the same size as the first image feature; The eigenvalues ​​corresponding to the joint point numbers in the second graph features and the third graph features are filled into the initial matrix to obtain the second graph features and the third graph features after graph depooling.

[0011] Furthermore, the temporal convolution includes four temporal convolution branches, wherein the convolution kernel size is Two-dimensional atrous convolution with dilation coefficients of 1, 2, 3, and 4 respectively. The results of the four temporal convolution branches are finally connected together to obtain multi-scale features.

[0012] This application proposes a skeleton action recognition method based on feature pyramid, and innovatively proposes a feature pyramid graph convolutional network (FP-GCN), which enhances the representation ability of the model by capturing the multi-scale spatial features of the skeleton sequence. Specifically, an attention-based graph pooling module is proposed, which can effectively compress the skeleton into multiple low-order subgraphs. These subgraphs serve as spatial representations of the skeleton at the corresponding level, reflecting local and fine-grained representations. The original skeleton and these subgraphs are combined to form a feature pyramid, in which the joints of each layer are unfolded in the same semantic space. In addition, a graph de-pooling module is introduced to restore the pooled subgraphs to their original topological structure. Compared with the existing technical solutions, this application achieves the highest recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flowchart of the skeleton action recognition method based on feature pyramid in this application.

[0014] Figure 2 Schematic diagram of the feature pyramid graph convolutional network structure of this application.

[0015] Figure 3 This is a schematic diagram of an attention-based graph pooling module according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] In one embodiment, Figure 1 As shown in FIG. 1 , a skeleton action recognition method based on feature pyramid is proposed, including: Step S1: input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton.

[0018] This embodiment proposes a feature pyramid graph convolutional network, such as Figure 2 As shown, it is used for skeleton action recognition. The feature extraction network includes a preset number of pyramid graph convolution blocks, and the specific number is determined by experiments. In a specific embodiment, the feature extraction network includes ten pyramid graph convolution (Pyramid GC) blocks, and the number of output channels of each block is 64, 64, 64, 64, 128, 128, 128, 256, 256 and 256 respectively, and the feature output by the previous pyramid graph convolution block is used as the input of the next pyramid graph convolution block.

[0019] The input features are processed in each pyramid graph convolution block as follows: Step 1.1: The input features of the pyramid graph convolution block are passed through three convolution layers to obtain the first features, second features, and third features of different scales.

[0020] The skeleton sequence to be detected , where C represents the number of channels of the joint, and The first feature is projected into the high-dimensional space and is respectively the number of frames of the skeleton sequence and the number of joint points of a single frame skeleton. , Second feature and the third feature : in, Represents the two-dimensional convolution of the i-th scale, where i=1,2,3.

[0021] It should be noted that the features in this embodiment correspond to the feature diagram in the figure, and will not be described in detail below.

[0022] Step 1.2: The first feature remains unchanged, and attention-based graph pooling operations are performed on the second and third features to obtain the corresponding first skeleton features, second skeleton features, and third skeleton features.

[0023] Specifically, the first feature Remain unchanged and directly used as the first skeleton feature, still expressed as .

[0024] Second feature or the third characteristic As input features, it is input to the attention-based graph pooling module, such as Figure 3 As shown, perform the following operations: right Perform linear changes to obtain the query matrix and key matrix : in, and are trainable weights, and are the trainable offsets, respectively.

[0025] Next, With the transposed Perform a dot multiplication operation to obtain the attention matrix between joints : .

[0026] Attention matrix between joints After using the ReLU activation function, sum up by row to get the weight score vector of each joint: in, represents the row sum function, For the The weight score vector of the joint points under the scale.

[0027] Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton : in, Indicates the number of selected joint points, Express After sorting the elements in the elements.

[0028] Finally, extract the joint point number from the input feature The corresponding rows and columns are pooled to obtain the feature map, which is the skeleton feature. and the third feature , input to the attention-based graph pooling module, and obtain the second skeleton features respectively and the third skeleton feature .

[0029] The first skeleton feature, the second skeleton feature, and the third skeleton feature describe the local topological relationship of the skeleton joints, and are stacked to form a skeleton pyramid structure [ , , ].

[0030] Step 1.3: The first skeleton feature, the second skeleton feature, and the third skeleton feature respectively extract the relationship features between the joints through graph convolution to obtain the corresponding first graph feature, the second graph feature, and the third graph feature.

[0031] Specifically, in this embodiment, the first skeleton feature, the second skeleton feature, and the third skeleton feature are respectively input into the corresponding graph convolution modules to perform the graph convolution operation. Generated skeleton features in three scale spaces , and The original skeleton topology and two local skeleton topologies are represented respectively, forming a feature pyramid structure. These three data are then input into three independent graph convolution modules to enhance the representation ability of skeleton features. The relationship features between joints are extracted through graph convolution to obtain high-dimensional graph features. , the second graph features And the third figure features ,in The number of joints in the input skeleton is the same as the number of joints in the input skeleton.

[0032] The graph convolution operation realizes the aggregation of spatial features. The graph convolution operation is a relatively mature technology in this field and will not be described in detail here.

[0033] Step 1.4: Through the image unpooling operation, the second image feature and the third image feature are restored to the same shape as the first image feature.

[0034] Specifically, the graph depooling operation of this embodiment is performed in the graph depooling module. In the graph depooling module, according to the joint point sequence number , ,Will , Input to the graph depooling module to restore the pooled features and ensure that the final features are consistent with the first graph features The number of joint points is the same. That is, construct a feature The same size of the initial matrix of all zeros, and Center point number The corresponding eigenvalues ​​are filled into the initial matrix. and Recovery and The topological form with the same number of joints obtains the second graph feature after graph depooling And the third figure features So far, , and It is a three-dimensional matrix of the same shape, and the dimensions represent the number of channels, frames, and joints.

[0035] The graph depooling module restores the feature graph of each layer of the pyramid to the original number of nodes. Step 1.5: Add the first image feature to the second image feature and the third image feature after image depooling to obtain a fused feature.

[0036] In this step, the first image feature The second image features after unpooling And the third figure features , add them together to get the fused feature Y. This step fuses these restored feature maps to construct a multi-scale spatial feature map.

[0037] Step 1.6: Perform temporal convolution on the fused features to obtain multi-scale features.

[0038] Perform temporal convolution (TCN) on the fused feature Y, that is, perform multi-scale two-dimensional hole convolution operations on the fused feature in the time dimension. The convolution kernel will slide along the time axis, and convolve the input data to obtain the skeleton data after extracting the temporal features. The convolution operation can capture the dependencies between adjacent frames and long-distance frames, and output a multi-scale feature map Z. In this step, the multi-scale spatial feature map passes through the temporal convolution network (TCN) module to complete the temporal feature aggregation.

[0039] The temporal convolution in this embodiment includes four temporal convolution branches, where the convolution kernel sizes are all Two-dimensional dilated convolution with dilation coefficients of 1, 2, 3, and 4. Convolution operations are performed on the frames [t−1, t, t+1], [t−2, t, t+2], [t−3, t, t+3], [t−4, t, t+4] of the skeleton sequence, where t is the current frame, effectively capturing features in different time ranges while increasing the receptive field through dilated convolution. The results of the four temporal convolution branches are finally connected together to obtain multi-scale features.

[0040] It should be noted that this application will also and Independent output for independent constraints at each scale feature.

[0041] Step S2: Frame fusion and individual fusion are performed on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector.

[0042] Specifically, the multi-scale features output by the feature extraction network are frame fused, that is, summed in the time dimension. Individual fusion is the summation operation in the node dimension. This step compresses the multi-scale feature matrix into a vector u with the number of channels, summarizing the temporal and spatial information of all joints in the entire sequence.

[0043] Step S3: After linear projection transformation of the channel number dimension vector, the recognition result is obtained through the classification function.

[0044] This step performs a linear projection transformation on the vector u and maps it to a vector v with the dimension of the action category. This process is implemented through a fully connected layer, which converts the high-dimensional feature vector u into a vector v representing each action category, preparing for the classification task.

[0045] The vector v is processed by the softmax function to obtain the confidence of each action category. The softmax function converts the output value after linear transformation into a probability distribution so that the sum of the confidence of each action category is 1, which represents the normalized confidence of each action category. The largest value among the confidences is selected as the predicted action category. The action category corresponding to the maximum confidence is the final prediction result of the model.

[0046] In this embodiment, after passing through all pyramid GC blocks and focusing on the multi-scale features of the human skeleton, the network uses global average pooling to compress features in the frame fusion layer and the human fusion layer. Finally, the model predicts the action category using the softmax function.

[0047] The experimental data of this application show that the method of this application has better interaction detection accuracy than other methods in the prior art. The experimental data are shown in Tables 1 and 2 below:

[0048] Table 1 Table 1 shows the comparison of the skeleton action recognition accuracy of the technical solution of the present application with the existing technical methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets. The results show that the method of the present application can effectively use fine-grained features to accurately recognize skeleton actions. X-Sub and X-View respectively represent different ways of dividing the dataset.

[0049] Table 2 shows the accuracy of the method of the present application on the NW-UCLA dataset compared with the accuracy of other prior art methods on the dataset.

[0050] Table 2 Table 2 shows that compared with the prior art, the accuracy of the technical solution of the present application has been significantly improved, by at least 1.1%.

[0051] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A skeleton action recognition method based on feature pyramid, characterized in that: The skeleton action recognition method based on feature pyramid includes: Step 1: Input the skeleton sequence to be detected into a feature extraction network composed of a preset number of pyramid graph convolution blocks to obtain multi-scale features of the human skeleton; Each pyramid graph convolution block performs the following operations: The input features of the pyramid graph convolution block are respectively passed through three convolution layers to obtain the first features, second features, and third features of different scales; The first feature remains unchanged, and the attention-based graph pooling operation is performed on the second feature and the third feature to obtain the corresponding first skeleton feature, second skeleton feature and third skeleton feature; The first skeleton feature, the second skeleton feature, and the third skeleton feature respectively extract the relationship features between the joints through graph convolution to obtain the corresponding first graph feature, the second graph feature, and the third graph feature; Through the graph unpooling operation, the second graph feature and the third graph feature are restored to the same shape as the first graph feature; Add the first image feature to the second image feature and the third image feature after image depooling to obtain a fused feature; Perform temporal convolution on the fused features to obtain multi-scale features; Step 2: Frame fusion and individual fusion are performed on the multi-scale features output by the feature extraction network to obtain a channel number dimension vector; Step 3: After linear projection of the channel number dimension vector, the recognition result is obtained through the classification function.

2. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The attention-based graph pooling operation includes: The second feature or the third feature is used as the input feature, and the input feature is linearly changed to obtain the query matrix and the key matrix respectively; Perform a dot multiplication operation on the query matrix and the transposed key matrix to obtain the attention matrix between joints; Use the ReLU activation function on the attention matrix between joints and sum them row by row to get the weight score vector of each joint; Sort and filter each element in the weight score vector to obtain the corresponding joint point number in the selected skeleton; Extract the rows and columns corresponding to the joint point numbers from the input features to obtain the pooled skeleton features.

3. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The frame fusion is a summation operation performed in the time dimension.

4. The skeleton action recognition method based on feature pyramid according to claim 1, characterized in that: The individual fusion is to perform a summation operation on the node dimension.

5. The method for skeleton action recognition based on feature pyramid according to claim 1, characterized in that: The step of restoring the second image feature and the third image feature to the same shape as the first image feature through the image depooling operation includes: Construct an initial matrix of all zeros with the same size as the first image feature; The eigenvalues ​​corresponding to the joint point numbers in the second graph features and the third graph features are filled into the initial matrix to obtain the second graph features and the third graph features after graph depooling.

6. The method for skeleton action recognition based on feature pyramid according to claim 1, characterized in that: The temporal convolution includes four temporal convolution branches, where the convolution kernel sizes are all Two-dimensional atrous convolution with dilation coefficients of 1, 2, 3, and 4 respectively. The results of the four temporal convolution branches are finally connected together to obtain multi-scale features.

Images