Human skeleton action recognition method based on spatial feature enhancement

Abstract

The application discloses a human skeleton action recognition method based on spatial feature enhancement, combines a designed spatial aggregation module with a spatial dynamic channel topology detailed graph convolution, combines a human body graph structure into a model of a multilayer perceptron, so as to meet specific field requirements of human skeleton-based action recognition, allow local and global spatial interaction, and enhance feature extraction of a spatial dimension. In addition, a multi-branch time convolution module is used for time modeling to extract multi-scale skeleton features, and human skeleton posture features processed through space and time are input into a full connection layer to obtain a final action recognition result. The application improves the accuracy of the human skeleton-based action recognition method and has strong robustness in aspects of scale differences of different human bodies, limb correlations of different actions and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method for human skeleton motion recognition based on spatial feature enhancement. Background Technology

[0002] Human skeleton motion recognition refers to the process of detecting and identifying the positions of key joints of the human body from images or videos using computer vision technology, and inferring human movements. As a crucial component and one of the most active research topics in computer vision, motion recognition has been studied for decades. Because actions can be used by humans to process tasks and express emotions, motion recognition can be used in a wide range of applications. With the continuous advancement of deep learning and artificial intelligence technologies, this technology has made significant progress in recent years, and this research direction is of great importance to many fields.

[0003] Human skeleton-based motion recognition relies on human pose estimation technology. Its basic principle is to extract the spatial location information of key points (such as head, shoulders, elbows, knees, etc.) from videos or images, thereby constructing a skeleton structure that reflects the human pose. This skeleton can be represented by a graph, where nodes represent key points and edges represent the connections between them.

[0004] With the increasing development of deep learning methods and their impressive performance in most other existing computer vision tasks, such as image classification, object detection, semantic segmentation, and pose estimation, deep learning methods for human pose recognition of skeleton sequences using recurrent neural networks (RNNs), convolutional neural networks (CNNs), and graph convolutional networks (GCNs) have also emerged. Although graph convolutional neural networks can learn the topological structure of human skeleton joints, the adjacency matrix in graph convolution is predefined and cannot effectively aggregate the spatial correlations between distant adjacent joints in the human structure. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses a human skeleton motion recognition method based on spatial feature enhancement, which solves the problem that current network models cannot effectively aggregate the spatial correlation between adjacent, distant joints in the human body structure.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] The human skeleton motion recognition method based on spatial feature enhancement includes the following steps:

[0008] S1. Obtain video sequence data of human skeleton joints, and process the human skeleton data by subtracting adjacent joints in the same frame and subtracting the same joints in adjacent frames, etc., to process the original human skeleton data into four different modalities of data: joints, bones, joint motion, and bone motion.

[0009] S2. Use a spatial partitioning strategy to define the adjacency matrix of the human skeleton joints, dividing the nodes into three parts, which will be used in the subsequent graph convolution space modeling process.

[0010] S3. In the process of feature extraction in the spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and use the channel-specific correlation of each channel to refine and model the channel topology.

[0011] S4. The human skeleton features processed by convolution of the channel topology refinement graph are input into the spatial aggregation module Mixer. The graph structure of the human body is combined into the multilayer perceptron model to meet the specific domain requirements of human skeleton-based action recognition. At the same time, it allows local and global spatial interaction and enhances the feature extraction of spatial dimensions.

[0012] S5. Input the human skeleton features modeled through spatial dimension into a multi-branch temporal convolutional network to extract features in the temporal dimension;

[0013] S6. The pose features extracted through spatial and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of a single modality.

[0014] S7. The results obtained from the four different modal data are weighted and fused to obtain the final action recognition result of the model.

[0015] In some embodiments, in S1, the processing of joint flow modal data is performed according to the following steps:

[0016] (1) Align the center joint of the human body in the first frame of the original human skeleton joint data with the origin of the 3D Cartesian coordinate system.

[0017] (2) Rotate all skeleton joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system.

[0018] Skeletal flow modal data is obtained by subtracting adjacent joint points in the same frame from the joint flow modal data, with the central joint point as the origin.

[0019] Joint motion flow modal data and skeletal motion flow modal data are obtained by subtracting the same joint points between adjacent frames of the corresponding modal data stream.

[0020] Furthermore, in S2, a spatial partitioning strategy is used to define the adjacency matrix of the human skeleton joints, dividing the nodes into three parts: the joint itself, neighboring nodes that are closer to the central joint than the joint itself, and other neighboring nodes of the current joint.

[0021]

[0022] Where r i It is the average distance from the center joint to joint i across all frames.

[0023] Furthermore, in S3, the channel topology refinement graph convolution dynamically and effectively models the channel topology. CTR-GC does not learn the topology of different channels independently, but rather learns it in a refined manner. Specifically, CTR-GC learns both the shared topology and channel-specific correlations simultaneously. The shared topology is a parameterized adjacency matrix that serves as a topological prior for all channels, providing general correlations between vertices. Channel-specific correlations are dynamically inferred for each sample, capturing the relationships between vertices within each channel. By refining the shared topology using channel-specific correlations, the separate modeling of each channel's topology is avoided, and some additional parameters are introduced, significantly reducing the difficulty of modeling the channel topology.

[0024] Topological shared graph convolution utilizes dynamic weights W for feature transformation and through a ij Aggregate v i The representations of the adjacent vertices are used to update their representations z. i Its formula is

[0025]

[0026] Dynamic topology-shared graph convolution dynamically infers the topology, thus exhibiting better generalization ability; akij represents the dynamic topological relationship between incises vi and vj, and depends on the input samples; the features of incises vi are represented as xi; the formula for dynamic topology-shared GC is:

[0027]

[0028] Where k represents the index of the input sample. and Let be the input and output features of the k-th sample; W is the dynamic weight. Finally, the channel topology R∈R is obtained by refining the shared topology A using channel-specific correlation Q. N×N×C′ :

[0029]

[0030] Here, α is a trainable scalar for adjusting the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

[0031] Furthermore, in S4, the Mixer module mainly consists of two blocks: Joint-Mlp-Block and Channel-Mlp-Block. The human skeleton features, after convolutional spatial modeling of the channel topology refinement graph, are input into the Mixer module. After format conversion, the human skeleton features undergo Layer Normalization, then are input into the Joint-Mlp-Block with residual connections for keypoint correlation modeling. After the same Layer Normalization, they are transposed and input into the Channel-Mlp-Block with residual connections for channel feature correlation modeling at the same keypoint. Finally, the processed human skeleton features are transposed, normalized using Layer Normalization, and output to a multi-branch temporal convolutional network for temporal feature extraction.

[0032] Furthermore, in S5, a multi-branch temporal convolutional network is used for temporal modeling. This network consists of four branches: a 1×1 convolutional branch, a max-pooling branch, and two one-dimensional temporal convolutions with kernels of 3 and dilation rates of 1 and 2, respectively. Each feature group is processed by a single branch, and the outputs of these four branches are concatenated and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network. This temporal modeling approach not only improves temporal modeling capabilities but also saves computational cost and parameters by reducing the channel width of each branch.

[0033] Furthermore, in S6, the pose features extracted through spatial and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of a single modality.

[0034] Furthermore, in section 7, we weighted and fused the results from four different modalities to obtain the final action recognition result of the model. The data flow fusion problem can be viewed as a function optimization process, with the optimal fusion effect as the objective function and each flow model as the decision variable, constructing a nonlinear programming equation based on fusion. Here, we use the control variable method. To fully utilize the fusion weights of the four flows for comparison, we performed a quantization process, incorporating initial values ​​greater than 0 into the constraint part of the model to reflect the importance of different flows.

[0035] The present invention provides a human skeleton motion recognition device based on spatial feature enhancement, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to perform operations according to the instructions to execute the steps of the method.

[0036] The present invention provides an apparatus comprising: a processor; a computer program; and the steps of the method performed when the computer program is processed.

[0037] The beneficial effects of the present invention are as follows: The human skeleton action recognition method based on spatial feature enhancement of the present invention improves the accuracy of skeleton-based human action recognition tasks without significantly increasing the parameters of the network model, and reduces the training time of the network model and improves the inference speed of the network model.

[0038] To verify the effectiveness of the spatial feature enhancement aggregation module in this invention, a Mixer spatial aggregation module and a multi-branch temporal convolutional neural network were added to the CTR-GCN basic network structure. The recognition rates on the NTU-RGB+D and NTU-RGB+D 120 datasets are shown in the table below:

[0039]

[0040] Attached Figure Description

[0041] Figure 1 This is a step framework diagram of the human skeleton motion recognition method based on spatial feature enhancement in an embodiment of the present invention;

[0042] Figure 2 This is a network model diagram of the human skeleton motion recognition method based on spatial feature enhancement in an embodiment of the present invention;

[0043] Figure 3 This is a temporal modeling diagram of the human skeleton motion recognition method based on spatial feature enhancement in this embodiment of the invention. Detailed Implementation

[0044] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0045] Example 1

[0046] like Figure 1 As shown, the human skeleton motion recognition method based on spatial feature enhancement includes the following steps:

[0047] S1. Obtain video sequence data of human skeleton joints, and process the human skeleton data by subtracting adjacent joints in the same frame and subtracting the same joints in adjacent frames, etc., to process the original human skeleton data into four different modalities of data: joints, bones, joint motion, and bone motion.

[0048] In S1, the joint flow modal data are processed according to the following steps:

[0049] (1) Align the center joint of the human body in the first frame of the original human skeleton joint data with the origin of the 3D Cartesian coordinate system.

[0050] (2) Rotate all skeleton joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system.

[0051] Skeletal flow modal data is obtained by subtracting adjacent joint points in the same frame from the central joint point as the origin.

[0052] The corresponding motion flow modal data is obtained by subtracting the same joint points between adjacent frames of joint flow modal data and skeletal flow modal data.

[0053] S2. Use a spatial partitioning strategy to define the adjacency matrix of the human skeleton joints, dividing the nodes into three parts, which will be used in the subsequent graph convolution space modeling process.

[0054] In S2, a spatial partitioning strategy is used to define the adjacency matrix of human skeleton joints, dividing the nodes into three parts: the joint itself, neighboring nodes that are closer to the central joint than the joint itself, and other neighboring nodes of the current joint.

[0055]

[0056] Where r i It is the average distance from the center joint to joint i across all frames.

[0057] S3, such as Figure 2 As shown, in the feature extraction process of the spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and use the channel-specific correlation of each channel to refine and model the channel topology.

[0058] In S3, the four pre-processed modal data are fed into the model for spatial and temporal modeling. The channel topology structure is dynamically learned through channel topology refinement graph convolution, and the correlation between skeleton joints is constructed.

[0059] Topological shared graph convolution utilizes dynamic weights W for feature transformation and through a ij Aggregate v i The representations of the adjacent vertices are used to update their representations z. i Its formula is

[0060]

[0061] Dynamic topology-shared graph convolution dynamically infers the topology, thus exhibiting better generalization ability; akij represents the dynamic topological relationship between incises vi and vj, and depends on the input samples; the features of incises vi are represented as xi; the formula for dynamic topology-shared GC is:

[0062]

[0063] Where k represents the index of the input sample. and Let be the input and output features of the k-th sample; W is the dynamic weight. Finally, the channel topology R∈R is obtained by refining the shared topology A using channel-specific correlation Q. N×N×C′ :

[0064]

[0065] Here, α is a trainable scalar for adjusting the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

[0066] S4, such as Figure 2 As shown, the human skeleton features processed by channel topology refinement graph convolution are input into the spatial aggregation module Mixer, which combines the graph structure of the human body into the multilayer perceptron model to meet the specific domain requirements of human skeleton-based action recognition, while allowing local and global spatial interaction and enhancing feature extraction in spatial dimensions. In S4, the human skeleton features are first normalized by Layer Normalization after format conversion, and then input into Joint-Mlp-Block with residual connections to model the correlation between joints. After the same Layer Normalization normalization process, they are transposed and input into Channel-Mlp-Block with residual connections to model the correlation of channel features at the same joint. Finally, the processed human skeleton features are transposed and normalized by Layer Normalization.

[0067] S5, such as Figure 3As shown, the human skeleton features modeled through spatial dimensions are input into a multi-branch temporal convolutional network for temporal feature extraction. In step S5, the spatially modeled human skeleton features are input into the multi-branch temporal convolutional network for temporal feature extraction. The multi-branch temporal convolutional network consists of four branches: a 1×1 convolutional branch, a max-pooling branch, and two one-dimensional temporal convolutions with kernels of 3 and dilation rates of 1 and 2, respectively. Each feature group is processed by a single branch, and the outputs of these four branches are concatenated and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network.

[0068] S6. The pose features extracted through spatial and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of a single modality.

[0069] S7. The results obtained from the four different modal data are weighted and fused to obtain the final action recognition result of the model.

[0070] Example 2

[0071] This embodiment provides a human skeleton motion recognition device based on spatial feature enhancement, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method according to Embodiment 1.

[0072] Example 3

[0073] This embodiment provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A human skeleton motion recognition method based on spatial feature enhancement, characterized in that, Includes the following steps: S1. Obtain video sequence data of human skeleton joints, and process the human skeleton data by subtracting adjacent joints in the same frame and subtracting the same joints in adjacent frames, so as to process the original human skeleton data into four different modalities: joints, bones, joint motion, and bone motion. S2. Use a spatial partitioning strategy to define the adjacency matrix of the human skeleton joints, dividing the nodes into three parts, which will be used in the subsequent graph convolution space modeling process. S3. In the process of feature extraction in the spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and use the channel-specific correlation of each channel to refine and model the channel topology. S4. The human skeleton features processed by channel topology refinement graph convolution are input into the spatial aggregation module Mixer, which combines the graph structure of the human body into the multilayer perceptron model to meet the needs of action recognition based on human skeletons, while allowing local and global spatial interaction and enhancing feature extraction in the spatial dimension; S4 In this module, the Mixer consists of two blocks: Joint-Mlp-Block and Channel-Mlp-Block. The human skeleton features, after being modeled in the convolutional space of the channel topology refinement graph, are input into the Mixer module. After format conversion, the human skeleton features are first normalized by Layer Normalization, and then input into the Joint-Mlp-Block with residual connections to model the correlation between joints. After the same Layer Normalization, they are transposed and input into the Channel-Mlp-Block with residual connections to model the correlation of channel features at the same joint. Finally, the processed human skeleton features are transposed and normalized by Layer Normalization and output to a multi-branch temporal convolutional network for temporal feature extraction. S5. The human skeleton features modeled through spatial dimension are input into a multi-branch temporal convolutional network for temporal feature extraction. In S5, a multi-branch temporal convolutional network is used for temporal modeling. The multi-branch temporal convolutional network consists of four branches: a 1×1 convolutional branch, a max pooling branch, and two one-dimensional temporal convolutions with a kernel size of 3 and dilation rates of 1 and 2, respectively. Each feature group is processed by a single branch, and the outputs of these four branches are connected together and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network. S6. The pose features extracted through spatial and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of a single modality. S7. The results obtained from the four different modal data are weighted and fused to obtain the final action recognition result of the model.

2. The human skeleton motion recognition method based on spatial feature enhancement according to claim 1, characterized in that, In S1, the processing of joint flow modal data follows these steps: S11: Align the central joint of the human body in the first frame of the original human skeleton joint data with the origin of the 3D Cartesian coordinate system. S12: Rotate all skeletal joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system; S13: Skeletal flow modal data is obtained by subtracting adjacent joint points in the same frame from the joint flow modal data with the central joint point as the origin; S14: Joint motion flow modal data and skeletal motion flow modal data are obtained by subtracting the same joint points between adjacent frames of the corresponding modal data stream.

3. The human skeleton motion recognition method based on spatial feature enhancement according to claim 1, characterized in that, In S2, a spatial partitioning strategy is used to define the adjacency matrix of the human skeleton joints, dividing the nodes into three parts: the joint itself, the neighboring nodes that are closer to the central joint than the joint itself, and the other neighboring nodes of the current joint. ； in It is the average distance from the center joint to joint i across all frames.

4. The human skeleton motion recognition method based on spatial feature enhancement according to claim 1, characterized in that, In S3, the channel topology refinement graph convolution dynamically and effectively models the channel topology; The shared topology is a parameterized adjacency matrix that serves as the topological prior for all channels and provides general correlations between vertices. For each sample, channel-specific correlations are dynamically inferred, and the relationships between vertices within each channel are captured; topological shared graph convolution utilizes dynamic weights W for feature transformation, and through a ij Aggregate v i The representations of the adjacent vertices are used to update their representations z. i Its formula is ； Dynamic topology-sharing graph convolution dynamically infers the topology, a k ij It is the key point v i v j The dynamic topological relationships between them, and depend on the input samples; key points v i The feature is represented as x i The formula for dynamic topology shared GC is: ； Where k represents the index of the input sample, and xkj and zkj are the input and output features of the k-th sample; W represents the dynamic weight; finally, the channel topology R∈R is obtained by refining the shared topology A using channel-specific correlation Q. 𝑁×𝑁×𝐶′ : ； Here, α is a trainable scalar for adjusting the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

5. The human skeleton motion recognition method based on spatial feature enhancement according to claim 1, characterized in that, In S6, the pose features extracted through spatial and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of a single modality.

6. The human skeleton motion recognition method based on spatial feature enhancement according to claim 1, characterized in that, In S7, the results obtained from four different modalities are weighted and fused to obtain the final action recognition result of the model; The control variable method is used; the initial values ​​of the four weights are greater than 0 and are included in the constraint part of the model to realize the importance of different flows.

7. A human skeleton motion recognition device based on spatial feature enhancement, characterized in that, It includes a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 6.

8. A device, characterized in that, include: processor; A computer program; when executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images