Human skeleton action recognition method based on spatial feature enhancement

Through the human skeleton action recognition method with spatial features enhancement, the channel topology is used to refine graph convolution and multi-branch time convolution network, combined with the Mixer module and the fully connected layer, the problem of difficulty in aggregating the spatial correlation of human structures is solved. Improved the accuracy of action recognition and model efficiency.

CN119007280AActive Publication Date: 2024-11-22NANJING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202410900464.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-22
Estimated Expiration
2044-07-05

AI Technical Summary

Technical Problem

Existing network models are difficult to effectively aggregate the spatial correlation between adjacent distant nodes in human structures, affecting the accuracy of human skeleton motion recognition.

Method used

Using the human skeleton action recognition method based on spatial feature enhancement, the adjacency matrix of the human skeleton nodes is defined through the spatial division strategy, the channel topology is used to refine graph convolution and multi-branch time convolution network, and combined with the Mixer module and the fully connected layer, extracted and fuse features of different modal data to improve identification accuracy.

Benefits of technology

Without increasing network model parameters, the accuracy of human movement recognition is improved, training time is reduced and inference speed is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007280A_ABST
    Figure CN119007280A_ABST
Patent Text Reader

Abstract

The invention discloses a human skeleton action recognition method based on spatial feature enhancement, which combines a designed spatial aggregation module with spatial dynamic channel topological refined graph convolution, and combines a graph structure of a human body into a model of a multi-layer perceptron so as to meet the specific field requirements of action recognition based on a human skeleton. And meanwhile, local and global space interaction is allowed, and feature extraction of spatial dimensions is enhanced. In addition, a multi-branch time convolution module is used in time modeling to extract multi-scale skeleton features, and human skeleton posture features after space and time processing are input to a full connection layer to obtain a final action recognition result. The accuracy of the motion recognition method based on the human body skeleton is improved, and the method has high robustness in the aspects of scale differences of different human bodies, limb correlation of different motions and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for human skeleton action recognition based on spatial feature enhancement. Background Art

[0002] Human skeleton action recognition refers to the process of using computer vision technology to detect and identify the positions of key joints of the human body from images or videos, and to infer human actions. As an extremely important component and the most active research topic in the field of computer vision, action recognition has been studied for decades. Since actions can be used by humans to handle affairs and express emotions, action recognition can be used in a wide range of application fields. With the continuous advancement of deep learning and artificial intelligence technology, this technology has made significant progress in recent years, and this research direction is of great significance to many fields.

[0003] Action recognition based on human skeleton relies on human posture estimation technology. Its basic principle is to extract the spatial position information of key points of the human body (such as head, shoulders, elbows, knees, etc.) from videos or images to construct a skeleton structure that reflects the human posture. This skeleton can be represented by a graph, where nodes represent key points and edges represent the connection relationship between them.

[0004] With the increasing development of deep learning methods and their impressive performance in most other existing computer vision tasks, such as image classification, object detection, semantic segmentation, pose estimation, etc., deep learning methods for human pose recognition from skeleton sequences using recurrent neural networks (RNNs), convolutional neural networks (CNNs), and graph convolutional networks (GCNs) have also emerged. Although graph convolutional neural networks can learn the topological structure of human skeleton joints, since the adjacency matrix in graph convolution is predefined, it cannot well aggregate the spatial correlation between adjacent and distant joints in the human structure. Summary of the invention

[0005] To solve the above problems, the present invention discloses a human skeleton action recognition method based on spatial feature enhancement to solve the problem that the current network model cannot well aggregate the spatial correlation between adjacent and distant joints in the human body structure.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The method for human skeleton action recognition based on spatial feature enhancement includes the following steps:

[0008] S1. Obtain video sequence data of human skeleton joints, perform processing such as subtracting adjacent joints in the same frame and subtracting the same joints in adjacent frames on the human skeleton data, and process the original human skeleton data into data of four different modes: joints, bones, joint motion, and bone motion;

[0009] S2. Use the space partitioning strategy to define the adjacency matrix of the human skeleton joints and divide the nodes into three parts for use in the subsequent graph convolution space modeling process;

[0010] S3. In the feature extraction process of the spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and refine the channel topology using the channel-specific correlation of each channel to model the channel topology.

[0011] S4. The human skeleton features processed by channel topology refinement graph convolution are input into the spatial aggregation module Mixer, and the human body graph structure is combined into the multi-layer perceptron model to meet the specific field requirements of action recognition based on the human skeleton, while allowing local and global spatial interaction to enhance feature extraction in the spatial dimension.

[0012] S5, inputting the human skeleton features modeled in the spatial dimension into a multi-branch temporal convolutional network to extract features in the temporal dimension;

[0013] S6, inputting the posture features extracted after spatial modeling and temporal modeling into the fully connected layer for decoding, and obtaining the prediction result of the action of the single modal data;

[0014] S7. Perform weighted fusion on the results obtained from the four different modal data to obtain the final action recognition result of the model.

[0015] In some embodiments, in S1, the processing of the joint flow modality data follows the following steps:

[0016] (1) aligning the central joint point of the person in the first frame of the original human skeleton joint point data with the origin of the 3D Cartesian coordinate system;

[0017] (2) Rotate all skeleton joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system.

[0018] The skeleton flow modal data is obtained by subtracting adjacent joint points in the same frame from the joint flow modal data with the central joint point as the origin.

[0019] The joint motion stream modal data and the skeleton motion stream modal data are obtained by subtracting the same joint points between adjacent frames of the corresponding modal data streams.

[0020] Furthermore, in S2, a space partitioning strategy is used to define the adjacency matrix of the human skeleton joints, and the nodes are divided into three parts, namely the joint itself, the neighboring nodes that are closer to the central joint than the joint itself, and other neighboring nodes of the current joint.

[0021]

[0022] where r i is the average distance from the center joint point to joint i over all frames.

[0023] Furthermore, in S3, the channel topology refinement graph convolution dynamically and effectively models the channel topology. Instead of learning the topology of different channels independently, CTR-GC learns the channel topology in a refined manner. Specifically, CTR-GC simultaneously learns shared topology and channel-specific correlations. The shared topology is a parameterized adjacency matrix that serves as a topological prior for all channels and provides universal correlations between vertices. Channel-specific correlations are dynamically inferred for each sample, and the relationship between vertices within each channel is captured. By refining the shared topology using channel-specific correlations, it avoids modeling the topological structure of each channel separately and introduces some additional parameters, which significantly reduces the difficulty of modeling the channel topology.

[0024] Topologically shared graph convolution uses dynamic weights W to transform features and passes a ij Aggregation i The representation of the adjacent vertices of to update its representation z i , whose formula is

[0025]

[0026] Dynamic topology sharing graph convolution dynamically infers the topological structure and therefore has better generalization ability; akij is the dynamic topological relationship between joints vi and vj and depends on the input sample; the feature of joint vi is represented as xi; the formula of dynamic topology sharing GC is:

[0027]

[0028] Where k represents the index of the input sample, and are the input and output features of the kth sample; W is the dynamic weight. Finally, the channel topology R∈R is obtained by refining the shared topology A using the channel-specific correlation Q N×N×C′ :

[0029]

[0030] Here, α is a trainable scalar that adjusts the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

[0031] Furthermore, in S4, the Mixer module is mainly composed of two blocks, namely Joint-Mlp-Block and Channel-Mlp-Block. The human skeleton features after channel topology refinement graph convolution space modeling are input into the Mixer module. After format conversion, the human skeleton features are first normalized by Layer Normalization, and then input into the Joint-Mlp-Block with residual connection for correlation modeling between joints. After the same Layer Normalization normalization, they are transposed and input into the Channel-Mlp-Block with residual connection for correlation modeling of channel features of the same joint. Finally, the processed human skeleton features are transposed, LayerNormalized, and output to the multi-branch temporal convolutional network for feature extraction in the time dimension.

[0032] Furthermore, in S5, a multi-branch temporal convolutional network is used for temporal modeling. The multi-branch temporal convolutional network used consists of four branches: a 1×1 convolution branch, a maximum pooling branch, and two one-dimensional temporal convolutions with a kernel of 3 and dilation rates of 1 and 2, respectively. Each feature group is processed with a single branch, and the outputs of the four branches are connected together and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network. This temporal modeling process not only improves the temporal modeling capability, but also saves computational cost and parameters due to the reduction of the channel width of each branch.

[0033] Furthermore, in S6, the posture features extracted after spatial modeling and temporal modeling are input into the fully connected layer for decoding to obtain the prediction result of the action of the single modal data.

[0034] Furthermore, in 7, we weightedly fuse the results obtained from the four different modal data to obtain the final action recognition result of the model. The data stream fusion problem can be regarded as a function optimization process, with the optimal fusion effect as the objective function and each stream model as the decision variable to construct a nonlinear programming equation based on fusion. Here we use the control variable method. In order to make full use of the fusion weights of the four streams for comparison, we carried out a quantization process and put the initial values ​​of the four weights greater than 0 into the constraint part of the model to realize the importance of different streams.

[0035] The present invention provides a human skeleton action recognition device based on spatial feature enhancement, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method.

[0036] The present invention provides a device, comprising: a processor; a computer program; and the steps of the method are executed by the processor when the computer program is executed.

[0037] The beneficial effects of the present invention are as follows: the human skeleton action recognition method based on spatial feature enhancement of the present invention improves the accuracy of skeleton-based human action recognition tasks without significantly increasing the parameters of the network model, reduces the training time of the network model, and improves the inference speed of the network model.

[0038] In order to verify the effectiveness of the aggregation module with spatial feature enhancement in the present invention, the CTR-GCN is used as the basic network structure, and the Mixer spatial aggregation module and the multi-branch temporal convolutional neural network are added. The recognition rates of the NTU-RGB+D and NTU-RGB+D 120 datasets are shown in the following table:

[0039]

[0040] BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a step framework diagram of a method for human skeleton action recognition based on spatial feature enhancement in an embodiment of the present invention;

[0042] Figure 2 A network model diagram of a method for human skeleton action recognition based on spatial feature enhancement in an embodiment of the present invention;

[0043] Figure 3 This is a time modeling diagram of a human skeleton action recognition method based on spatial feature enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0045] Example 1

[0046] like Figure 1 As shown, the human skeleton action recognition method based on spatial feature enhancement includes the following steps:

[0047] S1. Obtain video sequence data of human skeleton joints, perform processing such as subtracting adjacent joints in the same frame and subtracting the same joints in adjacent frames on the human skeleton data, and process the original human skeleton data into data of four different modes: joints, bones, joint motion, and bone motion;

[0048] In S1, the joint flow modal data is processed according to the following steps:

[0049] (1) aligning the central joint point of the person in the first frame of the original human skeleton joint point data with the origin of the 3D Cartesian coordinate system;

[0050] (2) Rotate all skeleton joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system.

[0051] The joint flow modal data is taken with the central joint point as the origin, and the adjacent joint points in the same frame are subtracted to obtain the bone flow modal data.

[0052] The same joint points between adjacent frames of joint flow modal data and skeleton flow modal data are subtracted to obtain the corresponding motion flow modal data.

[0053] S2. Use the space partitioning strategy to define the adjacency matrix of the human skeleton joints and divide the nodes into three parts for use in the subsequent graph convolution space modeling process;

[0054] In S2, a space partitioning strategy is used to define the adjacency matrix of the human skeleton joints, and the nodes are divided into three parts, namely the joint itself, the neighboring nodes that are closer to the central joint than the joint itself, and other neighboring nodes of the current joint.

[0055]

[0056] where r i is the average distance from the center joint point to joint i over all frames.

[0057] S3, such as Figure 2 As shown, in the feature extraction process of the spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and refine the channel topology using the channel-specific correlation of each channel to model the channel topology.

[0058] In S3, the processed four modal data are sent to the model for spatial and temporal modeling. The channel topology structure is dynamically learned through channel topology refinement graph convolution to build the correlation between skeleton joints.

[0059] Topologically shared graph convolution uses dynamic weights W to transform features and passes a ij Aggregation i The representation of the adjacent vertices of to update its representation z i , whose formula is

[0060]

[0061] Dynamic topology sharing graph convolution dynamically infers the topological structure and therefore has better generalization ability; akij is the dynamic topological relationship between joints vi and vj and depends on the input sample; the feature of joint vi is represented as xi; the formula of dynamic topology sharing GC is:

[0062]

[0063] Where k represents the index of the input sample, and are the input and output features of the kth sample; W is the dynamic weight. Finally, the channel topology R∈R is obtained by refining the shared topology A using the channel-specific correlation Q N×N×C′ :

[0064]

[0065] Here, α is a trainable scalar that adjusts the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

[0066] S4, such as Figure 2 As shown in the figure, the human skeleton features processed by the channel topology refinement graph convolution are input into the spatial aggregation module Mixer, and the graph structure of the human body is combined into the model of the multi-layer perceptron to meet the specific field requirements of action recognition based on the human skeleton, while allowing local and global spatial interactions and enhancing the feature extraction of the spatial dimension. In S4, the human skeleton features are first normalized by Layer Normalization after format conversion, and then input into the Joint-Mlp-Block with residual connection to model the correlation between joint points, and then transposed after the same LayerNormalization normalization processing, and input into the Channel-Mlp-Block with residual connection to model the correlation of channel features of the same joint point, and finally the processed human skeleton features are transposed and normalized by LayerNormalization.

[0067] S5, such as Figure 3As shown in FIG, the human skeleton features modeled in the spatial dimension are input into the multi-branch temporal convolutional network for feature extraction in the temporal dimension; in S5, the human skeleton features after spatial modeling are input into the multi-branch temporal convolutional network for feature extraction in the temporal dimension. The multi-branch temporal convolutional network used is composed of four branches: a 1×1 convolution branch, a maximum pooling branch, and two one-dimensional temporal convolutions with a kernel of 3 and dilation rates of 1 and 2, respectively. Each feature group is processed with a single branch, and the outputs of the four branches are connected together and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network.

[0068] S6, inputting the posture features extracted after spatial modeling and temporal modeling into the fully connected layer for decoding, and obtaining the prediction result of the action of the single modal data;

[0069] S7. Perform weighted fusion on the results obtained from the four different modal data to obtain the final action recognition result of the model.

[0070] Example 2

[0071] This embodiment provides a human skeleton action recognition device based on spatial feature enhancement, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method according to Example 1.

[0072] Example 3

[0073] This embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 are implemented.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A human skeleton action recognition method based on spatial feature enhancement, characterized in that: The steps include: S1. Obtain video sequence data of human skeleton joints, perform subtraction processing on the human skeleton data by subtracting adjacent joints in the same frame and the same joints in adjacent frames, and process the original human skeleton data into data of four different modes: joints, bones, joint movements, and bone movements; S2. Use the space partitioning strategy to define the adjacency matrix of the human skeleton joints and divide the nodes into three parts for use in the subsequent graph convolution space modeling process; S3. In the feature extraction process of spatial dimension, we use channel topology refinement graph convolution to dynamically learn different topological structures, aggregate joint features of different channels for human skeleton action recognition, and refine the channel topology using the channel-specific correlation of each channel; S4, input the human skeleton features processed by channel topology refinement graph convolution into the spatial aggregation module Mixer, and combine the human graph structure into the multi-layer perceptron model to meet the needs of the human skeleton-based action recognition field, while allowing local and global spatial interactions and enhancing feature extraction in the spatial dimension; S5, inputting the human skeleton features modeled in the spatial dimension into a multi-branch temporal convolutional network to extract features in the temporal dimension; S6, inputting the posture features extracted after spatial modeling and temporal modeling into the fully connected layer for decoding, and obtaining the prediction result of the action of the single modal data; S7. Perform weighted fusion on the results obtained from the four different modal data to obtain the final action recognition result of the model.

2. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S1, the processing of joint flow modal data follows the following steps: S11: aligning the central joint point of the person in the first frame of the original human skeleton joint point data with the origin of the 3D Cartesian coordinate system; S12: Rotate all skeleton joints so that the spine of the person in the first frame is parallel to the z-axis in the 3D Cartesian coordinate system; S13: The skeleton flow modal data is obtained by subtracting the adjacent joint points in the same frame from the joint flow modal data with the central joint point as the origin; S14: The joint motion stream modal data and the skeleton motion stream modal data are obtained by subtracting the same joint points between adjacent frames of the corresponding modal data streams.

3. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S2, the space partitioning strategy is used to define the adjacency matrix of the human skeleton joints, and the nodes are divided into three parts, namely the joint itself, the neighboring nodes closer to the central joint than the joint itself, and other neighboring nodes of the current joint; where r i is the average distance from the center joint point to joint i over all frames.

4. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S3, the channel topology refinement graph convolution dynamically and effectively models the channel topology; specifically, the shared topology is a parameterized adjacency matrix that serves as the topological prior for all channels and provides universal correlations between vertices. Dynamically infer channel-specific correlations for each sample and capture the relationship between vertices within each channel; topological shared graph convolution uses dynamic weights W for feature transformation and passes a ij Aggregation i The representation of the adjacent vertices of to update its representation z i , whose formula is Dynamic topology shared graph convolution dynamically infers the topology structure and thus has better generalization ability; a k ij is the joint point v i 、v j The dynamic topological relationship between them depends on the input sample; the joint point v i The feature is denoted as x i ; The formula for dynamic topology shared GC is: Where k represents the index of the input sample, and are the input features and output features of the kth sample; W is the dynamic weight. Finally, the channel topology is obtained by refining the shared topology A using the channel-specific correlation Q Here, α is a trainable scalar that adjusts the optimization strength, and the addition is performed in a broadcast manner, where A is added to each channel of α×Q.

5. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S4, the Mixer module consists of two blocks, namely Joint-Mlp-Block and Channel-Mlp-Block; the human skeleton features after channel topology refinement graph convolution space modeling are input into the Mixer module. After format conversion, the human skeleton features are first normalized by Layer Normalization, and then input into the Joint-Mlp-Block with residual connection for correlation modeling between joint points. After the same Layer Normalization normalization, they are transposed and input into the Channel-Mlp-Block with residual connection for correlation modeling of channel features of the same joint point. Finally, the processed human skeleton features are transposed and normalized by Layer Normalization and output to the multi-branch temporal convolutional network for feature extraction in the time dimension.

6. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S5, a multi-branch temporal convolutional network is used for temporal modeling. The multi-branch temporal convolutional network adopted consists of four branches: a 1×1 convolution branch, a maximum pooling branch, and two one-dimensional temporal convolutions with a kernel of 3 and dilation rates of 1 and 2 respectively; each feature group is processed with a single branch, and the outputs of the four branches are connected together and processed by another 1×1 convolution to form the output of the multi-branch convolutional neural network.

7. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S6, the posture features extracted after spatial modeling and temporal modeling are input into the fully connected layer for decoding to obtain the prediction results of the action of the single modal data.

8. The method for human skeleton action recognition based on spatial feature enhancement according to claim 1, characterized in that: In S7, the results obtained from the four different modal data are weighted and fused to obtain the final action recognition result of the model; Use the control variable method; put the initial values ​​of the four weights greater than 0 into the constraint part of the model to realize the importance of different flows.

9. A human skeleton action recognition device based on spatial feature enhancement, characterized in that: It comprises a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 8.

10. A device, characterized in that: include: Processor; computer program; when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Human body behavior recognition method and system based on multi-channel directed graph convolution

    CN116895097A

  • Action recognition method for enhancing spatio-temporal characteristics

    CN117315765A

  • Method and apparatus for video recognition

    US20230046066A1

Cited By

  • Three-dimensional human body posture estimation method based on enhanced topology sensing network

    CN119887928A