A three-dimensional gesture posture estimation method based on topology perception

Abstract

The present application relates to a kind of three-dimensional gesture posture estimation methods based on topological perception, comprising the following steps: obtaining the single depth map of three-dimensional gesture posture;The single depth map is input to three-dimensional gesture posture estimation network, and the three-dimensional joint point coordinate information of gesture posture is obtained;Wherein, the three-dimensional gesture posture estimation network includes: feature extraction module, for extracting the local feature of the image of the single depth map, and is converted into feature sequence;Encoder module is used to capture global context information based on the feature sequence;Decoder module is used to determine the relationship between joint nodes based on the global context information;Topological perception regression module is used to extract the semantic information of each layer in the decoder module to obtain the three-dimensional joint point coordinate information of gesture posture.The present application can realize accurate, robust three-dimensional gesture joint point position prediction when facing self-occlusion self-similarity problem.

Description

A Topology-Aware 3D Gesture Pose Estimation Method Technical Field

[0001] This invention relates to the field of three-dimensional gesture pose estimation technology, and in particular to a three-dimensional gesture pose estimation method based on topology awareness. Background Technology

[0002] Hands, as the primary medium for information transmission in daily human activities, play an indispensable role in numerous applications such as human-computer interaction, augmented reality, virtual reality, and third-person imitation learning. Accurate 3D hand gesture pose estimation using computer vision algorithms is a crucial component. With the development of high-precision depth cameras, depth map-based hand gesture pose estimation methods have made remarkable progress. However, existing methods still face significant challenges in cases of severe self-occlusion and high self-similarity.

[0003] The mainstream methods for 3D hand gesture estimation primarily utilize convolutional neural networks (CNNs). However, because CNNs operate within a fixed-scale window, their receptive field is limited by the window size, making it impossible to model long-distance dependencies. Therefore, these methods struggle to infer the gesture shape of occluded regions from the globally visible area. Transformer models based on self-attention mechanisms can overcome this limitation, directly modeling global dependencies between all nodes by calculating feature similarities between input nodes. However, since self-attention relies on node feature similarity, it ignores the inherent kinematic topology of the hand. This kinematic topology is an inherent property of the hand, explicitly revealing the intrinsic connections between nodes and constraining the maximum physiological range of joint movement. This is crucial for inferring the location of occluded joints; therefore, existing methods still have low accuracy in predicting the location of 3D hand gesture joints. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a topology-aware three-dimensional gesture pose estimation method that can achieve accurate and robust prediction of the position of three-dimensional gesture joints when facing self-occlusion and self-similarity problems.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a three-dimensional gesture pose estimation method based on topology awareness, comprising the following steps:

[0006] Obtain a single depth map of a 3D hand gesture pose;

[0007] The single depth map is input into a 3D gesture pose estimation network to obtain the 3D joint coordinate information of the gesture pose. The 3D gesture pose estimation network includes a feature extraction module, an encoder module, a decoder module, and a topology-aware regression module connected in sequence. The feature extraction module extracts local image features from the single depth map and converts them into feature sequences. The encoder module uses the feature sequences as input to capture global context information based on the feature sequences. The decoder module uses node queries as input to determine the relationships between joint nodes based on the global context information. The topology-aware regression module uses the concatenation result of the outputs of each layer in the decoder module as input to extract the semantic information of each layer in the decoder module to obtain the 3D joint coordinate information of the gesture pose.

[0008] The feature extraction module uses the ResNet network framework to extract local features of the single depth map to obtain a downsampled feature map. The downsampled feature map is then dimensionality-reduced by a convolutional layer and converted into a feature sequence through tensor flattening.

[0009] The encoder module includes N Transformer encoder layers. Each Transformer encoder layer includes a self-attention layer, a first normalization layer, a feedforward neural network layer, and a second normalization layer connected in sequence. The input of the self-attention layer is connected to the output of the first normalization layer, and the input of the feedforward neural network layer is connected to the output of the second normalization layer.

[0010] The decoder module comprises N Graphformer decoder layers, each including a node offset graph convolutional layer, a third normalization layer, a self-attention layer, a fourth normalization layer, a cross-attention layer, and a fifth normalization layer connected in sequence. The input of the node offset graph convolutional layer is connected to the output of the third normalization layer to decouple the node feature mapping process from the offset feature mapping process. The input of the self-attention layer is connected to the output of the fourth normalization layer. The input of the cross-attention layer is connected to the output of the fifth normalization layer.

[0011] The node offset graph convolutional layer uses a fixed adjacency matrix, which represents the hand kinematic topology information. The hand kinematic topology information includes 21 points and 20 edges, where each point corresponds to a hand joint, and each edge represents the physiological connection between the connected hand joints.

[0012] The expression for the node offset graph convolutional layer is: Among them, X (l) Let σ be the input feature of the l-th layer in the node offset map convolutional layer, and σ be the activation function. For the normalized adjacency matrix, Let A be a diagonal degree matrix, I be an adjacency matrix, and W1 and W2 be feature matrices, which map node features and offset features, respectively.

[0013] The topology-aware regression module includes a first semantic graph convolutional layer, a first ReLU layer, a second semantic graph convolutional layer, a second ReLU layer, a third semantic graph convolutional layer, and a 1×1 convolutional layer connected in sequence; the input of the first semantic graph convolutional layer is connected to the output of the second ReLU layer.

[0014] The expressions for the first semantic graph convolutional layer, the second semantic graph convolutional layer, and the third semantic graph convolutional layer are all: X′ (l+1) =σ(WX′) (l) ρ i (M⊙(A+I))), where X′ (l) Let σ be the input features of the l-th layer in the semantic graph convolutional layer, σ be the activation function, W be the feature matrix, and ρ be the input features of the l-th layer. i For the Softmax operation, M is the weighted matrix, A is the adjacency matrix, I is the identity matrix, and ⊙ represents element-wise multiplication.

[0015] Beneficial effects

[0016] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention uses a depth map as input, uses a graph convolutional neural network to introduce the inherent topological prior of the hand into the network, and uses Transformer to build global dependencies between nodes. When facing the self-occlusion and self-similarity problem, it can achieve accurate and robust prediction of the position of three-dimensional hand gesture joints. Attached Figure Description

[0017] Figure 1 is a flowchart of an embodiment of the present invention;

[0018] Figure 2 is a framework diagram of the Transformer encoder layer in an embodiment of the present invention;

[0019] Figure 3 is a framework diagram of the Graphformer decoder layer in an embodiment of the present invention;

[0020] Figure 4 is a kinematic topology diagram of the hand in an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of the node offset graph convolutional layer in an embodiment of the present invention;

[0022] Figure 6 is a framework diagram of the topology-aware regression module in an embodiment of the present invention;

[0023] Figure 7 is an example of three-dimensional hand gesture estimation under self-occlusion using the embodiments of the present invention;

[0024] Figure 8 is an example of three-dimensional hand gesture estimation under self-similar conditions using the embodiments of the present invention. Detailed Implementation

[0025] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0026] The embodiments of the present invention relate to a topology-aware three-dimensional gesture pose estimation method, which can alleviate the ambiguity problem caused by severe self-occlusion and high self-similarity.

[0027] In human visual perception, humans can accurately determine hand posture in various complex scenarios, primarily relying on a deep understanding of the interactive environment and rich prior knowledge of hand kinematics. These two types of information provide abundant constraints, effectively mitigating the ambiguity caused by occluded and similar joints. Based on this characteristic, this implementation utilizes a Transformer to model the global understanding of hand-scene interaction and a graph convolutional neural network to model the local kinematic connections between nodes, thereby constructing an end-to-end topology-aware 3D hand gesture estimation network.

[0028] As shown in Figure 1, the topology-aware 3D hand gesture estimation method of this embodiment includes: acquiring a single depth map of the 3D hand gesture pose; inputting the single depth map into a 3D hand gesture pose estimation network to obtain the 3D joint coordinate information of the hand gesture pose. The 3D hand gesture pose estimation network includes a feature extraction module, an encoder module, a decoder module, and a topology-aware regression module connected in sequence. The feature extraction module extracts local image features from the single depth map and converts them into a feature sequence. The encoder module uses the feature sequence as input to capture global context information based on the feature sequence. The decoder module uses node queries as input to determine the relationship between joint nodes based on the global context information. The topology-aware regression module uses the concatenation result of the outputs of each layer in the decoder module as input to extract the semantic information of each layer in the decoder module to obtain the 3D joint coordinate information of the hand gesture pose.

[0029] In this embodiment, the feature extraction module takes a 256×256 depth map as input and uses the ResNet network framework to extract local image features, resulting in an 8×8×2048 downsampled feature map. This downsampled feature map is then passed through a 1×1 convolutional layer to reduce the number of channels to 512 dimensions, and then transformed into a 64×256 feature sequence through tensor flattening.

[0030] The encoder module in this embodiment comprises four Transformer encoder layers. The framework structure of the Transformer encoder layer is shown in Figure 2, including a self-attention layer, a first normalization layer, a feedforward neural network layer, and a second normalization layer connected in sequence. The input of the self-attention layer is connected to the output of the first normalization layer, and the input of the feedforward neural network layer is connected to the output of the second normalization layer. This Transformer encoder layer mainly consists of a self-attention mechanism and a feedforward neural network, capturing global contextual information based on the input feature sequence. The Transformer encoder layer uses the feature sequence obtained from the feature extraction module as input. To maintain the relative spatial relationships of nodes in the feature sequence, sinusoidal positional encoding is introduced. The sinusoidal positional encoding is added to the feature sequence, and then sequentially passes through a series of self-attention layers, layer normalization operations, feedforward neural network layers, and layer normalization operations to obtain a global contextual feature of size 64×256.

[0031] The decoder module in this embodiment comprises four Graphformer decoder layers. This Graphformer decoder layer differs from the traditional Transformer decoder layer, which only includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer. Because it primarily relies on the self-attention layer to establish dependencies between nodes, it struggles to perceive the intrinsic connections between nodes inherent in the hand's kinematic topology. To overcome this limitation, the decoder module in this embodiment emphasizes the synergistic effect of the self-attention mechanism and the graph neural network, benefiting from long-distance dependencies and local topological connections of joints. Specifically, a kinematic topology graph of the hand is first constructed (as shown in Figure 4), containing 21 points and 20 edges. Each point in this topology graph corresponds to a hand joint, and each edge represents a physiological connection between the connected joints.

[0032] As shown in Figure 3, the Graphformer decoder layer in this embodiment includes a node offset graph convolutional layer, a third normalization layer, a self-attention layer, a fourth normalization layer, a cross-attention layer, and a fifth normalization layer connected in sequence. The input of the node offset graph convolutional layer is connected to the output of the third normalization layer to decouple the node feature mapping process and the offset feature mapping process. The input of the self-attention layer is connected to the output of the fourth normalization layer. The input of the cross-attention layer is connected to the output of the fifth normalization layer. Therefore, the Graphformer decoder layer in this embodiment consists of a node offset graph convolutional layer, a self-attention layer, and a cross-attention layer. The decoder layer takes node queries as input, where each node query corresponds one-to-one with a hand node, and each node query represents the positional encoding information of the node. Furthermore, since the node offset graph convolutional layer implements the non-linear mapping of node queries, the feedforward neural network layer used in the traditional Transformer decoder layer is removed.

[0033] In 3D hand gesture estimation tasks, node features themselves contain rich positional information, and neighboring nodes also provide rich and effective positional feature information for estimating offsets. This is especially crucial for occluded joints, where neighboring node positional information is of critical reference value. Inspired by this principle, the principle of the node offset graph convolutional layer in this implementation is shown in Figure 5, which decouples the node feature mapping process from the offset feature mapping process. Node feature mapping depends on the characteristics of the node itself, while offset feature mapping aggregates the offset information flowing from neighboring nodes and the current node to the central node. Furthermore, to better leverage the synergy with the sub-attention layer and accelerate model convergence, the node offset graph convolutional layer uses a fixed adjacency matrix. This adjacency matrix can represent the kinematic topology information of the hand; that is, when node i and node j are connected, the value at positions (i,j) and (j,i) in the adjacency matrix is ​​1. The formula for the node offset-based graph neural network of the l-th layer is as follows:

[0034]

[0035] Among them, X (l) Let σ be the input feature of the l-th layer, and σ be the activation function. The normalized adjacency matrix is ​​calculated using the following formula: Let A be the diagonal degree matrix, I be the adjacency matrix, and W1 be the identity matrix. W2 are feature matrices, which map node features and offset features, respectively. Note that in traditional graph convolutional neural networks, attention is allocated to the current node and its neighbors based on the degree matrix, thus weakening the guiding role of the node's own positional information.

[0036] The topological structure of hand joints is crucial for accurately inferring hand pose, especially in cases of severe self-occlusion and self-similarity. Existing regression modules employ multilayer perceptron networks, which independently regress the 3D coordinates of joints, ignoring spatial structural constraints between nodes. To address this issue, this implementation proposes a topology-aware regression module utilizing graph convolutional neural network (GCN) technology. This GCN provides a way to introduce topological priors, aggregating information from neighboring nodes under the guidance of the topological structure. However, traditional GCNs assign fixed attention weights to connections between nodes, neglecting the complex semantic relationships between adjacent nodes.

[0037] As shown in Figure 6, the topology-aware regression module of this embodiment includes a first semantic graph convolutional layer, a first ReLU layer, a second semantic graph convolutional layer, a second ReLU layer, a third semantic graph convolutional layer, and a 1×1 convolutional layer connected in sequence; the input of the first semantic graph convolutional layer is connected to the output of the second ReLU layer. Compared with traditional graph convolutional neural networks, semantic graph convolutional neural networks add a learned weighting matrix M to adaptively simulate the connection strength between joints, and its expression is:

[0038] X′ (l+1) =σ(WX′) (l) ρ i (M⊙(A+I)))

[0039] Where, X′ (l) Let σ be the input features of the l-th layer in the semantic graph convolutional layer, σ be the activation function, W be the feature matrix, and ρ be the input features of the l-th layer. i For the Softmax operation, M is the weighted matrix, A is the adjacency matrix, I is the identity matrix, and ⊙ represents element-wise multiplication.

[0040] Furthermore, to mitigate the oversmoothing issue caused by concatenating multiple semantic graph convolutional layers, a skip connection operation is employed. This involves concatenating the outputs of all Graphformer decoder layers and inputting them together into the topology-aware regression module. This encourages the regression module to implicitly extract semantic information contained in different decoder layers. Benefiting from the excellent properties of semantic graph convolution, the regression module in this implementation constrains the predicted hand pose to a more precise feature space under the guidance of the hand's topology.

[0041] The topology-aware 3D hand gesture pose estimation method of this embodiment not only fully learns the global information of hand joints, but also utilizes graph convolutional neural networks to fully leverage the inherent topological prior knowledge of the hand, effectively modeling the local kinematic connections between joints. This allows for accurate and robust prediction of 3D hand gesture joint positions even when facing self-occlusion and self-similarity problems. Compared with existing technologies, this embodiment achieves more accurate hand pose estimation under both self-occlusion and self-similarity conditions. Figures 7 and 8 respectively illustrate some examples of the 3D hand gesture pose estimation results under self-occlusion and self-similarity conditions.

Claims

1. A three-dimensional gesture pose estimation method based on topology awareness, characterized in that, Includes the following steps: A single depth map of a 3D hand gesture pose is obtained; the single depth map is input into a 3D hand gesture pose estimation network to obtain the 3D joint coordinate information of the hand gesture pose; wherein, the 3D hand gesture pose estimation network includes a feature extraction module, an encoder module, a decoder module, and a topology-aware regression module connected in sequence; the feature extraction module is used to extract local image features from the single depth map and convert them into feature sequences; the encoder module is used to capture global context information based on the feature sequences as input; the encoder module contains N Transformer encoder layers, each Transformer encoder layer including a self-attention layer, a first normalization layer, and a feedforward neural network connected in sequence. The network layer and the second normalization layer are connected. The input of the self-attention layer is connected to the output of the first normalization layer, and the input of the feedforward neural network layer is connected to the output of the second normalization layer. The decoder module takes node query as input and determines the relationship between joint nodes based on the global context information. The decoder module contains N Graphformer decoder layers, each including a node offset graph convolutional layer, a third normalization layer, a self-attention layer, a fourth normalization layer, a cross-attention layer, and a fifth normalization layer connected in sequence. The input of the node offset graph convolutional layer is connected to the output of the third normalization layer to decouple the node feature mapping process and the offset feature mapping process. The input of the self-attention layer is connected to the output of the fourth normalization layer; the input of the cross-attention layer is connected to the output of the fifth normalization layer; the topology-aware regression module takes the splicing result of the output of each layer in the decoder module as input, and is used to extract the semantic information of each layer in the decoder module to obtain the three-dimensional joint coordinate information of the gesture posture.

2. The three-dimensional gesture pose estimation method based on topology awareness according to claim 1, characterized in that, The feature extraction module uses the ResNet network framework to extract local features of the single depth map to obtain a downsampled feature map. The downsampled feature map is then dimensionality-reduced by a convolutional layer and converted into a feature sequence through tensor flattening.

3. The three-dimensional gesture pose estimation method based on topology awareness according to claim 1, characterized in that, The node offset graph convolutional layer uses a fixed adjacency matrix, which represents the hand kinematic topology information. The hand kinematic topology information includes 21 points and 20 edges, where each point corresponds to a hand joint, and each edge represents the physiological connection between the connected hand joints.

4. The topology-aware 3D gesture pose estimation method according to claim 1, characterized in that, The expression for the node offset graph convolutional layer is: Among them, X (l) Let σ be the input feature of the l-th layer in the node offset map convolutional layer, and σ be the activation function. For the normalized adjacency matrix, Let A be a diagonal degree matrix, I be an adjacency matrix, and W1 and W2 be feature matrices, which map node features and offset features, respectively.

5. The three-dimensional gesture pose estimation method based on topology awareness according to claim 1, characterized in that, The topology-aware regression module includes a first semantic graph convolutional layer, a first ReLU layer, a second semantic graph convolutional layer, a second ReLU layer, a third semantic graph convolutional layer, and a 1×1 convolutional layer connected in sequence; the input of the first semantic graph convolutional layer is connected to the output of the second ReLU layer.

6. The topology-aware 3D gesture pose estimation method according to claim 5, characterized in that, The expressions for the first semantic graph convolutional layer, the second semantic graph convolutional layer, and the third semantic graph convolutional layer are all: X′ (l+1) =σ(WX′) (l) ρ i (M⊙(A+I))), where X′ (l) Let σ be the input features of the l-th layer in the semantic graph convolutional layer, σ be the activation function, W be the feature matrix, and ρ be the input features of the l-th layer. i For the Softmax operation, M is the weighted matrix, A is the adjacency matrix, I is the identity matrix, and ⊙ represents element-wise multiplication.

Images