Self-adaptive graph three-dimensional hand posture estimation method based on depth map and point cloud fusion

Abstract

The invention discloses an adaptive graph three-dimensional hand attitude estimation method based on depth map and point cloud fusion, and relates to the field of three-dimensional hand attitude estimation. The method comprises the following steps: extracting 3D point features based on 3D point cloud data by using a point cloud global feature capturer, extracting local 2D visual features based on depth image data by using a 2D auto-encoder, and fusing the 3D point features and the local 2D visual features to form super point features for feature coding; joint point embedding initialization and patch initialization are carried out based on the super point features; a graph Transform is adopted and combined with an adaptive attention mechanism, super point features after joint point embedding initialization and patch initialization are decoded, local details and a motion corresponding relation are adaptively weighted, and hand posture estimation is carried out. According to the method, the 2D local visual information of the depth map and the complete three-dimensional geometric information provided by the 3D point cloud are fully utilized, and the modeling capability of the model is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional hand posture estimation, and specifically relates to an adaptive Figure 3 dimensional hand pose estimation method. Background Art

[0002] 3D hand pose estimation has become a key element in various human-computer interaction systems, providing the necessary key points for the human hand. Accurate positioning of the key points of the hand can accurately understand and effectively analyze human motion and behavior. With the advancement of deep learning technology and the popularity of low-cost depth cameras, 3D hand pose estimation based on depth images has made significant progress. However, when the hand is self-occluded (such as crossing fingers) or interacting with objects, the depth information may be occluded, resulting in loss or misestimation of joints. Self-occlusion and mutual occlusion of finger joints make accurate prediction more difficult. Existing 3D hand pose estimation methods have challenges in dealing with occlusion problems when the hand is self-occluded and interacting with objects. These methods often have difficulty in accurately estimating the position of the key points of the hand in complex occlusion situations, resulting in limited performance in practical applications and insufficient dynamic occlusion processing.

[0003] In recent years, deep learning-based methods have demonstrated the effectiveness of traditional convolutional neural networks (CNNs) in human pose estimation tasks. Subsequently, CNN architectures using different modes, such as point cloud, voxel representation, or multimodal combination, have been widely explored. For example, AWR (Adaptive Weighting Regression) is based on a fully convolutional 2D CNN architecture that can maintain the spatial structure of extracted features and better utilize local patterns in depth images. Under the guidance of an adaptive weight map, it aggregates different dense representation regions through discrete integration of all pixels, and finally generates accurate estimation results. Its network structure is as follows: Figure 1 However, there are still the following problems:

[0004] (1) Due to the noise and clutter problems in depth images, as well as the high self-similarity and self-occlusion problems between fingers. Although depth images can provide rich texture information, they still face challenges in dealing with hand self-occlusion and occlusion problems when interacting with objects. Although point cloud data can provide accurate 3D structural information, it lacks sufficient texture details, resulting in inaccurate hand posture estimation in complex backgrounds.

[0005] (2) Traditional 2D CNN-based methods are widely adopted due to their straightforward implementation. However, they suffer from inherent limitations, such as the inability to fully recognize the complexity of 3D structures and their dependence on camera viewpoint. To overcome these limitations, 3DCNN-based methods were introduced, which encapsulate volumetric data using a 3D voxelized representation of depth images. The disorder and irregularity of point cloud data complicate the direct application of traditional image processing techniques such as convolutional neural networks (CNNs). In addition, the sparsity and variability of point cloud data further increase the difficulty of extracting useful features from these data. Although methods such as Hand Pointnet (network structure such as Figure 2 These techniques have made progress in 3D hand pose estimation, but they require a lot of memory and computing resources. These methods suffer from the disorder and unstructure of point cloud data and are computationally inefficient.

[0006] In addition to CNNs, transformer-based methods have also shown feasibility in the field of hand pose estimation. Despite the remarkable success of these methods, it remains challenging to achieve accurate estimation in scenarios such as self-occlusion and hand-object occlusion. Recent studies have attempted to introduce static graph structures to capture the kinematic correspondence between keypoints, so that occluded keypoints can be reconstructed using the corresponding keypoint information. However, static graph structures fail to capture the dynamics of various occlusions.

[0007] The 3D global feature extraction module of other inventions uses PointNet++, which has achieved remarkable results in the 3D hand pose estimation task. However, the study found that these inventions provide an accurate description of the 3D structure based on the strategic analysis point cloud of PointNet. PointNet is a deep learning framework that is good at processing irregular and unstructured point clouds. It was originally used for hand pose estimation in HandPointNet. However, a significant disadvantage of point clouds is the overhead of convolution calculations for querying neighbors from dense point sets. Therefore, existing methods usually use sparse point clouds, which limits performance.

[0008] In order to bridge the shortcomings of images and point clouds, research currently uses multimodal representations, combining 2D depth images and 3D point clouds. By doing so, the model excels at extracting dense detail information while effectively capturing 3D spatial features, thereby facilitating accurate 3D hand pose estimation.

[0009] Based on this, the present invention designs a 3D global feature capturer Point Cloud Mamba (PCM) for capturing the global features and local area point features of the hand point cloud, aiming to capture global and local features at the same time. Through multi-stage processing and local group convolution, the network can capture local structural information; at the same time, through residual connection and multi-scale processing, the network can integrate local features to form a global feature representation; and the features obtained with the 2D autoencoder are fused into super point features for feature encoding. By introducing the adaptive attention mechanism technology, the weights of local geometric information and kinematic corresponding joint information are dynamically adjusted according to the visibility and occlusion of the hand joints, thereby enhancing the model's recognition ability for the hand joints. In addition, a data enhancement strategy PixDropout for depth maps is also applied to the invention to improve the accuracy of estimation tasks. The present invention makes full use of the 2D local visual information of the depth map and the complete three-dimensional geometric information provided by the 3D point cloud to optimize the modeling ability of the model. Summary of the invention

[0010] The purpose of the present invention is to provide an adaptive Figure 3 The proposed method aims to solve the problem that existing 3D hand pose estimation methods are difficult to accurately estimate the positions of hand key points in complex occlusion situations when dealing with hand self-occlusion and occlusion when interacting with objects, resulting in limited performance in practical applications and insufficient dynamic occlusion processing.

[0011] To achieve the above purpose, the technical solution adopted by the present invention is:

[0012] The first aspect of the present invention proposes an adaptive Figure 3 The method for estimating hand posture comprises the following steps:

[0013] S1. Use the point cloud global feature capturer to extract 3D point features based on 3D point cloud data, convert the 3D point cloud data into a 1D point sequence, and use a consistent traversal serialization strategy to make adjacent points in the sequence also adjacent in spatial position;

[0014] S2, using 2D autoencoder to extract local 2D visual features based on depth image data, and fusing 3D point features and local 2D visual features to form super point features for feature encoding;

[0015] S3, joint point embedding initialization and patch initialization based on super point features;

[0016] S4. Graph Transformer is used in combination with adaptive attention mechanism to decode the super-point features after joint embedding initialization and patch initialization, adaptively weight local details and motion correspondences, and perform hand posture estimation.

[0017] Preferably, the point cloud global feature capturer in S1 adopts Point Cloud Mamba; S1 is specifically as follows:

[0018] S101, providing various point cloud sequences through consistent traversal serialization strategy and its variants, and maintaining the relationship information between points by traversing multiple point sequences;

[0019] S102, introduce sequence hints to enable the Mamba layer to process point sequences generated by different serialization strategies;

[0020] S103, introducing a position coding method based on spatial coordinate mapping, and enhancing the spatial position information of the point through position coding.

[0021] Preferably, the S101 is specifically as follows:

[0022] First, the point cloud is sampled to convert the continuous space coordinates into discrete grid coordinates:

[0023]

[0024] in, and are the spatial coordinates of the point (i.e. the position of the point in three-dimensional space) and the grid coordinates, and N is the total number of grids;

[0025] Design an encoding function that maps coordinates to codes given two-dimensional coordinates; sort the sequence according to the codes to ensure that adjacent points are continuous in space;

[0026]

[0027] Among them, n 1 and n 2 Respectively represent the grid coordinates of the point in two dimensions, N represents the total number of grids, which is used to determine the resolution of the grid, and Code_func is a coding function. 2 The parity of determines how to calculate the encoding value;

[0028] Then, the code is calculated based on the grid coordinates of each point:

[0029]

[0030] in, is the grid coordinate of the point, indicating its position in the three-dimensional grid; the purpose of code is to map the three-dimensional grid coordinate to a unique position in the one-dimensional sequence; in this way, each point in the point cloud can be assigned a unique serial number according to its position in space; the point cloud is sorted according to the code and flattened into a one-dimensional sequence.

[0031] Furthermore, by exchanging the order of x, y, and z axes, six different serialization strategies are derived, namely “xyz”, “xzy”, “yxz”, “yzx”, “zxy”, and “zyx”; different serialization strategies are used for the input of different Mamba layers.

[0032] Preferably, the S102 is specifically as follows:

[0033] When combining multiple serialization strategies, an identifier is assigned to each serialization, which is used by the Mamba layer to identify the arrangement of the point cloud sequence;

[0034] Assign N to each serialization order p A learnable embedding is used as a sequence cue; before being processed by the Mamba layer, the point cloud sequence is added to the beginning and end of the sequence with the corresponding sequence cue; and a shared linear layer is assigned to all Mamba layers in each stage to map the sequence cue to the required channel size.

[0035] Preferably, the S103 is specifically as follows:

[0036] The position mapping function is characterized by the spatial information encoding of the points. The position embeddings corresponding to adjacent points in the sequence are similar. A private position embedding function is learned for each stage. Different Mamba layers in the same stage share the same position encoding function.

[0037] A shared position mapping function is used to map the spatial coordinates of a point to a position encoding with the same channel size as the feature;

[0038]

[0039] Among them, Embed pos Represents the position encoding projected from the spatial coordinates of the point; Linear is a linear transformation function and a fully connected layer that maps the three-dimensional spatial coordinates of the point to a high-dimensional space to obtain position embedding.

[0040] Preferably, the joint point embedding initialization in S3 is specifically as follows: feeding the super point features to two point set convolutional layers to generate a 3D global vector, and initializing the joint point embedding in sequence through three bias induction layers; the global vector is replicated J times and provided to the bias induction layer to generate embeddings for J individual joints;

[0041] Specifically, after obtaining the joint encoding, a linear transformation is applied to project the high-dimensional embedding into the 3D space as the initial position for collecting patches, and the k nearest super-point features are grouped with their corresponding features around each patch position to obtain a 3D point patch.

[0042] Preferably, the S4 is specifically as follows:

[0043] The joint embedding E is first enhanced by GCN to form the evolved joint embedding E′;

[0044] E′=ReLU(AEW)

[0045] Then each super point in each 3D point patch and its corresponding features are concatenated with the evolved joint point embedding to form the input of the adaptive attention mechanism; specifically as follows:

[0046] Q K =MLP([E′,F k ])

[0047] K = set_conv k ([E′,F k ])

[0048]

[0049] Among them, MLP is a multi-layer perceptron, set_conv k is the point set convolution layer, σ is the sigmoid activation function, [.,.] is the connection operation, and ⊙ is the Hadamard product;

[0050] A residual mechanism is introduced to accumulate the calculated value V and the input local points; the point set convolution is used as a feed-forward module to aggregate the calculated value V into the updated joint point embedding for position regression of each joint point;

[0051] Embed = set_conv F (V k +F k )+MAX F (V k +F k )

[0052] Among them, MAX F Represents the maximum pooling operation that pursues residual accumulation;

[0053] Finally, a trainable linear transformation is used to project the updated keypoint embeddings onto the 3D keypoint coordinates.

[0054] The second aspect of the present invention proposes an adaptive graph transformer model based on a point cloud state space model applied to the above method, comprising:

[0055] A super point feature encoding module, comprising a point cloud global feature capturer using Point Cloud Mamba and an autoencoder based on ConvNeXt; the point cloud global feature capturer introduces a consistent traversal serialization strategy, an order prompt, and a position encoding method based on spatial coordinate mapping; the super point feature encoding module is used to form a super point feature obtained by fusing 3D point features and local 2D visual features;

[0056] The joint embedding initialization module uses the joint embedding in HandFoldingNet to initialize the joint point embedding; it is used to initialize the joint point embedding and patch initialization of the super point feature;

[0057] Adaptive transformer decoding module; uses graph transformer and combines adaptive attention mechanism to dynamically adjust the weights of local geometric information and kinematic corresponding joint information according to the visibility and occlusion of hand joints.

[0058] Preferably, after collecting data for data enhancement, a training set is constructed, and the training set is used for model training; the data enhancement adopts the data enhancement method PixDropout of random pixel discarding, which randomly discards some pixels on the hand surface in the depth image to simulate the occlusion of the image recognition task.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) The present invention designs a point cloud global feature capturer PCM based on a state-space model. This module is specifically used to capture the global knowledge prior and local area fusion feature information of the 3D hand point cloud structure. PCM uses the core idea of ​​the Mamba architecture to convert 3D point cloud data into a 1D point sequence through the consistent traversal serialization (CTS) method, ensuring that adjacent points in the sequence are also adjacent in space, thereby retaining the spatial relationship information of the point cloud. In addition, PCM introduces order cues and a position encoding method based on spatial coordinate mapping to more accurately encode the spatial information of the points and effectively handle the disorder and irregularity of the point cloud data. And this module can more effectively fuse the 3D point features projected by 2D visual features into super point features with richer information.

[0061] (2) The present invention proposes an adaptive graph transformer (AGT), which combines the adaptive attention mechanism technology to dynamically adjust the weights of local geometric information and kinematic corresponding joint information according to the visibility and occlusion of the hand joints. AGT can learn the precise local geometric information of the hand and directly calculate the geometric features and kinematic topological information in the 3D key point space to avoid the 2D to 3D nonlinear problem. This design enables the model to adaptively focus on the kinematic topology or local information of the key points, enhancing the model's ability to recognize hand joints.

[0062] (3) The present invention proposes a novel data enhancement method, PixDropout, which is suitable for data enhancement of depth images. PixDropout simulates occlusion and self-occlusion in real visual conditions by randomly discarding some pixels in the input depth image, thereby increasing the amount and diversity of training data. This method is simple and effective, and can provide more diverse high-quality data for deep learning-based hand posture estimation methods, thereby enhancing the generalization ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 AWR network structure diagram in the background technology of the present invention;

[0064] Figure 2 The network structure diagram of Hand Pointnet in the background technology of the present invention;

[0065] Figure 3 A network structure diagram of the adaptive graph transformer PCM-AGT based on the point cloud state space model in the present invention;

[0066] Figure 4 A schematic diagram of obtaining a super point by fusing 3D point features and local 2D visual features in the present invention;

[0067] Figure 5 It is a network structure diagram of the point cloud global feature capturer PCM in the present invention;

[0068] Figure 6 It is a schematic diagram of the principle of the consistent serialization traversal strategy in the present invention;

[0069] Figure 7 A network structure diagram for order prompting in the present invention;

[0070] Figure 8 A network structure diagram of the adaptive graph transformer in the present invention;

[0071] Fig. 9 This is a schematic diagram of PixDropout data enhancement used in the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1:

[0074] Adaptive based on depth map and point cloud fusion Figure 3 The 3D hand posture estimation method is implemented using the adaptive graph transformer PCM-AGT based on the point cloud state space model. Its overall network structure is as follows: Figure 3 shown.

[0075] Adaptive based on depth map and point cloud fusion Figure 3 The hand posture estimation method is as follows:

[0076] Step 1: Construct a super point feature encoding module to extract 3D point features and local 2D visual features to form super point features for encoding;

[0077] The present invention combines two multimodal data types, depth map and point cloud, accepts depth image and point cloud as multimodal input, extracts 3D point features and local 2D visual features respectively through Point Cloud Mamba and ConvNeXt-based autoencoder, and then fuses these features to form super point features. Due to the disorder and irregularity of point cloud data, and in order to capture stronger super point features, Point Cloud Mamba (PCM) is used as the global feature extraction encoder of 3D point cloud, and the core idea of ​​state space models (SSMs), especially the Mamba architecture, is used to capture the global features of point cloud data while maintaining linear computational complexity. By proposing the Consistent Traverse Serialization (CTS) method, PCM can convert 3D point cloud data into 1D point sequence, ensuring that adjacent points in the sequence are also adjacent in space, thereby retaining the spatial relationship information of the point cloud. In addition, PCM introduces Order Prompts to help the model understand point cloud data in different serialization orders, as well as a position encoding method based on spatial coordinate mapping, which is more suitable for irregular point cloud data than traditional RoPE (Rotary Position Embedding) or learnable embedding. By directly mapping the spatial coordinates of the point to the position embedding of the same channel size as the feature, PCM can encode the spatial information of the point more accurately. The combination of these technologies enables PCM to effectively handle the disorder and irregularity of point cloud data, providing a strong support for the analysis and understanding of 3D point clouds.

[0078] (1) Point Cloud Mamba (PCM) architecture;

[0079] The disorder and irregularity of point cloud data make it complicated to directly apply traditional image processing techniques such as convolutional neural networks (CNNs), and it is difficult to fuse with 2D local visual features to obtain richer information. In addition, the sparsity and variability of point cloud data further increase the difficulty of extracting useful features from these data. The point cloud global feature capturer in the present invention adopts the core idea of ​​state space models (SSMs), especially the Mamba architecture, to capture the global features of point cloud data while maintaining linear computational complexity. The proposed point cloud state space encoder is as follows Figure 4-Figure 5 shown.

[0080] (2) Design consistent serialization traversal (Serialization strategy);

[0081] First, PCM provides various point cloud sequences through a consistent traversal serialization strategy and its variants, thereby maintaining the relationship information between points by traversing multiple point sequences. Second, PCM introduces order hints to help the Mamba layer better handle point sequences generated by different serialization methods, and enhances the spatial location information of points through position encoding. Finally, a more reasonable overall architecture is designed.

[0082] 1) Regarding serialization strategy;

[0083] It is crucial to convert 3-D point cloud data into a one-dimensional sequence that Mamba can process. Mamba can handle point cloud sequences arranged according to specific rules better than unordered point cloud sequences. The effect of randomly flattening point cloud data into a 1-D point sequence and feeding it to Mamba for modeling will be significantly reduced. On this basis, the present invention proposes a consistent traversal serialization (CTS) strategy, which ensures that adjacent points in the sequence are also adjacent in spatial position. Figure 6 It is illustrated that CTS converts a 3-D point cloud into a 1-D sequence.

[0084] First, the point cloud is sampled to convert the continuous space coordinates into discrete grid coordinates:

[0085]

[0086] in, and are the spatial coordinates of the point (i.e. the position of the point in three-dimensional space) and the grid coordinates, and N is the total number of grids.

[0087] The present invention designs a coding function, given two-dimensional coordinates, to map the coordinates to codes. Sorting the sequence according to the codes can ensure that adjacent points are continuous in space.

[0088]

[0089] Among them, n 1 and n 2 Respectively represent the grid coordinates of the point in two dimensions, N represents the total number of grids, which is used to determine the resolution of the grid, and Code_func is a coding function. 2 The parity of determines how to calculate the encoding value;

[0090] The present invention can then calculate the code based on the grid coordinates of each point:

[0091]

[0092] in, is the grid coordinate of the point, indicating its position in the three-dimensional grid; the purpose of code is to map the three-dimensional grid coordinate to a unique position in the one-dimensional sequence; in this way, each point in the point cloud can be assigned a unique serial number according to its position in space;

[0093] Sorting the point cloud according to the code allows it to be flattened into a 1D sequence. The performance of this simple serialization strategy is comparable to well-designed z-order and Hilbert-order serialization strategies.

[0094] In addition, by exchanging the order of the x, y, and z axes, six different serialization methods can be derived, which the present invention refers to as "xyz", "xzy", "yxz", "yzx", "zxy", and "zyx". These different serialization methods can be viewed as various point cloud observations from different spatial perspectives.

[0095] In addition, combining multiple serialization strategies can effectively help Mamba better model point cloud features. Specifically, the present invention uses different serialization strategies for the input of different Mamba layers, allowing Mamba to perceive the point cloud more comprehensively, thereby significantly exceeding a single serialization strategy.

[0096] 2) Design order prompt;

[0097] When combining multiple serialization strategies, an identifier needs to be assigned to each serialization. The identifier can help the Mamba layer identify the arrangement of point cloud sequences and better capture the point cloud features. A simple but effective sequential hint mechanism is proposed to achieve this goal, which is similar to the system message in large language models.

[0098] like Figure 7 As shown, the present invention allocates N p Learnable embeddings are used as sequential cues. Before being processed by the Mamba layer, the point cloud sequence has corresponding sequential cues added to the beginning and end of the sequence. Considering that the input feature dimensions of the Mamba layer at different stages may be different, the present invention also allocates a shared linear layer to all Mamba layers at each stage to map the sequential cues to the required channel size.

[0099] 3) Design positional embedding;

[0100] In sequence modeling, position encoding is crucial and widely used in text and image patch sequence modeling. For example, RoPE and learnable position embedding are widely used in text sequence modeling and image patch sequence modeling, respectively. However, the present invention finds that these position encoding methods are not suitable for point clouds due to their sparsity and irregular shapes. The spatial distance between any two adjacent points in a point cloud sequence varies greatly, so it is difficult to accurately represent the spatial gap between adjacent points using sequence position information. The present invention adopts a shared position mapping function to address this challenge. It maps the point space coordinates to a position encoding with the same channel size as the feature.

[0101]

[0102] Among them, Embed pos Refers to the position encoding projected from the point space coordinates; Linear is a linear transformation function, a fully connected layer that maps the three-dimensional spatial coordinates of the point to a high-dimensional space to obtain the position embedding. This simple position mapping function accurately encodes the spatial information of the point into features, and the position embeddings corresponding to adjacent points in the sequence are similar. Since the channel size of the feature varies at different stages of the network, a private position embedding function needs to be learned for each stage, and different Mamba layers at the same stage share the same position encoding function.

[0103] Step 2: Build a joint embedding initialization module to embed joint points and initialize patches;

[0104] Joint point embedding plays a crucial role in capturing the kinematic topology between joints. To this end, the concept of joint embedding originally proposed by HandFoldingNet can be used to obtain the initial joint embedding.

[0105] The joint embedding initialization module first feeds the superpoints with features F into two point set convolutional layers to generate a 3D global vector. Then, the joint point embeddings are initialized sequentially through three bias induction layers (BIL). The global vector is replicated J times and fed to the BIL; this operation generates embeddings for J individual joints. It is worth noting that BIL provides joint-independent biases, similar to learnable position encodings. After obtaining the joint encodings, a linear transformation is applied to project the high-dimensional embeddings into 3D space as the initial positions for collecting patches. The 3D point patches are obtained by grouping the k nearest superpoints with their corresponding features around each patch location.

[0106] Step 3: Build an adaptive graph transformer (AdaptiveGraphTransformer);

[0107] This paper proposes a multi-stage iterative refinement method of joint points based on adaptive kinematic and local geometric features of Transformer to adaptively weight local details and motion correspondences for accurate hand pose estimation under various occlusion conditions. The proposed transformer can learn the precise local geometric information of the hand, as a traditional local transformer directly calculates geometric features and kinematic topological information in 3D joint space. This design is to circumvent the 2D to 3D nonlinearity when running in 2D space. In addition, for accurate estimation, a novel adaptive attention module is integrated, which adaptively activates local geometric information or information from kinematically corresponding joints according to whether the joints are well presented or occluded. At the same time, the super point features encoded by PCM are combined with the graph transformer, which can simultaneously utilize the global feature capture ability of PCM and the modeling ability of the graph transformer for local neighborhood structure. This combination allows the model to capture local geometric structure in detail while maintaining global perception, thereby achieving better performance in 3D hand pose estimation.

[0108] The adaptive transformer decoder is a key component of the proposed framework, which queries local information based on the joint embeddings obtained by GCN based on kinematic correspondences to achieve accurate joint estimation. The joint embeddings are first enhanced by GCN to form an evolved embedding.

[0109] E′=ReLU(AEW)

[0110] Then, each super point and its corresponding features in each 3D patch are concatenated with the evolved joint embedding to form the input of the proposed novel attention mechanism of the Transformer.

[0111] Q K =MLP([E′,F k ])

[0112] K = set_conv k ([E′,F k ])

[0113]

[0114] Among them, MLP is a multi-layer perceptron, set_conv k is the point set convolution layer, σ is the sigmoid activation function, [.,.] is the connection operation, and ⊙ is the Hadamard product.

[0115] The present invention notices that the semantic difference between local point and key point embeddings affects the effectiveness of attention, and uses evolved key point embeddings to enhance the query (local point) Q, while the key (key point embedding) K is fused with local observations. In addition, unlike traditional attention that usually uses softmax dot product, the attention score of the present invention is obtained through Hadamard product with sigmoid as activation. In addition, due to the channel-level multiplication of Hadamard product, the proposed attention mechanism can be regarded as a multi-head attention that inherently uses the same number of attention heads as the channel depth. It is worth mentioning that attention produces two separate scores for local points and graph evolved key embeddings. Through these two scores, the proposed transformer is able to adaptively pay more attention to the kinematic topology or local information of the key points according to whether the key points are occluded by local points or well represented.

[0116] Following the standard attention block, a residual mechanism is introduced to accumulate the calculated value V and the input local point. Then, another point set convolution is used as a feed-forward module to aggregate the calculated value V into the updated joint point embedding for position regression of each joint point.

[0117] Embed = set_conv F (V k +F k )+MAX F (V k +F k )

[0118] Among them, MAX F represents the maximum pooling operation pursuing residual accumulation. Finally, a trainable linear transformation is adopted to project the updated keypoint embedding onto the 3D keypoint coordinates. To improve the estimation performance, the proposed transformer is stacked multiple times and the keypoint positions are iteratively optimized. The regressed positions from the previous transformer are used to group new patches as the input of the subsequent transformer stack.

[0119] The adaptive graph transformer (AGT) network architecture proposed in this invention is as follows: Figure 8 shown.

[0120] Step 4: Design a data enhancement method (PixDropout) to enhance the data and train the adaptive graph transformer PCM-AGT based on the point cloud state space model;

[0121] Data augmentation methods aim to increase the amount and diversity of training data by randomly creating novel and realistic data samples.

[0122] In recent years, data augmentation methods in vision, NLP, and speech have made significant progress. In the image domain, new samples are created by applying a set of transformations to available samples. These transformations can be roughly divided into two groups: ① geometric transformations and ② appearance transformations. A wide range of appearance transformations have been proposed recently to perform data augmentation on RGB images, such as color jittering, histogram equalization, and contrast adjustment. However, most of them are not applicable to depth images due to their different nature, limiting the set of data augmentation functions used by depth-based hand pose estimation methods, which mostly apply to geometric transformations such as random rotations, translations, and scaling.

[0123] In the present invention, a data enhancement method PixDropout is proposed based on the existing data enhancement method, such as Fig. 9 As shown in Figure 1, this is a simple yet effective appearance transfer that works well as a data augmentation function for depth images. Inspired by Dropout. While Dropout randomly selects and discards neurons in a neural network layer, PixDropout randomly discards some pixels on the hand surface in the input depth image. Most similar to previously proposed RGB image augmentation methods, they randomly sample and then mask rectangular regions of the input image to simulate occlusion for image recognition tasks. However, this method does not apply spatial priors (e.g., it samples individual pixels, not rectangular regions). Despite its simplicity, it is an effective data augmentation function for deep learning-based hand pose estimation methods.

[0124] The adaptive graph transformer model based on the point cloud state space model includes a super point feature encoding module, a joint embedding initialization module, and an adaptive transformer decoding module. After enhancing the training samples using the pixel random discarding data enhancement method PixDropout, a training set is constructed, and the adaptive graph transformer based on the point cloud state space model is trained using the training set.

[0125] In summary, the 3D global feature capturer PCM module of the 3D hand posture estimation algorithm based on the fusion of depth map and point cloud, the adaptive graph transformer AGT and the data enhancement method PixDropout for the depth map are the key technologies of the present invention. The present invention designs a point cloud global feature capturer PCM based on a state-space model. This module is specifically used to capture the global knowledge prior and local area fusion feature information of the 3D hand point cloud structure; at the same time, by introducing the adaptive attention mechanism technology, the weights of the local geometric information and the kinematic corresponding joint information are dynamically adjusted according to the visibility and occlusion of the hand joints, thereby enhancing the model's recognition ability for the hand joints. In addition, a data enhancement strategy PixDropout for the depth map is proposed and applied to the present invention to simulate the occlusion and self-occlusion in real visual conditions to obtain more diverse high-quality data.

[0126] Experimental verification:

[0127] In order to comprehensively evaluate the proposed algorithm, the present invention conducts extensive quantitative evaluation on the NYU dataset. In addition, the present invention is compared with seven advanced 3D hand pose estimation algorithms. The seven 3D hand pose estimation algorithms include three methods based on depth maps, three methods based on point clouds and the present invention's benchmark model depth map and point cloud fusion methods, namely AWR, SSRN, PHG, HandPointNet, HandFolding, HandR2N2, and IPNet. The implementation of these seven algorithms has been disclosed in the prior art, and the parameters set by the present invention are also consistent with the public parameters.

[0128] In terms of quantitative evaluation, the present invention has only one evaluation index, the mean keypoint error, which objectively evaluates the effect of posture estimation. The smaller the value of the mean keypoint error, the higher the accuracy of the estimation. The quantitative results of the NYU dataset are shown in Table 1.

[0129] Table 1 Quantitative comparison of the NYU dataset

[0130]

[0131]

[0132] As shown in Table 1, the proposed method shows significant advantages in average keypoint error. The results show that the proposed method achieves a new record of 7.06mm average distance error on the challenging NYU single-hand dataset. In addition, the results show that the proposed method greatly outperforms other 2D depth image based methods. This advantage is attributed to the direct processing of PCM-AGT in 3D space, which bypasses the highly nonlinear mapping problem associated with estimation from 2D images.

[0133] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An adaptive graph 3D hand posture estimation method based on depth map and point cloud fusion, characterized in that: The following steps are involved: S1. Use the point cloud global feature capturer to extract 3D point features based on 3D point cloud data, convert the 3D point cloud data into a 1D point sequence, and use a consistent traversal serialization strategy to make adjacent points in the sequence also adjacent in spatial position; S2, using 2D autoencoder to extract local 2D visual features based on depth image data, and fusing 3D point features and local 2D visual features to form super point features for feature encoding; S3, joint point embedding initialization and patch initialization based on super point features; S4. Graph Transformer is used in combination with adaptive attention mechanism to decode the super-point features after joint embedding initialization and patch initialization, adaptively weight local details and motion correspondences, and perform hand posture estimation.

2. The method for estimating three-dimensional hand posture based on adaptive graph based on depth map and point cloud fusion according to claim 1, characterized in that: The point cloud global feature capturer in S1 adopts Point Cloud Mamba; the details of S1 are as follows: S101, providing various point cloud sequences through consistent traversal serialization strategy and its variants, and maintaining the relationship information between points by traversing multiple point sequences; S102, introduce sequence hints to enable the Mamba layer to process point sequences generated by different serialization strategies; S103, introducing a position coding method based on spatial coordinate mapping, and enhancing the spatial position information of the point through position coding.

3. The method for estimating three-dimensional hand posture based on adaptive graph based on depth map and point cloud fusion according to claim 2, characterized in that: The S101 is specifically as follows: First, the point cloud is sampled to convert the continuous space coordinates into discrete grid coordinates: in, and are the spatial coordinates and grid coordinates of the point, and N is the total number of grids; Design an encoding function that maps coordinates to codes given two-dimensional coordinates; sort the sequence according to the codes to ensure that adjacent points are continuous in space; Where n1 and n2 represent the grid coordinates of the point in two dimensions respectively, N represents the total number of grids, which is used to determine the resolution of the grid, and Code_func is a coding function that determines how to calculate the coding value according to the parity of n2; Then, the code is calculated based on the grid coordinates of each point: in, is the grid coordinate of the point, indicating its position in the three-dimensional grid; the purpose of code is to map the three-dimensional grid coordinate to a unique position in the one-dimensional sequence; the point cloud is sorted according to the code and flattened into a one-dimensional sequence.

4. The method for adaptive graph 3D hand posture estimation based on depth map and point cloud fusion according to claim 3, characterized in that: By exchanging the order of x, y, and z axes, six different serialization strategies are derived, namely "xyz", "xzy", "yxz", "yzx", "zxy", and "zyx"; different serialization strategies are used for the input of different Mamba layers.

5. The method for adaptive graph 3D hand posture estimation based on depth map and point cloud fusion according to claim 4, characterized in that: The S102 is specifically as follows: When combining multiple serialization strategies, an identifier is assigned to each serialization, which is used by the Mamba layer to identify the arrangement of the point cloud sequence; Assign N to each serialization order p A learnable embedding is used as a sequence cue; before being processed by the Mamba layer, the point cloud sequence is added to the beginning and end of the sequence with the corresponding sequence cue; and a shared linear layer is assigned to all Mamba layers in each stage to map the sequence cue to the required channel size.

6. The method for adaptive graph 3D hand posture estimation based on depth map and point cloud fusion according to claim 5, characterized in that: The S103 is specifically as follows: A shared position mapping function is used to map the spatial coordinates of a point to a position encoding with the same channel size as the feature; Among them, Embed pos Represents the position encoding projected from the spatial coordinates of the point; Linear is a linear transformation function and a fully connected layer that maps the three-dimensional spatial coordinates of the point to a high-dimensional space to obtain position embedding.

7. The method for adaptive graph 3D hand posture estimation based on depth map and point cloud fusion according to claim 6, characterized in that: The joint point embedding initialization in S3 is specifically as follows: the super point features are fed to two point set convolutional layers to generate a 3D global vector, and the joint point embedding is initialized in sequence through three bias induction layers; the global vector is replicated J times and provided to the bias induction layer to generate embeddings for J individual joints; Specifically, after obtaining the joint encoding, a linear transformation is applied to project the high-dimensional embedding into the 3D space as the initial position for collecting patches, and the k nearest super-point features are grouped with their corresponding features around each patch position to obtain a 3D point patch.

8. The method for adaptive graph 3D hand posture estimation based on depth map and point cloud fusion according to claim 6, characterized in that: The S4 is specifically as follows: The joint embedding E is first enhanced by GCN to form the evolved joint embedding E′; E′=ReLU(AEW) Then each super point in each 3D point patch and its corresponding features are concatenated with the evolved joint point embedding to form the input of the adaptive attention mechanism; specifically as follows: Q K =MLP([E′,F k ]) K=set_conv k ([E′,F k ]) Among them, MLP is a multi-layer perceptron, set_conv k is the point set convolution layer, σ is the sigmoid activation function, [.,.] is the connection operation, and ⊙ is the Hadamard product; A residual mechanism is introduced to accumulate the calculated value V and the input local points; the point set convolution is used as a feed-forward module to aggregate the calculated value V into the updated joint point embedding for position regression of each joint point; Embes=set_conv F (V k +F k )+MAX F (V k +F k ) Among them, MAX F Represents the maximum pooling operation that pursues residual accumulation; Finally, a trainable linear transformation is used to project the updated keypoint embeddings onto the 3D keypoint coordinates.

9. An adaptive graph transformer model based on a point cloud state space model applied in the method according to any one of claims 1 to 8, characterized in that: include: A super point feature encoding module, comprising a point cloud global feature capturer using Point Cloud Mamba and an autoencoder based on ConvNeXt; the point cloud global feature capturer introduces a consistent traversal serialization strategy, an order prompt, and a position encoding method based on spatial coordinate mapping; the super point feature encoding module is used to form a super point feature obtained by fusing 3D point features and local 2D visual features; The joint embedding initialization module uses the joint embedding in HandFoldingNet to initialize the joint point embedding; it is used to initialize the joint point embedding and patch initialization of the super point feature; Adaptive transformer decoding module; uses graph transformer and combines adaptive attention mechanism to dynamically adjust the weights of local geometric information and kinematic corresponding joint information according to the visibility and occlusion of hand joints.

10. The adaptive graph transformer model based on point cloud state space model according to claim 9, characterized in that: After collecting data and performing data enhancement, a training set is constructed and the model is trained using the training set. The data enhancement method PixDropout is used to randomly discard pixels.

Images