A three-dimensional hand key point recovery method and system
By employing differentiable discrete coding and image-guided conditional generation mechanisms, combined with rotation-aligned quantization and diffusion models, the robustness problem of 3D gesture estimation in occluded scenarios is solved, achieving high-precision and stable 3D gesture recovery.
Patent Information
- Application Number
- CN202511537191.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2045-10-27
AI Technical Summary
Existing 3D gesture estimation methods struggle to consistently output high-quality prediction results in complex scenarios such as hand occlusion and blurred details, exhibiting insufficient robustness. In particular, recovery is difficult, errors are large, and stability is poor when key hand points are occluded.
Differentiable discrete coding technology and image-guided conditional generation mechanism are adopted. Differentiable paths are constructed through rotation alignment quantization method. Combined with diffusion model, the key points of 3D hand are gradually recovered. Pose reconstruction is performed using structured codebook and image-guided features.
It improves the recovery capability under occlusion conditions, enhances the accuracy of small joint recognition and the stability of overall estimation, and achieves high-precision 3D gesture recovery.
Smart Images
Figure CN121010989B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of three-dimensional gesture recognition technology, specifically to a method and system for restoring key points of a three-dimensional hand. Background Technology
[0002] 3D gesture estimation is an important research direction in computer vision and artificial intelligence, with wide-ranging value in applications such as intelligent interaction, human-computer interfaces, virtual reality, and robot control. By acquiring the positional information of key points of the hand in 3D space, natural and intuitive human-computer interaction can be achieved, such as gesture control in augmented reality environments or completing tasks with gesture commands in remote collaborative systems. Therefore, how to efficiently and accurately acquire the 3D pose of key hand points has become one of the core problems in this field.
[0003] Currently, mainstream 3D gesture estimation methods can be mainly divided into detection-based methods and regression-based methods. The former typically uses a deep learning network to output a heatmap or confidence map for each keypoint and infers the 3D position through spatial projection or camera parameters; the latter directly extracts global or local features from image or video frames and predicts the coordinates of keypoints in 3D space through a regression module. Both methods achieve good results under standard datasets and clear image conditions.
[0004] However, in practical applications, especially in complex scenarios where the hand is occluded, existing methods generally suffer from insufficient robustness. The hand structure is highly flexible and exhibits significant self-occlusion, with some key points potentially completely invisible in the image, posing a serious challenge to the model's generalization ability. Furthermore, some joints of the hand (such as fingertips) are small in scale and have weak texture, easily becoming obscured by background interference in the image, further exacerbating the recognition difficulty. Therefore, existing technologies often suffer from missed detections and false detections under occlusion conditions, and may even exhibit pose jumps within consecutive frames, making them unstable for complex interactive scenarios.
[0005] To overcome the aforementioned problems, researchers have recently attempted to introduce latent variable modeling and generative modeling techniques to enhance occlusion recovery capabilities by modeling pose distributions. Among these, discrete latent representations, due to their structured and composable characteristics, have become an effective means of improving occlusion recovery capabilities. However, traditional discretization methods are often non-differentiable, limiting gradient propagation capabilities during training.
[0006] In summary, existing 3D gesture estimation methods struggle to consistently deliver high-quality prediction results in scenarios with occluded hand keypoints and blurred details. To meet the dual demands of high accuracy and robustness in practical applications such as intelligent interaction, a novel 3D gesture estimation method is urgently needed. Summary of the Invention
[0007] To address the problems of difficulty, large errors, and poor stability in key point recovery under occlusion conditions in existing technologies, this invention provides a method and system for recovering 3D hand key points. It aims to overcome the problems of lost pose estimation information and incomplete key point recognition caused by occlusion, and achieve high-precision and robust recovery of 3D hand key points in complex occlusion scenarios.
[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:
[0009] In a first aspect, the present invention proposes a method for restoring key points of a three-dimensional hand, comprising the following steps:
[0010] Obtain the hand image and 3D hand key point coordinates;
[0011] Input the coordinates of the three-dimensional hand key points into the hand posture encoder to extract the latent structural feature vector;
[0012] Find the nearest neighbor codeword of the potential structural feature vector in the structured codebook, construct pseudo-quantization features by rotation and alignment, and use them to replace the real quantization results in training to achieve gradient propagation;
[0013] Image encoding is performed on hand images to extract image guidance features;
[0014] Conditional diffusion modeling is performed on the noisy pose codeword index. With the help of image-guided features, the discrete representation of the original pose is predicted and restored step by step to obtain the restored pose codeword index.
[0015] The recovered pose codeword index is looked up, mapped, and decoded to obtain the predicted coordinates of the 3D hand key points.
[0016] Furthermore, the hand posture encoder includes multiple local feature modeling units stacked together. Each local feature modeling unit includes a normalization layer, a joint spatial displacement modeling module, and a channel orientation feature enhancement module, and performs feature fusion through a residual connection structure.
[0017] Furthermore, the joint spatial displacement modeling module maps the input pose features to a high-dimensional latent space through a linear embedding operation, then segments the channel dimension and applies sliding offset operations of different scales within each channel segment to simulate the spatial adjacency relationship of hand joints.
[0018] Furthermore, finding the nearest neighbor codewords of the latent structural feature vectors in the structured codebook and constructing pseudo-quantization features through rotation alignment includes:
[0019] First, for each potential structural feature vector, find its nearest neighbor codeword vector in the structured codebook. Then, for each potential structural feature vector, perform rotation and scaling transformation to align the direction of each potential structural feature vector to its corresponding nearest neighbor codeword vector, thereby obtaining a pseudo-quantized alternative vector that can be used for training. Finally, form a direction-aligned pose code and obtain the quantized pose codeword index.
[0020] Furthermore, the noise-adding process for the attitude codeword index includes:
[0021] The occlusion probability is used to determine the occlusion probability of certain positions in the quantized attitude codeword index. Replaced with a uniform placeholder vector, or replaced with probability. Randomly replace with other valid codebooks, while retaining a certain percentage. The original pose codebook features; where K is the number of three-dimensional hand keypoints.
[0022] Furthermore, conditional diffusion modeling is performed on the noisy pose codeword index to progressively predict and reconstruct the discrete representation of the original pose with the aid of image-guided features, including:
[0023] A forward perturbation sequence is constructed based on the quantized attitude codeword index. In each iteration step, the perturbation state for the next step is sampled from the conditional probability distribution by combining the diffusion model with image-guided features. The original attitude codeword index is then recovered through progressive reverse iteration.
[0024] Furthermore, the diffusion model employs a multi-head attention-based denoiser to conditionally model the current perturbation state based on image guidance features and temporal context, predict the state distribution of the current perturbation state, and then determine the conditional probability based on the state distribution of the current perturbation state to perform stepwise sampling.
[0025] Furthermore, the loss function of the sampling process includes a variational lower bound and an auxiliary decoding target.
[0026] Furthermore, feature decoding is performed on the recovered pose codebook features to obtain the 3D hand keypoint reconstruction results, including:
[0027] By using a multi-layer local multilayer perceptron stacked decoder, the pose recovery task is performed on the recovered pose codebook features under occlusion or missing conditions, predicting the complete set of hand keypoint coordinates in 3D space.
[0028] Secondly, this invention proposes a three-dimensional hand key point recovery system, comprising:
[0029] The data acquisition module is used to acquire hand images and 3D hand key point coordinates;
[0030] The encoding module is used to input the coordinates of the three-dimensional hand key points into the posture encoder to extract the latent structural feature vector;
[0031] The pose quantization module searches for the nearest neighbor codewords of potential structural feature vectors in the structured codebook, constructs pseudo-quantized features through rotation and alignment, and uses them to replace the real quantization results in training to achieve gradient propagation.
[0032] The image feature extraction module is used to encode hand images and extract image guidance features;
[0033] The conditional diffusion module is used to perform conditional diffusion modeling on the noisy pose codeword index. With the help of image-guided features, it gradually predicts and restores the discrete representation of the original pose to obtain the restored pose codeword index.
[0034] The decoding module is used to look up and decode the recovered posture codeword index to obtain the predicted coordinates of the 3D hand key points.
[0035] The present invention has the following beneficial effects:
[0036] (1) The present invention combines differentiable discrete coding technology with image-guided conditional generation mechanism to improve the recovery capability under occlusion conditions and enhance the accuracy of small joint recognition and the stability of overall estimation.
[0037] (2) The vector quantization method based on the rotation trick proposed in this invention aligns the encoder output features to their corresponding nearest neighbor codeword directions by introducing a rotational affine transformation in the forward propagation stage, and inputs the alignment result into the decoder. This rotation process is treated as a constant during backpropagation, thereby constructing a differentiable path and indirectly optimizing the traditional non-differentiable quantization process. This method helps to improve the encoding expressive power while maintaining quantization accuracy and efficiency, and provides a stable training path for subsequent pose recovery based on the diffusion model. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the overall process of a three-dimensional hand key point recovery method;
[0039] Figure 2 This is a schematic diagram of the training process for a three-dimensional hand keypoint recovery method;
[0040] Figure 3 A schematic diagram of the reasoning process for a three-dimensional hand key point recovery method;
[0041] Figure 4 This is a schematic diagram of the principle of the rotation alignment quantization module. Detailed Implementation
[0042] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0043] Example 1
[0044] like Figure 1 As shown, an embodiment of the present invention provides a method for restoring key points of a three-dimensional hand, including the following steps S1 to S6:
[0045] S1. Obtain the hand image and the coordinates of the 3D hand key points;
[0046] S2. Input the coordinates of the three-dimensional hand key points into the posture encoder to extract the latent structural feature vector;
[0047] In an optional embodiment of the present invention, the hand posture encoder in step S2 includes multiple local feature modeling units stacked together. Each local feature modeling unit includes a normalization layer, a joint spatial displacement modeling module, and a channel direction feature enhancement module, and performs feature fusion through a residual connection structure.
[0048] The joint spatial displacement modeling module maps the input pose features to a high-dimensional latent space through a linear embedding operation, then segments the channel dimension and applies sliding offset operations of different scales within each channel segment to simulate the spatial adjacency relationship of hand joints.
[0049] In this embodiment, as Figure 2 As shown, to effectively extract the semantic structure information of the input 3D hand keypoints, the original 3D keypoint coordinates are input into the hand pose encoder for feature modeling. The 3D hand keypoints consist of J joints, each represented by 3D coordinates (x, y, z). The overall input format is as follows: The hand pose encoder is constructed using a multi-layer local perception structure, consisting of several stacked local feature modeling units. Each modeling unit includes a normalization layer (LayerNorm), a joint-wise shift modeling module, and a channel-wise feature enhancement module. Features are fused through a residual connection structure to preserve the original structural information from the multi-layer modeling process.
[0050] The joint spatial displacement modeling module maps the input pose features to a high-dimensional latent space through a linear embedding operation. It then segments the channel dimensions and applies sliding offset operations of different scales within each channel segment to simulate the spatial adjacency of hand joints, thus explicitly introducing local structure perception. This operation is achieved through parameterless segmented translation, effectively improving the model's ability to model geometric dependencies between joints while maintaining dimensionality consistency. The channel orientation feature enhancement module models and enhances the channel dimension features of each joint to improve the expressiveness and discriminative power of the joint representation. This module employs a multilayer perceptron (MLP) structure, processing the channel vectors of each joint independently. It first increases the dimensionality through linear transformation, then introduces a nonlinear expression through a nonlinear activation function, and finally restores the original dimension. This is then fused with the original input through a residual structure, achieving efficient channel feature modeling. The intermediate features output by each local feature modeling unit are passed as input to the next layer. After multiple layers are stacked, a set of structured latent representation features is output, denoted as... ,in Indicates the first i Feature encoding of each structural unit.
[0051] After processing by the hand pose encoder, the original 3D joint sequence is mapped into a set of continuous, dimension-consistent latent representation vectors, providing a structurally complete and generalizable intermediate representation space for subsequent discretization encoding and diffusion recovery.
[0052] S3. Find the nearest neighbor codeword of the potential structural feature vector in the structured codebook, construct pseudo-quantization features by rotation and alignment, and use them to replace the real quantization results in training to achieve gradient propagation.
[0053] Step S3 involves finding the nearest neighbor codewords of the potential structural feature vectors in the structured codebook and constructing pseudo-quantized features through rotation alignment, including:
[0054] First, for each latent feature vector, its nearest neighbor codeword vector is found in the structured codebook. Then, each latent feature encoding is oriented to its corresponding nearest neighbor codeword vector through rotation and scaling transformation, thereby obtaining a pseudo-quantized replacement vector that can be used for training. Finally, an orientation-aligned pose encoding is formed for subsequent reconstruction.
[0055] like Figure 4 As shown, the continuous latent features are encoded The input is fed into the Rotation-Aligned Tokenization Module (RATM). This module utilizes an affine rotation mechanism in the forward propagation phase to align the pose encoding output to the nearest neighbor codeword direction, and treats this as a constant during backpropagation, thus constructing a differentiable path to aid training. For each structural feature vector... Searching for it in the structured codebook Nearest Neighbor Codebook ,satisfy:
[0056]
[0057] The structured codebook is a set of learnable vectors obtained through training, used to discretely partition the input feature space. Specifically, the codebook is jointly trained by a pre-built hand pose encoder and a rotation-aligned quantization module (RATM). During training, a large amount of real 3D hand keypoint data is input into the encoder to extract continuous latent structural features. These features are then subjected to a differentiable discrete mapping using RATM. The discrete representation of each feature vector is determined based on its distance from the vectors in the codebook, and the codebook vectors are updated according to the quantization error. The structured codebook continuously adapts and adjusts during training, ensuring that its vector distribution fully covers the real 3D gesture space and possesses semantic discriminability, thereby achieving effective partitioning and representation of the structural latent space.
[0058] To overcome the limitation of non-differentiability in traditional quantization, this embodiment introduces a rotation and scaling transformation mechanism to transform each structural feature vector. Convert to a rotation-aligned vector that matches the direction of its corresponding codeword:
[0059]
[0060] in To transform the structural feature vector Rotate to nearest neighbor codebook Orthogonal matrix of directions This represents the magnitude scaling factor. The rotation matrix and scaling factor in the above transformation are treated as constants during backpropagation, avoiding the difficulty of calculating the gradient for non-differentiable quantization operations.
[0061] The final pseudo-quantization representation of the attitude is:
[0062]
[0063] It retains both discrete structure and gradient transitivity.
[0064] This invention proposes a vector quantization method based on the rotation trick. By introducing a rotational affine transformation in the forward propagation stage, the encoder output features are aligned to their corresponding nearest-neighbor codeword directions, and the alignment result is input into the decoder. This rotation process is treated as a constant during backpropagation, thus constructing a differentiable path and indirectly optimizing the traditional non-differentiable quantization process. This method maintains quantization accuracy and efficiency while improving encoding expressive power and providing a stable training path for subsequent pose recovery based on a diffusion model.
[0065] Pseudo-quantized pose representations are input to the decoder for pose reconstruction, and a supervision objective is introduced:
[0066]
[0067] in Let j be the coordinates of the reconstructed hand key point. This corresponds to the actual label.
[0068] During training, the commitment loss is also minimized jointly:
[0069]
[0070] in To stop the gradient operator.
[0071] This module significantly enhances the trainability and expressive power of the pose discretization process while maintaining efficient encoding capabilities.
[0072] In this embodiment, the training dataset used for model training can be derived from publicly available hand 3D pose datasets, including but not limited to FreiHAND, HO3D, and InterHand2.6M. These datasets all provide RGB images of the hand and corresponding 3D keypoint coordinate annotations, which can be used to train the hand pose encoding module, discrete quantization module, and diffusion recovery module in this invention. Public datasets cover diverse gestures, viewpoints, and occlusion scenarios, helping to improve the model's generalization ability and occlusion recovery performance. To enhance model robustness, this embodiment can also construct a training sample set based on public datasets, combining simulated synthetic data or performing data augmentation processing, thereby further improving the system's application capabilities in real-world complex scenarios.
[0073] S4. Perform image encoding on the hand image and extract image guidance features;
[0074] In an optional embodiment of the present invention, to guide the proper restoration of the occluded segment, step S3 uses an image encoder. Extract image guidance features corresponding to pose .
[0075] S5. Perform conditional diffusion modeling on the noisy pose codeword index. With the help of image-guided features, gradually predict and restore the discrete representation of the original pose to obtain the restored pose codeword index.
[0076] In an optional embodiment of the present invention, step S4 of adding noise to the attitude codeword index includes:
[0077] The occlusion probability is used to determine the occlusion probability of certain positions in the quantized attitude codeword index. Replaced with a uniform placeholder vector, or replaced with probability. Randomly replace with other valid codebooks, while retaining a certain percentage. The original pose codebook features; where K is the number of three-dimensional hand keypoints.
[0078] Step S4 performs conditional diffusion modeling on the noisy pose codeword index, and with the assistance of image-guided features, progressively predicts and reconstructs the discrete representation of the original pose, including:
[0079] A forward perturbation sequence is constructed based on the quantized attitude codeword index. In each iteration step, the diffusion model samples the next perturbation state from the conditional probability distribution using image-guided features. Through progressive reverse iteration, the original attitude codeword index is recovered and mapped to an attitude feature representation using a lookup table.
[0080] The diffusion model employs a multi-head attention-based denoiser to conditionally model the current perturbation state based on image guidance features and temporal context, predict the state distribution of the current perturbation state, and then determine the conditional probability based on the state distribution of the current perturbation state to perform stepwise sampling.
[0081] In this embodiment, after completing the structured discrete pose representation learning, to enhance the model's adaptability to occlusion, missing, or blurred joint inputs in complex scenes, a structural noise perturbation mechanism and image-guided conditional diffusion modeling are further introduced. During the training phase, the quantized pose codewords are first indexed... As the original input, through the perturbation function Construct a noisy version This is to simulate the invisibility and diverse uncertainties of attitude substructures. Specific perturbation methods include making partial vectors... With probability Replaced with a uniform placeholder vector or by probability Randomly replace with other valid codebooks, while retaining a certain percentage. The original pose structure. This mechanism can be achieved through the following transition matrix. Controlling each step of the disturbance process:
[0082]
[0083] in Control the occlusion ratio, Controlling the substitution ratio and noise scheduling based on the number of steps The linear change satisfies the prior assumption that the diffusion perturbation gradually increases.
[0084] The image guidance features are injected into the diffusion network along with the noisy pose input, forming the input pair for the conditional diffusion process. During the reverse reconstruction process, the model gradually recovers the occluded or replaced pose codebook based on image guidance information and contextual dependencies until a complete structural representation is reconstructed. Through this mechanism, the model can not only explicitly perceive the occlusion location but also learn the diverse possibilities of locally invisible joints, providing a solid foundation for subsequent temporal modeling and generative prediction.
[0085] In this embodiment, the noisy attitude codebook is represented. With image guidance features As a conditional input to the Pose-Guided Diffusion Module (PGDM), it performs a multi-step conditional backsampling process in the discrete state space to progressively recover missing or perturbed structural fragments. This module models the discrete diffusion process based on Markov chains and defines the forward perturbation sequence. ,in This represents the original discrete attitude index sequence. This represents the state of maximum perturbation; each step follows a preset transition probability matrix. Discrete perturbations at the label level are implemented, and occlusion and replacement mechanisms are explicitly modeled during the training phase. During the backsampling phase, the model learns from conditional probabilities... Generate the next state and iterate through the process. After denoising, the restored state is finally obtained. Then, the quantized attitude features are mapped back through a lookup table.
[0086] .
[0087] To improve prediction accuracy and sample diversity during the reverse process, a multi-head attention-based denoiser is introduced into the diffusion model structure, leveraging image-guided features and temporal context to denoise the current state. Perform conditional modeling to predict its initial state distribution. Then, based on this prediction, we can deduce... Stepwise sampling is performed. The Variational Lower Bound (VLB) is one of the core optimization objectives in the diffusion model, used to measure the model's ability to recover the original state at each denoising step. Its theoretical basis stems from variational inference, aiming to maximize the log-likelihood estimate of the initial true samples and provide gradient constraints on the rationality and coherence of the entire sampling sequence. In this embodiment, the Variational Lower Bound, by accumulating the prediction errors over multiple diffusion steps, guides the model to gradually approach the true discrete token distribution during the stepwise restoration process, improving generation stability and accuracy. The loss function for the entire sampling process is determined by the Variational Lower Bound. With auxiliary decoding target Together they form a whole, and the ultimate optimization goal is:
[0088]
[0089]
[0090] With the number of noise reduction steps As the probability decreases, the model updates its state at each step using the predicted distribution from the previous stage and integrates guiding information provided by the image encoder to progressively reconstruct a complete pose representation that conforms to semantic structure and human topological constraints. The final generated... The coordinates of the three-dimensional joints are then fed into the pre-trained decoder to reconstruct them. This method completes the pose recovery task under occlusion or missing conditions. This process not only demonstrates the model's ability to model uncertainties in occluded regions but also improves the accuracy and structural consistency of image-guided recovery, laying a robust foundation for human pose modeling under multimodal perception.
[0091] After completing conditional diffusion modeling and obtaining the recovered discrete attitude representation Then, the system maps the attitude index sequence back to the corresponding attitude codebook set. .
[0092] S6. Perform feature decoding on the recovered posture codebook features to obtain the 3D hand key point recovery results.
[0093] In an optional embodiment of the present invention, step S5 performs feature decoding on the recovered pose codebook features to obtain the three-dimensional hand keypoint recovery result, including:
[0094] By using a multi-layer local multilayer perceptron stacked decoder, the pose recovery task is performed on the recovered pose codebook features under occlusion or missing conditions, predicting the complete set of hand keypoint coordinates in 3D space.
[0095] In this embodiment, the recovered pose codebook features are input to a pre-trained decoding module to predict the complete set of hand keypoint coordinates in 3D space. The decoder's internal structure consists of stacked multi-layer local multilayer perceptrons, enabling it to adaptively model the spatial dependencies between structural units, ultimately outputting the keypoint set. ,in Let represent the three-dimensional coordinates of the j-th hand keypoint. This mapping process can be represented as:
[0096]
[0097] in Indicates the decoding function. This is the set of recovered pose features.
[0098] Since the preceding diffusion modeling fully utilizes image guidance and structural priors to model and repair occluded regions, the decoding stage can generate complete pose results that conform to anatomical structure and motion rationality without additional supervision signals. The final output keypoint set... The resulting 3D hand reconstruction can be directly used for downstream tasks such as action recognition and human-computer interaction. This module is structurally symmetrical to the encoding module, ensuring consistent information propagation and continuous semantic preservation.
[0099] Example 2
[0100] This invention provides a three-dimensional hand key point recovery system based on Embodiment 1, comprising:
[0101] The data acquisition module is used to acquire hand images and 3D hand key point coordinates;
[0102] The pose quantization module is used to encode and quantize the hand pose of the 3D hand key point coordinates to obtain pseudo-quantized pose codebook features.
[0103] The image feature extraction module is used to encode hand images and extract image guidance features;
[0104] The conditional diffusion module is used to add noise to the quantized attitude codeword index and then use the image-guided features as a conditional pair to jointly input the diffusion model to perform attitude diffusion recovery, thereby obtaining the recovered attitude codebook features.
[0105] The decoding module is used to perform feature decoding on the recovered posture codebook features to obtain the 3D hand key point recovery results.
[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0109] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.
[0110] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. A method for restoring key points of a three-dimensional hand, characterized in that, Includes the following steps: Obtain the hand image and 3D hand key point coordinates; Input the coordinates of the three-dimensional hand key points into the hand posture encoder to extract the latent structural feature vector; The nearest neighbor codewords of latent structural feature vectors are found in the structured codebook. Pseudo-quantized features are constructed by rotation and alignment to replace the real quantization results in training, thereby achieving gradient propagation. Specifically, for each latent structural feature vector, the nearest neighbor codeword vector is found in the structured codebook. Then, each latent structural feature vector is oriented to its corresponding nearest neighbor codeword vector by rotation and scaling transformation, thus obtaining pseudo-quantized replacement vectors that can be used for training. Finally, orientation-aligned pose encoding is formed, and the quantized pose codeword index is obtained. Image encoding is performed on hand images to extract image guidance features; Conditional diffusion modeling is performed on the noisy pose codeword index. With the help of image-guided features, the discrete representation of the original pose is predicted and restored step by step to obtain the restored pose codeword index. The recovered pose codeword index is looked up, mapped, and decoded to obtain the predicted coordinates of the 3D hand key points.
2. The method for restoring key points of a three-dimensional hand according to claim 1, characterized in that, The hand posture encoder includes multiple local feature modeling units stacked together. Each local feature modeling unit includes a normalization layer, a joint spatial displacement modeling module, and a channel orientation feature enhancement module, and performs feature fusion through a residual connection structure.
3. The method for restoring key points of a three-dimensional hand according to claim 2, characterized in that, The joint spatial displacement modeling module maps the input pose features to a high-dimensional latent space through a linear embedding operation, then segments the channel dimension and applies sliding offset operations of different scales within each channel segment to simulate the spatial adjacency relationship of hand joints.
4. The method for restoring key points of a three-dimensional hand according to claim 1, characterized in that, Noise-adding processing for the attitude codeword index includes: The occlusion probability is used to determine the occlusion probability of certain positions in the quantized attitude codeword index. Replaced with a uniform placeholder vector, or replaced with probability. Randomly replace with other valid codebooks, while retaining a certain percentage. The original pose codebook features; where K is the number of 3D hand keypoints.
5. The method for restoring key points of a three-dimensional hand according to claim 1, characterized in that, Conditional diffusion modeling is performed on the noisy pose codeword index. With the aid of image-guided features, the discrete representation of the original pose is predicted and reconstructed stepwise, including: A forward perturbation sequence is constructed based on the quantized attitude codeword index. In each iteration step, the perturbation state for the next step is sampled from the conditional probability distribution by combining the diffusion model with image-guided features. The original attitude codeword index is then recovered through progressive reverse iteration.
6. The method for restoring key points of a three-dimensional hand according to claim 5, characterized in that, The diffusion model employs a multi-head attention-based denoiser to conditionally model the current perturbation state based on image guidance features and temporal context, predict the state distribution of the current perturbation state, and then determine the conditional probability based on the state distribution of the current perturbation state to perform stepwise sampling.
7. A method for restoring key points of a three-dimensional hand according to claim 6, characterized in that, The loss function of the sampling process includes a variational lower bound and an auxiliary decoding target.
8. The method for restoring key points of a three-dimensional hand according to claim 1, characterized in that, The recovered pose codeword index is looked up, mapped, and decoded to obtain the predicted coordinates of the 3D hand key points, including: By using a multi-layer local multilayer perceptron stacked decoder, the pose recovery task is performed on the recovered pose codebook features under occlusion or missing conditions, predicting the complete set of hand keypoint coordinates in 3D space.
9. A three-dimensional hand key point recovery system based on the method of any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire hand images and 3D hand key point coordinates; The encoding module is used to input the coordinates of the three-dimensional hand key points into the posture encoder to extract the latent structural feature vector; The pose quantization module searches for the nearest neighbor codewords of potential structural feature vectors in the structured codebook, constructs pseudo-quantized features through rotation and alignment, and uses them to replace the real quantization results in training to achieve gradient propagation. The image feature extraction module is used to encode hand images and extract image guidance features; The conditional diffusion module is used to perform conditional diffusion modeling on the noisy pose codeword index. With the help of image-guided features, it gradually predicts and restores the discrete representation of the original pose to obtain the restored pose codeword index. The decoding module is used to look up and decode the recovered posture codeword index to obtain the predicted coordinates of the 3D hand key points.