Three-dimensional hand attitude estimation method based on frequency domain-space domain joint representation
By using a joint frequency-spatial representation method, a 3D hand pose estimation network is constructed, which solves the pose estimation error caused by occlusion in complex scenes and achieves more accurate hand pose estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI'AN POLYTECHNIC UNIVERSITY
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-24
AI Technical Summary
Existing 3D hand pose estimation methods suffer from occlusion interference in complex hand-object interaction scenarios, leading to the loss of detailed features such as fingertips and finger edges, resulting in pose estimation errors.
A frequency-spatial joint representation method is adopted. A three-dimensional hand pose estimation network is constructed. Local and global features are extracted by using a frequency-spatial collaborative feature decoupling module and a cross-domain fusion module, respectively. Feature fusion is then performed, and pose estimation is performed in combination with a MANO regressor.
Even when occluded, it can accurately capture detailed features such as finger edges and joint textures, while also grasping the overall outline of the hand, thus improving the accuracy of posture estimation.
Smart Images
Figure CN121921834A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer science, and more specifically, to a three-dimensional hand pose estimation method based on joint frequency-spatial domain representation. Background Technology
[0002] As a key technology in the field of human-computer interaction, 3D hand pose estimation has significant application value in scenarios such as intelligent driving, smart healthcare, and robot control. In intelligent driving systems, this technology enables accurate recognition of human-computer interaction commands by analyzing hand movement patterns; in the field of medical rehabilitation, posture prediction-based prosthetic control systems significantly improve patients' quality of life; and in the field of industrial robots, teaching-based robotic hand control systems achieve high-precision closed-loop control of grasping operations by capturing the operator's hand movement characteristics in real time. Notably, with the deepening development of the metaverse concept, this technology exhibits unique advantages in the field of virtual / augmented reality (VR / AR)—significantly enhancing the immersion and realism of virtual interaction scenarios by accurately reconstructing the topological structure of the user's hand skeleton.
[0003] From a technological evolution perspective, 3D hand pose estimation methods have undergone significant technological iterations. Early studies mostly employed traditional machine learning algorithms such as Support Vector Machines (SVM) and Random Forests for gesture feature modeling. While these methods exhibited good robustness in specific scenarios, they generally suffered from complex feature engineering and limited computational efficiency. In recent years, deep learning technology has driven breakthroughs in this field: vertex prediction frameworks based on the MANO parameterized hand model have achieved efficient pose reconstruction through low-dimensional representation learning; hybrid architectures integrating CNN-Transformer innovatively combine local feature extraction and global attention mechanisms; and some researchers have even proposed modeling the hand topology as a 3D graph structure, using Graph Convolutional Networks (GCNs) for spatial relationship modeling, which demonstrates stronger geometric representation capabilities than traditional convolutional neural networks. In recent years, 3D hand pose estimation methods based on monocular RGB images have achieved significant progress by combining deep convolutional networks with parameterized hand models (such as MANO). However, pose estimation in complex hand-object interaction scenarios still faces significant challenges. Severe occlusion interference can cause the loss of detailed features such as fingertips and finger edges during the learning process of deep convolutional networks, leading to pose estimation errors. Summary of the Invention
[0004] To overcome at least one deficiency in the prior art, this application provides a three-dimensional hand pose estimation method based on joint frequency-spatial domain representation.
[0005] Firstly, a three-dimensional hand pose estimation method based on joint frequency-spatial domain representation is provided, including: A 3D hand pose estimation network is constructed. The 3D hand pose estimation network includes a feature extractor, a frequency-spatial domain co-functionalized feature decoupling module, a cross-domain fusion module, and a MANO regressor. The frequency-spatial domain co-functionalized feature decoupling module includes a spatial local feature extraction branch, a spatial global feature extraction branch, and a frequency domain feature extraction branch. The 3D hand pose estimation network was trained based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses. During training, samples are input into a feature extractor to extract hand feature maps. These hand feature maps are then input into a frequency-spatial coordinating feature decoupling module. The spatial local feature extraction branch uses a pyramid pooling structure to extract local hand features, while the spatial global feature extraction branch uses a Swin Transformer to extract global hand features. The frequency domain feature extraction branch is based on a two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features. A cross-domain fusion module fuses high-frequency and local hand features to obtain high-frequency semantic features, fuses low-frequency and global hand features to obtain low-frequency semantic features, and then uses these high-frequency and low-frequency semantic features to obtain high-semantic hand features. These high-semantic hand features are then input into a MANO regressor to obtain hand pose estimation results. The RGB image of the hand pose to be processed is input into the trained 3D hand pose estimation network to obtain the hand pose estimation result.
[0006] In one embodiment, the spatial local feature extraction branch employs a pyramid pooling structure to extract local hand features, expressed by the following formula:
[0007]
[0008] in, This is a feature of pyramid pooling. Represents convolution. Indicates splicing, Represents hand feature map Perform max pooling with a kernel size of 5×5. Represents hand feature map Perform max pooling with a kernel size of 9×9. Represents hand feature map Perform max pooling with a kernel size of 13×13. This indicates batch normalization calculation. This refers to local hand features.
[0009] In one embodiment, the spatial global feature extraction branch uses the Swing Transformer to extract global hand features, expressed by the following formula:
[0010]
[0011] in, Features of the window after multi-head self-attention. This indicates that the window has multiple heads and self-attention. This indicates window partitioning operations. Represents convolution. Represents hand feature diagram, Represents global hand features. This indicates batch normalization calculation. Indicates splicing, Indicates to Perform horizontal stripe convolution. Indicates to Perform vertical stripe convolution.
[0012] In one embodiment, the frequency domain feature extraction branch is based on two-dimensional Haar wavelet transform to obtain low-frequency hand features and high-frequency hand features, including: Performing a two-dimensional Haar wavelet transform on the hand feature map yields one low-frequency component and three high-frequency components. Based on one low-frequency component and three high-frequency components, low-frequency hand features and high-frequency hand features are obtained using the following formula:
[0013]
[0014] in, Low-frequency hand features, This indicates batch normalization calculation. Represents convolution. Low-frequency components, High-frequency hand features Indicates splicing, There are three high-frequency components.
[0015] In one embodiment, the cross-domain fusion module fuses high-frequency hand features and local hand features to obtain high-frequency, high-semantic features, expressed by the following formula:
[0016]
[0017] in, High-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates high-frequency hand features Perform horizontal stripe convolution. Indicates high-frequency hand features Perform vertical stripe convolution. These are local hand features The corresponding query vector, key vector, and value vector. Indicates local hand features Perform horizontal stripe convolution. Indicates local hand features Perform vertical stripe convolution. It is a high-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
[0018] In one embodiment, the cross-domain fusion module fuses low-frequency hand features and global hand features to obtain low-frequency, high-semantic features, expressed by the following formula:
[0019]
[0020] in, Low-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates low-frequency hand features Perform horizontal stripe convolution. Indicates low-frequency hand features Perform vertical stripe convolution. These are global hand features. The corresponding query vector, key vector, and value vector. Indicates global hand features Perform horizontal stripe convolution. Indicates global hand features Perform vertical stripe convolution. It is a low-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
[0021] In one embodiment, high semantic hand features are obtained based on high-frequency high semantic features and low-frequency high semantic features, expressed by the following formula:
[0022] in, For high semantic hand features, Represents convolution. Indicates splicing, It is a low-frequency, high-semantic feature. These are high-frequency, high-semantic features.
[0023] Secondly, a three-dimensional hand pose estimation device based on joint frequency-spatial domain representation is provided, comprising: The network construction module is used to build a 3D hand pose estimation network. The 3D hand pose estimation network includes a feature extractor, a frequency-spatial domain co-functionalized feature decoupling module, a cross-domain fusion module, and a MANO regressor. The frequency-spatial domain co-functionalized feature decoupling module includes a spatial local feature extraction branch, a spatial global feature extraction branch, and a frequency domain feature extraction branch. The network training module is used to train the 3D hand pose estimation network based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses. During training, samples are input into a feature extractor to extract hand feature maps. These hand feature maps are then input into a frequency-spatial coordinating feature decoupling module. The spatial local feature extraction branch uses a pyramid pooling structure to extract local hand features, while the spatial global feature extraction branch uses a Swin Transformer to extract global hand features. The frequency domain feature extraction branch is based on a two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features. A cross-domain fusion module fuses high-frequency and local hand features to obtain high-frequency semantic features, fuses low-frequency and global hand features to obtain low-frequency semantic features, and then uses these high-frequency and low-frequency semantic features to obtain high-semantic hand features. These high-semantic hand features are then input into a MANO regressor to obtain hand pose estimation results. The estimation module is used to input the RGB image of the hand pose to be processed into the trained 3D hand pose estimation network to obtain the hand pose estimation result.
[0024] Compared with existing technologies, this application has the following advantages: The 3D hand pose estimation method based on joint frequency-spatial representation constructs a frequency-spatial collaborative feature decoupling mechanism, performing frequency-domain feature decoupling and spatial feature decoupling separately, followed by cross-domain feature fusion. Even under occlusion, it can capture details such as finger edges and joint textures while also grasping the overall hand contour. By comprehensively considering the characteristics of both frequency-domain and spatial features, this application achieves more accurate hand pose estimation results. Attached Figure Description
[0025] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a three-dimensional hand pose estimation method based on joint frequency-spatial domain representation is shown. Figure 2 A schematic diagram of a 3D hand pose estimation network is shown. Figure 3 A schematic diagram of the frequency domain feature extraction branch is shown; Figure 4 The hand pose estimation results of the method in this application are shown; Figure 5 The experimental comparison results are shown. Detailed Implementation
[0026] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.
[0027] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0028] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.
[0029] This application provides a three-dimensional hand pose estimation method based on joint frequency-spatial domain representation. Figure 1 A flowchart of a 3D hand pose estimation method based on joint frequency-spatial representation is shown. (See attached diagram) Figure 1 The method mainly includes the following steps: Step S1: Construct a 3D hand pose estimation network.
[0030] Figure 2 A schematic diagram of a 3D hand pose estimation network is shown. (See attached image) Figure 2The 3D hand pose estimation network includes a feature extractor, a frequency-spatial collaborative feature decoupling module, a cross-domain fusion module, and a MANO regressor; the frequency-spatial collaborative feature decoupling module includes a spatial local feature extraction branch, a spatial global feature extraction branch, and a frequency feature extraction branch.
[0031] Step S2: Train the 3D hand pose estimation network based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses.
[0032] When acquiring the training dataset, the hand pose RGB images are preprocessed, including alignment, cropping, and normalization.
[0033] During training, samples are input into a feature extractor for feature extraction to obtain hand feature maps; here, the feature extractor can be a ResNet50 network or a feature pyramid network.
[0034] The hand feature map is input into the frequency domain-spatial domain collaborative feature decoupling module. The spatial domain local feature extraction branch adopts a pyramid pooling structure to extract local hand features, the spatial domain global feature extraction branch adopts a Swing Transformer to extract global hand features, and the frequency domain feature extraction branch is based on two-dimensional Haar wavelet transform to obtain low-frequency hand features and high-frequency hand features. The cross-domain fusion module fuses high-frequency hand features and local hand features to obtain high-frequency high-semantic features, fuses low-frequency hand features and global hand features to obtain low-frequency high-semantic features, and obtains high-semantic hand features based on high-frequency high-semantic features and low-frequency high-semantic features. High-semantic hand features are input into the MANO regressor to obtain hand pose estimation results. Here, MANO (with Articulated and Non-rigid DefOrmations) is a parametric 3D hand model that is widely used in hand pose estimation tasks, aiming to recover the 3D shape and pose of the hand from a single 2D image or video sequence.
[0035] Step S3: Input the RGB image of the hand pose to be processed into the trained 3D hand pose estimation network to obtain the hand pose estimation result.
[0036] In this embodiment, a frequency-spatial collaborative feature decoupling mechanism is constructed. Frequency domain features are decoupled from spatial features separately, and then cross-domain feature fusion is performed. Even under occlusion, this mechanism can capture details such as finger edges and joint textures while also capturing the overall hand contour. By comprehensively considering the characteristics of both frequency and spatial features, this application achieves more accurate hand pose estimation results.
[0037] In one embodiment, the spatial local feature extraction branch employs a pyramid pooling structure, using max pooling and feature fusion to extract salient local hand features at different scales. Specifically, for the input hand feature map... Parallel three-level max pooling is performed, using symmetrical padding to maintain spatial dimensionality and avoid loss of edge information. The multi-scale pooling results are concatenated with the original features, and cross-scale interaction is achieved through convolution. This is expressed by the following formula:
[0038]
[0039] in, This is a feature of pyramid pooling. Represents convolution. Indicates splicing, Represents hand feature map Perform max pooling with a kernel size of 5×5. Represents hand feature map Perform max pooling with a kernel size of 9×9. Represents hand feature map Perform max pooling with a kernel size of 13×13. This indicates batch normalization calculation. This refers to local hand features.
[0040] In one embodiment, the spatial global feature extraction branch employs the Swing Transformer, replacing the traditional Swing Transformer's translation window operation with a horizontal-vertical stripe convolution group. This reduces computational complexity while enhancing long-distance relationships between hand joints, thereby improving the rationality of global pose estimation. Specifically, the hand feature map is convolutionally downsampled, then divided into windows, with each window undergoing multi-head self-attention to obtain window features. For information interaction between windows, a horizontal-vertical stripe convolution group replaces the Swing Transformer's window translation mechanism, enabling long-distance dependencies to be established between any windows. This is expressed by the following formula:
[0041]
[0042] in, Features of the window after multi-head self-attention. This indicates that the window has multiple heads and self-attention. This indicates window partitioning operations. Represents convolution. Represents hand feature diagram, Represents global hand features. This indicates batch normalization calculation. Indicates splicing, Indicates to Perform horizontal stripe convolution. Indicates to Perform vertical stripe convolution.
[0043] In one embodiment, Figure 3 The diagram illustrates the frequency domain feature extraction branch, which is based on two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features, including: A two-dimensional Haar wavelet transform is performed on the hand feature map to obtain one low-frequency component and three high-frequency components. The low-frequency component A represents the main contour of the hand and the spatial distribution of joints, while the high-frequency components H, V, and D encode detailed features such as fingertip texture and knuckle edges in three different directions.
[0044] Specifically, input hand feature map First, perform one-dimensional Haar wavelet decomposition on each row. Then, for the low-frequency component L and high-frequency component H obtained by the row Haar wavelet transform, perform one-dimensional Haar wavelet transform on each column respectively, expressed by the following formula:
[0045] in, This indicates that a Haar wavelet transform is performed on each row, where L represents the low-frequency component and H represents the high-frequency component. This indicates that the Haar wavelet transform is performed on each column, and the four components LL, LH, HL, and HH together constitute the complete result of the two-dimensional Haar wavelet transform. Generally, these four components are represented by A, H, V, and D, respectively.
[0046] Based on one low-frequency component and three high-frequency components, low-frequency hand features and high-frequency hand features are obtained using the following formula:
[0047]
[0048] in, Low-frequency hand features, This indicates batch normalization calculation. Represents convolution. Low-frequency components, High-frequency hand features Indicates splicing, There are three high-frequency components.
[0049] In this embodiment, a feature reconstruction strategy is designed to address the characteristics of hand pose tasks. The three high-frequency components H, V, and D are concatenated, followed by convolution and nonlinear operations. This process reduces the channel dimension to filter out redundant information as much as possible, enhancing the fine-grained response of the fingertip region and obtaining high-frequency hand features. Convolution is then performed on the low-frequency component A to obtain low-frequency hand features, maintaining the continuity of the hand contour. Haar wavelet transform, through adaptive high-low frequency band division, achieves precise alignment between hand semantics and frequency domain components while preserving spatial positioning accuracy.
[0050] In one embodiment, the cross-domain fusion module fuses high-frequency hand features and local hand features to obtain high-frequency, high-semantic features, expressed by the following formula:
[0051]
[0052] in, High-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates high-frequency hand features Perform horizontal stripe convolution. Indicates high-frequency hand features Perform vertical stripe convolution. These are local hand features The corresponding query vector, key vector, and value vector. Indicates local hand features Perform horizontal stripe convolution. Indicates local hand features Perform vertical stripe convolution. It is a high-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
[0053] Similarly, the cross-domain fusion module fuses low-frequency hand features and global hand features to obtain low-frequency, high-semantic features, expressed by the following formula:
[0054]
[0055] in, Low-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates low-frequency hand features Perform horizontal stripe convolution. Indicates low-frequency hand features Perform vertical stripe convolution. These are global hand features. The corresponding query vector, key vector, and value vector. Indicates global hand features Perform horizontal stripe convolution. Indicates global hand features Perform vertical stripe convolution. It is a low-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
[0056] Specifically, high-semantic hand features are obtained based on high-frequency high-semantic features and low-frequency high-semantic features, and are expressed by the following formula:
[0057] in, For high semantic hand features, Represents convolution. Indicates splicing, It is a low-frequency, high-semantic feature. These are high-frequency, high-semantic features.
[0058] Specifically, during training, the network parameters are continuously adjusted using a loss function to improve the accuracy of pose estimation.
[0059] The hand pose estimated by the 3D hand pose estimation network is compared with the labels in the training dataset. A loss function is calculated, and the network parameters are adjusted based on this loss value to achieve the best hand pose estimation prediction results. Loss function as follows:
[0060] in, , Representing the 2D and 3D keypoint losses respectively, using The loss is used to calculate the difference between the predicted 2D and 3D keypoint coordinates and the given true labels in the dataset; To represent the parameter loss of the MANO model, use The loss is used to calculate the MANO parameter loss; , , These are the weighting coefficients that balance each loss function.
[0061] To further verify the effectiveness of the method in this application, the following experimental analysis was conducted.
[0062] Select The RGB image is used to learn 3D hand pose estimation information through a hand pose estimation network, and then the estimated hand pose is obtained through a MANO regressor (parametric 3D hand model). Figure 4 The results of hand pose estimation by the method of this application are shown. The left side is the input RGB image, the middle side is the estimated 3D hand pose front view, and the right side is other views of the estimated 3D hand pose.
[0063] Furthermore, the effectiveness of the ablation experiment module was tested on the HO3Dv2 dataset, a benchmark dataset for 3D hand reconstruction and pose estimation. Figure 5 The experimental comparison results are shown. From... Figure 5 As can be seen, after applying the frequency-spatial domain collaborative feature decoupling mechanism and the cross-domain fusion module in sequence in the baseline, the percentage of correct key points in hand pose estimation is improved.
[0064] Employing the same inventive concept as the three-dimensional hand pose estimation method based on joint frequency-spatial representation, this embodiment also provides a corresponding three-dimensional hand pose estimation device based on joint frequency-spatial representation, comprising: The network construction module is used to build a 3D hand pose estimation network. The 3D hand pose estimation network includes a feature extractor, a frequency-spatial domain co-functionalized feature decoupling module, a cross-domain fusion module, and a MANO regressor. The frequency-spatial domain co-functionalized feature decoupling module includes a spatial local feature extraction branch, a spatial global feature extraction branch, and a frequency domain feature extraction branch. The network training module is used to train the 3D hand pose estimation network based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses. During training, samples are input into a feature extractor to extract hand feature maps. These hand feature maps are then input into a frequency-spatial coordinating feature decoupling module. The spatial local feature extraction branch uses a pyramid pooling structure to extract local hand features, while the spatial global feature extraction branch uses a Swin Transformer to extract global hand features. The frequency domain feature extraction branch is based on a two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features. A cross-domain fusion module fuses high-frequency and local hand features to obtain high-frequency semantic features, fuses low-frequency and global hand features to obtain low-frequency semantic features, and then uses these high-frequency and low-frequency semantic features to obtain high-semantic hand features. These high-semantic hand features are then input into a MANO regressor to obtain hand pose estimation results. The estimation module is used to input the RGB image of the hand pose to be processed into the trained 3D hand pose estimation network to obtain the hand pose estimation result.
[0065] The three-dimensional hand pose estimation device based on frequency domain-spatial domain joint representation in this embodiment has the same inventive concept as the three-dimensional hand pose estimation method based on frequency domain-spatial domain joint representation described above. Therefore, the specific implementation of this device can be found in the embodiment section of the three-dimensional hand pose estimation method based on frequency domain-spatial domain joint representation described above, and its technical effect corresponds to the technical effect of the above method, so it will not be repeated here.
[0066] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A three-dimensional hand pose estimation method based on joint frequency-spatial domain representation, characterized in that, include: A three-dimensional hand pose estimation network is constructed; the three-dimensional hand pose estimation network includes a feature extractor, a frequency domain-spatial domain collaborative feature decoupling module, a cross-domain fusion module, and a MANO regressor; the frequency domain-spatial domain collaborative feature decoupling module includes a spatial domain local feature extraction branch, a spatial domain global feature extraction branch, and a frequency domain feature extraction branch; The 3D hand pose estimation network is trained based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses. During training, samples are input to a feature extractor for feature extraction, resulting in a hand feature map. This hand feature map is then input to the frequency-spatial domain collaborative feature decoupling module. The spatial domain local feature extraction branch uses a pyramid pooling structure to extract local hand features, while the spatial domain global feature extraction branch uses a Swing Transformer to extract global hand features. The frequency domain feature extraction branch is based on a two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features. The cross-domain fusion module fuses the high-frequency and local hand features to obtain high-frequency semantic features, fuses the low-frequency and global hand features to obtain low-frequency semantic features, and obtains high-semantic hand features based on the high-frequency and low-frequency semantic features. These high-semantic hand features are then input to the MANO regressor to obtain the hand pose estimation result. The RGB image of the hand pose to be processed is input into the trained 3D hand pose estimation network to obtain the hand pose estimation result.
2. The method as described in claim 1, characterized in that, The spatial local feature extraction branch adopts a pyramid pooling structure to extract local hand features, expressed by the following formula: in, This is a feature of pyramid pooling. Represents convolution. Indicates splicing, Represents hand feature map Perform max pooling with a kernel size of 5×5. Represents hand feature map Perform max pooling with a kernel size of 9×9. Represents hand feature map Perform max pooling with a kernel size of 13×13. This indicates batch normalization calculation. This refers to local hand features.
3. The method as described in claim 1, characterized in that, The spatial global feature extraction branch uses SwingTransformer to extract global hand features, expressed by the following formula: in, Features of the window after multi-head self-attention. This indicates that the window has multiple heads and self-attention. This indicates window partitioning operations. Represents convolution. Represents hand features. Represents global hand features. This indicates batch normalization calculation. Indicates splicing, Indicates to Perform horizontal stripe convolution. Indicates to Perform vertical stripe convolution.
4. The method as described in claim 1, characterized in that, The frequency domain feature extraction branch is based on two-dimensional Haar wavelet transform to obtain low-frequency hand features and high-frequency hand features, including: Performing a two-dimensional Haar wavelet transform on the hand feature map yields one low-frequency component and three high-frequency components. Based on the one low-frequency component and three high-frequency components, the low-frequency hand features and high-frequency hand features are obtained using the following formula: in, Low-frequency hand features, This indicates batch normalization calculation. Represents convolution. Low-frequency components, High-frequency hand features Indicates splicing, There are three high-frequency components.
5. The method as described in claim 1, characterized in that, The cross-domain fusion module fuses the high-frequency hand features and the local hand features to obtain high-frequency, high-semantic features, which are expressed by the following formula: in, High-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates high-frequency hand features Perform horizontal stripe convolution. Indicates high-frequency hand features Perform vertical stripe convolution. These are local hand features The corresponding query vector, key vector, and value vector. Indicates local hand features Perform horizontal stripe convolution. Indicates local hand features Perform vertical stripe convolution. It is a high-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
6. The method as described in claim 1, characterized in that, The cross-domain fusion module fuses the low-frequency hand features and the global hand features to obtain low-frequency, high-semantic features, expressed by the following formula: in, Low-frequency hand features The corresponding query vector, key vector, and value vector. Represents convolution. Indicates low-frequency hand features Perform horizontal stripe convolution. Indicates low-frequency hand features Perform vertical stripe convolution. These are global hand features. The corresponding query vector, key vector, and value vector. Indicates global hand features Perform horizontal stripe convolution. Indicates global hand features Perform vertical stripe convolution. It is a low-frequency, high-semantic feature. This represents a multilayer perceptron. for The input dimension.
7. The method as described in claim 1, characterized in that, High-semantic hand features are obtained based on the high-frequency high-semantic features and the low-frequency high-semantic features, and are expressed by the following formula: in, For high semantic hand features, Represents convolution. Indicates splicing, It is a low-frequency, high-semantic feature. These are high-frequency, high-semantic features.
8. A three-dimensional hand pose estimation device based on joint frequency-spatial domain representation, characterized in that, include: A network construction module is used to construct a 3D hand pose estimation network. The 3D hand pose estimation network includes a feature extractor, a frequency-spatial domain collaborative feature decoupling module, a cross-domain fusion module, and a MANO regressor. The frequency-spatial domain collaborative feature decoupling module includes a spatial local feature extraction branch, a spatial global feature extraction branch, and a frequency domain feature extraction branch. The network training module is used to train the 3D hand pose estimation network based on the training dataset to obtain the trained 3D hand pose estimation network; the samples in the training dataset are RGB images of hand poses. During training, samples are input to a feature extractor for feature extraction, resulting in a hand feature map. This hand feature map is then input to the frequency-spatial domain collaborative feature decoupling module. The spatial domain local feature extraction branch uses a pyramid pooling structure to extract local hand features, while the spatial domain global feature extraction branch uses a Swing Transformer to extract global hand features. The frequency domain feature extraction branch is based on a two-dimensional Haar wavelet transform to obtain low-frequency and high-frequency hand features. The cross-domain fusion module fuses the high-frequency and local hand features to obtain high-frequency semantic features, fuses the low-frequency and global hand features to obtain low-frequency semantic features, and obtains high-semantic hand features based on the high-frequency and low-frequency semantic features. These high-semantic hand features are then input to the MANO regressor to obtain the hand pose estimation result. The estimation module is used to input the RGB image of the hand pose to be processed into the trained three-dimensional hand pose estimation network to obtain the hand pose estimation result.