A monocular RGB image-based three-dimensional hand pose estimation method

Abstract

The application discloses a three-dimensional hand posture estimation method based on monocular RGB images, designs a feature enhancement method which can explicitly introduce the inherent skeleton structure of the hand, and adaptively enhances the features of the joint nodes to be estimated by using the associated information, so that the accuracy of hand posture estimation is finally improved. The designed method is as follows: firstly, convolutional neural network is used to extract joint node level semantic features and skeleton level semantic features from the input hand image, and a feature fusion module is used to cross semantic aggregation of the two features; then, the feature adaptive enhancement module can adaptively enhance the related features of each joint node by using the associated information thereof; then, the joint node two-dimensional heat map and the relative depth map are obtained through the output layer, and a multi-stage optimization strategy is adopted to continuously refine the two-dimensional heat map and the depth map to estimate more accurate hand two-dimensional joint node coordinates and relative depth; finally, the final hand three-dimensional coordinate information is calculated by using the camera parameters.

Description

Technical Field

[0001] This invention relates to a method for estimating hand pose from monocular RGB images using inherent association information of hand joints, and belongs to the field of image processing. Background Technology

[0002] 3D hand pose estimation methods can be categorized into depth image-based and RGB image-based methods, depending on the data source. Due to the low cost and power consumption of RGB cameras, RGB image-based 3D hand pose estimation solutions are more popular than depth map-based solutions in many vision applications. However, compared to depth images, monocular RGB images have inherent drawbacks such as depth blur and illumination sensitivity, making the estimation of 3D hand pose from monocular RGB images a challenging problem.

[0003] 3D pose estimation from monocular RGB images can generally be categorized into heatmap-based detection methods, 2D pose to 3D pose enhancement methods, and parametric model fitting methods. Heatmap-based detection methods differ from regression methods that directly predict the 3D position of each keypoint. Instead, they predict the 3D Gaussian distribution of each keypoint in the heatmap representation. Post-processing then easily extracts the 3D keypoint coordinates from the heatmap. However, this method inevitably requires maintaining a relatively high-resolution heatmap during inference, naturally resulting in significant computational and memory overhead. Benefiting from numerous robust 2D pose estimation methods, many researchers have focused on 2D pose estimation to 3D enhancement methods. The entire process mainly consists of two parts: first, accurately estimating the 2D hand pose from the monocular RGB image; and then, regressing the final 3D pose coordinates using a low-complexity regression model. Unlike the two methods mentioned above, parametric model regression approximates the shape of the human hand using a parametric model. Most studies use the MANO modeling method, which maps pose and shape parameters to a triangular mesh to represent the shape of the human hand. Since the parametric model contains rich prior structures of the human hand, most works integrate the hand model into a differentiable layer of a neural network, automatically learning to map to MANO parameters for final keypoint estimation.

[0004] While existing 2D hand pose estimation methods boast high accuracy, they suffer from several drawbacks. These include the inherent lack of depth information in monocular RGB images, the mismatch between 2D pose estimation and 3D pose projection, and the fact that previous works treated all joints equally without considering their differences and the contextual relationships between them. These joint relationships can alleviate depth ambiguity and improve prediction accuracy to some extent. Therefore, it is necessary to investigate more robust 3D hand pose estimation methods for monocular RGB images to obtain accurate pose estimates. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of existing technologies and provide a hand pose estimation method based on monocular RGB images to obtain accurate hand joint coordinates. The hand 3D pose estimation task is transformed into two sub-tasks: 2D joint position estimation in a plane and 3D relative depth estimation of joints in space. The designed network consists of a visual feature extraction module, a semantic feature aggregation module, and a joint feature enhancement module, and employs a multi-stage optimization strategy to improve the accuracy of the final prediction.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] The present invention provides a three-dimensional hand pose estimation method based on monocular RGB images, comprising the following steps:

[0008] Step 1: Construct a 3D hand pose estimation network model, which consists of a visual feature extraction module, a semantic feature aggregation module, and a joint feature adaptive enhancement module.

[0009] Step 2: Input a monocular RGB image frame centered on the hand. After passing through the visual feature extraction module, a joint point localization information map and a skeleton association information map are generated in the visual feature extraction module. Then, a pre-trained ResNet18 is used as the feature extraction encoder to obtain the abstract semantic features of the image. Design a binary decoder based on the hourglass structure, using the joint point localization information map and the skeleton association information map as supervision, to obtain joint point-level features containing hand joint point localization information and skeleton-level features containing joint point association information.

[0010] Step 3: The obtained joint-level features and skeleton-level features are sent to the semantic feature aggregation module. The semantic feature fusion module adaptively fuses the captured skeleton-level features and joint-level features across semantic fusion to obtain aggregated semantic features that simultaneously contain both types of features.

[0011] Step 4: The aggregated semantic features are fed into the joint feature adaptive enhancement module that explicitly incorporates the inherent skeletal structure of the hand. For each hand joint to be detected, the relevant features are adaptively enhanced using the association information to obtain the enhanced features of each joint.

[0012] Step 5: For each enhanced feature of a joint, the predicted two-dimensional heatmap and relative depth map of the joint are obtained through the output layer. Then, a multi-stage iterative optimization method is used to continuously refine the two-dimensional heatmap and relative depth map of the joint. Then, the planar coordinates and relative depth values ​​of the joint are obtained through the decoding function. Finally, the three-dimensional coordinates of the hand joint are calculated through the camera parameters. At this point, the hand pose estimation is completed.

[0013] Furthermore, step 2 is specifically implemented as follows: First, a joint location information map and a skeleton association information map conforming to a Gaussian distribution are generated for network supervised training; then, a ResNet18 pre-trained on ImageNet is used as a feature extraction encoder to obtain abstract semantic features of the image; next, a binary decoder based on an hourglass structure is designed, using the joint location information map and the skeleton association map as supervision, and simultaneously outputting skeleton-level features F. b and key-level features F j To provide richer semantic information.

[0014] Furthermore, in step 3, the semantic feature aggregation module adaptively crosses the skeletal features F captured by semantic fusion. b and key-level features F j To obtain fusion features that simultaneously contain both semantics. The process is as follows:

[0015] (31) First, the skeletal features F output by the binary decoding based on the hourglass structure are... b and key-level features F j The concatenation is followed by a 3×3 convolution and a sigmoid activation function to obtain the weights of the corresponding features, as shown in the following formula:

[0016] W b W j =σ(conv) 3×3 (cat(F b ,F j ),θ1))

[0017] Where θ1 represents the parameters that the network needs to learn, F b ,F j W represents the skeletal-level features and keypoint-level features preceding the semantic aggregation module. b W jThis represents the learned weights of skeletal and joint-level features, σ represents the sigmoid activation function, and cat represents the concatenation operation.

[0018] (32) In the hourglass-structured binary decoding, the skeletal-level feature branches and the joint-level feature branches are cross-semantic adaptive feature fusion operations with residual connection structures, respectively. The joint-level weights W obtained in step (31) are used. j And bone-level weights W b The features obtained after cross-semantic fusion are calculated using the following formula:

[0019]

[0020] in This represents the bone-level and joint-level features after residual connection. This represents the dot product operation;

[0021] (33) Finally, the skeletal features after cross-semantic fusion are and key-level features The concatenated data is then fed into a 1×1 convolution to obtain the final aggregated semantic features.

[0022]

[0023] Where θ2 represents the parameters that the network needs to learn. This indicates the final fused features, where cat represents the concatenation operation and c represents the feature dimension.

[0024] Furthermore, the joint feature adaptive enhancement module in step 4 is implemented as follows:

[0025] First, construct the key-point association structure matrix. Where J represents the number of hand joints, and N represents the N joints defined as having association information with this joint; then the semantic features aggregated by the semantic aggregation module are... The features are divided into J groups, ensuring that each unique keypoint is assigned a unique feature. The term "unique" refers to the fact that each of the J joints is assigned a different characteristic. j∈0,...J-1, C represents the total feature dimension of all nodes, and c represents the feature dimension of each node;

[0026] The formula for the joint feature enhancement module is as follows:

[0027]

[0028]

[0029]

[0030] Where f j f represents the original features of the joint to be estimated. i ,i∈0,...N-1 represent the features of N key points that are related to the key point to be estimated, and θ represents the parameters that the network needs to learn. i∈0,...N-1 represents the association information of N key points that are learned and have a relationship with the key point to be estimated. i∈0,...N-1 represents the learned weight coefficients of N joints that are related to the joint to be estimated. This represents the enhanced features of the joints to be estimated.

[0031] Furthermore, in step 5, a multi-stage iterative optimization method is implemented as follows: after obtaining the enhanced joint-level features... Then, for each joint to be detected, its two-dimensional heatmap and relative depth map are predicted individually, using the following formula, where θ represents the parameters to be learned, and H... j D j These represent the predicted 2D heatmap and relative depth map of the key points, respectively:

[0032]

[0033] The predicted two-dimensional heat map H of the key points j Depth map D j semantic features enhanced by this key point The concatenated images are fed into the next stage of the network to learn the optimized 2D heatmap and depth map of the joints. Then, based on the optimized features, a more accurate pose estimation result can be learned, as shown in the following formula, where... Represent the semantic features of the key point to be detected at stage t, respectively: a two-dimensional heatmap and a depth map, φ. t+1 ψ represents the feature aggregation process in stage t+1. t+1 This represents the generation process of the two-dimensional heat map and depth map in stage t+1, which is consistent with the structure of the key feature enhancement module in stage t, but captures longer-range correlation information.

[0034]

[0035]

[0036] For each stage of prediction, the 2D joint heatmap and relative depth heatmap are decoded to obtain the UV coordinates of the joint in the pixel plane and the relative depth coordinates (z) of the joint. rel The formula is as follows:

[0037]

[0038]

[0039]

[0040] in This represents the normalized two-dimensional heatmap of the key points to be estimated, u j ,v j This represents the two-dimensional coordinates of the pixel plane of the joint point to be estimated. This represents the relative depth value of the joint to be estimated;

[0041] Finally, the final coordinates of the hand joints are calculated using camera parameters, and the calculation formula is as follows:

[0042]

[0043] Among them, z root K represents the root absolute depth coordinates, and K represents the camera intrinsic parameter matrix.

[0044] The advantages of this invention compared to the prior art are:

[0045] (1) Unlike previous methods that treat all hand joints equally, this invention considers the differences between different joints and innovatively introduces the inherent skeletal structure of the hand. For each joint to be detected, relevant features are adaptively extracted using its associated information to alleviate hand occlusion and depth blurring problems, thereby improving the accuracy of hand pose estimation. This invention achieves the above objectives by designing a binary decoder, a semantic aggregation module, and a joint feature adaptive enhancement module. The binary decoder and semantic aggregation module can generate rich joint-level and skeletal semantics. When estimating the two-dimensional planar coordinates and relative depth coordinates of a joint, the joint feature adaptive enhancement module adaptively extracts features related to the joint to be predicted from related joints. This means that each joint is regressed in its subspace and fully utilizes the features of related joints. This decoupled prediction method greatly reduces the difficulty of prediction and enhances the accuracy of estimation.

[0046] (2) This invention also employs a multi-stage iterative optimization strategy to capture long-range dependencies between hand joints. Guided by a coarse two-dimensional heatmap and depth map, the output two-dimensional heatmap and depth map are continuously refined through multiple iterations. The proposed iterative optimization strategy can adaptively capture different levels of correlation information for joints with different degrees of freedom, thereby improving the connection between different joints, alleviating the depth module and self-occlusion problems, and ultimately enhancing the accuracy of hand pose estimation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the process for a method of hand 3D pose estimation based on monocular RGB images according to the present invention;

[0049] Figure 2 This is a schematic diagram of the model for hand 3D pose estimation in monocular RGB images according to the present invention;

[0050] Figure 3 This is a visualization of the estimation results of the present invention;

[0051] Figure 4 This invention is compared with the results of some other methods on the publicly available STB dataset. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0053] This invention presents a robust method for hand 3D pose estimation based on monocular RGB images to obtain accurate hand keypoint coordinates. It also designs a feature enhancement method that explicitly incorporates the inherent skeletal structure of hand joints. This method adaptively enhances the features of the joints to be estimated using correlation information and suppresses irrelevant joint features. Simultaneously, a multi-stage optimization strategy is employed to continuously refine the 2D heatmap and depth map to estimate more accurate 2D hand joint coordinates and relative depth, ultimately improving the accuracy of hand pose estimation. This invention effectively utilizes the inherent correlation information of the joints.

[0054] like Figure 1 As shown, the method of the present invention includes the following steps:

[0055] Step 1: The monocular 3D hand pose estimation is considered as two subtasks: 2D joint position information estimation in the plane and 3D relative depth information estimation of joints in space. First, joint localization maps and skeletal association maps conforming to a Gaussian distribution are generated for network-supervised training. Then, a ResNet18 pre-trained on ImageNet is used as the feature extraction encoder to obtain high-level abstract semantic features. Finally, a binary decoder based on an hourglass structure is designed to simultaneously output skeletal-level features F. b and joint-level features F j Skeletal-level features F containing joint connection information b and joint-level features F containing joint location information j They can complement each other, and after subsequent feature fusion modules, fused features rich in both semantics can be obtained.

[0056] Step 2: The semantic feature fusion module can adaptively cross the skeletal features F captured by semantic fusion. b and key-level features F j To obtain fused features that simultaneously contain both semantics. The process is as follows: First, the skeletal features F b and joint features F j The concatenation is followed by a 3×3 convolution and a sigmoid activation function to obtain the weights of the corresponding features, as shown in the following formula:

[0057] W b W j =σ(conv) 3×3 (cat(F b ,F j ),θ1))

[0058] Where θ1 represents the parameters that the network needs to learn, F b ,F j W represents the skeletal features and articular features before aggregation. b W j σ represents the learned skeletal and joint feature weights, σ ​​represents the sigmoid activation function, and cat represents the concatenation operation.

[0059] Then, each branch is a cross-semantic adaptive feature fusion operation with its own residual connection structure, as shown in the following formula:

[0060]

[0061] in This represents the skeletal and articular features after residual connection. This indicates the dot product operation.

[0062] Finally, the concatenated features are fed into a 1×1 convolution to obtain the final aggregated semantic features.

[0063]

[0064] Where θ2 represents the parameters that the network needs to learn. This indicates the final fused features, and cat represents the concatenation operation.

[0065] Step 3: An adaptive enhancement module for joint features was designed that can explicitly incorporate the inherent skeletal structure of the hand. This module enhances relevant features and suppresses irrelevant features by extracting contextual information from the joints to be detected using the correlation information between joints. First, a joint correlation structure matrix is ​​constructed. Where J represents the number of hand joints, usually J=21, and N represents the N joints defined as having associated information with this joint. The aggregated semantic features will then be... The features are divided into J groups to ensure that each unique keypoint can be assigned a unique feature. in j∈0,...J-1, C represents the total feature dimension of all nodes, and c represents the feature dimension of each node. When estimating the two-dimensional planar coordinates and relative depth coordinates of a node, it adaptively extracts features related to the node j to be predicted from N related nodes. This means that each node j regresses within its subspace, fully utilizing the features of related nodes while suppressing irrelevant features. This decoupled prediction method significantly reduces the difficulty of prediction and enhances the accuracy of estimation. The formula for the node feature enhancement module is as follows:

[0066]

[0067]

[0068]

[0069] Where f j f represents the original features of the joint to be estimated. i ,i∈0,...N-1 represent the features of N key points that are related to the key point to be estimated, and θ represents the parameters that the network needs to learn. i∈0,...N-1 represents the association information of N key points that are learned and have a relationship with the key point to be estimated. i∈0,...N-1 represents the learned weight coefficients of N joints that are related to the joint to be estimated. This represents the enhanced features of the joints to be estimated.

[0070] Step 4: Finally, a multi-stage optimization method is used to continuously refine the joint heatmap and relative depth map to obtain more accurate and robust pose estimation results. Specifically, after obtaining the enhanced joint-level features... Then, the two-dimensional heatmap and relative depth map of each joint to be detected can be predicted individually, as shown in the following formula, where θ represents the parameters to be learned, and H... j D j These represent the predicted 2D heatmap and relative depth map of the key points, respectively:

[0071]

[0072] Step 5: Generate the predicted 2D heat map H of the key points. j Depth map D j semantic features enhanced by this key point The concatenated images are fed into the next stage of the network to learn the optimized 2D heatmap and depth map of the joints. It's important to note that subsequent optimizations continuously expand the joint relationships to capture longer-range dependencies. Then, based on the optimized features, a more accurate pose estimation result can be learned. The formula is as follows: Represent the semantic features of the key point to be detected at stage t, respectively: a two-dimensional heatmap and a depth map, φ. t+1 ψ represents the feature aggregation process in stage t+1. t+1 This represents the generation process of the 2D heatmap and depth map in stage t+1, which is consistent with the structure of the key feature enhancement module in stage t, but captures longer-range correlation information.

[0073]

[0074]

[0075] For each stage of prediction, the 2D joint heatmap and relative depth heatmap are decoded to obtain the UV coordinates of the joint in the pixel plane and the relative depth coordinates (z) of the joint. rel The formula is as follows, where This represents the normalized two-dimensional heatmap of the key points to be estimated, u j ,v j This represents the two-dimensional coordinates of the pixel plane of the joint point to be estimated. This represents the relative depth value of the joint to be estimated.

[0076]

[0077]

[0078]

[0079] Finally, the final coordinates of the hand joints can be calculated using camera parameters. The calculation formula is as follows, where z root This represents the root absolute depth coordinates, and K represents the camera intrinsic parameter matrix. At this point, the hand pose estimation process is complete.

[0080]

[0081] like Figure 2 As shown, the overall network framework of this invention is illustrated. First, an RGB image centered on the hand is input. After passing through the feature encoder and binary feature decoder in the left box, skeletal-level features and joint-level features are obtained respectively. Then, the semantic aggregation module integrates the two types of features across semantics. The right box represents the joint feature adaptive enhancement module, which first divides the aggregated semantic features into 21 equal groups. Then, for each joint to be detected, the relevant features can be adaptively enhanced using association information. After passing through the output layer, a two-dimensional heatmap and depth map are obtained. Then, an N-stage iterative optimization strategy is adopted to continuously refine the output two-dimensional heatmap and depth map. Finally, the two-dimensional pixel coordinates and spatial depth coordinates of the 21 joints are decoded respectively, and the final three-dimensional coordinates of the hand joints are obtained by calculating the camera parameters. Figure 2 Encode represents the encoder, Decode represents the decoder, Fusion represents the feature aggregation module, Enhance represents the feature adaptive enhancement module, and refine represents the multi-stage iterative optimization stage.

[0082] like Figure 3 As shown in the figure, the visualization of the estimation results of this invention is as follows: the left side represents the input image of a human hand, and the predicted two-dimensional joint coordinates are marked on it; the middle side represents the predicted joint coordinates; and the right side represents the actual joint coordinates. Figure 3 The accuracy of the detection results of this invention is visually demonstrated. Figure 3 In the diagram, `clr+pred_kp2d` indicates that the predicted 2D joint coordinates are annotated onto the original image, `Pred 3D` represents the predicted joint coordinates, and `3Dannotation` represents the actual joint coordinates. `Keypoint` represents the predicted hand joints, shown as circles in the image: `thumb` for thumb, `index` for index finger, `middle` for middle finger, `ring` for ring finger, and `pinky` for little finger.

[0083] Figure 4 This is a curve comparing the prediction results of this invention with other methods on the STB public dataset. 3D PCK represents the proportion of keypoints that are correctly estimated. It means that the proportion of the normalized distance between the detected keypoint and its corresponding real label is less than a set threshold. The larger the value, the better. Figure 4 The graph shows the prediction results of this invention on the STB public dataset compared with other methods. The results are superior to other existing methods. Error Thresholds represents the error threshold in mm. 3D PCK represents the proportion of the normalized distance between the detected keypoint and its corresponding ground truth label that is less than a set threshold. AUC represents the area under the 3D PCK curve. Ours represents the method of this invention. Ge represents the algorithm proposed by Ge et al. Yang et al. represents the algorithm proposed by Yang et al. Iqbal represents the algorithm proposed by Iqbal et al. Spurr et al., etc.

Claims

1. A three-dimensional hand pose estimation method based on monocular RGB images, characterized in that, Includes the following steps: Step 1: Construct a 3D hand pose estimation network model, which consists of a visual feature extraction module, a semantic feature aggregation module, and a joint feature adaptive enhancement module. Step 2: Input a monocular RGB image frame centered on the hand. After passing through the visual feature extraction module, the module generates a joint localization information map and a skeleton association information map. Then, a pre-trained ResNet18 is used as the feature extraction encoder to obtain the abstract semantic features of the image. Design a binary decoder based on an hourglass structure, using joint location information map and skeleton association information map as supervision, to obtain joint-level features containing hand joint location information and skeleton-level features containing joint association information. Step 3: The obtained joint-level features and skeleton-level features are sent to the semantic feature aggregation module. The semantic feature fusion module adaptively fuses the captured skeleton-level features and joint-level features across semantic fusion to obtain aggregated semantic features that simultaneously contain both types of features. Step 4: The aggregated semantic features are fed into the joint feature adaptive enhancement module that explicitly incorporates the inherent skeletal structure of the hand. For each hand joint to be detected, the relevant features are adaptively enhanced using the association information to obtain the enhanced features of each joint. Step 5: For each joint, the enhanced features are passed through the output layer to obtain the predicted joint 2D heat map and relative depth map. Then, a multi-stage iterative optimization method is used to continuously refine the joint 2D heat map and relative depth map. Then, the planar coordinates and relative depth values ​​of the joint are obtained through the decoding function. Finally, the 3D coordinates of the hand joints are calculated through the camera parameters. At this point, the hand pose estimation is completed. In step 5, a multi-stage iterative optimization method is implemented as follows: after obtaining the enhanced joint-level features... Then, for each joint to be detected, its two-dimensional heat map and relative depth map are predicted individually, using the following formula, where... This represents the parameters to be learned. These represent the predicted 2D heatmap and relative depth map of the key points, respectively. 2D heatmap of predicted key points Depth map semantic features enhanced by this key point The concatenated images are fed into the next stage of the network to learn the optimized 2D heatmap and depth map of the joints. Then, based on the optimized features, a more accurate pose estimation result can be learned, as shown in the following formula, where... These represent the number of joints to be detected at the [number]th node. Semantic features of the stage, 2D heatmap and depth map, Indicates the first Stage feature aggregation process, Indicates the first The generation process of the two-dimensional heatmap and depth map in the first stage is related to the second stage. The phase key feature enhancement module has a consistent structure, but it captures longer-range correlation information; Decoding the 2D joint heatmap and relative depth heatmap predicted at each stage yields the joint's position within the pixel plane. Coordinates, and the relative depth coordinates of the joints. The formula is as follows: in This represents the normalized two-dimensional heatmap of the key points to be estimated. This represents the two-dimensional coordinates of the pixel plane of the joint point to be estimated. This represents the relative depth value of the joint to be estimated; Finally, the final coordinates of the hand joints are calculated using camera parameters. The calculation formula is as follows. in, Represents the root absolute depth coordinates. This represents the camera intrinsic parameter matrix.

2. The method according to claim 1, characterized in that: Step 2 is implemented as follows: First, a joint location information map and a skeleton association information map conforming to a Gaussian distribution are generated for network supervised training; then, a ResNet18 pre-trained on ImageNet is used as a feature extraction encoder to obtain abstract semantic features of the image; finally, a binary decoder based on an hourglass structure is designed, using the joint location information map and the skeleton association map as supervision, and simultaneously outputting skeleton-level features. and key-level features To provide richer semantic information.

3. The method according to claim 1, characterized in that: In step 3, the semantic feature aggregation module adaptively crosses the skeletal features captured by semantic fusion. and key-level features To obtain fusion features that simultaneously contain both semantics. The process is as follows: (31) First, the skeletal features output by the binary decoding based on the hourglass structure are... and key-level features splicing, through Convolution and The activation function yields the weights of the corresponding features, as shown in the following formula: in This represents the parameters that the network needs to learn. This represents the skeletal-level features and keypoint-level features preceding the semantic aggregation module. This represents the learned weights of skeletal-level features and articulation-level features. express Activation function Indicates a splicing operation; (32) In the binary decoding based on the hourglass structure, the branches of the skeleton-level features and the joint-level features are cross-semantic adaptive feature fusion operations with residual connection structures. The joint-level weights obtained in step (31) are used. and skeletal weights The features obtained after cross-semantic fusion are calculated using the following formula: in This represents the bone-level and joint-level features after residual connection. This represents the dot product operation; (33) Finally, the skeletal features after cross-semantic fusion are and key-level features After splicing, it is sent in Convolution yields the final aggregate semantic features. ; in This represents the parameters that the network needs to learn. This indicates that the final fusion feature is obtained. represents the concatenation operation, and c represents the feature dimension.

4. The method according to claim 1, characterized in that: The joint feature adaptive enhancement module in step 4 is implemented as follows: First, construct the key-point association structure matrix. ,in This indicates the number of joints in the hand. This represents the defined information associated with that key. Each key point; then the semantic features aggregated by the semantic aggregation module. All feature dimensions are divided into Grouping ensures that each different keypoint is assigned a unique feature. The "unique" refers to the fact that each of the J joints is assigned a different characteristic. , This represents the total feature dimension of all relevant nodes. This represents the feature dimension of each joint; The formula for the joint feature enhancement module is as follows: in This represents the original features of the joint to be estimated. This indicates a relationship with the key point to be estimated. Features of each key point This represents the parameters that the network needs to learn. This indicates that the learned relationships are related to the key points to be estimated. Association information of each key point This indicates that the learned relationships are related to the key points to be estimated. Each key point weight coefficient This represents the enhanced features of the joints to be estimated.

Images