Hand pose estimation method based on palm structure saliency perception, related device

Abstract

The application provides a hand posture estimation method based on palm structure saliency perception and related equipment. The method comprises: acquiring a to-be-detected image containing a hand target; performing deep learning-based feature extraction processing on the to-be-detected image to obtain a multi-scale feature map; using a preset hand posture detection head network to perform multi-branch parallel feature mapping on the multi-scale feature map to respectively obtain palm center and edge key points, finger joint key points, and quality confidence scores corresponding to the key points, wherein the key points include the palm center and edge key points and the finger joint key points; and determining hand posture estimation information of the to-be-detected image according to the quality confidence scores, the palm center and edge key points, and the finger joint key points. The application can improve the accuracy of the hand posture estimation result.

Description

Technical Field

[0001] This application relates to the field of hand pose estimation technology, specifically to a hand pose estimation method and related equipment based on the saliency perception of palm structure. Background Technology

[0002] With the development of applications such as gesture interaction, virtual reality, robot manipulation, and rehabilitation analysis, computer vision-based hand pose estimation technology is widely used to identify hand positions and key point coordinates from images. Related technologies often employ deep learning frameworks such as YOLO-Pose and MediaPipe to perform object detection and joint key point regression on hand images.

[0003] In related technologies, a fixed paradigm of 21 hand joint keypoints is used, treating the palm region as the connection area between finger joints, and generating hand pose results mainly based on the coordinates of finger joints and fingertips. In scenarios where the palm is tilted, flexed, semi-clenched, or partially occluded, changes in the contour of the palm region will be transmitted to the joint point regression results, causing the output keypoints to drift, misalign, or jump, resulting in low accuracy of hand pose estimation results. Summary of the Invention

[0004] The embodiments of this application provide a hand pose estimation method and related equipment based on the saliency perception of palm structure, which aims to improve the accuracy of hand pose estimation results.

[0005] In a first aspect, embodiments of this application provide a hand pose estimation method based on saliency perception of hand structure, the method comprising:

[0006] Obtain the image to be detected that contains the hand target;

[0007] The image to be detected is subjected to feature extraction processing based on deep learning to obtain a multi-scale feature map;

[0008] Using a pre-defined hand pose detection head network, multi-branch parallel feature mapping is performed on the multi-scale feature map to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including the palm and edge key points and the finger joint key points.

[0009] Based on the quality confidence score, the palm and edge key points, and the finger joint key points, the hand pose estimation information of the image to be detected is determined.

[0010] In the above embodiments, multi-scale feature maps are obtained through deep learning feature extraction, and the hand pose detection head network is used to output palm and edge key points, finger joint key points and quality confidence scores in parallel. This allows palm contour information and finger joint information to participate in hand pose estimation, reducing key point drift, misalignment or jump under palm tilt, flexed palm, semi-fist or partial occlusion, thereby improving the accuracy of hand pose estimation results.

[0011] In one embodiment, the method utilizes a pre-defined hand pose detection head network to perform multi-branch parallel feature mapping on the multi-scale feature map, obtaining palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including:

[0012] Using the detection branch in the hand pose detection head network, single-class target detection is performed on the multi-scale feature map to obtain the hand target bounding box;

[0013] Using the key point regression branch in the hand pose detection head network, coordinate regression processing is performed on the multi-scale feature map based on the hand target bounding box to obtain the palm and edge key points and the finger joint key points;

[0014] Using the key point quality perception branch in the hand pose detection head network, the quality of the multi-scale feature map is evaluated based on the hand target bounding box, and the quality confidence scores corresponding to the multiple key points are obtained.

[0015] In the above embodiment, the bounding box of the hand target is obtained by the detection branch, and then the key point regression branch outputs the palm and edge key points and finger joint key points based on the bounding box of the hand target. The key point quality perception branch outputs the quality confidence score, so that the key point coordinates and quality information are generated around the hand target area, reducing the false detection of key points caused by background interference, thereby improving the stability of hand pose estimation in complex scenes.

[0016] In one embodiment, the hand pose detection head network is pre-trained through the following steps:

[0017] Obtain hand sample images containing real palm and edge key points, and real finger joint key points;

[0018] The hand sample image is subjected to feature extraction processing based on deep learning to obtain a multi-scale feature map of the sample;

[0019] The sample multi-scale feature map is input into the hand pose detection head network to be trained, and the predicted palm and edge key points, the predicted finger joint key points, and the prediction quality confidence scores corresponding to multiple predicted key points are output. The multiple predicted key points include the predicted palm and edge key points and the predicted finger joint key points.

[0020] Determine the first coordinate regression loss between the predicted palm and edge key points and the actual palm and edge key points, and determine the second coordinate regression loss between the predicted finger joint key points and the actual finger joint key points;

[0021] Based on the predicted quality confidence score, the first coordinate regression loss and the second coordinate regression loss are weighted to obtain the quality-perceived fusion loss.

[0022] The hand pose detection head network is obtained by training the model based on the quality-aware fusion loss.

[0023] In the above embodiments, training is supervised by real palm and edge key points and real finger joint key points, and the first coordinate regression loss and the second coordinate regression loss are weighted by the prediction quality confidence score to obtain the quality-aware fusion loss. This makes the training process distinguish the reliability of different key points, reduces the impact of occluded or blurred key points on model updates, and thus reduces key point drift, misalignment or jump.

[0024] In one embodiment, the step of weighting the first coordinate regression loss and the second coordinate regression loss based on the predicted quality confidence score to obtain the quality-perceived fusion loss includes:

[0025] Based on the prediction quality confidence score, the occlusion interference loss weight corresponding to the prediction key point is determined, and the occlusion interference loss weight is positively correlated with the prediction quality confidence score.

[0026] Using the occlusion interference loss weight, the coordinate regression loss of the corresponding predicted key point is calculated point by point to obtain the weighted single-point loss of the predicted key point. The coordinate regression loss of the corresponding predicted key point includes at least one of the first coordinate regression loss and the second coordinate regression loss.

[0027] The quality-perceived fusion loss is obtained by summing the weighted single-point losses of all the predicted key points.

[0028] In the above embodiments, the occlusion interference loss weight is determined based on the prediction quality confidence score, and the coordinate regression loss of the corresponding predicted key points is calculated point by point and then summed to obtain the quality-aware fusion loss. This allows key points with higher quality to receive stronger supervision, reduces the loss contribution of key points with lower quality, and suppresses the erroneous gradients introduced by occlusion and interference samples, thereby improving the accuracy of key point regression in occluded scenarios.

[0029] In one embodiment, training the model based on the quality-aware fusion loss to obtain the hand pose detection head network includes:

[0030] Construct a maximum bounding box based on the position coordinates of the key points of the actual finger joints;

[0031] The spatial area defined by the maximum outer bounding box is taken as the expected range of the palm.

[0032] Predicted palm and edge key points that are not within the expected range of the palm are considered invalid drift points;

[0033] Determine the positive out-of-bounds penalty loss of the invalid drift point relative to the boundary of the maximum bounding box;

[0034] The hand pose detection head network is obtained by training the model based on the positive out-of-bounds penalty loss and the quality-aware fusion loss.

[0035] In the above embodiments, a maximum bounding box is constructed based on the real finger joint key points, and the spatial area defined by the maximum bounding box is used as the expected range of the palm. Positive out-of-bounds penalty loss is determined for the predicted palm center and edge key points that are not within the expected range of the palm, so that the palm center and edge key points are constrained by the finger joint structure, reducing the out-of-bounds drift of the contour key points, thereby improving the rationality of the palm region pose estimation.

[0036] In one embodiment, the step of training the model based on the positive out-of-bounds penalty loss and the quality-aware fusion loss to obtain the hand pose detection head network includes:

[0037] Based on all the predicted palm and edge key points, the corresponding predicted palm geometric center coordinates are determined, and based on all the actual palm and edge key points, the corresponding actual palm geometric center coordinates are determined.

[0038] Based on all the predicted finger joint key points, determine the corresponding predicted finger joint geometric center coordinates, and based on all the actual finger joint key points, determine the corresponding actual finger joint geometric center coordinates.

[0039] Determine the predicted structural offset of the predicted finger joint geometric center coordinates relative to the predicted palm geometric center coordinates, and determine the actual structural offset of the actual finger joint geometric center coordinates relative to the actual palm geometric center coordinates;

[0040] Based on the difference between the predicted structural offset and the actual structural offset, a reverse global offset loss is determined to penalize the overall misalignment of local finger joints.

[0041] The hand pose detection head network is obtained by training the model based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss.

[0042] In the above embodiments, the geometric center coordinates of the palm are determined by predicting the key points of the palm and the edge and the actual key points of the palm and the edge. The predicted structural offset and the actual structural offset are calculated by combining the geometric center coordinates of the finger joints. Then, the reverse global offset loss is determined, so that the overall structure of the palm restricts the overall position of the finger joints in reverse, reducing the overall misalignment caused by local finger occlusion, thereby improving the stability of the posture result.

[0043] In one embodiment, the step of training the model based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss to obtain the hand pose detection head network includes:

[0044] The reverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss are jointly calculated to obtain the multi-task joint pose constraint loss.

[0045] The hand pose detection head network is obtained by training the model using the multi-task joint pose constraint loss and the backpropagation algorithm.

[0046] In the above embodiments, the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss are jointly calculated as a multi-task joint pose constraint loss. The hand pose detection head network is trained through the backpropagation algorithm, so that the key point quality, palm edge range, and finger joint global offset participate in the parameter update at the same time, reducing the accumulation of single-point regression error and thus improving the estimation accuracy in complex gestures and occlusion scenarios.

[0047] Secondly, embodiments of this application provide a hand pose estimation system based on palm structure saliency perception, comprising:

[0048] The image acquisition module is used to acquire the image to be detected, which contains the hand target.

[0049] The feature extraction module is used to perform deep learning-based feature extraction processing on the image to be detected to obtain a multi-scale feature map;

[0050] The parallel mapping module is used to perform multi-branch parallel feature mapping on the multi-scale feature map using a preset hand pose detection head network, to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including the palm and edge key points and the finger joint key points.

[0051] The pose determination module is used to determine the hand pose estimation information of the image to be detected based on the quality confidence score, the palm and edge key points, and the finger joint key points.

[0052] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the hand pose estimation method based on palm structure saliency perception as described above.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hand pose estimation method based on palm structure saliency perception as described above. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating the hand pose estimation method based on palm structure saliency perception provided in this application embodiment;

[0056] Figure 2 This is a schematic diagram of the distribution of 31 key hand points provided in an embodiment of this application;

[0057] Figure 3 This is a comparative diagram of the output effects of the 11-point model, 21-point model, and 31-point collaborative constraint model provided in the embodiments of this application;

[0058] Figure 4 This is a comparative diagram of the output effects of different models under occlusion and deformation scenarios provided in the embodiments of this application;

[0059] Figure 5 This is a schematic diagram of the hand pose estimation system based on palm structure saliency perception provided in an embodiment of this application;

[0060] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0061] It should be noted that, Figure 2 , Figure 3 , Figure 4 The accompanying drawings are in color to facilitate the differentiation of different elements using various colors. Figure 3 and Figure 4 Each image includes multiple different pictures to facilitate a direct comparison of the content within each image. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In addition, in the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0063] Firstly, embodiments of this application provide a hand pose estimation method based on saliency perception of hand structure, applied to a hand pose estimation system based on saliency perception of hand structure. The main implementer of this solution is the hand pose estimation system based on saliency perception of hand structure, which can run on electronic devices.

[0064] Below, refer to Figure 1 This paper provides a detailed description of the hand pose estimation method based on saliency perception of hand structure according to this application. Specifically, the hand pose estimation method based on saliency perception of hand structure may include:

[0065] S101. Obtain the image to be detected that contains the hand target.

[0066] In this embodiment, the image to be detected refers to image data containing at least one human hand. This image may originate from real-time capture by a camera or from an offline stored image file, where the hand occupies a portion of the image and may be subject to varying lighting conditions, motion blur, or partial occlusion.

[0067] In some embodiments of this application, in step S101, the current frame image can be read from the data buffer of the image acquisition device, or an image file can be loaded from a specified storage path to obtain an image to be detected containing the hand target. The input image needs to be preprocessed and adjusted, uniformly scaled to the fixed input size of the model and converted into an RGB three-channel image; the scaling adopts bilinear interpolation, the image pixel values ​​are normalized to the range of 0~1, and the size mismatch area is filled with a solid color background.

[0068] S102. Perform deep learning-based feature extraction processing on the image to be detected to obtain multi-scale feature maps.

[0069] In this embodiment, multi-scale feature maps refer to the set of feature tensors output at different spatial resolution levels after the image to be detected is input into the deep learning backbone network and the feature fusion neck network. Feature maps at different scales focus on capturing fine-grained local texture information and coarse-grained global semantic information in the image, respectively. The two complement each other and can improve the localization accuracy of subsequent keypoint regression.

[0070] In some embodiments of this application, in step S102, the image to be detected can be input into a lightweight convolutional backbone network. After multiple layers of convolution, batch normalization, and activation function operations, feature tensors are output at different downsampling rates. Subsequently, the feature tensors at each level can be input into a feature pyramid fusion network, and multi-scale feature fusion is performed through upsampling and cross-level concatenation operations to finally obtain multi-scale feature maps. For example, three scale feature maps with strides of 8, 16, and 32 can be output to detect hand targets of different sizes.

[0071] In some embodiments of this application, the lightweight convolutional backbone network specifically adopts the Backbone part of the Pose module in YOLOv8n. This Backbone mainly consists of convolutional layers, a C2f module, and an SPPF (Spatial Pyramid Pooling-Fast) module. The C2f module borrows the idea of ​​Cross Stage Partial (CSP) to enhance gradient flow through branch feature propagation and fusion, thereby maintaining good feature representation ability while reducing the number of parameters and computational cost.

[0072] The Feature Pyramid Fusion Network specifically adopts the Neck module built into the YOLOv8n architecture. This module uses a bidirectional feature aggregation structure of FPN (Feature Pyramid Network) + PAN, and borrows the bottom-up feature enhancement design idea from PANet (Path Aggregation Network). It completes cross-scale feature stitching through two paths: top-down sampling fusion and bottom-up sampling fusion. The top-down path supplements the shallow features with deep global semantics, while the bottom-up path supplements the deep features with shallow fine spatial texture. The YOLOv8n Neck maintains the original minimum number of channels configuration and does not add additional attention modules, dense connection layers, or other computational modules, resulting in low computational overhead for the single feature fusion link itself. This application fully retains the convolutional layers, channels, and stitching logic of the YOLOv8n native backbone network and the Neck feature fusion module. Only feature modulation logic is applied to the received Neck output features within the detection head. The core lightweight fusion structure remains unchanged, and only a three-branch reconstruction modification is performed on the downstream hand pose detection head.

[0073] S103. Using a pre-set hand pose detection head network, perform multi-branch parallel feature mapping on multi-scale feature maps to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including palm and edge key points and finger joint key points.

[0074] In this embodiment, the hand pose detection head network refers to a detection head structure that receives multi-scale feature maps as input and outputs different types of prediction results through multiple parallel branches. The network contains at least three branches that operate in parallel, each sharing input features but independently completing its own prediction task.

[0075] Palm and edge keypoints refer to feature points distributed in the palm area and the outer edge of the palm contour. These keypoints cover blind spots in the palm that traditional finger joint keypoints cannot cover, and are used to characterize the overall contour shape of the palm and palm tilt and scaling deformation information. For example, Figure 2As shown, there are 11 palm and edge-related points, numbered 0 to 10, distributed in the web area, the transition area between adjacent finger roots, the outer contour areas of both sides of the palm, the palm base contour area, and the wrist base / palm root area. These points are used to supplement the palm contour and palm base constraint information that is difficult to cover by traditional finger joint key points. Among them, one point located in the wrist base area shares spatial location with the palm root base point among the 21 finger joint key points. This shared point is counted as only one unique point in the total number of key points. Therefore, the 21 finger joint key points and the 10 non-shared palm and edge key points together constitute 31 unique key points, of which 11 palm and edge-related points are shown in the figure.

[0076] Key points of the finger joints refer to the skeletal feature points distributed at the joints of the five fingers of the hand, covering a total of 21 points including the palm base, the fourth joint of the thumb, the fourth joint of the index finger, the fourth joint of the middle finger, the fourth joint of the ring finger, and the fourth joint of the little finger. These key points are used to characterize the fine dynamic movements of the fingers, such as bending, extending, rotating, and opening and closing.

[0077] The quality confidence score is a scalar value output for each keypoint, used to evaluate the reliability of the prediction results for that keypoint. A higher score indicates better visibility of the area where the keypoint is located and a more reliable prediction result; a lower score indicates that the keypoint may be in an occluded, blurred, or noisy area, and the reliability of the predicted coordinates is lower.

[0078] In some embodiments of this application, such as Figure 2 As shown, the hand keypoint system includes finger joint keypoints and palm and edge keypoints. Figure 2 The blue key points and connecting lines in the diagram are used to characterize the skeletal structure of the finger joints, numbered 0 to 10 (in... Figure 2 Key points (displayed in red) Figure 2 The keypoints (displayed as green dots) are used to represent the palm and edge areas, including the web between the fingers, the transition area between the fingers, the edge of the palm, the outline of the palm bottom, the center of the palm, and the base of the wrist. It should be noted that among the keypoints numbered 0 to 10, the point located in the wrist base area and the palm base point among the finger joint keypoints are spatially shared points. This shared point is counted as only one unique point in the total number of keypoints. Therefore, after removing the duplicate count of this shared point from the 21 finger joint keypoints and 11 palm and edge keypoints, a total of 31 unique keypoints are formed.

[0079] In some embodiments of this application, in step S103, the multi-scale feature maps can be fed into three parallel branches of the hand pose detection head network for processing. The detection branch, keypoint regression branch, and keypoint quality perception branch can perform feature extraction based on shared multi-scale features. Specifically, the keypoint regression branch performs convolution operations on the multi-scale feature maps and outputs the two-dimensional coordinates and visibility markers of each keypoint, obtaining palm and edge keypoints and finger joint keypoints. The keypoint quality perception branch performs independent convolution operations on the multi-scale feature maps and outputs quality confidence scores corresponding to each of the multiple keypoints. There is a feature modulation dependency between the keypoint quality perception branch and the keypoint regression branch. The keypoint regression branch conditionally modulates the input features based on the soft visibility weights generated by the quality confidence scores, but this does not affect the joint training based on shared features. The three branches are executed in parallel during the same forward propagation process.

[0080] S104. Based on the quality confidence score, palm and edge key points, and finger joint key points, determine the hand pose estimation information of the image to be detected.

[0081] In this embodiment, hand pose estimation information refers to the final hand pose description data obtained after filtering and fusing keypoint coordinates based on the comprehensive quality confidence score. This data contains all keypoint coordinates after being weighted or filtered by the quality score, and can be directly used in downstream applications such as gesture recognition, hand rehabilitation analysis, or virtual reality interaction.

[0082] In some embodiments of this application, in step S104, the reliability of each keypoint can be screened based on the quality confidence score. Specifically, for keypoints with a quality confidence score higher than a preset quality threshold (e.g., 0.5), their predicted coordinates can be directly used as valid output; for keypoints with a quality confidence score lower than the preset quality threshold, they can be marked as low-confidence points and their contribution weight in subsequent pose fitting can be reduced. After screening, the palm and edge keypoints can be jointly organized with the finger joint keypoints to form complete hand pose estimation information covering the micro-motion features of the fingers and the macro-contour features of the palm. Since the hand pose detection head network is built based on a lightweight backbone network, the overall model parameter count and computational load are low. After training, it can be deployed on mobile terminals or embedded computing platforms for real-time inference, making it suitable for application scenarios requiring real-time hand pose estimation, such as gesture interaction, virtual reality and augmented reality, robot control, and hand rehabilitation analysis.

[0083] To illustrate the impact of different keypoint sets on the stability of hand pose estimation, a comparative example is provided where the same image to be detected is processed using an 11-point model that uses only palm and edge keypoints, a 21-point model that uses only finger joint keypoints, and the 31-point collaborative constraint model of this application. Figure 3 As shown, the 11-point model mainly reflects the palm outline and palm area position, but it is difficult to fully represent the finger joint posture; the 21-point model can represent the finger joint skeleton, but it is prone to local key point drift when the palm outline constraint is insufficient; the 31-point collaborative constraint model uses finger joint key points and palm and edge key points at the same time, and combines shared points to perform overall posture constraint, making the output hand posture estimation information more complete and stable.

[0084] As can be seen, the embodiments of this application obtain multi-scale feature maps through deep learning feature extraction, and use a hand pose detection head network to output palm and edge key points, finger joint key points and quality confidence scores in parallel. This allows palm contour information and finger joint information to participate in hand pose estimation, reducing key point drift, misalignment or jump under palm tilt, flexed palm, half fist or partial occlusion, thereby improving the accuracy of hand pose estimation results.

[0085] In the hand pose estimation process, the multi-branch parallel feature mapping within the hand pose detection head network involves three different levels of processing: object detection, coordinate regression, and quality assessment. Specifically, the hand pose detection head network performs multi-branch parallel feature mapping on multi-scale feature maps to obtain palm and edge keypoints, finger joint keypoints, and quality confidence scores corresponding to multiple keypoints, including:

[0086] S201. Using the detection branch in the hand pose detection head network, single-class target detection is performed on the multi-scale feature map to obtain the bounding box of the hand target.

[0087] In this embodiment, the detection branch refers to the lightweight classification and localization sub-network in the hand pose detection head network responsible for outputting the spatial location of the hand target. Since the hand pose estimation task only needs to detect a single type of target, the number of classification channels in the detection branch is fixed at 1. Compared with multi-class detection scenarios, the number of output channels of the classification convolutional layer can be compressed to a minimum, thereby reducing the number of parameters and floating-point operations in the classification branch. This ensures that the classification branch occupies a very low computational proportion in the overall hand pose detection head network and does not impose an additional burden on the inference speed of the keypoint regression branch and the keypoint quality perception branch.

[0088] The bounding box of the hand target refers to the coordinates of the rectangular region output by the detection branch. It is used to define the spatial range of the hand target in the image plane and to provide the region of interest for subsequent keypoint regression and quality assessment.

[0089] In some embodiments of this application, in step S201, multi-scale feature maps can be input into the convolutional layer in the detection branch. After channel compression and spatial prediction operations, the target confidence and bounding box offset are output at each spatial location of each scale feature map. Subsequently, non-maximum suppression processing can be performed on the candidate bounding boxes output at all scales to filter out redundant boxes with high overlap and retain the bounding box of the hand target with the highest confidence.

[0090] S202. Using the key point regression branch in the hand pose detection head network, coordinate regression processing is performed on the multi-scale feature map based on the hand target bounding box to obtain key points of the palm and edges, and key points of the finger joints.

[0091] In this embodiment, the keypoint regression branch refers to the sub-network in the hand pose detection head network responsible for outputting keypoint coordinates. This branch takes the feature region defined by the bounding box of the hand target as input, independently regresses its two-dimensional coordinates and visibility markers for each keypoint, and outputs a total of 31 points covering palm and edge keypoints and finger joint keypoints. Each point contains values ​​for three channels: horizontal coordinate, vertical coordinate, and visibility status.

[0092] In some embodiments of this application, in step S202, the corresponding feature region can be determined on the multi-scale feature map based on the bounding box of the hand target. This feature region is then input into the multi-layer convolutional structure of the keypoint regression branch for coordinate regression calculation. Before the feature is input into the convolutional structure of the keypoint regression branch, a soft visibility weight can be generated based on the quality confidence score output by the keypoint quality perception branch. This weight is then used to conditionally modulate the input features, enabling the regression branch to dynamically adjust the feature response intensity based on the quality assessment results. The keypoint regression branch outputs 31 sets of three-channel values. These 31 sets of outputs cover the unique points after deduplication of 21 finger joint keypoints and 11 palm and edge keypoints. Shared points are output only once and can be referenced by both keypoint sets. The coordinate values ​​of each keypoint are normalized relative to the bounding box of the hand target. When used, they can be restored to image coordinates based on the position and size of the bounding box of the hand target. At the model configuration level, the output shape of the keypoint regression branch can be set to [31, 3], where 31 represents the total number of palm and edge keypoints and finger joint keypoints, and 3 represents that each keypoint contains three channels: x-coordinate, y-coordinate, and visibility state. The visibility state channel outputs a binary value of 0 or 1, where 0 indicates that the keypoint is occluded but its position can be inferred, and 1 indicates that the keypoint is visible and can be directly observed.

[0093] In some embodiments of this application, before performing coordinate regression calculations, the keypoint regression branch may apply conditional modulation processing based on occlusion patterns to the input features. Specifically, a binary visibility mask can be generated based on the quality confidence score output by the keypoint quality perception branch, wherein the mask value corresponding to keypoints with a quality confidence score greater than or equal to a preset visibility threshold (e.g., 0.5) is 1, and the mask value corresponding to keypoints with a quality confidence score less than the preset visibility threshold is 0.

[0094] Subsequently, the binary visibility mask can be grouped according to the hand skeletal topology, and the visibility rates of the palm region, thumb skeletal chain, index finger skeletal chain, middle finger skeletal chain, ring finger skeletal chain, and little finger skeletal chain can be calculated separately. These six visibility rate values ​​are then combined to form a fragmented topological signature vector. The fragmented topological signature vector is a vector that compresses the visibility and invisibility states of 31 keypoints into a low-dimensional semantic representation based on the hand skeletal topology. Different self-occlusion patterns generated by different gestures correspond to different distributions of fragmented topological signature vector values, allowing the keypoint regression branch to obtain prior information about the current occlusion type before coordinate regression. The expression for calculating the visibility rate of each group is as follows:

[0095]

[0096] in, For the first Visibility of each skeletal group For the first Each skeleton group contains a set of keypoint indexes. For the first Binary visibility mask values ​​for key points This represents the number of keypoints contained in the group.

[0097] After generating the incomplete topological signature vector, this vector can be projected to the same dimension as the number of input feature channels through a single-layer linear mapping, and then passed through a Sigmoid activation function to obtain channel-level scaling coefficients. These channel-level scaling coefficients are then multiplied channel-by-channel with the input features of the keypoint regression branch to obtain the features modulated by occlusion conditions. The calculation expression is as follows:

[0098]

[0099]

[0100] in, This is the channel-level scaling factor. It is the Sigmoid activation function. The linear mapping weight matrix, This is an incomplete topological signature vector. The bias vector is a linear mapping. The original input features are the keypoint regression branch. The features are modulated by occlusion conditions. This is an element-wise multiplication operation. Through this conditional modulation process, when a skeletal chain is completely occluded, the channel-level scaling factor suppresses the channel response related to the local texture of that skeletal chain and enhances the weight of the global feature channel of the palm, guiding the keypoint regression branch to infer the position of the occluded keypoint from the global context information.

[0101] S203. Using the key point quality perception branch in the hand pose detection head network, the quality of multi-scale feature maps is evaluated based on the hand target bounding box, and the quality confidence scores corresponding to multiple key points are obtained.

[0102] In this embodiment, the keypoint quality perception branch refers to the third parallel sub-network in the hand pose detection head network, independent of the detection branch and the keypoint regression branch. This branch outputs a quality confidence score scalar value for each keypoint, which is used to measure whether there is occlusion, blurring, or noise interference in the image region where the keypoint is located, providing a basis for keypoint selection in the subsequent loss weighting and inference stages.

[0103] In some embodiments of this application, in step S203, the corresponding feature region can be determined on the multi-scale feature map based on the bounding box of the hand target. This feature region is then input into the convolutional layer of the keypoint quality-aware branch for quality evaluation. The keypoint quality-aware branch outputs 31 scalar values, corresponding to the quality confidence score for each point in the palm and edge keypoints and finger joint keypoints. The numerical range of the quality confidence score is normalized to between 0 and 1 using a sigmoid activation function. A value closer to 1 indicates a more reliable prediction result for the keypoint, while a value closer to 0 indicates that the keypoint is more likely to be affected by occlusion or blurring.

[0104] As can be seen, the embodiments of this application obtain the bounding box of the hand target through the detection branch, and then the key point regression branch outputs the palm and edge key points and finger joint key points based on the bounding box of the hand target. The key point quality perception branch outputs the quality confidence score, so that the key point coordinates and quality information are generated around the hand target area, reducing the false detection of key points caused by background interference, thereby improving the stability of hand pose estimation in complex scenes.

[0105] In some embodiments of this application, a pre-training step for the hand pose detection head network is provided. Specifically, the hand pose detection head network is pre-trained through the following steps:

[0106] S301. Obtain a hand sample image containing real palm and edge key points, and real finger joint key points.

[0107] In this embodiment, the hand sample image refers to pre-collected and manually annotated hand training data. The acquisition of hand sample images covers various lighting conditions (including strong light, weak light, backlight, and mixed light sources), various occlusion scenarios (including finger self-occlusion, object occlusion of the palm, and overlapping hands occlusion), and various hand postures (including open palm, flexed palm, lateral tilt, semi-clenched fist, fully clenched fist, and extended fingers), to ensure that the training data fully covers complex scenarios that may occur in real-world applications. Each hand sample image contains at least one human hand and includes the coordinates and visibility status of key points marked by the annotator in the palm area, palm edge area, and finger joint positions.

[0108] The true palm and edge key points refer to the coordinates and visibility of 11 feature points marked by the annotator on the hand sample image, located at positions such as the web between the fingers, the side edge of the palm, the palm bottom outline, the center of the palm, and the base of the wrist. Among them, the points located in the base of the wrist share spatial positions with the palm root base point in the true finger joint key points. The true finger joint key points refer to the coordinates and visibility of 21 skeletal points marked by the annotator on the hand sample image, located at the palm root base point and the joints of each of the five fingers.

[0109] It should be noted that, Figure 2 The points numbered 0 to 10 correspond to the palm and edge constraint-related points. One of these points is a spatially shared point with the palm root base point among the finger joint keypoints. This shared point shares the same coordinates and is not counted repeatedly in the total number of keypoints; therefore, the 10 non-shared palm and edge feature points and the 21 finger joint keypoints together form 31 unique keypoints.

[0110] In some embodiments of this application, in step S301, hand sample images and corresponding annotation files can be loaded from a hand pose dataset. The annotation files record the x-coordinate, y-coordinate, visibility state, and occlusion quality label for each of the 31 keypoints in a uniform format. The visibility state is used to identify the visibility of the keypoint in the image, expressed as a binary value: 0 indicates that the keypoint is occluded and cannot be directly observed, but its location can be inferred from the hand structure; 1 indicates that the keypoint is visible in the image and can be directly observed. For keypoints that exceed the image boundary or whose location cannot be determined, an independent validity label is used, and they are not included in the visibility state encoding. The occlusion quality label refers to the continuous numerical value or discrete level additionally marked by the annotator for each valid keypoint, used to quantify the observable quality of the area where the keypoint is located. A higher value indicates that the keypoint area is clearer and more reliable, while a lower value indicates that the keypoint area is more severely affected by occlusion, blurring, or noise. During the training phase, the occlusion quality label serves as the supervised target of the keypoint quality perception branch, guiding the keypoint quality perception branch to learn and output accurate prediction quality confidence scores. Keypoints excluded by validity markers are not included in the annotation of occlusion quality labels or the training of the quality-aware branch. During loading, data augmentation operations such as random flipping, random rotation, random scaling, and color jitter can be applied to hand sample images to enrich the diversity of training samples.

[0111] S302. Perform deep learning-based feature extraction processing on the hand sample image to obtain multi-scale feature maps of the sample.

[0112] In this embodiment, the multi-scale feature map of the samples refers to the set of feature tensors output at different spatial resolution levels after inputting the hand sample image into the backbone network and the feature fusion neck network with the same structure as the inference stage, which are used to provide input for the hand pose detection head network to be trained.

[0113] In some embodiments of this application, in step S302, the hand sample image after data augmentation can be input into a lightweight convolutional backbone network and a feature pyramid fusion network. After a forward propagation process that is completely consistent with the inference stage, a multi-scale feature map of the sample is obtained. The multi-scale feature map of the sample also contains multiple feature tensors with different amplitudes, and the spatial resolution and number of channels of each level of feature tensor are consistent with those in the inference stage.

[0114] S303. Input the multi-scale feature map of the sample into the hand pose detection head network to be trained, and output the predicted palm and edge key points, the predicted finger joint key points, and the prediction quality confidence scores corresponding to multiple predicted key points, including the predicted palm and edge key points and the predicted finger joint key points.

[0115] In this embodiment of the application, the predicted palm and edge key points refer to the predicted coordinates of the 11 palm and edge positions output by the key point regression branch in the hand pose detection head network to be trained.

[0116] Predicting finger joint keypoints refers to the predicted coordinates of the 21 finger joint positions output by the keypoint regression branch.

[0117] The prediction quality confidence score refers to the quality score output by the keypoint quality perception branch for each of the 31 predicted keypoints.

[0118] In some embodiments of this application, in step S303, the multi-scale feature map of the sample can be input into the hand pose detection head network to be trained. After parallel forward computation based on shared features by the detection branch, keypoint regression branch, and keypoint quality perception branch, the predicted palm and edge keypoints, predicted finger joint keypoints, and prediction quality confidence scores corresponding to multiple predicted keypoints are output respectively. The network structure and operation method of each branch are consistent with the aforementioned inference stage, the only difference being that intermediate gradient information needs to be retained during the training stage for subsequent backpropagation.

[0119] In some embodiments of this application, the keypoint regression branch employs a gradient-isolated dual-channel regression structure in the output layer. Specifically, the output layer of the keypoint regression branch contains two parallel convolutional layers: a visible point regression channel and an invisible point inference channel. The visible point regression channel is dedicated to performing accurate coordinate regression of keypoints in the visible region of the image, and its learning objective is to accurately fit the keypoint coordinates from clear visual texture features. The invisible point inference channel is dedicated to inferring the coordinates of occluded keypoints in the image, and its learning objective is to reasonably infer the position of the occluded keypoints from the occlusion pattern prior provided by global context features and incomplete topological signature vectors. The two channels share the occlusion conditionally modulated features as input, but each outputs an independent 31-point coordinate prediction result. The final output is fused according to a binary visibility mask, and its expression is:

[0120]

[0121] in, For the first The final predicted coordinates of each key point. For the first Binary visibility mask values ​​for key points The output of the visible point regression channel is the first Predict the coordinates of key points. The first inference channel output for invisible points Predict the coordinates of each key point.

[0122] During the training phase, the visible point regression channel only receives the gradient from the regression loss of visible keypoint coordinates, while the invisible point inference channel only receives the gradient from the regression loss of invisible keypoint coordinates and the bone segment vector field constraint loss. The gradient is isolated between the two channels through gradient truncation, so that the two learning objectives of accurately fitting the visible point coordinates and reasonably inferring the invisible point coordinates converge independently.

[0123] S304. Determine the first coordinate regression loss between the predicted palm and edge key points and the actual palm and edge key points, and determine the second coordinate regression loss between the predicted finger joint key points and the actual finger joint key points.

[0124] In this embodiment, the first coordinate regression loss refers to the loss value obtained by measuring the normalized coordinate deviation between the predicted palm and edge keypoints and the actual palm and edge keypoints based on OKS (Object Keypoint Similarity) exponential loss, used to measure the prediction accuracy of keypoints in the palm and edge regions. Similarly, the second coordinate regression loss refers to the loss value obtained by measuring the normalized coordinate deviation between the predicted finger joint keypoints and the actual finger joint keypoints based on OKS exponential loss, used to measure the prediction accuracy of keypoints in the finger joint regions.

[0125] In some embodiments of this application, step S304 uses OKS exponential loss to calculate keypoint deviation and filters out invalid points by visual masking; normalized distance loss is calculated point-by-point for 11 palm and edge keypoints and 21 finger joint keypoints respectively, and the losses of all valid points are directly summed without grouping mean processing. Shared points located in the wrist base region participate in point-by-point calculation according to the scale coefficient of their respective groups in the first and second coordinate regression losses, but are only included once in the subsequent accumulation of quality-perceived fusion loss, ensuring that the number of keypoints involved in the accumulation remains consistent with the 31 unique keypoints. Taking the palm and edge grouping loss as an example, its calculation expression is:

[0126]

[0127] in, The first coordinate regression loss, This is a set of indices for key points on the palm and edges. and The first The x and y coordinates of key points on the palm and edges are predicted. and The first The x and y coordinates of key points on the palm and edge.

[0128] For the first The training visual mask for each keypoint is generated directly from the visibility state in the annotations of the hand sample image. This mask is used only during the training phase to select valid keypoints for loss calculation. Specifically, when the visibility state of the keypoint in the annotation is 1 (indicating it is visible and can be directly observed), The value is 1, when the visibility state is 0 (indicating that it is occluded, cannot be directly observed, but its location can be inferred). A value of 0 indicates that only visible keypoints participate in the loss calculation. For keypoints identified by independent validity markers in the annotation file as exceeding the image boundary or whose location cannot be determined, the training visibility mask also has a value of 0 and does not participate in the loss calculation. It should be noted that the training visibility mask and the binary visibility mask generated based on the prediction quality confidence score in step S202 are two different masks: the training visibility mask comes from the manually annotated visibility state and is used for valid point selection in the loss calculation; the binary visibility mask comes from the prediction output of the keypoint quality-aware branch and is used to generate the incomplete topological signature vector and drive occlusion condition modulation.

[0129] The bounding box area of ​​the hand is labeled to normalize the coordinate deviation. For the first The scale coefficient of each keypoint reflects the consistency difference in the annotation of this type of keypoint; the larger the value, the higher the dispersion of the annotation and the greater the allowable deviation. The calculation method for the second coordinate regression loss is similar, except that the keypoint set is replaced with the finger joint keypoint set. The calculation logic is consistent.

[0130] S305. Based on the prediction quality confidence score, the first coordinate regression loss and the second coordinate regression loss are weighted to obtain the quality-perceived fusion loss.

[0131] In this embodiment, the quality-aware fusion loss refers to the overall loss obtained by multiplying the occlusion interference loss weight by the OKS single-point loss for each valid keypoint selected by the training visual mask, and then summing all weighted single-point losses. This loss allows keypoints with higher occlusion interference loss weights (i.e., higher prediction quality confidence scores) to receive a larger gradient contribution during training, while the contribution of keypoints with lower occlusion interference loss weights to the gradient is suppressed, thereby avoiding the negative impact of erroneous coordinates of occluded or blurred regions on model parameter updates.

[0132] In some embodiments of this application, in step S305, the OKS single-point loss can be weighted point-by-point based on the occlusion interference loss weight of each predicted key point and then summed to obtain the quality-aware fusion loss. The specific weighting calculation process will be described in detail in subsequent steps S401 to S403.

[0133] S306. The model is trained based on the quality-aware fusion loss to obtain the hand pose detection head network.

[0134] In some embodiments of this application, in step S306, the quality-aware fusion loss can be used as the objective function. The gradient of the quality-aware fusion loss relative to the parameters of each layer in the hand pose detection head network to be trained is calculated using the gradient descent optimization algorithm, and the network parameters are updated in the opposite direction of the gradient. After multiple rounds of iterative training until the quality-aware fusion loss converges to below a preset threshold or reaches a preset number of training rounds, the trained hand pose detection head network is obtained.

[0135] As can be seen, the embodiments of this application supervise training by using real palm and edge key points and real finger joint key points, and use prediction quality confidence scores to weight the first coordinate regression loss and the second coordinate regression loss to obtain quality-aware fusion loss. This makes the training process distinguish the reliability of different key points, reduces the impact of occlusion or blurry key points on model updates, and thus reduces key point drift, misalignment or jump.

[0136] In some embodiments of this application, the step of weighting the first coordinate regression loss and the second coordinate regression loss based on the predicted quality confidence score is refined. Specifically, the first coordinate regression loss and the second coordinate regression loss are weighted based on the predicted quality confidence score to obtain the quality-perceived fusion loss, including:

[0137] S401. Based on the prediction quality confidence score, determine the occlusion interference loss weight corresponding to the corresponding prediction key point. The occlusion interference loss weight is positively correlated with the corresponding prediction quality confidence score.

[0138] In this embodiment, the occlusion interference loss weight refers to the scalar weight value assigned to each predicted keypoint based on the prediction quality confidence score, used to control the contribution of that keypoint in the loss calculation. There is a positive correlation between the occlusion interference loss weight and the prediction quality confidence score; that is, keypoints with higher prediction quality confidence scores receive a larger occlusion interference loss weight, while keypoints with lower prediction quality confidence scores receive a smaller occlusion interference loss weight, thus compressing the contribution of keypoints located in occluded or blurred regions to the overall loss.

[0139] In some embodiments of this application, in step S401, the prediction quality confidence score of each predicted keypoint can be directly used as the occlusion interference loss weight for that keypoint. For example, when the prediction quality confidence score of a certain predicted keypoint is 0.9, its occlusion interference loss weight is 0.9; when the prediction quality confidence score of another predicted keypoint is 0.2, its occlusion interference loss weight is 0.2. In other implementations, the occlusion interference loss weight can also be obtained by applying a nonlinear mapping to the prediction quality confidence score, for example, by taking the square of the prediction quality confidence score as the occlusion interference loss weight, to further widen the weight gap between high-confidence points and low-confidence points. The calculation expression for the occlusion interference loss weight is:

[0140]

[0141] in, For the first The occlusion interference loss weights for each prediction key point For the first The prediction quality confidence score for each key prediction point To control the degree of weight differentiation, when When the weight of the occlusion interference loss is equal to 1, it is equal to the prediction quality confidence score itself. When the value is greater than 1, the weights of low confidence points are further compressed.

[0142] S402. Using the occlusion interference loss weight, the coordinate regression loss of the corresponding predicted key point is calculated point by point to obtain the weighted single-point loss of the predicted key point. The coordinate regression loss of the corresponding predicted key point includes at least one of the first coordinate regression loss and the second coordinate regression loss.

[0143] In this embodiment, the weighted single-point loss refers to the weighted loss value obtained by multiplying the coordinate regression loss of a valid keypoint by the occlusion interference loss weight of that keypoint. Only keypoints marked as valid by the training visual mask are included in the calculation. For predicted keypoints belonging to palm and edge keypoints, their single-point coordinate regression loss is the OKS single-point loss term corresponding to that point in the first coordinate regression loss summation formula; for predicted keypoints belonging to finger joint keypoints, their coordinate regression loss comes from the OKS single-point loss corresponding to that point in the second coordinate regression loss.

[0144] In some embodiments of this application, in step S402, the occlusion interference loss weight of each valid keypoint can be multiplied by the OKS single-point coordinate regression loss of that point to obtain the weighted single-point loss of that keypoint. Specifically, for the i-th valid keypoint, the calculation expression for its weighted single-point loss is:

[0145]

[0146] in, For the first Weighted single-point loss at each key point; For the first The occlusion interference loss weight of each key point; The first The x and y coordinates of each predicted key point; The first The x and y coordinates of each real key point; For the first The training visual mask for each key point, with occluded points having a value of 0 and not participating in the calculation; The bounding box area of ​​the hand is labeled for scale normalization of coordinate deviations; For the first The scale coefficients of each keypoint. The meanings of the symbols above are consistent with those in step S304. Through this point-by-point weighting mechanism, the loss contribution of occluded keypoints with low quality confidence is automatically compressed, while high-confidence clear keypoints retain complete gradient supervision.

[0147] S403. Sum the weighted single-point losses of all predicted key points to obtain the quality-perceived fusion loss.

[0148] In this embodiment of the application, summing the weighted single-point loss of all predicted keypoints means accumulating the weighted single-point loss values ​​of all valid keypoints after training the visual mask into a scalar value. This scalar value is the quality-aware fusion loss, which is used as the main supervision signal for the keypoint regression task during the training phase.

[0149] In some embodiments of this application, in step S403, the weighted single-point losses of all valid keypoints can be summed to obtain the quality-aware fusion loss. Its calculation expression is:

[0150]

[0151] in, For quality perception fusion loss, For the first Weighted single-point loss for each key prediction point To train a set of valid keypoint indices with a visual mask value of 1, For the first The training visual mask for each keypoint. The summation process does not group and calculate the mean for keypoints on the palm and edges or keypoints on the finger joints; instead, the weighted single-point loss of all valid keypoints is directly accumulated.

[0152] As can be seen, the embodiments of this application determine the occlusion interference loss weight based on the prediction quality confidence score, and calculate the coordinate regression loss of the corresponding predicted key points point by point, and then sum them to obtain the quality-aware fusion loss. This enables key points with higher quality to receive stronger supervision, reduces the loss contribution of key points with lower quality, suppresses the erroneous gradients introduced by occlusion and interference samples, and thus improves the accuracy of key point regression in occluded scenarios.

[0153] During model training, relying solely on quality-aware fusion loss for coordinate regression supervision is insufficient to effectively suppress abnormal drift at palm and edge keypoints. Therefore, some embodiments of this application introduce positive boundary violation penalty logic based on finger joint keypoints to construct spatial boundary constraints. Specifically, model training based on quality-aware fusion loss yields a hand pose detection head network, including:

[0154] S501. Construct the maximum bounding box based on the position coordinates of key points of real finger joints.

[0155] In this embodiment, the maximum bounding box refers to an axis-aligned rectangular region formed by taking the minimum and maximum values ​​of the horizontal coordinates of all real finger joint keypoints as the left and right boundaries, and the minimum and maximum values ​​of the vertical coordinates as the top and bottom boundaries. This rectangular region covers the spatial distribution range of all finger joint keypoints and is used to describe the maximum spread boundary of the finger joints on the image plane.

[0156] In some embodiments of this application, in step S501, the position coordinates of all 21 real finger joint key points in the current hand sample image can be traversed, and the minimum and maximum values ​​in the horizontal and vertical coordinate dimensions are calculated respectively. The minimum horizontal coordinate value is used as the left boundary of the maximum bounding box, the maximum horizontal coordinate value as the right boundary, the minimum vertical coordinate value as the upper boundary, and the maximum vertical coordinate value as the lower boundary, thereby constructing the maximum bounding box. Its expression is:

[0157]

[0158] in, The coordinates of the four boundaries of the largest bounding box. and The first The x and y coordinates of key points of a real finger joint Iterate through all 21 real finger joint key points.

[0159] S502, The spatial area defined by the largest outer bounding box is taken as the expected range of the palm.

[0160] In this embodiment, the expected range of the palm refers to a rectangular spatial region constrained by the four boundaries of the largest bounding box. This region represents the reasonable spatial range within which key points of various parts of the palm should fall under the current hand posture. Based on prior knowledge of the physiological structure of the hand, key points of the palm and its edges should not exceed the maximum spread boundary formed by the finger joints under a normal hand posture.

[0161] In some embodiments of this application, in step S502, the maximum bounding box constructed in step S501 can be directly used as the expected range of the palm. Considering the annotation error and the possibility that the edge points of the palm may be close to the outer edge of the finger joints, a preset boundary tolerance margin can be extended outward on each of the four boundaries of the maximum bounding box to obtain an appropriately enlarged expected range of the palm, so as to avoid misjudging the normal palm and edge key points that are close to the boundary.

[0162] S503. Predicted palm and edge key points that are not within the expected range of the palm are considered invalid drift points.

[0163] In this embodiment, invalid drift points refer to predicted palm and edge key points whose predicted coordinates fall outside the expected range of the palm. The predicted coordinates of these key points deviate from the reasonable space of the palm's physiological structure and are abnormal outputs generated during the network regression process, requiring additional loss penalties to pull them back to the reasonable area.

[0164] In some embodiments of this application, in step S503, it can be determined one by one whether the predicted coordinates of each predicted palm and edge key point fall within the expected range of the palm. Specifically, for the first... If a predicted palm and edge keypoint has a predicted x-coordinate that is less than the left boundary or greater than the right boundary of the expected palm area, or a predicted y-coordinate that is less than the upper boundary or greater than the lower boundary, then the predicted palm and edge keypoint is considered an invalid drift point. If all four boundary conditions are met, then the predicted palm and edge keypoint is within the normal range of prediction and is not treated as an invalid drift point.

[0165] S504. Determine the positive out-of-bounds penalty loss for invalid drift points relative to the boundary of the largest bounding box.

[0166] In this embodiment, the positive out-of-bounds penalty loss refers to the loss value obtained by accumulating the out-of-bounds distances of all invalid drift points beyond the boundary of the maximum bounding box. This loss value only penalizes the coordinate deviation in the out-of-bounds direction; predictive palm and edge keypoints within the expected palm range do not incur a positive out-of-bounds penalty loss.

[0167] In some embodiments of this application, in step S504, the out-of-bounds distance of each invalid drift point relative to the boundary of the maximum bounding box can be calculated. Specifically, for the horizontal axis direction, if the predicted horizontal axis of the invalid drift point is less than the left boundary, the out-of-bounds distance is the left boundary minus the predicted horizontal axis; if the predicted horizontal axis is greater than the right boundary, the out-of-bounds distance is the predicted horizontal axis minus the right boundary. The calculation method for the out-of-bounds distance in the vertical axis direction is similar. For each invalid drift point, the sum of the squares of its horizontal axis out-of-bounds distance and its vertical axis out-of-bounds distance can be used as the out-of-bounds penalty value for that point. Finally, the out-of-bounds penalty values ​​of all invalid drift points are summed and averaged to obtain the positive out-of-bounds penalty loss. Its calculation expression is:

[0168]

[0169] in, To punish losses for positive boundary crossings, The set of indices for all invalid drift points. The number of invalid drift points. For the first The out-of-bounds distance of each invalid drift point in the horizontal direction. For the first The out-of-bounds distance of each invalid drift point in the vertical direction. When there are no invalid drift points, the positive out-of-bounds penalty is zero.

[0170] S505. The hand pose detection head network is obtained by training the model based on the positive out-of-bounds penalty loss and the quality-aware fusion loss.

[0171] In some embodiments of this application, in step S505, the positive out-of-bounds penalty loss and the quality-aware fusion loss can be weighted and summed to obtain a joint loss that includes coordinate regression supervision and spatial boundary constraints. This joint loss is then used to update the parameters of each layer in the hand pose detection head network to be trained via backpropagation. The positive out-of-bounds penalty loss prompts the keypoint regression branch to learn to constrain the predicted coordinates of palm and edge keypoints within a reasonable spatial range formed by the finger joints, avoiding abnormal prediction outputs that exceed the physiological structure of the hand. After multiple rounds of iterative training until the joint loss converges, the hand pose detection head network is obtained.

[0172] As can be seen, the embodiments of this application construct the maximum bounding box based on the real finger joint key points, and take the spatial area defined by the maximum bounding box as the expected range of the palm. Positive out-of-bounds penalty loss is determined for the predicted palm center and edge key points that are not within the expected range of the palm, so that the palm center and edge key points are constrained by the finger joint structure, reducing the out-of-bounds drift of the contour key points, thereby improving the rationality of the palm region pose estimation.

[0173] Based on the positive out-of-bounds penalty loss constraining the palm and edge keypoints, when occlusion in the finger joint region causes the overall displacement of the finger joint keypoints, the global drift problem of the finger joint keypoints cannot be corrected by positive constraints alone. Therefore, some embodiments of this application also introduce a reverse global offset loss. Specifically, the model is trained based on the positive out-of-bounds penalty loss and the quality-aware fusion loss to obtain the hand pose detection head network, including:

[0174] S601. Based on all predicted palm and edge key points, determine the corresponding predicted palm geometric center coordinates, and based on all real palm and edge key points, determine the corresponding real palm geometric center coordinates.

[0175] In this embodiment, the predicted geometric center coordinates of the palm refer to the two-dimensional coordinate value obtained by taking the arithmetic mean of the coordinates of all predicted palm and edge keypoints, representing the spatial centroid position of the palm region predicted by the network. The true geometric center coordinates of the palm refer to the two-dimensional coordinate value obtained by taking the arithmetic mean of the coordinates of all true palm and edge keypoints, representing the true spatial centroid position of the palm region in the labeled data. Both are used together to construct a global spatial reference benchmark for the palm region.

[0176] In some embodiments of this application, in step S601, the abscissa component of the predicted palm geometric center coordinates is obtained by summing the abscissas of all predicted palm and edge key points and dividing by the total number of palm and edge key points; the ordinate component of the predicted palm geometric center coordinates is obtained by summing the ordinates of all predicted palm and edge key points and dividing by the total number of palm and edge key points. The calculation method for the actual palm geometric center coordinates is the same; the coordinates of the actual palm and edge key points are substituted into the same mean calculation process. The calculation expression is:

[0177]

[0178]

[0179] in, To predict the coordinates of the geometric center of the hand, The coordinates of the actual geometric center of the palm. The number of key points on the palm and edges. , For the first The x and y coordinates of key points on the palm and edges are predicted. , For the first The x and y coordinates of key points on the palm and edge.

[0180] S602. Based on all predicted finger joint key points, determine the corresponding predicted finger joint geometric center coordinates, and based on all real finger joint key points, determine the corresponding real finger joint geometric center coordinates.

[0181] In this embodiment, the predicted geometric center coordinates of the finger joint refer to the two-dimensional coordinate value obtained by taking the arithmetic mean of the coordinates of all predicted finger joint keypoints, representing the spatial centroid position of the finger joint region predicted by the network. The actual geometric center coordinates of the finger joint refer to the two-dimensional coordinate value obtained by taking the arithmetic mean of the coordinates of all actual finger joint keypoints, representing the actual spatial centroid position of the finger joint region in the labeled data.

[0182] In some embodiments of this application, in step S602, the abscissa component of the predicted finger joint geometric center coordinates can be obtained by summing the abscissas of all predicted finger joint key points and dividing by the total number of finger joint key points; the ordinate component of the predicted finger joint geometric center coordinates can be obtained by summing the ordinates of all predicted finger joint key points and dividing by the total number of finger joint key points. The calculation method for the actual finger joint geometric center coordinates is the same; it can be obtained by substituting the coordinates of the actual finger joint key points. The calculation expression is:

[0183]

[0184]

[0185] in, To predict the geometric center coordinates of the finger joints, The coordinates of the actual geometric center of the finger joint. The number of key points in the finger joints. , For the first Predict the x and y coordinates of key points of the finger joints. , For the first The x and y coordinates of key points of a real finger joint.

[0186] S603. Determine the predicted structural offset of the predicted finger joint geometric center coordinates relative to the predicted palm geometric center coordinates, and determine the true structural offset of the true finger joint geometric center coordinates relative to the true palm geometric center coordinates.

[0187] In this embodiment, the predicted structural offset refers to the two-dimensional coordinate difference vector between the predicted geometric center coordinates of the finger joints and the predicted geometric center coordinates of the palm, used to describe the spatial displacement relationship between the centroid of the finger joint region and the centroid of the palm region in the network prediction result. The true structural offset refers to the two-dimensional coordinate difference vector between the true geometric center coordinates of the finger joints and the true geometric center coordinates of the palm, used to describe the spatial displacement relationship between the centroid of the finger joint region and the centroid of the palm region in the true annotation. The two sets of offsets reflect the relative spatial structure between the finger joints and the palm, and this structural relationship should remain stable under normal hand posture.

[0188] In some embodiments of this application, in step S603, the abscissa of the predicted geometric center coordinates of the hand can be subtracted from the abscissa of the predicted geometric center coordinates of the finger joints to obtain the abscissa component of the predicted structural offset; the ordinate of the predicted geometric center coordinates of the hand can be subtracted from the ordinate of the predicted geometric center coordinates of the finger joints to obtain the ordinate component of the predicted structural offset. The calculation method of the actual structural offset is consistent with this. Its expression is:

[0189]

[0190]

[0191] in, To predict structural offset, This represents the actual structural offset. To predict the geometric center coordinates of the finger joints, To predict the coordinates of the geometric center of the hand, The coordinates of the actual geometric center of the finger joint. The coordinates are the geometric center coordinates of the actual palm.

[0192] S604. Based on the difference between the predicted structural offset and the actual structural offset, determine the inverse global offset loss used to penalize the overall misalignment of the local finger joints.

[0193] In this embodiment, the inverse global offset loss refers to the loss value obtained by measuring the difference between the predicted structural offset and the actual structural offset. When the finger joint keypoints are shifted as a whole due to occlusion or interference, the spatial relationship between the predicted finger joint geometric center coordinates and the predicted palm geometric center coordinates will deviate from the true value. The inverse global offset loss penalizes this overall misalignment, prompting the network to use the stable structural information of the palm region to correct the global drift of the finger joints.

[0194] In some embodiments of this application, in step S604, the squared Euclidean distance between the predicted structural offset and the actual structural offset can be calculated as the inverse global offset loss. Its calculation expression is:

[0195]

[0196] in, For reverse global offset loss, and These represent the abscissa and ordinate components of the predicted structural offset, respectively. and These represent the abscissa and ordinate components of the actual structural offset, respectively. The larger this loss value, the more severe the deviation of the relative spatial structure between the finger joints and the palm in the predicted result from the actual annotation.

[0197] S605. The hand pose detection head network is obtained by training the model based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss.

[0198] In some embodiments of this application, in step S605, the reverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss can be weighted and summed to obtain a joint loss that includes coordinate regression supervision, spatial boundary constraints, and global structure correction. The reverse global offset loss enables the network to learn and maintain the correct relative spatial relationship between the geometric centers of the finger joints and the geometric center of the palm. When a local finger is occluded, causing the overall displacement of the finger joint keypoints, the palm center and edge keypoints in the palm region have strong global stability due to their wide distribution and abundant redundant information, and can provide spatial anchor references for the finger joint region. Using this joint loss, the parameters of each layer in the hand pose detection head network to be trained are updated through the backpropagation algorithm. After multiple rounds of iterative training until the joint loss converges, the hand pose detection head network is obtained.

[0199] In some embodiments of this application, in addition to maintaining global spatial consistency through forward out-of-bounds penalty loss and reverse global offset loss, a bone segment vector field constraint loss within the local skeletal chain can be applied to the occluded key points to further accurately determine the direction and distance of the occluded key points on their skeletal chain, provided that the global coordinate system is already stable.

[0200] Specifically, for each occluded keypoint (i.e., a keypoint with a binary visibility mask value of 0), the system backtracks layer by layer along its bone chain in the hand's skeletal topology towards the palm to find the first visible ancestor keypoint (i.e., a keypoint with a binary visibility mask value of 1). If there is no visible ancestor keypoint on a certain bone chain, the bone segment vector field constraint of that occluded keypoint is skipped, and the positive out-of-bounds penalty loss and the reverse global offset loss are responsible for the fallback constraint.

[0201] For each occluded keypoint where a visible ancestor keypoint is found, calculate the predicted bone segment vector and the true bone segment vector pointing from the visible ancestor keypoint to that occluded keypoint:

[0202]

[0203]

[0204] in, For the first Predicted bone segment vectors of occluded key points For the first The true bone segment vector of the occluded key point and The first Predicted and actual coordinates of an occluded key point and These are the predicted and actual coordinates of the visible ancestor keypoint corresponding to the occluded keypoint, respectively.

[0205] Directional and length constraints are imposed on the predicted and actual bone segment vectors. The directional constraint uses cosine similarity to measure directional deviation.

[0206]

[0207] in, For the first The orientation constraint loss of the occluded keypoint is used to ensure that the orientation of the occluded keypoint is reasonable relative to the visible ancestor keypoint, so as to avoid prediction results that do not conform to the physiological structure of the hand, such as the finger breaking in the opposite direction. For the first Predicted bone segment vectors of occluded key points For the first The true bone segment vector of the occluded key point For L2 norm operations, in the numerator This is a vector dot product operation.

[0208] The length constraint uses the distance between the center point of the palm and the base point of the palm as the scale normalization benchmark to eliminate the scale variation caused by different distances of the hand in the image. The length constraint uses the SmoothL1 loss function to measure the deviation of the normalized bone segment length.

[0209]

[0210]

[0211] in, The scale-normalized reference distance, The coordinates of the center point of the palm. The coordinates of the base point are: For the first The length constraint loss of each occluded key point. For L2 norm operations, To smooth the L1 loss function, For the first Predicted bone segment vectors of occluded key points For the first The true bone segment vector of the occluded key point.

[0212] The orientation constraint loss and length constraint loss of all occluded keypoints with visible ancestor keypoints are summed to obtain the bone segment vector field constraint loss:

[0213]

[0214] in, For bone segment vector field constraint loss, This is the set of indexes of occluded keypoints that have visible ancestor keypoints. For the first The directional constraint loss of the occluded key points For the first The length constraint loss for each occluded keypoint. The bone segment vector field constraint loss can be multiplied by the corresponding loss weight coefficient and added to the multi-task joint pose constraint loss. The forward out-of-bounds penalty loss and the backward global offset loss ensure that keypoints do not go out of bounds or drift globally, while the bone segment vector field constraint loss provides fine-grained orientation and distance localization for occluded keypoints at the local skeletal chain level. The two constitute a hierarchical and progressive constraint system that is "globally stable and locally accurate". The bone segment vector field constraint loss only participates in loss calculation and gradient backpropagation during the training phase and does not perform additional calculations during the inference phase. After training, the network has internalized the skeletal chain orientation and length relationship into the network parameters.

[0215] As can be seen, the embodiments of this application determine the geometric center coordinates of the palm by predicting the key points of the palm and the edge and the actual key points of the palm and the edge, and calculate the predicted structural offset and the actual structural offset by combining the geometric center coordinates of the finger joints, thereby determining the reverse global offset loss, so that the overall structure of the palm restricts the overall position of the finger joints in reverse, reducing the overall misalignment caused by local finger occlusion, thereby improving the stability of the posture result.

[0216] During training, the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss supervise the hand pose detection head network from different dimensions. These three losses need to be integrated into a unified optimization objective to drive network parameter updates. Specifically, in some embodiments of this application, the model is trained based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss to obtain the hand pose detection head network, including:

[0217] S701. The reverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss are jointly calculated to obtain the multi-task joint pose constraint loss.

[0218] In this embodiment, the multi-task joint pose constraint loss refers to the scalar loss value obtained by weighting and summing the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss according to their respective loss weight coefficients. This loss value simultaneously includes the supervision signal of the keypoint coordinate regression accuracy, the constraint signal of the rationality of the spatial boundary of the palm and edge keypoints, and the correction signal of the relative structural consistency between the finger joints and the palm, serving as the overall optimization objective during the training phase of the hand pose detection head network.

[0219] In some embodiments of this application, in step S701, corresponding loss weight coefficients can be set for the reverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss, respectively. Then, the three are multiplied by their respective loss weight coefficients and summed to obtain the multi-task joint pose constraint loss. Its calculation expression is:

[0220]

[0221] in, For multi-task joint attitude constraint loss, For quality perception fusion loss, To punish losses for positive boundary crossings, For reverse global offset loss, , , These are the loss weight coefficients for the quality-aware fusion loss, the positive out-of-bounds penalty loss, and the backward global offset loss, respectively. These three loss weight coefficients are used to balance the contribution ratio of the three losses in gradient updates and can be adjusted based on the keypoint accuracy metrics on the validation set. For example, they can be... Set the primary loss benchmark to 1.0. Setting it to 0.5 is used to apply moderate spatial boundary constraints. It is set to 0.3 to provide an auxiliary global structural correction signal.

[0222] In some embodiments of this application, the classification loss and bounding box regression loss output by the detection branch can be included in the joint calculation during the calculation of the multi-task joint pose constraint loss. Specifically, the classification loss and bounding box regression loss of the detection branch can be weighted and summed with the above three pose-related losses to obtain a complete training loss covering target detection and keypoint pose estimation. Since hand detection is a single-class task, the numerical magnitude of the classification loss is small, and its impact on the overall training process is limited. However, its existence can ensure that the detection accuracy of the hand target bounding box is continuously optimized, providing a stable region of interest basis for keypoint regression.

[0223] S702. Using multi-task joint pose constraint loss, the model is trained through backpropagation algorithm to obtain the hand pose detection head network.

[0224] In this embodiment, the backpropagation algorithm refers to a computational process based on the chain rule, which propagates and accumulates the multi-task joint pose constraint loss layer by layer relative to the gradient of each learnable parameter in the hand pose detection head network. After obtaining the gradient of each layer's parameters through the backpropagation algorithm, the gradient descent optimizer is used to update the parameter values ​​in the opposite direction of the gradient, so that the multi-task joint pose constraint loss gradually decreases in subsequent iterations.

[0225] In some embodiments of this application, in step S702, the following operations may be performed in each training batch: the hand sample images of the current batch are forward propagated to obtain the prediction results, the multi-task joint pose constraint loss is calculated according to the aforementioned steps, and then the backpropagation algorithm is called to calculate the gradient of the multi-task joint pose constraint loss for each layer parameter of the detection branch, key point regression branch and key point quality perception branch in the hand pose detection head network, and then the gradient descent optimizer is used to update the parameters of each layer according to the gradient value.

[0226] After each training round, evaluation metrics across multiple dimensions can be calculated on the validation set to monitor the convergence trend of model accuracy. Specifically, the metrics for coordinate regression accuracy include: Mean Per Joint Position Error (MPJPE), which measures the average Euclidean distance between the predicted and true coordinates of all keypoints; Percentage of Correct Keypoints (PCK), which calculates the proportion of keypoints whose normalized distance between predicted and true coordinates is less than a preset normalization threshold, using the diagonal length of the hand bounding box as a normalization benchmark; and Normalized Mean Error (NME), which normalizes the pixel error of each keypoint by dividing it by the physiological length of the hand (distance from the wrist to the fingertip of the middle finger) to eliminate error measurement bias caused by differences in the distance and size of the hand within the image.

[0227] In one experimental example, a comparative test was conducted on different keypoint modeling schemes using the publicly available Rendered Hand Pose Dataset (RHD). The YOLOv8n model using only 21 finger joint keypoints achieved an MPJPE of 11.72 and a PCK@0.05 accuracy of 82.3%; the model using only 11 palm and edge-related points for independent inference achieved an MPJPE of 22.13 and a PCK@0.05 accuracy of 53.8%; while the proposed 31-point collaborative constraint model combined with a quality-aware branch reduced the MPJPE to 10.45 and improved the PCK@0.05 accuracy to 85.4%. This demonstrates that the proposed method, through the synergistic effect of finger joint keypoints, palm and edge keypoints, bidirectional geometric constraints, and a quality-aware branch, can reduce the average pixel error of keypoints and improve the keypoint prediction accuracy within a threshold range.

[0228] The metrics for the accuracy dimension of quality scoring include: the binary cross-entropy loss of quality scoring, used to calculate the difference between the predicted quality confidence score output by the keypoint quality perception branch and the actual visibility state. Its calculation expression is:

[0229]

[0230] in, For quality score binary cross-entropy loss, This represents the total number of key points. For the first The prediction quality confidence score for each key point For the first The true visibility state of each keypoint (value 0 or 1). This metric effectively measures the network's binary classification prediction ability for keypoint visibility. The visibility prediction accuracy dimension includes: visibility prediction accuracy, which calculates the proportion of keypoints whose prediction quality confidence score, after threshold binarization, matches the true visibility state to the total number of valid keypoints. Its calculation expression is:

[0231]

[0232] in, For visibility prediction accuracy, To predict the number of keypoints whose quality confidence scores, after being binarized with a threshold (e.g., 0.5), match the actual visibility status, This represents the total number of valid keypoints included in the statistics. This metric is used to evaluate the network's binary classification accuracy for keypoint visibility. All the above evaluation metrics are written to the training log after each training round to monitor the convergence status of coordinate regression accuracy, quality score accuracy, and visibility discrimination capability round by round. When the multi-task joint pose constraint loss no longer decreases on the validation set for several consecutive rounds, or reaches the preset maximum number of training rounds, training is stopped and the current network parameters are saved, resulting in the hand pose detection head network.

[0233] In some embodiments of this application, a quality scoring supervision loss can be introduced into the keypoint quality perception branch during model training. Specifically, the visibility state in the hand sample image annotation can be used as the supervision target for quality scoring. The binary cross-entropy loss between the predicted quality confidence score output by the keypoint quality perception branch and the actual visibility state is calculated, and this loss is multiplied by the corresponding loss weight coefficient and added to the multi-task joint pose constraint loss. The actual visibility state is binary encoded, with 1 indicating that the keypoint is visible and 0 indicating that the keypoint is occluded. Through this supervision signal, the keypoint quality perception branch can learn to distinguish between visible and occluded keypoints, outputting an accurate quality confidence score for each keypoint during the inference phase.

[0234] In this embodiment, the hand pose detection head network trained using multi-task joint pose constraint loss possesses a triple anti-occlusion collaborative capability during the inference phase. The first layer is structural complementarity anti-occlusion: when the finger area is occluded, the palm and edge keypoints, distributed in the center and outer contour of the palm, still retain global palm structural information; when the palm area is occluded, the finger joint keypoints, distributed across the finger joints, still retain local fine information. The second layer is quality-aware anti-occlusion: the keypoint quality-aware branch outputs a lower quality confidence score for keypoints in occluded areas, which reduces the contribution weight of these keypoints in pose fitting during inference, preventing erroneous coordinates from interfering with the final output. The third layer is bidirectional geometric constraint anti-occlusion: the forward out-of-bounds penalty loss uses the spatial boundary formed by the finger joints to constrain the palm and edge keypoints, while the reverse global offset loss uses the stability of the palm's geometric center to correct the overall drift of the finger joints. The collaboration of these two layers ensures that the output keypoint coordinates conform to the prior laws of hand physiological structure.

[0235] In scenarios involving occlusion or hand deformation, the output differences of models corresponding to different keypoint sets are more pronounced. For example... Figure 4As shown, when the hand target has local occlusion or deformation, the 11-point model only relies on the palm and edge key points, which is difficult to stably restore the finger joint posture; the 21-point model only relies on the finger joint key points, which is prone to local skeleton point displacement due to occlusion or deformation; the 31-point collaborative constraint model of this application uses the palm and edge key points to provide global contour reference and uses the finger joint key points to provide local skeleton information, and combines the quality-aware branch to suppress low-confidence key points, thereby reducing key point drift and maintaining posture structure stability.

[0236] As can be seen, the embodiments of this application jointly calculate the reverse global offset loss, the forward out-of-bounds penalty loss and the quality-aware fusion loss into a multi-task joint pose constraint loss, and train the hand pose detection head network through the backpropagation algorithm, so that the key point quality, the palm edge range and the finger joint global offset participate in the parameter update at the same time, reducing the accumulation of single-point regression error, thereby improving the estimation accuracy in complex gestures and occlusion scenarios.

[0237] Secondly, such as Figure 5 As shown, embodiments of this application provide a hand pose estimation system based on saliency perception of hand structure, including:

[0238] The image acquisition module is used to acquire the image to be detected, which contains the hand target.

[0239] The feature extraction module is used to perform deep learning-based feature extraction processing on the image to be detected, and obtain multi-scale feature maps.

[0240] The parallel mapping module is used to perform multi-branch parallel feature mapping on multi-scale feature maps using a preset hand pose detection head network, to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including palm and edge key points and finger joint key points.

[0241] The pose determination module is used to determine the hand pose estimation information of the image to be detected based on the quality confidence score, palm and edge key points, and finger joint key points.

[0242] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the hand pose estimation method based on palm structure saliency perception as described in any of the above embodiments.

[0243] Specifically, the electronic device can be a terminal device or a server equipped with a camera. The terminal device includes, but is not limited to, smartphones, tablets, virtual reality headsets, augmented reality glasses, robot control terminals, or rehabilitation training equipment, and the camera is used to acquire images of the target object, including the hand.

[0244] In some embodiments of this application, the electronic device further includes an image acquisition device for acquiring a video stream containing a hand target in real time, and inputting each frame of the video stream as an image to be detected into the processor to perform hand pose estimation processing.

[0245] In some embodiments of this application, the processor of the electronic device can be a general-purpose central processing unit, a graphics processing unit, or a neural network inference acceleration chip. When the processor is a mobile embedded processor, the lightweight structure of the hand pose detection head network allows the single-frame inference time to meet the requirements of real-time interaction.

[0246] In some embodiments of this application, the electronic device further includes a result output device for displaying hand posture estimation information on a display screen in the form of a visual skeleton overlay, or for transmitting hand posture estimation information to a downstream application module for gesture recognition, virtual reality interactive control, or rehabilitation exercise analysis.

[0247] Fourthly, embodiments of this application provide an electronic device for running a hand pose estimation system based on saliency perception of hand structure as described in any of the above embodiments. Figure 6 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0248] The electronic device includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 602 or a program loaded from storage portion 608 into Random Access Memory (RAM) 603, such as executing the hand pose estimation method based on palm structure saliency perception in the above embodiments. Various programs and data required for system operation are also stored in RAM 603. The CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. An Input / Output (I / O) interface 605 is also connected to bus 604.

[0249] The following components are connected to I / O interface 605: input section 606 including audio input devices, push-button switches, etc.; output section 607 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 608 including a hard disk, etc.; and communication section 609 including a network interface card such as a local area network (LAN) card, modem, etc. Communication section 609 performs communication processing via a network such as the Internet. Drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0250] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by CPU 601, it performs the various functions defined in this application.

[0251] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0252] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0253] Specifically, the electronic device of this embodiment includes a processor and a memory. The memory is coupled to one or more processors and is used to store computer program code. The computer program code includes computer instructions. One or more processors call the computer instructions to cause the electronic device to perform the method provided in the above embodiment.

[0254] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hand pose estimation method based on palm structure saliency perception as described in any of the above embodiments.

[0255] Sixthly, embodiments of this application provide a computer program product, including a computer program executed by a processor to implement the hand pose estimation method based on palm structure saliency perception as described in any of the above embodiments.

[0256] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A hand pose estimation method based on saliency perception of hand structure, characterized in that, The method includes: Obtain the image to be detected that contains the hand target; The image to be detected is subjected to feature extraction processing based on deep learning to obtain a multi-scale feature map; Using a pre-defined hand pose detection head network, multi-branch parallel feature mapping is performed on the multi-scale feature map to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including the palm and edge key points and the finger joint key points. Based on the quality confidence score, the palm and edge key points, and the finger joint key points, the hand pose estimation information of the image to be detected is determined.

2. The method as described in claim 1, characterized in that, The method utilizes a pre-defined hand pose detection head network to perform multi-branch parallel feature mapping on the multi-scale feature map, obtaining key points of the palm and edges, key points of finger joints, and quality confidence scores corresponding to multiple key points, including: Using the detection branch in the hand pose detection head network, single-class target detection is performed on the multi-scale feature map to obtain the hand target bounding box; Using the key point regression branch in the hand pose detection head network, coordinate regression processing is performed on the multi-scale feature map based on the hand target bounding box to obtain the palm and edge key points and the finger joint key points; Using the key point quality perception branch in the hand pose detection head network, the quality of the multi-scale feature map is evaluated based on the hand target bounding box, and the quality confidence scores corresponding to the multiple key points are obtained.

3. The method as described in claim 2, characterized in that, The hand pose detection head network is pre-trained through the following steps: Obtain hand sample images containing real palm and edge key points, and real finger joint key points; The hand sample image is subjected to feature extraction processing based on deep learning to obtain a multi-scale feature map of the sample; The sample multi-scale feature map is input into the hand pose detection head network to be trained, and the predicted palm and edge key points, the predicted finger joint key points, and the prediction quality confidence scores corresponding to multiple predicted key points are output. The multiple predicted key points include the predicted palm and edge key points and the predicted finger joint key points. Determine the first coordinate regression loss between the predicted palm and edge key points and the actual palm and edge key points, and determine the second coordinate regression loss between the predicted finger joint key points and the actual finger joint key points; Based on the predicted quality confidence score, the first coordinate regression loss and the second coordinate regression loss are weighted to obtain the quality-perceived fusion loss. The hand pose detection head network is obtained by training the model based on the quality-aware fusion loss.

4. The method as described in claim 3, characterized in that, The step of weighting the first coordinate regression loss and the second coordinate regression loss based on the predicted quality confidence score to obtain the quality-perceived fusion loss includes: Based on the prediction quality confidence score, the occlusion interference loss weight corresponding to the prediction key point is determined, and the occlusion interference loss weight is positively correlated with the prediction quality confidence score. Using the occlusion interference loss weight, the coordinate regression loss of the corresponding predicted key point is calculated point by point to obtain the weighted single-point loss of the predicted key point. The coordinate regression loss of the corresponding predicted key point includes at least one of the first coordinate regression loss and the second coordinate regression loss. The quality-perceived fusion loss is obtained by summing the weighted single-point losses of all the predicted key points.

5. The method as described in claim 3, characterized in that, The process of training the model based on the quality-aware fusion loss to obtain the hand pose detection head network includes: Construct a maximum bounding box based on the position coordinates of the key points of the actual finger joints; The spatial area defined by the maximum outer bounding box is taken as the expected range of the palm. Predicted palm and edge key points that are not within the expected range of the palm are considered invalid drift points; Determine the positive out-of-bounds penalty loss of the invalid drift point relative to the boundary of the maximum bounding box; The hand pose detection head network is obtained by training the model based on the positive out-of-bounds penalty loss and the quality-aware fusion loss.

6. The method as described in claim 5, characterized in that, The process of training the model based on the positive out-of-bounds penalty loss and the quality-aware fusion loss to obtain the hand pose detection head network includes: Based on all the predicted palm and edge key points, the corresponding predicted palm geometric center coordinates are determined, and based on all the actual palm and edge key points, the corresponding actual palm geometric center coordinates are determined. Based on all the predicted finger joint key points, determine the corresponding predicted finger joint geometric center coordinates, and based on all the actual finger joint key points, determine the corresponding actual finger joint geometric center coordinates. Determine the predicted structural offset of the predicted finger joint geometric center coordinates relative to the predicted palm geometric center coordinates, and determine the actual structural offset of the actual finger joint geometric center coordinates relative to the actual palm geometric center coordinates; Based on the difference between the predicted structural offset and the actual structural offset, a reverse global offset loss is determined to penalize the overall misalignment of local finger joints. The hand pose detection head network is obtained by training the model based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss.

7. The method as described in claim 6, characterized in that, The hand pose detection head network is obtained by training the model based on the inverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss, including: The reverse global offset loss, the forward out-of-bounds penalty loss, and the quality-aware fusion loss are jointly calculated to obtain the multi-task joint pose constraint loss. The hand pose detection head network is obtained by training the model using the multi-task joint pose constraint loss and the backpropagation algorithm.

8. A hand pose estimation system based on saliency perception of hand structure, characterized in that, The system includes: The image acquisition module is used to acquire the image to be detected, which contains the hand target. The feature extraction module is used to perform deep learning-based feature extraction processing on the image to be detected to obtain a multi-scale feature map; The parallel mapping module is used to perform multi-branch parallel feature mapping on the multi-scale feature map using a preset hand pose detection head network, to obtain palm and edge key points, finger joint key points, and quality confidence scores corresponding to multiple key points, including the palm and edge key points and the finger joint key points. The pose determination module is used to determine the hand pose estimation information of the image to be detected based on the quality confidence score, the palm and edge key points, and the finger joint key points.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the hand pose estimation method based on palm structure saliency perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the hand pose estimation method based on palm structure saliency perception as described in any one of claims 1 to 7.

Images