Hand detection method and apparatus, non-transitory storage medium, and electronic device

Abstract

The application discloses a hand detection method and device, a nonvolatile storage medium and an electronic device. The method comprises: processing an image by using a computer vision model to obtain a plurality of prediction boxes and a left-hand confidence score and a right-hand confidence score corresponding to each prediction box; marking a left-hand prediction box and a right-hand prediction box in the plurality of prediction boxes; copying a target prediction box to obtain a first copy and a second copy, marking the first copy as a left-hand prediction box, and marking the second copy as a right-hand prediction box; performing feature modulation processing on a left-hand prediction box set to obtain key point coordinates corresponding to a hand region in the left-hand prediction box; and performing feature modulation processing on a right-hand prediction box set to obtain key point coordinates corresponding to a hand region in the right-hand prediction box. The application solves the technical problems of detection confusion and key point identity confusion caused by the difficulty in accurately distinguishing left and right hands in hand detection and hand key point positioning.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a hand detection method, apparatus, non-volatile storage medium, and electronic device. Background Technology

[0002] In the field of computer vision, especially for the design of human-computer interaction interfaces for augmented reality (AR), virtual reality (VR), and smart devices, vision-based gesture recognition technology occupies a central position. The crucial first step is accurately detecting the hand in an image and locating its key points, which is relatively easy to achieve in single-handed scenarios. However, when faced with complex scenarios, especially situations involving close interaction between both hands, traditional technologies face significant challenges.

[0003] Currently, mainstream hand detection technologies primarily employ either a top-down or bottom-up strategy. The top-down approach first uses an object detector to identify the bounding boxes of the hands, then independently estimates keypoints within each box. The bottom-up approach, on the other hand, estimates all keypoints first and then groups them for classification. While the top-down method generally performs better in detection accuracy, its detector is based on a softmax classifier, and this mutually exclusive property makes it ineffective when handling overlapping hands. When hands are very close, crossed, or overlapping, the detector struggles to accurately determine the boundaries of the hands in the same overlapping area, thus easily leading to missed or false detections, especially in highly overlapping scenarios such as prayer gestures.

[0004] Even if two hand regions can be detected, traditional keypoint estimation networks may fail to correctly distinguish between the left and right hands due to the similarity of the input images, outputting two sets of mirrored keypoint coordinates, leading to an identity-switch problem. This confusion not only affects the accuracy of gesture recognition but may also mislead subsequent interaction logic, especially in scenarios requiring precise identification of left and right hand operations.

[0005] To address the challenge of detecting hands and locating key points in a single-frame image, some solutions rely on the temporal information of the video sequence, using tracking to improve the detection results for a single frame. However, this temporal-dependent approach becomes ineffective in single-image applications, while also increasing system complexity and latency, limiting its application on edge computing devices with high real-time requirements or power sensitivity.

[0006] In summary, existing vision-based hand detection and keypoint localization technologies have significant limitations and shortcomings when handling complex scenes, especially those involving close hand-to-hand interaction. These shortcomings affect the practical application of the technologies, particularly in scenarios such as AR, VR, and smart device interaction, directly impacting the user experience. Summary of the Invention

[0007] This application provides a hand detection method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problems of detection confusion and key point identity confusion caused by the difficulty in accurately distinguishing between the left and right hands due to hand detection and key point localization.

[0008] According to one aspect of this application, a hand detection method is provided, comprising: acquiring an image to be processed; processing the image using a computer vision model to obtain multiple prediction boxes for hand regions in the image, and a left-hand confidence score and a right-hand confidence score corresponding to each prediction box, wherein the left-hand confidence score is used to quantify the probability that the hand region is the left hand, and the right-hand confidence score is used to quantify the probability that the hand region is the right hand; among the multiple prediction boxes, marking prediction boxes with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold as left-hand prediction boxes belonging to the left hand, and marking prediction boxes with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold as right-hand prediction boxes belonging to the right hand. The target predicted bounding box is copied to obtain a first copy and a second copy. The first copy is labeled as a left-handed predicted bounding box, and the second copy is labeled as a right-handed predicted bounding box. The target predicted bounding box is a predicted bounding box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold. The left-handed predicted bounding box set, which includes multiple left-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-handed predicted bounding box. The right-handed predicted bounding box set, which includes multiple right-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-handed predicted bounding box. The labeling information in the left-handed and right-handed predicted bounding box sets is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0009] Optionally, the computer vision model includes multiple parallel classification heads, including a left-hand binary classifier and a right-hand binary classifier, wherein the left-hand binary classifier is used to determine whether the hand region is the left hand and the right-hand binary classifier is used to determine whether the hand region is the right hand.

[0010] Optionally, a pose estimation model is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction box. This includes: cropping the image centered on the left-hand prediction box and scaling the cropped image to a preset size to obtain a first image; processing the first image using a feature extraction backbone network to obtain a first image feature map with spatial dimensions; converting the marker information in the left-hand prediction box into a first identity feature vector using an identity encoder, and adjusting the first identity feature vector to a first identity feature tensor consistent with the spatial dimensions of the first image feature map through a shape change operation; generating a first channel weighting parameter and a first spatial attention parameter based on the first identity feature tensor; weighting the first image feature map by channel dimensions using the first channel weighting parameter, and modulating the first image feature map by spatial attention dimensions using the first spatial attention parameter to obtain a first target feature map; performing an upsampling operation on the first target feature map using a keypoint decoder to restore the spatial resolution of the first target feature map to obtain a first keypoint heatmap; and determining the keypoint coordinates corresponding to the hand region within the left-hand prediction box based on the first keypoint heatmap.

[0011] Optionally, a pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box. This includes: cropping the image centered on the right-hand prediction box and scaling the cropped image to a preset size to obtain a second image; processing the second image using a feature extraction backbone network to obtain a second image feature map with spatial dimensions; converting the marker information in the right-hand prediction box into a second identity feature vector using an identity encoder, and adjusting the second identity feature vector to a second identity feature tensor with the same spatial dimensions as the second image feature map through a shape transformation operation; generating second channel weighting parameters and second spatial attention parameters based on the second identity feature tensor; applying channel dimension weighting to the second image feature map using the second channel weighting parameters, and applying spatial attention dimension modulation to the second image feature map using the second spatial attention parameters to obtain a second target feature map; performing an upsampling operation on the second target feature map using a keypoint decoder to restore the spatial resolution of the second target feature map to obtain a second keypoint heatmap; and determining the keypoint coordinates corresponding to the hand region within the right-hand prediction box based on the second keypoint heatmap.

[0012] Optionally, a pose estimation model is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction box. This includes: obtaining the first geometric parameters and first labeling information of the left-hand prediction box; encoding the first geometric parameters into a first geometric feature vector, wherein the first geometric feature vector includes the position and scale information of the left-hand prediction box; encoding the first labeling information into a first semantic feature vector, wherein the first semantic feature vector includes the category attribute and confidence information of the left-hand prediction box; fusing the first geometric feature vector and the first semantic feature vector to obtain a first context representation; and using a parameter generation network to process the first context representation. The process involves processing to obtain first feature modulation parameters, which include first channel weighting coefficients and a first spatial transformation matrix. Based on the first geometric parameters, a corresponding first region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The first channel weighting coefficients are applied to the channel dimension of the first region feature map to recalibrate the importance of different feature channels, and the first spatial transformation matrix is ​​applied to the spatial dimension of the first region feature map to perform attention weighting on different spatial positions within the left-hand prediction box. The modulated first region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region within the left-hand prediction box.

[0013] Optionally, a pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction boxes. This includes: obtaining the second geometric parameters and second labeling information of the right-hand prediction boxes; encoding the second geometric parameters into a second geometric feature vector, wherein the second geometric feature vector includes the position and scale information of the right-hand prediction boxes; encoding the second labeling information into a second semantic feature vector, wherein the second semantic feature vector includes the category attribute and confidence information of the right-hand prediction boxes; fusing the second geometric feature vector and the second semantic feature vector to obtain a second context representation; and using a parameter generation network to process the second context representation. The process involves processing to obtain the second feature modulation parameters, which include: second channel weighting coefficients and a second spatial transformation matrix. Based on the second geometric parameters, the corresponding second region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The second channel weighting coefficients are applied to the channel dimension of the second region feature map to recalibrate the importance of different feature channels, and the second spatial transformation matrix is ​​applied to the spatial dimension of the second region feature map to perform attention weighting on different spatial positions within the right-hand prediction box. The modulated second region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box.

[0014] Optionally, before performing feature modulation processing on the set of left-handed prediction boxes including multiple left-handed prediction boxes using the pose estimation model, the method further includes: performing a first non-maximum suppression operation on the left-handed prediction boxes marked as belonging to the left hand to obtain a set of left-handed prediction boxes; and performing a second non-maximum suppression operation on the right-handed prediction boxes marked as belonging to the right hand to obtain a set of right-handed prediction boxes, wherein the first non-maximum suppression operation and the second non-maximum suppression operation are independent operations.

[0015] Optionally, the pose estimation model is trained using the following method: The classification loss is determined by summing the left-hand and right-hand classification losses, where both are calculated separately using the binary cross-entropy loss function; the bounding box regression loss is determined by the regression difference between the predicted bounding box coordinate parameters and the ground truth hand bounding box coordinate parameters; the target loss is determined by the difference between the confidence score of the predicted bounding box and the target label calculated based on the intersection-union ratio (IUU) of the predicted and ground truth hand bounding boxes; the loss function is determined based on the classification loss, bounding box regression loss, and target loss; and the initial pose estimation model is trained on the training set, obtaining the pose estimation model when the loss function meets the preset convergence condition.

[0016] According to another aspect of this application, a hand detection device is also provided, comprising: an acquisition module for acquiring an image to be processed; a processing module for processing the image using a computer vision model to obtain multiple prediction boxes for a hand region in the image and a left-hand confidence score and a right-hand confidence score corresponding to each prediction box, wherein the left-hand confidence score is used to quantify the probability that the hand region is the left hand, and the right-hand confidence score is used to quantify the probability that the hand region is the right hand; and a first labeling module for labeling prediction boxes with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold as left-hand prediction boxes belonging to the left hand, and predict boxes with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold as right-hand prediction boxes belonging to the right hand. The second labeling module is used to copy the target predicted box to obtain a first copy and a second copy. The first copy is labeled as a left-handed predicted box, and the second copy is labeled as a right-handed predicted box. The target predicted box is a predicted box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold. The first parsing module is used to perform feature modulation processing on the set of left-handed predicted boxes, which includes multiple left-handed predicted boxes, using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-handed predicted box. The second parsing module is used to perform feature modulation processing on the set of right-handed predicted boxes, which includes multiple right-handed predicted boxes, using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-handed predicted box. The labeling information in the set of left-handed and right-handed predicted boxes is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0017] According to another aspect of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above hand detection method.

[0018] According to another aspect of this application, an electronic device is also provided, comprising: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described hand detection method during runtime.

[0019] According to another aspect of this application, a computer program is also provided, wherein the computer program, when executed by a processor, implements the above-described hand detection method.

[0020] According to another aspect of this application, a computer program product is also provided, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program that, when executed by a processor, implements the above-described hand detection method.

[0021] In this application, the following steps are taken: First, an image to be processed is acquired. Then, a computer vision model is used to process the image, resulting in multiple predicted bounding boxes for the hand region in the image, along with left-hand and right-hand confidence scores for each predicted bounding box. The left-hand confidence score quantifies the probability that the hand region is the left hand, and the right-hand confidence score quantifies the probability that the hand region is the right hand. Among the multiple predicted bounding boxes, those with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold are marked as left-hand predicted boxes, and those with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold are marked as right-hand predicted boxes. Finally, the target predicted bounding box is copied to obtain a first copy and a second copy. The first copy is marked as a left-hand predicted box, and the second copy is marked as a right-hand predicted box. The target predicted bounding box is defined as one with a left-hand confidence score greater than the first threshold and a right-hand confidence score greater than the second threshold. The system employs a threshold-based prediction bounding box method. A pose estimation model is used to modulate the features of a set of left-hand prediction bounding boxes (including multiple left-hand prediction bounding boxes) to obtain the keypoint coordinates corresponding to the hand regions within the left-hand prediction bounding boxes. Similarly, a pose estimation model is used to modulate the features of a set of right-hand prediction bounding boxes (including multiple right-hand prediction bounding boxes) to obtain the keypoint coordinates corresponding to the hand regions within the right-hand prediction bounding boxes. The labeled information in both the left-hand and right-hand prediction bounding box sets is used to modulate the feature representation of the intermediate layer of the pose estimation model. By adopting a collaborative strategy of multi-label hand detection and identity-aware keypoint localization, the system achieves efficient and accurate detection and differentiation of left and right hands, as well as precise localization of their keypoints in complex scenarios. This significantly improves the robustness and accuracy of hand detection and keypoint localization, thereby solving the technical problems of detection confusion and keypoint identity confusion caused by the difficulty in accurately distinguishing between left and right hands during hand detection and keypoint localization. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of a hand detection method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the output head structure of a multi-label hand detection model according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram illustrating the injection of identity information into an identity-aware hand keypoint estimation network according to an embodiment of this application;

[0026] Figure 4This is a flowchart of another hand detection method according to an embodiment of this application;

[0027] Figure 5 This is a structural diagram of a hand detection device according to an embodiment of this application;

[0028] Figure 6 This is a hardware structure block diagram of a computer terminal for a hand detection method according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to an embodiment of this application, a method embodiment for hand detection is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart of a hand detection method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0033] Step S101: Obtain the image to be processed.

[0034] It should be noted that, Figure 1The steps shown are based entirely on single-frame image information and do not rely on video timing for tracking and error correction. This can significantly reduce processing complexity and latency, making this application widely applicable to single-image analysis, interactive scenarios with high real-time requirements, and power-sensitive edge computing devices.

[0035] Step S102: The image is processed using a computer vision model to obtain multiple prediction boxes for the hand region in the image, as well as the left-hand confidence score and right-hand confidence score corresponding to each prediction box. The left-hand confidence score is used to quantify the probability that the hand region is the left hand, and the right-hand confidence score is used to quantify the probability that the hand region is the right hand.

[0036] In step S102, a computer vision model is used to perform depth analysis on the acquired image to be processed. The computer vision model is trained to identify and enclose all hand regions appearing in the image, achieved by generating a series of prediction boxes. For each prediction box, not only are the coordinates of the bounding box provided, but two specific confidence scores are also calculated and output: a left-hand confidence score and a right-hand confidence score. The left-hand confidence score quantifies and represents the probability that the hand region within the prediction box corresponds to the left hand; similarly, the right-hand confidence score quantifies and represents the probability that the hand region within the prediction box corresponds to the right hand.

[0037] It is worth noting that the computer vision model in step S102 can employ a multi-label hand detection model. Specifically, this model includes two independent binary classifiers: a left-hand binary classifier and a right-hand binary classifier. The left-hand binary classifier determines the confidence score that the predicted bounding box represents the left hand, while the right-hand binary classifier determines the confidence score that the predicted bounding box represents the right hand. Both classifiers use the sigmoid activation function to ensure that the left-hand and right-hand confidence scores independently and non-exclusively reflect whether the hand within the predicted bounding box is left or right. This design allows the model to handle situations where multiple hand regions may exist in an image simultaneously, providing more robust hand detection output, especially in complex scenes with overlapping or interacting hands.

[0038] Figure 2 This is a schematic diagram of the output head structure of a multi-label hand detection model according to an embodiment of this application. Figure 2 As can be seen in the diagram, the core structure of the output head uses a parallel Sigmoid classifier instead of the traditional Softmax classifier. The parallel Sigmoid classifier allows the model to independently determine the left and right hands in the hand region of the image simultaneously, overcoming the limitations of the Softmax classifier due to its mutually exclusive nature, which prevents it from simultaneously providing high-confidence predictions for the left and right hands in overlapping or interactive regions.

[0039] Specifically, the output head of the multi-label hand detection model includes two independent binary sigmoid classifiers: a left-hand binary classifier and a right-hand binary classifier. The outputs of both classifiers are activated by the sigmoid function, meaning they can independently output confidence scores between 0 and 1 without affecting each other.

[0040] In the output head of the multi-label hand detection model, each detection bounding box is associated with two sigmoid classifiers. First, the input image is analyzed through a shared feature extraction network to obtain feature vectors that can be used for classification and localization. Then, the feature vectors are fed into parallel left-hand and right-hand binary classifiers, each of which outputs a score vector representing confidence, denoted as the left-hand confidence score Vleft and the right-hand confidence score Vright, respectively.

[0041] It's important to note that because sigmoid activation is used instead of softmax, Vleft and Vright can vary independently, without being constrained by each other. This means that the same image region may simultaneously have high confidence scores for both the left and right hands, which is particularly important in scenarios where hands overlap or interact closely. The model can flexibly handle the uncertainty of hand positions and will not ignore either hand due to the mutual exclusion nature of individual classifiers.

[0042] In step S103, among the multiple prediction boxes, prediction boxes with a left-hand confidence score greater than the first threshold and a right-hand confidence score less than the second threshold are marked as left-hand prediction boxes belonging to the left hand, and prediction boxes with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold are marked as right-hand prediction boxes belonging to the right hand.

[0043] In step S103, the acquired set of predicted bounding boxes is processed to distinguish and label left-handed and right-handed predicted bounding boxes. Specifically, the left-handed confidence score of each predicted bounding box is compared with a preset first threshold, while its right-handed confidence score is compared with another preset second threshold. When the left-handed confidence score of a predicted bounding box is higher than the first threshold and its right-handed confidence score is lower than the second threshold, the predicted bounding box is explicitly labeled as "left-handed," thus classifying it as a left-handed predicted bounding box. Conversely, when the right-handed confidence score of a predicted bounding box is higher than the second threshold and its left-handed confidence score is lower than the first threshold, the predicted bounding box is labeled as "right-handed," thus classifying it as a right-handed predicted bounding box.

[0044] By applying specific threshold rules, the left and right hand identities of the hand bounding boxes can be accurately distinguished, avoiding the identity confusion that may occur when traditional methods handle overlapping or interacting hands. This process not only makes full use of the output information of the multi-label detection model, but also provides clear identity labels for subsequent keypoint localization steps.

[0045] Optionally, for predicted bounding boxes where the left-hand confidence score is less than the first threshold and the right-hand confidence score is less than the second threshold, the following steps can be selectively performed: 1. Ignore the predicted bounding box: If the preset thresholds are derived from extensive experimental tuning and the detection model's performance is quite stable in practical applications, the most direct approach is to exclude the predicted bounding box, assuming it does not contain any hand information. 2. Refinement analysis: To further improve the comprehensiveness of detection and reduce potential missed detections, the low-confidence predicted bounding box can be sent to the subsequent refinement analysis module. The refinement analysis module performs the following steps: re-cropping and resizing the region, applying additional hand keypoint localization models or other advanced vision algorithms to attempt to discover hand cues. 3. Feedback to model training: Collecting these low-confidence predicted bounding box cases and using them as part of the training data for subsequent model iterations and optimization can improve the model's overall generalization ability and detection accuracy, ensuring the model can better adapt to various complex scenarios.

[0046] Step S104: Copy the target prediction box to obtain a first copy and a second copy. Mark the first copy as a left-handed prediction box and the second copy as a right-handed prediction box. The target prediction box is a prediction box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold.

[0047] In step S104, specific predicted bounding boxes whose left-hand confidence scores exceed a preset first threshold, and whose right-hand confidence scores also exceed a preset second threshold, are referred to as target predicted bounding boxes, and a copying and labeling strategy is implemented. First, a copying operation of the target predicted bounding boxes is performed, resulting in two identical copies: a first copy and a second copy. The first copy is assigned the label "left-hand predicted bounding box," indicating that the hand region in the first copy is identified as left-handed. Similarly, the second copy is assigned the label "right-hand predicted bounding box," indicating that the hand region in the second copy is identified as right-handed.

[0048] By implementing the above copying and labeling process, even when hand region features are highly similar or overlapping, the model can ensure that each hand can be processed and identified separately, providing an important prerequisite for subsequent independent keypoint localization. Step S104 effectively solves the problem of difficulty in distinguishing highly overlapping hand identities in traditional techniques, and can improve the robustness and accuracy of hand detection and keypoint localization technology in complex scenes.

[0049] Step S105: Use the pose estimation model to perform feature modulation processing on the set of left-hand prediction boxes, which includes multiple left-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region in the left-hand prediction box.

[0050] The set of left-hand prediction boxes includes all prediction boxes identified and labeled as left-hand in the previous step. A pose estimation model is applied to the set of left-hand prediction boxes for feature modulation. During the pose estimation model's processing, the hand image region within the left-hand prediction box, along with the left-hand identification information, is input together. This identification information can be presented in One-Hot encoding format and fused with the network feature map through a selected mechanism (such as channel concatenation or conditional normalization). This guides the pose estimation model to focus on extracting deep morphological cues associated with left-hand features. Through this feature modulation processing, the pose estimation model can more accurately estimate the keypoint coordinates of the hand region within the left-hand prediction box, ensuring the accuracy of keypoint localization and the correct identification of left and right hands.

[0051] Step S106: The pose estimation model is used to perform feature modulation processing on the set of right-hand prediction boxes, which includes multiple right-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region in the right-hand prediction box. The label information in the set of left-hand prediction boxes and the set of right-hand prediction boxes is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0052] Specifically, the feature map of the pose estimation model is modulated using the label information of each prediction box in the right-hand prediction box set, i.e., the absolute identity of the "right hand". The modulation process can be implemented in various ways, such as concatenating the identity information with the feature map along the channel dimension, or using it as a parameter in the conditional normalization layer to adjust the feature map, thereby enabling the pose estimation model to learn feature patterns closely related to the right-hand identity. After feature modulation, the pose estimation model can more accurately estimate the coordinates of key points in the hand region within the right-hand prediction box, while ensuring the accuracy and consistency of key point localization for both the left and right hands.

[0053] The above steps involve acquiring the image to be processed; processing the image using a computer vision model to obtain multiple predicted bounding boxes for the hand region in the image, along with left-hand and right-hand confidence scores for each predicted bounding box. The left-hand confidence score quantifies the probability that the hand region is the left hand, and the right-hand confidence score quantifies the probability that the hand region is the right hand. Among the multiple predicted bounding boxes, those with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold are marked as left-hand predicted bounding boxes, and those with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold are marked as right-hand predicted bounding boxes. The target predicted bounding box is copied to obtain a first copy and a second copy. The first copy is marked as a left-hand predicted bounding box, and the second copy is marked as a right-hand predicted bounding box. The target predicted bounding box is the left-hand predicted bounding box. Predicted bounding boxes with hand confidence scores greater than a first threshold and right-hand confidence scores greater than a second threshold are selected. A pose estimation model is used to modulate the features of a set of left-hand predicted bounding boxes (including multiple left-hand predicted bounding boxes) to obtain the keypoint coordinates corresponding to the hand regions within the left-hand predicted bounding boxes. Similarly, a pose estimation model is used to modulate the features of a set of right-hand predicted bounding boxes (including multiple right-hand predicted bounding boxes) to obtain the keypoint coordinates corresponding to the hand regions within the right-hand predicted bounding boxes. The labeled information in both the left-hand and right-hand predicted bounding box sets is used to modulate the feature representation of the intermediate layer of the pose estimation model. By employing a collaborative strategy of multi-label hand detection and identity-aware keypoint localization, the goal of efficiently and accurately detecting and distinguishing between left and right hands and precisely locating their keypoints in complex scenes is achieved. This significantly improves the robustness and accuracy of hand detection and keypoint localization.

[0054] The following are Figure 1 The steps shown are illustrated and explained by way of example.

[0055] According to some optional embodiments of this application, the computer vision model includes multiple parallel classification heads, which include a left-hand binary classifier and a right-hand binary classifier, wherein the left-hand binary classifier is used to determine whether the hand region is the left hand and the right-hand binary classifier is used to determine whether the hand region is the right hand.

[0056] Specifically, a left-hand binary classifier is applied to each detected hand region. By analyzing the features of that region, the model outputs a confidence score representing the left hand, reflecting the probability that the hand region is the left hand. Similarly, a right-hand binary classifier is applied to the same hand region to estimate the confidence score that the region is the right hand, reflecting the probability that the hand region belongs to the right hand. It is worth noting that the outputs of the left-hand and right-hand binary classifiers use the Sigmoid activation function, ensuring that the output values ​​of the two classifiers are independent. This allows the model to assign high left-hand and high right-hand confidence scores to a hand region simultaneously when dealing with overlapping or complex hand scenes, providing flexibility for subsequent processing and analysis. By using the left-hand and right-hand binary classifiers in parallel, the computer vision model can effectively distinguish and identify the left and right hands in an image without temporal information, enhancing the overall hand detection and keypoint localization capabilities.

[0057] It should be noted that this embodiment employs a multi-label detection model with parallel dual-Sigmoid classification heads, allowing for simultaneous prediction of high left-hand and high right-hand confidence for the same image region. This inherently supports the representation of overlapping hand regions in the model structure. Combined with post-processing rules specifically designed for this purpose, the problem of missed detections in closely spaced hand scenes is fundamentally solved. Experiments show that the detection recall rate in overlapping hand scenes can be improved from approximately 65% ​​of traditional methods to over 91%.

[0058] According to some alternative embodiments of this application, the key point coordinates corresponding to the hand region in the left-hand prediction box are obtained by performing feature modulation processing on a set of left-hand prediction boxes including multiple left-hand prediction boxes using a pose estimation model. This can be achieved by the following method: cropping the image with the left-hand prediction box as the center and scaling the cropped image to a preset size to obtain a first image; processing the first image using a feature extraction backbone network to obtain a first image feature map with spatial dimension; converting the marker information in the left-hand prediction box into a first identity feature vector using an identity encoder, and adjusting the first identity feature vector to a first identity feature tensor consistent with the spatial dimension of the first image feature map through a shape change operation; generating a first channel weighting parameter and a first spatial attention parameter based on the first identity feature tensor; weighting the first image feature map by channel dimension using the first channel weighting parameter, and modulating the first image feature map by spatial attention dimension using the first spatial attention parameter to obtain a first target feature map; performing an upsampling operation on the first target feature map using a key point decoder to restore the spatial resolution of the first target feature map to obtain a first key point heatmap; and determining the key point coordinates corresponding to the hand region in the left-hand prediction box based on the first key point heatmap.

[0059] Figure 3This is a schematic diagram of identity information injection into an identity-aware hand keypoint estimation network according to an embodiment of this application. The following is in conjunction with... Figure 3 This section details how to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction frame.

[0060] 1. Image cropping centered on the left-hand prediction bounding box: First, the left-hand prediction bounding box generated by the pose estimation model (multi-label hand detection model) is used to crop a local region including the left hand from the original image. This cropping operation ensures that the hand is the focus of attention in the image, reducing background interference.

[0061] 2. Scale the cropped image to a preset size: The cropped left-hand image is further scaled to the preset fixed size required by the model input to obtain the first image.

[0062] 3. Process the first image using a feature extraction backbone network: Input the first image into a feature extraction backbone network, such as HRNet-W32, to obtain the first image feature map. The first image feature map includes depth information, which is the basis for subsequent keypoint localization.

[0063] 4. Use an identity encoder to convert the label information in the left-hand prediction box into a first identity feature vector: Convert the left-hand identity label (e.g., [1,0]) into a high-dimensional vector through One-Hot encoding, and map it to a dimension compatible with the first image feature map through a fully connected layer (FC layer) to obtain the first identity feature vector.

[0064] 5. Adjust the first identity feature vector to a first identity feature tensor with the same spatial dimension as the first image feature map through a shape transformation operation: Reshape the first identity feature vector into a tensor that matches the size of the feature map. By copying and expanding the first identity feature vector, ensure that the first identity feature vector can be concatenated with the first image feature map in the channel dimension, thereby attaching hand identity information at each spatial location.

[0065] 6. Generate first channel weighted parameters and first spatial attention parameters based on the first identity feature tensor: Further utilize the first identity feature tensor to generate two parameters: the first channel weighted parameters and the first spatial attention parameters. These parameters represent the importance of the left-hand identity to each channel in the feature map and the attention distribution across spatial regions.

[0066] 7. Use the first channel weighting parameter to perform channel dimension weighting on the first image feature map: The weights of each channel in the first image feature map can be adjusted to make the network pay more attention to features related to the localization of left-hand key points.

[0067] 8. Modulate the spatial attention dimension of the first image feature map using the first spatial attention parameter: By modulating the spatial attention distribution of the first image feature map, the information expression of the hand region can be enhanced and the interference of non-hand regions can be suppressed.

[0068] 9. Obtain the first target feature map: After the above channel weighting and spatial attention modulation, a feature map that integrates the left-hand identity information is generated, which is the first target feature map.

[0069] 10. Perform upsampling operation on the first target feature map using the key point decoder: send the first target feature map into the key point decoder, and restore the spatial resolution of the feature map through the upsampling operation to obtain the first key point heatmap.

[0070] 11. Based on the first key point heatmap, determine the coordinates of the key points corresponding to the hand region in the left-hand prediction box: Each pixel in the heatmap represents the confidence level of the corresponding hand key point. By finding the peak position in the heatmap, the precise coordinates of each hand key point can be determined, thereby realizing the localization of the hand key points.

[0071] On the other hand, using a pose estimation model to perform feature modulation processing on a set of left-hand prediction boxes including multiple left-hand prediction boxes to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction box can also be achieved through the following method: obtaining the first geometric parameters and first label information of the left-hand prediction box; encoding the first geometric parameters into a first geometric feature vector, wherein the first geometric feature vector includes: the position and scale information of the left-hand prediction box; encoding the first label information into a first semantic feature vector, wherein the first semantic feature vector includes: the category attribute and confidence information of the left-hand prediction box; fusing the first geometric feature vector and the first semantic feature vector to obtain a first context representation; and using a parameter generation network to process the first context representation. The following characterization process is performed to obtain the first feature modulation parameters, which include: first channel weighting coefficients and a first spatial transformation matrix. Based on the first geometric parameters, the corresponding first region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The first channel weighting coefficients are applied to the channel dimension of the first region feature map to recalibrate the importance of different feature channels, and the first spatial transformation matrix is ​​applied to the spatial dimension of the first region feature map to perform attention weighting on different spatial positions within the left-hand prediction box. The modulated first region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-hand prediction box.

[0072] In this embodiment, firstly, the first geometric parameters of the left-hand prediction box are accurately extracted from the output of the pose estimation model. These first geometric parameters include specific location information and scale attributes. The extracted geometric parameters are then converted into a digitized first geometric feature vector, which includes the location and size information of the left-hand prediction box in the image.

[0073] Secondly, the first labeling information associated with the left-handed predicted bounding box is obtained. This first labeling information includes the category attribute of the left-handed predicted bounding box and its corresponding confidence score. The first labeling information is then transformed into a first semantic feature vector, which not only reflects the category to which the predicted bounding box belongs (i.e., left-handed) but also quantifies the model's certainty regarding this classification.

[0074] Then, the first geometric feature vector and the first semantic feature vector are fused to generate a first contextual representation that integrates position, scale, and confidence information. This fusion process ensures that the model can understand and represent the features of the hand from multiple dimensions.

[0075] Next, a parameter generation network is used to perform deep processing on the first context representation to obtain the first feature modulation parameters, which include first channel weighting coefficients and a first spatial transformation matrix. The channel weighting coefficients are used to guide the network to recalibrate the importance of different feature channels, and the spatial transformation matrix is ​​used to apply attention weights to specific spatial locations, thereby strengthening the feature representation of key regions.

[0076] Furthermore, based on the position and scale information of the left-hand bounding box, a first region feature map is extracted from the intermediate layer of the pose estimation model, where the first region feature map includes the range of the left-hand bounding box. The first channel weighting coefficient is applied to the channel dimension of the first region feature map to dynamically adjust the importance of different feature channels; simultaneously, the first spatial transformation matrix is ​​applied to the spatial dimension of the first region feature map to enhance the model's ability to perceive and locate key points within the left-hand bounding box.

[0077] Finally, the successor layer of the pose estimation model receives the modulated first region feature map and performs a detailed analysis of the feature-modulated image region by utilizing pre-trained network weights, thereby obtaining the precise coordinates of the key points of the hand within the left hand prediction box.

[0078] It should be noted that this embodiment injects absolute hand identity information (a set of left / right hand prediction boxes, each carrying labeled information) as a condition into the keypoint estimation network, modulating the left and right hands at the network feature level, forcing the network to learn identity-related feature representations. This ensures that the left hand always outputs keypoints that conform to the anatomical structure of the left hand, and the right hand does the same, fundamentally solving the keypoint identity confusion problem and reducing the identity error rate to below 1%.

[0079] In some optional embodiments of this application, the pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes including multiple right-hand prediction boxes to obtain the key point coordinates corresponding to the hand region in the right-hand prediction box. This can be achieved by the following method: cropping the image with the right-hand prediction box as the center and scaling the cropped image to a preset size to obtain a second image; processing the second image using a feature extraction backbone network to obtain a second image feature map with spatial dimension; converting the marker information in the right-hand prediction box into a second identity feature vector using an identity encoder, and adjusting the second identity feature vector to a second identity feature tensor with the same spatial dimension as the second image feature map through a shape change operation; generating a second channel weighting parameter and a second spatial attention parameter based on the second identity feature tensor; weighting the second image feature map by channel dimension using the second channel weighting parameter, and modulating the second image feature map by spatial attention dimension using the second spatial attention parameter to obtain a second target feature map; performing an upsampling operation on the second target feature map using a key point decoder to restore the spatial resolution of the second target feature map to obtain a second key point heatmap; and determining the key point coordinates corresponding to the hand region in the right-hand prediction box based on the second key point heatmap.

[0080] It should be noted that the method for generating the key point coordinates corresponding to the hand region in the right-hand prediction frame can be found in the method for generating the key point coordinates corresponding to the hand region in the left-hand prediction frame, as they are consistent in their technical implementation principles.

[0081] On the other hand, using a pose estimation model to perform feature modulation processing on a set of right-hand prediction boxes including multiple right-hand prediction boxes to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box can also be achieved through the following method: obtaining the second geometric parameters and second labeling information of the right-hand prediction box; encoding the second geometric parameters into a second geometric feature vector, wherein the second geometric feature vector includes: the position and scale information of the right-hand prediction box; encoding the second labeling information into a second semantic feature vector, wherein the second semantic feature vector includes: the category attribute and confidence information of the right-hand prediction box; fusing the second geometric feature vector and the second semantic feature vector to obtain a second context representation; and using a parameter generation network to process the second context representation. The following characterization process yields the second feature modulation parameters, which include: second channel weighting coefficients and a second spatial transformation matrix. Based on the second geometric parameters, the corresponding second region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The second channel weighting coefficients are applied to the channel dimension of the second region feature map to recalibrate the importance of different feature channels, and the second spatial transformation matrix is ​​applied to the spatial dimension of the second region feature map to perform attention weighting on different spatial positions within the right-hand prediction box. The modulated second region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box.

[0082] It should be noted that the method for generating the key point coordinates corresponding to the hand region in the right-hand prediction frame can be found in the method for generating the key point coordinates corresponding to the hand region in the left-hand prediction frame, as they are consistent in their technical implementation principles.

[0083] As some optional embodiments of this application, before performing feature modulation processing on a set of left-handed prediction boxes including multiple left-handed prediction boxes using the pose estimation model, the following steps may also be performed: performing a first non-maximum suppression operation on the left-handed prediction boxes marked as belonging to the left hand to obtain a set of left-handed prediction boxes; performing a second non-maximum suppression operation on the right-handed prediction boxes marked as belonging to the right hand to obtain a set of right-handed prediction boxes, wherein the first non-maximum suppression operation and the second non-maximum suppression operation are independent operations.

[0084] Understandably, before using the pose estimation model for feature modulation, non-maximum suppression (NMS) needs to be performed on the predicted bounding boxes output by the detection model to further optimize and filter the prediction results. First, for all predicted bounding boxes labeled as left-handed, i.e., these boxes are initially determined to contain left-handed regions, a first NMS operation is performed. This process aims to reduce duplicate predicted bounding boxes, especially when overlapping detections exist in the left-handed region. By comparing the degree of overlap between predicted bounding boxes and combining the confidence score of each predicted bounding box, the first NMS operation selects the predicted bounding box with the highest confidence and the least overlap, thus obtaining the set of left-handed predicted bounding boxes. The set of left-handed predicted bounding boxes includes the most likely left-handed regions, and the predicted bounding boxes are independent of each other, with no overlap or an acceptable degree of overlap. Then, a similar strategy is used to process the predicted bounding boxes labeled as right-handed. A second NMS operation is performed, the goal of which is also to eliminate overlapping predicted bounding boxes, ensuring the accuracy and robustness of the final set of right-handed predicted bounding boxes. The second NMS operation is independent of the first NMS operation, meaning that the processing of the left-handed and right-handed predicted bounding box sets is performed in parallel and does not affect each other. By performing the first and second NMS operations on the left-hand and right-hand prediction boxes respectively, redundancy in the prediction box set can be effectively reduced, improving the efficiency and accuracy of subsequent key point localization.

[0085] As some alternative embodiments of this application, the pose estimation model is trained by the following methods: A classification loss is determined based on the sum of the left-hand and right-hand classification losses, wherein both the left-hand and right-hand classification losses are calculated separately using the binary cross-entropy loss function; a bounding box regression loss is determined based on the regression difference between the predicted bounding box coordinate parameters and the ground truth hand bounding box coordinate parameters; a target loss is determined based on the difference between the confidence score of the predicted bounding box and the target label calculated based on the intersection-union ratio of the predicted bounding box and the ground truth hand bounding box; a loss function is determined based on the classification loss, bounding box regression loss, and target loss; and an initial pose estimation model is trained based on the training set, obtaining the pose estimation model when the loss function satisfies a preset convergence condition.

[0086] In this embodiment, the sum of the left-hand classification loss and the right-hand classification loss is first set as the overall classification loss. The left-hand and right-hand classification losses are calculated independently using a binary cross-entropy loss function. This mechanism ensures the model's accurate learning of left-hand and right-hand classification tasks, enhancing its identity perception capabilities.

[0087] Secondly, the bounding box regression loss is determined by comparing the difference between the predicted bounding box coordinates and the actual hand annotation bounding box coordinates. The bounding box regression loss reflects the accuracy of the model in locating the hand region, prompting the model to optimize bounding box predictions and achieve more accurate hand detection.

[0088] Furthermore, the target loss is determined by the difference between the target label calculated based on the confidence score of the predicted bounding box and the intersection-over-union (IoU) of the ground truth hand bounding box. The target loss ensures that the model can correctly assess the presence or absence of a hand within the predicted bounding box, further improving the robustness of the detection.

[0089] A comprehensive loss function is constructed by combining classification loss, bounding box regression loss, and target recognition loss. This loss function comprehensively measures the model's overall performance in terms of classification accuracy, localization precision, and target recognition ability, and is a key indicator for model optimization during training.

[0090] Based on the aforementioned loss function framework, the initial pose estimation model is iteratively trained using the training set. In each iteration, the model predicts hand detection boxes and classification results based on the input image, then calculates the difference between the actual prediction and the ground truth annotation, and updates the model parameters through backpropagation to minimize the loss function. When the loss function value changes steadily during training, i.e., the preset convergence condition is met, it means that the model has learned stably and no longer improves significantly. At this point, training is complete, and a mature hand pose estimation model is obtained. This pose estimation model has the ability to robustly detect and locate hands in complex scenes and distinguish between left and right hands, which is the cornerstone for achieving accurate hand keypoint localization.

[0091] Figure 4 This is a flowchart of another hand detection method according to an embodiment of this application, such as... Figure 4 As shown, the method includes the following steps:

[0092] Step S401, Input image preprocessing: Preprocess the original input image, including: adjusting the size, standardizing the format, and ensuring that it meets the requirements of the model input.

[0093] Step S402, Multi-label Hand Detection Model: Perform multi-label hand detection on the preprocessed input image to obtain a series of predicted boxes. Each predicted box includes bounding box coordinates, target confidence score, left-hand confidence score, and right-hand confidence score. The multi-label hand detection model innovatively uses two independent sigmoid classifiers, each responsible for determining whether each predicted box contains a left or right hand. Compared to the Softmax classifier, it can better handle the complex situation of overlapping hands.

[0094] Step S403, Identity Determination and Post-processing: Based on the predicted bounding box scores output by the multi-label hand detection model, identity is determined. If the left-hand confidence score is greater than a preset first threshold while the right-hand score is lower than a second threshold, the predicted bounding box is labeled as a left hand; conversely, if the right-hand confidence score is greater than the second threshold while the left-hand score is lower than the first threshold, it is labeled as a right hand. For predicted bounding boxes where both left-hand and right-hand confidence scores exceed the thresholds, two copies are generated, labeled as left hand and right hand respectively. Then, a first non-maximum suppression operation is performed on all predicted bounding boxes labeled as left hand, and a second non-maximum suppression operation is performed on predicted bounding boxes labeled as right hand, resulting in the final set of left-hand and right-hand predicted bounding boxes. The post-processing step aims to eliminate redundant detection boxes, ensuring that each hand has only one optimal detection result.

[0095] In step S404, non-maximum suppression is performed independently on all predicted bounding boxes marked as left-handed and those marked as right-handed to obtain the final left-handed and right-handed detection results.

[0096] In step S405, for each detection box in the left-hand and right-hand detection results, the image area is cropped with that box as the center and scaled to a fixed size.

[0097] Step S406, Identity-Aware Hand Keypoint Estimation Model: The cropped and scaled image regions (first image and second image) along with the corresponding identity information (labeled as left or right hand) are input into the keypoint estimation model. The model modulates its feature representation through the identity information, thereby accurately distinguishing the keypoints (21 keypoint coordinates) of the left and right hands. The input of the identity-aware hand keypoint estimation model includes not only the hand image but also the hand identity label represented by One-Hot encoding. The hand identity label is converted into a high-dimensional embedding vector through a fully connected layer and fused with the image features by concatenation or conditional normalization.

[0098] The following is a detailed implementation example illustrating the implementation details of the hand detection method proposed in this application.

[0099] The experimental setup is as follows: Hardware environment: Training and testing were conducted on a workstation equipped with an NVIDIA GeForce RTX 4090 GPU (24GB VRAM), an Intel i9-13900K CPU, and 64GB of RAM. Software environment: Ubuntu 20.04 operating system, PyTorch 1.12.1 deep learning framework, and CUDA 11.6 acceleration library were used. Dataset: A fusion dataset was used for training. This included public datasets such as COCO-Hand and Freihand, as well as a self-synthesized two-handed interaction dataset. The total number of training images was approximately 150,000, and the validation set consisted of 20,000 images. The self-synthesized data was generated using the Blender engine, focusing on creating challenging samples such as hands touching, crossed, and clenched fists. Data annotation: All hand instances were annotated with bounding boxes and their absolute identities (left / right), as well as the precise coordinates of 21 hand keypoints.

[0100] The model implementation details are as follows: Multi-label detection model (computer vision model): An improvement upon the YOLOv5s baseline model. The input image size is 640x640. The original classification head is replaced with two parallel convolutional layers (kernel=1), each connected to a Sigmoid activation function. Training hyperparameters: initial learning rate 0.01, using a cosine annealing scheduler, batch size 128, SGD optimizer, momentum 0.937, weight decay 0.0005, trained for 300 epochs. Keypoint estimation model (pose estimation model): A lightweight HRNet-W32 is used as the backbone network. The input image size is 256x256. Identity labels are converted into 2D one-hot vectors, mapped to 32-dimensional embedding vectors through a fully connected layer, and then spatially copied and concatenated with the feature map output from the backbone (downsampled by 4 times, with 32 channels) to form a 64-channel feature map before being input into the decoder. Training hyperparameters: initial learning rate 0.001, Adam optimizer, batch size 64, training for 200 epochs.

[0101] Performance metrics and results are as follows. The model was evaluated on the reserved test set and achieved the following metrics: Detection performance: On the hand detection task, the mean precision (mAP@0.5) reached 0.928. Specifically, the recall for overlapping hands (IoU>0.3) improved from 65% in the traditional Softmax method to 91% in this solution. Keypoint performance: The mean precision (MeanPerJointPositionError, MPJPE) for hand keypoint estimation was 5.2 pixels (on a 256x256 image). The keypoint identity confusion error rate was less than 0.5%. Inference speed: On an RTX4090, the end-to-end processing time (including detection and keypoint estimation) for a single 640x640 image averaged 35 milliseconds (approximately 28 FPS), meeting the requirements of real-time applications.

[0102] The above implementation examples demonstrate that the hand detection method provided in this application can efficiently and accurately solve the problem of hand detection and localization in a single frame image, and exhibits excellent robustness, especially in extremely overlapping scenes.

[0103] Figure 5 This is a structural diagram of a hand detection device according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes:

[0104] The acquisition module 51 is used to acquire the image to be processed.

[0105] The processing module 52 is used to process the image using a computer vision model to obtain multiple predicted boxes for the hand region in the image, as well as the left-hand confidence score and right-hand confidence score corresponding to each predicted box. The left-hand confidence score is used to quantify the probability that the hand region is the left hand, and the right-hand confidence score is used to quantify the probability that the hand region is the right hand.

[0106] The first marking module 53 is used to mark, among multiple prediction boxes, prediction boxes whose left-hand confidence scores are greater than a first threshold and whose right-hand confidence scores are less than a second threshold as left-hand prediction boxes, and prediction boxes whose right-hand confidence scores are greater than the second threshold and whose left-hand confidence scores are less than the first threshold as right-hand prediction boxes.

[0107] The second labeling module 54 is used to copy the target prediction box to obtain a first copy and a second copy. The first copy is labeled as a left-handed prediction box and the second copy is labeled as a right-handed prediction box. The target prediction box is a prediction box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold.

[0108] The first parsing module 55 is used to perform feature modulation processing on a set of left-hand prediction boxes, which includes multiple left-hand prediction boxes, using a pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-hand prediction box.

[0109] The second parsing module 56 is used to perform feature modulation processing on the set of right-hand prediction boxes, which includes multiple right-hand prediction boxes, using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-hand prediction box. The label information in the set of left-hand prediction boxes and the set of right-hand prediction boxes is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0110] Optionally, the computer vision model includes multiple parallel classification heads, including a left-hand binary classifier and a right-hand binary classifier, wherein the left-hand binary classifier is used to determine whether the hand region is the left hand and the right-hand binary classifier is used to determine whether the hand region is the right hand.

[0111] Optionally, a pose estimation model is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction box. Specifically, this includes the following steps: cropping the image centered on the left-hand prediction box and scaling the cropped image to a preset size to obtain a first image; processing the first image using a feature extraction backbone network to obtain a first image feature map with spatial dimensions; converting the marker information in the left-hand prediction box into a first identity feature vector using an identity encoder, and adjusting the first identity feature vector to a first identity feature tensor consistent with the spatial dimensions of the first image feature map through a shape transformation operation; generating first channel weighting parameters and first spatial attention parameters based on the first identity feature tensor; weighting the first image feature map by channel dimensions using the first channel weighting parameters, and modulating the first image feature map by spatial attention dimensions using the first spatial attention parameters to obtain a first target feature map; performing an upsampling operation on the first target feature map using a keypoint decoder to restore the spatial resolution of the first target feature map, obtaining a first keypoint heatmap; and determining the keypoint coordinates corresponding to the hand region within the left-hand prediction box based on the first keypoint heatmap.

[0112] Optionally, a pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box. Specifically, this includes the following steps: cropping the image centered on the right-hand prediction box and scaling the cropped image to a preset size to obtain a second image; processing the second image using a feature extraction backbone network to obtain a second image feature map with spatial dimensions; converting the marker information in the right-hand prediction box into a second identity feature vector using an identity encoder, and adjusting the second identity feature vector to a second identity feature tensor with the same spatial dimension as the second image feature map through a shape transformation operation; generating second channel weighting parameters and second spatial attention parameters based on the second identity feature tensor; applying channel dimension weighting to the second image feature map using the second channel weighting parameters, and applying spatial attention dimension modulation to the second image feature map using the second spatial attention parameters to obtain a second target feature map; performing upsampling on the second target feature map using a keypoint decoder to restore the spatial resolution of the second target feature map, obtaining a second keypoint heatmap; and determining the keypoint coordinates corresponding to the hand region within the right-hand prediction box based on the second keypoint heatmap.

[0113] Optionally, a pose estimation model is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the left-hand prediction box. Specifically, this includes the following steps: obtaining the first geometric parameters and first labeling information of the left-hand prediction box; encoding the first geometric parameters into a first geometric feature vector, wherein the first geometric feature vector includes: the position and scale information of the left-hand prediction box; encoding the first labeling information into a first semantic feature vector, wherein the first semantic feature vector includes: the category attribute and confidence information of the left-hand prediction box; fusing the first geometric feature vector and the first semantic feature vector to obtain a first context representation; and using a parameter generation network to process the first context... The representation is processed to obtain the first feature modulation parameters, which include: first channel weighting coefficients and a first spatial transformation matrix. Based on the first geometric parameters, the corresponding first region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The first channel weighting coefficients are applied to the channel dimension of the first region feature map to recalibrate the importance of different feature channels, and the first spatial transformation matrix is ​​applied to the spatial dimension of the first region feature map to perform attention weighting on different spatial positions within the left-hand prediction box. The modulated first region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-hand prediction box.

[0114] Optionally, a pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the keypoint coordinates corresponding to the hand region within the right-hand prediction box. Specifically, this includes the following steps: obtaining the second geometric parameters and second labeling information of the right-hand prediction box; encoding the second geometric parameters into a second geometric feature vector, wherein the second geometric feature vector includes: the position and scale information of the right-hand prediction box; encoding the second labeling information into a second semantic feature vector, wherein the second semantic feature vector includes: the category attribute and confidence information of the right-hand prediction box; fusing the second geometric feature vector and the second semantic feature vector to obtain a second context representation; and using a parameter generation network to process the second context. The representation is processed to obtain the second feature modulation parameters, which include: second channel weighting coefficients and a second spatial transformation matrix. Based on the second geometric parameters, the corresponding second region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model. The second channel weighting coefficients are applied to the channel dimension of the second region feature map to recalibrate the importance of different feature channels, and the second spatial transformation matrix is ​​applied to the spatial dimension of the second region feature map to perform attention weighting on different spatial positions within the right-hand prediction box. The modulated second region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region within the right-hand prediction box.

[0115] Optionally, the hand detection device is further configured to perform the following steps before performing feature modulation processing on a set of left-hand prediction boxes including multiple left-hand prediction boxes using a pose estimation model: performing a first non-maximum suppression operation on the left-hand prediction boxes marked as belonging to the left hand to obtain a set of left-hand prediction boxes; performing a second non-maximum suppression operation on the right-hand prediction boxes marked as belonging to the right hand to obtain a set of right-hand prediction boxes, wherein the first non-maximum suppression operation and the second non-maximum suppression operation are independent operations.

[0116] Optionally, the pose estimation model is trained using the following method: The classification loss is determined by summing the left-hand and right-hand classification losses, where both are calculated separately using the binary cross-entropy loss function; the bounding box regression loss is determined by the regression difference between the predicted bounding box coordinate parameters and the ground truth hand bounding box coordinate parameters; the target loss is determined by the difference between the confidence score of the predicted bounding box and the target label calculated based on the intersection-union ratio (IUU) of the predicted and ground truth hand bounding boxes; the loss function is determined based on the classification loss, bounding box regression loss, and target loss; and the initial pose estimation model is trained on the training set, obtaining the pose estimation model when the loss function meets the preset convergence condition.

[0117] It should be noted that the modules in the above figure can be program modules (for example, a set of program instructions to implement a specific function 5) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.

[0118] It should be noted that, Figure 5 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.

[0119] Figure 6 A hardware block diagram of a computer terminal for implementing a hand detection method is shown. Figure 6 As shown, the computer terminal 60 may include one or more processors 602 (shown as 602a, 602b, ..., 602n in the figure) 602 (processor 602 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 604 for storing data, and a transmission module 606 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 60 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0120] It should be noted that the aforementioned one or more processors 602 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 60. As described in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0121] The memory 604 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the hand detection method in this embodiment. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, thereby realizing the aforementioned hand detection method. The memory 604 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 604 may further include memory remotely located relative to the processor 602, and these remote memories can be connected to the computer terminal 60 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The transmission module 606 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 60. In one example, the transmission module 606 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 606 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0123] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 60.

[0124] It should be noted here that, in some optional embodiments, the above... Figure 6 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 6 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0125] It should be noted that, Figure 6 The computer terminal shown is used to execute Figure 1 The hand detection method shown above means that the relevant explanations in the above command execution method also apply to this electronic device, and will not be repeated here.

[0126] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-mentioned hand detection method.

[0127] A program on a non-volatile storage medium performs the following functions: acquires an image to be processed; processes the image using a computer vision model to obtain multiple predicted bounding boxes for the hand region in the image, along with a left-hand confidence score and a right-hand confidence score for each predicted bounding box, where the left-hand confidence score quantifies the probability that the hand region is the left hand, and the right-hand confidence score quantifies the probability that the hand region is the right hand; among the multiple predicted bounding boxes, predicting boxes with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold are marked as left-hand predicted boxes, and predicting boxes with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold are marked as right-hand predicted boxes; copies... The target predicted bounding box is obtained as a first copy and a second copy. The first copy is labeled as a left-handed predicted bounding box, and the second copy is labeled as a right-handed predicted bounding box. The target predicted bounding box is a predicted bounding box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold. The left-handed predicted bounding box set, which includes multiple left-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-handed predicted bounding box. The right-handed predicted bounding box set, which includes multiple right-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-handed predicted bounding box. The labeling information in the left-handed and right-handed predicted bounding box sets is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0128] This application also provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described hand detection method when it runs.

[0129] The processor runs a program that performs the following functions: acquiring an image to be processed; processing the image using a computer vision model to obtain multiple predicted bounding boxes for the hand region in the image, along with a left-hand confidence score and a right-hand confidence score for each predicted bounding box, where the left-hand confidence score quantifies the probability that the hand region is the left hand, and the right-hand confidence score quantifies the probability that the hand region is the right hand; among the multiple predicted bounding boxes, marking predicted bounding boxes with a left-hand confidence score greater than a first threshold and a right-hand confidence score less than a second threshold as left-hand predicted bounding boxes, and marking predicted bounding boxes with a right-hand confidence score greater than the second threshold and a left-hand confidence score less than the first threshold as right-hand predicted bounding boxes; copying the target... The first and second copies of the predicted bounding boxes are obtained by labeling the predicted bounding boxes. The first copy is labeled as a left-handed predicted bounding box, and the second copy is labeled as a right-handed predicted bounding box. The target predicted bounding box is the predicted bounding box whose left-handed confidence score is greater than a first threshold and whose right-handed confidence score is greater than a second threshold. The left-handed predicted bounding box set, which includes multiple left-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-handed predicted bounding box. The right-handed predicted bounding box set, which includes multiple right-handed predicted bounding boxes, is subjected to feature modulation processing using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-handed predicted bounding box. The labeling information in the left-handed and right-handed predicted bounding box sets is used to modulate the feature representation of the intermediate layer of the pose estimation model.

[0130] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] In the above embodiments of this application, the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with relevant laws, regulations and standards, take necessary protective measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0137] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A hand detection method, characterized in that, include: Obtain the image to be processed; The image is processed using a computer vision model to obtain multiple predicted bounding boxes for the hand region in the image, as well as a left-hand confidence score and a right-hand confidence score for each predicted bounding box. The left-hand confidence score is used to quantify the probability that the hand region is a left hand, and the right-hand confidence score is used to quantify the probability that the hand region is a right hand. In the plurality of prediction boxes, for prediction boxes where the left-hand confidence score is greater than the first threshold and the right-hand confidence score is less than the second threshold, the prediction box is assigned the identity label of being left-handed, thus obtaining a left-hand prediction box; and for prediction boxes where the right-hand confidence score is greater than the second threshold and the left-hand confidence score is less than the first threshold, the prediction box is assigned the identity label of being right-handed, thus obtaining a right-hand prediction box. The target prediction box is copied to obtain a first copy and a second copy. The first copy is marked as the left-hand prediction box and the second copy is marked as the right-hand prediction box. The target prediction box is the prediction box whose left-hand confidence score is greater than the first threshold and whose right-hand confidence score is greater than the second threshold. The process involves using a pose estimation model to perform feature modulation processing on a set of left-hand prediction boxes, including multiple predicted left-hand boxes, to obtain the keypoint coordinates corresponding to the hand region within the predicted left-hand boxes. This includes: cropping the image centered on the predicted left-hand boxes and scaling the cropped image to a preset size to obtain a first image; processing the first image using a feature extraction backbone network to obtain a first image feature map with spatial dimensions; converting the marker information in the predicted left-hand boxes into a first identity feature vector using an identity encoder, and adjusting the first identity feature vector to a first identity feature tensor with the same spatial dimensions as the first image feature map through a shape transformation operation; generating a first channel weighting parameter and a first spatial attention parameter based on the first identity feature tensor; applying channel dimension weighting to the first image feature map using the first channel weighting parameter and spatial attention dimension modulation to the first image feature map using the first spatial attention parameter to obtain a first target feature map; performing an upsampling operation on the first target feature map using a keypoint decoder to restore the spatial resolution of the first target feature map, obtaining a first keypoint heatmap; and determining the keypoint coordinates corresponding to the hand region within the predicted left-hand boxes based on the first keypoint heatmap. The pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, which includes multiple right-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region in the right-hand prediction box. The label information in the set of left-hand prediction boxes and the set of right-hand prediction boxes is used to modulate the feature representation of the intermediate layer of the pose estimation model.

2. The method according to claim 1, characterized in that, The computer vision model includes multiple parallel classification heads, each including a left-hand binary classifier and a right-hand binary classifier. The left-hand binary classifier is used to determine whether the hand region is a left hand, and the right-hand binary classifier is used to determine whether the hand region is a right hand.

3. The method according to claim 1, characterized in that, The pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region within the right-hand prediction box, including: The image is cropped with the right-hand prediction box as the center, and the cropped image is scaled to a preset size to obtain a second image; The second image is processed using a feature extraction backbone network to obtain a second image feature map with spatial dimension; The marking information in the right-hand prediction box is converted into a second identity feature vector using an identity encoder, and the second identity feature vector is adjusted into a second identity feature tensor with the same dimension as the second image feature graph space through a shape change operation; Generate second channel weighted parameters and second spatial attention parameters based on the second identity feature tensor; The second image feature map is weighted by channel dimension using the second channel weighting parameter, and the second image feature map is modulated by spatial attention dimension using the second spatial attention parameter to obtain the second target feature map; The second target feature map is upsampled using a keypoint decoder to restore its spatial resolution and obtain a second keypoint heatmap. Based on the second key point heatmap, the key point coordinates corresponding to the hand region in the right-hand prediction frame are determined.

4. The method according to claim 1, characterized in that, The pose estimation model is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region within the left-hand prediction boxes, including: Obtain the first geometric parameters and first marker information of the left-hand prediction box; The first geometric parameter is encoded into a first geometric feature vector, wherein the first geometric feature vector includes: the position and scale information of the left-hand prediction box; The first label information is encoded into a first semantic feature vector, wherein the first semantic feature vector includes: the category attribute and confidence information of the left-hand prediction box; The first geometric feature vector and the first semantic feature vector are fused to obtain the first context representation; The first context representation is processed using a parameter generation network to obtain a first feature modulation parameter, wherein the first feature modulation parameter includes: a first channel weighting coefficient and a first spatial transformation matrix; Based on the first geometric parameters, the corresponding first region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model; The first channel weighting coefficient is applied to the channel dimension of the first region feature map to recalibrate the importance of different feature channels, and the first spatial transformation matrix is ​​applied to the spatial dimension of the first region feature map to perform attention weighting on different spatial positions within the left-hand prediction box. The modulated first region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-hand prediction frame.

5. The method according to claim 1, characterized in that, The pose estimation model is used to perform feature modulation processing on a set of right-hand prediction boxes, including multiple right-hand prediction boxes, to obtain the key point coordinates corresponding to the hand region within the right-hand prediction box, including: Obtain the second geometric parameters and second marker information of the right-hand prediction box; The second geometric parameter is encoded into a second geometric feature vector, wherein the second geometric feature vector includes: the position and scale information of the right-hand prediction box; The second labeling information is encoded into a second semantic feature vector, wherein the second semantic feature vector includes: the category attribute and confidence information of the right-hand prediction box; The second geometric feature vector and the second semantic feature vector are fused to obtain the second context representation; The second context representation is processed using a parameter generation network to obtain the second feature modulation parameters, wherein the second feature modulation parameters include: second channel weighting coefficients and a second spatial transformation matrix; Based on the second geometric parameters, the corresponding second region feature map is extracted from the full-image feature map generated by the intermediate layer of the pose estimation model; The second channel weighting coefficient is applied to the channel dimension of the second region feature map to recalibrate the importance of different feature channels, and the second spatial transformation matrix is ​​applied to the spatial dimension of the second region feature map to perform attention weighting on different spatial positions within the right-hand prediction box. The modulated second region feature map is analyzed using the successor layer of the intermediate layer of the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-hand prediction frame.

6. The method according to claim 1, characterized in that, Before performing feature modulation processing on the set of left-hand prediction boxes, which includes multiple left-hand prediction boxes, using a pose estimation model, the method further includes: Perform a first nonmaximum suppression operation on the left-handed prediction boxes marked as belonging to the left hand to obtain the set of left-handed prediction boxes; A second nonmaximum suppression operation is performed on the right-handed prediction boxes marked as belonging to the right-hand side to obtain the set of right-handed prediction boxes, wherein the first nonmaximum suppression operation and the second nonmaximum suppression operation are independent operations.

7. The method according to claim 1, characterized in that, The attitude estimation model was trained using the following method: The classification loss is determined by summing the left-hand classification loss and the right-hand classification loss, wherein the left-hand classification loss and the right-hand classification loss are calculated separately using the binary cross-entropy loss function. The bounding box regression loss is determined based on the regression difference between the predicted bounding box coordinate parameters and the actual hand annotation bounding box coordinate parameters. The target loss is determined based on the difference between the confidence score of the predicted bounding box and the target label calculated based on the intersection-union ratio of the predicted bounding box and the real hand annotation box; The loss function is determined based on the classification loss, the bounding box regression loss, and the targeting loss; The initial pose estimation model is trained based on the training set, and the pose estimation model is obtained when the loss function satisfies the preset convergence condition.

8. A hand detection device, characterized in that, include: The acquisition module is used to acquire the image to be processed; The processing module is used to process the image using a computer vision model to obtain multiple prediction boxes for the hand region in the image, as well as a left-hand confidence score and a right-hand confidence score corresponding to each prediction box. The left-hand confidence score is used to quantify the probability that the hand region is a left hand, and the right-hand confidence score is used to quantify the probability that the hand region is a right hand. The first labeling module is used to assign the label "belonging to left-hand" to a prediction box in the plurality of prediction boxes where the left-hand confidence score is greater than a first threshold and the right-hand confidence score is less than a second threshold, thereby obtaining a left-hand prediction box; and to assign the label "belonging to right-hand" to a prediction box in the plurality of prediction boxes where the right-hand confidence score is greater than the second threshold and the left-hand confidence score is less than the first threshold, thereby obtaining a right-hand prediction box. The second labeling module is used to copy the target prediction box to obtain a first copy and a second copy, label the first copy as the left-hand prediction box, and label the second copy as the right-hand prediction box, wherein the target prediction box is a prediction box whose left-hand confidence score is greater than the first threshold and whose right-hand confidence score is greater than the second threshold; The first parsing module is used to perform feature modulation processing on a set of left-hand prediction boxes, including multiple left-hand prediction boxes, using a pose estimation model to obtain the key point coordinates corresponding to the hand region in the left-hand prediction box. This includes: cropping the image centered on the left-hand prediction box and scaling the cropped image to a preset size to obtain a first image; processing the first image using a feature extraction backbone network to obtain a first image feature map with spatial dimensions; converting the marker information in the left-hand prediction box into a first identity feature vector using an identity encoder, and adjusting the first identity feature vector to a first identity feature tensor consistent with the spatial dimension of the first image feature map through a shape change operation; generating a first channel weighting parameter and a first spatial attention parameter based on the first identity feature tensor; weighting the first image feature map by channel dimensions using the first channel weighting parameter, and modulating the first image feature map by spatial attention dimensions using the first spatial attention parameter to obtain a first target feature map; performing an upsampling operation on the first target feature map using a key point decoder to restore the spatial resolution of the first target feature map to obtain a first key point heatmap; and determining the key point coordinates corresponding to the hand region in the left-hand prediction box based on the first key point heatmap. The second parsing module is used to perform feature modulation processing on the set of right-hand prediction boxes, which includes multiple right-hand prediction boxes, using the pose estimation model to obtain the key point coordinates corresponding to the hand region in the right-hand prediction box. The labeling information in the set of left-hand prediction boxes and the set of right-hand prediction boxes is used to modulate the feature representation of the intermediate layer of the pose estimation model.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the hand detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the hand detection method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hand detection method according to any one of claims 1 to 7.

Images