Hand posture reconstruction methods, devices, systems, equipment, media, and procedures.

Abstract

This disclosure provides a method, apparatus, system, device, medium, and program product for hand pose reconstruction. The method includes: acquiring an image sequence simultaneously captured from at least two angles; determining the hand region in each frame of the image sequence; wherein the hand region of at least a portion of the images in the image sequence is determined based on the hand regions of adjacent images; identifying hand pose parameters in the hand regions; and constructing a three-dimensional hand model based on the hand pose parameters. This disclosure, based on multiple views, avoids keypoint drift and shape loss caused by occlusion of the real hand during the three-dimensional hand model construction process, and has the advantages of low computational load and short inference time. Furthermore, since the hand region of some images in the image sequence is determined based on the hand regions of their adjacent images, target recognition is not required for each frame, thereby reducing the frequency of target recognition and further improving the speed and efficiency of three-dimensional hand model construction.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional reconstruction technology, and in particular to a method, apparatus, system, device, medium and program product for reconstructing hand posture. Background Technology

[0002] In hand-object interaction applications, the construction of 3D hand models is of great significance for gesture recognition, virtual reality, and posture tracking. Single-view 3D hand models are often prone to keypoint drift and shape loss due to hand occlusion. Multi-view model construction can alleviate occlusion problems to some extent, and its applications are becoming increasingly widespread, leading to a growing number of multi-view models used for this purpose.

[0003] Currently, multi-view models used to construct multi-view 3D models are obtained by training neural networks with images acquired from different perspectives as training samples. In practical applications, images acquired from different perspectives need to be input into the multi-view model simultaneously. If the image acquisition angle in practical applications is different from the acquisition angle of the training samples, it will lead to a large error in the output results of the multi-view model. In addition, the multi-view model requires a long inference time, and its real-time performance can no longer meet the current requirements. Summary of the Invention

[0004] The technical problem to be solved by this disclosure is to overcome the above-mentioned defects in the prior art and provide a hand posture reconstruction method, apparatus, system, device, medium and program product.

[0005] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0006] Firstly, a method for reconstructing hand posture is provided, including:

[0007] Acquire an image sequence that is simultaneously captured from at least two angles, the image sequence comprising one or more frames of images;

[0008] Determine the hand region in each frame of the image sequence; wherein the hand regions of at least a portion of the images in the image sequence are determined based on the hand regions of adjacent images;

[0009] Identify hand gesture parameters in the hand region;

[0010] A three-dimensional hand model is constructed based on the hand posture parameters.

[0011] Optionally, determining the hand region in each frame of the image sequence includes:

[0012] If the difference in hand pose parameters between the current frame image and the adjacent previous target frame images is greater than or equal to a first difference threshold, target detection is performed on the current frame image to determine the hand region in the current frame image;

[0013] If the difference in hand pose parameters between the current frame image and the images of the adjacent previous target frames is less than a first difference threshold, the hand region of the current frame image is determined by scaling the target ratio based on the border size of the hand region of the adjacent previous frame image; or, the hand region of the current frame image is determined by scaling the target ratio based on the weighted result of the border size of the hand region of the images of the adjacent target frames.

[0014] Optionally, the target scale is determined according to at least one of the following: a preset scaling factor, a first dynamic scaling factor determined based on the distance change between adjacent frame images, and a second dynamic scaling factor determined based on the flexion and extension state of the fingers in the image.

[0015] Optionally, in response to the difference in hand pose parameters between the current frame image and the adjacent previous target frame images being less than a first difference threshold and reaching a preset period, target detection is performed on the current frame image to determine the hand region in the current frame image.

[0016] Optionally, before the step of identifying hand gesture parameters in the hand region, the method further includes:

[0017] The probability of a real hand being contained in the hand region is determined based on a hand correction model, and hand regions with a probability less than a probability threshold are filtered out.

[0018] Optionally, the hand pose parameters include the pixel coordinates and camera coordinates of key hand points; constructing a 3D hand model based on the hand pose parameters includes:

[0019] For images taken at the same time but from different angles, determine the pixel coordinates and camera coordinates of the key hand points in each hand region, and determine the world coordinates of the key hand points at the corresponding time based on the pixel coordinates and camera coordinates;

[0020] The three-dimensional hand model is constructed based on the world coordinates.

[0021] Optionally, the hand pose parameters are determined based on a single-view hand detection model; the single-view hand detection model includes a keypoint detection sub-model and a chirality detection sub-model; the keypoint detection sub-model is used to identify the pixel coordinates and camera coordinates of key points of the hand in the hand region, and the chirality detection sub-model is used to identify the chirality in the image;

[0022] Determine the world coordinates of the hand keypoints at the corresponding moment, including:

[0023] Match hand key points with consistent chirality from images taken at the same time but from different angles;

[0024] Based on the pixel coordinates of the matched hand keypoints and the camera coordinates, the world coordinates of the hand keypoints at the corresponding moment are determined.

[0025] Optionally, the world coordinates of the hand key points include absolute root coordinates; the hand pose reconstruction method further includes:

[0026] By combining PnP (Perspective-n-Point) and triangulation, we construct the constraint relationship between pixel coordinates, camera coordinates, and world coordinates of hand key points corresponding to the same physical location in images taken at the same time but from different angles.

[0027] The absolute root coordinates are obtained by solving based on the constraints.

[0028] Optionally, the Hungarian Algorithm (HA) is used to match all hand keypoints in images taken at the same time but from different angles to determine the hand keypoints corresponding to the same physical location in the images taken at the same time but from different angles.

[0029] Optionally, constructing a three-dimensional hand model based on the hand pose parameters includes:

[0030] The hand pose parameters are optimized, and a three-dimensional hand model is constructed based on the optimized hand pose parameters; wherein the optimization process includes at least one of the following:

[0031] For each image captured from the same angle, in response to the difference between the hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image being greater than the second difference threshold, the hand pose parameters of the hand region in the i-th frame image are replaced with the weighted result of the hand pose parameters of the hand region in the i-th frame image and the hand pose parameters of the corresponding hand region in the previous n frames.

[0032] In response to the fact that the difference between the hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is greater than the third difference threshold, the hand pose parameters of the corresponding hand region in the i-th frame image are replaced with the hand pose parameters of the hand region in the (i-1)-th frame image taken at the same angle.

[0033] In response to the fact that the difference in hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image is less than or equal to a second difference threshold and the difference in hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is less than or equal to a third difference threshold, the hand pose parameters of the hand region in the i-th frame image are smoothed compared with the hand pose parameters of the corresponding hand region in the previous n frames.

[0034] Secondly, a three-dimensional reconstruction device for hand posture is provided, comprising:

[0035] The acquisition module is used to acquire image sequences captured simultaneously from at least two angles.

[0036] A determining module is used to determine the hand region in each frame of the image sequence; wherein the hand regions of at least some images in the image sequence are determined based on the hand regions of adjacent images;

[0037] The recognition module is used to recognize hand gesture parameters in the hand region;

[0038] A construction module is used to construct a three-dimensional hand model based on the hand posture parameters.

[0039] Thirdly, a three-dimensional reconstruction system for hand posture is provided, comprising:

[0040] One or more processors, said processors being configured to implement the hand pose reconstruction method as described in any one of the first aspects.

[0041] Optionally, the hand pose reconstruction system further includes: at least two cameras, which are communicatively connected to the one or more processors, and the at least two cameras are used to simultaneously capture images of a real hand from different angles to obtain an image sequence;

[0042] And / or, the processor includes a GPU (Graphics Processing Unit).

[0043] Fourthly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the hand posture reconstruction method described in any one of the first aspects.

[0044] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the hand posture reconstruction method described in any one of the first aspects.

[0045] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the hand posture reconstruction method described in any one aspect.

[0046] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0047] The positive improvements of this disclosure are as follows: This disclosure uses a mechanism to process and merge multi-view images to construct a 3D hand model, avoiding keypoint drift and shape loss caused by occlusion of the real hand during the 3D hand model construction process. Compared to directly training a multi-view model, this disclosure requires less computation and has a shorter inference time. Furthermore, it avoids errors in model recognition caused by differences between the shooting angles of training sample images and actual application shooting angles. Moreover, the hand region in some images of the image sequence is determined based on the hand regions of their adjacent images, eliminating the need for target recognition in each frame, thereby reducing the frequency of target recognition and further improving the speed and efficiency of 3D hand model construction. Attached Figure Description

[0048] Figure 1a A schematic diagram of the architecture of a hand posture reconstruction system provided as an exemplary embodiment of this disclosure;

[0049] Figure 1b A flowchart illustrating a hand pose reconstruction method provided as an exemplary embodiment of this disclosure;

[0050] Figure 2a A flowchart of another hand pose reconstruction method provided as an exemplary embodiment of this disclosure;

[0051] Figure 2b A flowchart illustrating the determination of hand bounding box using a static mode during hand pose reconstruction, provided as an exemplary embodiment of this disclosure;

[0052] Figure 2c A flowchart illustrating the determination of hand bounding boxes using a dynamic pattern during hand pose reconstruction, provided as an exemplary embodiment of this disclosure;

[0053] Figure 2d A flowchart illustrating the determination of hand pose parameters during hand pose reconstruction, provided as an exemplary embodiment of this disclosure;

[0054] Figure 2e A flowchart illustrating the optimization of hand pose parameters during hand pose reconstruction, provided as an exemplary embodiment of this disclosure;

[0055] Figure 3 A schematic diagram illustrating the effect of estimating finger extension state, provided as an exemplary embodiment of this disclosure;

[0056] Figure 4 A schematic diagram illustrating the effect of extending the border of a finger, provided as an exemplary embodiment of this disclosure;

[0057] Figure 5 A schematic diagram illustrating an application scenario of a hand pose reconstruction method provided as an exemplary embodiment of this disclosure;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0059] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0060] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0061] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of images all comply with relevant laws and regulations and do not violate public order and good morals.

[0062] Reconstructing 3D hand models from single-view RGB images is challenging due to factors such as depth blur and occlusion. To address these issues, current research focuses on reconstructing 3D hand models using multi-view images. This approach integrates information from multiple perspectives to effectively reduce detection errors and enhance robustness and accuracy. However, multi-view methods also suffer from problems such as cross-view data association between objects, especially when multiple objects and cameras are involved, leading to high computational costs and posing challenges to real-time requirements.

[0063] To address the aforementioned issues, this disclosure provides a hand pose reconstruction system and method that effectively and robustly overcomes the occlusion problem and improves the speed and efficiency of 3D hand model construction.

[0064] Figure 1a This is a schematic diagram of the architecture of a hand pose reconstruction system provided as an exemplary embodiment of this disclosure. Figure 1aAs shown, the system includes a server 11 and several electronic devices. The electronic devices may be, for example, the camera 12 shown in the figure. The electronic devices may also be mobile phones with cameras, tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc. The embodiments disclosed herein do not limit this.

[0065] Each electronic device interacts with the server 11 via a network, which can be a wired or wireless network, such as Wi-Fi (Wireless Local Area Network) or 4G (Fourth Generation Mobile Communication Technology).

[0066] Server 11 can be a physical server or a virtual server. Server 11 can run server-side programs for a certain application to implement the relevant business functions of the application. For example, when server 11 runs a hand posture reconstruction program, it can implement the relevant business functions of hand posture reconstruction.

[0067] In the technical solution of this embodiment, the server 11 cooperates with electronic devices: several electronic devices acquire image sequences simultaneously captured from different angles, and the server 11 acquires the image sequences and performs hand pose reconstruction. The following description uses the acquisition of image sequences using a camera as an example to further illustrate the hand pose reconstruction process.

[0068] Figure 1b A flowchart of a hand pose reconstruction method provided for an exemplary embodiment of this disclosure is included, the method comprising the following steps:

[0069] Step 101: Obtain an image sequence that has been captured simultaneously from at least two angles.

[0070] In one implementation, at least two calibrated cameras are positioned around the hand to be reconstructed in 3D. Each camera is triggered to simultaneously capture images of the hand (i.e., the real hand) from different angles. A set of image sequences is obtained for each shooting angle, i.e., multi-view images are acquired. Each set of image sequences contains multiple frames of images (RGB images) taken at different times, thereby capturing the dynamic movement trajectory of the real hand.

[0071] The hand pose reconstruction method in this embodiment is applied to a server, meaning that steps 101-104 are executed by the server. In one implementation, the server directly receives the image sequence captured by the camera and uses this image sequence as the data basis for hand pose reconstruction, achieving multi-view gesture tracking. In another implementation, the camera stores the captured image sequence in a memory, and the server retrieves the image sequence from the memory as the data basis for hand pose reconstruction. The memory can be the server's built-in memory or an external memory independent of the server.

[0072] It should be noted that the number of cameras, the frame rate of the cameras, and the layout can be set according to actual needs; the number of real hands can be one or more. This disclosure does not impose any particular limitations on this.

[0073] Step 102: Determine the hand region in each frame of the image sequence; wherein the hand region of at least some images in the image sequence is determined based on the hand regions of adjacent images.

[0074] Since images captured by a camera include not only the actual hand but also the environmental background, especially when the actual hand appears in front of a skin-colored object or in a complex scene, directly estimating hand pose parameters from the complete image captured by the camera may be inaccurate and inefficient. In this embodiment, before performing step 103 to identify hand pose parameters in the hand region, the hand region in the image is first determined, that is, the region of interest (ROI) including the hand is isolated from the image. This eliminates the interference of irrelevant background information on hand pose parameter identification, greatly simplifying the task of hand pose parameter estimation.

[0075] Understandably, for each set of image sequences, the hand region of each frame within the image sequence is determined separately. The determination of the hand region in the images of each set of image sequences can be performed synchronously or asynchronously.

[0076] Since target recognition in images is time-consuming and energy-intensive, this embodiment performs target recognition on some images in the image sequence to determine the hand region, while the hand region of the remaining images is determined based on the hand regions of their neighboring images, in order to reduce the frequency of target recognition.

[0077] Step 103: Identify hand gesture parameters in the hand region.

[0078] Each frame of an image can contain one or more hand regions. When each frame contains multiple hand regions, the hand pose parameters for each region are identified separately. The identification of hand pose parameters for each hand region can be performed synchronously or asynchronously.

[0079] The hand pose parameters include the 2D coordinates (pixel coordinates) and 3D coordinates (camera coordinates) of the hand key points. The number of hand key points can be flexibly customized according to actual application needs. For example, 21 core joint points of industry standards can be selected as hand key points. These 21 core joint points fully cover the palm base, palm center, finger bases, finger joints, and fingertips.

[0080] Step 104: Construct a three-dimensional hand model based on hand posture parameters.

[0081] Understandably, since the images are captured simultaneously from at least two angles, at a single capture moment, images from at least two angles can be obtained. In step 104, a 3D hand model is constructed based on the hand gesture parameters identified from these at least two angle images. For images acquired at different times, multiple 3D hand models corresponding to each time moment can be constructed, thereby achieving linkage between the 3D hand model and the real hand, i.e., realizing multi-view gesture tracking. The 3D hand model can be represented using a mesh.

[0082] The following example, using image sequences from two angles to construct a 3D hand model, further illustrates hand pose reconstruction. (See [link to relevant documentation]). Figure 2a Step 102 is implemented using a hand detector, step 103 using a hand estimator, and step 104 using a state manager. First, images (Image A and Image B) from two synchronized and calibrated cameras (Camera A and Camera B) are input into the hand detector to detect all possible hand regions in the images, identifying and isolating these regions with bounding boxes. Next, the hand estimator estimates the original hand pose parameters within the hand regions, including the 2D coordinates (pixel coordinates) and 3D coordinates (camera coordinates) of the hand keypoints. Finally, the state manager aligns the original hand pose parameters at different times and across multiple camera views to obtain optimized hand pose parameters, thus completing the hand pose reconstruction.

[0083] In this embodiment, images simultaneously captured from at least two angles are used as the data foundation. A mechanism is employed to process and merge these multi-view images to construct a 3D hand model. This reduces keypoint drift and shape loss caused by occlusion of the real hand during the 3D hand model construction process. Furthermore, compared to directly training a multi-view model, this method has lower computational complexity and shorter inference time, and avoids significant model errors due to differences between the shooting angles of training sample images and those in the actual application scenario. Additionally, the hand region in some images of the image sequence is determined based on the hand regions of their adjacent images, eliminating the need for target recognition in each frame. This reduces the frequency of target recognition, further improving the speed and efficiency of 3D hand model construction.

[0084] In one embodiment, step 102 is implemented using a hand detector. The hand detector can be a pre-trained YOLO-World (a real-time open vocabulary object detection model), or it can be trained using image samples labeled with hand regions on YOLO-World or other neural network architectures.

[0085] The open-vocabulary features of the YOLO-World model increase the flexibility of the location estimation task, allowing for the detection of arbitrary objects with text cues. Furthermore, the YOLO-World model utilizes a reparameterizable visual-language path aggregation network to better integrate image and text features, bridging the gap between visual and text representations. By regressing bounding boxes and classification probabilities in a single step and using them as a hand detector, the YOLO-World model maintains high inference accuracy while achieving fast inference speed.

[0086] In one embodiment, for most scenarios, hand movement is slow and smooth compared to the camera's frame rate. In the i-th frame (the current frame), the hand state is close to that of the previous frame or several adjacent frames. The hand region in the previous frame or several adjacent frames can be tracked to estimate the hand region in the current frame, thereby reducing the frequency of hand detector execution. The hand state can be determined by hand pose parameters, characterized by at least one of the following: palm orientation, hand nature (also known as eccentric hand nature), hand joint position, and hand flexion / extension state. The palm orientation is defined as the vector from the wrist to the base of the middle finger, describing the hand's orientation in space. Hand nature (eccentric hand nature) is the hand type, used to distinguish whether the current hand is left or right. Hand joint position is represented by 2D pixel coordinates or 3D spatial coordinates of hand keypoints. Hand flexion / extension state describes the degree of bending and extension of the fingers / knuckles, reflecting the hand's movement pattern.

[0087] Understandably, the hand regions of the images in the image sequence are determined according to the order in which the images are acquired. The current frame image is also the image of the hand region to be determined.

[0088] The tracking mechanism is explained below. Step 102 includes the following steps:

[0089] Step 102-1: In response to the difference in hand pose parameters between the current frame image and the adjacent previous target frame images being greater than or equal to a first difference threshold, target detection is performed on the current frame image to determine the hand region in the current frame image.

[0090] The target frame number n can be set according to the actual situation. The first difference threshold can be set according to the actual situation.

[0091] If the difference in hand pose parameters between the current frame and adjacent previous target frames is greater than or equal to a first difference threshold, it indicates that the actual hand movement is fast, or that the hand states in adjacent previous target frames are contradictory, violating the assumptions of smoothness and consistency. In this case, a hand detector is used to identify the target in the previous frame to determine the hand region in the current frame. This method of determining the hand region is called the static mode.

[0092] Understandably, for images that are just acquired, such as the first frame and the second frame, since there are no adjacent previous target frames, the hand region is determined through target recognition.

[0093] Step 102-2: In response to the fact that the difference in hand pose parameters between the current frame image and the images of the adjacent previous target frames is less than a first difference threshold, the hand region of the current frame image is determined by scaling the target ratio based on the border size of the hand region of the adjacent previous frame image.

[0094] If the difference in hand pose parameters between the current frame and the images of the preceding target frames is less than a first difference threshold, it indicates that the real hand movement is slow and smooth, or that the hand poses in the images of the preceding target frames are consistent and do not violate the assumptions of smoothness and consistency. The bounding box of the current frame can then be estimated based on the hand region of the preceding frame. Using this method to determine the hand region is called the dynamic mode.

[0095] In other implementations, step 102-2 is replaced with step 102-2'.

[0096] Step 102-2': Determine the hand region of the current frame image by scaling the target ratio based on the weighted result of the border size of the hand region of the images of adjacent target frames.

[0097] In one embodiment, the difference in hand pose parameters in an image is characterized by Euclidean distance, and the first difference threshold is also the first distance threshold. For example, before identifying the i-th frame image (the image to be identified), the Euclidean distance between the hand pose parameters of the previous n frames adjacent to the i-th frame image is calculated. Taking n=2 as an example, this means calculating the Euclidean distance between the hand pose parameters of the (i-1)-th frame image and the (i-2)-th frame image. If the Euclidean distance between the hand pose parameters of the (i-1)-th frame image and the (i-2)-th frame image is less than the first distance threshold, it indicates that the actual hand movement is slow, and the difference between the hand region in the i-th frame image and the hand region in the (i-1)-th frame image should be small. The hand region in the i-th frame image can be determined based on the hand region in the (i-1)-th frame image and / or the (i-2)-th frame image, thus eliminating the need for target recognition in the i-th frame image.

[0098] In other implementations, the differences in hand pose parameters can also be represented based on at least one of the following: Manhattan distance, cosine similarity, Hamming distance, etc. Specifically, for coordinate data such as joints included in hand pose parameters, Euclidean distance and Manhattan distance can be used to represent their spatial position differences; for vector data such as direction vectors and angle features included in hand pose parameters, cosine similarity (cosine distance) can be used to represent their direction / angle differences; for the discretized state features of hand pose parameters, Hamming distance can be used to represent their discrete state differences.

[0099] In one embodiment, to ensure that the dynamic mode does not deviate in the wrong direction (e.g., inferences based on incorrect previous results may lose track of the tracked hand), and to quickly resolve situations where the hand leaves the camera's field of view or a new hand enters the camera's field of view, this embodiment periodically performs a hand detector even in dynamic mode. Specifically, in response to the difference in hand pose parameters between the current frame image and the images of the adjacent previous target frames being less than a first difference threshold and reaching a preset period, target detection is performed on the current frame image to determine the hand region in the current frame image.

[0100] The preset cycle can be set according to the actual situation.

[0101] In one embodiment, the target scale is determined empirically. The border of the previous hand region is scaled to the target scale. Taking a target scaling scale of 1.2 as an example, the border of the previous hand region is expanded outward by 1.2 to serve as the border of the current frame image. Here, the border of the previous hand region is the border of the previous frame image or a weighted result of the borders of several previous frames. If the border of the hand region is represented by a rectangle, the weighted result of the border is also the weighted result of the coordinates of the four vertices of the rectangle.

[0102] In one embodiment, the target scale is a dynamic value determined based on the actual hand movements. The target scale is determined based on at least one of the following: a preset scaling factor, a first dynamic scaling factor determined based on the distance changes between adjacent frames, and a second dynamic scaling factor determined based on the flexion and extension states of the fingers in the image. The first dynamic scaling factor is related to the global movement speed of the hand. The second dynamic scaling factor is related to the local movement speed of the hand, i.e., the flexion and extension states of the fingers.

[0103] In one embodiment, a preset scaling factor Determined based on experience.

[0104] In one embodiment, the first dynamic scaling factor Characterized by the following function:

[0105] ;

[0106] in, The norm represents the coordinate change of key points of the hand in two adjacent frames, where α and β are positive coefficients. For a fast-moving hand, a larger bounding box, based on the above function, allows for greater estimation error to catch up with the rapid movement.

[0107] In one embodiment, the finger flexion / extension state (whether the hand is open or closed) is checked based on the difference between the current frame image i and the two adjacent previous frames. Assuming the background is approximately static, the bbox content of the (i-1)th frame image is subtracted from the bbox content (pixels within the bbox) to cancel out most background values ​​to 0. The finger flexion / extension state is then determined based on the amount of non-zero values ​​near the boundary (which can be determined by a soft or hard threshold).

[0108] The second dynamic scaling factor s2 expands the border of the extended finger. Additional space is also provided for the direction of finger extension within the border. Finger joints clustered near the boundary indicate that the finger extends from that side. The second dynamic scaling factor s2 can be estimated using the average distance between the finger joints and the boundary.

[0109] Figure 3 This is a schematic diagram illustrating the effect of estimating finger extension state as an exemplary embodiment of this disclosure. See also... Figure 3 The i-1 frame image (the magenta box in the left image) is subtracted from the i-2 frame image (the magenta box in the middle image), and then the approximately static background cancellation is set to 0 (the green area in the right image). If the pixel elements in the blue area of ​​the right image are set to 0, the only non-zero pixel represents the outstretched finger (the white box in the right image). The blue area in the right image is the intersection of the bounding boxes of the two frames (the i-1 frame image and the i-2 frame image).

[0110] The above methods can help estimate whether a larger bounding box is needed or whether a smaller bounding box is sufficient. They can also provide a rough estimate of finger flexion and extension.

[0111] Figure 4 This is a schematic diagram illustrating the effect of extending the border of a finger, as provided in an exemplary embodiment of this disclosure. See also... Figure 4 In the left image, most finger joints (green dots representing key points of the finger) cluster near the top boundary, indicating finger flexion. In the next frame, space is added above them, as... Figure 4 As shown in the middle image, the finger joints are also clustered near the top edge, but slightly further away and more dispersed. Therefore, less space is added to the top of the next bbox, as shown in the right image.

[0112] In one embodiment, the product of a preset scaling factor, a first dynamic scaling factor, and a second dynamic scaling factor is determined as the target scale. The formula for the target scale s is as follows:

[0113] .

[0114] For example, left, top, right, and bottom represent the border of the hand region in the previous frame image. , , , This represents the border of the hand region in the current frame image. , , , If we represent the possible expansions in each direction, then we have:

[0115] ;

[0116] Among them, coefficient .

[0117] In one embodiment, an upper limit is set for border scaling to avoid excessive sensitivity to changes in border size, as expressed by the following formula:

[0118] ;

[0119] in, , Configure it according to your actual needs.

[0120] In one embodiment, to ensure that dynamic estimation does not deviate in the wrong direction (e.g., inferences based on incorrect previous results may lose track of the hand), and to quickly resolve situations where the hand leaves the camera's field of view or a new hand enters the camera's field of view, the hand detector is executed periodically in dynamic mode. If the hand pose in the current frame contradicts the hand pose in the previous frame, violating the assumptions of smoothness and consistency, the hand detector is also run in the next round to correct the error.

[0121] In one embodiment, before step 103, the method further includes: determining the probability that a hand region contains a real hand based on a hand correction model, and filtering out hand regions with a probability less than a probability threshold.

[0122] The output of the hand detector is input into the hand correction model to determine the probability that the hand region contains a real hand. If the probability is less than the probability threshold, it means that the hand region is not a hand region, and the bounding box output by YOLO World is likely misidentified and should be filtered out; if the probability is greater than or equal to the probability threshold, it means that the hand region contains a real hand and can be used for 3D hand model construction. The probability threshold can be set according to the actual situation.

[0123] Because hand detectors can make overconfident predictions, they may classify irrelevant objects as target objects. For example, a hand detector may fail to distinguish a real human hand from other non-hand objects whose shape and color are similar to a human hand. This can be exacerbated when detecting small objects from large images. In this embodiment, a hand correction model is used to correct the hand regions identified by the hand detector, filtering out misidentified hand regions to improve the effectiveness of 3D hand model construction.

[0124] The hand correction model can be implemented, but is not limited to, by adding an image classifier to the visual language model of MobileCLIP. In this embodiment, a list of positive prompts for describing human hands (such as "photo of a human hand", "photo of a hand with blurred edges") and a list of negative prompts for describing non-hand objects (such as "photo of a robot arm") are constructed, and the probability of each text description is calculated using an image classifier. To distinguish between hand objects (valid data) and non-hand objects (noise), the hand region in the image is compared with n positive prompts (represented by probabilities). ) and m negative prompts (representing the probability) The comparisons are performed, and binary classification is performed using the total probability.

[0125] .

[0126] MobileCLIP achieves an accuracy-latency tradeoff and maintains high performance even on edge devices with limited computing resources, meeting the real-time requirements of hand pose reconstruction.

[0127] Furthermore, when used in conjunction with the YOLO-World model, MobileCLIP can compensate for the YOLO-World model's shortcomings in detecting small objects. The strong generalization capabilities of both the YOLO-World model and MobileCLIP enable support for a wide variety of tasks and scenarios. The YOLO-World model enables fast detection to ensure efficiency, while MobileCLIP filters out false positives to ensure accuracy.

[0128] See Figure 2b The following describes the implementation method for determining the hand region in the current frame image based on static mode and by fusing MobileCLIP and YOLO-World: Obtain a new image (i.e., the current frame image), first call YOLO-World to identify and locate the hand region in the new image to obtain the initial bounding boxes in the new image, then call MobileCLIP to filter the initial bounding boxes, filtering out non-hand bounding boxes, and finally determining the hand bounding box corresponding to the current frame image.

[0129] See Figure 2c The following describes the implementation method for determining the hand region in the current frame image based on dynamic mode: The initial bounding boxes (bboxes) of the current frame image are determined based on the hand key points of the preceding n frames. Then, it is determined whether the hand movement speed is too fast and whether the flexion / extension state is the same based on the preceding n frames. If the hand movement speed is not too fast and the flexion / extension state is the same, the initial bboxes are scaled to obtain the hand bounding boxes corresponding to the current frame image. Otherwise, the bboxes of the current frame image are determined based on static mode. It should be noted that the execution order of the two judgment conditions—hand movement speed and flexion / extension state—is not limited to the order shown in the figure (first judging hand movement speed, then judging flexion / extension state). In other implementations, flexion / extension state can be judged first, then hand movement speed, or both can be executed simultaneously.

[0130] In one embodiment, TensorRT optimization improves model inference speed by 2-3 times, enabling real-time inference capabilities on conventional devices.

[0131] In one embodiment, the hand pose parameters include the pixel coordinates and camera coordinates of hand key points. Pixel coordinates are the two-dimensional coordinates of the hand key points in the image coordinate system, and camera coordinates are the three-dimensional coordinates of the hand key points in the camera coordinate system. Step 104 includes:

[0132] Step 104-1: Determine the world coordinates of the hand key points at the corresponding time based on the pixel coordinates and camera coordinates of the hand key points in images taken at different angles at the same time.

[0133] Step 104-2: Construct a three-dimensional hand model based on world coordinates.

[0134] In one embodiment, hand keypoints are determined based on a single-view hand detection model. The input to the single-view hand detection model is a single-view image (i.e., an image taken by each camera at a different angle), and the output of the single-view hand detection model is the pixel coordinates and camera coordinates of the hand keypoints.

[0135] In this embodiment, the advantages of high efficiency and strong generalization ability of the single-view hand detection model are utilized. The single-view hand detection model is used to process images from different cameras, which can improve the real-time performance, robustness and deployment flexibility of hand pose reconstruction.

[0136] In one embodiment, a lightweight version of the SimpleHand model is used as the base model for the 3D hand modeling task. SimpleHand achieves excellent accuracy while maintaining high inference speed, estimating the pixel coordinates and camera coordinates of hand keypoints from images in real time.

[0137] In one embodiment, TensorRT optimization and memory caching further optimize the inference speed of the single-view hand detection model by approximately 2x.

[0138] Chirality (left or right hand) is another important parameter required for constructing a 3D hand model. In one embodiment, a chirality detection model is trained based on a one-dimensional convolutional neural network (1D CNN). This model leverages its lightweight and efficient handling of sequential data to quickly distinguish between left and right hands, facilitating subsequent registration of hands in images from different viewpoints.

[0139] In one embodiment, a chirality detection model is trained on the FreiHAND dataset using flattened MANO-style 3D vertices as input. The optimal checkpoint achieves perfect accuracy on the FreiHAND evaluation dataset, with a size of only 111 KB.

[0140] In one embodiment, chirality detection is integrated into the post-processing stage of keypoint detection. The single-view hand detection model includes a keypoint detection sub-model and a chirality detection sub-model. The keypoint detection sub-model is used to identify the pixel coordinates and camera coordinates of hand keypoints in the image, and the chirality detection sub-model is used to identify chirality in the image.

[0141] In this embodiment, chirality detection is integrated into the post-processing stage of keypoint detection. Both keypoint detection and chirality recognition are implemented on the GPU, which can reduce CPU-GPU data transfer.

[0142] In step 104-1, determining the world coordinates of the hand key points at the corresponding time includes: matching hand key points with consistent chirality from images taken at the same time but from different angles, and determining the world coordinates of the hand key points at the corresponding time based on the pixel coordinates and camera coordinates of the matched hand key points.

[0143] For example, images A and B are taken at the same time from two different angles, depicting a pair of hands. Both images A and B contain a left hand and a right hand. To determine the world coordinates, a chirality detection model is used to identify the chirality of the hands in images A and B respectively. The left hand in image A is matched with the left hand in image B, and the world coordinates of the left hand's keypoints are determined based on the pixel coordinates and camera coordinates of the keypoints in both images. Similarly, the right hand in image A is matched with the right hand in image B, and the world coordinates of the right hand's keypoints are determined based on the pixel coordinates and camera coordinates of the keypoints in both images.

[0144] In one embodiment, the world coordinates of hand keypoints include absolute root coordinates, which are determined jointly by Perspective-n-Point (PnP) and triangulation. Specifically: by combining PnP and triangulation, constraints are constructed between the pixel coordinates, camera coordinates, and world coordinates of hand keypoints corresponding to the same physical location in images taken at the same time but from different angles. The absolute root coordinates are then obtained based on these constraints, enabling a root recovery strategy compatible with both single-view and multi-view tasks. The constraint relationships are represented as follows:

[0145] ;

[0146] in, Represents the pixel coordinates of key points on the hand. The root-relative camera coordinates representing key points of the hand. World coordinates representing key hand points The root coordinates represent the absolute root coordinates in camera space, where K represents the internal camera matrix, R represents the external rotation matrix, and T represents the external translation vector. Root-relative camera coordinates refer to the relative positions of each hand keypoint within a camera coordinate system established with the root joint as the origin. Root-relative camera coordinates are used because hand detection models based on RGB images (two-dimensional images) typically only provide root-relative coordinates, as two-dimensional images do not contain depth information.

[0147] In this embodiment, one of the key points of the hand, such as the wrist or the base of the middle finger, is taken as the root joint, and the positions of all other key points of the hand relative to the root joint are estimated. If the actual coordinates of the root joint can be obtained, then the actual coordinates of all hand key points (root recovery) can be calculated based on the root coordinates and the coordinates relative to the root joint. Understandably, each hand has one root joint, and if there are two hands in an image, then each hand has its own root joint (the two hands are calculated independently and cannot share a single root joint).

[0148] In this embodiment, PnP and triangulation methods are used in combination to match hand projections and recover absolute root coordinates, so as to make full use of the 2D-3D and 3D-3D information conveyed in 2D and 3D hand poses.

[0149] For single-view tasks, the absolute root coordinates are obtained using the first equation relating pixel coordinates and camera coordinates. The absolute root coordinates can also be obtained by solving an equation that incorporates information from any two hand keypoints. For multi-view, multi-hand estimation, multi-view methods cannot be directly applied. Figure 3The angle measurement method (second equation) is used because hand poses detected from different camera views have not yet been matched. In this embodiment, to correlate these hand poses, the absolute root coordinates in the corresponding individual views are first estimated to coarsely recover the world coordinates of the hand keypoints. Then, the hand poses are aligned grouped across views using the coarse world coordinates of the hand keypoints. Finally, multi-view constraints are applied to those hands registered from multiple views to recalculate the absolute root coordinates, thereby integrating information from different camera views. In the final step, the results are obtained by jointly solving the first and second equations to utilize 2D-3D (pixel coordinates and camera coordinates) and 3D-3D (camera coordinates and world coordinates) relationships and achieve better robustness and accuracy.

[0150] In one embodiment, considering the limited number of hand keypoints (taking 21 hand keypoints as an example), a more direct and concise exhaustive method is used instead of an algorithm like RANSAC (Random Sample Consensus) to find the optimal absolute root coordinates. In this embodiment, the equation coefficients can be placed in a tensor and solved using a linear algebra solver.

[0151] In single-view root recovery, constraints from the root (for ease of calculation) and all other hand keypoints are sequentially combined to construct a system of equations:

[0152] ;

[0153] The first equation in the above system of equations describes the root joint. , , The offset is zero.

[0154] in, The root relative camera coordinates represent different hand key points. and These represent the pixel coordinates of the root and different hand keypoints, respectively. The solution is replaced back into the equation, and the difference between the left and right sides of the equation is calculated (the left and right sides of the equation may no longer be equal due to errors in pixel coordinate and camera coordinate estimation). The solution with the smallest error is determined as the final solution.

[0155] In multi-view root recovery, the subscripts 0, 1, ..., m-1 are used to represent the view index, and the joint system of equations is given by the following:

[0156] ;

[0157] Here, m represents the number of cameras / viewpoints. Each hand keypoint has a set of root relative coordinates in each viewpoint. For m viewpoints, there are m 2D-2D equations and m 3D-3D equations, for a total of m*2 equations. This set of equations lists all m*2 equations related to one of the hand keypoints.

[0158] Taking 21 points representing the key points of a hand as an example, 21 such systems of equations can be formulated for all 21 key points, yielding 21 solutions. Each solution is replaced with the system of equations for all joints, and the error, caused by inconsistencies between the left and right sides of the equations, is calculated. The solution with the minimum error is selected to determine the absolute root coordinates.

[0159] Before determining the absolute root coordinates, a multi-view hand matching task needs to be performed, which involves registering hand keypoints representing the same physical location in the multi-view images and grouping them into a single hand group. World coordinates are then determined based on the hand keypoints within the same hand group.

[0160] In one embodiment, the Hungarian Algorithm (HA) is used in the multi-view hand matching task, reducing the worst-case time complexity from... Optimized to This method efficiently solves the problem of "one-to-one matching of hands in multiple views." Specifically, it uses the Hungarian algorithm to match all hand keypoints in images taken at the same time but from different angles to determine the hand keypoints corresponding to the same physical location in the images taken at the same time but from different angles. In multi-view hand matching, the elements of the cost matrix are used to quantify the matching cost between different hands in the multi-view hand matching task. Specifically, the matching cost between the "current hand to be assigned" and the "hand group" is quantified by distance.

[0161] Hand Group: This refers to the set of observations in different views that are identified as the "same physical hand". For example, if hand A is detected in view A and hand B is detected in view B, and they are grouped together by a matching algorithm, then the group contains hand A and hand B, representing the same hand in reality.

[0162] The hand-to-hand distance is defined as the average of all hand-to-hand distances:

[0163] ;

[0164] Where, d i It is the distance between the "current hand to be assigned" and the i-th hand from the existing hand group consisting of n hands.

[0165] To calculate hand-to-hand distances, the coordinates of hand keypoints within a hand group are transformed to the same coordinate system to calculate the distances between keypoints representing different hands. Taking a hand group containing two hands as an example, the coordinates of one hand are transformed to the camera space of the other hand to calculate the distance between the hand keypoints of the two hands. The same operation is then repeated for the second hand. Next, the distances from the two transformation directions are averaged.

[0166] ;

[0167] In this definition, uppercase letters represent world coordinates, lowercase letters represent camera coordinates, and different camera spaces are distinguished by apostrophes. The order of coordinate transformation is from one camera space to world space, and then to another camera space. Due to estimation errors, this definition is robust when two hands are close together in one camera view but far apart in another.

[0168] The hands estimated from the multi-view images are sequentially assigned to hand groups, and the match with the minimum total cost is taken as the final solution.

[0169] In one embodiment, after achieving optimal allocation using the Hungarian algorithm, intra-group information is then fused using a weighted average (and inconsistent samples are penalized), ultimately improving the accuracy and robustness of hand estimation.

[0170] In one embodiment, hand keypoints are also managed for state. The hand state is represented by hand pose parameters, which include at least one of the following: two-dimensional coordinates and three-dimensional coordinates of the hand keypoints, hand position, and finger flexion / extension state. The consistency of the hand state is evaluated by comparing it with the hand state in the current frame image with the hand state in images from the same camera several previous target frames (temporal comparison) and the hand state in the current frame image from different cameras (spatial comparison). If the consistency assumption is not met, the hand pose parameters are optimized, and a three-dimensional hand model is constructed based on the optimized hand pose parameters to improve the robustness of hand pose reconstruction.

[0171] See Figure 2dThe process of hand pose reconstruction is further explained as follows: The simpleHand model is used to estimate the 2D pixel coordinates of the hand joints, as well as the 3D root relative coordinates of the hand joints and vertices in camera space. The HandTypeClassifer classification model is then used to determine the hand's chirality, i.e., whether it is a left or right hand. Next, single-hand root node recovery is performed, estimating the absolute root coordinates in a single view. Then, it is determined whether it is a single-view pose reconstruction scene. If yes, the result is directly output. If no, multi-view registration and multi-hand root node recovery are performed, and the hands from different viewpoints are fused. The absolute root coordinates are recalculated, and the final result is output. The output includes the hand's 2D joints, 3D joints, 3D vertices, and chirality.

[0172] See Figure 2e Hand gesture parameter optimization processing includes at least one of the following:

[0173] (1) Time-based optimization. For each image captured from the same angle, in response to the difference in hand pose parameters of the matching hand region between the i-th frame and the (i-1)-th frame being greater than a second difference threshold, the hand pose parameters of the hand region in the i-th frame are replaced with the weighted result of the hand pose parameters of the hand region in the i-th frame and the hand pose parameters of the corresponding hand regions in the previous n frames. That is, if the hand state in the current frame conflicts with the previous state, the weighted average result of the hand pose parameters in the previous n frames is used to correct the hand pose parameters of the i-th frame, so that the hand state in the current frame is adjusted towards the previous reliable state. Figure 2e The example uses n=1, but in practice, n can be set according to the requirements.

[0174] (2) Spatial Comparison-Based Optimization. In response to a difference greater than a third difference threshold between the hand pose parameters of the matching hand region in the i-th frame image and an image taken at the same time but from a different angle, the hand pose parameters of the corresponding hand region in the i-th frame image are replaced with those from the (i-1)-th frame image taken at the same angle. That is, if the hand state in the first-view image conflicts with the hand state in the second-view image—for example, if the difference in the shape of the two hands exceeds a threshold—then the hand pose parameters of the i-th frame image in the first-view image are replaced with those from the (i-1)-th frame image in the first-view image, and the hand pose parameters of the i-th frame image in the second-view image are replaced with those from the (i-1)-th frame image in the second-view image, effectively replacing them with the previous consistent state.

[0175] (3) In response to the fact that the difference in hand pose parameters of the matching hand regions in the i-th frame image and the (i-1)-th frame image is less than or equal to a second difference threshold, and the difference in hand pose parameters of the matching hand regions in the i-th frame image and the image taken at the same time but from a different angle is less than or equal to a third difference threshold, the hand pose parameters of the hand regions in the i-th frame image are smoothed compared to the hand pose parameters of the corresponding hand regions in the previous n frames. That is, for images where the hand state is consistent in both time and space, the hand pose parameters are smoothed. Specifically, the coordinates are smoothed by weighted averaging with the coordinates of the previous state. Represented as newly estimated coordinates (camera coordinates and / or pixel coordinates) from the current frame. Represented as coordinates from the previous frame. The new coordinates are represented as the norm of the coordinate change between the current frame and the previous frame, and then smoothed out by... Given, among which It is a positive coefficient. For large changes... The new coordinates preserve the estimated results; for small changes

[0176] The new coordinates are approximately equal to the previous coordinates. Small changes are eliminated, while large changes are preserved. In other implementations, this mechanism is also applied to smooth the border of the hand area.

[0177] In this embodiment, self-conflicts and cross-view conflicts are monitored in real time to enhance robustness and ensure consistency of hand pose parameters from previous frames and other camera images. Consistency is used to determine whether hand tracking is still effective, whether to clear the tracking history, and whether to run the hand detector in the next frame.

[0178] In one embodiment, all models involved in this embodiment run on the GPU to achieve real-time performance, and absolute root coordinate calculation is also supported and executed on the GPU. Some tracking mechanisms, such as determining the dynamic pattern of the hand region, Hungarian algorithm matching, and state management, are mainly executed on the CPU to reduce the burden on the GPU.

[0179] For detailed descriptions of YOLO-World, MobileCLIP, and SimpleHand used in the above embodiments, please refer to the relevant technical descriptions, which will not be repeated here.

[0180] To evaluate performance, the hand pose reconstruction method of this embodiment was tested using images captured from the robot environment and the FreiHAND evaluation dataset. In the table below, FPS (frames per second) is calculated using the total runtime of the entire pipeline, including all components: the hand detector, the hand estimator, and the state manager. For multi-view datasets, multi-view frames are used to measure the frame rate (e.g., for two-view data, 1 frame = 2 images). Input images are passed to the pipeline as a real-time stream, with a batch size set to 1 (single-view / multi-view frames).

[0181] The runtime for the different types of data is shown in Table 1. The data consists of 1280×720 resolution images captured by a camera. The tests were conducted on a mobile GPU, specifically an NVIDIA GeForce RTX4060 Mobile.

[0182] Table 1

[0183] Dataset model FPS Single view, single hand Static mode 45 Single view, single hand Dynamic mode 82 Dual view, one hand Static mode 28 Dual view, one hand Dynamic mode 54 Dual view with both hands Static mode 26 Dual view with both hands Dynamic mode 34

[0184] Table 1 shows the runtime of the multi-view real-time hand pose reconstruction pipeline of this embodiment, measured on a mobile GPU. As can be seen from the table, the processing speed of the dynamic mode is greater than that of the static mode, exceeding the real-time threshold (24 to 30 FPS). In this embodiment, the frame rate for dual-view hand pose reconstruction using an NVIDIA GeForce RTX 4090 GPU was 61 FPS, significantly faster than that of the mobile GPU.

[0185] Table 2 shows the comparison of the running time of the method provided in this embodiment with existing SOTA (State of The Art) methods, including MobRecon, SimpleHand, H2ONet (single-frame input version), SiMA-Hand, and MLPHand.

[0186] Table 2

[0187] method FPS Test device MobRecon 80 NVIDIA RTX 2080 Ti SimpleHand 70 NVIDIA RTX 2080 Ti H2ONet 52 NVIDIA RTX 2080 Ti SiMA-Hand 48 NVIDIA RTX 2080 Ti MLPHand 71 NVIDIA RTX 3090 The method in this embodiment (static mode) 53 NVIDIA RTX 4060 Mobile The method in this embodiment (dynamic mode) 70 NVIDIA RTX 4060 Mobile

[0188] As shown in Table 2, the method provided in this implementation achieves comparable or better efficiency compared to the entire pipeline (including hand detection, hand joint estimation, and state management) and the state-of-the-art (SOTA) model (which only includes the mesh reconstruction (3D hand joint estimation) component).

[0189] Next, images with information loss are used to evaluate the robustness of the method in this embodiment. Figure 5The image shows examples of using single-view pose reconstruction and multi-view pose reconstruction (using a dual-view example) to handle detection failure, truncation, and occlusion. In the left column, images from two cameras are processed as single-view inputs without any multi-view information association. The single-view mechanism fails to address challenging cases when information loss is severe; see the areas marked with red circles. In the right column, the hand is successfully reconstructed using multiple camera views to supplement missing or incorrect information.

[0190] Corresponding to the aforementioned hand posture reconstruction method embodiments, this disclosure also provides embodiments of hand posture reconstruction devices.

[0191] The device includes:

[0192] The acquisition module is used to acquire image sequences captured simultaneously from at least two angles.

[0193] A determining module is used to determine the hand region in each frame of the image sequence; wherein the hand regions of at least some images in the image sequence are determined based on the hand regions of adjacent images;

[0194] The recognition module is used to recognize hand gesture parameters in the hand region;

[0195] A construction module is used to construct a three-dimensional hand model based on the hand posture parameters.

[0196] The device in this embodiment is highly modular, with each module being replaceable and easily upgradable to more effective models for different real-world application scenarios.

[0197] In one embodiment, the determining module is specifically used for:

[0198] If the difference in hand pose parameters between the current frame image and the adjacent previous target frame images is greater than or equal to a first difference threshold, target detection is performed on the current frame image to determine the hand region in the current frame image;

[0199] If the difference in hand pose parameters between the current frame image and the images of the adjacent previous target frames is less than a first difference threshold, the hand region of the current frame image is determined by scaling the target ratio based on the border size of the hand region of the adjacent previous frame image; or, the hand region of the current frame image is determined by scaling the target ratio based on the weighted result of the border size of the hand region of the images of the adjacent target frames.

[0200] In one embodiment, the target scale is determined according to at least one of the following: a preset scaling factor, a first dynamic scaling factor determined based on the distance change between adjacent frame images, and a second dynamic scaling factor determined based on the flexion and extension state of the fingers in the image.

[0201] In one embodiment, in response to the difference in hand pose parameters between the current frame image and the adjacent previous target frame images being less than a first difference threshold and reaching a preset period, target detection is performed on the current frame image to determine the hand region in the current frame image.

[0202] In one embodiment, it also includes:

[0203] The correction module is used to determine the probability that the hand region contains a real hand based on the hand correction model, and to filter out hand regions with a probability less than a probability threshold.

[0204] In one embodiment, the hand pose parameters include pixel coordinates and camera coordinates of hand key points; the building module has features for:

[0205] For images taken at the same time but from different angles, determine the pixel coordinates and camera coordinates of the key hand points in each hand region, and determine the world coordinates of the key hand points at the corresponding time based on the pixel coordinates and camera coordinates;

[0206] The three-dimensional hand model is constructed based on the world coordinates.

[0207] In one embodiment, the hand pose parameters are determined based on a single-view hand detection model; the single-view hand detection model includes a keypoint detection sub-model and a chirality detection sub-model; the keypoint detection sub-model is used to identify the pixel coordinates and camera coordinates of key points of the hand in the hand region, and the chirality detection sub-model is used to identify the chirality in the image;

[0208] When determining the world coordinates of the hand keypoints at a given moment, the construction module is specifically used for:

[0209] Match hand key points with consistent chirality from images taken at the same time but from different angles;

[0210] Based on the pixel coordinates of the matched hand keypoints and the camera coordinates, the world coordinates of the hand keypoints at the corresponding moment are determined.

[0211] In one embodiment, the world coordinates of the hand keypoints include absolute root coordinates; the device further includes:

[0212] The constraint module is used to combine PnP and triangulation to construct the constraint relationship between the pixel coordinates, camera coordinates and world coordinates of the hand key points corresponding to the same physical location in images taken at the same time but from different angles; and to solve for the absolute root coordinates based on the constraint relationship.

[0213] In one embodiment, the Hungarian algorithm is used to match all hand keypoints in images taken at the same time but from different angles to determine the hand keypoints corresponding to the same physical location in the images taken at the same time but from different angles.

[0214] In one embodiment, the building module is specifically used for:

[0215] The hand pose parameters are optimized, and a three-dimensional hand model is constructed based on the optimized hand pose parameters; wherein the optimization process includes at least one of the following:

[0216] For each image captured from the same angle, in response to the difference between the hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image being greater than the second difference threshold, the hand pose parameters of the hand region in the i-th frame image are replaced with the weighted result of the hand pose parameters of the hand region in the i-th frame image and the hand pose parameters of the corresponding hand region in the previous n frames.

[0217] In response to the fact that the difference between the hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is greater than the third difference threshold, the hand pose parameters of the corresponding hand region in the i-th frame image are replaced with the hand pose parameters of the hand region in the (i-1)-th frame image taken at the same angle.

[0218] In response to the fact that the difference in hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image is less than or equal to a second difference threshold and the difference in hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is less than or equal to a third difference threshold, the hand pose parameters of the hand region in the i-th frame image are smoothed compared with the hand pose parameters of the corresponding hand region in the previous n frames.

[0219] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0220] This disclosure also provides a hand posture reconstruction system, including:

[0221] One or more processors, the processors being used to implement the hand pose reconstruction method provided in any of the above embodiments.

[0222] Optionally, the hand pose reconstruction system also includes at least two cameras that are communicatively connected to the processor. These cameras are used to simultaneously capture images of the real hand from different angles to obtain an image sequence.

[0223] In one embodiment, the processor is a GPU, which implements the hand pose reconstruction method provided in any of the above embodiments.

[0224] In one embodiment, the processor is a CPU and a GPU, which work together to implement the hand pose reconstruction method provided in any of the above embodiments. For example, all models involved in the above embodiments run on the GPU to achieve real-time performance, and absolute root coordinate calculation is also supported and executed on the GPU. Some tracking mechanisms, such as determining the dynamic pattern of the hand region, Hungarian algorithm matching, and state management, are mainly executed on the CPU to reduce the burden on the GPU.

[0225] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the hand posture reconstruction method described in any of the above embodiments. Figure 6 The electronic device 60 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0226] like Figure 6 As shown, the electronic device 60 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0227] Bus 63 includes a data bus, an address bus, and a control bus.

[0228] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0229] The memory 62 may also include a program tool 625 (or utility) having a set (at least one) program module 624, such program module 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0230] The processor 61 executes various functional applications and data processing by running computer programs stored in the memory 62, such as the hand posture reconstruction method provided in any of the above embodiments.

[0231] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 65. Furthermore, electronic device 60 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of electronic device 60 via bus 63. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0232] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0233] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hand posture reconstruction method provided in any of the above embodiments.

[0234] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0235] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the hand posture reconstruction method described in any of the above embodiments.

[0236] This disclosure also provides a computer program that, when executed by a processor, implements the hand posture reconstruction method described in any of the preceding embodiments.

[0237] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0238] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for reconstructing hand posture, characterized in that, include: Acquire an image sequence that is simultaneously captured from at least two angles, the image sequence comprising one or more frames of images; Determine the hand region in each frame of the image sequence; wherein the hand regions of at least a portion of the images in the image sequence are determined based on the hand regions of adjacent images; Identify hand pose parameters in the hand region; the hand pose parameters include pixel coordinates of hand key points and / or camera coordinates; A three-dimensional hand model is constructed based on the hand posture parameters.

2. The hand posture reconstruction method according to claim 1, characterized in that, Determining the hand region in each frame of the image sequence includes: If the difference in hand pose parameters between the current frame image and the adjacent previous target frame images is greater than or equal to a first difference threshold, target detection is performed on the current frame image to determine the hand region in the current frame image; If the difference in hand pose parameters between the current frame image and the images of the adjacent previous target frames is less than a first difference threshold, the hand region of the current frame image is determined by scaling the target ratio based on the border size of the hand region of the adjacent previous frame image; or, the hand region of the current frame image is determined by scaling the target ratio based on the weighted result of the border size of the hand region of the images of the adjacent target frames.

3. The hand posture reconstruction method according to claim 2, characterized in that, The target scale is determined according to at least one of the following: a preset scaling factor, a first dynamic scaling factor determined based on the distance change between adjacent frame images, and a second dynamic scaling factor determined based on the flexion and extension state of the fingers in the image.

4. The hand posture reconstruction method according to claim 2, characterized in that, In response to the difference in hand pose parameters between the current frame image and the adjacent previous target frame images being less than a first difference threshold and reaching a preset period, target detection is performed on the current frame image to determine the hand region in the current frame image.

5. The hand posture reconstruction method according to claim 1, characterized in that, Prior to the step of identifying hand gesture parameters in the hand region, the method further includes: The probability of a real hand being contained in the hand region is determined based on a hand correction model, and hand regions with a probability less than a probability threshold are filtered out.

6. The hand posture reconstruction method according to any one of claims 1-5, characterized in that, Constructing a three-dimensional hand model based on the hand posture parameters includes: For images taken at the same time but from different angles, determine the pixel coordinates and camera coordinates of the key hand points in each hand region, and determine the world coordinates of the key hand points at the corresponding time based on the pixel coordinates and camera coordinates; The three-dimensional hand model is constructed based on the world coordinates.

7. The hand posture reconstruction method according to claim 6, characterized in that, The hand pose parameters are determined based on a single-view hand detection model; the single-view hand detection model includes a key point detection sub-model and a chirality detection sub-model; the key point detection sub-model is used to identify the pixel coordinates and camera coordinates of key points of the hand in the hand region, and the chirality detection sub-model is used to identify the chirality in the image; Determine the world coordinates of the hand keypoints at the corresponding moment, including: Match hand key points with consistent chirality from images taken at the same time but from different angles; Based on the pixel coordinates of the matched hand keypoints and the camera coordinates, the world coordinates of the hand keypoints at the corresponding moment are determined.

8. The hand posture reconstruction method according to claim 6, characterized in that, The world coordinates of the hand key points include absolute root coordinates; the hand pose reconstruction method also includes: By combining PnP and triangulation, we construct the constraint relationship between pixel coordinates, camera coordinates and world coordinates of hand key points corresponding to the same physical location in images taken at the same time but from different angles. The absolute root coordinates are obtained by solving based on the constraints.

9. The hand posture reconstruction method according to claim 8, characterized in that, The Hungarian algorithm is used to match all hand key points in images taken at the same time but from different angles to determine the hand key points corresponding to the same physical location in the images taken at the same time but from different angles.

10. The hand posture reconstruction method according to any one of claims 1-5 and 7-9, characterized in that, Constructing a three-dimensional hand model based on the hand posture parameters includes: The hand pose parameters are optimized, and a three-dimensional hand model is constructed based on the optimized hand pose parameters; wherein the optimization process includes at least one of the following: For each image captured from the same angle, in response to the difference between the hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image being greater than the second difference threshold, the hand pose parameters of the hand region in the i-th frame image are replaced with the weighted result of the hand pose parameters of the hand region in the i-th frame image and the hand pose parameters of the corresponding hand region in the previous n frames. In response to the fact that the difference between the hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is greater than the third difference threshold, the hand pose parameters of the corresponding hand region in the i-th frame image are replaced with the hand pose parameters of the hand region in the (i-1)-th frame image taken at the same angle. In response to the fact that the difference in hand pose parameters of the matching hand region in the i-th frame image and the (i-1)-th frame image is less than or equal to a second difference threshold and the difference in hand pose parameters of the matching hand region in the i-th frame image and the image taken at the same time but from a different angle is less than or equal to a third difference threshold, the hand pose parameters of the hand region in the i-th frame image are smoothed compared with the hand pose parameters of the corresponding hand region in the previous n frames.

11. A hand posture reconstruction device, characterized in that, include: The acquisition module is used to acquire image sequences captured simultaneously from at least two angles. A determining module is used to determine the hand region in each frame of the image sequence; wherein the hand regions of at least some images in the image sequence are determined based on the hand regions of adjacent images; The recognition module is used to recognize hand gesture parameters in the hand region; A construction module is used to construct a three-dimensional hand model based on the hand posture parameters.

12. A hand posture reconstruction system, characterized in that, include: One or more processors, said processors being configured to implement the hand pose reconstruction method according to any one of claims 1 to 10.

13. The hand posture reconstruction system according to claim 12, characterized in that, The hand pose reconstruction system further includes: at least two cameras, which are communicatively connected to the one or more processors, and the at least two cameras are used to simultaneously capture images of a real hand from different angles to obtain an image sequence; And / or, the processor includes a GPU.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the hand posture reconstruction method according to any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hand posture reconstruction method according to any one of claims 1 to 10.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hand posture reconstruction method as described in any one of claims 1 to 10.

Images