A hand three-dimensional reconstruction method, device and reconstruction network training method

Abstract

The application provides a hand three-dimensional reconstruction method and device and a reconstruction network training method. Based on camera internal parameters, a color image and an original depth image aligned with the color image in a pixel coordinate system, initial hand parameterized model parameters are obtained. According to the camera internal parameters, an initial hand three-dimensional grid represented by the initial hand parameterized model parameters is projected into a two-dimensional image corresponding to the color image, a model depth image and a model visibility image having the same pixel coordinate system as the original depth image are generated. Based on the original depth image, the model depth image and the model visibility image, pixels in a hand region are divided into a plurality of pixel partitions. A hybrid depth image is generated based on the plurality of pixel partitions. The initial hand parameterized model parameters are corrected based on the color image and the hybrid depth image, and target hand parameterized model parameters are obtained. The application improves the robustness and spatial positioning accuracy of a hand reconstruction algorithm in a complex scene.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, and in particular to a method, device and reconstruction network training method for 3D reconstruction of a hand. Background Technology

[0002] In applications such as interactive tracking, virtual-real fusion, and robot teleoperation, how to stably and accurately reproduce the three-dimensional pose and geometry of a hand using image data acquired by a depth camera is a key technical requirement we currently face.

[0003] To meet these requirements, existing hand 3D reconstruction schemes typically input the extracted color and depth image information into a network model for single-stage parameter regression; or convert the depth map into a point cloud and then perform multimodal fusion with image features to output hand pose parameters or mesh shape. In parameter optimization or result refinement, the conventional strategy is to treat the entire original depth map or the entire hand observation area as a unified object for global feature extraction and processing.

[0004] However, existing reconstruction schemes have significant limitations in practical applications. Limited by the operating characteristics of consumer-grade depth sensors, the raw depth images acquired in complex scenarios such as rapid hand movements, partial occlusion, reflective materials, and close-up photography are prone to depth holes, boundary spikes, flying point noise, and local depth distortion. Because existing schemes typically use an architecture that treats the entire depth map as a unified input for feature extraction, it is difficult to effectively identify and isolate the inherent differences in observation reliability within the depth data itself in the early stages. This makes the feature extraction process highly susceptible to direct interference from the aforementioned unreliable depth noise. This processing mode, which fails to effectively distinguish between reliable and unreliable data observations, allows local erroneous observation information to propagate uncontrollably and affect the global parameter estimation process, ultimately resulting in severely insufficient robustness of hand reconstruction in complex scenes and a high susceptibility to parameter drift. Summary of the Invention

[0005] This application provides a method, apparatus, and reconstruction network training method for three-dimensional hand reconstruction, aiming to solve the problem that existing hand reconstruction schemes fail to effectively distinguish between reliable and unreliable data observation processing modes, resulting in insufficient robustness of hand reconstruction in complex scenarios and easy parameter drift.

[0006] In a first aspect, this application provides a method for three-dimensional reconstruction of a hand, comprising: Based on camera intrinsics, color images, and original depth images aligned with the color images in pixel coordinates, initial hand parameterization model parameters are obtained. Based on the camera intrinsic parameters, the initial 3D mesh of the hand, represented by the initial parametric model parameters, is projected onto the 2D image corresponding to the color image to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial 3D mesh of the hand relative to the current camera at each pixel position; Based on the original depth image, the model depth map, and the model visibility map, the pixels to be processed in the hand region are divided into multiple pixel partitions; wherein, the multiple pixel partitions include a retention area, a replacement area, and a conflict area; A mixed depth map is generated based on the multiple pixel partitions; The initial hand parametric model parameters are corrected based on the color image and the mixed depth map to obtain the target hand parametric model parameters. The hand is then reconstructed based on the target hand parametric model parameters to obtain the position of the 3D hand mesh in the camera coordinate system.

[0007] As one embodiment, the step of projecting the initial 3D mesh of the hand, represented by the initial hand parameterized model parameters, onto the 2D image corresponding to the color image based on the camera intrinsic parameters, to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image, includes: Based on the camera intrinsic parameters, multiple mesh patches in the initial three-dimensional mesh of the hand are projected onto the two-dimensional image coordinate system corresponding to the color image to obtain the projection area of ​​each mesh patch; The projected grid patches are rasterized to determine the pixel positions covered by each grid patch, and the candidate model depth value is calculated for the pixel positions covered by the grid patches. When multiple candidate model depth values ​​correspond to the same pixel location, the candidate model depth value closest to the current camera optical center is selected by using a depth buffer method as the model depth value of the pixel location to generate the model depth map; Based on the depth test results of each mesh patch at the corresponding pixel position, and the relationship between the normal of each mesh patch and the current camera viewing direction, the visibility at each pixel position is determined to generate the model visibility map.

[0008] As one embodiment, the step of dividing the pixels to be processed within the hand region into multiple pixel partitions based on the original depth image, the model depth map, and the model visibility map includes: The pixels with valid original depth values ​​in the original depth image and the pixels with valid model depth values ​​in the model depth image and whose visibility in the model visibility image reaches a preset visibility threshold are identified as the pixels to be processed. Based on the original depth image, the model depth map, and the model visibility map, multiple evaluation parameters are determined for each pixel to be processed; Based on the plurality of evaluation parameters corresponding to each pixel to be processed, the pixel to be processed is divided into the reserved area, the replacement area, or the conflict area.

[0009] As one embodiment, before determining the plurality of evaluation parameters corresponding to each of the pixels to be processed, the method further includes: Based on the projection coverage area of ​​the initial three-dimensional hand mesh in the two-dimensional image, a model hand contour map is generated; Based on the color image and the original depth image, initial three-dimensional key points of the hand and the initial confidence level corresponding to the initial three-dimensional key points of the hand are obtained; Accordingly, determining the multiple evaluation parameters corresponding to each pixel to be processed includes: Determine the pixel depth validity and pixel neighborhood depth fluctuation amplitude corresponding to the pixel to be processed in the original depth image; The pixel depth difference is determined based on the difference between the original depth value corresponding to the pixel to be processed in the original depth image and the model depth value of the corresponding pixel position in the model depth image; The initial confidence scores corresponding to the initial three-dimensional key points of the hand are mapped to the corresponding pixel positions within the hand region to obtain pixel-level confidence scores. Pixel-level edge consistency is determined based on the positional relationship between the edge pixel positions in the color image and the edge pixel positions in the original depth image. The validity of the pixel-level model depth is determined based on whether the corresponding pixel position in the model depth map has a valid model depth value. Pixel-level model visibility is determined based on the visibility of the corresponding pixel position in the model visibility map. Based on the positional deviation between the model hand contour map and the edge pixel positions in the color image, and the positional deviation between the model hand contour map and the edge pixel positions in the original depth image, the boundary burr area and the boundary defect area within the hand region are determined. The pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel depth difference, pixel-level confidence, pixel-level edge consistency, pixel-level model depth validity, pixel-level model visibility, boundary spur region, and boundary defect region corresponding to the same pixel to be processed are used as multiple evaluation parameters corresponding to the pixel to be processed.

[0010] As one embodiment, dividing the pixel to be processed into the reserved region, the replacement region, or the conflict region based on the plurality of evaluation parameters corresponding to each pixel to be processed includes: The pixel to be processed is assigned to the reserved area if any of the following conditions are met: The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary spur area, and the pixel-level model depth validity is invalid; The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary spur area, the pixel-level model depth validity is valid, and the pixel-level model visibility does not reach the preset visibility threshold. The pixel depth validity is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, the pixel depth difference does not exceed the depth difference threshold, the pixel-level confidence reaches the local confidence threshold, the pixel-level edge consistency meets the preset consistency condition, and the pixel to be processed is not in the boundary burr area. When the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed meets at least one of the following conditions, the pixel to be processed is assigned to the replacement region: The pixel depth validity is invalid; The original depth value corresponding to the pixel to be processed is a hole value or an anomaly value; The pixel neighborhood depth fluctuation amplitude exceeds the depth fluctuation threshold; The pixel to be processed is located within the boundary defect area; When the pixel depth validity is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed is not assigned to the reserved area or the replacement area, the pixel to be processed is assigned to the conflict area.

[0011] As one embodiment, generating a blended depth map based on the plurality of pixel partitions includes: The depth value of each pixel position within the reserved area is determined as the original depth value of the corresponding pixel position in the original depth image; The depth value of each pixel position within the replacement area is determined as the model depth value of the corresponding pixel position in the model depth map; For pixels within the conflict zone, determine the pixel-level fusion weight corresponding to the pixel, and based on the pixel-level fusion weight, perform a weighted summation of the original depth value and the model depth value at the corresponding pixel position in the original depth image and the model depth image to obtain the mixed depth value of the pixel within the conflict zone. For pixels within the hand region that are not identified as the pixels to be processed, retain the depth value or depth invalidation marker of the corresponding pixel position in the original depth image; The mixed depth map is generated based on the depth values ​​of each pixel position in the reserved area, the replacement area, and the conflict area, as well as the depth values ​​or depth invalid markers of pixel positions that are not identified as the pixel to be processed.

[0012] As one embodiment, determining the pixel-level fusion weight corresponding to the pixel includes: The reliability of the original depth is determined based on the pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel-level edge consistency, and pixel-level confidence level corresponding to the pixel. The reliability of the model depth is determined based on the pixel-level model depth validity, pixel-level model visibility, pixel-level confidence, and pixel depth difference corresponding to the pixel. The pixel-level fusion weights corresponding to the original depth values ​​are determined based on the proportional relationship between the original depth reliability and the model depth reliability.

[0013] As one embodiment, before correcting the initial hand parametric model parameters based on the color image and the mixed depth map, the method further includes: A depth difference map is constructed based on the difference between the blended depth value of each pixel position in the blended depth map and the original depth value of the corresponding pixel position in the original depth image; Accordingly, the step of correcting the initial hand parametric model parameters based on the color image and the mixed depth map to obtain the target hand parametric model parameters includes: Feature encoding is performed on the color image, the original depth image, the mixed depth map, the depth difference map, and the model visibility map, respectively, and the obtained encoded features are fused to obtain fused features; Based on the fusion features and the initial hand parameterization model parameters, determine the residual correction amounts of multiple parameters and the update confidence levels corresponding to each local region of the hand; The parameters of the initial hand parameterized model are updated based on the residual corrections of the multiple parameters and the updated confidence level to obtain the parameters of the target hand parameterized model.

[0014] As an example, updating the initial hand parameterization model parameters based on the multiple parameter residual corrections and the update confidence to obtain the target hand parameterization model parameters includes: The initial hand parameterization model parameters are updated based on the residual correction values ​​of the multiple parameters to obtain the intermediate corrected hand parameterization model parameters; Based on the updated confidence levels corresponding to each local hand region, low-confidence local hand regions with updated confidence levels below the updated confidence level threshold are identified. Local closed-loop optimization is then performed on the intermediate corrected hand parameterization model parameters corresponding to the low-confidence local hand regions to obtain the target hand parameterization model parameters.

[0015] As an example, the step of performing local closed-loop optimization on the intermediate-corrected hand parametric model parameters corresponding to the low-confidence local region of the hand to obtain the target hand parametric model parameters includes: The intermediate-calibrated hand parameterized model parameters are used as the initial values ​​for iteration, and the hand joint parameters or local mesh parameters corresponding to the low-confidence local region of the hand are determined as the parameters to be optimized. Based on the intermediate corrected hand parameterized model parameters, a corrected model depth map, a corrected model visibility map, and a corrected model hand contour map are generated. The observed hand contour is determined based on at least one of the color image and the original depth image; A joint optimization objective function is constructed, comprising an observation consistency term and a hand prior constraint term. The observation consistency term includes at least one of a pixel depth consistency term and a contour consistency term, wherein the pixel depth consistency term is the pixel depth consistency between the corrected model depth map and the mixed depth map, weighted based on the corrected model visibility map; the contour consistency term is the contour consistency between the corrected model hand contour map and the observed hand contour; and the hand prior constraint term includes at least one of a hand bone length consistency term, a joint range of motion constraint term, and a self-collision penalty term. While keeping the parameters corresponding to the local hand region whose update confidence reaches the update confidence threshold unchanged, the parameters to be optimized are iteratively adjusted with the goal of reducing the joint optimization objective function, so as to obtain the parameterized model parameters of the target hand.

[0016] As one embodiment, the plurality of parameter residual corrections include shape parameter corrections, global translation corrections, global rotation corrections, and attitude parameter corrections; The step of updating the initial hand parameterization model parameters based on the multiple parameter residual corrections to obtain intermediate corrected hand parameterization model parameters includes: The shape parameter correction amount is linearly added to the initial hand shape parameter in the initial hand parameterized model parameters to obtain the corrected hand shape parameter; the global translation correction amount is linearly added to the initial global translation parameter in the initial hand parameterized model parameters to obtain the corrected global translation parameter. By performing an exponential mapping operation from rotation vector to rotation matrix or a rotation composition operation, the global rotation correction is combined with the initial global rotation parameters in the initial hand parameterization model parameters to obtain the corrected global rotation parameters. Based on the posture parameter correction amount, the initial hand posture parameters in the initial hand parameterized model parameters are updated by linear addition or rotation compound update to obtain the corrected hand posture parameters. Based on the corrected hand shape parameters, the corrected global translation parameters, the corrected global rotation parameters, and the corrected hand posture parameters, intermediate corrected hand parameterized model parameters are obtained.

[0017] As one embodiment, before determining the pixels to be processed within the hand region, the method further includes: Obtain a hand region mask based on the color image and the original depth image; The hand region is determined based on the hand region mask and the model hand contour map.

[0018] As one embodiment, determining the hand region based on the hand region mask and the model hand contour map includes: The morphological expansion process is performed on the hand contour of the model according to the preset expansion radius. The candidate hand region is obtained by determining the union of the expanded model hand contour map and the hand region mask; Based on at least one of the edges in the color image and the edges in the original depth image, the boundary of the candidate hand region is aligned and corrected to obtain the hand region.

[0019] Secondly, this application also provides a three-dimensional reconstruction device for the hand, comprising: The initial parameter acquisition module is used to acquire initial hand parameterized model parameters based on camera intrinsic parameters, color image, and original depth image aligned with the color image in pixel coordinate system; The projection module is used to project the initial three-dimensional mesh of the hand, represented by the parameters of the initial parametric hand model, onto the two-dimensional image corresponding to the color image based on the camera intrinsic parameters, thereby generating a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial three-dimensional hand mesh relative to the current camera at each pixel position; A pixel partitioning module is used to divide the pixels to be processed in the hand region into multiple pixel partitions based on the original depth image, the model depth map, and the model visibility map; wherein the multiple pixel partitions include a retention area, a replacement area, and a conflict area; A hybrid depth map generation module is used to generate a hybrid depth map based on the reserved area, the replacement area, and the conflict area; The parameter correction module is used to correct the initial hand parameterization model parameters based on the color image and the mixed depth map to obtain the target hand parameterization model parameters.

[0020] Thirdly, this application also provides a method for training a three-dimensional hand reconstruction network, wherein the three-dimensional hand reconstruction network includes a first reconstruction network and a second reconstruction network, and the method includes: Training samples are obtained, including sample camera intrinsic parameters, color sample images, depth sample images aligned with the color sample images in the pixel coordinate system, and ground truth parameters of the hand parameterized model; wherein, the depth sample images are obtained by performing depth defect enhancement on the original depth sample images. The sample camera intrinsic parameters, the color sample image, and the depth sample image are input into the first reconstruction network to obtain the initial hand parameter prediction values; Based on the intrinsic parameters of the sample camera, the initial three-dimensional mesh of the hand, represented by the predicted initial hand parameters, is projected onto the two-dimensional image corresponding to the color sample image to generate a model depth sample map and a model visibility sample map. Based on the depth sample image, the model depth sample map, and the model visibility sample map, the sample pixels in the hand area are divided into a reserved area, a replacement area, or a conflict area, and a hybrid depth sample map is generated based on the different sample pixel partitions. A depth difference sample map is generated based on the difference between the depth values ​​at corresponding pixel positions in the hybrid depth sample map and the depth sample image. The color sample image, the depth sample image, the mixed depth sample image, the depth difference sample image, the model visibility sample image, and the initial hand parameter prediction value are input into the second reconstruction network to obtain the parameter residual prediction value, and the corrected hand parameter prediction value is obtained based on the parameter residual prediction value and the initial hand parameter prediction value. Based on the differences between the initial predicted hand parameters and the ground truth parameters of the hand parameterized model, and the differences between the corrected predicted hand parameters and the ground truth parameters of the hand parameterized model, a training loss is constructed. Based on the training loss, the first reconstruction network and the second reconstruction network are trained in stages or jointly until the preset convergence condition is met, and the trained hand 3D reconstruction network is obtained.

[0021] Fourthly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described three-dimensional hand reconstruction methods and / or the above-described training method for the three-dimensional hand reconstruction network.

[0022] Fifthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described three-dimensional hand reconstruction methods and / or the above-described training method for a three-dimensional hand reconstruction network.

[0023] Sixthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described three-dimensional hand reconstruction methods and / or the above-described training method for a three-dimensional hand reconstruction network. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts illustrating the three-dimensional reconstruction method for the hand provided in this application; Figure 2 This is the second flowchart illustrating the three-dimensional reconstruction method for the hand provided in this application; Figure 3 This is a schematic diagram of the model reprojection and pixel partitioning of the hand region provided in this application; Figure 4 This is a schematic diagram illustrating the generation of the hybrid depth map and depth difference map provided in this application; Figure 5 This is a schematic diagram of the structure of the second reconstruction network performing residual correction provided in this application; Figure 6This is a schematic diagram of selective closed-loop correction for low-confidence critical points and low-confidence local regions provided in this application; Figure 7 This is a schematic diagram of the structure of the three-dimensional hand reconstruction device provided in this application; Figure 8 This is a flowchart of the hand 3D reconstruction network training method provided in this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0028] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0029] The following description, in conjunction with the accompanying drawings, describes the hand 3D reconstruction method, apparatus, and reconstruction network training method provided in this application.

[0030] It should be noted that the hand 3D reconstruction method provided in this application embodiment is based on a hand 3D reconstruction device. The hand 3D reconstruction method provided in this application embodiment can improve the stability and accuracy of hand reconstruction under conditions of depth holes, boundary burrs, local occlusion and low quality depth, and is suitable for scenarios such as interactive tracking, virtual-real fusion and robot teleoperation.

[0031] This application describes a hand three-dimensional reconstruction method using a hand three-dimensional reconstruction device as the execution subject as an example.

[0032] Figure 1 This is one of the flowcharts illustrating the three-dimensional hand reconstruction method provided in this application. For example... Figure 1 As shown, the hand three-dimensional reconstruction method provided in this application includes: S110: Based on camera intrinsic parameters, color image, and original depth image aligned with the color image in pixel coordinate system, obtain initial hand parameterization model parameters.

[0033] S120: Based on the camera intrinsic parameters, project the initial hand 3D mesh represented by the initial hand parameterized model parameters onto the 2D image corresponding to the color image to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial hand 3D mesh relative to the current camera at each pixel position.

[0034] The hand 3D reconstruction method of this application is implemented through a hand 3D reconstruction network, which includes a first reconstruction network and a second reconstruction network.

[0035] The first reconstruction network can be a convolutional network, a Transformer network, or a hybrid network combining convolution and Transformer. The first reconstruction network is used to obtain initial hand parameterized model parameters, initial hand 3D keypoint coordinates, and initial confidence levels of the initial hand 3D keypoint coordinates based on camera intrinsics, a color image, and an original depth image aligned with the color image in pixel coordinates.

[0036] Please combine Figure 2 and Figure 3 The input data for the first reconstruction network includes a pixel-aligned color image (RGB image, Ir) 101, a raw depth image (Do) 102, and camera intrinsics (K) 103. The camera intrinsics include at least the horizontal focal length fx, the vertical focal length fy, the principal point x-coordinate cx, and the principal point y-coordinate cy. Since the RGB image and the depth image are pixel-aligned, the RGB value and depth value at the same pixel location x=(u,v) correspond to the same imaging ray.

[0037] For valid pixels in the original depth map, they can be back-projected into 3D points in the camera coordinate system based on camera intrinsic parameters: X=(u-cx)Do(u,v) / fx; Y=(v-cy)Do(u,v) / fy; Z=Do(u,v) (1).

[0038] This yields the 3D coordinate map Po corresponding to the original depth map. In practical implementation, the input to the first reconstruction network can include an RGB three-channel image, a normalized depth map, a depth validity mask, the 3D coordinate map, and camera intrinsic parameter encoding.

[0039] The output data of the first reconstruction network includes initial hand parametric model parameters, initial hand 3D keypoint coordinates, and initial confidence scores for the initial hand 3D keypoint coordinates. The initial hand parametric model parameters include at least initial hand pose parameters, initial hand shape parameters, initial global rotation parameters, and initial global translation parameters.

[0040] In one possible implementation, the output data of the first reconstruction network also includes a hand region mask, hand contour, and / or keypoint visibility. The hand region mask is used to limit subsequent pixel partitioning to the hand region only. The hand contour is used to determine whether there are burrs, defects, or inconsistencies with RGB edges in the original depth boundary. Keypoint visibility is used to determine whether local areas such as fingers and palms may be occluded by objects or self-occluded. The hand region mask, hand contour, and / or keypoint visibility are used to reduce the original depth weight of unreliable regions during the pixel partitioning stage (see the explanation of S130 below), and to identify low-confidence keypoints or low-confidence local regions that require local optimization during the selective local loop closure optimization iteration stage (see the explanation of S150 below).

[0041] In one possible implementation, the first reconstruction network includes a color (RGB) feature encoding branch, a depth geometric feature encoding branch, a camera intrinsic parameter encoding branch, and a multi-task output head, which includes a parametric regression head, a keypoint regression head, and a confidence prediction head.

[0042] The RGB feature encoding branch is used to extract hand texture, edges, and semantic features of the hand region. The depth geometry feature encoding branch is used to extract the 3D shape, depth continuity, and depth hole features of the hand. The camera intrinsic parameter encoding branch is used to encode the focal length, principal point position, and pixel ray direction into feature vectors or pixel-level positional codes to avoid scale and viewpoint ambiguities caused by different camera intrinsic parameters.

[0043] After being concatenated, attention-based fusion, or channel-wise weighted fusion, the features of each branch are input into the parametric regression head, keypoint regression head, and confidence prediction head. The parametric regression head outputs the initial hand parametric model parameters Θ0. The keypoint regression head outputs a preset number of initial hand 3D keypoint coordinates, preferably 21 hand keypoint coordinates J0={J0j}. The confidence prediction head outputs the initial confidence C0 of the initial hand 3D keypoint coordinates.

[0044] In one possible implementation, when the parametric hand model uses a 3D Hand Model for Articulated and Non-rigid Objects (MANO), the initial hand parametric model parameters Θ0 include at least the initial hand pose parameter θ0, the initial hand shape parameter β0, the initial global rotation parameter R0, and the initial global translation parameter t0. The hand pose parameter describes the bending and rotational states of the finger joints, the hand shape parameter describes the palm width, finger length, and overall hand shape differences, and the global rotation and global translation parameters describe the spatial pose of the entire hand in the camera coordinate system.

[0045] In one possible implementation, the keypoint regression head can directly regress the 3D coordinates in the camera coordinate system.

[0046] In another possible implementation, the keypoint regression head first outputs a two-dimensional heatmap and relative depth for each keypoint, and then combines this with backprojection of camera intrinsic parameters to obtain the three-dimensional keypoint coordinates. Specifically, for keypoint j, the keypoint regression head outputs a two-dimensional heatmap Hj and a depth value zj, obtains the pixel coordinates (uj, vj) through soft-argmax or the maximum response position, and then obtains the three-dimensional coordinates of the keypoint in the camera coordinate system using the following equation (2). : (2); in, This represents the inverse matrix of the camera intrinsic parameters. Represents two-dimensional pixel coordinates.

[0047] The initial confidence level C0 can be a keypoint-level confidence level, a local region-level confidence level, or a combination of both. The keypoint-level confidence level C0j can be determined jointly by heatmap peak values, heatmap distribution variance, effective local depth ratio, consistency between predicted keypoints and model keypoints, and whether keypoints are occluded. Alternatively, it can be directly output by the network using the sigmoid activation function. The local region-level confidence level can be obtained by mapping the keypoint confidence level to the corresponding finger segment, palm grid block, or pixel region. For example, for a certain local grid block Gk, the average or weighted average of its associated keypoint confidence levels can be used as the local region-level confidence level.

[0048] After obtaining the initial parametric model parameters of the hand, a specific initial hand network is constructed in virtual space based on these parameters. This initial hand network can be represented by a 3D mesh surface, voxel representation, or point cloud set. Subsequently, based on the actual camera imaging model, the constructed initial hand network is inversely geometrically mapped to the current 2D image viewpoint to achieve reprojection. Please refer to [reference needed]. Figure 2 The model is reprojected. The reprojection process simulates the principles of ray casting or differentiable rendering, flattening the model features in three-dimensional space onto a two-dimensional pixel plane. Subsequently, differentiable rendering or rasterization rendering is used to generate hand depth prior information, including a model depth map, visibility map, and hand contour map corresponding to the hand region. This hand depth prior information is a clean hand reference information that completely masks background noise from the real scene.

[0049] The parametric hand model is used to establish a computable relationship between the initial parametric hand model parameters, the initial hand network, and the reprojected depth prior. Specifically, the initial parametric hand model parameters Θ0 output by the first reconstruction network are input into the MANO model to generate an initial hand mesh M0=(V0,F1), where V0 is the set of mesh vertices and F1 is the topology of mesh triangles. The MANO model reprojects the initial hand mesh into the camera coordinate system under the constraints of the real camera intrinsic parameters, generating a hand model depth map Dm, a visibility map Vm, and a model hand contour map Sm. In addition, the MANO model provides structural constraints such as bone length, joint range of motion, and self-collision during local closed-loop optimization iterations, ensuring that local corrections in low-confidence regions do not compromise the overall rationality of the hand, as explained in S150 below.

[0050] The model depth map Dm is a representation of the hand depth derived from the initial hand geometry prior. It is used to constrain the low-quality depth of the hand region, rather than providing general depth completion for the entire scene. The visibility map Vm is a binary or probabilistic map used to indicate whether a pixel or mesh surface is a surface visible from the current 2D image viewpoint; for example, after z-buffer rasterization, pixels occupied by the nearest triangle are assigned a value of 1, while pixels that are self-occluded, backlit, or not covered by the model are assigned a value of 0. The model hand contour map Sm is a binary map of the hand region after the model is projected onto the image plane. Its boundaries can be obtained by performing a morphological gradient on the model hand contour map Sm.

[0051] S130: Based on the original depth image, the model depth map, and the model visibility map, the pixels to be processed within the hand region are divided into multiple pixel partitions. These multiple pixel partitions include a reserved area, a replacement area, and a conflict area.

[0052] Specifically, the original depth image, the color image, and the prior information on hand depth are comprehensively compared from multiple dimensions. Based on the multi-dimensional feature comparison results, the pixels within the hand region are clearly divided into three isolated pixel partitions: a reserved area, a replacement area, and a conflict area. Please refer to [reference needed]. Figure 2 .

[0053] In practice, the reserved area is used to summarize high-quality pixels that are completely reliable in terms of original depth image measurement and have a high degree of consistency with the prior model; the replacement area is used to summarize incomplete pixels that have obvious holes or serious noise anomalies in the original depth image; and the conflict area is used to summarize edge pixels that fall between the above two, where there is a certain deviation between the observed data and the model data and the reliability is difficult to determine by a single factor.

[0054] S140: Generate a mixed depth map based on the multiple pixel partitions.

[0055] After the region is divided, a corresponding depth value processing scheme is executed for the characteristics exhibited by different pixel partitions. For example, for the reserved region, the depth value of the original depth image can be directly adopted; for the replacement region, hand depth prior information can be used for filling; for the conflict region, adaptive weighted fusion can be performed based on the reliability confidence level of both.

[0056] The processed depth values ​​of each pixel partition are then reassembled and integrated to output a mixed depth map Dh that eliminates hole and flying point interference while retaining realistic and reliable details. Please refer to [reference needed]. Figure 2 .

[0057] S150: Based on the color image and the mixed depth map, the parameters of the initial hand parametric model are corrected to obtain the parameters of the target hand parametric model. Based on the parameters of the target hand parametric model, hand reconstruction is performed to obtain the position of the three-dimensional hand mesh in the camera coordinate system.

[0058] Please combine Figure 2 This step is implemented through a second reconstruction network. The second reconstruction network receives the RGB image Ir, the original depth image Do, the blended depth map Dh, the visibility map Vm, the initial hand parameterized model parameters Θ0, and optional hand region masks, model hand contour maps Sm, initial hand 3D keypoint coordinates J0, and initial keypoint heatmaps. Unlike re-estimation from zero, the second reconstruction network uses residual regression, outputting only multiple parameter residual corrections (including shape parameter corrections, global translation corrections, global rotation corrections, and pose parameter corrections). These parameter residual corrections are superimposed or compounded onto the initial hand parameterized model parameters to complete the correction update. After correction by the second reconstruction network, the final hand parameterized model parameters can be output. Figure 2(shown as MANO parameters), final 3D keypoint coordinates of the hand ( Figure 2 (Shown as 3D key points). The final parametric model parameters of the hand can be directly applied to downstream tasks such as high-precision virtual reality hand interaction tracking or remote control of robotic dexterity hands.

[0059] This application's embodiments inversely map the roughly estimated initial parametric model parameters of the hand to prior hand depth information in a two-dimensional view, performing reprojection depth completion only within the hand region to avoid irrelevant error propagation caused by general depth completion across the entire scene. Furthermore, it utilizes this prior information in conjunction with the original data to construct a three-state pixel partitioning mechanism including a reserved area, a replacement area, and a conflict area, robustly addressing depth holes, boundary defects, and local occlusion, achieving refined separation of reliable observations from anomalous noise in the original depth image. By using partitioned differential processing to obtain a hybrid depth map, it addresses the depth holes and boundary defects easily generated by consumer-grade depth sensors in scenarios with hand occlusion and rapid movement. The high-quality hybrid depth map guides subsequent parameter correction updates, ensuring that the estimation of the final parametric model parameters is free from interference from unreliable features, significantly improving the robustness and spatial positioning accuracy of the hand reconstruction algorithm in complex scenes. Under conditions of complex interaction, close-range shooting, boundary occlusion, and low-quality depth, this application's embodiments can improve the parameter estimation accuracy and key point positioning stability of the parametric hand model, demonstrating significant engineering application value.

[0060] In one possible implementation, in S120, based on the camera intrinsic parameters, the initial hand 3D mesh represented by the initial hand parameterized model parameters is projected onto the 2D image corresponding to the color image to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image, including: S1201: Based on the camera intrinsic parameters, project multiple mesh patches in the initial three-dimensional hand mesh onto the two-dimensional image coordinate system corresponding to the color image to obtain the projection area of ​​each mesh patch.

[0061] S1202: Rasterize each projected grid patch to determine the pixel positions covered by each grid patch, and calculate the candidate model depth value for the pixel positions covered by the grid patch.

[0062] S1203: When multiple candidate model depth values ​​correspond to the same pixel position, the candidate model depth value closest to the current camera optical center is selected by using a depth buffer method as the model depth value of the pixel position to generate the model depth map.

[0063] S1204: Based on the depth test results of each mesh patch at the corresponding pixel position, and the relationship between the normal of each mesh patch and the current camera viewing direction, determine the visibility at each pixel position to generate the model visibility map.

[0064] After obtaining the initial parametric hand model parameters Θ0, input them into the parametric hand model to generate the initial 3D hand mesh M0=(V0,F). For mesh vertices vi∈V0, transform them to the camera coordinate system according to the initial global rotation parameter R0 and the initial global translation parameter t0 to obtain the corresponding coordinates vic. vic=R0 vi+t0; where vic=(Xi,Yi,Zi); Subsequently, the coordinates vic are projected onto the two-dimensional image coordinate system corresponding to the color image based on the camera intrinsic parameter K, to obtain the corresponding two-dimensional pixel coordinates (ui, vi), which are the mesh vertices of the initial three-dimensional mesh of the hand: ui=fx Xi / Zi+cx; vi=fy Yi / Zi+cy.

[0065] The initial 3D hand mesh comprises multiple mesh vertices and multiple mesh patches formed by connecting these vertices. Each mesh patch consists of at least three mesh vertices from the initial 3D hand mesh and represents a local surface region of the initial 3D hand mesh. For a parametric hand model represented using a triangular mesh, each mesh patch is a triangular patch composed of three mesh vertices, and these multiple mesh patches collectively constitute the palm surface and finger surface of the initial 3D hand mesh. Thus, each mesh patch obtained from projection corresponds to a projection region.

[0066] Finally, the projected mesh is rasterized. For any pixel x, if at least one model triangle covers the pixel, the model surface closest to the current 2D image viewpoint (i.e., the closest visible triangle to the camera) is selected using a depth buffer, and the pixel's model depth value Dm(x) is assigned the depth value of that nearest model surface. If no model triangle covers the pixel, the pixel's model depth value Dm(x) is set to an invalid value.

[0067] When multiple candidate model depth values ​​correspond to the same pixel location, the candidate model depth value closest to the current camera optical center is selected as the model depth value for that pixel location using a depth buffering method.

[0068] The model depth map Dm is constructed by combining the model depth values ​​at all pixel locations. Therefore, the model depth map Dm is a hand depth representation derived from the initial hand geometry prior and has the same resolution as the original depth map.

[0069] The visibility map Vm indicates whether the model surface is visible from the current camera viewpoint (i.e., the 2D image viewpoint). The visibility map Vm limits the scope of the model's prior knowledge. When the visibility of a pixel Vm(x) = 1, it means that the pixel corresponds to a visible surface of the model; when the visibility of a pixel Vm(x) = 0, it means that the pixel is not covered by the model or belongs to a self-occluded / invisible area.

[0070] In one possible implementation, the visibility of each pixel is determined based on the depth test results at the corresponding pixel location and the relationship between the normal of each mesh facet and the current camera viewing direction, in order to generate a model visibility map. If pixel x is covered by a model triangle facet, and the model triangle facet passes the depth buffer test, is not occluded by other model surfaces, and is not a back facet, then the visibility of the pixel Vm(x) = 1; otherwise, Vm(x) = 0.

[0071] This application embodiment explicitly projects and calculates the abstract three-dimensional spatial structure of the hand, and decomposes it into a model depth map, a visibility map, and a model hand contour map, providing multi-dimensional geometric prior references for subsequent processing. In particular, the visibility map can keenly perceive and mark the finger mesh patches that are self-occluded or on the back side of the hand from the current camera viewpoint, effectively preventing the invisible model surface depth hidden on the back from being erroneously extracted and used by the network and contaminating the real surface observation data. This greatly improves the safety and guidance accuracy of the application of three-dimensional geometric prior information in complex gesture occlusion environments.

[0072] In one possible implementation, before determining the pixels to be processed within the hand region, the method further includes obtaining the hand region: Q10: Obtain the hand region mask based on the color image and the original depth image. The hand region mask is obtained through a first reconstruction network, as explained above.

[0073] Q20: Determine the hand region based on the hand region mask and the model hand contour map.

[0074] Specifically, the hand region mask generated under the data-driven system and the model hand contour map generated under the physical geometry-driven system are subjected to spatial overlap analysis and boundary fusion processing. This allows the semantic-level observed hand range predicted by the network model to be complementaryly correlated with the absolute geometric boundary, thereby dynamically and accurately cropping the target scope of the hand to be processed, i.e., the hand region, in the whole image.

[0075] This application establishes a dual-constraint region delimitation mechanism by combining a hand region mask reflecting two-dimensional observation semantics extracted from a color image and the original depth image, and a model hand contour map reflecting absolute geometric boundaries generated by three-dimensional spatial structure projection. This mechanism strictly limits subsequent pixel partitioning and mixed depth calculations to be carried out only within a highly certain hand region, shielding it from interference from irrelevant depth information in the background environment and preventing background depth errors from being introduced into the depth completion calculation system. This not only significantly eliminates the unnecessary computational overhead caused by full-image depth completion for the entire scene, effectively saving the resource consumption of the computing system, but also significantly improves the purity of local pixel partitioning processing, providing an extremely reliable domain guarantee for finally achieving high-precision hand parameter correction.

[0076] In one possible implementation, in Q20, determining the hand region based on the hand region mask and the model hand contour map includes: Q21: Perform morphological expansion processing on the hand contour of the model according to the preset expansion radius.

[0077] Q22: Determine the union of the expanded model hand contour map and the hand region mask to obtain the candidate hand region.

[0078] Q23: Based on at least one of the edges in the color image and the edges in the original depth image, the boundary of the candidate hand region is aligned and corrected to obtain the hand region.

[0079] In this embodiment, the model's hand contour image is morphologically expanded by a preset radius and then combined with the hand region mask extraction to construct a candidate hand region with sufficient spatial inclusiveness, ensuring that no real hand pixels are excluded. Furthermore, by combining the edges of the real color image or the original depth image for alignment correction, the final enclosed hand region can closely fit the real boundary of the physical surface. This dynamic positioning mechanism of first expanding and then aligning and contracting not only accommodates the slight deviations inherent in the initial prediction stage, but also implements strict truncation at the physical boundary, avoiding unnecessary hole filling calculations for the background depth of the entire scene, and cutting off the path of background environment-independent errors propagating to the hand reconstruction process.

[0080] In one possible implementation, in S130, dividing the pixels to be processed within the hand region into multiple pixel partitions based on the original depth image, the model depth map, and the model visibility map includes: S1301: The pixels with valid original depth values ​​in the original depth image and the pixels with valid model depth values ​​in the model depth image and whose visibility in the model visibility image reaches a preset visibility threshold are determined as the pixels to be processed.

[0081] S1302: Based on the original depth image, the model depth map, and the model visibility map, determine multiple evaluation parameters corresponding to each pixel to be processed.

[0082] Evaluation parameters are a series of discriminative indicators used to quantify the quality of various data features, observation reliability, and prior agreement at a specific pixel.

[0083] Specifically, evaluation parameters for the validity of the representation can be determined by reading the status codes of the camera's underlying driver feedback, evaluation parameters for the overlap of the representation edges can be determined by calculating the spatial offset between the color image gradient and the model contour gradient, and evaluation parameters for the spatial consistency of the representation can be determined by extracting the absolute or relative error of the depth level.

[0084] S1303: Based on the plurality of evaluation parameters corresponding to each pixel to be processed, the pixel to be processed is divided into the reserved area, the replacement area, or the conflict area.

[0085] Specifically, this can be achieved by setting up a multi-level conditional judgment logic tree. For example, when multiple evaluation parameters of a pixel simultaneously satisfy the conditions of being valid, highly consistent, and having low error, the pixel is assigned to the reserved region. Alternatively, a lightweight classifier based on a support vector machine or multilayer perceptron can be pre-trained, using the concatenated evaluation parameters as input feature vectors. The classifier then directly outputs the category label of the pixel belonging to the three regions mentioned above, thus completing the region assignment operation. This multi-parameter cross-validation partitioning method ensures that each pixel within the hand region can be objectively and accurately assigned to the most suitable processing channel based on the complex imaging conditions of its location.

[0086] This application embodiment constructs a comprehensive evaluation parameter with multiple dimensions for each independent pixel in the hand region by combining the original depth image, color image, and hand depth prior information. This breaks away from the drawbacks of traditional methods that treat the entire hand as a uniform, coarse object. Furthermore, based on the above-mentioned multi-dimensional evaluation parameters, cross-validation and fine-tuning of multi-source information are performed on each pixel, and the pixels are accurately diverted to retention areas, replacement areas, or conflict areas with distinctly different properties. This achieves explicit hard isolation of real and reliable observation data from abnormal and unreliable noise interference at the pixel level. This fine-tuning partitioning mechanism based on the composite judgment of multi-dimensional evaluation parameters can not only fully activate and utilize the spatial geometric prior knowledge generated in the initial stage, but also accurately cut off the chain of erroneous observation information propagating to the subsequent processing network from the source. This significantly improves the robustness and scientific nature of feature fusion of the entire reconstruction algorithm when facing complex imaging conditions such as depth holes and boundary defects.

[0087] In one possible implementation, before determining the multiple evaluation parameters corresponding to each pixel to be processed, the method further includes: Based on the projection coverage area of ​​the initial three-dimensional hand mesh in the two-dimensional image, a model hand contour map is generated; The initial confidence level of the initial three-dimensional key point coordinates of the hand is obtained based on the color image and the original depth image.

[0088] Specifically, the model hand contour map Sm is a binary region map formed after projecting the model mesh. If pixel x is covered by the projection of any model triangle, then Sm(x)=1; otherwise, Sm(x)=0.

[0089] Furthermore, the model hand contour boundary can be obtained by performing dilation, erosion, and morphological gradient operations on the model hand contour map Sm. This model hand contour boundary is used to determine the hand region Oh and is subsequently used for boundary burr detection and contour consistency constraints.

[0090] The process of obtaining the initial confidence level is explained in the above description.

[0091] Accordingly, in S1302, determining the multiple evaluation parameters corresponding to each pixel to be processed includes: S13021: Determine the pixel depth validity Mo(x) and pixel neighborhood depth fluctuation amplitude σz(x) for each pixel in the original depth image.

[0092] Specifically, if the depth value Do(x) of pixel x in the original depth image is not empty, not 0, not a number (NaN), and is within the effective range of the depth camera, then Mo(x) = 1; otherwise, Mo(x) = 0.

[0093] Based on the above, abnormal flying points are identified by combining local depth continuity. For example, in the neighborhood centered on pixel x, if the difference between Do(x) and the median value of the neighborhood is greater than a preset threshold, then the pixel is identified as a noise abnormal pixel.

[0094] The pixel neighborhood depth fluctuation amplitude is used to characterize the depth continuity between the pixel to be processed and its neighboring effective depth pixels. Specifically, for the pixel to be processed x, a neighborhood window N(x) of a preset size is constructed with x as the center. The neighborhood window N(x) can be a 3×3, 5×5 or other preset size pixel window. Pixels with effective original depth values ​​are selected from the neighborhood window N(x) to obtain the effective neighborhood pixel set Nv(x).

[0095] In one implementation, a neighborhood reference depth value Dmed(x) is determined based on the original depth values ​​of each pixel in the effective neighborhood pixel set Nv(x), wherein Dmed(x) can be the median, mean, or weighted mean of the original depth values ​​in the effective neighborhood pixel set Nv(x). Further, the difference between the original depth value Do(x) of the pixel to be processed x and the neighborhood reference depth value Dmed(x) is determined as the pixel neighborhood depth fluctuation amplitude σz(x) of the pixel to be processed x, i.e.: σz(x) = |Do(x) - Dmed(x)|; Where σz(x) represents the depth fluctuation amplitude of the neighborhood of pixel x, Do(x) represents the original depth value of pixel x in the original depth image, and Dmed(x) represents the neighborhood reference depth value of pixel x.

[0096] If the pixel neighborhood depth fluctuation amplitude σz(x) is greater than the preset depth fluctuation threshold, it indicates that the pixel to be processed x has an abnormal depth jump relative to its neighboring pixels, and the pixel to be processed x can be judged as an abnormal noise pixel or its original depth reliability can be reduced; if the pixel neighborhood depth fluctuation amplitude σz(x) does not exceed the preset depth fluctuation threshold, it indicates that the pixel to be processed x has good continuity with the neighborhood depth, and its original depth reliability can be improved.

[0097] S13022: Determine the pixel depth difference ΔD(x) based on the difference between the original depth value corresponding to the pixel to be processed in the original depth image and the model depth value of the corresponding pixel position in the model depth image.

[0098] For a valid pixel x in the original depth image, the difference between the original depth value Do(x) corresponding to the pixel to be processed in the original depth image and the model depth value Dm(x) at the corresponding pixel position in the model depth image is taken as the pixel depth difference ΔD(x) of pixel x: ΔD(x)=|Do(x)-Dm(x)| (3); For invalid pixels in the original depth image, pixel depth consistency is not calculated or the pixel depth consistency is set to a preset large value.

[0099] S13023: Map the initial confidence scores corresponding to the initial three-dimensional key points of the hand to the corresponding pixel positions within the hand region to obtain pixel-level confidence scores Cp(x).

[0100] Specifically, based on the pixel position of pixel x in the model's hand contour map, its corresponding mesh triangle or nearest key point can be determined, and then the confidence level of the corresponding finger segment or palm mesh block can be assigned to the pixel to obtain pixel-level confidence.

[0101] If a pixel corresponds to multiple local regions, a distance-weighted average method can be used to obtain the pixel-level confidence level.

[0102] S13024: Determine pixel-level edge consistency E(x) based on the positional relationship between the edge pixel positions in the color image and the edge pixel positions in the original depth image.

[0103] If a color image has a clear edge near pixel x, and the original depth map also has a depth edge at the corresponding pixel position, then the pixel-level edges are considered to be consistent, i.e., E(x)=1; otherwise, E(x)=0.

[0104] If the deviation between the edge pixel position of the color image and the edge pixel position of the original depth image exceeds a preset range, it is considered that there is a boundary burr area or a boundary defect area.

[0105] S13025: Determine the validity of the pixel-level model depth based on whether the corresponding pixel position in the model depth map has a valid model depth value.

[0106] Specifically, if the model depth value Dm(x) corresponding to pixel x in the model depth map is not empty or invalid, and is within the preset valid depth range, then the pixel-level model depth validity Mm(x) of pixel x is determined to be valid, i.e., Mm(x) = 1; otherwise, the pixel-level model depth validity Mm(x) of pixel x is determined to be invalid, i.e., Mm(x) = 0.

[0107] S13026: Determine the pixel-level model visibility based on the visibility of the corresponding pixel position in the model visibility map.

[0108] Specifically, if the visibility Vm(x) corresponding to pixel x in the model visibility map is greater than or equal to the preset visibility threshold τv, then the pixel-level model visibility of pixel x is determined to meet the preset visibility condition; otherwise, the pixel-level model visibility of pixel x is determined not to meet the preset visibility condition.

[0109] Among them, pixel-level model depth validity Mm(x) is used to characterize whether pixel x in the model depth map has a valid model depth value; pixel-level model visibility Vm(x) is used to characterize whether the corresponding model surface in the model depth map is visible under the current camera view, and is used to limit only the model depth visible under the current camera view to participate in replacement or fusion, so as to prevent invisible fingers or back surfaces from erroneously affecting the original observation.

[0110] S13027: Based on the positional deviation between the edge pixel positions of the model hand contour map and the color image, and the positional deviation between the edge pixel positions of the model hand contour map and the original depth image, determine the boundary burr area and the boundary defect area within the hand region.

[0111] Specifically, firstly, high-frequency features are extracted from the color image and the original depth image using an edge detection operator to obtain a set of two-dimensional texture edge pixels and a set of depth abrupt change edge pixels. Then, the spatial Euclidean distance or directional gradient distance between the edge pixels of the model's hand contour map and the two-dimensional texture edge pixels, and between the edge pixels of the model's hand contour map and the depth abrupt change edge pixels, is calculated, and this distance value is used as the positional deviation.

[0112] Boundary burr regions represent redundant noise patches that abruptly extend into the background beyond the actual physical boundary of the hand, while boundary defect regions represent missing blocks that are abnormally recessed into the hand because the original depth observation results fail to reach the actual physical boundary.

[0113] Specifically, when the depth edge or effective depth connected region in the original depth image overflows outward relative to the edge of the model hand contour map and / or the hand edge in the color image, and the overflow deviation exceeds a preset overflow threshold, the corresponding pixel connected region is determined as a boundary spur region. When the depth edge in the original depth image shrinks inward relative to the edge of the model hand contour map and / or the hand edge in the color image, and the shrinkage deviation exceeds a preset shrinkage threshold, the corresponding pixel connected region in the model hand contour map and / or the hand edge in the color image that lacks a valid original depth value is determined as a boundary defect region.

[0114] S13028: The pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel depth difference, pixel-level confidence, pixel-level edge consistency, pixel-level model depth validity, pixel-level model visibility, boundary spur region, and boundary defect region corresponding to the same pixel to be processed are used as multiple evaluation parameters corresponding to the pixel to be processed.

[0115] This application embodiment aggregates pixel depth validity and pixel-level model validity, which reflect the physical state of the underlying data, combines pixel depth consistency and pixel-level edge consistency, boundary spur regions and boundary defect regions, which reflect the cross-modal geometric alignment state, and integrates pixel-level confidence, which reflects the inherent uncertainty of the network, to construct a rigorous and three-dimensional multi-dimensional health check index library, providing a quantitative data foundation for achieving accurate pixel-level classification and hard isolation in the future.

[0116] In one possible implementation, in S1303, dividing the pixel to be processed into the reserved area, the replacement area, or the conflict area based on the plurality of evaluation parameters corresponding to each pixel to be processed includes: S13031: The pixel to be processed is assigned to the reserved area Okeep when any of the following conditions are met: Scenario 1: The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary spur area, and the pixel-level model depth validity is invalid; Scenario 2: The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary burr area, the pixel-level model depth validity is valid, and the pixel-level model visibility does not reach the preset visibility threshold. Scenario 3: The pixel depth validity of the pixel to be processed is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, the pixel depth difference does not exceed the depth difference threshold, the pixel-level confidence reaches the local confidence threshold, the pixel-level edge consistency meets the preset consistency condition, and the pixel to be processed is not in the boundary burr area.

[0117] Specifically, the pixel set of the reserved region Okeep can be determined based on any of the above scenarios. For scenario three, the reserved region Okeep can be determined under the following conditions: Okeep={x∈Oh|Mo(x)=1,Mm(x)=1,Vm(x)≥τv,ΔD(x)≤τ1,Cp(x)≥τ2,E(x)=1,x B} (4); Where Mo(x) represents the pixel depth validity of pixel x, Mm(x) represents the pixel-level model depth validity of pixel x, Vm(x) represents the pixel-level model visibility of pixel x, τv represents the preset visibility threshold, ΔD(x) represents the pixel depth difference, τ1 represents the depth difference threshold, Cp(x) represents the pixel-level confidence, τ2 represents the local confidence threshold, E(x) represents the pixel-level edge consistency, E(x)=1 indicates that the pixel-level edge consistency meets the preset consistency condition, and B represents the boundary spur region.

[0118] S13032: When the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed meets at least one of the following conditions, the pixel to be processed is assigned to the replacement area: The pixel depth validity is invalid; The original depth value corresponding to the pixel to be processed is a hole value or an anomaly value; The pixel neighborhood depth fluctuation amplitude exceeds the depth fluctuation threshold; The pixel to be processed is located within the boundary defect area.

[0119] Specifically, the pixel set of the replacement region Oreplace is determined based on the following conditions: Oreplace={x∈Oh|Mm(x)=1,Vm(x)≥τv,[Mo(x)=0 or x is a deep hole pixel or x is an abnormal depth pixel or σz(x)>τz or x∈Bmiss]} (5; Where Mm(x) represents the pixel-level model depth validity of pixel x, Vm(x) represents the pixel-level model visibility of pixel x, τv is the visibility threshold, Mo(x) represents the pixel depth validity of pixel x, σz(x) represents the pixel neighborhood depth fluctuation amplitude of pixel x, τz represents the depth fluctuation threshold, and Bmiss represents the boundary defect region.

[0120] S13033: When the pixel depth validity is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed is not assigned to the reserved area or the replacement area, the pixel to be processed is assigned to the conflict area.

[0121] Specifically, the pixel set of the conflict zone Oconflict is determined based on the following conditions: Oconflict={x∈Oh|Mo(x)=1,Mm(x)=1,Vm(x)≥τv,x Okeep,x Oreplace}(6; Okeep represents the reserved area, and Oreplace represents the replacement area.

[0122] Figure 3 An example of a pixel partition is given, which includes two replacement regions, one reserved region, and one conflict region.

[0123] The embodiments of this application set multi-dimensional threshold conditions for pixel depth validity, depth consistency, confidence, and boundary distortion status, and accurately and forcibly divide pixels into reserved areas, replacement areas, or conflict areas. This strict area determination standard ensures that observed pixels have a clear belonging area.

[0124] In one possible implementation, in step S140, generating the blended depth map based on the plurality of pixel partitions includes: P10: The depth value of each pixel position in the reserved area is determined as the original depth value of the corresponding pixel position in the original depth image, and the depth value of each pixel position in the replacement area is determined as the model depth value of the corresponding pixel position in the model depth map.

[0125] Pixels located in the retention region Okeep have valid original depths that are consistent with the model prior, so their original depths can be directly retained, i.e., Dh(x) = Do(x). Pixels located in the replacement region Oreplace have missing or obviously unreliable original depths, and the model depth is visible and reliable at these pixels, so the model depth is used for depth filling, i.e., Dh(x) = Dm(x).

[0126] P20: For pixels within the conflict zone, determine the pixel-level fusion weight corresponding to the pixel, and perform a weighted summation of the original depth value and model depth value of the corresponding pixel position in the original depth image and the model depth image based on the pixel-level fusion weight to obtain the mixed depth value of the pixel within the conflict zone.

[0127] Pixels within the conflict zone (Oconflict) are those with significant differences between the original depth and the model depth, inconsistent boundaries, moderate confidence, unstable visibility, or uncertain local occlusion. For these pixels, neither the original depth nor the model depth is directly used; instead, a weighted fusion is performed, i.e.: Dh(x)=w(x) Do(x)+(1-w(x)) Dm(x) (7; Where w(x) is the pixel-level fusion weight (please refer to...) Figure 4 The depth is determined by the original depth reliability, visibility, and depth difference.

[0128] P30: For pixels in the hand region that are not identified as the pixels to be processed, retain the depth value or depth invalidation mark of the corresponding pixel position in the original depth image.

[0129] P40: Based on the depth values ​​of each pixel in the reserved area, the replacement area, and the conflict area, as well as the depth values ​​or depth invalid markers of pixel positions not identified as the pixel to be processed, the mixed depth map is generated. Figure 4 An example of a mixed depth map is given.

[0130] This application embodiment performs differentiated depth assignment and weighting processing based on pixel partitioning determination results, and seamlessly stitches them into a complete hybrid depth map. This allows the processed depth image to maintain high accuracy in the real physical world in the preserved areas, and to achieve topological structure filling in the missing areas using pure geometric priors. This effectively fills in the depth discontinuities caused by sensor hardware defects. The conflict area integrates real physical observations and geometric priors, providing a continuous, complete and geometrically reasonable data foundation for the subsequent network.

[0131] In one possible implementation, in P20, determining the pixel-level fusion weight corresponding to the pixel includes: P21: Based on the pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel-level edge consistency, and pixel-level confidence level corresponding to the pixel, determine the original depth reliability ro(x): ·E(x)·Cp(x) (8; Where Mo(x) represents the pixel depth validity of pixel x; σz(x) represents the fluctuation range of the pixel neighborhood depth corresponding to pixel x; represents the preset scale parameter used to normalize the depth fluctuation amplitude; E(x) represents the pixel-level edge consistency corresponding to pixel x; Cp(x) represents the pixel-level confidence corresponding to pixel x; exp(·) represents the exponential function.

[0132] Wherein, Mo(x) can be a binary quantity, Mo(x)=1 when the original depth value at pixel x is valid, otherwise Mo(x)=0; E(x) can be a binary quantity or a continuous score with a value range of [0,1], used to characterize the consistency between the edge of the color image and the edge of the original depth image; Cp(x) has a value range of [0,1], used to characterize the prediction reliability of the local hand region where pixel x is located.

[0133] As σz(x) increases, A decrease in ro(x) indicates that the original depth fluctuates more within the neighborhood of pixel x, and the reliability of the original depth is lower. When the original depth is valid, the neighborhood depth fluctuation is small, the edge consistency is high, and the pixel-level confidence is high, ro(x) is larger, and the mixed depth map tends to retain the original depth value at pixel x.

[0134] P22: Based on the pixel-level model depth validity, pixel-level model visibility, pixel-level confidence, and pixel depth difference corresponding to the pixel, determine the model depth reliability rm(x): (9); in, This represents the preset scale parameter used to normalize pixel depth differences.

[0135] If the original depth has holes, burrs, or discontinuities, while the model depth is visible and consistent with the surroundings, then rm(x) is larger, and the blended depth is closer to the model depth.

[0136] P23: Based on the proportional relationship between the original depth reliability and the model depth reliability, determine the pixel-level fusion weight w(x) corresponding to the original depth value: w(x)=(ro(x)+ε) / (ro(x)+rm(x)+2ε) (10); Where ro(x) represents the original depth reliability corresponding to pixel x, rm(x) represents the model depth reliability corresponding to pixel x, and ε is a minimal constant to prevent the denominator from being zero.

[0137] Accordingly, the fusion weights corresponding to the model depth values ​​are 1-w(x), that is: 1-w(x)=(rm(x)+ε) / (ro(x)+rm(x)+2ε) (11); After obtaining the pixel-level fusion weights, the blending depth value of pixel x within the conflict zone is determined according to the following formula: Dh(x)=w(x)·Do(x)+(1-w(x))·Dm(x) (12); Where Dh(x) represents the blended depth value at pixel x, Do(x) represents the original depth value at pixel x in the original depth image, and Dm(x) represents the model depth value at pixel x in the model depth image.

[0138] Therefore, when the original depth is relatively smooth, the edges are well-fitted, and the confidence level is high, ro(x) is larger, w(x) increases accordingly, and the mixed depth value is closer to the original depth value; when the original depth is distorted but the model depth reliability is high, rm(x) is larger, 1-w(x) increases accordingly, and the mixed depth value is closer to the model depth value.

[0139] This application decouples the reliability influencing factors of the original depth and the model prior at the micro level, and constructs a dynamic pixel-level fusion weight allocation algorithm that includes local depth fluctuations, edge consistency and model visibility. This enables the fusion process of conflict areas to perform adaptive soft transition calculations based on the current physical degradation degree of pixels, eliminating the abrupt changes and artifact distortions caused by different depth sources at the splicing boundary, and ensuring the smoothness and coherence of the mixed depth map in the three-dimensional spatial topology.

[0140] In one possible implementation, before correcting the initial hand parametric model parameters based on the color image and the mixed depth map (S150), the method further includes: Based on the difference between the blended depth value at each pixel location in the blended depth map and the original depth value at the corresponding pixel location in the original depth image, a depth difference map is constructed to explicitly represent the replaced or corrected depth region and its correction magnitude.

[0141] Accordingly, in S150, the initial hand parametric model parameters are corrected based on the color image and the mixed depth map to obtain the target hand parametric model parameters, including: S1501: Perform feature encoding on the color image, the original depth image, the mixed depth map, the depth difference map, and the model visibility map respectively, and perform fusion processing on the obtained encoded features to obtain fused features; S1502: Based on the fusion features and the initial hand parameterization model parameters, determine the residual correction amounts of multiple parameters and the update confidence levels corresponding to each local region of the hand; S1503: Update the initial hand parameterization model parameters based on the multiple parameter residual corrections and the update confidence level to obtain the target hand parameterization model parameters.

[0142] Please combine Figure 2 To enable the second reconstruction network to explicitly perceive which regions are corrected and the magnitude of the correction, embodiments of this application construct a depth difference map Dd based on the hybrid depth map and the original depth map. Figure 4 An example of a depth difference map is given. The depth difference map is used to characterize the local correction results of the model reprojection depth addition on the original depth, explicitly representing which pixels have been added or corrected, and the magnitude of the correction.

[0143] Specifically, for each pixel, if the depth value of the original depth map is valid, then the depth difference of the pixel is Dd(x) = Dh(x) - Do(x); if the depth value of the original depth map is invalid, then Dd(x) = Dh(x) or is set to a preset fill value as required by the implementation.

[0144] The RGB image, original depth image, blended depth map, depth difference map, and visibility map are then input into the second reconstruction network, and the parameter residual correction is output in a residual manner.

[0145] Specifically, please combine Figure 5 The second reconstruction network can adopt a multi-branch encoding-feature fusion-residual output structure.

[0146] The second reconstruction network comprises a feature encoding and fusion module and a residual and confidence output head. The feature encoding and fusion module employs a multi-path input encoding structure. The first path is an RGB encoding branch, used to extract image features such as hand texture, finger gap edges, contours, and occlusion boundaries. The second path is a raw depth encoding branch, used to extract local raw depth geometry information, holes, flying spots, and boundary defects from the raw depth map. The third path is a hybrid depth encoding branch, used to extract the hand geometry after partitioning and depth filling. The fourth path is a depth difference encoding branch, used to extract the location and correction magnitude of the depth-filled region. The fifth path is a visibility encoding branch, used to indicate to the hand network which pixels have prior model visibility and reliability, and which pixels have prior model invisibility or instability.

[0147] The feature encoding and fusion module fuses the encoded features from various channels through channel concatenation, cross-modal attention, feature pyramid fusion, or Transformer encoders to obtain the fused feature F. The fused feature F is then input into the residual and confidence output header.

[0148] The residual and confidence output headers include multiple residual output headers and confidence output headers. The multiple residual output headers include an attitude parameter residual header, a shape parameter residual header, a global pose residual header, and a keypoint residual header, used to output the corresponding residual corrections. Specifically, the attitude parameter residual header outputs the attitude parameter correction Δθ, the shape parameter residual header outputs the shape parameter correction Δβ, the global pose residual header outputs the global rotation correction Δr and the global translation correction Δt, and the keypoint residual header outputs the keypoint correction ΔJ={ΔJj}. The confidence output header outputs the updated confidence score Cu, which can include the keypoint-level confidence score Cuj and the local region-level confidence score Cup. Therefore, the obtained parameter residual corrections include shape parameter corrections, global translation corrections, global rotation corrections, and attitude parameter corrections, while also outputting keypoint corrections and updated confidence scores.

[0149] Subsequently, the initial hand parameterization model parameters are updated based on the multiple parameter residual corrections and update confidence levels to obtain the target hand parameterization model parameters.

[0150] This application embodiment constructs a hybrid depth map and a depth difference map, enabling the second reconstruction network to not only receive the repaired depth information, but also to perceive which regions have been replaced and the extent of replacement, thereby improving the effectiveness of residual correction.

[0151] In one possible implementation, in S1330, the initial hand parameterization model parameters are updated based on the multiple parameter residual corrections and the update confidence level to obtain the target hand parameterization model parameters, including: S1331: Update the initial hand parameterization model parameters based on the residual corrections of the multiple parameters to obtain intermediate corrected hand parameterization model parameters.

[0152] S1332: Based on the updated confidence level corresponding to each local region of the hand, determine the low-confidence local regions of the hand whose updated confidence level is lower than the updated confidence level threshold, and perform local closed-loop optimization on the intermediate corrected hand parameterization model parameters corresponding to the low-confidence local regions of the hand to obtain the target hand parameterization model parameters.

[0153] In the process of updating the initial hand parametric model parameters based on multiple parameter residual correction values, the initial hand parametric model parameters are first updated based on the multiple parameter residual correction values ​​to obtain intermediate corrected hand parametric model parameters. This superposition update belongs to parameter correction driven by a feedforward neural network. It mathematically weights or directly adds the initial hand parametric model parameters roughly predicted by the first reconstruction network to the residual compensation values ​​extracted by the second reconstruction network for the mixed depth map, thereby generating a corrected hand parametric model parameter that more closely approximates the physical observation facts.

[0154] Simple forward correction may not perfectly handle situations where fingers severely occlude each other. Based on this consideration, after obtaining the corrected hand parametric model parameters, the local hand regions with update confidence scores below the update confidence threshold are further extracted. Local closed-loop optimization iterations are then applied to these local hand regions to obtain the final hand parametric model parameters. Please refer to [reference needed]. Figure 2 Selective closed-loop correction in [the context].

[0155] Suppose that the updated output of the second reconstruction network includes the corrected 3D keypoint coordinates of the hand J1 and the updated confidence level Cu.

[0156] If the confidence scores Cuj for all keypoints and Cup for local regions are both higher than the corresponding update confidence thresholds, it indicates that the residual correction results of the second reconstruction network are reliable and can be directly adopted. Therefore, the hand 3D reconstruction results include: the final parameterized model parameters of the hand. Final hand 3D key point coordinates .

[0157] If at least one keypoint-level confidence score Cuj and / or local region-level confidence score Cup is lower than the update confidence threshold, it indicates that the keypoint or local region may still have occlusion, missing depth, or unstable prediction. Therefore, a low-confidence set is constructed, including a low-confidence keypoint set Lj and a low-confidence local region set Lp. The low-confidence keypoint set Lj = {j|Cuj<τc} and the low-confidence local region set Lp = {p|Cup<τp} (τc and τp are the keypoint update confidence threshold and the local region update confidence threshold, respectively). Therefore, please combine this with... Figure 6 Based on the low-confidence set, the high-confidence region and low-confidence region 502 on the unoptimized hand network 501 are determined. Then, local closed-loop optimization iteration 503 is performed through closed-loop feedback path 505 to update the low-confidence region 502, thereby obtaining the optimized hand network 504.

[0158] In one possible implementation, during local closed-loop optimization iteration, the corrected hand parametric model parameters Θ1 are used as initial values. Local closed-loop optimization iteration is performed only on the corresponding pose parameters, joint angles, and necessary translational compensations to obtain the optimal values ​​Θopt for the hand parametric model parameters, and then... Based on this, the final 3D keypoint coordinates of the hand are determined. The preferred result is obtained by re-regression using the target MANO model: (Where, Regressor represents the keypoint regression function or linear mapping matrix that comes with the MANO model), to ensure that the final parameterized hand model and the final 3D keypoint coordinates of the hand are consistent.

[0159] This application embodiment deeply combines forward network residual regression with backward selective closed-loop optimization, and uses local region update confidence as a hard decision switch to trigger optimization, realizing a highly targeted adaptive repair strategy. This selective local closed-loop optimization mechanism actively freezes the parameter matrix of high confidence regions at the algorithm level, reduces unnecessary perturbations to high confidence regions, reduces the risk of parameter drift caused by global optimization, and endows the reconstruction system with local stable tracking capability in complex occlusion scenarios.

[0160] In one possible implementation, the plurality of parameter residual corrections include shape parameter corrections, global translation corrections, global rotation corrections, and attitude parameter corrections; Accordingly, in S1331, the initial hand parameterization model parameters are updated based on the multiple parameter residual correction values ​​to obtain intermediate corrected hand parameterization model parameters, including: S13311: Linearly add the shape parameter correction amount Δβ to the initial hand shape parameter β0 in the initial hand parameterized model parameters to obtain the corrected hand shape parameters; linearly add the global translation correction amount Δt to the initial global translation parameter t0 in the initial hand parameterized model parameters to obtain the corrected global translation parameters.

[0161] For shape parameters, global translation parameters, and key point coordinates, the corrected hand shape parameters β1, corrected global translation parameters t1, and corrected hand 3D key point coordinates J1 can usually be obtained by linearly adding the "original quantity + correction quantity": β1=β0+Δβ, t1=t0+Δt, J1=J0+ΔJ.

[0162] S13312: By performing an exponential mapping operation from rotation vector to rotation matrix or a rotation composition operation, the global rotation correction Δr is combined with the initial global rotation parameter R0 in the initial hand parameterized model parameters to obtain the corrected global rotation parameter. .

[0163] For global rotation parameters, a rotation increment composition is used: ·R0, where, This represents the exponential mapping from a rotation vector to a rotation matrix.

[0164] S13313: Based on the posture parameter correction amount Δθ, the initial hand posture parameter θ0 in the initial hand parameterized model parameters is updated by linear addition or rotation compound update to obtain the corrected hand posture parameter θ1.

[0165] For attitude parameters, if axis angles or Principal Component Analysis (PCA) attitude vectors are used, they can be linearly added under small residual conditions to obtain the corrected hand attitude parameters θ1: θ1 = θ0 + Δθ. If rotation matrices or 6D rotation representations are used, rotational composite updates are performed to avoid illegal attitudes caused by direct addition of rotation spaces.

[0166] S13314: Based on the corrected hand shape parameters, the corrected global translation parameters, the corrected global rotation parameters, and the corrected hand posture parameters, intermediate corrected hand parameterized model parameters are obtained.

[0167] After updating the initial hand parametric model parameters by superimposing the residual corrections of the multiple parameters, the intermediate corrected hand parametric model parameters Θ1={θ1,β1,R1,t1} and the corrected three-dimensional key point coordinates J1 of the hand are obtained.

[0168] In another possible implementation, the corrected parametric model parameters Θ1 of the hand can be input into the MANO model to regenerate the hand mesh and the model's 3D keypoints J1_mano. The corrected 3D keypoint coordinates J1j corresponding to each pixel j are then consistently fused with the model's 3D keypoints J1_mano,j, resulting in the corrected and fused 3D keypoint coordinates J1j_fuse. For example: J1j_fuse=αjJ1j+(1-αj)J1_mano,j (13); Here, αj is the weight, which can be determined by the keypoint confidence level Cuj.

[0169] This application embodiment ensures that the pose of the finally reconstructed 3D hand model is forcibly constrained within a reasonable and effective state space that absolutely conforms to human skeletal anatomy and the real physical world by performing physically isolated differential updates on the residual correction amounts of each parameter.

[0170] In one possible implementation, S1332, the step of performing local closed-loop optimization on the intermediate-corrected hand parametric model parameters corresponding to the low-confidence local region of the hand to obtain the target hand parametric model parameters includes: S13321: Use the intermediate correction hand parameterized model parameter Θ1 as the initial value for iteration, and determine the hand joint parameters or local mesh parameters corresponding to the local area of ​​the hand as the parameters to be optimized ΘL.

[0171] S13322: Based on the intermediate corrected hand parameterized model parameter Θ1, generate the corrected model depth map Dr(Θ1), the corrected model visibility map, and the corrected model hand contour map Sr(Θ1).

[0172] S13323: Determine the observed hand contour based on at least one of the color image and the original depth image.

[0173] Specifically, in one implementation, the color image is segmented into a hand region to obtain an observed hand mask, and the observed hand mask is subjected to boundary extraction to obtain a first observed contour.

[0174] In another implementation, a depth hand region is determined based on the effective depth region and depth abrupt change edges in the original depth image, and the boundary of the depth hand region is extracted to obtain a second observation contour.

[0175] In another implementation, the first observed contour and the second observed contour are fused to obtain the third observed hand contour. The fusion includes at least one of the following: union, intersection, confidence-weighted fusion, or edge distance-based filtering.

[0176] Preferably, when the confidence level of the hand region segmentation in the color image is higher than a preset segmentation confidence threshold, the first observation contour is used as the observed hand contour; when the confidence level of the hand region segmentation in the color image is lower than the preset segmentation confidence threshold and the effective depth region in the original depth image meets a preset integrity condition, the second observation contour is used as the observed hand contour; when both are available, the third observation contour is used as the observed hand contour.

[0177] S13324: Construct a joint optimization objective function L(ΘL). The joint optimization objective function includes an observation consistency term and a hand prior constraint term; the observation consistency term includes at least one of a pixel depth consistency term Ldepth and a contour consistency term Lcontour: the pixel depth consistency term Ldepth is the pixel depth consistency between the corrected model depth map and the mixed depth map, weighted based on the corrected model visibility map; the contour consistency term Lcontour is the contour consistency between the corrected model hand contour map and the observed hand contour; the hand prior constraint term includes at least one of a hand bone length consistency term Lbone, a joint range of motion constraint term Lrange, and a self-collision penalty term Lcollision. S13325: While keeping the parameters corresponding to the local hand region whose update confidence reaches the update confidence threshold unchanged, the parameters to be optimized are iteratively adjusted with the goal of reducing the joint optimization objective function to obtain the parameterized model parameters of the target hand.

[0178] For low-confidence keypoints, the local hand region can be mapped from the low-confidence keypoint to the corresponding finger segment, palm mesh block, or model vertex set. For example, when the confidence of the distal keypoint of the index finger is low, only the distal joint angle of the index finger, the adjacent intermediate joint angle, and the corresponding mesh vertex are considered as the objects to be optimized, while the palm and other fingers are frozen or subject to strong constraints.

[0179] Specifically, the local closed-loop optimization iteration can be solved using gradient descent, Adam, L-BFGS, Levenberg-Marquardt, or other nonlinear least squares methods. The number of optimization iterations can be set to a fixed number, or it can be stopped early when the loss decreases less than a threshold. After optimization, the optimal values ​​of the hand parameterized model parameters Θopt are obtained.

[0180] The preset iteration termination conditions include at least one of the following: reaching a preset number of iterations, the decrease in the joint optimization objective function being less than a preset loss decrease threshold, or the change in the parameter to be optimized being less than a preset parameter change threshold.

[0181] In one possible implementation, selective local loop closure optimization iteration is performed based on the following objective function: L(ΘL)=λ1 Ldepth+λ2 Lcontour+λ3 Lbone+λ4 Lrange+λ5 Lcollision+λ6 Lreg (14); Wherein, Ldepth represents the pixel depth consistency term between the corrected model depth map and the hybrid depth map. The pixel depth consistency term is weighted based on the corrected model visibility map and is used to apply depth consistency constraints only to the model surfaces visible from the current camera viewpoint, avoiding the incorrect constraint of invisible model surfaces to the observation depth position. Specifically, for pixel positions with visibility of 0 in the corrected model visibility map, the corresponding depth consistency weight is 0; for pixel positions with visibility greater than 0, the corresponding depth consistency weight is determined according to the visibility and is used to weight the depth difference between the corrected model depth map and the hybrid depth map. Lcontour represents the contour consistency between the corrected model hand contour map and the observed contour. Lbone represents hand bone length consistency, Lrange represents joint range of motion constraint, Lcollision represents self-collision penalty term, Lreg represents regularization term, and λ1, λ2, λ3, λ4, λ5, and λ6 are weight coefficients, which can be obtained through validation set parameter tuning, or normalized according to the initial values ​​of each loss term, or learned using an uncertainty weighting method.

[0182] After obtaining the updated parameterized model parameters of the hand in each iteration, a new model depth map Dr(Θ) and a model hand contour map Sr(Θ) are obtained by re-rendering.

[0183] The formula for calculating the pixel depth consistency Ldepth of pixel x is as follows: (15); Where OL is the set of pixels projected onto the image plane from the local region of the hand, Vr(x) is the visibility map generated by reprojection based on the corrected parameterized hand model parameters Θ1, and ρ(·) is the robust loss function, such as SmoothL1 loss or Huber loss. By weighting the robust loss function with Vr(x), erroneous constraints on the invisible model surface can be avoided.

[0184] In contour consistency Lcontour, the observed contour can come from the boundary of the hand region mask output by the first reconstruction network, or from the fusion result of the RGB image edges and the effective boundary of the original depth map. For pixel x, the formula for calculating contour consistency Lcontour is as follows: (16); Where Sr(Θ) is the corrected model hand contour map, Sobs is the observed contour, and Dist(·) is the distance from pixel x to the nearest observed contour point.

[0185] Lbone, a hand bone length consistency parameter, is used to constrain the optimized bone lengths so that they do not deviate from the bone lengths estimated by the shape parameters of the parametric hand model or the user's hand shape. (17); Where B is the set of hand skeleton connections, Ji(Θ) is the coordinate of key point i, Jj(Θ) is the coordinate of key point j, and lij is the reference bone segment length corresponding to the bone segment length between key point i and key point j. The reference bone segment length can be calculated from the skeleton generated by the shape parameter β1 of the parameterized hand model, or it can be estimated from the statistical average of the training set or during the user initialization stage.

[0186] The Lrange joint range of motion constraint is used to limit the joint angle to a reasonable range of motion. (18); in, θkmin and θkmax represent the actual range of motion of joint k; they can be obtained based on the posture prior of the parameterized hand model, the statistical range of joint angles in the training data, or the anatomical constraints of the human hand.

[0187] The self-collision penalty term Lcollision is used to prevent unreasonable interpenetration between fingers and between fingers and the palm. It can be calculated using the minimum distance between mesh triangular faces, the capsule approximation model, or the symbolic distance field. For example, when the distance between two non-adjacent local mesh blocks is less than a safety threshold, it is included in the self-collision penalty term. (19); Where C is the pair of mesh blocks that need to be detected for collision (e.g., mesh blocks a and b), d(Ma,Mb) is the minimum distance between two mesh blocks a and b, and dsafe is the safe distance threshold.

[0188] The regularization term Lreg is used to prevent global parameter drift caused by low-confidence local optimization. (20); Where ΘL represents the current parameter to be optimized during the iteration process, and ΘL,0 represents the initial value of the parameter to be optimized at the start of the local closed-loop optimization. The initial value of the parameter to be optimized, ΘL,0, is extracted from the hand joint parameters or local mesh parameters corresponding to the low-confidence local region of the intermediate calibration hand parameterized model parameter Θ1.

[0189] With the goal of minimizing the joint optimization objective function L(ΘL), the parameters to be optimized are iteratively adjusted, and the parameters of the hand parameterized model after iteration are used as the final parameters of the hand parameterized model.

[0190] This application innovatively links and nonlinearly fuses data fidelity constraints (pixel depth consistency and contour consistency) reflecting 2D visual fit with hard priors of anatomical structure (hand skeleton length, joint range of motion constraints, and self-collision penalty for preventing clipping) reflecting the physical limits of the human body in three dimensions. This constructs a rigorous joint optimization objective function, ensuring that each local micro-gesture posture in the final output not only fits the real edge details in image observation, but also follows the limits of real human skeletal physiological activity in the three-dimensional physical space. This avoids fatal geometric collapse defects in high-precision reconstruction fields, such as reverse finger breakage and multiple fingers clipping and merging with each other, in virtual hand models.

[0191] Based on the above, this application also provides a hand three-dimensional reconstruction device. The hand three-dimensional reconstruction device and the above-described hand three-dimensional reconstruction method can be referred to in correspondence with each other.

[0192] As an example, such as Figure 7 As shown, the hand three-dimensional reconstruction device provided in this application includes: The initial parameter acquisition module 710 is used to acquire initial hand parameterized model parameters based on camera intrinsic parameters, color image, and original depth image aligned with the color image in pixel coordinate system. The projection module 720 is used to project the initial three-dimensional mesh of the hand, represented by the parameters of the initial parametric hand model, onto the two-dimensional image corresponding to the color image according to the camera intrinsic parameters, and generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial three-dimensional hand mesh relative to the current camera at each pixel position; The pixel partitioning module 730 is used to divide the pixels to be processed in the hand region into multiple pixel partitions based on the original depth image, the model depth map, and the model visibility map; wherein the multiple pixel partitions include a retention area, a replacement area, and a conflict area; The hybrid depth map generation module 740 is used to generate a hybrid depth map based on the reserved area, the replacement area, and the conflict area; The parameter correction module 750 is used to correct the initial hand parameterization model parameters based on the color image and the mixed depth map to obtain the target hand parameterization model parameters.

[0193] This application's embodiments inversely map the roughly estimated initial parametric model parameters of the hand to prior hand depth information in a two-dimensional view, performing reprojection depth completion only within the hand region to avoid irrelevant error propagation caused by general depth completion across the entire scene. Furthermore, it utilizes this prior information in conjunction with the original data to construct a three-state pixel partitioning mechanism including a reserved area, a replacement area, and a conflict area, robustly addressing depth holes, boundary defects, and local occlusion, achieving refined separation of reliable observations from anomalous noise in the original depth image. By using partitioned differential processing to obtain a hybrid depth map, it addresses the depth holes and boundary defects easily generated by consumer-grade depth sensors in scenarios with hand occlusion and rapid movement. The high-quality hybrid depth map guides subsequent parameter correction updates, ensuring that the estimation of the final parametric model parameters is free from interference from unreliable features, significantly improving the robustness and spatial positioning accuracy of the hand reconstruction algorithm in complex scenes. Under conditions of complex interaction, close-range shooting, boundary occlusion, and low-quality depth, this application's embodiments can improve the parameter estimation accuracy and key point positioning stability of the parametric hand model, demonstrating significant engineering application value.

[0194] Based on the above, this application also provides a method for training a three-dimensional reconstruction network for the hand, wherein the three-dimensional reconstruction network for the hand includes a first reconstruction network and a second reconstruction network.

[0195] The training data for the first reconstruction network includes RGB images, depth images, camera intrinsics, 3D keypoint annotations for the hand, MANO parameter annotations, hand region masks or contour annotations, and keypoint visibility annotations. Data sources can include: real RGB-D acquired data, data generated by multi-view or motion capture system calibration, publicly available RGB-D hand or hand-object interaction datasets, and synthetic RGB-D data rendered based on the MANO model. Before training, the depth images are preprocessed with random holes, boundary defects, flying spot noise, occlusion blocks, depth jitter, and reflection distortion to obtain preprocessed depth sample images, ensuring that the first reconstruction network can still output stable initial results under low-quality depth conditions.

[0196] During the training phase, the first and second reconstruction networks can be trained jointly using 3D keypoint loss, MANO parameter loss, depth consistency loss, contour consistency loss, and confidence supervision loss. This can be achieved through phased training, joint training, or a combination of both. During training, the enhanced depth map can be used as the network input, while the unenhanced or highly accurate labeled keypoints and MANO parameters serve as the supervision targets.

[0197] In the phased training approach, the first reconstruction network is trained first. Training samples include RGB images, depth images, camera intrinsics, ground truth values ​​of 3D keypoints of the hand, ground truth values ​​of MANO parameters, ground truth values ​​of hand region masks or contours, and ground truth values ​​of keypoint visibility. For data without ground truth values ​​of MANO parameters, pseudo-labels can be obtained by fitting the MANO model through multi-view triangulation, motion capture results, or known 3D keypoints.

[0198] The loss function LN1 of the first reconstruction network is expressed as follows: LN1=μ1 LJ3D+μ2 LJ2D+μ3 LMANO+μ4 Lmask+μ5 Lconf(21); Among them, LJ3D is the 3D keypoint loss, LJ2D is the keypoint reprojection loss, LMANO is the MANO parameter loss, Lmask is the hand region mask or contour loss, and Lconf is the confidence supervision loss. The confidence supervision target can be automatically generated based on the prediction error. For example, when the prediction error of the j-th keypoint is less than a threshold, its confidence label is set to 1; otherwise, it is set to 0. Alternatively, it can be set to... , where ej is the keypoint error. μ1, μ2, μ3, μ4, and μ5 represent weighting coefficients, which can be obtained by parameter tuning of the validation set, or set according to the depth camera noise model, keypoint error statistics, and the numerical scale of each loss term.

[0199] After the first reconstruction network is trained, it is frozen and used to generate initial MANO parameters, initial keypoints, and initial confidence scores on the training samples. It is then used to further generate model depth maps, visibility maps, hand contour maps, blended depth maps, and depth difference maps. The second reconstruction network is then trained. The supervision objective of the second reconstruction network is the residual ΔΘgt relative to the output of the first reconstruction network. ΔΘgt=Θgt⊖Θ0, ΔJgt=Jgt-J0 (22); Where ΔΘgt represents the residual parameters of the hand parameterized model, Θgt represents the final hand parameterized model parameters output by the second reconstruction network during training, ⊖ represents the parameter difference, ΔJgt represents the residual coordinates of the hand's 3D keypoints, and Jgt represents the final coordinates of the hand's 3D keypoints output by the second reconstruction network during training.

[0200] The loss function LN2 of the second reconstruction network is expressed as: LN2=ν1 LΔΘ+ν2 LΔJ+ν3 LJ3Dupd+ν4 Ldepthupd+ν5 Lconfupd (23); Where LΔΘ is the parameter residual loss, LΔJ is the keypoint residual loss, LJ3Dupd is the updated keypoint loss, Ldepthupd is the depth consistency loss between the updated model and the hybrid depth map, Lconfupd is the update confidence loss, and ν1, ν2, ν3, ν4, and ν5 are weight coefficients, which can be obtained by parameter tuning of the validation set, or set according to the depth camera noise model, keypoint error statistics, and the numerical scale of each loss term.

[0201] In the joint training approach, after the first and second reconstruction networks are pre-trained separately, the first reconstruction network, model reprojection, pixel partitioning, the blended depth generation module, and the second reconstruction network are combined into a single training graph. If reprojection uses differentiable rendering and pixel partitioning uses continuous soft weights, the depth consistency loss and contour consistency loss can be backpropagated to the first and second reconstruction networks. If pixel partitioning uses hard thresholding, the gradient can be stopped on the partitioning result, and only the network parameters before and after partitioning can be trained. If a soft mask is used, backpropagation can be achieved through differentiable rendering and soft blending weights.

[0202] The total loss Ltotal from joint training can be expressed as: Ltotal=LN1+LN2+eta1Ldepth+eta2Lcontour+eta3Lprior+eta4Lconf (24); Lprior includes prior constraints such as bone length, joint range of motion, and self-collision. η1, η2, η3, and η4 are weighting coefficients, which can be obtained by parameter tuning of the validation set or set according to the depth camera noise model, key point error statistics, and the numerical scale of each loss term.

[0203] Through the above training, the first reconstruction network can output a stable initial estimate under low-quality depth input, and the second reconstruction network can learn the residual correction capability for the depth-filling region and the difference region.

[0204] Based on the above, such as Figure 8 As shown, the training method for joint training provided in this application includes: S810: Obtain training samples, which include sample camera intrinsic parameters, color sample images, depth sample images aligned with the color sample images in the pixel coordinate system, and ground truth parameters of the hand parameterized model, wherein the depth sample images are obtained by enhancing the depth defects of the original depth sample images. S820: Input the sample camera intrinsic parameters, the color sample image, and the depth sample image into the first reconstruction network to obtain the initial hand parameter prediction values; S830: Based on the intrinsic parameters of the sample camera, project the initial hand 3D mesh represented by the initial hand parameter prediction value onto the 2D image corresponding to the color sample image to generate a model depth sample map and a model visibility sample map; S840: Based on the depth sample image, the model depth sample map, and the model visibility sample map, the sample pixels in the hand area are divided into a retention area, a replacement area, or a conflict area, and a mixed depth sample map is generated based on the different sample pixel partitions. S850: Generate a depth difference sample map based on the difference between the depth values ​​at corresponding pixel positions in the hybrid depth sample map and the depth sample image; S860: Input the color sample image, the depth sample image, the mixed depth sample image, the depth difference sample image, the model visibility sample image, and the initial hand parameter prediction value into the second reconstruction network to obtain the parameter residual prediction value, and obtain the corrected hand parameter prediction value based on the parameter residual prediction value and the initial hand parameter prediction value. S870: Based on the difference between the initial predicted hand parameters and the true parameters of the hand parameterized model, and the difference between the corrected predicted hand parameters and the true parameters of the hand parameterized model, construct the training loss; S880: Based on the training loss, perform phased training or joint training on the first reconstruction network and the second reconstruction network until the preset convergence condition is met, and obtain the trained hand 3D reconstruction network.

[0205] This application's embodiments inversely map the roughly estimated initial parametric model parameters of the hand to prior hand depth information in a two-dimensional view, performing reprojection depth completion only within the hand region to avoid irrelevant error propagation caused by general depth completion across the entire scene. Furthermore, it utilizes this prior information in conjunction with the original data to construct a three-state pixel partitioning mechanism including a reserved area, a replacement area, and a conflict area, robustly addressing depth holes, boundary defects, and local occlusion, achieving refined separation of reliable observations from anomalous noise in the original depth image. By using partitioned differential processing to obtain a hybrid depth map, it addresses the depth holes and boundary defects easily generated by consumer-grade depth sensors in scenarios with hand occlusion and rapid movement. The high-quality hybrid depth map guides subsequent parameter correction updates, ensuring that the estimation of the final parametric model parameters is free from interference from unreliable features, significantly improving the robustness and spatial positioning accuracy of the hand reconstruction algorithm in complex scenes. Under conditions of complex interaction, close-range shooting, boundary occlusion, and low-quality depth, this application's embodiments can improve the parameter estimation accuracy and key point positioning stability of the parametric hand model, demonstrating significant engineering application value.

[0206] Figure 9 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute the aforementioned hand 3D reconstruction method and / or implement the aforementioned hand 3D reconstruction network training method.

[0207] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0208] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the hand three-dimensional reconstruction method provided in the above embodiments and / or implement the above-described hand three-dimensional reconstruction network training method.

[0209] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the hand three-dimensional reconstruction method provided in the above embodiments and / or to implement the training method of the hand three-dimensional reconstruction network described above.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0211] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for three-dimensional reconstruction of a hand, characterized in that, include: Based on camera intrinsics, color images, and original depth images aligned with the color images in pixel coordinates, initial hand parameterization model parameters are obtained. Based on the camera intrinsic parameters, the initial 3D mesh of the hand, represented by the initial parametric model parameters, is projected onto the 2D image corresponding to the color image to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial 3D mesh of the hand relative to the current camera at each pixel position; Based on the original depth image, the model depth map, and the model visibility map, the pixels to be processed in the hand region are divided into multiple pixel partitions; wherein, the multiple pixel partitions include a retention area, a replacement area, and a conflict area; A mixed depth map is generated based on the multiple pixel partitions; The initial hand parametric model parameters are corrected based on the color image and the mixed depth map to obtain the target hand parametric model parameters. The hand is then reconstructed based on the target hand parametric model parameters to obtain the position of the 3D hand mesh in the camera coordinate system.

2. The method for three-dimensional reconstruction of the hand according to claim 1, characterized in that, The step of projecting the initial 3D mesh of the hand, represented by the initial parametric model parameters, onto the 2D image corresponding to the color image based on the camera intrinsic parameters, to generate a model depth map and a model visibility map with the same pixel coordinate system as the original depth image, includes: Based on the camera intrinsic parameters, multiple mesh patches in the initial three-dimensional hand mesh are projected onto the two-dimensional image coordinate system corresponding to the color image to obtain the projection area of ​​each mesh patch; The projected grid patches are rasterized to determine the pixel positions covered by each grid patch, and the candidate model depth value is calculated for the pixel positions covered by the grid patches. When multiple candidate model depth values ​​correspond to the same pixel location, the candidate model depth value closest to the current camera optical center is selected by using a depth buffer method as the model depth value of the pixel location to generate the model depth map; Based on the depth test results of each mesh patch at the corresponding pixel position, and the relationship between the normal of each mesh patch and the current camera viewing direction, the visibility at each pixel position is determined to generate the model visibility map.

3. The method for three-dimensional reconstruction of the hand according to claim 1, characterized in that, Based on the original depth image, the model depth map, and the model visibility map, the pixels to be processed within the hand region are divided into multiple pixel partitions, including: The pixels with valid original depth values ​​in the original depth image and the pixels with valid model depth values ​​in the model depth image and whose visibility in the model visibility image reaches a preset visibility threshold are identified as the pixels to be processed. Based on the original depth image, the model depth map, and the model visibility map, multiple evaluation parameters are determined for each pixel to be processed; Based on the plurality of evaluation parameters corresponding to each pixel to be processed, the pixel to be processed is divided into the reserved area, the replacement area, or the conflict area.

4. The method for three-dimensional reconstruction of the hand according to claim 3, characterized in that, Before determining the multiple evaluation parameters corresponding to each pixel to be processed, the method further includes: Based on the projection coverage area of ​​the initial three-dimensional hand mesh in the two-dimensional image, a model hand contour map is generated; Based on the color image and the original depth image, initial three-dimensional key points of the hand and the initial confidence level corresponding to the initial three-dimensional key points of the hand are obtained; Accordingly, determining the multiple evaluation parameters corresponding to each pixel to be processed includes: Determine the pixel depth validity and pixel neighborhood depth fluctuation amplitude corresponding to the pixel to be processed in the original depth image; The pixel depth difference is determined based on the difference between the original depth value corresponding to the pixel to be processed in the original depth image and the model depth value of the corresponding pixel position in the model depth image; The initial confidence scores corresponding to the initial three-dimensional key points of the hand are mapped to the corresponding pixel positions within the hand region to obtain pixel-level confidence scores. Pixel-level edge consistency is determined based on the positional relationship between the edge pixel positions in the color image and the edge pixel positions in the original depth image. The validity of the pixel-level model depth is determined based on whether the corresponding pixel position in the model depth map has a valid model depth value. Pixel-level model visibility is determined based on the visibility of the corresponding pixel position in the model visibility map. Based on the positional deviation between the model hand contour map and the edge pixel positions in the color image, and the positional deviation between the model hand contour map and the edge pixel positions in the original depth image, the boundary burr area and the boundary defect area within the hand region are determined. The pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel depth difference, pixel-level confidence, pixel-level edge consistency, pixel-level model depth validity, pixel-level model visibility, boundary spur region, and boundary defect region corresponding to the same pixel to be processed are used as multiple evaluation parameters corresponding to the pixel to be processed.

5. The method for three-dimensional reconstruction of the hand according to claim 4, characterized in that, The step of dividing the pixel to be processed into the reserved region, the replacement region, or the conflict region based on the plurality of evaluation parameters corresponding to each pixel to be processed includes: The pixel to be processed is assigned to the reserved area if any of the following conditions are met: The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary spur area, and the pixel-level model depth validity is invalid; The pixel depth validity of the pixel to be processed is valid, the pixel to be processed is not in the boundary spur area, the pixel-level model depth validity is valid, and the pixel-level model visibility does not reach the preset visibility threshold. The pixel depth validity is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, the pixel depth difference does not exceed the depth difference threshold, the pixel-level confidence reaches the local confidence threshold, the pixel-level edge consistency meets the preset consistency condition, and the pixel to be processed is not in the boundary burr area. When the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed meets at least one of the following conditions, the pixel to be processed is assigned to the replacement region: The pixel depth validity is invalid; The original depth value corresponding to the pixel to be processed is a hole value or an anomaly value; The pixel neighborhood depth fluctuation amplitude exceeds the depth fluctuation threshold; The pixel to be processed is located within the boundary defect area; When the pixel depth validity is valid, the pixel-level model depth validity is valid, the pixel-level model visibility reaches the preset visibility threshold, and the pixel to be processed is not assigned to the reserved area or the replacement area, the pixel to be processed is assigned to the conflict area.

6. The method for three-dimensional reconstruction of the hand according to claim 5, characterized in that, The generation of the blended depth map based on the multiple pixel partitions includes: The depth value of each pixel position within the reserved area is determined as the original depth value of the corresponding pixel position in the original depth image; The depth value of each pixel position within the replacement area is determined as the model depth value of the corresponding pixel position in the model depth map; For pixels within the conflict zone, determine the pixel-level fusion weight corresponding to the pixel, and based on the pixel-level fusion weight, perform a weighted summation of the original depth value and the model depth value at the corresponding pixel position in the original depth image and the model depth image to obtain the mixed depth value of the pixel within the conflict zone. For pixels within the hand region that are not identified as the pixels to be processed, retain the depth value or depth invalidation marker of the corresponding pixel position in the original depth image; The mixed depth map is generated based on the depth values ​​of each pixel position in the reserved area, the replacement area, and the conflict area, as well as the depth values ​​or depth invalid markers of pixel positions that are not identified as the pixel to be processed.

7. The method for three-dimensional reconstruction of the hand according to claim 6, characterized in that, Determining the pixel-level fusion weight corresponding to the pixel includes: The reliability of the original depth is determined based on the pixel depth validity, pixel neighborhood depth fluctuation amplitude, pixel-level edge consistency, and pixel-level confidence level corresponding to the pixel. The reliability of the model depth is determined based on the pixel-level model depth validity, pixel-level model visibility, pixel-level confidence, and pixel depth difference corresponding to the pixel. The pixel-level fusion weights corresponding to the original depth values ​​are determined based on the proportional relationship between the original depth reliability and the model depth reliability.

8. The method for three-dimensional reconstruction of the hand according to claim 1, characterized in that, Before correcting the parameters of the initial hand parametric model based on the color image and the mixed depth map, the method further includes: A depth difference map is constructed based on the difference between the blended depth value of each pixel position in the blended depth map and the original depth value of the corresponding pixel position in the original depth image; Accordingly, the step of correcting the initial hand parametric model parameters based on the color image and the mixed depth map to obtain the target hand parametric model parameters includes: Feature encoding is performed on the color image, the original depth image, the mixed depth map, the depth difference map, and the model visibility map, respectively, and the obtained encoded features are fused to obtain fused features; Based on the fusion features and the initial hand parameterization model parameters, determine the residual correction amounts of multiple parameters and the update confidence levels corresponding to each local region of the hand; The parameters of the initial hand parameterized model are updated based on the residual corrections of the multiple parameters and the updated confidence level to obtain the parameters of the target hand parameterized model.

9. The method for three-dimensional reconstruction of the hand according to claim 8, characterized in that, The step of updating the initial hand parameterization model parameters based on the multiple parameter residual corrections and the updated confidence level to obtain the target hand parameterization model parameters includes: The initial hand parameterization model parameters are updated based on the residual correction values ​​of the multiple parameters to obtain the intermediate corrected hand parameterization model parameters; Based on the updated confidence levels corresponding to each local hand region, low-confidence local hand regions with updated confidence levels below the updated confidence level threshold are identified. Local closed-loop optimization is then performed on the intermediate corrected hand parameterization model parameters corresponding to the low-confidence local hand regions to obtain the target hand parameterization model parameters.

10. The method for three-dimensional reconstruction of the hand according to claim 9, characterized in that, The step of performing local closed-loop optimization on the intermediate-corrected hand parametric model parameters corresponding to the low-confidence local region of the hand to obtain the target hand parametric model parameters includes: The intermediate-calibrated hand parameterized model parameters are used as the initial values ​​for iteration, and the hand joint parameters or local mesh parameters corresponding to the low-confidence local region of the hand are determined as the parameters to be optimized. Based on the intermediate corrected hand parameterized model parameters, a corrected model depth map, a corrected model visibility map, and a corrected model hand contour map are generated. The observed hand contour is determined based on at least one of the color image and the original depth image; A joint optimization objective function is constructed, comprising an observation consistency term and a hand prior constraint term. The observation consistency term includes at least one of a pixel depth consistency term and a contour consistency term, wherein the pixel depth consistency term is the pixel depth consistency between the corrected model depth map and the mixed depth map, weighted based on the corrected model visibility map; the contour consistency term is the contour consistency between the corrected model hand contour map and the observed hand contour; and the hand prior constraint term includes at least one of a hand bone length consistency term, a joint range of motion constraint term, and a self-collision penalty term. While keeping the parameters corresponding to the local hand region whose update confidence reaches the update confidence threshold unchanged, the parameters to be optimized are iteratively adjusted with the goal of reducing the joint optimization objective function, so as to obtain the parameterized model parameters of the target hand.

11. The method for three-dimensional reconstruction of the hand according to claim 8, characterized in that, The residual corrections for the multiple parameters include shape parameter corrections, global translation corrections, global rotation corrections, and attitude parameter corrections. The step of updating the initial hand parameterization model parameters based on the multiple parameter residual corrections to obtain intermediate corrected hand parameterization model parameters includes: The shape parameter correction amount is linearly added to the initial hand shape parameter in the initial hand parameterized model parameters to obtain the corrected hand shape parameter; the global translation correction amount is linearly added to the initial global translation parameter in the initial hand parameterized model parameters to obtain the corrected global translation parameter. By performing an exponential mapping operation from rotation vector to rotation matrix or a rotation composition operation, the global rotation correction is combined with the initial global rotation parameters in the initial hand parameterization model parameters to obtain the corrected global rotation parameters. Based on the posture parameter correction amount, the initial hand posture parameters in the initial hand parameterized model parameters are updated by linear addition or rotation compound update to obtain the corrected hand posture parameters. Based on the corrected hand shape parameters, the corrected global translation parameters, the corrected global rotation parameters, and the corrected hand posture parameters, intermediate corrected hand parameterized model parameters are obtained.

12. The method for three-dimensional reconstruction of the hand according to claim 4, characterized in that, Before determining the pixels to be processed within the hand region, the process further includes: Obtain a hand region mask based on the color image and the original depth image; The hand region is determined based on the hand region mask and the model hand contour map.

13. The method for three-dimensional reconstruction of the hand according to claim 12, characterized in that, Determining the hand region based on the hand region mask and the model hand contour map includes: The morphological expansion process is performed on the hand contour of the model according to the preset expansion radius. The candidate hand region is obtained by determining the union of the expanded model hand contour map and the hand region mask; Based on at least one of the edges in the color image and the edges in the original depth image, the boundary of the candidate hand region is aligned and corrected to obtain the hand region.

14. A three-dimensional reconstruction device for a hand, characterized in that, include: The initial parameter acquisition module is used to acquire initial hand parameterization model parameters based on camera intrinsic parameters, color image, and original depth image aligned with the color image in pixel coordinate system; The projection module is used to project the initial three-dimensional mesh of the hand, represented by the parameters of the initial parametric hand model, onto the two-dimensional image corresponding to the color image based on the camera intrinsic parameters, thereby generating a model depth map and a model visibility map with the same pixel coordinate system as the original depth image; wherein, the model visibility map is used to represent the visibility of the model surface corresponding to the initial three-dimensional hand mesh relative to the current camera at each pixel position; A pixel partitioning module is used to divide the pixels to be processed in the hand region into multiple pixel partitions based on the original depth image, the model depth map, and the model visibility map; wherein the multiple pixel partitions include a retention area, a replacement area, and a conflict area; A hybrid depth map generation module is used to generate a hybrid depth map based on the reserved area, the replacement area, and the conflict area; The parameter correction module is used to correct the initial hand parameterization model parameters based on the color image and the mixed depth map to obtain the target hand parameterization model parameters.

15. A method for training a three-dimensional reconstruction network for the hand, characterized in that, The hand 3D reconstruction network includes a first reconstruction network and a second reconstruction network, and the method includes: Training samples are obtained, including sample camera intrinsic parameters, color sample images, depth sample images aligned with the color sample images in the pixel coordinate system, and ground truth parameters of the hand parameterized model; wherein, the depth sample images are obtained by performing depth defect enhancement on the original depth sample images. The sample camera intrinsic parameters, the color sample image, and the depth sample image are input into the first reconstruction network to obtain the initial hand parameter prediction values; Based on the intrinsic parameters of the sample camera, the initial three-dimensional mesh of the hand, represented by the predicted initial hand parameters, is projected onto the two-dimensional image corresponding to the color sample image to generate a model depth sample map and a model visibility sample map. Based on the depth sample image, the model depth sample map, and the model visibility sample map, the sample pixels in the hand area are divided into a reserved area, a replacement area, or a conflict area, and a hybrid depth sample map is generated based on the different sample pixel partitions. A depth difference sample map is generated based on the difference between the depth values ​​at corresponding pixel positions in the hybrid depth sample map and the depth sample image. The color sample image, the depth sample image, the mixed depth sample image, the depth difference sample image, the model visibility sample image, and the initial hand parameter prediction value are input into the second reconstruction network to obtain the parameter residual prediction value, and the corrected hand parameter prediction value is obtained based on the parameter residual prediction value and the initial hand parameter prediction value. Based on the differences between the initial predicted hand parameters and the ground truth parameters of the hand parameterized model, and the differences between the corrected predicted hand parameters and the ground truth parameters of the hand parameterized model, a training loss is constructed. Based on the training loss, the first reconstruction network and the second reconstruction network are trained in stages or jointly until the preset convergence condition is met, and the trained hand 3D reconstruction network is obtained.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the hand three-dimensional reconstruction method according to any one of claims 1 to 13 and / or the hand three-dimensional reconstruction network training method according to claim 15.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hand three-dimensional reconstruction method according to any one of claims 1 to 13 and / or the hand three-dimensional reconstruction network training method according to claim 15.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hand three-dimensional reconstruction method according to any one of claims 1 to 13 and / or the hand three-dimensional reconstruction network training method according to claim 15.

Images