Hand three-dimensional mesh generation method and system based on bone constraint and scale estimation

Abstract

The application belongs to the technical field of image data processing, and discloses a hand three-dimensional grid generation method and system based on skeleton constraint and scale estimation, which comprises the following steps: acquiring a hand image, and performing hand key point detection to obtain a plurality of hand key points and three-dimensional pixel coordinates of each hand key point; performing skeleton segment segmentation on a predefined hand template grid, and combining with finger joint space position segmentation to obtain a plurality of hand local sub-grids; estimating hand depth information; based on hand image segmentation and projection mask matching, combining with hand depth information, solving geometric scale parameters of each local sub-grid, correcting key point coordinates according to skeleton physiological constraint, and combining geometric scale parameters and mapping relationship, deforming and combining each local sub-grid to generate a hand three-dimensional grid. The application has better deployment adaptability, and is suitable for mobile terminals, AR devices and edge computing platforms.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method and system for generating three-dimensional meshes of the hand based on skeletal constraints and scale estimation. Background Technology

[0002] With the development of augmented reality (AR), virtual reality (VR), digital human interaction, human-computer interaction, and mobile vision applications, vision-based hand tracking technology has been widely used.

[0003] Existing methods for 3D hand reconstruction mainly include: a) methods based on parametric models, such as the MANO model, which generate hand meshes by optimizing unit direction vectors, shape parameters, and global transformation parameters; b) methods based on deep learning to directly predict hand meshes, which directly regress 3D vertices through neural networks.

[0004] Among them, the parametric model method, due to its strong constraints on human body structure, is widely used in the fields of digital humans and virtual interaction.

[0005] However, existing technologies still have the following defects and shortcomings: Parametric models rely on global shape parameters and are difficult to adapt to local geometric changes; MANO optimization processes typically rely on a large number of iterative calculations, which makes it difficult to meet the real-time operation requirements of mobile terminals and edge devices. Depth errors in 3D keypoints cause geometric distortion of MANO's mesh (unnatural finger bending). MANO parametric models are computationally complex and time-consuming to render. Deep learning mesh prediction models are computationally intensive and difficult to deploy in AR glasses, mobile terminals, and edge computing devices.

[0006] Therefore, how to provide a method and system for generating 3D hand meshes based on skeletal constraints and scale estimation is an urgent problem to be solved. Summary of the Invention

[0007] This invention provides a method and system for generating 3D hand meshes based on skeletal constraints and scale estimation, in order to solve the problems mentioned above in the prior art.

[0008] According to a first aspect of the present invention, a method for generating a three-dimensional mesh of a hand based on skeletal constraints and scale estimation is provided.

[0009] In one embodiment, a method for generating a 3D mesh of a hand based on skeletal constraints and scale estimation includes the following steps: A hand image is acquired, and a lightweight detection network is used to detect key points of the hand image to obtain several key points of the hand and the three-dimensional pixel coordinates of each key point. Based on the preset skinning weights, the predefined hand template mesh is segmented into bone segments, and combined with the spatial position of the finger joints, multiple local sub-meshes of the hand are obtained to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment. Based on the three-dimensional pixel coordinates of key points of the hand, the unit direction vector of each bone segment is calculated, and the depth information of the hand is estimated based on the camera imaging parameters and the pixel scale of the hand in the image. Based on hand image segmentation and projection mask matching, and combined with the depth information of the hand, the geometric scale parameters of each local sub-mesh are solved. The coordinates of key points are corrected according to the physiological constraints of the skeleton. Combining the geometric scale parameters and mapping relationship, the local sub-mesh is deformed and combined to generate a 3D mesh of the hand.

[0010] According to a second aspect of the present invention, a three-dimensional mesh generation system for the hand based on skeletal constraints and scale estimation is provided.

[0011] In one embodiment, a hand 3D mesh generation system based on skeletal constraints and scale estimation includes: The image detection module is used to acquire hand images and perform hand keypoint detection on the hand images based on a lightweight detection network to obtain several hand keypoints and the three-dimensional pixel coordinates of each hand keypoint. The relation mapping module is used to segment the predefined hand template mesh into bone segments according to the preset skin weights, and combine the spatial position of the finger joints to obtain multiple local sub-mesh of the hand, so as to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment; The depth calculation module is used to calculate the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of the key points of the hand, and to estimate the depth information of the hand based on the camera imaging parameters and the pixel scale of the hand in the image. The mesh generation module is used to solve the geometric scale parameters of each local sub-mesh based on hand image segmentation and projection mask matching, combined with the depth information of the hand, correct the key point coordinates according to the skeletal physiological constraints, and combine the geometric scale parameters and mapping relationship to deform and combine each local sub-mesh to generate a 3D mesh of the hand.

[0012] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: 1. This invention adopts a reconstruction method based on 21 hand key points to directly drive segmented meshes. By establishing a mapping relationship between key points and local meshes, the pose transformation of each local mesh is realized without the need for MANO model parameter optimization. Compared with traditional parametric reconstruction methods, this invention simplifies the hand mesh generation process, reduces computational complexity, improves real-time reconstruction efficiency, and has better deployment adaptability, making it suitable for mobile devices, AR devices, and edge computing platforms.

[0013] 2. This invention projects an initialized full-hand template mesh onto an image plane, and performs segmentation masking with the real hand. Loss optimization involves jointly estimating the scaling factors of each local sub-mesh to achieve the best match between the projected contour and the real segmentation contour at the pixel level, thereby indirectly and accurately deducing the local width geometry of the fingers and palm. This initialization process only needs to be executed once in the first frame. Subsequent frames fix the width scale parameters of each region and only update the spatial position and length of the local mesh, which ensures personalized hand shape adaptation without affecting the real-time frame rate.

[0014] 3. This invention supports the generation of local meshes by selecting target areas according to application requirements, such as individual finger segment meshes, finger and palm combination meshes, etc., and is driven independently through the corresponding skeletal structure. Compared with the complete Hand Mesh rendering method, this invention reduces the number of vertices and bone transformations involved in the calculation, reduces the amount of rendering computation, and improves the real-time rendering efficiency of mobile devices.

[0015] 4. This invention establishes a hand skeleton topology structure, transforms the connection relationship between key points into a skeleton chain with fixed structural constraints, and maintains the following during the Mesh generation process: skeleton connection relationship; correct finger bending direction; skeleton length constraint; avoids unreasonable deformation caused by directly using absolute three-dimensional coordinates with errors. Therefore, even if there is a certain Z-direction error in the input key points, it can maintain the finger posture that conforms to the human body structure law, and improve the reconstruction stability under complex action conditions.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 This is a flowchart illustrating a method for generating a 3D mesh of a hand based on skeletal constraints and scale estimation, according to an exemplary embodiment. Figure 2This is a block diagram illustrating the principle of a hand 3D mesh generation system based on skeletal constraints and scale estimation, according to an exemplary embodiment. Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0019] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0020] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0021] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0022] Figure 1 An embodiment of the hand 3D mesh generation method based on skeletal constraints and scale estimation of the present invention is shown.

[0023] In this optional embodiment, the method for generating a 3D mesh of the hand based on skeletal constraints and scale estimation includes the following steps: Step S101: Obtain a hand image and perform hand keypoint detection on the hand image based on a lightweight detection network to obtain several hand keypoints and the three-dimensional pixel coordinates of each hand keypoint. Step S102: According to the preset skinning weight, the predefined hand template mesh is segmented into bone segments, and combined with the spatial position of the finger joints, multiple local sub-mesh of the hand are obtained to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment. When performing subsequent skinning deformation, the unit direction vector of the bone segment corresponding to each local sub-mesh is called according to the binding mapping relationship to perform skinning calculation, so that each local sub-mesh deforms synchronously with its bound bone segment. Step S103: Calculate the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of the key points of the hand, and estimate the depth information of the hand based on the camera imaging parameters and the pixel scale of the hand in the image. Step S104: Based on hand image segmentation and projection mask matching, and combined with the depth information of the hand, solve the geometric scale parameters of each local sub-mesh, correct the key point coordinates according to the skeletal physiological constraints, and combine the geometric scale parameters and mapping relationship to deform and combine each local sub-mesh to generate a three-dimensional hand mesh.

[0024] In this optional embodiment, acquiring a hand image and performing hand keypoint detection on the hand image based on a lightweight detection network to obtain several hand keypoints and the three-dimensional pixel coordinates of each hand keypoint includes the following steps: preprocessing the acquired hand image including color space conversion, image scaling, and pixel normalization to obtain a preprocessed image; inputting the preprocessed image into a hand detection network for inference and outputting a palm bounding box; cropping the region of interest based on the palm bounding box and normalizing the region of interest; inputting the normalized region of interest into a preset keypoint regression network, outputting the normalized three-dimensional relative coordinates of the keypoints, and obtaining several hand keypoints and the three-dimensional pixel coordinates of each hand keypoint by performing an inverse coordinate transformation on the normalized three-dimensional relative coordinates.

[0025] In this optional embodiment, the predefined hand template mesh is segmented into bone segments according to preset skinning weights, and further segmented by the spatial position of finger joints to obtain multiple local hand sub-mesh. The process of establishing a binding mapping relationship between each local sub-mesh and its corresponding bone segment includes the following steps: obtaining a predefined hand template mesh and its corresponding skinning weight matrix. The hand template mesh contains multiple vertices and triangular faces. The skinning weight matrix is ​​used to establish a weight mapping relationship between each vertex and each hand bone. Each vertex corresponds to a weight vector containing all bone weight values. Based on the maximum value of each vertex in each bone weight, the bone region to which the vertex belongs is determined, and the hand template mesh is initially divided into a palm region and several finger regions. For each finger region, the four joint positions of the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and fingertip are used to divide the fingers into joints, forming a local hand sub-mesh containing a palm region and several joint regions. Each local sub-mesh is bound to its corresponding bone segment to establish a binding mapping relationship between each local sub-mesh and its corresponding bone segment.

[0026] In this optional embodiment, the calculation of the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of the hand key points, and the estimation of the depth information of the hand based on the camera imaging parameters and the pixel scale of the hand in the image, includes the following steps: Based on the three-dimensional pixel coordinates of the hand key points, and combined with the hierarchical connection relationship of the hand joints (i.e., the connection order from the wrist joint, palm, metacarpophalangeal joints of each finger, proximal interphalangeal joints, distal interphalangeal joints to the fingertips), a directed skeletal chain composed of multiple bone segments is established; each bone segment is defined by its parent joint and child joints. For each bone segment in the directed skeletal chain, the unit direction vector of each bone segment is calculated based on the three-dimensional coordinates of its parent joint and child joints. The algorithm calculates the skeletal unit direction vector for each bone segment. Using the skeletal unit direction vector as one axis of the local coordinate system, and combining it with the vertically upward reference vector of the world coordinate system, a local coordinate system is constructed. A local rotation matrix is ​​established by cross product of the coordinate axes. Euler angles are calculated using the local rotation matrix to obtain the 3D rotation angle of the bone. Based on the 3D pixel coordinates of the hand keypoints, and combined with the preset camera focal length, the standard actual width of the hand template mesh, and the pixel width of the palm, the wrist point depth value is calculated according to the projection relationship of similar triangles to obtain the depth information of the hand. The depth information is used for the initialization of the 3D keypoints of the hand.

[0027] In this optional embodiment, the expression for calculating the skeletal unit direction vector of each bone segment is: ; In the formula, The skeletal unit direction vector; The three-dimensional coordinates of the sub-joint; The three-dimensional coordinates of the parent joint; The three-dimensional Euclidean distance between the parent and child joints.

[0028] In this optional embodiment, establishing a local rotation matrix through coordinate axis cross product includes: setting the skeleton unit direction vector as the horizontal axis of the local coordinate system; performing a cross product between the horizontal axis and the vertically upward reference vector of the world coordinate system, and then normalizing the result to obtain the vertical axis of the local coordinate system; performing a cross product between the obtained vertical axis of the local coordinate system and the horizontal axis of the local coordinate system, and then normalizing the result to obtain the vertical axis of the local coordinate system; and combining the three orthogonal unit vectors of the horizontal axis, vertical axis, and vertical axis (i.e., X-axis, Y-axis, and Z-axis) of the local coordinate system to generate a local rotation matrix.

[0029] In this optional embodiment, a lightweight segmentation network is used to perform binary segmentation of the region of interest (ROI) of the hand image to obtain a realistic hand segmentation mask. Based on the hand's depth information, depth initialization is performed on hand keypoints to obtain the 3D global coordinates of each keypoint in the camera coordinate system; the depth information is fixed after initialization. Based on the 3D global coordinates of each keypoint in the camera coordinate system, the initial spatial position of the hand template mesh in the camera coordinate system is determined. According to the initial spatial position and based on camera intrinsic parameters, the current complete hand template mesh is projected to generate a corresponding binary projection mask. Combined with the realistic hand segmentation mask, an optimization objective function is constructed. With the goal of minimizing the optimization objective function, length and thickness coefficients are assigned to each local sub-mesh, and the Lucas-Kan algorithm is used. The ADE algorithm synchronously updates the length and thickness coefficients of each local sub-mesh (each coefficient has an initial value of 1). After iterative convergence, the geometric scale parameters of each local sub-mesh are obtained. The geometric scale parameters of each local sub-mesh are locked, and the 3D coordinates of the hand key points are optimized and corrected based on the constraints of the skeletal physiological structure. Based on the corrected coordinates of the hand key points, the unit direction vector of each bone segment is recalculated to obtain the updated unit direction vector of each bone segment. According to the mapping relationship, the bone segment and its skin weight corresponding to each local sub-mesh are determined. Based on the updated unit direction vector of each bone segment, combined with the bone segment and its skin weight corresponding to each local sub-mesh, linear hybrid skin deformation is performed on each local sub-mesh. The deformed local sub-meshes are combined according to the topology of the hand skeleton to generate the 3D mesh of the hand.

[0030] Specifically, the initial spatial position of the hand template mesh in the camera coordinate system is determined based on the three-dimensional global coordinates of each hand key point in the camera coordinate system. This includes: using the wrist point depth value obtained from depth estimation as the root node depth of the hand; back-projecting the three-dimensional pixel coordinates of each hand key point to the three-dimensional space of the camera coordinate system according to the camera intrinsic parameters to obtain the three-dimensional global coordinates of each hand key point; using the three-dimensional global coordinates of the wrist joint as the initial positioning origin of the palm sub-mesh; determining the initial orientation of the corresponding sub-mesh according to the unit direction vector of each bone segment; and sequentially positioning each local sub-mesh to the corresponding joint to complete the determination of the initial spatial position of the hand template mesh in the camera coordinate system.

[0031] Furthermore, when generating the corresponding binary projection mask, the projection processing can be implemented using the existing 3D rendering pipeline. This is achieved by configuring the camera intrinsics as the projection parameters of the rendering camera and rendering to an off-screen buffer, thus obtaining the binary projection mask. Specifically, for each vertex in the current hand template mesh, let its 3D coordinates in the camera coordinate system be (X, Y, Z), and according to the camera intrinsics (focal length...)... , and main points , Projecting it through perspective onto the image plane yields two-dimensional pixel coordinates. ), , For each triangular facet in the template mesh, the internal region of the triangular facet is rasterized and filled according to the two-dimensional pixel coordinates of its three vertices after projection. The filled pixels are marked as the hand region. After traversing all triangular facets, the binary projection mask of the hand template mesh is obtained.

[0032] In this optional embodiment, after the depth initialization is completed, the scale adaptation of subsequent frames includes the following steps: calculating the three-dimensional coordinates of the camera coordinate system based on the wrist point depth value determined in the initialization stage and the three-dimensional pixel coordinates of the current hand key points, which is used to update the spatial position of the hand template mesh; calculating and saving the initial pixel length of each bone segment based on the three-dimensional pixel coordinates of the hand key points in the initialization stage; calculating the current pixel length of each bone segment based on the three-dimensional pixel coordinates of the hand key points in each subsequent frame, obtaining the overall scaling factor based on the ratio of the current pixel length to the corresponding bone segment length in the initial stage, and scaling the corresponding local sub-mesh as a whole, that is, multiplying the overall scaling factor by the coordinate offset of the sub-mesh vertex relative to its corresponding joint point. After scaling, the length and thickness change proportionally. The mesh vertices are recalculated from the original template each time, and the scaling is not cumulative.

[0033] Specifically, the optimization and correction include: using four-point coplanar constraints to correct the depth offset of key points of the fingers and eliminate finger deformities and bends; using bone length consistency constraints, calculating the fixed length ratio between each bone segment based on the hand template mesh that has been scaled and adapted, generating user-specific bone physiological proportions, and when a bone segment length is detected to deviate from the preset proportion threshold, correcting the depth value of the corresponding key point along the camera optical axis. Each time an initialization phase is added, an initialization phase is used to customize the hand mesh model for various hand shapes. In this optional embodiment, the initial spatial position of the hand template mesh in the camera coordinate system is determined based on the 3D global coordinates. The length and thickness coefficients of each local sub-mesh are solved, and frame-by-frame scale adaptation is performed after initialization, including the following steps: the wrist point depth value obtained from depth estimation is used as the root node depth of the hand, and this depth value is used for subsequent rendering; the 3D pixel coordinates of each hand key point are back-projected to the 3D space of the camera coordinate system according to the camera intrinsic parameters to obtain the 3D global coordinates of each hand key point; the initial orientation of the corresponding sub-mesh is determined according to the unit direction vector of each bone segment to complete the initial spatial positioning of the hand template mesh in the camera coordinate system; each local sub-mesh is equipped with a length and thickness coefficient, and the initial value of each coefficient is 1. For any vertex in a local sub-mesh, its offset vector relative to the joint point of that sub-mesh is set as follows. The offset vector is decomposed into axial components along the bone direction. and the radial component perpendicular to the bone direction The transformed vertex offset is: ; In the formula, This is the length coefficient of the subgrid. This is the coarseness coefficient; A segmentation network is used to perform binary segmentation of the region of interest (ROI) in the hand image, obtaining a realistic hand segmentation mask. Based on camera intrinsics, the current hand template mesh is projected onto the image plane to generate a binary projection mask. This is achieved by utilizing existing 3D rendering pipelines (such as OpenGL / WebGL), which configure the camera intrinsics to the projection parameters of the rendering camera and render to an off-screen buffer. The intersection-union ratio (IUU) of the projection mask and the realistic hand segmentation mask is used as the optimization objective. An iterative optimization algorithm, Lucas-Kanade, is employed to synchronously update the length and thickness coefficients of each local sub-mesh. After iterative convergence, the best-fitting hand template model is obtained. When two consecutive rounds of iteration converge... Convergence occurs when the loss difference is less than a preset threshold or the number of iterations reaches the upper limit. After convergence, the best-fitting hand template model is obtained. The length and thickness coefficients remain unchanged after initialization. During the initialization phase, the initial length of each bone segment is calculated and saved based on the three-dimensional coordinates of the hand key points. In each subsequent frame, the three-dimensional coordinates of the camera coordinate system are calculated based on the wrist point depth value determined in the initialization phase and the three-dimensional pixel coordinates of the current hand key points, which are used to update the spatial position of the hand template mesh. In each subsequent frame, the current pixel length of each bone segment is calculated based on the three-dimensional pixel coordinates of the hand key points. The overall scaling factor is obtained based on the ratio of the current length to the length of the corresponding bone segment in the initial phase. The corresponding local sub-mesh is scaled as a whole. The mesh vertices are recalculated from the original template each time, and the scaling is not cumulative.

[0034] In this optional embodiment, the skeletal physiological structure constraints include: Four-point coplanar constraints are used to correct depth offset of key finger points to eliminate finger deformities and bends. Bone length consistency constraint is used to detect and correct key point depth distortion based on the user's specific bone proportions; Temporal smoothing constraints are used to smooth the coordinates of key points in consecutive frames in order to suppress inter-frame tracking jitter.

[0035] Figure 2 An embodiment of the hand 3D mesh generation system based on skeletal constraints and scale estimation of the present invention is shown.

[0036] In this optional embodiment, a hand 3D mesh generation system based on skeletal constraints and scale estimation includes: The image detection module 201 is used to acquire a hand image and perform hand key point detection on the hand image based on a lightweight detection network to obtain several hand key points and the three-dimensional pixel coordinates of each hand key point. The relation mapping module 202 is used to segment the predefined hand template mesh into bone segments according to the preset skin weights, and combine the spatial position of the finger joints to obtain multiple local sub-mesh of the hand, so as to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment. The depth calculation module 203 is used to calculate the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of the key points of the hand, and to estimate the depth information of the hand based on the camera imaging parameters and the pixel scale of the hand in the image. The mesh generation module 204 is used to solve the geometric scale parameters of each local sub-mesh based on hand image segmentation and projection mask matching, combined with the depth information of the hand, correct the key point coordinates according to the skeletal physiological constraints, and combine the geometric scale parameters and mapping relationship to deform and combine each local sub-mesh to generate a three-dimensional mesh of the hand.

[0037] To facilitate understanding of the above technical solutions of this invention, the following further explains the technical solutions of this invention from the perspective of architecture and principle. Specifically, this invention achieves rapid conversion of hand key points to parameterized hand model parameters by constructing a processing flow of key point detection, pose initialization, rapid parameter solving, local enhancement optimization, and mesh generation, thereby improving reconstruction speed and stability, and enhancing the fitting accuracy of the finger region. The specific implementation flow is as follows: 1. Obtain stable 3D key points for the hand; A lightweight hand detection network (such as the MediaPipe Hands detector) is used to detect the input image, which is a visible light RGB monocular image; the detection process is: image preprocessing → Blaze lightweight hand detection model inference; Image preprocessing includes: Color space conversion: Converts the input BGR format image to RGB format.

[0038] Image scaling: Scales the image to the model's fixed input size of 192×192 pixels.

[0039] Pixel normalization: Normalizes pixel values ​​to the range [0.0, 1.0].

[0040] The network default input is 192×192, NMS threshold is 0.3, confidence threshold is 0.5, low-confidence invalid boxes are filtered out, and the hand bounding box is output. The region of interest (ROI) is cropped based on the bounding box. The original image is cropped using the bounding box coordinates and the boundary is expanded outward by 200% to prevent hand truncation. The image is then scaled to a fixed resolution (e.g., 192×192) and mean and variance are normalized. The ROI is input into a lightweight keypoint regression network (the network consists of a MobileNetV2 backbone, deconvolutional layers, and a fully connected output head; during inference, convolutions extract features, and the fully connected layers regress the normalized XYZ relative coordinates of 21 keypoints, outputting the relative coordinates of 21 3D keypoints) to predict the normalized coordinates. Using the inverse transform of the bounding box, the inverse transform steps are as follows: (1) Multiply the normalized coordinates of the 192-dimensional area by the width and height of the ROI to obtain the pixel coordinates in the local coordinate system of the ROI; (2) By superimposing the top-left offset, the original image coordinate system can be restored. The top-left offset refers to the pixel coordinates of the top-left corner of the hand bounding box output by the hand detection network in the original input image. This value is the starting position of ROI cropping and also the offset required to translate from the local coordinate system of the ROI back to the global coordinate system of the original image in the subsequent inverse coordinate transformation.

[0041] Ultimately, 21 key hand points were identified: Each key point contains: X coordinate; Y coordinate; Z coordinate; thus forming a three-dimensional skeleton representation of the hand.

[0042] A topological structure of the hand skeleton was established based on 21 key points of the hand; A standard MANO hand template mesh (containing 1538 vertices and 3072 triangles) was selected, and the region was divided based on the predefined skinning weights of MANO. The MANO skinning weights correspond to the mapping relationship between the 21 hand bones (including the carpal bones, 5 metacarpal bones, and the three phalanges of the 5 fingers) and the 1538 vertices, forming a 1538×21 weight matrix. The bone weight of each vertex ranges from [0,1], and the sum of all bone weights is 1. A single vertex is affected by a maximum of 4 bones, and the remaining weights are 0.

[0043] The region to which each vertex belongs is determined by the maximum value among the 21 bone weights. The bone corresponding to the maximum weight is used as the basis for region assignment: the carpal bones and metacarpal bones correspond to the palm region, and each phalanx corresponds to the corresponding finger region. Thus, the template mesh is initially divided into 1 palm region and 5 finger regions.

[0044] For each finger, the phalanges are further divided based on the positions of four joints: MCP (metacarpophalangeal joint), PIP (proximal interphalangeal joint), DIP (distal phalangeal joint), and TIP (tip joint). Using the line connecting MCP and TIP as the finger's main axis, two cutting points are determined based on the projection positions of the PIP and DIP joints along the finger's main axis. Cutting planes perpendicular to the finger's main axis are then constructed, dividing the finger into three regions: proximal phalanges, intermediate phalanges, and distal phalanges.

[0045] Ultimately, the MANO hand template was divided into one palm region and 15 finger joint regions, forming a total of 16 independent sub-mesh, which provides a foundation for subsequent local scale optimization, personalized hand shape adjustment, and real-time hand driving.

[0046] 2. Establish the mapping relationship between each local mesh and its corresponding skeletal chain; steps: (1) Each of the 16 sub-grids is bound to its corresponding exclusive bone segment; the palm is bound to the wrist bone and all metacarpal bones; each phalanx is bound to the corresponding interphalangeal bone; (2) Generate an index lookup table: subgrid ID Bone number; Vertex ID Bind bone ID. Subsequent use: During pose calculation, only the bone angles bound to the sub-mesh are used for LBS deformation; during on-demand rendering, the corresponding sub-mesh is retrieved based on the selected bone, skipping the calculation of other meshes; During real-time driving, the joint rotation angles are solved in step 103 based on the calculated joint rotation angles and global scaling factor, and the global scaling factor is obtained through iterative optimization. The linear hybrid skin (LBS) algorithm is used to transform the vertices of each sub-mesh to achieve the pose transformation of the Mesh.

[0047] The complete calculation process of Linear Blended Skin (LBS) is as follows: Read the original vertex coordinates of the current submesh, the bone weights bound to the vertex, and the rotation and translation transformation matrices corresponding to each bone; solve for the vertex coordinates after deformation according to the weighted superposition rule.

[0048] The calculation formula is: Vertex deformation coordinates = ∑ (bone transformation matrix × corresponding bone weight × original vertex coordinates).

[0049] The core function of this deformation transformation is to drive the three-dimensional mesh to rotate and bend synchronously with the hand bones, highly replicating the natural flexion and extension posture of a real person's hand; After the above calculations, the vertex coordinates of the 3D hand mesh after all deformations can be output, completing the real-time driving of the 3D hand model by the key points; the whole process does not require iterative solution of the three types of global parameters of the MANO model, such as shape, pose, and trans, which greatly reduces the computational overhead. Supports on-demand loading: Based on the upper-layer application identifier (e.g., only the index and middle fingers need to be displayed), skinning calculations are performed only on the corresponding sub-grids, while other sub-grids are skipped, thereby reducing computing power consumption.

[0050] 3. Rotation angle and scale calculation: Quickly calculate the rotation angle of each bone, hand width and initial depth information based on key points of the hand, providing pose and scale parameters for segmented Hand Mesh; The specific steps are as follows: Based on the hand joint connection rules (e.g., wrist → palm → each finger MCP → PIP → DIP → TIP), a directed skeletal chain is established. For each bone segment; ; in, The skeletal unit direction vector; The three-dimensional coordinates of the sub-joint; The three-dimensional coordinates of the parent joint; The three-dimensional Euclidean distance between the parent and child joints; The rotation angle of each bone is calculated by constructing an orthogonal local coordinate system (Frame). The required data for the calculation includes: bone orientation vectors. Vertical upward reference vector of the world coordinate system Among them, the direction vector Calculated from the coordinates of the start and end points of the skeleton; During calculation, the direction vector is used. The X-axis (horizontal axis) serves as the local coordinate system, with the world coordinate system as the vertically upward reference vector. Construct a local coordinate system when the bone direction vector is... When parallel or antiparallel, the horizontal direction vector of the world coordinate system is used as an alternative reference vector. This is included in the calculation to prevent the cross product result from being a zero vector, which would cause normalization to fail. First, the local Z-axis (vertical axis) is obtained by performing a cross product between the X-axis and the reference vector. ; In the formula, The selected reference vector; The local Y-axis (vertical axis) is obtained by cross product of the X-axis and Z-axis. ; Finally, a local rotation matrix consisting of three orthogonal unit vectors X, Y, and Z is obtained, and Euler angles are calculated using the rotation matrix to obtain the three-dimensional rotation angle of the bone.

[0051] in, The direction vector of the skeleton; Here, X represents the vertically upward reference vector in the world coordinate system; X, Y, and Z represent the three orthogonal coordinate axes of the local coordinate system; the rotation matrix represents the orientation of the skeleton relative to the world coordinate system; and the Euler angles represent the rotation angles of the skeleton around the three coordinate axes. Through the above calculations, the complete three-dimensional orientation of each skeleton can be obtained, including torsional information about the skeleton's axes.

[0052] The pixel lengths of the finger skeleton and palm skeleton were calculated using Euclidean distance based on 21 three-dimensional key points of the hand.

[0053] Using the preset camera focal length f (Focal length can be obtained using the device's factory default parameters or through camera calibration) and the standard actual width of the hand template grid. Combined with the detected palm pixel width in the image The initial depth of the wrist point is estimated. This includes the palm pixel width. The depth Z of the wrist point can be approximated by the pixel distance between the metacarpophalangeal joint (MCP) of the index finger and the metacarpophalangeal joint (MCP) of the little finger. Based on the projection relationship of similar triangles, the formula for calculating the wrist point depth Z is as follows: ; Where Z is the estimated depth value in the wrist point camera coordinate system; f The focal length of the camera; This is the standard actual width corresponding to the hand grid in the template; Let be the pixel width corresponding to the palm region in the image. By substituting the above parameters, the initial depth value of the wrist point can be calculated, thus initializing the 3D keypoints of the hand based on the principle of similar triangles. Finally, we obtain: the 3D rotation angle of each bone; the pixel length of the palm and finger skeletons; and the depth Z of the wrist point in the camera coordinate system.

[0054] 4. Initialize hand scale parameters using hand segmentation results to improve reconstruction stability under different hand shapes and shooting distances, thus completing the hand customization function. The specific implementation process is as follows: The lightweight segmentation network Yolov8n-seg is used to perform binary segmentation on the ROI hand bounding box screenshot; Yolov8n-seg is equipped with a lightweight instance segmentation head; input resolution is 192. 192 3. The network outputs a single-channel probability map, which is then binarized with a threshold of 0.5. Points above the threshold are marked as white for the hand, and points below the threshold are marked as black background, thus obtaining the hand mask. The current complete hand template mesh (including all 16 sub-regions) is projected onto the image plane according to the camera intrinsic parameters to generate a corresponding binary projection mask. During initialization, the scaling factors for all sub-regions are set using a preset template factor, with length and thickness coefficients respectively. The initial value of each coefficient is 1. The scaling factors are updated in the template mesh as follows: For any vertex in a local sub-mesh, let its offset vector relative to the joint of that sub-mesh be... The offset vector is decomposed into axial components along the bone direction. and the radial component perpendicular to the bone direction The transformed vertex offset is: ; In the formula, This is the length coefficient of the subgrid. This is the coarseness coefficient; Then, the Lucas-Kanade iterative optimization strategy is adopted to optimize the average intersection-union ratio (IUU) of the projection mask and the real hand segmentation mask. The objective function is set as the optimization objective function, and the scaling factors of 16 local sub-meshes are solved simultaneously; complete iterative process: (1) After updating the hand mesh template by using the scaling factor of each sub-mesh in the current round, the complete hand template mesh is projected onto the image plane to generate a model projection mask; (2) Calculate the whole Loss value; ; in, The intersection ratio of the hands; A region mask generated for projection of the Hand3d model; The hand mask obtained from the segmentation network; (3) Update the scaling factor of each local region using the gradient descent algorithm; ; in, For the first t During the nth iteration i Scaling factor for each region; This is the updated scaling factor; To optimize step size; for The gradient with respect to the scaling factor.

[0055] (4) Repeated iterative updates, when two consecutive rounds When the loss difference is less than 0.001 or the number of iterations reaches the maximum limit (50 times), the optimization is considered to have converged, and the iterative calculation is stopped. After the iteration converges, the scaling factor corresponding to each sub-mesh represents the local geometric scale of the palm and each finger segment; by multiplying the vertex coordinates of the standard template mesh with the scaling factor of the corresponding region, the thickness of the palm and the length and width of each finger segment can be adjusted to match the actual hand shape of the current user.

[0056] After scale optimization is completed, all scaling factors are locked, and the individual hand size initialization is completed. Subsequently, various upper-layer business software (including gesture interaction, AR virtual wearables, hand motion capture, 3D hand animation, etc.) can load and render specified local sub-mesh as needed (e.g., only rendering the index and middle fingers), without having to repeatedly perform scale optimization calculations. This can stably preserve personalized hand shape features while significantly reducing real-time rendering computing power consumption.

[0057] It should be further noted that this invention also incorporates joint optimization constraints, the core function of which is to address two major defects in the generation of the 3D hand mesh: depth direction distortion at key points and temporal jitter in consecutive frames of hand gestures. By collaboratively correcting hand joint coordinates through three layers of physical constraints, the naturalness of the reconstructed pose and tracking stability are improved. Specifically, this includes three sets of constraint mechanisms: The four-point coplanar constraint is used to optimize depth. During the iterative optimization process, the depth values ​​of key points in the current frame are corrected to reduce Z prediction error. Bone length consistency constraint is used to verify and correct the depth. It serves as a verification layer after the iterative optimization converges. If distortion is detected, a secondary optimization is triggered. One Euro filter is used for timing smoothing. It filters the frame-by-frame key point coordinates of the module output to reduce jitter. Ultimately, it can output the final 3D keypoint coordinates.

[0058] It should also be noted that a One-Euro Filter temporal smoothing mechanism is used to stabilize animation effects by addressing high-frequency jitter at key points in consecutive frames; this mechanism relies on the frame-by-frame coordinates of the key points obtained earlier and the hand shape parameters from the scale module.

[0059] Filtering implementation: Frame-by-frame calculations are performed on the XYZ axes of 21 key points. The smoothing intensity is adaptively adjusted according to the motion speed, with weaker smoothing for high-speed motion and stronger noise reduction for low-speed motion. The current coordinates are corrected by weighted averaging of historical frames. In continuous frame optimization, this constraint not only penalizes single-frame 2D reprojection errors but also dynamically suppresses high-frequency jitter components through adaptive frequency truncation.

[0060] The four-point coplanar constraint, based on the physiological rule of four coplanar points in the human finger, corrects the abnormal bending caused by the Z-axis depth offset of key points. For each finger, the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), and fingertip tip (TIP) are selected as a set of spatial vertices. From the perspective of human physiological structure, these four joints are always within the same bending plane during movement. If visual detection generates depth noise, it will cause some key points to shift their Z-coordinates, causing the four points to deviate from the unified plane, resulting in unnatural finger twisting and bending. Correction is achieved by calculating the flatness of the spatial quadrilateral formed by these four points. The specific logic is as follows: For each finger, the four joints—MCP, PIP, DIP, and TIP—are selected as spatial constraint points, and their three-dimensional coordinates are represented as follows: ; in, Indicates the first i The three-dimensional coordinates of each finger joint. , , These represent the coordinate components of the joint in three-dimensional space. A spatial plane fitting is performed based on the four joint points to obtain the corresponding plane equation: ; Where A, B, and C represent the spatial plane normal vector parameters, and D represents the plane offset.

[0061] Calculate the perpendicular distance from each key point to the plane, and construct the four-point coplanar constraint loss function: ; In the formula, The loss function is a four-point coplanar constraint.

[0062] Iterative optimization of keypoint depth coordinates based on the gradient of the loss function: ; in, Indicates the first t During the nth iteration i Depth coordinates of key points; Indicates the updated depth coordinates; Indicates the optimization step size; This represents the gradient of the loss function with respect to the depth coordinates.

[0063] This process gradually restores the four joints to a curved plane that conforms to the physiological structure of the human finger, thereby reducing finger deformities caused by depth errors.

[0064] Bone length consistency constraints are used for Z-axis distortion detection based on the initial prior of hand segmentation results. First, a mesh adapted to the user's hand shape is created, the actual 3D lengths of all metacarpals and phalanges are measured, and the fixed length ratios between each bone segment are calculated to generate the user-specific physiological bone proportions, which serve as a rigid correction benchmark. During optimization iterations, if the 3D Euclidean distance of a bone suddenly deviates from the preset proportion threshold, perspective distortion along the Z-axis (depth direction) is immediately identified. At this point, the constraint function forces the gradient descent focus direction to be locked on the camera optical axis (Z-axis), and the depth values ​​of key nodes are adjusted in a targeted manner based on the acquisition of hand key points until the bone length is restored to the preset physiological proportions.

[0065] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0066] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0068] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0070] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method for generating 3D hand meshes based on skeletal constraints and scale estimation, characterized in that, Includes the following steps: A hand image is acquired, and a lightweight detection network is used to detect key points of the hand image to obtain several key points of the hand and the three-dimensional pixel coordinates of each key point. Based on the preset skinning weights, the predefined hand template mesh is segmented into bone segments, and combined with the spatial position of the finger joints, multiple local sub-meshes of the hand are obtained to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment. Based on the three-dimensional pixel coordinates of key points of the hand, the unit direction vector of each bone segment is calculated, and the depth information of the hand is estimated based on the camera imaging parameters and the pixel scale of the hand in the image. Based on hand image segmentation and projection mask matching, and combined with the depth information of the hand, the geometric scale parameters of each local sub-mesh are solved. The coordinates of key points are corrected according to the physiological constraints of the skeleton. Combining the geometric scale parameters and mapping relationship, the local sub-mesh is deformed and combined to generate a 3D mesh of the hand.

2. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 1, characterized in that, The process of acquiring a hand image and detecting hand key points based on a lightweight detection network to obtain several hand key points and the three-dimensional pixel coordinates of each hand key point includes the following steps: The acquired hand image is preprocessed, including color space conversion, image scaling, and pixel normalization, to obtain the preprocessed image. The preprocessed image is input into the hand detection network for inference and outputs the hand bounding box; The region of interest is cropped based on the bounding box of the palm, and then normalized. The normalized region of interest is input into a preset keypoint regression network, which outputs the normalized 3D relative coordinates of the keypoints. By performing an inverse coordinate transformation on the normalized 3D relative coordinates, several hand keypoints and the 3D pixel coordinates of each hand keypoint are obtained.

3. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 1, characterized in that, The process of segmenting the predefined hand template mesh into bone segments based on preset skinning weights and combining this with the spatial position of finger joints to obtain multiple local hand sub-meshes, and establishing a binding mapping relationship between each local sub-mesh and its corresponding bone segment, includes the following steps: Obtain a predefined hand template mesh and its corresponding skinning weight matrix. The hand template mesh contains multiple vertices and triangular faces. The skinning weight matrix is ​​used to establish the weight mapping relationship between each vertex and each hand bone. Each vertex corresponds to a weight vector containing the weight values ​​of all bones. Based on the maximum value of each vertex in each bone weight, the bone region to which the vertex belongs is determined, and the hand template mesh is initially divided into a palm region and several finger regions. For each finger region, the finger joints are divided using the four joint positions of the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and fingertip, forming a local subgrid of the hand that includes one palm region and several finger joint regions. Each local submesh is bound to its corresponding bone segment to establish a binding mapping relationship between each local submesh and its corresponding bone segment.

4. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 1, characterized in that, The process of calculating the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of key points of the hand, and estimating the depth information of the hand based on camera imaging parameters and the pixel scale of the hand in the image, includes the following steps: Based on the three-dimensional pixel coordinates of key points of the hand and combined with the hierarchical connection relationship of the hand joints, a directed skeletal chain composed of multiple bone segments is established. For each bone segment in a directed skeletal chain, calculate the skeletal unit direction vector of each bone segment based on the three-dimensional coordinates of its parent and child joints. Using the skeletal unit direction vector as one coordinate axis of the local coordinate system, and combining it with the vertically upward reference vector of the world coordinate system, a local coordinate system is constructed, and a local rotation matrix is ​​established through the cross product of the coordinate axes. The three-dimensional rotation angles of the skeleton are obtained by solving the Euler angles using a local rotation matrix. Based on the three-dimensional pixel coordinates of the key points of the hand, and combined with the preset camera focal length, the standard actual width of the hand template mesh, and the pixel width of the palm, the depth value of the wrist point is calculated according to the similar triangle projection relationship, thereby obtaining the depth information of the hand; the depth information is used for the initialization of the three-dimensional key points of the hand.

5. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 4, characterized in that, The expression for calculating the skeletal unit direction vector of each bone segment is as follows: ； In the formula, The skeletal unit direction vector; The three-dimensional coordinates of the sub-joint; The three-dimensional coordinates of the parent joint; The three-dimensional Euclidean distance between the parent and child joints.

6. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 4, characterized in that, The process of establishing the local rotation matrix through the cross product of coordinate axes includes: Set the skeletal unit direction vector as the horizontal axis of the local coordinate system, perform a cross product between the horizontal axis and the vertical upward reference vector of the world coordinate system, and then normalize the result to obtain the vertical axis of the local coordinate system. The vertical axis of the local coordinate system is cross-multiplied with the horizontal axis of the local coordinate system, and then normalized to obtain the vertical axis of the local coordinate system. The local rotation matrix is ​​generated by combining the three orthogonal unit vectors of the local coordinate system: the horizontal axis, the vertical axis, and the y-axis.

7. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 1, characterized in that, The process of generating a 3D hand mesh by segmenting and matching hand images with projection masks, combining hand depth information, solving for the geometric scale parameters of each local sub-mesh, correcting key point coordinates according to skeletal physiological constraints, and combining the geometric scale parameters and mapping relationships to deform and combine each local sub-mesh includes the following steps: A lightweight segmentation network is used to perform binary segmentation of the region of interest in the hand image to obtain a true hand segmentation mask; Depth initialization of hand key points is performed based on the depth information of the hand to obtain the three-dimensional global coordinates of each hand key point in the camera coordinate system. The depth information is fixed after initialization. Based on the three-dimensional global coordinates of each hand key point in the camera coordinate system, the initial spatial position of the hand template mesh in the camera coordinate system is determined. Based on the initial spatial position and camera intrinsic parameters, the current complete hand template mesh is projected to generate a corresponding binary projection mask. Combined with the real hand segmentation mask, an optimization objective function is constructed. With the goal of minimizing the objective function, each local subgrid is equipped with length and coarseness coefficients, and the Lucas-Kanade algorithm is used to update the length and coarseness coefficients of each local subgrid synchronously. After iterative convergence, the geometric scale parameters of each local subgrid are obtained. The geometric scale parameters of each local sub-mesh are locked, and the three-dimensional coordinates of the hand key points are optimized and corrected based on the constraints of the skeletal physiological structure. Based on the corrected hand key point coordinates, the unit direction vector of each bone segment is recalculated to obtain the updated unit direction vector of each bone segment. Based on the mapping relationship, determine the bone segments and their skinning weights corresponding to each local sub-mesh; Based on the updated unit direction vector of each bone segment, combined with the bone segment and its skinning weight corresponding to each local sub-mesh, linear hybrid skinning deformation is performed on each local sub-mesh. The deformed local sub-meshes are then combined according to the topology of the hand bones to generate a 3D hand mesh.

8. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 7, characterized in that, After the depth initialization is completed, the scale adaptation of subsequent frames includes the following steps: The 3D coordinates of the camera coordinate system are calculated by combining the wrist point depth value determined in the initialization phase with the 3D pixel coordinates of the current hand key points, which is used to update the spatial position of the hand template mesh; During the initialization phase, the initial pixel length of each bone segment is calculated and saved based on the 3D pixel coordinates of the hand key points. In each subsequent frame, the current pixel length of each bone segment is calculated based on the 3D pixel coordinates of the hand key points. The overall scaling factor is obtained based on the ratio of the current pixel length to the corresponding bone segment length in the initial phase, and the corresponding local sub-mesh is scaled as a whole.

9. The method for generating a 3D hand mesh based on skeletal constraints and scale estimation according to claim 7, characterized in that, The skeletal physiological structure constraints include: Four-point coplanar constraints are used to correct depth offset of key finger points to eliminate finger deformities and bends. Bone length consistency constraint is used to detect and correct key point depth distortion based on the user's specific bone proportions; Temporal smoothing constraints are used to smooth the coordinates of key points in consecutive frames in order to suppress inter-frame tracking jitter.

10. A hand 3D mesh generation system based on skeletal constraints and scale estimation, characterized in that, The system includes: The image detection module is used to acquire hand images and perform hand keypoint detection on the hand images based on a lightweight detection network to obtain several hand keypoints and the three-dimensional pixel coordinates of each hand keypoint. The relation mapping module is used to segment the predefined hand template mesh into bone segments according to the preset skin weights, and combine the spatial position of the finger joints to obtain multiple local sub-mesh of the hand, so as to establish the binding mapping relationship between each local sub-mesh and the corresponding bone segment; The depth calculation module is used to calculate the unit direction vector of each bone segment based on the three-dimensional pixel coordinates of the key points of the hand, and to estimate the depth information of the hand based on the camera imaging parameters and the pixel scale of the hand in the image. The mesh generation module is used to solve the geometric scale parameters of each local sub-mesh based on hand image segmentation and projection mask matching, combined with the depth information of the hand, correct the key point coordinates according to the skeletal physiological constraints, and combine the geometric scale parameters and mapping relationship to deform and combine each local sub-mesh to generate a 3D mesh of the hand.

Images