A low-resolution three-dimensional hand shape depth learning reconstruction method based on UV-Map

Abstract

The application discloses a low-resolution three-dimensional hand shape reconstruction method based on UV-Map, relates to the field of image processing, and particularly relates to a three-dimensional hand reconstruction method in the field of computer vision.The application has the advantages and beneficial effects of performance improvement, reliability improvement, cost reduction, process simplification, energy saving and environmental protection and the like.The application considers a local image block of a single frame of hand, the proposed method considers using UV-Map to express the hand, and the influence of the background is removed, so that the reconstruction precision is higher, and the recovery of hand details is more accurate.

Description

Technical Field

[0001] This invention relates to the field of image processing, and particularly to a method for three-dimensional hand reconstruction in the field of computer vision. Background Technology

[0002] 3D human body reconstruction has always been a research hotspot in the field of computer vision, and is widely used in virtual reality, human-computer interaction, and motion capture. As an important organ of the human body, the hand is receiving increasing attention in human body reconstruction. However, hand images have low resolution in human body images, typically only accounting for a fraction of a normal human body image. Therefore, reconstruction is difficult and has low accuracy. Existing patent inventions, such as "Low-resolution multi-view hand reconstruction method based on human hand parameter model" (CN201811071125.8), aim to solve the problem of hand reconstruction at low resolution, but it relies on multi-view images of the human body and depends on the parametric model of the hand, resulting in low accuracy in reconstructing hand details. Related patents, such as "A high-precision 3D reconstruction method for hand skeleton based on a single depth camera" (CN202010112945.8) and "Hand motion reconstruction method and device" (CN201910646553.7), use a depth camera to reconstruct the hand skeleton, while the latter performs hand motion reconstruction. The patent "Method, Apparatus, Electronic Device and Storage Medium for Reconstructing Three-Dimensional Model of Hand" (CN202011384288.9) is for reconstructing hands from RGBD images. Summary of the Invention

[0003] The purpose of this invention is to propose a low-resolution hand reconstruction method based on UV-Map. Compared with existing hand reconstruction methods, the proposed method targets local hand image modules in a single frame of normal human body image. It utilizes the three-dimensional representation of UV-Map and employs a deep learning network to reconstruct the three-dimensional hand shape from a monocular low-resolution hand image.

[0004] The technical solution of this invention is a low-resolution 3D hand shape reconstruction method based on UV-Map, the method comprising:

[0005] Step 1: Detect the hand in the human image, adjust it to an image rectangle centered on the hand, and extract a low-resolution local color image of the hand.

[0006] Step 2: Generate multi-resolution images: Use super-resolution networks SRNet1 and SRNet2 to perform super-resolution processing on the low-resolution image, generating multiple high-resolution images of different sizes. The conversion formula is as follows:

[0007]

[0008]

[0009] Step 3: Obtain the hand outline and extract the hand region image;

[0010] The hand contour is extracted from the image using the SegNet1 hand contour extraction network. Then, by segmenting the hand images at different resolutions using the hand contour, background interference is removed to obtain images containing hand region information. The specific expression is as follows:

[0011]

[0012]

[0013]

[0014]

[0015] in It is by Obtained by downsampling It is by Obtained by downsampling;

[0016] Step 4: Establish the Encoder-Decoder network structure UVNet1, UVNet2, and UVNet3. As input, M high ∈R 3×4d×4d M med ∈R 3×2d×2d M low ∈R 3×d×d As output, the hand region image is regressed to a UV-Map at the same resolution: M high M med M low The specific expression is as follows:

[0017]

[0018]

[0019]

[0020] Step 5: Fuse multi-resolution UV-Map;

[0021] Super-resolution networks SRNet3 and SRNet4 were used to improve UV-MapM. med M low Upsampling is performed, learnable fusion weights are used, and fusion is carried out linearly to obtain a fused UV-Map image M. UV∈R 3×4d×4d The specific meaning is as follows:

[0022]

[0023]

[0024] Where w i =sigmoid(p i ), p i These are learnable parameters in the network, adjusted to the range [0,1] using the sigmoid function;

[0025] Step 6: Denoising UV-Map;

[0026] A denoising network, RefNet, is built to remove noise from the UV-Map, resulting in the denoised UV-Map, M. RUV ∈R 3×4d×4d The conversion formula is as follows:

[0027]

[0028] Step 7: Reconstruct the hand shape;

[0029] From image M RUV Sampling is performed to extract the RGB color information of the corresponding UV points and resolve it into the XYZ coordinates of the 3D mesh vertices, i.e., P∈R. 778×3 Then, based on the vertices of the 3D mesh, the 3D joint coordinates J∈R are obtained by projection through the joint regression matrix in the MANO model. 21×3 Specifically, it is expressed as follows:

[0030] P = Sampling(M) RUV )

[0031] J = Proj(P)

[0032] Where Sampling(·) represents sampling at a specified point, and Proj(·) represents regressing the coordinates of a 3D joint from a 3D mesh point;

[0033] Step 8: Jointly train the neural network;

[0034] The training of the entire neural network is divided into two stages: the pre-training stage, in which the segmentation network and the super-resolution network are trained separately, using the pre-trained segmentation network and super-resolution network SRNet1, SRNet2, SRNet3, SRNet4 and SegNet1; after the pre-training is completed, the parameters of the segmentation network and the super-resolution network will be fixed and will not be updated; in the formal training stage, the end-to-end training of the UV-Map prediction network UVNet1, UVNet2, UVNet3, the learnable fusion weights, and the UV-Map denoising network RefNet will be learned.

[0035] The loss function for the formal training phase is as follows:

[0036] L=α1L uv +α2L uv-smooth +α3L mesh-smooth +α4L refine

[0037] Where α1, α2, α3, and α4 represent the weights of each loss, and L uv The UV-Map image loss is used to constrain the training of the UV-Map prediction network and fusion parameters, and the predicted UV-Map M will be calculated. refine and the real UV-Map L1 loss between W uv For weighted graphs; L uv-smooth The UV-Map smoothing loss is used to reduce the difference between each pixel on the predicted UV-Map and its surrounding pixels, thereby producing a smoothing effect. A contour map I of the UV-Map is also introduced. UV-Mask To limit the smooth region and reduce the impact of the boundary region; L mesh-smooth The 3D mesh smoothing loss is used to smooth the reconstructed 3D hand shape, calculating the error between the predicted mesh vertices and surrounding vertices; L refine The UV-Map denoising loss is used to constrain the training of the UV-Map denoising network. Mean squared error is used to calculate the denoised UV-Map M. refine With real UV-Map The error between;

[0038]

[0039]

[0040]

[0041]

[0042] Where W and H represent the length and width of the image patch, β i,jP represents the weight of the ij-th element in the UV-map, where N represents the three channels. i,j This represents the brightness estimate at position ij. Let P represent the true value of the brightness at position ij, |.|1 represents the first norm, and P i Let P represent the i-th point on the 3D mesh. j Let GT represent the adjacent points of the i-th point on the 3D mesh, where the superscript GT indicates the truth value, P represents the vertex, and V represents the number of vertices.

[0043] Step 9: Use the trained neural network to recognize the gestures in the image to be recognized.

[0044] Furthermore, following step 4, an additional step is taken to use UV-Map to map the 3D vertex coordinates to a continuous 2D image by leveraging the vertex connectivity in the 3D hand mesh.

[0045] Furthermore, in step 8,

[0046] α1=1, α2=0.1, α3=0.1, α4=1.

[0047] The advantages and beneficial effects of this invention include improved performance, increased reliability, reduced cost, simplified process, energy saving, and environmental friendliness. This invention considers local image patches of the hand within a single frame. The proposed method employs UV-map for hand representation and removes the influence of the background, thus achieving higher reconstruction accuracy and more accurate restoration of hand details. Implementation

[0048] Step 1: Acquire a single low-resolution color image of the hand;

[0049] Step 2: Generate multi-resolution images;

[0050] This study employs deep learning to perform super-resolution operations, using the MDSR (Multi-Scale Deep Super-Resolution Network) as its backbone. For input low-resolution images, it generates high-resolution images at 2x and 4x the original size. This network is a modification of ResNet, removing unnecessary redundant structures to improve stability. Since most parameters can be shared across different scales, only a few layers need adjustment to obtain outputs at multiple resolutions. With a sufficient dataset of high-resolution hand images, the super-resolution network is retrained to make the model more targeted and better able to recover detailed information in the hand region.

[0051] Step 3: Obtain the hand outline and extract the hand region image.

[0052] Deep learning methods are employed for hand contour detection, using a hand segmentation network (SegNet) based on U-Net. Thanks to the simple structure of U-Net, the model parameters can be reduced by adjusting the number of network layers and the number of channels per layer, thus achieving lightweight design. This also makes the network easier to train on small datasets or across different datasets to meet diverse needs.

[0053] Step 4: Convert the hand region image into a UV-Map at the corresponding resolution.

[0054] While UV-Map prediction is an image domain task, it is not pixel-level prediction. The pixels in the input image and the pixel positions in the UV-Map are not one-to-one. Therefore, an encoder-decoder network (UVNet) will be used, but skip layers will not be used to connect the encoder and decoder. For the encoder, ResNet34 will be used as the backbone for feature extraction, while a combination of deconvolution and convolution will be used in the decoder stage to gradually generate a UV-Map with the same resolution as the input image. Step 5: Multi-resolution UV-Map Fusion

[0055] The fusion process proceeds step-by-step from low to high resolution, using pixel stacking to enrich pixel information without altering dimensionality. To ensure the range of pixel values ​​remains constant after stacking, a weighted summation method is employed. Furthermore, due to the diversity of the dataset and the varying proportions of effective features at different resolutions, learnable fusion parameters are introduced during the fusion stage, allowing the network to train and obtain the optimal fusion ratio.

[0056] Step 6: Denoising UV-Map

[0057] The fused UV-Map inevitably contains some "noise points" (i.e., pixels whose colors differ significantly from their surroundings). When these noise points overlap with sampling points, they can significantly affect the reconstructed hand shape. Referring to deep learning methods in image denoising, a UV-Map denoising network (RefineNet) was designed to denoise the uneven regions in the UV-Map. Utilizing the characteristics of convolutional neural networks, it smooths out noise points in the UV-Map and suppresses interference from boundary regions. Simultaneously, leveraging the characteristics of deep learning, the network is trained to pay closer attention to noise near the UV-Map sampling points.

[0058] Step 7: Reconstruct the hand shape

[0059] By leveraging the connection relationships between mesh vertices in the MANO hand model, a mapping from the 3D mesh of the hand shape to a UV-Map was achieved. During this process, the correspondence between 3D vertices and UV points was also recorded. Therefore, the RGB pixel values ​​of the corresponding UV points were parsed into XYZ coordinate values, resulting in the coordinates of 778 mesh vertices. Then, the coordinates of 21 joints were obtained using the transformation matrix of the MANO model.

[0060] Step 8: Train the neural network

[0061] Based on the above steps, a low-resolution hand reconstruction framework based on UV-Map is constructed, which includes multiple components such as a super-resolution network (SRNet), a hand segmentation network (SegNet), a UV-Map prediction network (UVNet), learnable fusion weights W, and a UV-Map denoising network (RefineNet). Training is performed in two phases: first, the super-resolution network and the hand segmentation network are trained separately, and then the remaining components are trained end-to-end.

[0062] Step 9: UV-Map Generation

[0063] The mapping from a 3D hand shape to a 2D UV-Map will employ the Least Squares Conformal Mapping (LSCM) method in computer graphics to obtain the mapping relationship between the 3D vertices of the hand shape and the UV points on the UV-Map. The RGB channels of the 2D image correspond precisely to the XYZ axes of the 3D coordinate system. Therefore, the vertex coordinates of the 3D hand shape can be mapped to the corresponding UV point's three-channel pixel values ​​on the UV-Map, and the gaps between points on the UV-Map are filled using image interpolation.

[0064] To further evaluate the validity of the patent, a quantitative error analysis experiment was conducted. The FreiHAND dataset was used, and the input images were down-resolution processed to obtain multiple low-resolution images. The average grid point error and average joint point error were used as evaluation criteria. The average grid point error was the average Euclidean distance between points on the grid, and the average joint point position error was the average Euclidean distance error of the joints. The results are shown in Table 1.

[0065] Table 1. Quantitative analysis, reconstruction error (unit: cm)

[0066] 16x16 1.59 1.48 32x32 1.49 1.39 48x48 1.19 1.12 64x64 1.18 1.10

Claims

1. A low-resolution 3D hand shape reconstruction method based on UV-Map, the method comprising: Step 1: Detect the hand in the human image, adjust it to an image rectangle centered on the hand, and extract a low-resolution local color image of the hand. ; Step 2: Generate multi-resolution images: using super-resolution networks Super-resolution processing is performed on low-resolution images to generate multiple high-resolution images of different sizes. , The conversion formula is as follows: ； ； Step 3: Obtain the hand outline and extract the hand region image; Hand contour extraction network Obtain the hand contour from the image. Then, by segmenting the hand images at different resolutions using the hand contour, background interference is removed to obtain images of the hand region. The specific expression is as follows: ； in It is by Obtained by downsampling It is by Obtained by downsampling; Step 4: Establish the Encoder-Decoder network structure ,Will As input, As output, the hand region images are regressed to UV-Maps at the same resolution: The specific expression is as follows: ； Step 5: Fuse multi-resolution UV-Map; Using super-resolution networks UV-Map Upsampling is performed, learnable fusion weights are used, and fusion is carried out linearly to obtain a fused UV-Map image. The specific meaning is as follows: ； in , These are learnable parameters in the network, adjusted to the range [0,1] using the sigmoid function; Step 6: Denoising UV-Map; Establish a denoising network for UV-Map To remove noise from the UV-Map, we obtain the denoised UV-Map, i.e. The conversion formula is as follows: ； Step 7: Reconstruct the hand shape; From the image Sampling is performed to extract the RGB color information of the corresponding UV points, and this information is then parsed into the XYZ coordinates of the 3D mesh vertices. ; Then, based on the vertices of the 3D mesh, the coordinates of the 3D joints are obtained by projecting them onto the joint regression matrix in the MANO model. Specifically, it is expressed as follows: ； in, Indicates sampling at a specified point. This represents the regression from 3D mesh points to 3D joint coordinates; Step 8: Jointly train the neural network; The training of the entire neural network is divided into two stages: a pre-training stage, in which the segmentation network and the super-resolution network are trained separately, using the already trained segmentation network and super-resolution network. and After pre-training, the parameters of the segmentation network and the super-resolution network will be fixed and will no longer be updated; during the formal training phase, the end-to-end UV-Map prediction network will be trained. Learnable fusion weights, UV-Map denoising network Three parts; The loss function for the formal training phase is as follows: ； in, , , , This indicates the weight of each loss. The UV-Map image loss is used to constrain the training of the UV-Map prediction network and fusion parameters, and the predicted UV-Map will be calculated. and the real UV-Map L1 loss between For weighted graphs; The UV-Map smoothing loss is used to reduce the difference between each pixel in the predicted UV-Map and its surrounding pixels, thereby producing a smoothing effect, and a contour map of the UV-Map is also introduced. This limits the smooth area and reduces the impact of the boundary area; The 3D mesh smoothing loss is used to smooth the reconstructed 3D hand shape and calculate the error between the predicted mesh vertices and the surrounding vertices. The UV-Map denoising loss is used to constrain the training of the UV-Map denoising network, and the mean squared error is used to calculate the denoised UV-Map. With real UV-Map The error between; ； ； ； ； Where W and H represent the length and width of the image patch, This represents the weight of the ij-th element in the UV-map, where N represents the number of channels. This represents the brightness estimate at position ij. This represents the true value of the brightness at position ij. Describes the norm 1. Represents the i-th point on the 3D mesh. Let GT represent the adjacent points of the i-th point on the 3D mesh, where the superscript GT indicates the truth value, P represents the vertex, and V represents the number of vertices. Step 9: Use the trained neural network to recognize the gestures in the image to be recognized.

2. The low-resolution 3D hand shape reconstruction method based on UV-Map as described in claim 1, characterized in that, For the additional step after step 4, the 3D vertex coordinates are mapped to a continuous 2D image using UV-Map in the form of vertex connections in the 3D hand mesh.

3. The low-resolution 3D hand shape reconstruction method based on UV-Map as described in claim 1, characterized in that, In step 8 。

Images