Method and apparatus for human hand reconstruction

Abstract

The application relates to the technical field of three-dimensional reconstruction, and provides a human hand reconstruction method and device, which inputs a single-view or multi-view human hand RGB image into a human hand reconstruction network to obtain a human hand three-dimensional model with texture. The human hand reconstruction network comprises an encoder and a decoder. The encoder extracts feature maps of different scales and human hand feature vectors from each human hand RGB image. The decoder performs graph convolution and upsampling operations on the human hand feature vectors for several times to generate a human hand geometric model. Since the human hand information contained in the feature maps of different scales is different in richness, the human hand information extracted from the feature maps of different scales corresponding to each human hand RGB image is fused with the corresponding human hand feature vectors during the graph convolution operation, so that the target human hand geometric vertex set after the upsampling operation contains more human hand global information and detail information, thereby improving the accuracy and authenticity of the human hand three-dimensional model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, and provides a hand reconstruction method and device. BACKGROUND

[0002] In human three-dimensional reconstruction, the parts that are mainly concerned are the face and the hand. The face is used to convey expressions, and the hand is used to convey actions. In particular, in a real-time three-dimensional reconstruction scene with interaction, the reconstruction effect of the hand directly affects the overall reconstruction quality.

[0003] For example, in a virtual game scene of human-computer interaction, the motion parameters of the hand are captured through gesture recognition technology, and the hand three-dimensional model is driven to move through the motion parameters to realize interaction with virtual objects. If the hand three-dimensional model reconstruction quality is poor, it may lead to the hand being unable to accurately trigger instructions to virtual objects, and the interactive experience is reduced.

[0004] With the development of computer vision, hand reconstruction based on images has become a research hotspot. However, the hand is the most flexible organ of the human body, and has characteristics such as fast action speed, large posture distribution range, and serious self-occlusion, which makes it difficult to accurately obtain the accurate shape and posture of the hand from images, increasing the difficulty of hand reconstruction in the field of computer vision.

[0005] Therefore, it is an urgent problem to quickly and accurately reconstruct a high-quality hand three-dimensional model through visual images. SUMMARY

[0006] The present application provides a hand reconstruction method and device for improving the quality of hand reconstruction.

[0007] In one aspect, the present application provides a hand reconstruction method, comprising:

[0008] obtaining a hand RGB image from at least one perspective;

[0009] extracting feature maps of different scales and a hand feature vector from the hand RGB image through an encoder in a trained hand reconstruction network;

[0010] when the number of the hand RGB images is one, performing a plurality of times of graph convolution operations on the hand feature vector extracted from the hand RGB image through a decoder in the hand reconstruction network to obtain a target hand geometric vertex set; wherein in the plurality of times of graph convolution operations, the hand information extracted from the feature maps of different scales corresponding to the hand RGB image is respectively fused with the hand feature vector corresponding to the hand RGB image;

[0011] obtaining a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set;

[0012] According to at least one human hand RGB image, the human hand geometry model is rendered to obtain a human hand three-dimensional model with a texture map.

[0013] Optionally, when the number of the human hand RGB images is multiple:

[0014] For each human hand feature vector extracted from the human hand RGB image, a plurality of graph convolution operations are performed on the decoder in the human hand reconstruction network to obtain an initial human hand geometry vertex set; wherein, in the plurality of graph convolution operations, the human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is respectively fused with the human hand feature vector corresponding to the human hand RGB image.

[0015] According to the parameters of the RGB camera for collecting the human hand RGB image under multiple perspectives, the initial human hand geometry vertex sets corresponding to the multiple human hand RGB images are fused to obtain target human hand geometry vertex information.

[0016] Optionally, the parameters of the multiple perspective RGB cameras are determined by the following method:

[0017] According to the human hand feature vector extracted from each human hand RGB image, the parameters of the corresponding RGB camera are calculated through at least one fully connected layer and an activation function of the decoder; or

[0018] According to the calibration object images collected by the RGB cameras under multiple perspectives, the parameters of the RGB cameras under each perspective are calibrated in advance;

[0019] wherein, the parameters of each RGB camera include a rotation matrix and a translation vector of the human hand relative to the RGB camera, which are used to unify the geometry vertices in the multiple initial human hand geometry vertex sets to a coordinate system when the multiple initial human hand geometry vertex sets are fused.

[0020] Optionally, in the plurality of graph convolution operations, the human hand information corresponding to the feature maps of different scales extracted from the human hand RGB image is respectively fused with the human hand feature vector corresponding to the human hand RGB image, and the scale of the feature map corresponding to the human hand information fused by the human hand feature vector increases with the increase of the number of graph convolution operations.

[0021] Optionally, the human hand geometry model is obtained according to the three-dimensional coordinates of the geometry vertices in the target human hand geometry vertex set, comprising:

[0022] The three-dimensional coordinates of the human hand skeleton nodes are obtained by performing inverse graph convolution operation and full connection operation on the geometry vertices in the target human hand geometry vertex set;

[0023] According to the preset skin parameters, the three-dimensional coordinate changes of the human hand skeleton node are driven to obtain a driven human hand geometric model.

[0024] Optionally, an upsampling operation is performed after each graph convolution operation to increase the number of geometric vertices obtained after the graph convolution operation.

[0025] In another aspect, the present application provides a reconstruction device, comprising a processor, a memory, a display screen and a communication interface, the communication interface, the display screen, the memory and the processor are connected through a bus;

[0026] The memory comprises a data storage unit and a program storage unit, the program storage unit stores a computer program, and the processor executes the following operations according to the computer program:

[0027] Through the communication interface, the human hand RGB image collected by the RGB camera of at least one perspective is acquired and stored to the data storage unit;

[0028] Through the encoder in the trained human hand reconstruction network, the feature maps of different scales and the human hand feature vectors are extracted from the human hand RGB image;

[0029] When the number of human hand RGB images is one, the human hand feature vector extracted in the human hand RGB image is subjected to several times of graph convolution operation through the decoder in the human hand reconstruction network to obtain a target human hand geometric vertex set; wherein, the human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is respectively fused with the human hand feature vector corresponding to the human hand RGB image in the several times of graph convolution operation;

[0030] According to the three-dimensional coordinates of the geometric vertices in the target human hand geometric vertex set, a human hand geometric model is obtained;

[0031] According to at least one human hand RGB image, the human hand geometric model is subjected to differentiable rendering to obtain a human hand three-dimensional model after texture mapping, and the display screen is displayed.

[0032] Optionally, when the number of human hand RGB images is more than one, the processor further executes:

[0033] For each human hand feature vector extracted in the human hand RGB image, several times of graph convolution operation are performed through the decoder in the human hand reconstruction network to obtain an initial human hand geometric vertex set; wherein, the human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is respectively fused with the human hand feature vector corresponding to the human hand RGB image in the several times of graph convolution operation;

[0034] According to parameters of the RGB cameras for capturing the hand RGB images under multiple perspectives, the initial hand geometric vertex sets corresponding to the hand RGB images are fused to obtain target hand geometric vertex information.

[0035] Optionally, the processor determines the parameters of the RGB cameras under each perspective by:

[0036] According to the hand feature vectors extracted from each hand RGB image, the parameters of the corresponding RGB cameras are calculated through at least one fully connected layer and an activation function of the decoder; or

[0037] According to the calibration object images captured by the RGB cameras under multiple perspectives, the parameters of the RGB cameras under each perspective are calibrated in advance.

[0038] The parameters of each RGB camera include a rotation matrix and a translation vector of the hand relative to the RGB camera, which are used to unify the geometric vertices in the multiple initial hand geometric vertex sets to a coordinate system when the multiple initial hand geometric vertex sets are fused.

[0039] Optionally, in several times of graph convolution operations, the hand information extracted from the feature maps of different scales corresponding to the hand RGB images is fused with the hand feature vectors corresponding to the hand RGB images respectively, and the scale of the feature map corresponding to the hand information fused by the hand feature vector increases with the increase of the number of times of graph convolution operations.

[0040] Optionally, the processor obtains a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set, and the specific operation is as follows:

[0041] The three-dimensional coordinates of the hand skeleton nodes are obtained by performing reverse graph convolution operations and full connection operations on the geometric vertices in the target hand geometric vertex set.

[0042] According to pre-set skinning parameters, the three-dimensional coordinates of the hand skeleton nodes are driven to change to obtain a driven hand geometric model.

[0043] Optionally, the processor performs up-sampling operations after each graph convolution operation to increase the number of geometric vertices obtained after the graph convolution operation.

[0044] In another aspect, the embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used to make a computer device execute the hand reconstruction method provided by the embodiment of the present application.

[0045] The application provides a human hand reconstruction method and device, wherein the trained human hand reconstruction network comprises an encoder and a decoder, the encoder is used for extracting feature maps of different scales and a human hand feature vector from a human hand RGB image of each view angle, and the decoder is used for performing a plurality of graph convolution operations on the extracted human hand feature vector to obtain a target human hand geometric vertex set rich in geometric vertices so as to improve the accuracy of a human hand model; and in the plurality of graph convolution operations, human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is fused with the human hand feature vector corresponding to the human hand RGB image respectively, since the feature maps of different scales are obtained by encoding human hand RGB images of different resolutions, the richness of the human hand information extracted from the feature maps of different scales is different, by fusing the human hand information extracted from the feature maps of different scales into the human hand feature vector, the target human hand geometric vertex set contains more global information and detail information, so that a human hand geometric model consistent with a real human hand is obtained, and when the human hand geometric model is rendered in a differentiable manner, the authenticity of the human hand three-dimensional model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0047] Figure 1 A method flowchart for human hand reconstruction based on a single-view human hand RGB image is provided for the embodiments of the present application.

[0048] Figure 2 A human hand information fusion diagram after graph convolution of geometric vertices and feature maps is provided for the embodiments of the present application.

[0049] Figure 3 A graph convolution and upsampling process diagram of a single human hand RGB image is provided for the embodiments of the present application.

[0050] Figure 4 A method flowchart for generating a human hand geometric model is provided for the embodiments of the present application.

[0051] Figure 5 A relationship diagram of different numbers of geometric vertices in a target human hand geometric vertex set and model fineness is provided for the embodiments of the present application.

[0052] Figure 6 A structure diagram of a single-view human hand reconstruction network is provided for the embodiments of the present application.

[0053] Figure 7 A method flowchart for hand reconstruction based on multi-view RGB images of a hand is provided for an embodiment of the present application.

[0054] Figure 8 An extraction process diagram of feature maps and hand feature vectors of multiple RGB images of a hand is provided for an embodiment of the present application.

[0055] Figure 9 A fusion process diagram of multiple initial geometric vertex sets of a hand is provided for an embodiment of the present application.

[0056] Figure 10 A structure diagram of a multi-view hand reconstruction network is provided for an embodiment of the present application.

[0057] Figure 11 A hardware structure diagram of a reconstruction device is provided for an embodiment of the present application.

[0058] Figure 12 A functional structure diagram of a reconstruction device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following explains the terms of the embodiments of the present application.

[0060] Graph Convolutional Networks (GCN): A hand mesh is composed of multiple geometric vertices to form a triangular surface model, the geometric vertices and the geometric vertices are connected to each other to form a graph. Graph convolution is a process of transmitting information between adjacent vertices. After graph convolution operation, each geometric vertex has information of its adjacent vertices.

[0061] Differentiable rendering: a relatively new research technology, usually combined with deep learning framework. Compared with traditional rendering technology, differentiable rendering can make the entire rendering process differentiable, so that the gradient can be back propagated, thereby constantly updating the deep learning network parameters, which helps to build an end-to-end network model; at the same time, for a given input image of a view, the two-dimensional semantic information obtained by differentiable rendering of the three-dimensional geometry can form a self-looping supervision network with the input image corresponding to the view, without additional expensive supervision. At present, the open source differentiable rendering library (such as Pytorch3D, OpenDR, etc.) has provided an interface for differentiable rendering of three-dimensional geometry, which is easy to use.

[0062] The following summarizes the design idea of the embodiments of the present application.

[0063] Hand reconstruction is an important and fundamental problem in computer graphics, which has a wide range of applications in virtual / augmented reality, human-computer interaction and artificial intelligence. For example, in the field of virtual reality, by reconstructing the geometry and motion of the human hand, the interaction between the hand and virtual objects is realized, and the reality of the human in the virtual environment is enhanced. In the field of human-computer interaction, the hand gesture recognition technology that has emerged in recent years can replace the traditional keyboard and mouse to issue instructions by tracking the motion of the hand and mapping the specific motion of the hand to the instruction set, realizing portable and intelligent human-computer interaction. In the field of artificial intelligence, by collecting and reconstructing the motion of the hand in real time and combining with the method of machine learning, automatic translation of sign language can be realized. Due to the wide application prospect of hand reconstruction technology, it has very high scientific research and application value.

[0064] Since hand reconstruction has a wide range of application scenarios, it is a research hotspot to use ordinary RGB cameras to collect images to conveniently and quickly reconstruct hands and estimate hand poses, instead of using heavy and high-cost hardware devices such as inertial navigation and reflective tracking.

[0065] At present, the method of hand reconstruction is mainly a template-based parameterized estimation method. This method depends on the MANO template, which is a differentiable parameterized model of hand shape and pose given by statistical analysis of a large number of real hand scan data. When reconstructing the hand, by inputting a pose vector and a shape vector, a complete hand surface model and the three-dimensional coordinates of the embedded skeleton joint can be obtained, wherein the pose vector and the shape vector of the hand can be estimated from the hand image by using the deep learning method. However, as one of the most flexible parts of the human body, the hand has complexity and diversity, and there may be occlusion between fingers. Therefore, in order to accurately obtain the pose vector and the shape vector of the hand, it is generally necessary to start from the open hand pose and perform time series tracking. For other gestures, the estimated parameters are poor, and the real hand model may not be reconstructed.

[0066] In view of this, the embodiment of the present application provides a human hand reconstruction method and device, which can reconstruct a high-precision and relatively real human hand three-dimensional model based on single-view or multi-view human hand RGB images by using a human hand reconstruction network built by a neural network. The human hand reconstruction network includes an encoder and a decoder. The encoder is used to extract feature maps of different scales and human hand feature vectors from human hand RGB images of each view. The decoder is used to perform several times of graph convolution operations on the extracted human hand feature vectors, and in the several times of graph convolution operations, human hand information extracted from the feature maps of different scales corresponding to the human hand RGB images is respectively fused with the human hand feature vectors corresponding to the human hand RGB images, so that the target human hand geometric vertex set contains more global information and detail information, thereby obtaining a high-precision human hand geometric model consistent with a real human hand, and when the human hand geometric model is rendered in a differentiable manner, the reality of the human hand three-dimensional model is improved. The entire reconstruction process does not need to obtain depth data and scanning data of the human hand, nor does it need to perform time sequence tracking from a human hand opening gesture. The human hand reconstruction network can predict the posture of the human hand by itself according to human hand geometric vertices obtained after processing of human hand RGB images in a case where the shape of the human hand is unknown, thereby realizing a lightweight, efficient and real human hand reconstruction method.

[0067] The human hand reconstruction network provided by the embodiment of the present application can be deployed on a server for virtual / augmented reality holographic communication, including but not limited to a micro server, a cloud server, a server cluster, and can also be deployed on a client with an interactive function, such as a notebook computer, a desktop computer, a smart phone, a tablet, VR glasses, AR glasses and the like. The server and the client are collectively referred to as a reconstruction device.

[0068] The embodiment of the present application can be reconstructed based on single-view human hand RGB images or multi-view human hand RGB images.

[0069] In the following, taking single-view human hand RGB images as an example, the human hand reconstruction method provided by the embodiment of the present application is described.

[0070] Referring to Figure 1 The method flowchart for human hand reconstruction based on single-view images provided by the embodiment of the present application is executed by a reconstruction device, and mainly includes the following steps:

[0071] S101: Obtain single-view human hand RGB images.

[0072] In S101, the reconstruction device can be self-equipped with an RGB camera or can perform data transmission with an externally independently deployed RGB camera. After the RGB camera collects a frame of human hand RGB images, the human hand RGB images are transmitted to the reconstruction device.

[0073] S102: Using the encoder in the trained hand reconstruction network, feature maps of different scales and hand feature vectors are extracted from the RGB image of the hand.

[0074] In the embodiments of this application, the encoder in the hand reconstruction network can be built using a deep neural network. During execution S102, an RGB image of the hand is input into the encoder, which outputs a long vector (i.e., a hand feature vector) and several feature maps of different scales. These feature maps of different scales can represent the encoded features of RGB images of the hand at different resolutions.

[0075] It should be noted that the embodiments of this application do not impose restrictive requirements on the specific neural network used by the encoder, including but not limited to VGG, ResNet, HRnet, etc.

[0076] For example, taking a ResNet network used by the encoder, when the input RGB image of a hand has a resolution of 256*256 pixels, after passing through multiple layers of the ResNet network structure, it outputs three feature maps of different scales, with sizes of 32*32 pixels, 16*16 pixels, and 8*8 pixels, respectively. Each pixel contains 256-dimensional hand information, forming three-dimensional matrices of sizes 256*32*32, 256*16*16, and 256*8*8, respectively. Simultaneously, in the last layer of the ResNet network, after pooling layers and multilayer perceptron (MLP) operations, a 2048-dimensional hand feature vector is also output. This hand feature vector contains global information about the RGB image of the hand.

[0077] S103: By using the decoder in the hand reconstruction network, perform several graph convolution operations on the hand feature vector extracted from the RGB image of the hand to obtain the geometric vertex set of the target hand.

[0078] In S103, the decoder in the hand reconstruction network first reconstructs the extracted global hand feature vector into sparse geometric vertex features in the hand mesh corresponding to the graph convolution operation. The hand performs several graph convolution operations of the same level on the geometric vertex features. In these graph convolution operations, the decoder gradually obtains the three-dimensional coordinates of the sparse geometric vertices. Using a projection method or a self-attention mechanism, the hand information is extracted from the feature maps of different scales corresponding to the RGB image of the hand based on the three-dimensional coordinates of each geometric vertex. The extracted hand information is then fused with the hand feature vector.

[0079] like Figure 2 The image shown is a schematic diagram illustrating the fusion of geometric vertices after graph convolution and hand information extracted from the feature map, as provided in an embodiment of this application.Figure 2 For a fusion process, the information of the human hand mesh is defined as a matrix of P=N*f size, where the i-th row Pi represents the information of the i-th geometric vertex. The graph convolution (GCN) can pass information between different vertices, and through projection or self-attention mechanism, the information F related to the human hand RGB image can be obtained i , and then the information F in the image is fused into the information P of the i-th geometric vertex through a fully connected layer, to obtain the coordinates V of the i-th geometric vertex in the three-dimensional space i i i 3d .

[0080] When performing S103, an upsampling operation is performed after each graph convolution operation to increase the number of geometric vertices obtained after each graph convolution operation. The more the number of geometric vertices, the more detailed the human hand model.

[0081] In the several times of graph convolution operations in S103, when the human hand information extracted from the feature maps of different scales is respectively fused with the human hand feature vector, the scale of the feature map corresponding to the human hand information fused by the human hand feature vector increases with the increase of the number of graph convolution operations. This is because the small resolution human hand RGB image and the human hand mesh with fewer geometric vertices contain more global information, while the high resolution human hand RGB image and the human hand mesh with more geometric vertices contain more detailed information.

[0082] In specific implementation, in S103, the human hand feature vector is reorganized into an N*f matrix through graph convolution operation, where N is the number of geometric vertices in the human hand mesh, and f is the information dimension of each geometric vertex. The human hand information is extracted from the feature maps of different scales, and the human hand information is fused with the geometric vertex information after several times of graph convolution operation. After fusion, an upsampling operation is performed, and after each upsampling operation, the number of geometric vertices in the human hand mesh is doubled. After several times of graph convolution and upsampling operation, the target human hand geometric vertex set is obtained. The number of feature maps and the number of graph convolution operations can be the same or different.

[0083] For example, taking three scales of feature maps and five times of graph convolution operation as an example, referring to Figure 3 ​​After the first graph convolution operation, an N*f matrix is obtained. Then, hand information is extracted from the 8*8 feature map, and the hand information is fused with the geometric vertex information after the first graph convolution operation. After the fusion, a first upsampling operation is performed to obtain a 2N*f matrix, that is, the number of geometric vertices in the hand mesh is doubled to 2N. Subsequently, a second graph convolution and upsampling operation are performed on the result of the first upsampling to obtain a 4N*f matrix. Further, a third graph convolution operation is performed on the result of the second upsampling, and hand information is extracted from the 16*16 feature map. The hand information is fused with the geometric vertex information after the third graph convolution operation. After the fusion, a third upsampling operation is performed to obtain an 8N*f matrix. A fourth graph convolution and upsampling operation are performed on the result of the third upsampling to obtain a 16N*f matrix. Finally, a fifth graph convolution operation is performed on the result of the fourth upsampling, and hand information is extracted from the 32*32 feature map. The hand information is fused with the geometric vertex information after the fourth graph convolution operation. After the fusion, a fifth upsampling operation is performed to obtain a 32N*f matrix.

[0084] Optionally, the hand information includes three-dimensional coordinates and semantic information of the geometric vertices.

[0085] With the increase of the number of graph convolution and upsampling, the information transmission between the geometric vertices is richer, the number of geometric vertices is larger, and the hand model obtained is more accurate.

[0086] It should be noted that the number of feature maps and the number of graph convolution operations can be set according to actual needs, and the embodiments of the present application do not have restrictive requirements.

[0087] S104: Obtain a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set.

[0088] When S104 is performed, the specific process is described in Figure 4 , which mainly includes the following steps:

[0089] S1041: Perform a reverse graph convolution operation and a full connection operation on the geometric vertices in the target hand geometric vertex set to obtain three-dimensional coordinates of hand skeleton nodes.

[0090] S1042: Drive the three-dimensional coordinates of the hand skeleton nodes to change according to a pre-set skinning parameter to obtain a driven hand geometric model.

[0091] Participate Figure 5 The relationship diagram between the number of geometric vertices in the target hand geometric vertex set provided by the embodiments of the present application and the accuracy of the model is shown in the figure, wherein the sparser the number of geometric vertices is, the lower the accuracy of the model is. For example, Figure 5It is shown that the number of geometric vertices in the target hand geometry vertex set increases from left to right, and the hand geometry model becomes finer in turn.

[0092] The reconstructed hand geometry model of the embodiment of the present application has semantic information and an embedded skeleton. Therefore, in actual application, the hand geometry model can be driven by motion parameters and circumscribed on a human body three-dimensional model to realize a full-body virtual character effect. The driving method can adopt a dual quaternion skinning driving method.

[0093] S105: Microdifferentiable rendering is performed on the hand geometry model according to the hand RGB image to obtain a hand three-dimensional model with a texture map.

[0094] For the texture, in S105, a microdifferentiable rendering method is adopted to construct a pixel-by-pixel color constraint according to the hand RGB image to generate a hand texture map. After mapping, a hand three-dimensional model is obtained.

[0095] Referring to Figure 6 The structural diagram of the single-view-based hand reconstruction network provided by the embodiment of the present application is shown in FIG. 6. For a hand RGB image collected by a single-view RGB camera, an encoder composed of a ResNet network outputs feature maps with sizes of 32*32, 16*16 and 8*8 pixels and a 2048-dimensional hand feature vector. The hand feature vector is input into a decoder. The decoder performs 5 convolution operations on the hand feature vector. In the 1st, 3rd and 5th layers, 8*8, 16*16 and 32*32 feature maps extracted from the hand information are introduced into the graph convolution layers, respectively. The number of geometric vertices obtained by upsampling after each graph convolution operation is 63, 126, 252, 504 and 1008, respectively. The model becomes finer in turn. Finally, a hand three-dimensional model with a texture is output by microdifferentiable rendering.

[0096] The hand reconstruction method provided by the embodiment of the present application is not only suitable for single-view hand RGB images but also suitable for multi-view hand RGB images. The hand reconstruction method process provided by the embodiment of the present application is described below by taking multi-view hand RGB images as an example. Referring to Figure 7 , the method mainly includes the following steps:

[0097] S701: Obtain a plurality of hand RGB images from different views.

[0098] In S701, a plurality of RGB cameras capture RGB images from different views of the hand, and transmit the captured hand RGB images to a reconstruction device for three-dimensional reconstruction of the hand by the reconstruction device.

[0099] S702: For each hand RGB image, the encoder in the trained hand reconstruction network is used to extract feature maps of different scales and a hand feature vector from the hand RGB image.

[0100] For example, referring to Figure 8 , assuming there are N hand RGB images (i.e. N views), the encoder is a ResNet network, and after inputting each hand RGB image into the encoder, 32*32, 16*16, 8*8 pixel size feature maps and a 2048-dimensional hand feature vector are output respectively.

[0101] S703: For each hand feature vector extracted from each hand RGB image, the decoder in the hand reconstruction network is used to perform several graph convolution operations to obtain an initial hand geometric vertex set.

[0102] In S703, after each hand RGB image is encoded by the encoder, a hand feature vector can be obtained. By using the decoder, several graph convolution operations are performed on each hand feature vector, and after each graph convolution operation, an upsampling operation is performed to increase the number of geometric vertices obtained after the graph convolution operation, thereby obtaining an initial hand geometric vertex set corresponding to the hand feature vector extracted from each hand RGB image. In the several graph convolution operations, the hand information extracted from the feature maps of different scales corresponding to each hand RGB image is fused with the hand feature vector corresponding to the hand RGB image, respectively, and the scale of the feature map corresponding to the hand information fused by the hand feature vector increases with the number of graph convolution operations.

[0103] The process of performing graph convolution and upsampling operations on the feature maps of different scales and the hand feature vector extracted from each hand RGB image is described in the section on the hand reconstruction method based on a single-view image.

[0104] S704: According to the parameters of the RGB camera that acquires the hand RGB images from multiple views, the initial hand geometric vertex sets corresponding to the multiple hand RGB images are fused to obtain target hand geometric vertex information.

[0105] Since the multiple RGB cameras are located at different views of the hand, the multiple hand RGB images acquired contain different visible parts of the hand, so that the hand feature vector extracted from each hand RGB image contains different hand features. By multi-view feature fusion, more accurate reconstruction results can be obtained.

[0106] For example, referring to Figure 9Fig. 6 shows a schematic diagram of a hand feature vector graph convolution operation after the initial hand geometry vertex set fusion process of the embodiment of the present application. The 2048-dimensional hand feature vector 1 of a hand RGB image is input into the decoder. The decoder performs 5-layer graph convolution operation on the hand feature vector 1. In each layer of the graph convolution operation, the hand information extracted from the 8*8, 16*16, and 32*32 features in the hand RGB image is fused with the results of the 1st, 3rd, and 5th layer graph convolution, respectively, to obtain the initial hand geometry vertex set 1. Figure 1 Fig. 6 shows a schematic diagram of a hand feature vector graph convolution operation after the initial hand geometry vertex set fusion process of the embodiment of the present application. The 2048-dimensional hand feature vector 1 of a hand RGB image is input into the decoder. The decoder performs 5-layer graph convolution operation on the hand feature vector 1. In each layer of the graph convolution operation, the hand information extracted from the 8*8, 16*16, and 32*32 features in the hand RGB image is fused with the results of the 1st, 3rd, and 5th layer graph convolution, respectively, to obtain the initial hand geometry vertex set 1. Figure 2 Fig. 6 shows a schematic diagram of a hand feature vector graph convolution operation after the initial hand geometry vertex set fusion process of the embodiment of the present application. The 2048-dimensional hand feature vector 1 of a hand RGB image is input into the decoder. The decoder performs 5-layer graph convolution operation on the hand feature vector 1. In each layer of the graph convolution operation, the hand information extracted from the 8*8, 16*16, and 32*32 features in the hand RGB image is fused with the results of the 1st, 3rd, and 5th layer graph convolution, respectively, to obtain the initial hand geometry vertex set 1.

[0107] In S704, when fusing multiple initial hand geometry vertex sets, the parameters of each RGB camera need to be known, including the rotation matrix and translation vector of the hand relative to the corresponding RGB camera, which are used to unify the geometry vertices in the multiple initial hand geometry vertex sets to a coordinate system. The fusion method is not limited in the embodiment of the present application, and can be fused by using the max-pooling method or the self-attention mechanism.

[0108] The determination method of the parameters of each RGB camera has two methods, i.e., based on hand feature vector determination and based on camera calibration determination. When based on hand feature vector determination, the hand feature vector extracted from each hand RGB image is used to calculate the parameters of each RGB camera in real time through at least one fully connected layer and an activation function of the decoder. When based on camera calibration determination, the parameters of the RGB camera at each view are calibrated in advance according to the calibration object images collected by the RGB cameras at multiple views.

[0109] S705: Obtain a hand geometry model according to the three-dimensional coordinates of the geometry vertices in the target hand geometry vertex set.

[0110] In specific implementation, first, the geometric vertices in the target hand geometry vertex set are subjected to reverse graph convolution operation and full connection operation to obtain three-dimensional coordinates of the hand skeleton nodes, and then the three-dimensional coordinates of the hand skeleton nodes are driven to change according to the pre-set skinning parameters to obtain the driven hand geometry model.

[0111] In S705, the target hand geometry vertex set obtained based on the multi-view hand RGB image contains hand information seen from different angles, compared with the target hand geometry vertex set obtained based on the single-view hand RGB image in S104, which improves the accuracy and authenticity of the hand geometry model.

[0112] S706: Differentiable rendering is performed on the hand geometry model according to the multiple hand RGB images to obtain the hand three-dimensional model with texture mapping.

[0113] In S706, the hand texture map generated by differentiable rendering using the multiple hand RGB images contains more texture information, so that the hand three-dimensional model after mapping is more realistic.

[0114] Referring to Figure 10 The structure diagram of the multi-view hand reconstruction network provided by the embodiment of the present application is described as follows. For the hand RGB images collected by the RGB cameras from multiple angles, an encoder composed of a ResNet network outputs feature maps with sizes of 32*32, 16*16 and 8*8 pixels and a 2048-dimensional hand feature vector, respectively. The multiple hand feature vectors are input into a decoder, and the decoder performs 5 convolution operations on each hand feature vector. In the first, third and fifth layers of the graph convolution layer, 8*8, 16*16 and 32*32 hand information extracted from the corresponding feature maps is introduced, respectively. The number of geometric vertices obtained by upsampling after each graph convolution operation is 63, 126, 252, 504 and 1008, respectively. Through a full connection layer, the parameters of each camera relative to the hand are determined according to the multiple hand feature vectors, and the parameters are used to fuse the results of the final upsampling of the hand feature vectors by using the maximum pooling or self-attention mechanism to obtain a target hand geometry vertex set. Finally, a hand three-dimensional model with texture is output by differentiable rendering.

[0115] The method for reconstructing a human hand provided in the embodiments of the present application can be realized by one or more common RGB cameras and a civilian-level reconstruction device (for example, a computer configured with a GTX series graphics card) having data processing capability. The human hand RGB image collected by the RGB camera in single view or multiple views is input, and a human hand three-dimensional model with texture is output through a human hand reconstruction network and a differentiable rendering. The human hand three-dimensional model is represented as a three-dimensional mesh model, and the mesh model is composed of three-dimensional coordinates of surface vertices, adjacency relationships between triangular facets composed of surface vertices, and a texture map. The method only needs the human hand RGB image in single view or multiple views, does not need a scanning model of the human hand, and does not need depth information, so that the reconstruction of the human hand can be realized. The surface geometry of the human hand is reconstructed, and the texture feature of the human hand is also reconstructed. The network can predict the pose and shape of the human hand according to the single or multiple human hand RGB images without knowing the shape of the human hand, and in the graph convolution process, the human hand information extracted from the feature maps of different scales is introduced in different levels, so that the reconstructed human hand three-dimensional model is more accurate and realistic than the three-dimensional model reconstructed based on the MANO template.

[0116] Based on the same technical concept, the embodiments of the present application provide a reconstruction device. The reconstruction device can be a notebook computer, a desktop computer, a smart phone, a tablet, a VR glasses, an AR glasses and the like client having data processing capability, and can also be a server for realizing a remote interaction process, including but not limited to a micro server, a cloud server, a server cluster and the like. The reconstruction device can realize the steps of the human hand reconstruction method in the above embodiments, and can achieve the same technical effects.

[0117] Referring to Figure 11 The reconstruction device includes a processor 1101, a memory 1102, a display screen 1103 and a communication interface 1104, and the communication interface 1104, the display screen 1103, the memory 1102 and the processor 1101 are connected through a bus 1105;

[0118] The memory 1102 includes a data storage unit and a program storage unit, and the program storage unit stores a computer program. The processor 1101 executes the following operations according to the computer program:

[0119] The human hand RGB image collected by the RGB camera in at least one view is acquired through the communication interface 1104 and stored in the data storage unit;

[0120] The feature maps of different scales and the human hand feature vector are extracted from the human hand RGB image through the encoder in the trained human hand reconstruction network;

[0121] When the number of the human hand RGB images is one, the human hand feature vector extracted from the human hand RGB image is subjected to a plurality of graph convolution operations by a decoder in the human hand reconstruction network to obtain a target human hand geometric vertex set; wherein in the plurality of graph convolution operations, the human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is fused with the human hand feature vector corresponding to the human hand RGB image respectively;

[0122] A human hand geometric model is obtained according to the three-dimensional coordinates of the geometric vertices in the target human hand geometric vertex set;

[0123] The human hand geometric model is subjected to differentiable rendering according to at least one human hand RGB image to obtain a human hand three-dimensional model after texture mapping, and the human hand three-dimensional model is displayed through the display screen 1103.

[0124] Optionally, when the number of the human hand RGB images is multiple, the processor 1101 further performs:

[0125] For the human hand feature vector extracted from each of the human hand RGB images, a plurality of graph convolution operations are performed by a decoder in the human hand reconstruction network to obtain an initial human hand geometric vertex set; wherein in the plurality of graph convolution operations, the human hand information extracted from the feature maps of different scales corresponding to the human hand RGB image is fused with the human hand feature vector corresponding to the human hand RGB image respectively;

[0126] According to the parameters of the RGB cameras that capture the human hand RGB images under multiple perspectives, the initial human hand geometric vertex sets corresponding to the multiple human hand RGB images are fused to obtain target human hand geometric vertex information.

[0127] Optionally, the processor 1101 determines the parameters of the RGB cameras of each perspective by the following way:

[0128] According to the human hand feature vector extracted from each of the human hand RGB images, the parameters of the corresponding RGB camera are calculated by at least one fully connected layer and an activation function of the decoder; or

[0129] According to the calibration object images captured by the RGB cameras of multiple perspectives, the parameters of the RGB cameras of each perspective are calibrated in advance;

[0130] wherein the parameters of each RGB camera include a rotation matrix and a translation vector of the human hand relative to the RGB camera, which are used to unify the geometric vertices in the plurality of initial human hand geometric vertex sets to one coordinate system when the plurality of initial human hand geometric vertex sets are fused.

[0131] Optionally, in the several times of graph convolution operations, the hand information extracted from the feature maps of different scales corresponding to the hand RGB image is respectively fused with the hand feature vector corresponding to the hand RGB image, and the scale of the feature map corresponding to the hand information of the hand feature vector fusion increases with the increase of the number of graph convolution operations.

[0132] Optionally, the processor 1101 obtains a hand geometry model according to the three-dimensional coordinates of the geometric vertices in the target hand geometry vertex set, and the specific operation is as follows:

[0133] The three-dimensional coordinates of the hand skeleton nodes are obtained by performing the inverse graph convolution operation and the full connection operation on the geometric vertices in the target hand geometry vertex set.

[0134] According to the pre-set skinning parameters, the three-dimensional coordinates of the hand skeleton nodes are driven to change to obtain a driven hand geometry model.

[0135] Optionally, the processor 1101 performs an upsampling operation after each graph convolution operation to increase the number of geometric vertices obtained after the graph convolution operation.

[0136] It should be noted that, Figure 11 This is only an example, and the reconstruction device also includes speakers, microphones, power supplies, audio processors, and other common devices of interactive devices, which are not shown.

[0137] The embodiments of the present application Figure 11 The processor involved in the embodiments of the present application can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0138] Referring to Figure 12 A functional structure diagram of a reconstruction device provided by the embodiments of the present application is shown in FIG. 12, and the reconstruction device mainly includes an acquisition module 1201, a feature extraction module 1202, a graph convolution operation module 1203, a geometry reconstruction module 1204, and a texture reconstruction module 1205, wherein:

[0139] The acquisition module 1201 is configured to acquire at least one hand RGB image from a certain view angle.

[0140] The feature extraction module 1202 is configured to extract a feature map of different scales and a hand feature vector from the hand RGB image by using an encoder in a trained hand reconstruction network.

[0141] The graph convolution operation module 1203 is configured to, when the number of hand RGB images is one, perform a plurality of times of graph convolution operations on the hand feature vector extracted from the hand RGB image by using a decoder in the hand reconstruction network, to obtain a target hand geometric vertex set; wherein in the plurality of times of graph convolution operations, the hand information extracted from the feature map of different scales corresponding to the hand RGB image is fused with the hand feature vector corresponding to the hand RGB image respectively.

[0142] The geometric reconstruction module 1204 is configured to obtain a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set.

[0143] The texture reconstruction module 1205 is configured to perform differentiable rendering on the hand geometric model according to at least one hand RGB image, to obtain a hand three-dimensional model with a texture map.

[0144] Optionally, when the number of hand RGB images is more than one, the graph convolution operation module 1203 of the reconstruction device is further configured to, for the hand feature vector extracted from each hand RGB image, perform a plurality of times of graph convolution operations on the hand feature vector by using the decoder in the hand reconstruction network, to obtain an initial hand geometric vertex set; wherein in the plurality of times of graph convolution operations, the hand information extracted from the feature map of different scales corresponding to the hand RGB image is fused with the hand feature vector corresponding to the hand RGB image respectively.

[0145] At this time, the reconstruction device further includes a fusion module 1206 configured to:

[0146] According to the parameters of the RGB camera used to acquire the hand RGB images from multiple view angles, the initial hand geometric vertex sets corresponding to the plurality of hand RGB images are fused to obtain target hand geometric vertex information.

[0147] The specific implementation of each functional module is described in the foregoing embodiments, which will not be repeated here.

[0148] The embodiment of the present application further provides a computer readable storage medium for storing some instructions, which are executed to complete the hand reconstruction method in the foregoing embodiments.

[0149] The embodiment of the present application further provides a computer program product for storing a computer program, which is used for executing the hand reconstruction method in the foregoing embodiment.

[0150] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0152] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0154] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method of human hand reconstruction, characterized by, The method comprises: acquiring a hand RGB image; extracting a feature map of different scales and a hand feature vector from the hand RGB image through an encoder in a trained hand reconstruction network; reforming the hand feature vector corresponding to the hand RGB image into sparse geometric vertex features in a hand grid through a decoder in the hand reconstruction network, and performing a plurality of graph convolution operations of the same level on the sparse geometric vertex features to obtain a target hand geometric vertex set; wherein the output of the previous graph convolution operation is input into the next graph convolution operation after being up-sampled, and at least one of the plurality of graph convolution operations extracts hand information of each geometric vertex from a feature map of a scale corresponding to the number of the graph convolution operation, and fuses each hand information into the corresponding geometric vertex feature obtained by the graph convolution operation; the scale of the feature map corresponding to the hand information increases with the number of graph convolution operations; obtaining a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set; performing differentiable rendering on the hand geometric model according to at least one hand RGB image to obtain a hand three-dimensional model with a texture map, wherein the gradient of the parameters is reversely transmitted to adjust the parameters of the hand reconstruction network, and the two-dimensional semantic information obtained by the differentiable rendering and the hand RGB image of the corresponding view form a self-loop supervision network.

2. The method of claim 1, wherein, When a plurality of hand RGB images are acquired from different views: acquiring a target hand geometric vertex set corresponding to each hand RGB image; merging a plurality of target hand geometric vertex sets into a new target hand geometric vertex set according to the parameters of the RGB cameras that acquire the hand RGB images under multiple views.

3. The method of claim 2, wherein, The parameters of the RGB cameras under multiple views are determined by: calculating the parameters of the corresponding RGB camera through at least one fully connected layer and an activation function of the decoder according to the hand feature vector extracted from each hand RGB image; or pre-calibrating the parameters of the RGB camera under each view according to the calibration object images acquired by the RGB cameras under multiple views; wherein the parameters of each RGB camera include a rotation matrix and a translation vector of the hand relative to the RGB camera, which are used to unify the geometric vertices in a plurality of target hand geometric vertex sets to a coordinate system when the plurality of target hand geometric vertex sets are fused.

4. The method of any one of claims 1-3, wherein, The method of obtaining a hand geometric model according to the three-dimensional coordinates of the geometric vertices in the target hand geometric vertex set comprises: performing inverse graph convolution operation and full connection operation on the geometric vertices in the target hand geometric vertex set to obtain the three-dimensional coordinates of hand skeleton nodes; driving the three-dimensional coordinates of the hand skeleton nodes to change according to pre-set skinning parameters to obtain a driven hand geometric model.

5. A reconfigurable device, characterized by The device comprises a processor, a memory, a display screen and a communication interface, and the communication interface, the display screen, the memory and the processor are connected through a bus. The memory includes a data storage unit and a program storage unit, the program storage unit stores a computer program, and the processor executes the following operations according to the computer program: Through the communication interface, at least one RGB image of a human hand collected by an RGB camera is obtained, and stored in the data storage unit; Through the trained encoder in the human hand reconstruction network, feature maps of different scales and a human hand feature vector are extracted from the human hand RGB image; Through the decoder in the human hand reconstruction network, the human hand feature vector corresponding to the human hand RGB image is reorganized into sparse geometric vertex features in a human hand grid, and a target human hand geometric vertex set is obtained by performing a number of graph convolution operations of the same level on the sparse geometric vertex features; wherein the output of the last graph convolution operation is input into the next graph convolution operation after being up-sampled, and at least one of the number of graph convolution operations extracts human hand information of each geometric vertex from the feature map of the scale corresponding to the number of the graph convolution operation, and fuses each human hand information into the corresponding geometric vertex feature obtained by the graph convolution operation; the scale of the feature map corresponding to the human hand information increases with the increase of the number of graph convolution operations; A human hand geometric model is obtained according to the three-dimensional coordinates of the geometric vertices in the target human hand geometric vertex set; According to the human hand geometric model, a differentiable rendering is performed on at least one human hand RGB image to obtain a human hand three-dimensional model with a texture map, and the human hand three-dimensional model is displayed on the display screen.

6. The reconfigurable device of claim 5, wherein, When multiple human hand RGB images are obtained from different perspectives, the processor further executes: Obtain the target human hand geometric vertex set corresponding to each human hand RGB image; According to the parameters of the RGB camera for collecting the human hand RGB image under multiple perspectives, the multiple target human hand geometric vertex sets are merged into a new target human hand geometric vertex set.

Images