Interactive two-hand three-dimensional reconstruction method based on monocular RGB picture

Abstract

The invention discloses an interactive two-hand three-dimensional reconstruction method based on a monocular RGB picture. According to the method, in order to remodel the problems of autocorrelation of hands and mutual shielding of the hands existing in three dimensions of the two hands from RGB pictures, through a self-similarity removing sub-module and a shielding removing sub-module designed in the process of extracting the priori knowledge of the hands, the priori knowledge of the hands and the joint features of the hands are fused, and the related features of the hands are enhanced. And then fusing the hand global features with priori knowledge, and regressing 10-dimensional shape parameters and 3-dimensional camera parameters of the hand. And splicing the fused left and right hand features to obtain space double-wrist joint offset information, and performing feature enhancement by using the joint features to regress 45-dimensional pose parameters of the hand. And finally, obtaining three-dimensional grid vertexes of the left hand and the right hand by using the regressed parameters, and adding space double-wrist joint offset information by taking a root node, namely a wrist, as a center, so as to realize visual three-dimensional reconstruction of the interactive double hands in the space.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and relates to a hand feature method in a monocular RGB image, and specifically to an interactive two-handed three-dimensional reconstruction method based on a monocular RGB image. Background Art

[0002] The task of reconstructing the 3D mesh of hands from RGB images has a wide range of application scenarios in virtual reality, augmented reality, gesture control, human-computer interaction, animation production, medical rehabilitation and other fields.

[0003] For the two-hand reconstruction task, some methods use the variable autoencoder (VAE) technology to build a prior model of two-hand interaction, and transform the reconstruction task of the interactive two-handed into conditional sampling from the hand prior. Although this method improves the accuracy of two-hand reconstruction to a certain extent, it ignores the naturalness and comprehensibility of the two-hand reconstruction results, resulting in unreasonable reconstructed hand meshes and postures that exceed the limits of the human body.

[0004] Some methods improve the accuracy of the model by adding additional information such as depth or using multi-view technology, but this also increases the computational complexity. The computing power consumption and delay issues are particularly prominent in practical applications. Some other methods use the attention mechanism to solve the problems of hand self-similarity and mutual occlusion faced by hand reconstruction. Although this can improve the performance of the model to a certain extent, the accuracy of the reconstructed hands is not ideal. Summary of the invention

[0005] The present invention proposes an interactive two-handed 3D reconstruction method based on monocular RGB images. By processing the self-similarity of hands and the mutual occlusion of hands in the process of extracting prior knowledge of hands, the method not only effectively solves the problem of unnatural two-handed reconstruction results caused by the hand reconstruction method based on prior knowledge, but also improves the accuracy of two-handed reconstruction.

[0006] An interactive two-handed 3D reconstruction method based on a monocular RGB image specifically comprises the following steps: Step 1: Image data processing: Resize the hand RGB images to a uniform size.

[0007] Step 2: Extract global hand features and hand joint coordinates: The adjusted RGB image of the hand is input into the image feature extraction network to obtain the latent space features of both hands. Then convolution operations are performed separately to extract the global features F of the left and right hands. L 、F R , based on the left and right hand global features F L 、F RThe left and right hand heat map features are obtained respectively, and the joint positions are predicted according to the left and right hand heat map features to obtain the left and right hand joint point coordinates.

[0008] Step 3: Extracting hand prior knowledge: Use the self-similarity removal submodule and the occlusion removal submodule to extract the global features F from the left and right hands. L 、F R The self-similar submodule includes a cascaded normalization layer, a multi-head self-attention module, a linear layer, a normalization layer and a multi-layer perceptron, which respectively perform the left and right hand global features F L 、F R After processing, they are input into the de-occlusion submodule together:

[0009] (1) (2) in, Indicates the hand logo, L represents the left hand, R represents the right hand, Norm() represents the norm layer, Atten() represents the multi-head self-attention processing, They represent the query matrix, key matrix, and value matrix of multi-head self-attention, MLP() represents multi-layer perceptron, and Linear() represents linear layer. represents the hand features extracted by the self-similar submodule, f h Represents the global features of the hand after multi-head self-attention processing.

[0010] The de-occlusion submodule includes two branches, one for processing left and one for processing right hand features. , , using the multi-head self-attention mechanism to interact and output the left and right hand prior knowledge priorL and priorR: (3) (4) (5) (6) (7) (8) Among them, F R→L and F L→R They represent the hand prior features of the right hand and the left hand extracted after attention processing, and Dropout() represents the discard layer.

[0011] Step 4: Regression of hand model MANO parameters: s4.1. The left and right hand global features F obtained in step 2 L 、F R Respectively with the prior knowledge F of the left and right hands in step 3 L→R and F R→L The fusion obtains the hand prior knowledge with global information, and regresses it into the 10-dimensional shape parameters and 3-dimensional camera parameters of the left and right hands required by the hand model MANO.

[0012] s4.2. Perform feature splicing on the hand prior knowledge with global information in s4.1 to obtain the spatial double wrist joint offset information.

[0013] s4.3. Input the joint features of the left and right hands obtained in step 2 together with the hand prior knowledge with global information in s4.1 into the fusion module, strengthen the hand features through grid sampling and self-attention mechanism, and then perform feature enhancement processing with the left and right hand joint features to obtain the global hand prior knowledge with joint information, and regress it into the 45-dimensional pose parameters of the left and right hands required by the hand model MANO.

[0014] s4.4. The regressed MANO parameters of the left and right hands are input into the MANO model to obtain the 3D mesh vertices of the left and right hands respectively. Then, with the root node, i.e. the wrist, as the center, the offset of both hands in space is added to the left hand to realize the visualized 3D reconstruction of the interactive two hands in space.

[0015] Step 5: Model training and reconstruction Use L1 loss to calculate the loss between the reconstruction result in step 4 and the input hand RGB image annotation. After averaging all losses, perform backpropagation to optimize model parameters, save the best weights of MPJPE and MPVPE indicators, and complete model training.

[0016] The RGB image of the hand to be reconstructed is input into the trained model to complete the visual 3D reconstruction of the interacting hands in space.

[0017] The present invention has the following beneficial effects: The present invention proposes a network model for extracting and regressing interactive two-hand three-dimensional grids based on single RGB image features. Compared with other methods of hand reconstruction, the self-similarity and mutual occlusion problems of hand reconstruction are taken into account in the process of extracting hand prior knowledge, and functional modules for removing self-similarity and removing occlusion are designed respectively. The self-similarity removal submodule uses the self-attention mechanism to learn the internal relationship of a single hand, and reshapes a natural and accurate single-hand grid while solving the hand self-similarity problem; the occlusion removal submodule makes full use of the cross-hand attention mechanism to learn the spatial connection and position characteristics between the two hands, which not only alleviates the reasoning difficulty caused by mutual occlusion of the hands, but also accurately locates the spatial position of the two hands. The two modules complement each other, and ultimately make the reshaped hand grid more realistic. At the same time, this method only relies on RGB images, does not require additional multi-modal inputs such as depth information, has a low implementation cost, and has great market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart of the interactive two-handed 3D reconstruction method.

[0019] Figure 2 Schematic diagram of the structure of the self-similar submodule.

[0020] Figure 3 Schematic diagram of the structure of the de-occlusion submodule.

[0021] Figure 4 This is the visualization result on the InterHand2.6M dataset in the embodiment. DETAILED DESCRIPTION

[0022] The present invention will be further explained below with reference to the accompanying drawings; An interactive two-handed 3D reconstruction method based on monocular RGB images, such as Figure 1 As shown in the figure, the extraction module is responsible for extracting and processing hand features in a dual-stream manner. It not only learns hand prior knowledge through the prior knowledge module, but also processes potential hand joint features. The regression module is used to process hand joint features and regress the parameters of the MANO hand model. The specific steps are as follows:

[0023] Step 1: Image data processing: The hand RGB images are resized to a uniform size to meet the input requirements of the image feature extraction network.

[0024] Step 2: Extract global hand features and hand joint coordinates: The pre-trained ResNet50 network on the ImageNet dataset is used as the image feature extraction network. Therefore, the size of the hand RGB image is uniformly adjusted to 256*256 and the number of channels is 3. The adjusted hand RGB image is input into the pre-trained ResNet50 network to obtain the double-handed latent space features with a size of 8*8 and a number of channels of 2048. Then, three convolution operations with a convolution kernel of 1 and a stride of 1 are performed in sequence, and finally the left and right hand global features F with a dimension of 512 are extracted. L 、F R Then, the left and right global features F L 、F R A convolution operation with a kernel of 1 and a stride of 1 is performed to obtain the left and right hand heat map features with a channel number of 168 respectively. After adjusting the shapes of the left and right hand heat map features from [168, 8, 8] to [21, 8, 8, 8], the heat map features are converted into probability distributions through a three-dimensional softmax operation, and the 21 joint positions of the hand are predicted to obtain the joint point coordinates of the left and right hands.

[0025] Step 3: Extracting hand prior knowledge: The left and right hand global features F obtained in step 2 are L 、F R Flatten it and input it into the self-similarity removal submodule and the occlusion removal submodule in turn, such as Figure 2 and Figure 3 As shown. Finally, after the linear layer processing, it is restored to its original shape, and the prior knowledge F of the left and right hands is obtained. L→R and F R→L .

[0026] The self-similar submodule includes a cascaded normalization layer, a multi-head self-attention module, a linear layer, a normalization layer and a multi-layer perceptron, respectively, for the left and right global features F L 、F R After processing, they are input into the de-occlusion submodule together: (1) (2) in, Indicates the hand logo, L represents the left hand, R represents the right hand, Norm() represents the norm layer, Atten() represents the multi-head self-attention processing, They represent the query matrix, key matrix, and value matrix of multi-head self-attention, MLP() represents multi-layer perceptron, and Linear() represents linear layer. represents the hand features extracted by the self-similar submodule, f h Represents the global features of the hand after multi-head self-attention processing.

[0027] The de-occlusion submodule includes two branches, one for processing left and one for processing right hand features. , , using the multi-head self-attention mechanism to interact and output the left and right hand prior knowledge priorL and priorR: (3) (4) (5) (6) (7) (8) where F R→L and F L→R They represent the hand prior features of the right hand and the left hand extracted after attention processing, and Dropout() represents the discard layer.

[0028] Step 4: Regression of hand model MANO parameters: s4.1. The left and right hand global features F obtained in step 2 L 、F R Respectively with the prior knowledge F of the left and right hands in step 3 L→R and F R→L The fusion obtains the hand prior knowledge with global information, and regresses it into the 10-dimensional shape parameters and 3-dimensional camera parameters of the left and right hands required by the hand model MANO.

[0029] s4.2. Perform feature splicing on the hand prior knowledge with global information in s4.1 to obtain the spatial double wrist joint offset information.

[0030] s4.3, normalize the joint point coordinates of the left and right hands obtained in step 2 and map them to the image coordinate system. Then input them into the fusion module together with the hand prior knowledge with global information in s4.1, strengthen the hand features through grid sampling and self-attention mechanism, and then perform feature enhancement processing with the joint point coordinates to obtain the global hand prior knowledge with joint point information, and regress it into the 45-dimensional pose parameters of the left and right hands required by the hand model MANO.

[0031] s4.4. Input the regressed MANO parameters of the left and right hands into the MANO model to obtain the 3D mesh vertices of the left and right hands respectively. Then, add the spatial offset of both hands centered on the root node to the left hand to realize the visualized 3D reconstruction of the interactive hands in space.

[0032] Step 5: Model training and reconstruction s5.1. Use L1 loss to calculate the loss between the spatial displacement of the two wrist joints obtained in step 4, MANO's pose parameters, MANO's shape parameters, joint point camera parameters, joint point MANO camera parameters, joint image, MANO's joint image and the input hand RGB image annotation. After averaging all the losses, perform back propagation.

[0033] s5.2. In order to better observe the training status of the model, use the TensorBoard tool to observe the decrease in loss. At the same time, add an early stopping mechanism. If the model has been trained for 5 consecutive rounds and the loss has not decreased, the early stopping mechanism is triggered to terminate the model training.

[0034] s5.3. Save the best weights of MPJPE and MPVPE indicators to complete model training. Input the RGB image of the hand to be reconstructed into the trained model to complete the visual 3D reconstruction of the interactive hands in space.

[0035] In order to verify the beneficial effects of this method, it was compared with mainstream reconstruction methods on the InterHand2.6M dataset, and MPJPE (Mean Per Joint Position Error) and MPVPE (Mean Per VertexPosition Error) were used as evaluation indicators. These two evaluation indicators focus on the average Euclidean distance between the predicted hand joints and vertices and their annotations, respectively. The smaller the data, the higher the accuracy of the reconstruction.

[0036] The experiment was conducted on a server equipped with a single NVIDIA A6000 graphics card, based on the PyTorch framework and using the Adam optimizer to optimize the training process. The total number of training rounds was 30, the initial learning rate was set to 0.0001, and the learning rate was reduced in the 15th and 17th rounds of training, respectively. The overall training took about 60 hours. The comparative experimental results are shown in Table 1:

[0037] Table 1 Among them, all represents the complete data test set, single represents the single-hand subset, and two represents the double-hand subset. According to the data in Table 1, in each hand type dataset, the evaluation index data of this method is the lowest, indicating that the reconstruction results of this method are closer to the annotation of the original RGB hand image than other mainstream methods. The visual 3D reconstruction results of this method on the InterHand2.6M dataset are shown in Figure 1. Figure 4As shown in the figure. Images of two types of hand cases, one hand and two hands, are selected, and the results of hand reconstruction are visualized from different perspectives. As can be seen from the figure, in the scene where the two hands interact closely, the reconstruction result is still good, and there is no mesh interlacing. In the scene where the one-hand is occluded by environmental objects, the reconstruction result is still ideal and is little affected by environmental occlusion.

[0038] In summary, compared with other methods, this method can effectively alleviate the hand self-similarity problem and mutual occlusion problem faced by interactive two-hand reconstruction based on RGB images, and can reshape a relatively natural and reasonable hand mesh while improving the accuracy of two-hand reconstruction.

Claims

1. An interactive two-handed 3D reconstruction method based on monocular RGB images, characterized by: The specific steps include: Step 1: Adjust the hand RGB image to a uniform size; Step 2: Input the adjusted hand RGB image into the image feature extraction network to obtain the latent space features of both hands; then perform convolution operations respectively to extract the global features F of the left and right hands. L 、F R , and predict the coordinates of the joint points of the left and right hands; Step 3: Use the self-similarity removal submodule and the occlusion removal submodule to extract the global features F from the left and right hands. L 、F R Extracting hand prior knowledge from The self-similar submodule includes a cascaded normalization layer, a multi-head self-attention module, a linear layer, a normalization layer and a multi-layer perceptron, respectively, for the left and right global features F L 、F R After processing, the left and right hand features are obtained , , and then input them into the de-occlusion submodule together, using the multi-head self-attention mechanism for feature interaction, and outputting the left and right hand prior knowledge priorL and priorR; Step 4: Using the global features and prior knowledge obtained in steps 2 and 3, regress the MANO parameters of the hand model to achieve a visual 3D reconstruction of the interacting hands in space; Step 5: Calculate the loss between the reconstruction result in step 4 and the input hand RGB image annotation, average all losses and perform back propagation to optimize model parameters and complete model training; input the hand RGB image to be reconstructed into the trained model to complete the visual 3D reconstruction of the interacting hands in space.

2. The interactive two-handed 3D reconstruction method based on a monocular RGB image as claimed in claim 1, characterized in that: The image feature extraction network is a pre-trained ResNet50 network on the ImageNet dataset.

3. The interactive two-handed 3D reconstruction method based on monocular RGB images as claimed in claim 1 or 2, characterized in that: The double-handed latent space features output by the image feature extraction network are subjected to three convolution operations with a convolution kernel of 1 and a step size of 1, and the left and right hand global features F with a dimension of 512 are obtained. L 、F R ; Then the left and right global features F L 、F R A convolution operation with a kernel of 1 and a stride of 1 is performed to obtain the left and right hand heat map features with a channel number of 168 respectively. After adjusting the shapes of the left and right hand heat map features from [168, 8, 8] to [21, 8, 8, 8], the heat map features are converted into probability distributions through a three-dimensional softmax operation, and the 21 joint positions of the hand are predicted to obtain the joint point coordinates of the left and right hands.

4. The interactive two-handed 3D reconstruction method based on monocular RGB images as claimed in claim 1 or 2, characterized in that: The method of obtaining the prior knowledge of the left and right hands priorL and priorR through the deocclusion submodule is as follows: Among them, F R→L and F L→R They represent the hand prior features of the right hand and the left hand extracted after attention processing, Dropout() represents the dropout layer; Norm() represents the norm layer, and Atten() represents the multi-head self-attention processing. Represent the query matrix, key matrix and value matrix of multi-head self-attention respectively, Indicates the hand logo, L represents the left hand and R represents the right hand; MLP() represents multi-layer perceptron and Linear() represents linear layer.

5. The interactive two-handed 3D reconstruction method based on monocular RGB images as claimed in claim 1, characterized in that: Step 4 is as follows: s4.

1. The left and right hand global features F obtained in step 2 L 、F R Respectively with the prior knowledge F of the left and right hands in step 3 L→R and F R→L Fusion, to obtain the hand prior knowledge with global information; then regress it into the 10-dimensional shape parameters and 3-dimensional camera parameters of the left and right hands required by the hand model MANO; s4.2, feature splicing of the hand prior knowledge with global information in s4.1 to obtain the spatial double wrist joint offset information; s4.

3. The joint features of the left and right hands obtained in step 2 are input into the fusion module together with the hand prior knowledge with global information in s4.

1. The hand features are enhanced through grid sampling and self-attention mechanism. The features are then enhanced with the joint features of the left and right hands to obtain the global hand prior knowledge with joint information, which is then regressed into the 45-dimensional pose parameters of the left and right hands required by the hand model MANO. s4.

4. The regressed MANO parameters of the left and right hands are input into the MANO model to obtain the 3D mesh vertices of the left and right hands respectively. Then, with the root node, i.e. the wrist, as the center, the offset of both hands in space is added to the left hand to realize the visualized 3D reconstruction of the interactive two hands in space.

6. The interactive two-handed 3D reconstruction method based on monocular RGB images as claimed in claim 1, characterized in that: Use L1 loss to calculate the loss between the reconstruction result in step 4 and the input hand RGB image annotation, and save the weights of the corresponding model when the MPJPE and MPVPE indicators are the best.

7. The interactive two-handed 3D reconstruction method based on monocular RGB images as claimed in claim 6, characterized in that: Use the TensorBoard tool to observe the decrease in loss. At the same time, add an early stopping mechanism. If the model has been trained for five consecutive rounds and the loss has not decreased, the early stopping mechanism is triggered to terminate the model training.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1, 2, 5, 6, and 7.

Images