Lightweight monocular RGB image two-hand reconstruction method based on perception enhancement

Abstract

The invention discloses a lightweight monocular RGB image two-hand reconstruction method based on perception enhancement. Firstly, potential spatial features of both hands are extracted from a hand RGB picture by using an image feature extraction network, and then single-hand global features of a left hand and a right hand are separated. Space attention and channel attention are applied to the single-hand global features respectively, and space and channel dimension enhancement is achieved. And respectively fusing the single-hand global features and then adding to obtain single-hand hand features with complementary clues. And finally, carrying out shape adjustment and self-attention enhancement to obtain enhanced single-hand features, carrying out adaptive feature fusion on the single-hand global features, outputting single-hand adaptive hand features with clear features, and carrying out regression to obtain hand model MANO parameters so as to realize visual three-dimensional reconstruction of two interactive hands in space. According to the method, the performance and the calculation efficiency of remodeling the two three-dimensional hands from the monocular picture are balanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and relates to a method for reconstructing visualized three-dimensional double hands from monocular RGB pictures, in particular to a lightweight monocular RGB image double hand reconstruction method based on perception enhancement. BACKGROUND

[0002] Reconstructing three-dimensional double hands from RGB images has a wide range of application scenarios in virtual reality, augmented reality, gesture control, human-computer interaction, animation production, medical rehabilitation and other fields. Although current methods have achieved theoretical breakthroughs through hand parameterized modeling and end-to-end hand vertex regression, they still face two major challenges in actual deployment. On the one hand, they are limited by the dependence on professional hand capture devices in laboratory environments; on the other hand, complex real-world interference can significantly reduce the model reconstruction performance and generalization ability. Some methods improve the model accuracy by adding depth and other picture additional information and or using multi-view technology, but at the same time, the computational complexity is also high. Due to the limitations of memory and computing resources in mobile device environments, resource-intensive or sequential time frame methods will result in high computational cost, making them unsuitable for resource-constrained edge devices, and the delay problem is particularly prominent in actual applications. These problems hinder the large-scale application of models in unrestricted natural environments. Therefore, it is urgent to develop a lightweight three-dimensional interactive double hand reconstruction method suitable for mobile platforms. SUMMARY

[0003] The application proposes a lightweight monocular RGB image double hand reconstruction method based on perception enhancement, which maintains the parameter quantity and computational quantity of the model at a low level through lightweight interactive hand perception enhancement, while ensuring a high hand reconstruction accuracy.

[0004] A lightweight monocular RGB image double hand reconstruction method based on perception enhancement, specifically comprising the following steps:

[0005] Step 1: Picture data processing:

[0006] Adjust the hand RGB picture to a uniform size to meet the input requirements of the image feature extraction network.

[0007] Step 2: Extract hand global features and hand joint coordinates:

[0008] s2.1, use a pre-trained TinyViT-5M network as an image feature extraction network, input the adjusted hand RGB picture into the pre-trained TinyViT-5M network, and obtain double hand latent space features F.

[0009] s2.2, perform convolution operation on the double hand latent space features F to separate left and right hand global features F L , FR .

[0010] s2.3, respectively, the single-hand global feature F L , F R apply spatial attention, the most prominent single-hand feature GMP s (F h ) and the average distribution of single-hand feature GAP s (F h ) are spliced and fused, and then the fused feature Y h is subjected to convolution operation to obtain enhanced single-hand spatial feature . represents the identity of the hand, L represents the left hand, and R represents the right hand.

[0011] s2.4, similarly, respectively, the single-hand global feature F L , F R apply channel attention, the most prominent single-hand feature GMP c (F h ) and the average distribution of single-hand feature GAP c (F h ) are subjected to convolution operation respectively, and then element-wise addition fusion is performed to obtain enhanced single-hand channel feature .

[0012] s2.5, in order to further improve the pixel-level hand feature representation ability, the single-hand spatial feature , the single-hand channel feature and the single-hand global feature F h obtained in steps s2.3 and s2.4 are respectively element-wise multiplied and fused, and then the fused features are added to obtain a single-hand hand feature Y f h .

[0013] The single-hand hand feature Y f h with complementary clues is subjected to shape adjustment and self-attention enhancement to obtain enhanced single-hand feature Y e h .

[0014] Finally, the enhanced single-hand feature Y e h is element-wise multiplied with the single-hand global feature F h to realize adaptive feature fusion, so as to obtain a single-hand feature with clearer hand feature, and convolution operation is performed to obtain a single-hand adaptive hand feature Y a h .

[0015] s2.5, adaptive hand feature Y of left and right hands respectively a L a R Convolution operation is performed to obtain left and right hand heat map features and convert them into probability distribution through three-dimensional Softmax operation to obtain the joint coordinates of left and right hands.

[0016] Step 3: Regression hand model MANO parameters:

[0017] s3.1, adaptive hand feature Y of single hand obtained in step s2.4 is respectively concatenated with the left and right hand features Y obtained in step s2.4. a h Single hand shape parameters and 3D camera parameters required by the hand model MANO are regressed.

[0018] s3.2, left and right hand adaptive hand features Y obtained in step s2.4 are concatenated. a L a R Spatial double-wrist joint offset information is regressed.

[0019] s3.3, left and right hand joint coordinates obtained in step s2.5 are normalized and mapped to the image coordinate system. Then, the left and right hand adaptive hand features Y obtained in step s2.4 are concatenated. a L a R Hand features with joint information are obtained through grid sampling, and left and right hand pose parameters required by the hand model MANO are regressed.

[0020] s3.4, the regressed left and right hand parameters are input into the MANO model to obtain the three-dimensional grid vertices of the left and right hands, respectively. Then, the spatial double-wrist joint offset information centered on the root node is added to the left hand to realize the visual three-dimensional reconstruction of the interactive double hands in space.

[0021] Step 4: model training and reconstruction

[0022] The loss between the spatial displacement of the double-wrist joint obtained in step 4, the pose parameters of MANO, the shape parameters of MANO, the joint camera parameters, the joint MANO camera parameters, the joint image, the joint image of MANO and the input hand RGB picture label is calculated, and the model parameters and weights are trained through back propagation. The hand RGB image to be reconstructed is input into the trained model to complete the visual three-dimensional reconstruction of the interactive double hands in space.

[0023] The present application has the following beneficial effects: ​​​

[0024] 1、The method is based on monocular RGB image for three-dimensional interactive double hand reconstruction, only relying on monocular RGB image, without additional depth map or multi-view image and other multi-modal input, greatly reducing the system cost and deployment threshold.

[0025] 2、Through the interactive hand perception enhancement scheme, the spatial, channel and pixel multiple attention mechanisms are fused, the expression ability of hand features and the modeling ability of complex interactive actions are effectively improved.

[0026] 3、An adaptive feature enhancement is designed, which further enhances the expression of hand joint features by using residual processing, so that the method has stronger robustness and accuracy in different scenes.

[0027] 4、In the reconstruction stage, the grid regression function combining the heat map guided joint positioning and feature sampling is used to realize the high-precision regression of MANO parameters and the end-to-end prediction of three-dimensional pose and shape.

[0028] 5、The overall structure of the model proposed by the method is light, and the parameter quantity and calculation quantity are significantly lower than those of existing methods, which has high efficient deployment capability on edge devices and mobile platforms. Experimental results show that the method greatly reduces the consumption of computing resources while maintaining high hand reconstruction accuracy, and has good engineering practical value and broad market application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The flow chart of the interactive double hand three-dimensional reconstruction method.

[0030] Figure 2 The structure diagram of spatial attention.

[0031] Figure 3 The structure diagram of channel attention.

[0032] Figure 4 The fusion diagram of spatial and channel attention.

[0033] Figure 5 The visualization result of the embodiment on the dataset InterHand2.6M. DETAILED DESCRIPTION

[0034] The present application will be further explained and described below in conjunction with the accompanying drawings;

[0035] As Figure 1 shown, a lightweight monocular RGB image double hand reconstruction method based on perception enhancement specifically includes the following steps:

[0036] Step 1: picture data processing:

[0037] The size of the hand RGB image is uniformly adjusted to 224*224, and the channel number is 3, so as to meet the input requirements of the image feature extraction network.

[0038] Step 2: Extract hand global features and hand joint coordinates:

[0039] s2.1, use the pre-trained TinyViT-5M network on the ImageNet dataset as the image feature extraction network, input the adjusted hand RGB image into the pre-trained TinyViT-5M network, and get the hand latent space feature F with size 7*7 and channel number 320.

[0040] s2.2, respectively, the hand latent space feature F is subjected to two convolution operations with convolution kernel 1 and step 1, and finally separated to get the left and right hand global features F with size 7*7 and channel number 320. L R .

[0041] s2.3, respectively, the left and right hand global features F L R obtained in step s2.2 are applied spatial attention, as shown in Figure 2 , to get enhanced single hand spatial features .

[0042]

[0043]

[0044]

[0045]

[0046] Wherein, GMP s () represents global maximum pooling in spatial direction, GAP s () represents global average pooling in spatial direction, represents splicing operation, W p is convolution weight, C represents channel number, max() represents maximum value, * represents matrix multiplication, F h:c represents the c-th channel of single hand global feature F h , represents the identification of hand, L represents left hand, and R represents right hand.

[0047] s2.4, similarly, respectively, the left and right hand global features F L R obtained in step s2.2 are applied channel attention, as shown in Figure 3 , to get enhanced single hand channel features​​​ :

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] where GMP c () denotes global max pooling in channel direction, GAP c () denotes global average pooling in channel direction, H and W represent the height and width of F h respectively, W ps is the convolution weight matrix corresponding to the convolution operation, + denotes element-wise addition between tensors, (i,j) represents the pixel coordinates on F h feature, and k represents the channel index.

[0054] s2.5, as shown in Figure 4 , the single-hand spatial feature , the single-hand channel feature and the single-hand global feature F h are element-wise multiplied and fused, and then the fused features are added to obtain the single-hand hand feature Y f h :

[0055]

[0056] where, denotes element-wise multiplication.

[0057] Then, the hand feature Y f h with complementary clues is reshaped in tensor shape, and then is enhanced by self-attention mechanism and normalized to obtain the feature Y e :

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] where the superscript T denotes transpose, Reshape() denotes reshaping of a tensor shape, Softmax() denotes normalization, Q, K, V represent query matrix, key matrix, value matrix respectively, W q , W k , W v represent the corresponding weight matrix respectively, γ represents a learnable channel-level scaling parameter, initialized to zero, d = C / 8 is the scaling factor of attention.

[0064] Finally, the enhanced single-hand feature Y e h is element-wise multiplied by the single-hand global feature F h , and a convolution operation with a kernel size of 1 and a stride of 1 is performed to obtain the adaptive hand feature Y a h of the single hand.

[0065]

[0066] where W e is the convolution weight.

[0067] s2.5, the adaptive hand features Y a h of the left and right hands are respectively subjected to a convolution operation with a kernel size of 1 and a stride of 1, and left and right hand heat map features with a channel number of 147 are obtained. After adjusting the shape of the left and right hand heat map features from [147, 7, 7] to [21, 7, 7, 7], the heat map features are converted into probability distribution through a three-dimensional Softmax() operation, and the 21 joint positions of the hand are predicted to obtain the left and right hand joint coordinates.

[0068] Step 3: Regression of hand model MANO parameters:

[0069] s3.1, the left and right hand adaptive hand features Y a h obtained in step s2.4 are used to regress the left and right hand 10-dimensional shape parameters and 3-dimensional camera parameters required by the hand model MANO.

[0070] s3.2, the left and right hand adaptive hand features Y a h obtained in step s2.4 are concatenated to regress the spatial double-hand wrist joint offset information.

[0071] s3.3, the left and right hand joint coordinates obtained in step s2.5 are normalized and mapped to the image coordinate system. Then, the left and right hand adaptive hand features Y a h, the hand features with joint information are obtained by grid sampling, and the left and right hand 45-dimensional pose parameters required by the hand model MANO are regressed.

[0072] s3.4, the regressed left and right hand MANO parameters are input into the MANO model to obtain the three-dimensional grid vertices of the left and right hands, and then the spatial double-wrist joint offset information centered on the root node is added to the left hand to realize the visual three-dimensional reconstruction of the interactive double hands in space.

[0073] Step 4: Model training and reconstruction

[0074] s4.1, the L1 loss is used to calculate the loss between the spatial displacement of the double-wrist joint obtained in step 4, the pose parameters of MANO, the shape parameters of MANO, the joint camera parameters, the joint MANO camera parameters, the joint image, the joint image of MANO and the input hand RGB picture label, and all the losses are averaged and backpropagated.

[0075] s4.2, in order to better observe the training of the model, the TensorBoard tool is used to observe the decline of the loss. At the same time, the early stopping mechanism is added, when the model is continuously trained for 5 rounds and the loss has not decreased, the early stopping mechanism is triggered and the model training is terminated.

[0076] s4.3, save the best weight of MPJPE and MPVPE indicators, complete the model training. Input the hand RGB image to be reconstructed into the trained model to complete the visual three-dimensional reconstruction of the interactive double hands in space.

[0077] In order to verify the beneficial effects of the method, the mainstream reconstruction method is compared on the InterHand2.6M dataset, and MPJPE (Mean Per Joint Position Error) and MPVPE (Mean Per Vertex Position Error) are used as evaluation indicators, which focus on the average Euclidean distance between the predicted hand joints and vertices and their labels, and the smaller the data is, the higher the reconstruction accuracy is.

[0078] The experiment is carried out on a server with a single Nvidia A6000 graphics card, based on the PyTorch framework and using Adam optimizer and CosineLR learning rate scheduler to optimize the training process. The total training rounds are 80, the batch size is 128, the initial learning rate is set to 0.0006, and the overall training costs about 120 hours.

[0079] The comparative experimental results are shown in Table 1. Among them, Params represents the parameter quantity, the unit is million (M), FLOPs represents the floating point operation number of one forward propagation of the model, the unit is billion (G), Memory represents the memory resource size required by the model inference, the unit is megabytes (MB). all represents the complete data test set, single represents the single hand subset, two represents the double hand subset, the units of indicators MPJPE and MPVPE are millimeters (mm).

[0080] Table 1

[0081]

[0082] According to the data in Table 1, it can be known that the method realizes effective reduction of the parameter quantity Params, the floating point operation number FLOPs and the memory usage Memory, and the reconstruction accuracy reaches 7.94 mm MPJPE.

[0083] The reconstruction results of the hand are visualized from different perspectives according to some cases as shown in Figure 5 In the scene of close double hand interaction, the reconstruction result of the method is good, and can be used as a feasible solution for augmented reality (AR) and virtual reality (VR) interfaces and mobile devices requiring low delay hand interaction understanding.

Claims

1. A lightweight monocular RGB image hand reconstruction method based on perception enhancement, characterized in that: An image feature extraction network is used to extract the latent spatial features F of both hands from an RGB image of the hand. Then, a convolution operation is performed to separate the global features F of the left and right hands. L F R ; Global features F of one hand h By applying spatial attention, enhanced single-handed spatial features are obtained. ; Global features F of one hand h By applying channel attention, enhanced single-handed channel features are obtained. ;in, The symbols representing hands, L for left hand and R for right hand, respectively enhance the spatial features of a single hand. Single-handed channel characteristics With single-handed global features F h The features are then fused and added together to obtain the single-hand hand features Y with complementary cues. f h The enhanced single-handed feature Y is obtained after shape adjustment and self-attention enhancement. e h The enhanced single-handed feature Y e h With single-handed global features F h Adaptive feature fusion and convolution are performed to obtain the adaptive hand features Y of one hand. a h ; Adaptive hand features Y from a single hand a h The MANO parameters of the hand model are obtained by regression, and the interactive hands are visualized in three-dimensional space for reconstruction.

2. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 1, characterized in that: Using a pre-trained TinyViT-5M network as the image feature extraction network, the RGB images of the hands were adjusted to a uniform size and then input into the pre-trained TinyViT-5M network to extract the latent spatial features F of both hands.

3. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 1, characterized in that: Global features F of one hand h Global max pooling and global average pooling are performed separately in the spatial direction to obtain the most prominent one-handed features (GMP) in the spatial direction. s (F h ) and the uniformly distributed single-handed feature GAP s (F h The two are then concatenated and merged before convolution is performed to obtain enhanced single-handed spatial features. .

4. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 1, characterized in that: Global features F of one hand h Global max pooling and global average pooling are performed separately along the channel direction, and then GMP is applied to the most prominent one-handed features along the channel direction. c (F h ) and the uniformly distributed single-handed feature GAP c (F h Perform convolution operations on the two data points, then fuse them element-wise to obtain the enhanced single-handed channel features. .

5. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 1, characterized in that: Adaptive hand features Y from a single hand a h The MANO parameters of the hand model are obtained by regression, and the interactive hands are visualized in three-dimensional space for reconstruction.

6. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 5, characterized in that: Adaptive hand features Y for one hand a h Convolutional operations are performed to obtain heatmap features of a single hand, which are then converted into a probability distribution using a 3D Softmax() operation to predict the coordinates of the joints of the single hand. The coordinates of the joints of the left and right hands are normalized and mapped to the image coordinate system for adaptive hand features Y of the single hand. a h Mesh sampling is performed to regress the hand pose parameters required for the hand model MANO.

7. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 5, characterized in that: Adaptive hand features Y from a single hand a h The hand shape parameters and camera parameters required for the MANO hand model are obtained through regression.

8. The lightweight monocular RGB image hand reconstruction method based on perception enhancement as described in claim 5, characterized in that: Adaptive hand features Y for left and right hands a L Y a R The data is stitched together to regress the spatial offset information of the two wrist joints. The regressed hand parameters are then input into the MANO model to obtain the three-dimensional mesh vertices of the left and right hands. The spatial offset information of the two wrist joints centered on the root node is then added to the left hand to realize the visualized three-dimensional reconstruction of the interactive hands in space.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 8.

Images