Interactive hand reconstruction method based on context awareness and spatial position attention

Abstract

The invention discloses an interactive hand reconstruction method based on context perception and spatial position attention, and the method comprises the steps: extracting context information of a depth map, a central position heat map and an edge contour, which are related to a global space, based on a collected single color image containing two hands; connecting the extracted global space-related depth map, the central position heat map and the context information of the edge contour, and extracting context sensing features; based on the context sensing features, obtaining space attention features between joint points of the two hands; and inputting the spatial attention features into a three-dimensional reconstruction model, and regressing the three-dimensional coordinates of the vertexes on the surfaces of the two hands to obtain a result after reconstruction of the interactive hand. According to the method, interaction information and shielding clues between the two hands are effectively utilized, so that accurate reconstruction of the two hands in a complex interaction scene is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information processing, and in particular relates to an interactive hand reconstruction method based on context perception and spatial position attention. Background Art

[0002] With the rapid development of technologies such as virtual reality, augmented reality, and human-computer interaction, hand 3D reconstruction has become a research hotspot in the field of computer vision. Gestures are one of the most important and intuitive ways of human interaction. Through hand movements and postures, people can easily express their intentions and convey rich information. In human-computer interaction systems, accurate recognition and understanding of hand gestures and movements can provide computer systems with more natural and efficient control commands, greatly improving user experience. Virtual reality technology aims to create and experience a virtual world, providing users with an immersive sense of immersion and interaction. Reconstructing and mapping the user's hands into a virtual environment in real time and interacting with virtual objects is an important means to enhance the virtual reality experience and enhance the user's sense of immersion. Interactive hand reconstruction requires modeling and understanding of the movements between the two hands. Frequent interactions and occlusions between the two hands lead to serious self-occlusion and mutual occlusion problems. Self-occlusion of the hand is mainly caused by the bending and overlapping of the fingers of a single hand, while mutual occlusion comes from the contact and overlap between the two hands. These occlusions make many key points and areas of the hand invisible in the image, which brings great difficulties to the inference of hand posture and shape. Secondly, unlike the relatively limited gesture categories in one-hand scenarios, two-hand interactions often involve more diverse and complex gestures, which places higher requirements on the generalization ability of the reconstruction model. Reconstructing interactive hands from an RGB image faces complex challenges. On the one hand, if the two-hand reconstruction task is simply regarded as two independent one-hand tasks, not only will these important interaction information be ignored, but also the visual information of some hand areas will be lost due to occlusion, and ultimately an accurate three-dimensional mesh cannot be reconstructed; on the other hand. When reconstructing in a top-down manner, it is usually necessary to use a bounding box detection module to identify the hand position and then perform cropping. This processing method will lose the global spatial relationship information between the hands in the original image to a certain extent. Summary of the invention

[0003] To solve the above technical problems, the present invention proposes an interactive hand reconstruction method based on context perception and spatial position attention to achieve accurate reconstruction of both hands in complex interactive scenes.

[0004] To achieve the above objectives, the present invention provides an interactive hand reconstruction method based on context perception and spatial position attention, comprising:

[0005] Based on a single color image containing both hands, contextual information of depth map, center position heat map and edge contour related to the global space is extracted;

[0006] The extracted global space-related depth map, center position heat map and edge contour context information are connected in the channel dimension to extract context-aware features;

[0007] Performing affine transformation on the context-aware features to obtain spatial attention features between joints of both hands;

[0008] The spatial attention features are input into a three-dimensional reconstruction model to obtain the three-dimensional coordinates of the vertices of the surfaces of both hands, thereby obtaining a result of interactive hand reconstruction.

[0009] Optionally, extracting the center location heat map includes:

[0010] The collected single color image containing both hands is processed through the ResNet-50 network to obtain a fine-grained feature map;

[0011] Upsampling the fine-grained feature map by deconvolution to obtain a fine-grained feature map with a spatial resolution reduced to one quarter;

[0012] A convolution of a preset size is used to predict a hand center heat map, a center coordinate is extracted from the hand center heat map by a soft-argmax algorithm, and a fine-grained feature map with a spatial resolution reduced to one-fourth is sampled based on the center coordinate to extract a feature vector;

[0013] The size of the hand bounding box is predicted according to the feature vector, and the hand area is cropped to obtain a center position heat map.

[0014] Optionally, obtaining spatial attention features between hand joints includes:

[0015] Inputting the context-aware features into a ResNet-50 network to obtain cropped hand image features;

[0016] The cropped hand image features are passed through the convolution layer and the soft-argmax algorithm to obtain the 2.5D joint point coordinates;

[0017] Interpolate the 2.5D joint point coordinates with the image features to obtain a spatial feature vector;

[0018] Perform multi-head self-attention processing on each hand to obtain the query features, key features, and value features of each hand, and obtain the self-attention weighted joint features;

[0019] According to the query features of one hand, the key features and value features of the other hand are obtained through multi-head attention, and the cross-attention features of the left and right hands are obtained according to the key features and value features of both hands;

[0020] The hand features of cross attention and self-attention are fused through MLP to obtain the spatial attention features between the joints of both hands.

[0021] Optionally, the method for obtaining the cross attention features of the left and right hands is:

[0022]

[0023] in, Represents interactive attention features encoding right-to-left-hand correlations; Represents the interactive attention features encoding left-hand to right-hand correlation; d k represents the normalization constant; softmax represents the softmax algorithm; Q L represents the left-hand query feature; K R represents the right-hand key feature; T represents the matrix transpose; V R Represents the right-hand value feature; K L Indicates left-hand key features.

[0024] Optionally, the method of fusing the hand features of cross attention and self attention through MLP to obtain the spatial attention features between the joints of both hands is as follows:

[0025]

[0026] in, represents the left-hand spatial attention feature; MLP represents the MLP network; Represents the self-attention weighted joint features of the left hand; Represents interactive attention features encoding right-to-left-hand correlations; represents the right hand spatial attention characteristics; Represents the self-attention weighted joint features of the right hand; Representation of interactive attention features encoding left-to-right-hand correlations.

[0027] Optionally, the results of interactive hand reconstruction include:

[0028] Inputting the spatial attention features between the joints of the two hands into a parameter regression model to obtain the posture;

[0029] The features extracted by Resnet18 are used to regress shape parameters and weak perspective camera parameters;

[0030] The shape parameters and the weak perspective camera parameters are calculated through the MANO model to obtain the left and right hand coordinates after interactive reconstruction.

[0031] Optionally, the method for obtaining the left and right hand coordinates after interactive reconstruction is:

[0032] O L=M(β left ,θ left )+ζO L，root

[0033] O R =M(β right ,θ right )+ζO R，root +δ

[0034] Where M(·) represents the MANO model; ζ represents the three-dimensional joint point regression matrix; O L and O R denote the vertex coordinates of the left and right hands respectively, ζO L，root and ζO R，root Respectively represent the positions of the left and right hand root nodes, δ represents the position change of the right hand relative to the left hand; β left represents the left-hand shape parameter; θ left represents the left-handed weak perspective camera parameters; β right represents the right-hand shape parameter; θ right Represents the right-handed weak perspective camera parameters.

[0035] Technical effect of the invention: The present invention discloses an interactive hand reconstruction method based on context perception and spatial position attention, which selects depth maps, center position heat maps and edge contours related to the global space as context information; uses a context perception module to effectively integrate these context information and incorporate them into the hand reconstruction process, captures the spatial relationship of the internal joints of a single hand through a self-attention mechanism, and uses a cross-attention mechanism to process the interaction and occlusion between the two hands, obtains a more comprehensive and semantic feature representation from the entire image, and effectively utilizes the interaction information and occlusion clues between the two hands, thereby achieving accurate reconstruction of the two hands in complex interactive scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 A flowchart of an interactive hand reconstruction method based on context perception and spatial position attention according to an embodiment of the present invention;

[0038] Figure 2 It is a structural schematic diagram of a feature extractor according to an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the structure of a hand bounding box detection module according to an embodiment of the present invention;

[0040] Figure 4A schematic diagram of the structure of a context awareness module according to an embodiment of the present invention;

[0041] Figure 5 It is a schematic diagram of the structure of the spatial position attention module according to an embodiment of the present invention;

[0042] Figure 6 Schematic diagram of the structure of a 3D reconstruction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] like Figure 1 As shown, this embodiment provides an interactive hand reconstruction method based on context perception and spatial position attention, including:

[0046] Input a single color image containing both hands to the feature extractor to extract contextual information such as depth map, center position heat map and edge contour related to the global space;

[0047] The feature information of the depth map, center position heat map and edge contour are connected on the channel and input into the context perception module;

[0048] Input context-aware features into the spatial position attention module to capture the spatial position relationship between the joints of both hands;

[0049] The feature vector that passes through the spatial position attention module is passed through the three-dimensional reconstruction module to obtain the three-dimensional coordinates of the vertices of the surfaces of both hands and obtain the result of interactive hand reconstruction.

[0050] Specifically, Figure 2 As shown, the present invention provides an interactive hand reconstruction system based on context perception and spatial position attention, including a feature extractor structure, a context perception module, a spatial position attention module and a three-dimensional reconstruction module; the feature extraction module comprises three parts, namely, a Unet network extracts a depth map related to the global space from the image, a ResNet-50 as the backbone network extracts a center position heat map and a canny edge detection algorithm is used to obtain the context information of the edge contour; the context perception module applies affine transformation to the intermediate features of the network; the spatial position attention module uses self-attention and cross-attention to capture the spatial position relationship between the joints of both hands.

[0051] Further, such as Figure 3 As shown, extracting the center position heat map includes a hand bounding box detection module, which consists of a ResNet-50 network, a deconvolution layer, a convolution layer, and a fully connected layer. The specific steps include:

[0052] The original image is processed by ResNet-50 to obtain a fine-grained feature map

[0053] Through deconvolution of F bbox Upsampling is performed to reduce the spatial resolution to the original image. Right now

[0054] Apply a 1×1 convolution to predict the hand center heatmap H center , the soft-argmax algorithm is used to extract the center coordinates from the center heat map, and the fine-grained feature map F is constructed based on these coordinates. bbox Perform sampling and extract feature vectors;

[0055] The feature vector is input into the fully connected layer to predict the size of the hand bounding box and achieve accurate cropping of the hand area to obtain the cropped image I crop .

[0056] Further, such as Figure 4 As shown in the figure, the context-aware module uses ResNet-50 as the backbone network and applies affine transformation to the intermediate features of the network. The input of the affine transformation includes the target feature map X and the context information C. The specific steps of the affine transformation include:

[0057] The module processes the context information C through an embedding layer fe, embeds C into the resolution matching the target feature map X through a convolutional network, and generates a latent vector w;

[0058] After the embedding layer, we continue with two separate convolutions f scale and f bias The scale parameter γ and the bias parameter are learned separately, and the target feature map X is normalized by these two parameters.

[0059] Further, such as Figure 5 As shown in FIG. 1 , the spatial attention features between the joints of both hands are obtained through the spatial position attention module, where the spatial position attention module is composed of a 2.5D joint point coordinate estimation module and a spatial position attention module. The specific steps include:

[0060] The cropped hand image is input into the Resnet50 network to obtain the cropped hand image feature F h ;

[0061] Cropping hand image features F h The 2.5D joint point coordinates obtained after the convolution layer and the soft-argmax algorithm Where J represents the number of joint points; 2.5D joint point coordinates are a combination of two-dimensional plane coordinates (x, y) and an extra dimension representing depth (usually z). Since they are estimated from a single-view image, they may not be directly converted into complete three-dimensional space coordinates;

[0062] The 2.5D joint point coordinates P h With image feature F i Interpolate and finally get the spatial eigenvector

[0063] Perform multi-head self-attention on each hand to obtain the query feature Q of each hand h , key feature K h Sum value feature V h , and obtain the joint features weighted by self-attention

[0064] Use the query features from one hand to obtain the key features and value features from the other hand through multi-head attention;

[0065] The characteristics of the crossed attention of the left and right hands are:

[0066]

[0067] in, Represents interactive attention features encoding right-to-left-hand correlations; Represents the interactive attention features encoding left-hand to right-hand correlation; d k represents the normalization constant; softmax represents the softmax algorithm; Q L represents the left-hand query vector; K R represents the right-hand key feature; T represents the matrix transpose; V R Represents the right-hand value feature; K L Indicates left-hand key features;

[0068] The hand features of cross attention and self attention are fused through MLP to obtain the final hand spatial attention features. and

[0069]

[0070] in, represents the left-hand spatial attention feature; MLP represents the MLP network; Represents the joint features of the left hand self-attention weighted; Represents interactive attention features encoding right-to-left-hand correlations; represents the right hand spatial attention characteristics; Represents the right hand self-attention weighted joint features; Representation of interactive attention features encoding left-to-right-hand correlations.

[0071] Further, such as Figure 6 As shown, the results after interactive hand reconstruction include:

[0072] The spatial attention feature F between the joints of both hands pos′ Input parameter regression model to obtain posture θ;

[0073] The features extracted by Resnet18 are used to regress the shape parameter β and the weak perspective camera parameter π;

[0074] The shape parameters and weak perspective camera parameters are calculated through the MANO model to obtain the left and right hand coordinates after interactive reconstruction. The calculation is as follows:

[0075] O L =M(β left ,θ left )+ζO L，root

[0076] O R =M(β right ,θ right )+ζO R，root +δ

[0077] Where M(·) represents the MANO model; ζ represents the three-dimensional joint point regression matrix; O L and O R denote the vertex coordinates of the left and right hands respectively, ζO L，root and ζO R，root Respectively represent the positions of the left and right hand root nodes, δ represents the position change of the right hand relative to the left hand; β left represents the left-hand shape parameter; θ left represents the left-handed weak perspective camera parameters; β right represents the right-hand shape parameter; θ right Represents the right-handed weak perspective camera parameters.

[0078] The present invention discloses an interactive hand reconstruction method based on context perception and spatial position attention, which selects depth maps, center position heat maps and edge contours related to the global space as context information; uses a context perception module to effectively integrate these context information and incorporate them into the hand reconstruction process, captures the spatial relationship of the internal joints of a single hand through a self-attention mechanism, and uses a cross-attention mechanism to process the interaction and occlusion between the two hands, thereby obtaining a more comprehensive and semantic feature representation from the entire image, effectively utilizing the interaction information and occlusion clues between the two hands, and thus achieving accurate reconstruction of the two hands in complex interactive scenes.

[0079] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An interactive hand reconstruction method based on context awareness and spatial position attention, characterized in that: include: Based on a single color image containing both hands, contextual information of depth map, center position heat map and edge contour related to the global space is extracted; The extracted global space-related depth map, center position heat map and edge contour context information are connected in the channel dimension to extract context-aware features; Performing affine transformation on the context-aware features to obtain spatial attention features between joints of both hands; The spatial attention features are input into a three-dimensional reconstruction model to obtain the three-dimensional coordinates of the vertices of the surfaces of both hands, thereby obtaining a result of interactive hand reconstruction.

2. The interactive hand reconstruction method based on context awareness and spatial position attention as claimed in claim 1, characterized in that: Extracting the center location heat map includes: The collected single color image containing both hands is processed through the ResNet-50 network to obtain a fine-grained feature map; Upsampling the fine-grained feature map by deconvolution to obtain a fine-grained feature map with a spatial resolution reduced to one quarter; A convolution of a preset size is used to predict a hand center heat map, a center coordinate is extracted from the hand center heat map by a soft-argmax algorithm, and a fine-grained feature map with a spatial resolution reduced to one-fourth is sampled based on the center coordinate to extract a feature vector; The size of the hand bounding box is predicted according to the feature vector, and the hand area is cropped to obtain a center position heat map.

3. The interactive hand reconstruction method based on context perception and spatial position attention as claimed in claim 1, characterized in that: The spatial attention features between the joints of both hands are obtained including: Inputting the context-aware features into a ResNet-50 network to obtain cropped hand image features; The cropped hand image features are passed through the convolution layer and the soft-argmax algorithm to obtain the 2.5D joint point coordinates; Interpolate the 2.5D joint point coordinates with the image features to obtain a spatial feature vector; Perform multi-head self-attention processing on each hand to obtain the query features, key features, and value features of each hand, and obtain the self-attention weighted joint features; According to the query features of one hand, the key features and value features of the other hand are obtained through multi-head attention, and the cross-attention features of the left and right hands are obtained according to the key features and value features of both hands; The hand features of cross attention and self-attention are fused through MLP to obtain the spatial attention features between the joints of both hands.

4. The interactive hand reconstruction method based on context awareness and spatial position attention as claimed in claim 3, characterized in that: The method to obtain the cross attention features of the left and right hands is: in, Represents interactive attention features encoding right-to-left-hand correlations; Represents the interactive attention features encoding left-hand to right-hand correlation; d k represents the normalization constant; softmax represents the softmax algorithm; Q L represents the left-hand query feature; K R represents the right-hand key feature; T represents the matrix transpose; V R Represents the right-hand value feature; K L Indicates left-hand key features.

5. The interactive hand reconstruction method based on context perception and spatial position attention as claimed in claim 3, characterized in that: By fusing the hand features of cross attention and self attention through MLP, the method of obtaining the spatial attention features between the joints of both hands is as follows: in, represents the left-hand spatial attention feature; MLP represents the MLP network; Represents the self-attention weighted joint features of the left hand; Represents interactive attention features encoding right-to-left-hand correlations; represents the right hand spatial attention characteristics; Represents the self-attention weighted joint features of the right hand; Representation of interactive attention features encoding left-to-right-hand correlations.

6. The interactive hand reconstruction method based on context awareness and spatial position attention as claimed in claim 1, characterized in that: The results after interactive hand reconstruction include: Inputting the spatial attention features between the joints of the two hands into a parameter regression model to obtain the posture; The features extracted by Resnet18 are used to regress shape parameters and weak perspective camera parameters; The shape parameters and the weak perspective camera parameters are calculated through the MANO model to obtain the left and right hand coordinates after interactive reconstruction.

7. The interactive hand reconstruction method based on context awareness and spatial position attention as claimed in claim 1, characterized in that: The method to obtain the left and right hand coordinates after interactive reconstruction is: the L =M(β left ,the left )+ζO L，root The R =M(β right ,the right )+ζO R，root +d Where M(·) represents the MANO model; ζ represents the three-dimensional joint point regression matrix; O L and O R denote the vertex coordinates of the left and right hands respectively, ζO L，root and ζO R，root Respectively represent the positions of the left and right hand root nodes, δ represents the position change of the right hand relative to the left hand; β left represents the left-hand shape parameter; θ left represents the left-handed weak perspective camera parameters; β right represents the right-hand shape parameter; θ right Represents the right-handed weak perspective camera parameters.

Images