A hand-object interaction reconstruction method based on single-view RGBD fusion network

Abstract

The present invention relates to a method for reconstructing hand-object interaction based on a single-view RGBD fusion network, the steps of which include: converting the depth map of the aligned RGBD image of the hand-object interaction into a point cloud, and applying a feature encoder to extract RGB features and point features therefrom; fusing the extracted RGB features and point features to generate dense RGBD features; using an SDF feature encoder to generate hand-object shape coding from the dense RGBD features, and predicting the hand-object posture parameters and object position through a hand-object geometric feature estimation module; processing the rotation and translation of each three-dimensional query point relative to the wrist and the translation of the object through a geometric Fourier feature encoding module to obtain a hand-object geometric coding; and using a hand and object SDF decoder to estimate the SDF of each three-dimensional query point using the hand-object shape coding and the hand-object geometric coding to reconstruct the hand and object shapes. Compared with the prior art, the present invention more fully integrates RGB and depth information, and realizes geometrically precise reconstruction of hand-object interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of image information processing, and in particular to a hand-object interaction reconstruction method based on a single-view RGBD fusion network. Background Art

[0002] Hand-object interaction is crucial for understanding human behavior as it primarily involves manipulating the physical environment with hands. This interaction facilitates various interactive applications such as robotic learning, virtual reality, and augmented reality. Recently, hand-object joint reconstruction has attracted increasing attention due to its ability to provide detailed contact information. However, reconstructing hand-object interactions from a single image remains a challenge due to the inherent complexity of some hand-object interaction scenarios, such as mutual occlusions and changes in object shape and pose during the interaction.

[0003] Learning-based paradigms are able to extract detailed interaction information by directly reconstructing the pose and shape of the hand and object from single-view images without prior knowledge of the object category. Current methods can generally be divided into two categories: those utilizing parametric mesh models and those based on implicit representations. Given the ill-posed nature of the problem, caused by mutual occlusions and varying object shapes, parametric mesh model-based methods encode prior knowledge of the hand to reduce ambiguity in hand-object reconstruction. Such methods typically involve recovering a 3D hand model by estimating the hand pose and leveraging a parametric hand mesh model, MANO. However, such methods can only reconstruct simple 3D objects with limited accuracy and may not accurately capture the details of the hand-object interaction. Neural implicit representations have recently made progress in the field of joint hand-object reconstruction, as they are able to generalize across a variety of shape resolutions and topologies. A common approach to neural implicit representation for hand-object interaction reconstruction is the signed distance field (SDF). This approach involves learning an implicit shape code, which is then fed into an SDF decoder, typically a multilayer perceptron (MLP), along with a query point to generate an SDF. However, this approach does not incorporate knowledge about hand and object poses into the SDF, resulting in unrealistic hand-object meshes. At the same time, due to the loss of geometric information caused by perspective projection, the performance of RGB-based methods is often limited in challenging scenes, such as those with poor lighting conditions and complex object shapes. Therefore, capturing the geometric details of human hands interacting with different objects in an end-to-end manner remains a major challenge. Summary of the invention

[0004] The purpose of the present invention is to overcome the defect of the above-mentioned prior art that it is unable to accurately capture the details of the interaction between the hand and the object and to provide a hand-object interaction reconstruction method based on a single-view RGBD fusion network, especially for complex object shapes, to achieve detailed reconstruction of the hand-object.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A hand-object interaction reconstruction method based on a single-view RGBD fusion network, the method steps comprising:

[0007] Obtain aligned RGBD images of hand-object interaction, convert the depth map into a point cloud, and apply a two-stream RGB-point cloud feature encoder to extract RGB features and point features from the RGB image and point cloud data;

[0008] The adaptive bidirectional fusion module utilizes the complementarity of appearance and geometric information and the sensitivity of different feature layers to the object reconstruction task to fuse the RGB features and point features to generate dense RGBD features;

[0009] The SDF feature encoder is used to generate the hand shape encoding from the dense RGBD features, and the hand geometry feature estimation module is used to predict the hand posture parameters and object position.

[0010] The rotation and translation of each three-dimensional query point relative to the wrist and the translation of the object are normalized through the geometric Fourier feature encoding module and converted into a standard normalized coordinate system to obtain the hand-object geometric encoding;

[0011] The hand and object SDF decoders are used to estimate the SDF of each 3D query point from the hand-object shape encoding and the hand-object geometry encoding to reconstruct the hand and object shapes.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] The present invention uses adaptive bidirectional RGBD feature fusion (ABF) and geometric Fourier feature encoding (GFFE) to refine RGBD features. An adaptive bidirectional fusion module is used to fuse two modal information, RGB image and point cloud data, and the complementarity of appearance and geometric information and the sensitivity of different feature layers to the hand-object reconstruction task are used to generate dense RGBD features, and then the generated dense RGBD features are used to generate hand-object shape coding; at the same time, a hand-object geometric feature estimation module is used to predict the posture parameters of the hand-object, and a geometric Fourier feature encoding module is used to convert the rotation and translation of each three-dimensional query point relative to the wrist and the translation of the object to obtain the hand-object geometric coding. The above scheme can fuse RGB and depth information to achieve geometrically fine hand-object interaction reconstruction. In addition, the present invention enhances the neural implicit representation of hand-object interaction in both feature extraction and encoding stages, so that the SDF decoder can focus on the geometric details of the object and improve the accuracy of hand-object reconstruction. The hand-object reconstruction results obtained by the present invention are higher than the prior art in indicators such as hand-object shape error. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1It is a flowchart of the hand-object interaction reconstruction method based on the single-view RGBD fusion network of the present invention;

[0015] Figure 2 A schematic diagram of the structure of a bidirectional fusion module in one embodiment of the present invention;

[0016] Figure 3 It is a schematic diagram of the structure of an adaptive layer fusion module in one embodiment of the present invention;

[0017] Figure 4 This is a comparison diagram between the true value and the reconstruction result of the hand-object reconstruction in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0019] Example 1

[0020] The present invention proposes a hand-object interaction reconstruction method based on a single-view RGBD fusion network to achieve detailed reconstruction of the hand-object, especially complex object shapes.

[0021] The proposed method utilizes adaptive bidirectional RGBD feature fusion (ABF) and geometric Fourier feature encoding (GFFE) to refine RGBD features, and enhances the neural implicit representation of hand-object interaction in both feature extraction and encoding stages, enabling the SDF decoder to focus on the geometric details of objects.

[0022] like Figure 1 As shown, the hand-object interaction reconstruction method based on the single-view RGBD fusion network proposed in the present invention includes the following steps:

[0023] Step 1: Given an aligned RGBD image of hand-object interaction, use the camera's intrinsic parameter matrix θ to convert the depth map into a point cloud and apply a two-stream RGB-point cloud feature encoder f TSRPE Extracting RGB features for hand-object reconstruction from RGB images and point cloud data and point features

[0024] Step 2: Adaptive bidirectional fusion module f ABF Fusion of the above RGB features and point features The two modal information utilizes the complementarity of appearance and geometric information and the sensitivity of different feature layers to the object reconstruction task to generate dense RGBD features F rgbd .

[0025] Step 3: SDF feature encoder utilizes dense RGBD features F rgbd Generate the hand shape code z, and the hand geometric feature estimation module predicts the hand posture parameters T hr and t o ;

[0026] Step 4: Geometric Fourier feature encoding module f GFFE Each 3D query point Normalize with respect to the wrist rotation and translation and the object translation, transforming them into a standard normalized coordinate system and Get the hand-object geometry code g h and g o ;

[0027] Step 5: Hand and Object SDF Decoder f HODecoder Estimate each 3D query point using a fine-grained hand-object shape encoding z Reconstruct the hand and object shape SDF h and SDF o .

[0028] In step 1, the two-stream RGB-point cloud feature encoder f TSRPE It consists of a convolutional neural network and a Transformer to extract RGB features from images and point clouds respectively. and point features

[0029] In step 2, the adaptive bidirectional fusion module f ABF Fusion of RGB features and point features

[0030] In this embodiment, the Figure 2 , the adaptive bidirectional fusion module f in step 2 ABF Using the complementarity of appearance and geometric information, the process is as follows:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] Among them, K r2p represents the nearest neighbor size for aggregating RGB features, K p2rIndicates the nearest neighbor size used to aggregate point features.

[0037] In this embodiment, the Figure 3 , the adaptive bidirectional fusion module f in step 2 ABF Using the sensitivity of different feature layers to the object reconstruction task, dense RGBD features are generated. The process is as follows:

[0038]

[0039]

[0040] Among them, ⊙ represents element-wise multiplication, σ represents the sigmoid function, and Concat is the concatenation operation along the channel dimension. Each modality branch can utilize local and global information from other modality branches to promote its representation learning and obtain the shape information of the interaction between the hand and the object.

[0041] In step 3, the SDF feature encoder consists of 1×1 convolution and fully connected layers.

[0042] In step 3, the hand-object geometric feature estimation module predicts the hand-object posture parameters T hr and the object position t o , the process is as follows:

[0043] The hand encoder composed of fully connected layers takes RGB-D features as input to regress the hand pose parameters θ p and the hand shape parameter β h . Then, the hand posture parameter θ p and shape parameter β h The vertex coordinates V, hand joint coordinates J, and hand posture parameters θ of the hand model are obtained through network model regression. h , global rotation of hand joint T hr and the global rotation center t h To obtain the object position, the voxel heat map H is subjected to a soft maximum operation to predict the center pixel position probability map of the object Then from Get voxel coordinates as follows:

[0044]

[0045]

[0046]

[0047] Among them, W input and H input is the width and height of the input image, W hm , H hm and Dhm is the width, height and depth of the heat map H, R d represents the depth radius relative to the wrist joint estimated based on the training data. op Use the camera intrinsic parameter θ and wrist joint coordinates get.

[0048]

[0049] Among them, c x and c y represents the coordinates of the camera's principal point, and f x and f y Indicates focal length.

[0050] In step 3, the loss function of the hand-object geometric feature estimation module is defined as follows:

[0051]

[0052] Among them, μ J , μ θ and μ β They are the hand joint coordinates J and the hand posture parameters θ p and the hand shape parameter β h The loss coefficient, is the true value of the hand joint coordinates, is the zero vector, It is the true value of the center of mass coordinate of the object in the hand relative coordinate.

[0053] In step 4, the geometric Fourier feature encoding module f GFFE Each 3D query point Normalize the rotation and translation of the wrist and the object to convert them into a standard normalized coordinate system and Get the hand-object geometry code g h and g o ;

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] in, From Gaussian distribution The modulation frequency obtained by sampling, σ is the variance of the Gaussian distribution. m represents the number of Fourier kernels.

[0060] In step 5, the hand and object SDF decoder f HODecoder Utilize fine-grained shape encoding z and hand geometry g h and g o Encode the estimated 3D query point Reconstruct the hand and object shape SDF h and SDF o , the process is as follows:

[0061] SDF h =f h (z,g h ),

[0062] SDF o =f o (z,g o )

[0063] In step 5, the hand and object SDF decoder f HODecoder It includes a hand SDF decoder and an object SDF decoder, both of which are constructed by fully connected layers, and their loss functions are defined as follows:

[0064]

[0065] in, and are the L1 distance losses of the predicted and true values ​​of the signed distances of the hand and the object, μ rech and μ reco are the loss coefficients for the signed distances of the hand and the object, respectively.

[0066] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.

[0067] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0068] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A hand-object interaction reconstruction method based on a single-view RGBD fusion network, characterized in that: The method steps include: Obtain aligned RGBD images of hand-object interaction, convert the depth map into a point cloud, and apply a two-stream RGB-point cloud feature encoder to extract RGB features and point features from the RGB image and point cloud data; The adaptive bidirectional fusion module utilizes the complementarity of appearance and geometric information and the sensitivity of different feature layers to the object reconstruction task to fuse the RGB features and point features to generate dense RGBD features; the adaptive bidirectional fusion module f ABF Using the complementarity of appearance and geometric information, the process is as follows: in, is the RGB feature, is the point feature, K r2p represents the nearest neighbor size for aggregating RGB features, K p2r Represents the nearest neighbor size used to aggregate point features; The adaptive bidirectional fusion module f ABF Using the sensitivity of different feature layers to the object reconstruction task, dense RGBD features F are generated rgbd , the process is as follows: Among them, ⊙ represents element multiplication, σ represents the sigmoid function, and Concat represents the concatenation operation along the channel dimension; The SDF feature encoder is used to generate the hand shape encoding from the dense RGBD features, and the hand geometry feature estimation module is used to predict the hand posture parameters and object position; the hand geometry feature estimation module predicts the hand posture parameters T hr and the object position t o , the process is as follows: The hand encoder composed of fully connected layers takes dense RGBD features as input and regresses the hand posture parameters θ p and the hand shape parameter β h ; The hand posture parameter θ p and shape parameter β h The vertex coordinates V, hand joint coordinates J, and hand posture parameters θ of the hand model are obtained through network model regression. h , global rotation of hand joint T hr and the global rotation center t h ; In order to obtain the object position, a soft maximum operation is applied to the voxel heat map H to predict the probability map of the center pixel position of the object. And get the three-dimensional pixel coordinates The details are as follows: Among them, W input and H input is the width and height of the input image, W hm , H hm and D hm is the width, height and depth of the heat map H, R d represents the depth radius relative to the wrist joint estimated based on the training data; The object position t o Use the camera intrinsic parameter θ and wrist joint coordinates Obtained, as follows: Among them, c x and c y represents the coordinates of the camera principal point, f x and f y Indicates focal length; The loss function of the hand-object geometric feature estimation module is defined as follows: Among them, μ J , μ θ and μ β They are the hand joint coordinates J and the hand posture parameters θ p and the hand shape parameter β h The loss coefficient of is the true value of the hand joint coordinates, is the true value of the center of mass coordinate of the object in the hand-relative coordinates; The rotation and translation of each three-dimensional query point relative to the wrist and the translation of the object are normalized through the geometric Fourier feature encoding module and converted into a standard normalized coordinate system to obtain the hand-object geometric encoding; The hand and object SDF decoder is used to estimate the SDF of each 3D query point using the hand-object shape encoding and hand-object geometry encoding to reconstruct the hand and object shapes.

2. According to claim 1, a hand-object interaction reconstruction method based on a single-view RGBD fusion network is characterized in that: The dual-stream RGB-point cloud feature encoder f TSRPE It consists of a convolutional neural network and a Transformer to extract RGB features and point features from images and point clouds, respectively.

3. According to claim 1, a method for reconstructing hand-object interaction based on a single-view RGBD fusion network is characterized in that: The SDF feature encoder consists of 1×1 convolutional and fully connected layers.

4. According to claim 1, a hand-object interaction reconstruction method based on a single-view RGBD fusion network is characterized in that: The geometric Fourier feature encoding module f GFFE Each 3D query point Normalized relative to the wrist rotation and translation and the object translation and converted to a standard normalized coordinate system and And obtain the hand-object geometry code g h and g o ; Among them, T hr represents the global rotation of the hand joint, From Gaussian distribution The modulation frequency obtained by sampling, σ is the variance of the Gaussian distribution.

5. According to claim 4, a method for reconstructing hand-object interaction based on a single-view RGBD fusion network is characterized in that: The hand and object SDF decoder f HODecoder It includes an SDF decoder for a hand and an SDF decoder for an object, wherein the SDF decoder for the hand and the SDF decoder for the object are both constructed by a fully connected layer; The hand SDF decoder and the object SDF decoder are respectively encoded by the hand geometry g h and geometry coding o and shape encoding z estimates for each 3D query point Reconstruct the hand and object shape SDF h and SDF o : SDF h =f h (z,g h ) SDF o =f o (z,g o )。 6. The hand-object interaction reconstruction method based on a single-view RGBD fusion network according to claim 5 is characterized in that: The hand and object SDF decoder f HODecoder The loss function is defined as follows: in, and are the L1 distance losses of the predicted and true values ​​of the signed distances of the hand and the object, μ re and μ reco are the loss coefficients for the predicted and true values ​​of the signed distances of the hand and object, respectively.

Images