A 3D interactive hand reconstruction method and system based on implicit representation

Through the three-dimensional interactive hand reconstruction method based on implicit surface representation, the parameterized implicit neural network and physical optimization scheme are used to solve the problem of chirality and unlimited number in a single color image, and high-quality physical rationality is achieved The reconstruction of hand shape and posture solves the problems of spatial entanglement and penetration in the prior art.

CN114998520BActive Publication Date: 2024-11-12SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210619894.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-02
Publication Date
2024-11-12
Estimated Expiration
2042-06-02

AI Technical Summary

Technical Problem

The prior art is difficult to effectively reconstruct the physically reasonable hand shape and posture from a single color image without limiting chirality and hand quantity, and there are problems of spatial entanglement and penetration, resulting in low reconstruction quality.

Method used

Using a three-dimensional interactive hand reconstruction method based on implicit surface representation, a parameterized implicit neural network is built, query strategies and conditional design is carried out through image features, combined with a physical optimization solution to solve the problem of mutual penetration of instances under implicit representation, and iteratively optimizes until it is optimized. The maximum penetration depth is less than 2mm.

Benefits of technology

High-quality reconstruction of any chirality and number of hands from a single color image is achieved, improving the physical authenticity and efficiency of the reconstruction results, and ensuring the physical rationality of the interactive hands.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114998520B_ABST
    Figure CN114998520B_ABST
Patent Text Reader

Abstract

The present invention proposes a method and system for reconstructing a three-dimensional interactive hand based on implicit representation, including the following steps: 1. Building a feature extraction neural network, first obtaining global and regional features from a single input color image, and then further obtaining instance and joint features based on the input image and regional features; 2. Using the obtained regional features, instance features and joint features, selecting query points according to the query strategy, and constructing query conditions; 3. Constructing a parameterized implicit neural network, and performing implicit reconstruction based on the query points and query conditions; 4. Physically optimizing the reconstructed model, penalizing unreasonable penetration, adjusting and updating the reconstructed model; 5. Iteratively optimizing the reconstructed model until the maximum penetration depth is less than 2mm, and then using the optimization result as the final reconstruction result of the interactive hand. The present invention only needs a single color image to reconstruct any chirality and number of hands in the image, realizing an end-to-end modeling method, and improving the quality of three-dimensional gesture and shape reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and computer graphics, and in particular to a three-dimensional interactive hand reconstruction method and system based on implicit representation. Background Art

[0002] In recent years, 3D gesture and shape reconstruction technology has become a hot research direction, with broad application prospects in the fields of virtual reality, robot control, and somatosensory games. In real daily life, interactive hands are everywhere and play an important role in information exchange between people, such as single-person emotional expression and multi-person collaboration. Just like human activities in real life, the interaction between hands in virtual reality is also an important content. Therefore, it is necessary to study the reconstruction of interactive hands. Compared with single hands, due to the interaction, interactive hands are usually severely occluded, in close contact with each other, and have similar texture features. In addition, the solution space of interactive hands has greater degrees of freedom.

[0003] Early methods tend to utilize additional depth information or more perspectives as input. However, they always fit the input data to a specific hand shape template without personalized changes and cannot adapt to existing learning-based methods. Due to the ubiquitous nature of monocular color images, methods using monocular color images are preferable to methods using multi-camera and depth cameras. The construction of the Interhand2.6M dataset brought data support to this problem and subsequently promoted the birth of a series of methods for reconstructing interacting hands from monocular color images. However, these interactive hand reconstruction works still have the following problems:

[0004] 1) They usually rely on the assumption that there is only one left hand and one right hand in the image, reformulating the problem as the regression of MANO parameters of two hands. This limits the number of chirality and hands in the image. 2) The reconstructed interacting hands have physically unreasonable problems such as spatial entanglement and interpenetration, resulting in low reconstruction quality. Summary of the invention

[0005] The technical problem to be solved by the present invention is to reconstruct physically reasonable hand shapes and postures from a single color image without restrictions on handedness and the number of hands.

[0006] In order to solve the above technical problems, the present invention proposes a 3D interactive hand reconstruction method based on implicit surface representation. The method constructs a parameterized implicit neural network, which can classify 3D query points in the normative space of interactive hands using image features as query strategies and conditions. According to image clues, new query strategies and query conditions are designed to improve query efficiency and reduce query space. In addition, an effective physical optimization scheme is provided to solve the problem of mutual penetration between instances under implicit representation, effectively ensuring the physical authenticity of the reconstruction results.

[0007] The three-dimensional interactive hand reconstruction method based on implicit representation proposed in the present invention comprises the following steps:

[0008] Step 1: Build a feature extraction neural network to first obtain global features from a single input color image, divide the global features into multiple regional features based on the connected parts, and then further obtain instance features and joint features based on the input image and regional features;

[0009] Step 2, using the global features, regional features and joint features obtained in step 1, select query points according to the query strategy and construct query conditions;

[0010] Step 3: Build a parameterized implicit neural network and perform implicit reconstruction based on query points and query conditions;

[0011] Step 4, physically optimize the reconstructed model, penalize unreasonable penetration, and adjust and update the reconstructed model;

[0012] Step 5, iteratively optimize the reconstructed model until the maximum penetration depth is less than 2 mm, and then use the optimization result as the final reconstruction result of the interactive hand.

[0013] Furthermore, step 1 uses two encoder-decoder neural networks with ResNet18 as the backbone, and the acquired features are supervised during the network training process.

[0014] Furthermore, the training can constrain each feature through the following loss function:

[0015] L feature =L Global +L instance +L joint

[0016] The first term of the loss function L Global Indicates constraints on the global mask and global Z-map, and the second term of the loss function L instance Indicates that the instance mask is constrained, and the third term of the loss function L joint Indicates supervision of the joint positioning map and Z-map. Global , Linstance and L joint The formulas are defined as follows:

[0017]

[0018]

[0019]

[0020] L Global The loss function and are the network prediction values ​​of the global mask and global Z-map, respectively. and is the corresponding truth value; L instance The loss function and denote the network prediction value and true value of the kth instance mask in region i respectively; L joint The loss function and are the localization map of the joint in region i and the network prediction value of the Z-map, Yes and is the corresponding truth value.

[0021] Furthermore, in step 1, for the input color image I containing the interacting hands, the first encoder-decoder network is first used to obtain the global mask G M and the global Z-mapG Z Then, use the bounding box coordinates of each region to crop I, G M and G Z To obtain regional information, that is, regional image Region Mask area Next, according to and Using a second encoder-decoder network, localization maps of visible joints are extracted in parallel for each region i and and the mask E of the visible instances (k) k represents the number of visible instance masks in region i.

[0022] Further, in step 2, for each region Construct sampling space d according to query strategy (i) , and sample points in the sampling space. Combine all the sampled points as query points and normalize them into a unit cube. In addition, for each region, obtain the region embedding r from the feature extraction in step 1 (i) , each instance E in the region(k) Send it to the multi-layer perceptron to encode and get the instance embedding e (k) . (i) and e (k) As a query condition for implicit reconstruction.

[0023] Furthermore, in step 3, based on the query point and the query condition, a parameterized implicit neural network is used for implicit reconstruction. The parameterized implicit neural network can be regarded as an implicit function, and the formula is as follows:

[0024]

[0025] Among them, p is the query point, r and e are the region embedding and instance embedding respectively, which are used as query conditions, and τ represents the output occupancy value.

[0026] The neural network is supervised using the cross entropy loss between the predicted occupancy and the true occupancy. In addition, we define a collision loss to penalize the collision of the hand surface. Considering that a query point belongs to at most one hand instance, the penetration loss is defined as:

[0027]

[0028] Among them, Ω represents all query points and k is equal to the number of hands in each area.

[0029] Furthermore, in step 4, after the hand surface is implicitly reconstructed, a physical rationality check is performed on the reconstructed interactive hand to determine whether the reconstructed interactive hand model has penetration. If penetration exists, a physical optimization method is used to optimize the reconstructed model.

[0030] Furthermore, in step 5, considering that the hand is non-rigid, a penetration of less than 2 mm is allowed. The model is reconstructed according to the iterative optimization of step 4 until the depth d is less than 2 mm.

[0031] Furthermore, the system includes a feature extraction module, an implicit reconstruction module, and a physical optimization module. The feature extraction module includes two units: a global feature extraction unit and a regional feature extraction unit, which obtains a global mask G from a single input image. M , global Z-mapG Z , Regional Image Region Mask and Region The instance and joint feature estimation unit extracts the localization map of the visible joints for each region i in parallel based on the region features. and And the visible instance mask E (k) The implicit reconstruction module consists of two units: query point and query condition acquisition unit. For each region Select query points according to the query strategy and obtain the region embedding r from the feature extraction module (i) , through the multi-layer perceptron from the instance mask E of the region (k) The instance embedding {e (k)}, r (i) and e (k) As the query condition for implicit reconstruction; the implicit surface reconstruction unit obtains the occupancy value of each query point through a parameterized implicit neural network, and further extracts the reconstructed surface. The physical optimization module determines whether the reconstructed model has penetration. If there is penetration, the physical optimization method is used to optimize the reconstructed model, and the reconstructed model is iteratively optimized until the maximum penetration depth is less than 2mm. The optimization result is used as the final reconstruction result of the interactive hand.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] The present invention proposes a 3D interactive hand reconstruction method based on implicit representation, which can reconstruct hands of any chirality and number in the image with only a single color image, realizes an end-to-end modeling approach, and improves the quality of 3D gesture and shape reconstruction.

[0034] In addition, new query strategies and query conditions are designed based on image clues, which improves query efficiency and reduces the query space.

[0035] In addition, for physically unreasonable reconstruction results, an effective physical optimization scheme is provided to solve the problem of mutual penetration between instances under implicit representation, which effectively ensures the physical authenticity of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a three-dimensional interactive hand reconstruction method according to an embodiment of the present invention;

[0037] Figure 2 is a schematic structural diagram of a three-dimensional interactive hand reconstruction system in an embodiment of the present invention;

[0038] Figure 3 1 is a diagram of a physical optimization solution in an embodiment of the present invention, wherein (a) is a force calculation for each pixel, and (b) is a force calculation for each instance.

[0039] Figure 4 It is a reconstruction effect diagram that can be achieved by the present invention, wherein (a) column of images is the input image, (b) column of images is the real mask, (c) column of images is the predicted mask, (d) column of images is the implicit reconstruction result, and (e) two columns of images are the results of the two viewing angles after optimization. DETAILED DESCRIPTION

[0040] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings and embodiments, thereby combing the methods used in the present invention and the implementation effects, so that users can have a clearer understanding of the present invention. It is worth noting that the features of the embodiments of the present invention can be combined with each other without conflict, and the technical solutions formed are all within the protection scope of the present invention.

[0041] In addition, the flowcharts shown in the drawings can be executed in a computer as a series of continuous instructions, and the flow order may be appropriately modified in some cases.

[0042] Embodiment 1

[0043] Figure 1 This is a flowchart of a three-dimensional interactive hand reconstruction method based on implicit representation according to the first embodiment of the present invention. Figure 1 Describe each step in detail.

[0044] Step S110, building a feature extraction neural network to obtain corresponding features from the input color image.

[0045] In order to obtain the corresponding features from a single color image, a feature extraction network is built. Figure 2 As shown in S1, the feature extraction network consists of two encoder-decoder neural networks with ResNet18 as the backbone. For the input color image I containing the interacting hands, the first encoder-decoder network is used to obtain the global mask G M and the global Z-mapG Z Then, use the bounding box coordinates of each region to crop I and G M and G Z To obtain regional information, that is, regional image Region Mask area Next, according to and Using a second encoder-decoder network, localization maps of visible joints are extracted in parallel for each region i and And the visible instance mask E (k) . Where k represents the number of visible instance masks in region i.

[0046] In order to achieve the above functions, the feature extraction network is pre-trained using the following loss function:

[0047] L feature =L Global +L instance +L joint

[0048] The first term of the loss function L GlobalIndicates supervision of the global mask and global Z-map, the second term of the loss function L instance Indicates supervision of instance masks, the third term of the loss function L joint Indicates supervision of the joint positioning map and Z-map. Global , L instance and L joint The formulas are defined as follows:

[0049]

[0050]

[0051]

[0052] L Global The loss function and are the network prediction values ​​of the global mask and global Z-map, respectively. and is the corresponding truth value; L instance The loss function and denote the network prediction value and true value of the kth instance mask in region i respectively; L joint The loss function and are the localization map of the joint in region i and the network prediction value of the Z-map, Yes and is the corresponding truth value.

[0053] Step S120, using the acquired features, selecting query points according to the query strategy, and constructing query conditions.

[0054] The specific process of query point selection is as follows: For each area First find the corresponding query silhouette and maximum query depth Then, use As the bottom area, As the height, construct the sampling space d (i) . Sample points uniformly in this space. In addition, consider each visible joint as an anchor. Select some more sampling points near each anchor. Specifically, use the Gaussian mixture distribution (x (j) ,σ), where each visible joint coordinate x (j) As the center, σ as the variance of each dimension. In the experiment of the present invention, σ is set to 2.5cm. All sampling points are combined as query points and normalized into a unit cube. The specific process of constructing the query condition is as follows: For each area from Figure 2 Get the region embedding r from the S1 feature extraction (i) For each instance E in the region (k) Send it to the multi-layer perceptron to encode and get the instance embedding e (k) . (i) and e (k) As a query condition for implicit reconstruction.

[0055] Step S130, building a parameterized implicit neural network, and performing implicit reconstruction based on the query points and query conditions.

[0056] like Figure 2 As shown in S2, based on the query point and query condition, a parameterized neural network is established It is considered as an implicit function and the formula is as follows:

[0057]

[0058] Among them, p is the query point, r and e are the region embedding and instance embedding respectively, which are used as query conditions, and τ represents the output occupancy value.

[0059] In the experimental stage of the present invention, the hand surface is implicitly represented by an isosurface with τ = 0.5. If the occupancy of the query point is less than 0.5, it means that the point is outside the surface, and if the occupancy value is greater than 0.5, the point is inside the surface.

[0060] We supervise the neural network using the cross entropy loss between the predicted occupancy and the true occupancy. In addition, we define a collision loss to penalize the collision of the hand surface. Considering that a query point belongs to at most one hand instance, the penetration loss is defined as:

[0061]

[0062] Among them, Ω represents all query points and k is equal to the number of hands in each area.

[0063] The parameterized implicit neural network consists of three convolutional layers and nine fully connected layers. It is trained using the SGD optimizer with a batch size of 64. The query points are not fixed and are resampled in each iteration.

[0064] Step S140 , performing a physical rationality check on the reconstructed interactive hand, and optimizing and updating the reconstructed model.

[0065] After implicitly reconstructing the hand surface, the reconstructed interactive hand is physically checked for plausibility and unreasonable penetration is penalized to better align with the real gesture. Figure 3 As shown, the penetration depth is first calculated by projecting the area mask in the positive Z direction. The observed rays should continuously pass through a hand, and the number of intersections should be even. Suppose there is a ray that passes through the implicitly reconstructed region of two interacting hands A and B. Along this ray, record all intersections as Where N is the number of intersection points. For the intersection points on the surface of hand A if and If the surfaces do not belong to the same hand, it means that penetration has occurred at this location. In this case, find the next intersection point on the surface of hand A. Define the penetration depth as the intersection point and The distance between the two pixels. Then, the force is calculated for each pixel. In the optimization stage, it is hoped that the hand posture can be adjusted to any direction, not parallel to the light. Considering that the force is directional and the magnitude of the force is positively correlated with the penetration depth, the location where the penetration occurs can generate a repulsive force like a spring. In this case, the hand is considered as a rigid body. The relationship between the penetration depth and the corresponding repulsive force is defined as:

[0066]

[0067] Among them, f (u,v) represents the force generated by the light emitted along the pixel (u, v) due to penetration, t represents the number of penetrations, λ is the related weight, d t represents the tth penetration depth.

[0068] Add up the repulsive forces of all rays. Due to the different repulsive forces of each ray, the direction of the resultant force F may deviate from the z-axis direction. The formula for calculating the resultant force is as follows:

[0069]

[0070] Based on the resultant force F, calculate the average penetration depth Then, for each hand, move in a direction parallel to F To penalize extreme adjustments, a projection loss is used, in which orthogonal projection is used.

[0071]

[0072] Among them, H pose represents the optimized interaction hand, π(H pose ) is its 2D projection, R M Represents the region mask estimated by the encoder-decoder.

[0073] Step S150, iteratively optimize the reconstructed model until the maximum penetration depth is less than 2 mm, and use the optimization result as the final reconstruction result.

[0074] Considering that the hand is non-rigid, a penetration of less than 2 mm is allowed. The model is reconstructed according to the iterative optimization in step S140 until the depth d is less than 2 mm.

[0075] In this embodiment 1, the input color image can contain any chirality and number of hands. The experimental results are as follows: Figure 4 As shown. Among them, Figure 4 The first column represents the input color image, the second and third columns represent the true mask and the predicted mask respectively, the fourth column is the implicit reconstruction result, and the fifth column represents the optimized reconstruction result.

[0076] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

Claims

1. A 3D interactive hand reconstruction method based on implicit representation, characterized by: The method comprises the following steps: Step 1: Build a feature extraction neural network to first obtain global and regional features from a single input color image, and then further obtain instance features and joint features based on the input image and regional features; Step 2, using the features obtained in step 1, select query points according to the query strategy and construct query conditions; Step 3: Build a parameterized implicit neural network and perform implicit reconstruction based on query points and query conditions; Step 4, physically optimize the reconstructed model, penalize unreasonable penetration, and adjust and update the reconstructed model; Step 5, iteratively optimize the reconstructed model until the maximum penetration depth is less than 2 mm, and then use the optimization result as the final reconstruction result of the interactive hand; In step 2, for each region Construct sampling space d according to query strategy (i) , and sample points in the sampling space; combine all the sampled points as query points and normalize them into a unit cube; in addition, for each region, obtain the region embedding r from the feature extraction in step 1 (i) , each instance E in the region (k) Send it to the multi-layer perceptron to encode and get the instance embedding e (k) ; r (i) and e (k) As a query condition for implicit reconstruction; In step 3, based on the query point and the query condition, a parameterized implicit neural network is used for implicit reconstruction; the parameterized implicit neural network is regarded as an implicit function, and the formula is as follows: Where p is the query point, r and e are the region embedding and instance embedding respectively, which are used as query conditions, and τ represents the output occupancy value; The cross entropy loss between the predicted occupancy and the true occupancy is used to supervise the neural network. In addition, a collision loss is defined to penalize the collision of the hand surface. Considering that a query point belongs to at most one single hand instance, the penetration loss is defined as: Among them, Ω represents all query points and k is equal to the number of hands in each area.

2. The three-dimensional interactive hand reconstruction method based on implicit representation according to claim 1, characterized in that: The step 1 uses two encoder-decoder neural networks with ResNet18 as the backbone, and the acquired features are supervised during the network training process.

3. The implicit representation-based 3D interactive hand reconstruction method according to claim 2, characterized in that: The training constrains each feature through the following loss function: L feature =L Global +L instance +L joint The first term of the loss function L Global Indicates constraints on the global mask and global Z-map, and the second term of the loss function L instance Indicates that the instance mask is constrained, and the third term of the loss function L joint Indicates supervision of the joint positioning map and Z-map; L Global , L instance and L joint The formulas are defined as follows: L Global The loss function and are the network prediction values ​​of the global mask and global Z-map, respectively. and is the corresponding truth value; L instance The loss function and denote the network prediction value and true value of the kth instance mask in region i respectively; L joint The loss function and are the localization map of the joint in region i and the network prediction value of the Z-map, Yes and is the corresponding truth value.

4. The three-dimensional interactive hand reconstruction method based on implicit representation according to claim 1, characterized in that: In step 1, for the input color image I containing the interacting hands, the first encoder-decoder network is first used to obtain the global mask G M and the global Z-mapG Z ; Then, use the bounding box coordinates of each region to crop I, G M and G Z To obtain regional information, that is, regional image Region Mask area Next, according to and Using a second encoder-decoder network, localization maps of visible joints are extracted in parallel for each region i and and the mask E of the visible instances (k) ; k represents the number of visible instance masks in region i.

5. The three-dimensional interactive hand reconstruction method based on implicit representation according to claim 1, characterized in that: In step 4, after the hand surface is obtained by implicit reconstruction, a physical rationality check is performed on the reconstructed interactive hand to determine whether the reconstructed interactive hand model has penetration. If penetration exists, the reconstructed model is optimized using a physical optimization method.

6. The three-dimensional interactive hand reconstruction method based on implicit representation according to claim 1, characterized in that: In step 5, considering that the hand is non-rigid, penetration within 2 mm is allowed; the model is reconstructed according to the iterative optimization of step 4 until the depth d is less than 2 mm.

7. A 3D interactive hand reconstruction system based on implicit representation, characterized in that: The method according to any one of claims 1 to 6 is adopted, wherein the system comprises a feature extraction module, an implicit reconstruction module, and a physical optimization module; The feature extraction module consists of two units: global and regional feature extraction units, which obtain a global mask G from a single input image. M , global Z-mapG Z , Regional Image Region Mask and Region The instance and joint feature estimation unit extracts the localization map of the visible joints for each region i in parallel based on the region features. and And the visible instance mask E (k) ; The implicit reconstruction module includes two units: query point and query condition acquisition unit. For each region Select query points according to the query strategy and obtain the region embedding r from the feature extraction module (i) , through the multi-layer perceptron from the instance mask E of the region (k) The instance embedding {e (k) }, r (i) and e (k) As the query condition for implicit reconstruction; the implicit surface reconstruction unit obtains the occupancy value of each query point through a parameterized implicit neural network, and further extracts the reconstructed surface; The physical optimization module determines whether the reconstructed model has penetration. If penetration exists, the reconstructed model is optimized using a physical optimization method. The reconstructed model is iteratively optimized until the maximum penetration depth is less than 2 mm. The optimization result is used as the final reconstruction result of the interactive hand.