A 3D human body mesh completion method based on implicit neural field representation

Abstract

The present invention provides a 3D human mesh completion method based on implicit neural field representation, which relates to the technical field of 3D reconstruction technology. The method includes the following steps: establishing a human mesh completion model, establishing an implicit neural field, training the human mesh completion model, sampling the space where the complete human mesh is located to obtain sampling points, calculating the truncated signed distance function and color corresponding to the sampling points as labels; obtaining color information based on the color of the closest patch to the three-dimensional mesh surface from the spatial point corresponding to the sampling point; sampling the sparse 3D point cloud position, color, and surface normal vector data of the missing human mesh surface to be completed and inputting them into the trained model to establish an implicit neural field, and then extracting the complete human mesh using the MarchingCubes algorithm. This paper proposes a mesh recovery network that reconstructs the 3D shape of the complete structure and restores texture details based on the input partial 3D shape and corresponding texture information.

Description

Technical Field

[0001] The present invention relates to the field of 3D reconstruction technology, and in particular to a 3D human body mesh completion method based on implicit neural field representation. Background Art

[0002] Three-dimensional reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects. It is the basis for processing, operating and analyzing its properties in a computer environment, and is also a key technology for establishing virtual reality in computers to express the objective world.

[0003] Existing data-driven learning methods for 3D object reconstruction are generally based on representations such as point clouds, voxels, or triangular meshes. The first method uses an autoencoder algorithm to learn linear blend skinning (LBS) to recover joint shapes from point clouds. The second method is a voxel-based method that uses a three-dimensional voxelized mesh as one of the shape representations and establishes a one-to-one mapping between the input and the voxelized shape. The third method is through a triangular mesh representation. The fourth method is based on a deep implicit function method that uses a neural network to implicitly encode 3D shapes, such as DeepSDF. These 3D object reconstruction works generally use the position of a 3D space point and the implicit encoding as the input of the implicit field to predict the occupancy probability or signed distance value corresponding to the 3D space query point.

[0004] However, existing 3D object reconstruction methods have shortcomings. Methods for recovering joint shapes from point clouds, although point clouds are driven by high efficiency and high flexibility, do not provide any topological information, which brings challenges to reconstruction. Voxel-based methods are usually limited by memory size and can only reconstruct low-resolution discrete surfaces. Methods represented by triangular meshes have limited surface details and fixed topological structures, which are not conducive to the reconstruction of fine structures and are insufficient to represent high-quality shapes of objects, human bodies, etc. Methods based on deep implicit functions are difficult for more complex shapes. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to propose a 3D human body mesh completion method based on implicit neural field representation, by introducing a unified representation of the human body's implicit neural field to solve the technical problem that the existing human body mesh topological structure is inconsistent and difficult to repair.

[0006] The technical means adopted in the present invention are as follows:

[0007] A 3D human body mesh completion method based on implicit neural field representation, the method comprising:

[0008] Building a human mesh completion model includes the following steps:

[0009] Sampling the missing human body mesh surface to obtain sparse 3D point cloud position, color, and surface normal vector data, using the sparse 3D point cloud position, color, and surface normal vector data as input to the human body mesh completion model, and normalizing the sparse 3D point cloud position to a standard space and performing voxel block division to obtain sparse 3D point cloud information;

[0010] The sparse 3D point cloud information is encoded and diffused through the structured implicit coding network Points2VoxNet to obtain multi-scale geometric feature volumes and texture feature volumes;

[0011] Establish an implicit neural field. For any point x in space, use trilinear interpolation on the geometric feature volume and texture feature volume to obtain the geometric features and texture features of point x. Input the geometric features and texture features of point x into the fully connected network for decoding to obtain the geometric representation and texture representation of point x.

[0012] Training the human body mesh completion model includes the following steps:

[0013] Sampling the space where the complete human body grid is located to obtain sampling points, and calculating the truncated signed distance function and color corresponding to the sampling points as labels;

[0014] The color information is obtained according to the color of the nearest patch from the spatial point corresponding to the sampling point to the three-dimensional grid surface;

[0015] The sampling points are input into the implicit neural field to predict the truncated signed distance function and color of the sampling points, and the optimized objective function between the predicted and true labels is calculated to train and optimize the model.

[0016] Get the trained human mesh completion model;

[0017] The trained human body mesh completion model is used to complete the human body mesh to be completed, including the following steps:

[0018] Given a missing human body mesh to be completed;

[0019] The sparse 3D point cloud position, color and surface normal vector data of the missing human body mesh surface to be completed are sampled and input into the trained model to establish an implicit neural field, and then the complete human body mesh is extracted using the Marching Cubes algorithm.

[0020] Furthermore, it also includes:

[0021] Define implicit representations before building implicit neural fields;

[0022] Using signed distance field as implicit representation, we get implicit representation formula;

[0023] Transforming implicit fields into explicit representations that can ultimately be presented;

[0024] Get sdf i The value is truncated and truncation is performed to obtain tsdf i value;

[0025] According to the sdf i Values ​​and tsdf i It is worth to obtain the implicit field formula for 3D shape reconstruction.

[0026] Furthermore, the formulas for the multi-scale geometric feature volume V and texture feature volume T are:

[0027]

[0028] Among them, φ is the Points2VoxNet network, M is the number of points in the sparse 3D point cloud, and p k is the 3D position of the kth point in the sparse 3D point cloud, is the RGB value of the point, is the normal vector corresponding to the point.

[0029] Furthermore, the formulas for geometric feature coding g(x) and texture feature coding t(x) are:

[0030]

[0031] in, is trilinear interpolation, and x is the 3D position of any point in space.

[0032] Furthermore, the formula for geometric representation tsdf and texture representation rgb is:

[0033] tsdf=M g (x, g(x))

[0034] rgb=M t (x, g(x), t(x)).

[0035] Among them, M g is the geometric decoding network, M t For texture decoding network.

[0036] Furthermore, the formula of the trained human mesh completion model is:

[0037]

[0038] Among them, x i is the sampling point of the space where the complete human body grid is located, N is the number of sampling points, λ g is the geometric regression loss weight, λ tRecover loss weights for textures.

[0039] Furthermore, the implicit field for 3D shape reconstruction is:

[0040]

[0041] Where x∈R 3 is a point in 3D space, sdf is the signed distance of point x, which represents the directed distance from point x to the nearest surface point, tsdf is the truncated signed distance of point x; c is the cutoff threshold of the truncated signed distance, and θ is the implicit field network parameter.

[0042] Furthermore, sampling the space where the complete human body grid is located to obtain sampling points includes sampling near the surface of the complete human body grid and uniformly sampling in the entire standard space.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] Our method adopts a multi-scale 3D shape representation and uses structured implicit coding to reconstruct the 3D shape of the complete structure from sparse 3D information such as partial point clouds, preserving the global structure while retaining fine-scale details even for complex poses.

[0045] This invention completes the 3D image from partial information and restores the surface texture at the same time. Reconstructing non-rigid objects has always been challenging, such as high flexibility and complex structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 Flow chart of the method of the present invention.

[0048] Figure 2 This is the algorithm framework diagram of the present invention.

[0049] Figure 3 This is the human body grid completion diagram of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] like Figure 1 As shown, the present invention provides a 3D human body mesh completion method based on implicit neural field representation.

[0053] Although many works focus on 3D reconstruction from a single or multiple perspectives, the present invention focuses on achieving three-dimensional shape reconstruction and surface texture recovery from sparse 3D shape representations. Since sparse 3D shape representations have the disadvantages of lack of surface details, discrete low resolution, and incomplete and missing structures, for many application scenarios, what is needed is a continuous, complete, high-resolution shape representation to reconstruct the complete 3D shape and restore its surface texture from a discrete partial 3D shape representation with missing surface details. To achieve this goal, the present invention proposes a mesh recovery network to reconstruct the 3D shape of its complete structure and restore the texture details based on the input partial 3D shape and the corresponding texture information. In order to promote the understanding of the model proposed by the present invention, the present invention first introduces implicit representation in the first part, introduces structured implicit coding in the second part, and finally introduces the shape reconstruction and texture recovery of this method.

[0054] Implicit Representation

[0055] There are many types of 3D shape representations, which can be divided into two categories: explicit representation and implicit representation. Compared with explicit representation, implicit field has the advantages of continuity and no resolution limitation. Therefore, many works are based on implicit representation to perform 3D modeling of fine static object shapes. There are many different representations for implicit representation, such as signed distance field or binary occupancy field, but these representations are conceptually similar. The present invention uses a signed distance field as an implicit representation. Specifically, the implicit field used in the present invention can represent the signed closest distance sdf of any point in 3D space relative to the 3D surface. The symbols + / - represent that the spatial point is outside / inside the 3D surface, and its absolute value is equal to the closest distance between the spatial point and the 3D surface.

[0056] f(p i , z) = sdf i :p i ∈R 3

[0057] where f(·) is an implicit function, given any 3D point p in space i And implicit encoding, the implicit function can predict the sdfi value corresponding to the point. When the sdfi value is equal to 0, the point is located on the surface of the 3D object to be reconstructed. Therefore, in order to convert the implicit field into an explicit representation that can be displayed in the end, the points on the decision boundary surface (sdfi=0) in the implicit field can be used to reconstruct its 3D shape through Marching Cubes. In addition, compared to explicit representation, implicit representation is not limited by resolution and can query any point to achieve surface reconstruction of 3D shapes with more complex topological structures. In the training stage of the present invention, it is necessary to prepare a series of data pairs ψ in advance. The data pairs consist of 3D sampling points and corresponding sdf values, that is:

[0058] Ψ={(p i , sdf i ):f θ (p i , z g , z t )=sdf i

[0059] where z g Represents geometric encoding, z t Represents texture encoding, θ is the implicit field network parameter. However, considering that the spatial sampling points far away from the surface are not very useful for model fitting, and make the calculation and model fitting pressure greater, sdf i The value is truncated at [-c, c], where c = 0.01. i Make a distinction and express it as tsdfi , the sdf of the points far from the surface i Truncating the value will make the model focus more on recovering surface details, so the implicit field for 3D shape reconstruction is represented as:

[0060]

[0061] Structured implicit encoding

[0062] Since the present invention realizes the simultaneous reconstruction of 3D shape and restoration of surface texture, it is inspired by Neural Body and binds structured implicit coding to the complete 3D model surface. Unlike Neural Body which uses SMPL model as 3D surface, the training sample data is a partial 3D mesh, and the topological structure between the data is different. It is unrealistic to use vertices as input. In addition, directly implicitly encoding the surface space points can not only control the spatial position and relationship of the implicit coding, but also make the model pay attention to its surface texture details, so that the model can ensure the spatial position during geometric coding and feature alignment during texture coding. Therefore, in the training stage, the input part of the 3D surface is uniformly sampled, and the joint implicit field of geometric coding and texture coding is obtained by the input 3D spatial point position. The coding of other points in the space is obtained by querying. In order to obtain a continuous implicit field, that is, the geometric coding and texture coding of any point in the space can be queried, which is very simple. The method is to perform trilinear interpolation directly, but experiments have observed that since the input 3D space points are relatively sparse, the codes corresponding to most space points will be zero. To solve this problem, a code diffusion operation is used. Based on sparse 3D convolution, Points2VoxNet is proposed to effectively process the input structured implicit code and diffuse the implicit code to other points in space. First, the sparse structured implicit code of the input is preliminarily constructed to obtain the corresponding implicit code volume. The structured implicit code is diffused to other empty voxel blocks through sparse convolution. Finally, given any 3D space point x, the corresponding geometric code z is obtained by querying the corresponding implicit code voxel. g And texture encoding z t , specifically defined as:

[0063] z g , z t =φ(x,x c , x norm )

[0064] Where x represents the position of the 3D space point, x c Indicates the RGB value of the point, x norm Represents the normal vector corresponding to the point.

[0065] Shape reconstruction and texture restoration

[0066] Figure 2 The geometry and texture regression of any point in 3D space are summarized and implemented through different MLP networks. For shape geometry regression, the tsdf value of the spatial point is predicted by geometric implicit coding, which is specifically defined as:

[0067] tsdf i =M g (x i , z g )

[0068] Geometric regression only needs to ensure spatial position and relationship, but for texture regression, in addition to the texture implicit encoding controlling the surface color, spatial position is also required to ensure feature alignment, so as to better ensure structural consistency and spatiality. The specific definition is:

[0069] RGB i =M t (x i , z g , z t )

[0070] The method jointly trains geometric regression and texture recovery using L2 loss:

[0071]

[0072] where λ g ,λ t Represents the hyperparameters of the geometric regression and texture recovery loss functions, respectively, and are set to 10 2 and 1.

[0073] Example

[0074] Human body mesh completion Figure 3As shown, this embodiment first takes part of the 3D point cloud as the input of the entire network, including the point cloud position, color and surface normal vector. The sparse point cloud is first normalized to the standard space and divided into voxel blocks with a resolution of (224, 224, 224). Then, the sparse point cloud information is encoded and diffused through the structured implicit coding network Points2VoxNet to obtain a joint implicit field of geometric coding and texture coding. This field can encode the geometric information and texture information of any point in the space, where the geometric coding and texture coding dimensions are both set to 64. The network is divided into 5 layers with the number of channels being ( 32, 64, 128, 256, 256), each layer is downsampled, reducing the resolution from the original 224 to 7. Then, multi-scale feature fusion is performed to preserve global and local information. The implicit field of each scale is queried separately, and the multi-scale spatial encoding is aggregated to form a 736-dimensional encoding. Finally, the geometric and texture encoding of the spatial point is obtained through a linear layer. Finally, the Points2Nerf network is used to achieve final shape reconstruction and texture recovery. Specifically, the geometric encoding is predicted with the point position, and the geometry and texture joint encoding is predicted with the point position. The entire network is trained using Adam optimization, with an initial learning rate set to 0.0005 and cosine annealing as the learning rate. Convergence is achieved after 30 epochs on two 2080Ti.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D human body mesh completion method based on implicit neural field representation, characterized in that: The method comprises: Building a human mesh completion model includes the following steps: Sampling the missing human body mesh surface to obtain sparse 3D point cloud position, color, and surface normal vector data, using the sparse 3D point cloud position, color, and surface normal vector data as input to the human body mesh completion model, and normalizing the sparse 3D point cloud position to a standard space and performing voxel block division to obtain sparse 3D point cloud information; The sparse 3D point cloud information is encoded and diffused through the structured implicit coding network Points2VoxNet to obtain multi-scale geometric feature volumes and texture feature volumes; The encoding and diffusion method of the structured implicit coding network Points2VoxNet is as follows: the implicit coding is diffused to other points in space. First, the sparse structured implicit coding of the input is preliminarily constructed to obtain the corresponding implicit coding volume. The structured implicit coding is diffused to other empty voxel blocks through sparse convolution. Finally, given any 3D space point , obtain the corresponding geometric code by querying the corresponding implicit code voxel and texture encoding , specifically defined as: in Represents the position of a point in 3D space, Indicates the RGB value of the point, Represents the normal vector corresponding to the point; Establish an implicit neural field, for any point in space , use trilinear interpolation to get points for geometric and texture features The geometric features and texture features of the point The geometric features and texture features of the point are input into the fully connected network for decoding, and the point Geometric representation and texture representation; Training the human body mesh completion model includes the following steps: Sampling the space where the complete human body grid is located to obtain sampling points, and calculating the truncated signed distance function and color corresponding to the sampling points as labels; The color information is obtained according to the color of the nearest patch from the spatial point corresponding to the sampling point to the three-dimensional grid surface; The sampling points are input into the implicit neural field to predict the truncated signed distance function and color of the sampling points, and the optimization objective function between the predicted and true labels is calculated to train and optimize the model. Get the trained human mesh completion model; The trained human body mesh completion model is used to complete the human body mesh to be completed, including the following steps: Given a missing human body mesh to be completed; The sparse 3D point cloud position, color and surface normal vector data of the missing human body mesh surface to be completed are sampled and input into the trained model to establish an implicit neural field, and then the complete human body mesh is extracted using the Marching Cubes algorithm.

2. The 3D human body mesh completion method based on implicit neural field representation according to claim 1, characterized in that: Also includes: Define implicit representations before building implicit neural fields; Using signed distance field as implicit representation, we get implicit representation formula; Transforming implicit fields into explicit representations that can ultimately be presented; Get The value is truncated and the value; for point The signed distance, for point The truncated signed distance of According to the Value and It is worth to obtain the implicit field formula for 3D shape reconstruction.

3. The 3D human body mesh completion method based on implicit neural field representation according to claim 2, characterized in that: Multi-scale geometric features and texture features The formula is: in, For the Points2VoxNet network, is the number of points in the sparse 3D point cloud, is the first The 3D position of the point, is the RGB value of the point, is the normal vector corresponding to the point.

4. The 3D human body mesh completion method based on implicit neural field representation according to claim 3, characterized in that: Geometric feature encoding and texture feature encoding The formula is: in, is trilinear interpolation, is the 3D position of any point in space.

5. The 3D human body mesh completion method based on implicit neural field representation according to claim 4, characterized in that: Geometric representation With texture representation The formula is: in, is the geometric decoding network, For texture decoding network.

6. The 3D human body mesh completion method based on implicit neural field representation according to claim 5, characterized in that: The formula of the trained human mesh completion model is: in, To sample the space where the complete human body grid is located, is the number of sampling points, is the geometric regression loss weight, Recover loss weights for textures.

7. The 3D human body mesh completion method based on implicit neural field representation according to claim 6, characterized in that: The implicit field for 3D shape reconstruction is: in, is a point in 3D space, for point The signed distance of the point The signed distance to the nearest surface point, for point The truncated signed distance of is the cutoff threshold for the truncation signed distance, is the implicit field network parameter.

8. The 3D human body mesh completion method based on implicit neural field representation according to claim 1, characterized in that: The sampling of the space where the complete human body grid is located to obtain sampling points includes sampling near the surface of the complete human body grid and uniformly sampling in the entire standard space.

Images