Method for reconstructing geometry and material of human body based on sparse flash image

Abstract

The invention discloses a method for reconstructing human body geometry and material based on a sparse flash image, and the method comprises the steps: firstly constructing and pre-training a geometric material prediction network which comprises an image feature extraction network, a geometric prediction network and a material prediction network; then, a flash image of a target person is shot, light rays are emitted from random pixels of each view angle, intersection points of the light rays and a scene are obtained by using a spherical tracking algorithm and a pre-trained geometric prediction network, and materials of the intersection points are obtained by using a pre-trained material prediction network; and based on the geometric and material attributes of the intersection points, the colors of the intersection points are calculated by using physics-based differentiable rendering, and the geometric material prediction network is further optimized by taking the color close to the actual pixel color as a target. And finally, extracting geometry and materials of the human body model by using the optimized network. According to the method, high-quality human body digital assets capable of being re-illuminated can be obtained in a relatively short time only by using the multi-view flash images and the camera pose information shot by the consumer-level equipment under dark light.

Description

Technical Field

[0001] The invention relates to the field of human body three-dimensional modeling, and in particular to a method for reconstructing human body geometry and material based on sparse flash images. Background Art

[0002] Reconstructing the human body from images has been a long-standing problem in computer vision and graphics. It has a wide range of applications in the film industry, games, and virtual reality, enabling individuals to be displayed in virtual worlds. High-fidelity reconstructions usually require complex hardware, such as commercial body scanners. In addition, surface textures can be acquired through traditional photometric stereo equipment, such as light field acquisition equipment with programmable lights and densely synchronized cameras. However, these devices are usually too complex and expensive, which hinders their use in consumer applications. In recent years, the rapid development of deep learning has led to an increasing number of studies that have begun to adopt data-driven methods for human body modeling. This type of method learns strong priors from human body scan data, thereby reducing the need for complex shooting hardware to sparse or single cameras. These methods use neural networks to extract features, then project 3D points to each view, sample features in a pixel-aligned manner, reason about geometry and appearance, and finally use marching cubes for geometry extraction. Vertex colors can be obtained through a trained appearance prediction network to achieve human digitization. Among these neural network-based methods, some work has demonstrated new capabilities to reconstruct re-illuminated human bodies from a single image, and can simultaneously predict lighting, albedo, and geometry. Another body of work has demonstrated the ability to reconstruct the human body from sparse images. However, when the body pose, camera viewpoint, and lighting conditions deviate significantly from the training data distribution, it can be observed that the reconstruction quality of such neural network-based methods is significantly reduced. Another active research area on 3D reconstruction optimizes each scene and models each scene through a neural implicit field. For example, some work introduced an optimization-based scheme that takes flash images as input to improve the reconstruction quality. However, a common disadvantage of these optimization-based methods is that they are time-consuming, for example, about 10 hours for optimization of a single scene, which is mainly due to the need to collect dense multi-view images and re-optimize each scene. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and provide a method for reconstructing human geometry and material based on sparse flash images, so that high-quality, re-lightable human digital assets can be quickly created using consumer-grade equipment.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A method for reconstructing human body geometry and material based on sparse flash images, comprising the following steps:

[0006] Step 1: Obtain a public human scan dataset. For each human model, take the average coordinates of all vertices of the human model as the origin, normalize each human model to the unit cube, use all vertices and normals of the normalized human model as geometric data, and use the human model texture as material data; and use N camera parameters {π1,π2,…,π N} Render flash images of different view angles around each human model {I1,I2,…,I N}; The flash image refers to an image taken by placing a point light source at the camera position;

[0007] Step 2: Construct and pre-train a geometric material prediction network, wherein the geometric material prediction network includes an image feature extraction network, a geometric prediction network, and a material prediction network; wherein,

[0008] The image feature extraction network is a U-Net architecture, whose input is an image of H×W×3 and whose output is a feature block of H×W×128; wherein H represents the height of the input image and W represents the width of the input image;

[0009] The geometric prediction network includes a geometric transformer and a geometric multilayer perceptron, wherein the geometric transformer includes two self-attention modules, each of which includes a multi-head attention network and a position-encoded forward network, the multi-head attention network includes three input linear layers, a Softmax layer and an output linear layer, and the position-encoded forward network includes a linear layer and an activation layer; the input of the geometric transformer is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points, and the output is a fused 129-dimensional feature; the input of the geometric transformer is obtained in the following manner: given the spatial coordinates of any point in a unit cube and the camera pose corresponding to K perspective images, the camera space coordinates of the projection point are obtained by projecting the point to K perspective cameras, and the depth value Z of the camera space coordinates of the K projection points can be obtained at this time; then for the camera space coordinates of the K projection points, the feature blocks output by the image feature extraction network are sampled by bilinear interpolation to obtain K 128-dimensional feature vectors; the geometric multilayer perceptron includes five linear layers, the input is the fused 129-dimensional feature output by the geometric transformer, and the output is the directed distance corresponding to each point, with a dimension of 1;

[0010] The material prediction network includes a material transformer and a material multilayer perceptron. The structure, input dimension, and output dimension of the material transformer are the same as those of the geometric transformer. The input of the material multilayer perceptron is the fused 129-dimensional features output by the material transformer, and the output is the material coefficient corresponding to the position of each point, with a dimension of 7.

[0011] Step 3: Take an image of the target person, emit light from random pixels of each shooting angle, use the spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to get the intersection of the light and the scene, and get the material of the intersection according to the material prediction network pre-trained in step 2; based on the geometry and material properties of the intersection, use physically based differentiable rendering to calculate the color, with the goal of getting close to the actual pixel color, further train and optimize the geometry material prediction network, so that the geometry material prediction network outputs more accurate geometry and material;

[0012] Step 4: At uniform grid points in the unit cube, the directed distance is calculated based on the optimized geometry prediction network obtained in step 3, and then the triangular mesh is extracted using the marching cube algorithm; for the vertices of the triangular mesh, the material information of the vertices is predicted using the optimized material prediction network obtained in step 3; finally, the UV is automatically unfolded to obtain the material map.

[0013] Furthermore, the image feature extraction network includes a 4-layer convolutional encoder, a 4-layer convolutional decoder and a convolutional output layer, and jump connections are used between the convolutional encoders and convolutional decoders at the same level.

[0014] Furthermore, the output dimensions of the five linear layers of the geometric multilayer perceptron are 1024, 512, 256, 128, and 1, respectively, and the middle three linear layers and the input are skipped.

[0015] Furthermore, the output dimensions of the five linear layers of the material multilayer perceptron are 1024, 512, 256, 128, and 7, respectively, and the middle three linear layers and the input are skipped.

[0016] Furthermore, the pre-trained loss function of the step two includes geometric loss and material loss; the geometric loss is the loss between the actual directed distance information of a point in a given unit cube and the directed distance predicted by the geometric prediction network; the material loss is the loss between the material information of a point in a given unit cube and the material output by the material prediction network.

[0017] Furthermore, the expression of the pre-training loss function of step 2 is as follows:

[0018] L pretrain =L sdf +λ nml L nml+λ eik L eik +λ aln L aln +L app

[0019]

[0020] Among them, L sdf represents the directed distance loss function, L nml represents the normal loss function, L eik represents the regularized loss function, L aln represents the alignment loss function, L app represents the material loss function, λ l ,λ nml ,λ eik ,λ aln Respectively represent the weights of the corresponding loss functions; d i is the sampling point x i The predicted directed distance. i is the label of the sampling point, 0 and 1 represent inside and outside the surface respectively; τ is a learnable parameter, BCE is the binary cross entropy function, and φ is the sigmoid function; is the normal direction of the sampling point; x j For x i The projection point to the zero value surface, represents the gradient and δ is a hyperparameter.

[0021] Furthermore, the step three includes the following sub-steps:

[0022] (3.1) Shooting data: In low light conditions, use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives {I k}, solve the camera parameters {π k}, segment the person and background to obtain the segmentation map {m k};

[0023] (3.2) Ray casting and intersection: For each camera viewpoint, randomly sample the pixel p in the image captured in step (3.1), calculate the direction of the light, and emit the ray to the scene; use the spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface, and query the geometric prediction network at each step in the iteration to obtain the directed distance of the step. If the directed distance is less than the set threshold, it is considered that the intersection is found, otherwise the ray is allowed to move forward the directed distance, and iterate until the algorithm exits to obtain the intersection position of the ray and the human body, which is recorded as x0;

[0024] (3.3) Optimization through differentiable rendering: Reparameterize the intersection position x0 obtained in step (3.2) so that it can propagate the gradient to the geometric material prediction network:

[0025]

[0026] Among them, v represents the light vector, n represents the normal direction, and d0 represents the directed distance at x0;

[0027] The material coefficient corresponding to the x position is obtained by querying the material prediction network pre-trained in step 2;

[0028] Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color C of the parameterized intersection position x :

[0029] C x =L i (v, x)f r (v, v, x)(v·n)

[0030] Among them, L i (v, x) represents the incident light intensity, f r represents the bidirectional reflectance distribution function, C x Indicates the rendering color at x when viewed from the direction of sight v;

[0031] The image loss is obtained by calculating the difference between the color of image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding points to conform to the standard signed distance field to prevent the optimization from not conforming to the definition of the directed distance. The image loss and the geometric regularization loss are minimized by the gradient descent method to obtain the gradient of the neural network parameters, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field.

[0032] Furthermore, in step 3, the geometric material prediction network is further trained to optimize the loss function L ft The expression is:

[0033] L ft =L c +λ mask L mask +λ eik L eik +λ aln L aln +λ sp L sp +λ smt L smt

[0034]

[0035]

[0036] Among them, L c represents the color loss function, L mask represents the mask loss function, L sp represents the sparsity loss function, L smt represents the smooth loss function; λ c , mask , eik , aln , sp , smt They represent the weights of each loss function respectively.

[0037] Furthermore, step 4 includes the following sub-steps:

[0038] (4.1) Divide the space evenly into L 3 Voxel, get the spatial position of the target voxel vertex;

[0039] (4.2) Substitute L from step (4.1) 3 The vertex position of the voxel and the image captured in step 3 are input into the geometric prediction network optimized in step 3 to obtain the directed distance at the vertex position;

[0040] (4.3) Based on the voxel vertex position of step (4.1) and the directed distance at the vertex position obtained in step (4.2), the marching cube algorithm is used to calculate the triangular mesh; then the material information of the vertices of the triangular mesh is predicted using the optimized material prediction network; finally, the open source software Blender is used to automatically unfold the UV and export the material map.

[0041] A 3D human body reconstruction system based on sparse perspective flash images, the system is used to implement a method for reconstructing human body geometry and material based on sparse flash images, the system comprising:

[0042] A rendering module, used for rendering a data set including a flash RGB image using the acquired three-dimensional human body model;

[0043] A pre-training module, used to use the data set as training data and pre-train the geometric material prediction network using a gradient descent method;

[0044] The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed, and use the differentiable rendering framework and gradient descent method to optimize the pre-trained geometric material prediction network again;

[0045] The geometry and material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry and material prediction network into a three-dimensional human body model with texture material.

[0046] The beneficial effects of the present invention are as follows:

[0047] The method of the present invention only utilizes multi-view flash images and camera pose information taken under dim light by consumer-grade devices, obtains an initial solution based on a pre-trained network, and fine-tunes and optimizes the network for a single person. High-quality reconstruction results can be obtained in a relatively short time, which effectively makes up for the poor robustness and accuracy of pre-training methods, while avoiding the disadvantages of pure optimization methods that require re-optimization of each scene and are too slow. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the method of reconstructing human body geometry and texture based on sparse flash images of the present invention.

[0049] Figure 2 Schematic diagram of the image feature extraction network.

[0050] Figure 3 Schematic diagram of geometry transformer and material transformer.

[0051] Figure 4 Schematic diagram of a geometric multi-layer perceptron.

[0052] Figure 5 Schematic diagram of a material multilayer perceptron.

[0053] Figure 6 It is a rendering of an embodiment of the present invention, wherein the dashed box on the left is the input flash image, the dashed box in the middle is the normal of the output triangular mesh, the lighting effect of the triangular mesh white model, and the base color (Base Color) of the triangular mesh from top to bottom; the dashed box on the right is the re-illumination performance of the output digitized human body model under different ambient lights.

[0054] Figure 7 1 is a rendering of a digital human body asset reconstructed according to an embodiment of the present invention under different lighting conditions; from left to right: the lighting effect of point light source position 1, the lighting effect of point light source position 2, the lighting effect of ambient light 1, and the lighting effect of ambient light 2.

[0055] Figure 81 is a rendering of a digital human body asset reconstructed by an embodiment of the present invention under test viewing angles and lighting conditions; from left to right: a photographed image at a test viewing angle, a rendered color image obtained by an embodiment of the present invention, a rendered normal image, diffuse albedo, roughness, specular albedo, and specular shading. DETAILED DESCRIPTION

[0056] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] Technical term explanation:

[0058] 1. Marching Cubes

[0059] 2. Transformer

[0060] 3. Multi-Layer Perceptron (MLP)

[0061] 4. Signed Distance Field (SDF)

[0062] 5. Field of view (FOV)

[0063] The core idea of ​​the method for reconstructing human body geometry and texture based on sparse flash images of the present invention is: using sparse multi-perspective flash images, obtaining an initial solution based on a pre-trained network, fine-tuning and optimizing the network for a single person, and obtaining high-quality reconstruction results in a relatively short time.

[0064] like Figure 1 As shown, the method for reconstructing human body geometry and material based on sparse flash images according to an embodiment of the present invention comprises the following steps:

[0065] Step 1: Create training data: Based on the public human body scan data set, preprocess and normalize it into a unit cube as geometric data; use the renderer to render the geometric data to obtain multi-view image data. The image data and geometric data pairs are used for subsequent supervised learning.

[0066] In this embodiment, a public human body scan data set is obtained. For each human body model M, the average coordinates of all vertices of each human body model are taken as the origin, and each human body model M is normalized to a unit cube. All vertices and normals of the normalized human body model are used as geometric data, and the texture of the human body model is used as material data. N} Render flash images {I1, I2, ..., I N The flash image refers to an image taken by placing a point light source at the camera position. In this embodiment, N=60, the camera is located at a distance of about 2 meters from the model, and the FOV is about 65 degrees.

[0067] Step 2: Build and pre-train the geometric material prediction network, which includes three sub-modules: image feature extraction network, geometric prediction network, and material prediction network.

[0068] Image feature extraction network The input is an H×W×3 image, and the output is a H×W×128 feature block, where H represents the height of the input image and W represents the width of the input image. Image feature extraction network The architecture is U-Net, which consists of a 4-layer convolutional encoder, a 4-layer convolutional decoder, and a convolutional output layer. The encoder and decoder at the same level use jump connections. The detailed structure is shown in the attached figure. Figure 2 shown.

[0069] The geometry prediction network consists of a geometry transformer T geo and Geometric Multilayer Perceptron Composition, where T geo The network structure is as follows Figure 3 As shown, The network structure is as follows Figure 4 Given the spatial coordinates of any point in the unit cube and the camera poses corresponding to the K viewpoint images, the camera space coordinates of the projection point are obtained by projecting the point to the K viewpoint cameras (the depth value Z of the camera space coordinates of the K projection points can be obtained at this time), and then the image feature extraction network is sampled through bilinear interpolation. The output feature blocks are K 128-dimensional feature vectors. The K 128-dimensional feature vectors and the depth values ​​Z of the camera space coordinates of the K projection points are used as the input of the geometric prediction network, that is, the geometric transformer T geo Input of the geometric transformer T geo The output of is the fused 129-dimensional features. Figure 3 As shown, the geometric transformer T geoIt contains two self-attention modules, each of which consists of a multi-head attention network (three input linear layers, Softmax layer, output linear layer) and a position encoding forward network (linear layer, activation layer). Figure 4 As shown, the geometric multi-layer perceptron The input is the 129-dimensional feature output by the geometric transformer, and the output is the directed distance corresponding to the point position, with a dimension of 1. The geometric multilayer perceptron consists of five linear layers, with output dimensions of 1024, 512, 256, 128, and 1. The middle three-layer network and the input are skipped.

[0070] The material prediction network consists of a material transformer T app and material multilayer perceptron The input of the material prediction network is the same as that of the geometry prediction network, which is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points. app The structure, input dimension, output dimension and geometric transformer T geo The input is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points, and the output is the fused 129-dimensional feature output by the material transformer. app It contains two self-attention modules, each of which consists of a multi-head attention network (three linear layers, Softmax layer, output linear layer) and a position encoding forward network (linear layer, activation layer). Figure 5 As shown, the material multi-layer perceptron The input is the 129-dimensional feature output by the material transformer, and the output is the material coefficient corresponding to the point position, with a dimension of 7. The material multilayer perceptron consists of five linear layers, and the output dimensions are 1024, 512, 256, 128, and 7 respectively. The middle three-layer network and the input are skipped.

[0071] The pre-training of the geometric material prediction network specifically includes:

[0072] (2.1) Randomly sample K perspectives from the image rendered in step 1.1 to obtain images {I1, I2, ..., I k}, and camera parameters {π1, π2, ..., π k}, using image feature extraction network Extract multi-view feature blocks {V1, V2, ..., V k}.

[0073] (2.2) Randomly select points inside the unit cube and randomly select points near the model surface to obtain the point set {x}, each point passes through the camera parameter {π k} Projection to obtain the projection coordinates {X k}; For each projection coordinate {X k}, multi-view feature blocks {V k}, and obtain K 128-dimensional features {v k}.

[0074] (2.3) The K 128-dimensional features {v k} and the depth value z after projection of the point to form a 129-dimensional feature, which is input to T geo and T app , after the attention mechanism fuses the multi-view features, we get the fused 1×129-dimensional geometric features and the fused 1×129-dimensional material features. The fused 129-dimensional geometric feature input Output a scalar d, which represents the signed distance from the point to the geometric surface. The fused 129-dimensional material feature input Output material coefficient, representing the material corresponding to the point position.

[0075] (2.4) For each point in the point set {x}, the geometric loss is calculated using the actual directed distance information of the point and the directed distance predicted by the geometric prediction network; for each point sampled near the surface in the point set {x}, the material information of the point and the output of the material prediction network are used to calculate the material loss; the above losses are used to supervise the training of the image feature extraction network, the geometric prediction network and the material prediction network using the gradient descent method. As one implementation method, the loss function L of the pre-trained geometric material prediction network is pretrain The expression is as follows:

[0076] L pretrain =L sdf +λ nml L nml +λ eik L eik +λ aln L aln +L app

[0077]

[0078] Among them, L sdf , L nml , L eik , L aln For supervised geometry, L app Used to monitor materials. sdf Represents the signed distance loss function, the purpose of which is to constrain the predicted signed distance of points near the surface to be close to 0, and to constrain the predicted signed distance of points in the space to be negative inside and positive outside. L nml represents the normal loss function, which is used to constrain the predicted normal of points near the surface to be close to the normal of the real geometry; L eikrepresents the regularized loss function, which is used to constrain the gradient of the directed distance to 1; L aln represents the alignment loss function, which is used to constrain the normal of a spatial point to be parallel to the normal of the projection point of the spatial point to the zero-value surface; L app Represents the material loss function, which is used to constrain the predicted material to be close to the real material. l ,λ nml ,λ eik ,λ aln Represent the weights of each loss function. i is the sampling point x i The predicted directed distance. i is the label of the sampling point (0 / 1 represents inside / outside the surface), τ is a learnable parameter, BCE is the binary cross entropy function, and φ is the sigmoid function. is the normal direction of the sampling point. j For x i The projection point to the zero value surface, represents the gradient. δ is a hyperparameter, which is set to 10 in this embodiment. In this embodiment, λ l =0.2,λ nml =0.1,λ eik =0.1,λ aln =0.01.

[0079] Step 3: Joint optimization based on differentiable rendering: Shoot data, emit light from random pixels of each shooting perspective, use the spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to get the intersection of the light and the scene, and the material of the intersection is obtained according to the material prediction network pre-trained in step 2. Based on the geometry and material properties of the intersection, use physically based differentiable rendering to calculate the color, with the goal of approaching the actual pixel color, and further train and optimize the geometry material prediction network, so that the geometry material prediction network outputs more accurate geometry and material. Step 3 specifically includes the following sub-steps:

[0080] (3.1) Shooting data: In low light conditions, use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives {I k}, and solve the camera parameters {π k}, segment the person and background to obtain the segmentation map {m k}.

[0081] (3.2) Ray casting and intersection: For each camera view, randomly sample the pixel p in the image captured in step (3.1), calculate the light direction, and emit the ray to the scene. Use the spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface, and query the geometric prediction network at each step in the iteration. Get the directed distance of this step. If the directed distance is less than the set threshold, it is considered that the intersection is found. Otherwise, let the ray move forward the directed distance. Iterate until the algorithm exits and get the intersection position of the ray and the human body, recorded as x0.

[0082] (3.3) Optimization through differentiable rendering: The intersection position x0 obtained in step (3.2) is reparameterized so that it can propagate the gradient to the material prediction network.

[0083]

[0084] Among them, v represents the light vector, n represents the normal direction, and d0 represents the signed distance at x0.

[0085] The material prediction network pre-trained in step 2 Query and obtain the material coefficient corresponding to the x position.

[0086] Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color C of the parameterized intersection position x :

[0087] C x =L i (v, x)f r (v, v, x)(v·n)

[0088] Among them, L i (v, x) represents the incident light intensity, f r represents the bidirectional reflectance distribution function, C x Represents the rendering color at x when viewed from the direction of view v.

[0089] The image loss is obtained by calculating the difference between the color of the image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding point to conform to the standard signed distance field to prevent the optimization from not conforming to the definition of the signed distance. By minimizing the image loss and the geometric regularization loss by the gradient descent method, the gradient of the neural network parameters can be obtained, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field. As one implementation method, the expression of the loss function here is:

[0090] L ft =L c+λ mask L mask +λ eik L eik +λ aln L aln +λ sp L sp +λ smt L smt

[0091]

[0092] Among them, L c represents the color loss function, which is used to constrain the calculated color to be close to the actual pixel color; L mask represents the mask loss function, which is used to constrain the directed distance of points outside the mask to be greater than 0; L sp represents the sparsity loss function, which is used to constrain the predicted color to be as sparse as possible; L smt represents the smoothness loss function, which is used to constrain the predicted color to be as smooth as possible; L eik and L aln The meaning and pre-training loss function L pretrain L eik , L aln Same. c ,λ mask ,λ eik ,λ aln ,λ sp ,λ smt Represent the weights of each loss function respectively.

[0093] Step 4: Export geometry and materials: At uniform grid points in the unit cube, the directed distance is calculated based on the optimized geometry prediction network obtained in step 3, and then the triangular mesh is extracted using the marching cube algorithm; for the vertices of the triangular mesh, the material information of the vertices is predicted using the optimized material prediction network obtained in step 3. Finally, the open source software Blender (http: / / www.blender.org) is used to automatically unfold UVs and export material maps. Step 4 includes the following sub-steps:

[0094] (4.1) Uniform sampling of voxel vertices: Divide the space evenly into L 3 voxel, and get the spatial position of the target voxel vertex.

[0095] (4.2) Query the optimized geometric prediction network to obtain the directed distance of the voxel vertex: Substitute L in step (4.1) 3 The vertex position of the voxel and the image taken in step (3.1) are used as input, and the geometric prediction network optimized in step 3 is run to obtain the directed distance at the vertex position.

[0096] (4.3) Extracting triangular mesh and material: Based on the voxel vertex position in step (4.1) and the directed distance at the vertex position obtained in step (4.2), the triangular mesh is calculated using the Marching Cubes algorithm. Then, the material information of the vertices of the triangular mesh is predicted using the optimized material prediction network. Finally, the open source software Blender (http: / / www.blender.org) is used to automatically unfold UVs and export material maps.

[0097] A three-dimensional human body reconstruction system based on sparse perspective flash images according to another embodiment of the present invention comprises:

[0098] A rendering module, used for rendering a data set including a flash RGB image using the acquired three-dimensional human body model;

[0099] A pre-training module is used to use the data set as training data and pre-train the geometric material prediction network using the gradient descent method; the geometric material prediction network includes an image feature extraction network, a geometric prediction network, and a material prediction network;

[0100] The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed, and optimize the geometric material prediction network again using the differentiable rendering framework and gradient descent method;

[0101] The geometry material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry material prediction network into a three-dimensional human body model with texture material.

[0102] The specific implementation process of each module corresponds to each sub-step in the method.

[0103] Training example: On a desktop computer equipped with an Intel I5-4430 CPU, an Nvidia GeForce GTX 1080ti graphics processor (11GB) and 16GB of memory, the method of the embodiment of the present invention is implemented using the PyTorch programming language, and the model is trained on a desktop computer equipped with an Nvidia GeForce GTX 3090 graphics processor (24GB) and 128GB of memory. The inventor uses the database Thuman2.0 (https: / / github.com / ytrock / THuman2.0-Dataset / ) and the database Custom Human (https: / / custom-humans.github.io / ) to train the model.

[0104] Example time consumption: Pre-training time is about 24 hours, and only one training is required on the database data. It takes about 20 minutes to shoot sparse perspective human images and camera pose estimation (colmap), and about 10 minutes for joint optimization.

[0105] Effect example: In this embodiment of the present invention, multiple flash photos are used to reconstruct the geometric texture of the human body. Different characters are rendered using Blender's renderer under different lighting conditions. The re-lighting effect is as follows: Figure 6 (dashed box on the right), Figure 7 As shown in the figure; the geometry, appearance, material (albedo, roughness, etc.) effects of the reconstructed character, such as Figure 8 The present invention can quickly reconstruct triangular meshes and materials from photos of people taken from sparse perspectives, and the reconstructed geometry and materials can realistically restore the real geometry and appearance of the person ( Figures 6 to 8 ), the reconstruction results on human bodies of various clothing and genders prove the robustness of the present invention ( Figures 6 to 8 ).

[0106] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A method for reconstructing human body geometry and texture based on sparse flash images, characterized in that: The following steps are involved: Step 1: Obtain a public human scan dataset. For each human model, take the average coordinates of all vertices of the human model as the origin, normalize each human model to the unit cube, use all vertices and normals of the normalized human model as geometric data, and use the human model texture as material data; and use N camera parameters {π1,π2,…,π N } Render flash images of different view angles around each human model {I1,I2,…,I N }; The flash image refers to an image taken by placing a point light source at the camera position; Step 2: Construct and pre-train a geometric material prediction network, wherein the geometric material prediction network includes an image feature extraction network, a geometric prediction network, and a material prediction network; wherein, The image feature extraction network is a U-Net architecture, whose input is an image of H×W×3 and whose output is a feature block of H×W×128; wherein H represents the height of the input image and W represents the width of the input image; The geometric prediction network includes a geometric transformer and a geometric multilayer perceptron, wherein the geometric transformer includes two self-attention modules, each of which includes a multi-head attention network and a position-encoded forward network, the multi-head attention network includes three input linear layers, a Softmax layer and an output linear layer, and the position-encoded forward network includes a linear layer and an activation layer; the input of the geometric transformer is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points, and the output is a fused 129-dimensional feature; the input of the geometric transformer is obtained in the following manner: given the spatial coordinates of any point in a unit cube and the camera pose corresponding to K perspective images, the camera space coordinates of the projection point are obtained by projecting the point to K perspective cameras, and the depth value Z of the camera space coordinates of the K projection points can be obtained at this time; then for the camera space coordinates of the K projection points, the feature blocks output by the image feature extraction network are sampled by bilinear interpolation to obtain K 128-dimensional feature vectors; the geometric multilayer perceptron includes five linear layers, the input is the fused 129-dimensional feature output by the geometric transformer, and the output is the directed distance corresponding to each point, with a dimension of 1; The material prediction network includes a material transformer and a material multilayer perceptron. The structure, input dimension, and output dimension of the material transformer are the same as those of the geometric transformer. The input of the material multilayer perceptron is the fused 129-dimensional features output by the material transformer, and the output is the material coefficient corresponding to the position of each point, with a dimension of 7. Step 3: Take an image of the target person, emit light from random pixels of each shooting angle, use the spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to get the intersection of the light and the scene, and get the material of the intersection according to the material prediction network pre-trained in step 2; based on the geometry and material properties of the intersection, use physically based differentiable rendering to calculate the color, with the goal of getting close to the actual pixel color, further train and optimize the geometry material prediction network, so that the geometry material prediction network outputs more accurate geometry and material; Step 4: At uniform grid points in the unit cube, the directed distance is calculated based on the optimized geometry prediction network obtained in step 3, and then the triangular mesh is extracted using the marching cube algorithm; for the vertices of the triangular mesh, the material information of the vertices is predicted using the optimized material prediction network obtained in step 3; finally, the UV is automatically unfolded to obtain the material map.

2. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The image feature extraction network includes a 4-layer convolutional encoder, a 4-layer convolutional decoder and a convolutional output layer, and jump connections are used between the convolutional encoders and convolutional decoders at the same level.

3. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The output dimensions of the five linear layers of the geometric multilayer perceptron are 1024, 512, 256, 128, and 1, respectively, and the middle three linear layers and the input are skipped.

4. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The output dimensions of the five linear layers of the material multilayer perceptron are 1024, 512, 256, 128, and 7, respectively, and the middle three linear layers and the input are skipped.

5. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The pre-trained loss function of step 2 includes geometric loss and material loss; the geometric loss is the loss between the actual directed distance information of a point in a given unit cube and the directed distance predicted by the geometric prediction network; the material loss is the loss between the material information of a point in a given unit cube and the material output by the material prediction network.

6. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 5, characterized in that: The expression of the pre-training loss function of step 2 is as follows: L pretrain =L sdf +λ nml L nml +λ eik L eik +λ aln L aln +L app Among them, L sdf represents the directed distance loss function, L nml represents the normal loss function, L eik represents the regularized loss function, L aln represents the alignment loss function, L app represents the material loss function, λ l ,λ nml ,λ eik ,λ aln Respectively represent the weights of the corresponding loss functions; d i is the sampling point x i The predicted directed distance. i is the label of the sampling point, 0 and 1 represent inside and outside the surface respectively; τ is a learnable parameter, BCE is the binary cross entropy function, and φ is the sigmoid function; is the normal direction of the sampling point; x j For x i The projection point to the zero value surface, represents the gradient and δ is a hyperparameter.

7. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The step three includes the following sub-steps: (3.1) Shooting data: In low light conditions, use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives {I k }, solve the camera parameters {π k }, segment the person and background to obtain the segmentation map {m k }; (3.2) Ray casting and intersection: For each camera viewpoint, randomly sample the pixel p in the image captured in step (3.1), calculate the direction of the light, and emit the ray to the scene; use the spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface, and query the geometric prediction network at each step in the iteration to obtain the directed distance of the step. If the directed distance is less than the set threshold, it is considered that the intersection is found, otherwise the ray is allowed to move forward the directed distance, and iterate until the algorithm exits to obtain the intersection position of the ray and the human body, which is recorded as x0; (3.3) Optimization through differentiable rendering: Reparameterize the intersection position x0 obtained in step (3.2) so that it can propagate the gradient to the geometric material prediction network: Among them, v represents the light vector, n represents the normal direction, and d0 represents the directed distance at x0; The material coefficient corresponding to the x position is obtained by querying the material prediction network pre-trained in step 2; Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color C of the parameterized intersection position x : C x =L i (v,x)f r (v,v,x)(v·n) Among them, L i (v,x) represents the incident light intensity, f r represents the bidirectional reflectance distribution function, C x Indicates the rendering color at x when viewed from the direction of sight v; The image loss is obtained by calculating the difference between the color of image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding points to conform to the standard signed distance field to prevent the optimization from not conforming to the definition of the directed distance. The image loss and the geometric regularization loss are minimized by the gradient descent method to obtain the gradient of the neural network parameters, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field.

8. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 7, characterized in that: In step 3, the geometric material prediction network is further trained to optimize the loss function L ft The expression is: L ft =L c +λ mask L mask +λ eik L eik +λ aln L aln +λ sp L sp +λ smt L smt Among them, L c represents the color loss function, L mask represents the mask loss function, L sp represents the sparsity loss function, L smt represents the smooth loss function; λ c , mask , eik , aln , sp , smt Represent the weights of each loss function respectively.

9. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: Step 4 includes the following sub-steps: (4.1) Divide the space evenly into L 3 Voxel, get the spatial position of the target voxel vertex; (4.2) Substitute L from step (4.1) 3 The vertex position of the voxel and the image captured in step 3 are input into the geometric prediction network optimized in step 3 to obtain the directed distance at the vertex position; (4.3) Based on the voxel vertex position of step (4.1) and the directed distance at the vertex position obtained in step (4.2), the marching cube algorithm is used to calculate the triangular mesh; then the material information of the vertices of the triangular mesh is predicted using the optimized material prediction network; finally, the open source software Blender is used to automatically unfold the UV and export the material map.

10. A 3D human body reconstruction system based on sparse perspective flash images, characterized in that: The system is used to implement the method for reconstructing human body geometry and material based on sparse flash images as described in any one of claims 1 to 9, and the system comprises: A rendering module, used for rendering a data set including a flash RGB image using the acquired three-dimensional human body model; A pre-training module, used to use the data set as training data and pre-train the geometric material prediction network using a gradient descent method; The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed, and use the differentiable rendering framework and gradient descent method to optimize the pre-trained geometric material prediction network again; The geometry material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry material prediction network into a three-dimensional human body model with texture material.

Images