A method for reconstructing human geometry and material from sparse flash images

Abstract

The application discloses a method for reconstructing human geometry and material based on sparse flash images, which comprises the following steps: firstly, constructing and pre-training a geometry and material prediction network comprising an image feature extraction network, a geometry prediction network and a material prediction network; then, taking flash images of a target person, emitting light from random pixels at each view angle, obtaining the intersection of the light and the scene by using a spherical tracking algorithm and the pre-trained geometry prediction network, and obtaining the material of the intersection by using the pre-trained material prediction network; based on the geometry and material properties of the intersection, calculating the color of the intersection by using a physically-based differentiable rendering, further optimizing the geometry and material prediction network by taking the actual pixel color as the target; and finally, extracting the geometry and material of the human model by using the optimized network. The application can obtain high-quality and relightable human digital assets in a short time by only using multi-view flash images and camera pose information taken by a consumer-level device in dark light.

Description

Technical Field

[0001] The present invention relates to the field of human body three-dimensional modeling, and in particular to a method for reconstructing human body geometry and material based on sparse flash images. Background Art

[0002] Reconstructing the human body from images has been a long-standing problem in computer vision and graphics. It has widespread applications in the film industry, gaming, and virtual reality, enabling individuals to be displayed in virtual worlds. High-fidelity reconstruction typically requires complex hardware, such as commercially available body scanners. Alternatively, surface textures can be acquired using traditional photometric stereo equipment, such as light field acquisition devices with programmable lights and densely synchronized cameras. However, these devices are often too complex and expensive, hindering their use in consumer applications. In recent years, the rapid development of deep learning has led to a growing body of research employing data-driven approaches to modeling the human body. These methods learn strong priors from body scan data, reducing the need for complex capture hardware to sparse or single cameras. These methods utilize neural networks to extract features, then project 3D points onto each view, sampling features in a pixel-aligned manner, inferring geometry and appearance. Finally, geometry extraction is performed using marching cubes, and vertex colors can be derived using a trained appearance prediction network, thus digitizing the human body. Several of these neural network-based methods have demonstrated the ability to reconstruct relightable human bodies from a single image, simultaneously predicting lighting, albedo, and geometry. Another body of work has demonstrated the ability to reconstruct the human body from sparse images. However, when the body pose, camera viewpoint, and lighting conditions deviate significantly from the training data distribution, it can be observed that the reconstruction quality of such neural network-based methods is significantly reduced. Another active research area on 3D reconstruction optimizes each scene and models each scene through a neural implicit field. For example, some work has introduced an optimization-based scheme that uses flash images as input to improve the reconstruction quality. However, a common disadvantage of these optimization-based methods is that they are time-consuming. For example, the optimization for a single scene takes about 10 hours, which is mainly due to the need to collect dense multi-view images and re-optimize each scene. Summary of the Invention

[0003] The purpose of the present invention is to address the deficiencies of the existing technology and provide a method for reconstructing human body geometry and texture based on sparse flash images, so that high-quality, re-lightable human body digital assets can be quickly created using consumer-grade equipment.

[0004] The object of the present invention is achieved through the following technical solutions:

[0005] A method for reconstructing human body geometry and texture based on sparse flash images, comprising the following steps:

[0006] Step 1: Obtain a public human body scan dataset. For each human body model, normalize it to the unit cube with the average coordinates of all vertices of the human body model as the origin. Use all vertices and normals of the normalized human body model as geometric data and the human body model texture as material data. And use N camera parameters {π1,π2,…,π N} Render flash images {I1, I2, ..., I N The flash image refers to an image captured by placing a point light source at the camera position;

[0007] Step 2: Construct and pre-train a geometry material prediction network, which includes an image feature extraction network, a geometry prediction network, and a material prediction network; wherein,

[0008] The image feature extraction network is a U-Net architecture, whose input is an H×W×3 image and the output is a H×W×128 feature block; where H represents the height of the input image and W represents the width of the input image;

[0009] The geometric prediction network includes a geometric transformer and a geometric multi-layer perceptron, wherein the geometric transformer includes two self-attention modules, each of which includes a multi-head attention network and a position-encoded forward network, the multi-head attention network includes three input linear layers, a Softmax layer, and an output linear layer, and the position-encoded forward network includes a linear layer and an activation layer; the input of the geometric transformer is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points, and the output is a fused 129-dimensional feature; the input of the geometric transformer is obtained as follows: given the spatial coordinates of any point in a unit cube and the camera pose corresponding to K perspective images, the point is projected onto K perspective cameras to obtain the camera space coordinates of the projection point, and the depth value Z of the camera space coordinates of the K projection points can be obtained at this time; then, for the camera space coordinates of the K projection points, the feature blocks output by the image feature extraction network are sampled by bilinear interpolation to obtain K 128-dimensional feature vectors; the geometric multi-layer perceptron includes five linear layers, the input is the fused 129-dimensional feature output by the geometric transformer, and the output is the directed distance corresponding to each point, with a dimension of 1;

[0010] The material prediction network includes a material transformer and a material multilayer perceptron. The structure, input dimension, and output dimension of the material transformer are the same as those of the geometric transformer. The input of the material multilayer perceptron is the fused 129-dimensional features output by the material transformer, and the output is the material coefficient corresponding to the position of each point, with a dimension of 7.

[0011] Step 3: Capture an image of the target person and emit light from random pixels at each viewing angle. Use a spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to determine the intersection of the light and the scene. The material of the intersection is determined using the material prediction network pre-trained in step 2. Based on the geometry and material properties of the intersection, physically based differentiable rendering is used to calculate the color. With the goal of approximating the actual pixel color, the geometry and material prediction network is further trained and optimized to output more accurate geometry and material.

[0012] Step 4: At uniform grid points in the unit cube, the directed distance is calculated based on the optimized geometry prediction network obtained in step 3, and then the triangular mesh is extracted using the marching cube algorithm. For the vertices of the triangular mesh, the material information of the vertices is predicted using the optimized material prediction network obtained in step 3. Finally, the UV is automatically unfolded to obtain the material map.

[0013] Furthermore, the image feature extraction network includes a 4-layer convolutional encoder, a 4-layer convolutional decoder and a convolutional output layer, and jump connections are used between the convolutional encoders and convolutional decoders at the same level.

[0014] Furthermore, the output dimensions of the five linear layers of the geometric multilayer perceptron are 1024, 512, 256, 128, and 1, respectively, and the middle three linear layers and the input are skipped.

[0015] Furthermore, the output dimensions of the five linear layers of the material multilayer perceptron are 1024, 512, 256, 128, and 7, respectively, and the middle three linear layers and the input are skipped.

[0016] Furthermore, the pre-trained loss function of step 2 includes geometric loss and material loss; the geometric loss is the loss between the actual directed distance information of a point in a given unit cube and the directed distance predicted by the geometric prediction network; the material loss is the loss between the material information of a point in a given unit cube and the material output by the material prediction network.

[0017] Furthermore, the expression of the loss function of the pre-training in step 2 is as follows:

[0018] L pretrain =L sdf +λ nml L nml+λ eik L eik +λ aln L aln +L app

[0019]

[0020] Among them, L sdf Represents the directed distance loss function, L nml represents the normal loss function, L eik represents the regularized loss function, L aln represents the alignment loss function, L app represents the material loss function, λ l ,λ nml ,λ eik ,λ aln Respectively represent the weights of the corresponding loss functions; d i is the sampling point x i The predicted directed distance. i is the label of the sampling point, 0 and 1 represent inside and outside the surface respectively; τ is a learnable parameter, BCE is the binary cross entropy function, and φ is the sigmoid function; is the normal direction of the sampling point; x j is x i The projection point to the zero value surface, represents the gradient and δ is a hyperparameter.

[0021] Furthermore, the step three includes the following sub-steps:

[0022] (3.1) Shooting data: Under low-light conditions, use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives {I k}, solve the camera parameters {π k}, segment the person and background to obtain the segmentation map {m k};

[0023] (3.2) Ray casting and intersection: For each camera viewpoint, randomly sample pixel p in the image captured in step (3.1), calculate the light direction, and emit the ray to the scene; use a spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface. At each step in the iteration, query the geometric prediction network to obtain the directed distance of that step. If the directed distance is less than the set threshold, the intersection is considered to be found. Otherwise, the ray is allowed to advance the directed distance. Iterate until the algorithm exits and obtain the intersection position of the ray and the human body, which is recorded as x0;

[0024] (3.3) Optimize by differentiable rendering: Reparameterize the intersection position x0 obtained in step (3.2) so that it can propagate the gradient to the geometric material prediction network:

[0025]

[0026] Among them, v represents the light vector, n represents the normal direction, and d0 represents the directed distance at x0;

[0027] The material coefficient corresponding to the x position is obtained by querying the material prediction network pre-trained in step 2;

[0028] Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color C of the parameterized intersection position x :

[0029] C x =L i (v, x)f r (v, v, x)(v·n)

[0030] Among them, L i (v, x) represents the incident light intensity, f r represents the bidirectional reflectance distribution function, C x Indicates the rendering color at x when viewed from the direction of sight v;

[0031] The image loss is obtained by calculating the difference between the color of the image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding points to conform to the standard signed distance field, preventing the optimization from not conforming to the definition of the signed distance. The image loss and geometric regularization loss are minimized by the gradient descent method to obtain the gradient of the neural network parameters, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field.

[0032] Furthermore, in step 3, the geometric material prediction network is further trained to optimize the loss function L ft The expression is:

[0033] L ft =L c +λ mask L mask +λ eik L eik +λ aln L aln +λ sp L sp +λ smt L smt

[0034]

[0035]

[0036] Among them, L c Represents the color loss function, L mask represents the mask loss function, L sp represents the sparsity loss function, L smt represents the smooth loss function; λ c ,λ mask ,λ eik ,λ aln ,λ sp ,λ smt Represent the weights of each loss function respectively.

[0037] Furthermore, step 4 includes the following sub-steps:

[0038] (4.1) Divide the space evenly into L 3 Voxel, get the spatial position of the target voxel vertex;

[0039] (4.2) Replace L in step (4.1) 3 The vertex position of the voxel and the image captured in step 3 are input into the geometric prediction network optimized in step 3 to obtain the directed distance at the vertex position;

[0040] (4.3) Based on the voxel vertex positions in step (4.1) and the directed distances at the vertex positions obtained in step (4.2), a triangular mesh is calculated using the marching cube algorithm; then, the optimized material prediction network is used to predict the material information of the vertices of the triangular mesh; finally, the open source software Blender is used to automatically unfold UVs and export the material map.

[0041] A 3D human body reconstruction system based on sparse perspective flash images, the system is used to implement a method for reconstructing human body geometry and material based on sparse flash images, the system comprising:

[0042] a rendering module, configured to render a data set including flash RGB images using the acquired three-dimensional human body model;

[0043] A pre-training module, configured to use the dataset as training data and pre-train the geometric material prediction network using a gradient descent method;

[0044] The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed and use the differentiable rendering framework and gradient descent method to further optimize the pre-trained geometric material prediction network;

[0045] The geometry and material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry and material prediction network into a three-dimensional human body model with textured material.

[0046] The beneficial effects of the present invention are as follows:

[0047] The method of the present invention only uses multi-view flash images and camera pose information taken under low light by consumer-grade devices, obtains an initial solution based on a pre-trained network, and fine-tunes and optimizes the network for a single person. High-quality reconstruction results can be obtained in a relatively short time, effectively compensating for the poor robustness and poor accuracy of pre-training methods, while avoiding the disadvantage of pure optimization methods that require re-optimization for each scene and are too slow. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the method for reconstructing human body geometry and texture based on sparse flash images of the present invention.

[0049] Figure 2 Schematic diagram of the image feature extraction network.

[0050] Figure 3 Schematic diagram of the geometry transformer and material transformer.

[0051] Figure 4 Schematic diagram of a geometric multilayer perceptron.

[0052] Figure 5 Schematic diagram of the material multi-layer perceptron.

[0053] Figure 6 This is a rendering of an embodiment of the present invention, where the dashed box on the left is the input flash image, the dashed box in the middle is the normal of the output triangular mesh, the lighting effect of the triangular mesh white model, and the base color of the triangular mesh from top to bottom; the dashed box on the right is the re-illumination performance of the output digitized human body model under different ambient light.

[0054] Figure 7 These are renderings of the digital human body asset reconstructed under different lighting conditions according to an embodiment of the present invention; from left to right, they are: the lighting effect of point light source position 1, the lighting effect of point light source position 2, the lighting effect of ambient light 1, and the lighting effect of ambient light 2.

[0055] Figure 8The following are the renderings of the digital human asset reconstructed under the test viewing angle and lighting conditions according to an embodiment of the present invention. From left to right, they are: a captured image at the test viewing angle, a rendered color image obtained according to an embodiment of the present invention, a rendered normal image, diffuse albedo, roughness, specular albedo, and specular shading. DETAILED DESCRIPTION

[0056] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Technical term explanation:

[0058] 1. Marching Cubes

[0059] 2. Transformer

[0060] 3. Multi-Layer Perceptron (MLP)

[0061] 4. Signed Distance Field (SDF)

[0062] 5. Field of view (FOV)

[0063] The core idea of ​​the method of reconstructing human body geometry and texture based on sparse flash images of the present invention is: using sparse multi-perspective flash images, obtaining an initial solution based on a pre-trained network, fine-tuning and optimizing the network for a single person, and obtaining high-quality reconstruction results in a relatively short time.

[0064] like Figure 1 As shown, the method for reconstructing human body geometry and texture based on sparse flash images according to an embodiment of the present invention includes the following steps:

[0065] Step 1: Create training data: Based on a publicly available human body scan dataset, preprocess and normalize it to a unit cube to serve as geometric data. Use a renderer to render this geometric data to generate multi-view image data. The resulting image and geometric data pairs are then used for subsequent supervised learning.

[0066] In this embodiment, a public human body scan dataset is obtained. For each human body model M, the average coordinates of all vertices of each human body model are used as the origin, and each human body model M is normalized to the unit cube. All vertices and normals of the normalized human body model are used as geometric data, and the texture of the human body model is used as material data. N} Render flash images {I1, I2, ..., I N A flash image is an image captured by placing a point light source at the camera location. In this embodiment, N = 60, the camera is located approximately 2 meters from the model, and the FOV is approximately 65 degrees.

[0067] Step 2: Build and pre-train the geometry and material prediction network, which includes three sub-modules: image feature extraction network, geometry prediction network, and material prediction network.

[0068] Image feature extraction network The input is an H×W×3 image and the output is a H×W×128 feature block, where H represents the height of the input image and W represents the width of the input image. Image Feature Extraction Network The architecture is U-Net, which consists of a 4-layer convolutional encoder, a 4-layer convolutional decoder, and a convolutional output layer. The encoder and decoder at the same level use jump connections. The detailed structure is shown in the attached figure. Figure 2 shown.

[0069] The geometry prediction network consists of the geometry transformer T geo and geometric multilayer perceptrons Composition, where T geo The network structure is as follows Figure 3 As shown, The network structure is as follows Figure 4 Given the spatial coordinates of any point in the unit cube and the camera poses corresponding to the K view images, the camera space coordinates of the projection point are obtained by projecting the point onto the K view cameras (the depth value Z of the camera space coordinates of the K projection points can be obtained at this time), and then the image feature extraction network is sampled by bilinear interpolation. The output feature block is K 128-dimensional feature vectors. The K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of the K projection points are used as the input of the geometric prediction network, that is, the geometric transformer T geo Input of the geometric transformer T geo The output of is the fused 129-dimensional features. Figure 3 As shown, the geometric transformer T geoIt contains two self-attention modules, each of which consists of a multi-head attention network (three input linear layers, a softmax layer, and an output linear layer) and a position-encoded feedforward network (linear layer and activation layer). Figure 4 As shown, the geometric multilayer perceptron The input is the 129-dimensional feature output by the geometric transformer, and the output is the directed distance corresponding to the point position, with a dimension of 1. The geometric multilayer perceptron consists of five linear layers, with output dimensions of 1024, 512, 256, 128, and 1 respectively. The middle three layers of the network and the input are skipped.

[0070] The material prediction network consists of the material transformer T app and material multilayer perceptron The input of the material prediction network is the same as that of the geometry prediction network, which is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points. app The structure, input dimension, output dimension and geometric transformer T geo The input is K 128-dimensional feature vectors and the depth value Z of the camera space coordinates of K projection points, and the output is the fused 129-dimensional feature output by the material transformer. app It contains two self-attention modules, each of which consists of a multi-head attention network (three linear layers, a softmax layer, and an output linear layer) and a position-encoded feedforward network (linear layer and activation layer). Figure 5 As shown, the material multi-layer perceptron The input is the 129-dimensional feature output by the material transformer, and the output is the material coefficient corresponding to the point position, with a dimension of 7. The material multilayer perceptron consists of five linear layers, with output dimensions of 1024, 512, 256, 128, and 7 respectively. The middle three layers are connected to the input by skip connections.

[0071] The pre-training of the geometric material prediction network specifically includes:

[0072] (2.1) Randomly sample K perspectives from the image rendered in step 1.1 to obtain images {I1, I2, ..., I k}, and camera parameters {π1, π2, ..., π k}, using image feature extraction network Extract multi-view feature blocks {V1, V2, ..., V k}.

[0073] (2.2) Randomly select points inside the unit cube and randomly select points near the model surface to obtain the point set {x}, each point passes through the camera parameter {π k}Projection to obtain the projection coordinates {X k}; For each projection coordinate {X k}, sampling multi-view feature blocks {V k}, get K 128-dimensional features {v k}.

[0074] (2.3) The K 128-dimensional features of each point {v k} and the depth value z after projection of the point to form a 129-dimensional feature, which is input to T geo and T app , after the attention mechanism fuses the multi-view features, we get the fused 1×129-dimensional geometric features and the fused 1×129-dimensional material features. The fused 129-dimensional geometric feature input Output a scalar d, representing the directed distance from the point to the geometric surface. Output material coefficient, representing the material corresponding to the point position.

[0075] (2.4) For each point in the point set {x}, the geometric loss is calculated using the actual directed distance information of the point and the directed distance predicted by the geometric prediction network; for the points sampled near the surface in the point set {x}, the material information of the point and the output of the material prediction network are used to calculate the material loss; the above losses are used to supervise the training of the image feature extraction network, the geometric prediction network, and the material prediction network using the gradient descent method. As one implementation method, the loss function L of the pre-trained geometric material prediction network is pretrain The expression is as follows:

[0076] L pretrain =L sdf +λ nml L nml +λ eik L eik +λ aln L aln +L app

[0077]

[0078] Among them, L sdf , L nml , L eik , L aln For supervised geometry, L app Used to monitor materials. sdf L represents the signed distance loss function, the purpose of which is to constrain the predicted signed distance of points near the surface to be close to 0, and to constrain the predicted signed distance of spatial points to be negative inside and positive outside. nml represents the normal loss function, which is used to constrain the predicted normal of points near the surface to be close to the normal of the real geometry; L eikRepresents the regularized loss function, which is used to constrain the gradient of the directed distance to be 1; L aln represents the alignment loss function, which is used to constrain the normal of a spatial point to be parallel to the normal of the projection point of the spatial point to the zero-value surface; L app Represents the material loss function, which is used to constrain the predicted material to be close to the real material. l ,λ nml ,λ eik ,λ aln Represent the weights of each loss function. i is the sampling point x i The predicted directed distance. i is the label of the sampling point (0 / 1 represents inside / outside the surface), τ is a learnable parameter, BCE is the binary cross entropy function, and φ is the sigmoid function. is the normal direction of the sampling point. j is x i The projection point to the zero value surface, represents the gradient. δ is a hyperparameter, which is set to 10 in this embodiment. In this embodiment, λ l =0.2,λ nml =0.1,λ eik =0.1,λ aln =0.01.

[0079] Step 3: Joint optimization based on differentiable rendering: Shoot data, emit light from random pixels of each shooting angle, use the spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to obtain the intersection of the light and the scene, and the material of the intersection is obtained according to the material prediction network pre-trained in step 2. Based on the geometry and material properties of the intersection, use physically based differentiable rendering to calculate the color, with the goal of approaching the actual pixel color, and further train and optimize the geometry and material prediction network to make the geometry and material prediction network output more accurate geometry and material. Step 3 specifically includes the following sub-steps:

[0080] (3.1) Shooting data: Under low-light conditions, use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives {I k}, and solve the camera parameters {π k}, segment the person and background to obtain the segmentation map {m k}.

[0081] (3.2) Ray casting and intersection: For each camera viewpoint, randomly sample pixel p in the image captured in step (3.1), calculate the light direction, and emit the ray to the scene. Use the spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface, and query the geometric prediction network at each step in the iteration. Get the directed distance of this step. If the directed distance is less than the set threshold, it is considered that the intersection point is found. Otherwise, let the ray move forward the directed distance. Iterate until the algorithm exits and get the intersection position of the ray and the human body, which is recorded as x0.

[0082] (3.3) Optimize through differentiable rendering: Reparameterize the intersection position x0 obtained in step (3.2) so that it can propagate the gradient to the material prediction network.

[0083]

[0084] Among them, v represents the light vector, n represents the normal direction, and d0 represents the signed distance at x0.

[0085] The material prediction network pre-trained in step 2 Query and obtain the material coefficient corresponding to the x position.

[0086] Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color C of the parameterized intersection position x :

[0087] C x =L i (v, x)f r (v, v, x)(v·n)

[0088] Among them, L i (v, x) represents the incident light intensity, f r represents the bidirectional reflectance distribution function, C x Indicates the rendering color at x when viewed from the view direction v.

[0089] The image loss is obtained by calculating the difference between the color of the image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding point to conform to the standard signed distance field, preventing the optimization from not conforming to the definition of the signed distance. By minimizing the image loss and the geometric regularization loss by the gradient descent method, the gradient of the neural network parameters can be obtained, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field. As one of the implementation methods, the expression of the loss function here is:

[0090] L ft =L c+λ mask L mask +λ eik L eik +λ aln L aln +λ sp L sp +λ smt L smt

[0091]

[0092] Among them, L c represents the color loss function, which is used to constrain the calculated color to be close to the actual pixel color; L mask represents the mask loss function, which is used to constrain the directed distance of points outside the mask to be greater than 0; L sp Represents the sparsity loss function, which is used to constrain the predicted color to be as sparse as possible; L smt represents the smoothness loss function, which is used to constrain the predicted color to be as smooth as possible; L eik and L aln The meaning of and pre-training loss function L pretrain L in eik , L aln Same. c ,λ mask ,λ eik ,λ aln ,λ sp ,λ smt Represent the weights of each loss function respectively.

[0093] Step 4: Exporting Geometry and Materials: At uniform grid points within the unit cube, directed distances are calculated using the optimized geometry prediction network from Step 3. A triangular mesh is then extracted using the marching cubes algorithm. For each triangular mesh vertex, the optimized material prediction network from Step 3 is used to predict the vertex material information. Finally, the open-source software Blender (http: / / www.blender.org) is used to automatically unwrap UVs and export the texture maps. Step 4 includes the following sub-steps:

[0094] (4.1) Uniform sampling of voxel vertices: Divide the space evenly into L 3 voxel, get the spatial position of the target voxel vertex.

[0095] (4.2) Query the optimized geometric prediction network to obtain the directed distance of the voxel vertex: L in step (4.1) 3 The vertex position of the voxel and the image taken in step (3.1) are used as input, and the geometric prediction network optimized in step 3 is run to obtain the directed distance at the vertex position.

[0096] (4.3) Extracting the triangular mesh and material: Based on the voxel vertex positions from step (4.1) and the directed distances at the vertex positions obtained from step (4.2), the Marching Cubes algorithm is used to calculate the triangular mesh. The optimized material prediction network is then used to predict the material information for the vertices of the triangular mesh. Finally, the open-source software Blender (http: / / www.blender.org) is used to automatically unwrap the UVs and export the material maps.

[0097] A 3D human body reconstruction system based on sparse perspective flash images according to another embodiment of the present invention includes:

[0098] a rendering module, for rendering a data set including flash RGB images using the acquired three-dimensional human body model;

[0099] The pre-training module is used to use the dataset as training data and pre-train the geometry and material prediction network using the gradient descent method; the geometry and material prediction network includes the image feature extraction network, the geometry prediction network, and the material prediction network;

[0100] The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed and further optimize the geometric material prediction network using the differentiable rendering framework and gradient descent method;

[0101] The geometry and material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry and material prediction network into a three-dimensional human body model with textured material.

[0102] The specific implementation process of each module corresponds to each sub-step in the method.

[0103] Training Example: The methods of the present invention were implemented using the PyTorch programming language on a desktop computer equipped with an Intel i5-4430 CPU, an Nvidia GeForce GTX 1080ti graphics processor (11GB), and 16GB of RAM. The model was trained on a desktop computer equipped with an Nvidia GeForce GTX 3090 graphics processor (24GB) and 128GB of RAM. The inventors used the Thuman2.0 database (https: / / github.com / ytrock / THuman2.0-Dataset / ) and the Custom Human database (https: / / custom-humans.github.io / ) to train the model.

[0104] Example duration: Pre-training took approximately 24 hours, with only one training run on the database data. Capturing sparse-viewpoint person images and estimating camera poses (colmap) took approximately 20 minutes, and joint optimization took approximately 10 minutes.

[0105] Effect example: Implementing the present invention, multiple flash photos are used to reconstruct the human body geometry. Different characters are rendered using Blender's renderer under different lighting conditions. The re-lighting effect is as follows: Figure 6 (dashed box on the right), Figure 7 As shown in the figure; the geometry, appearance, material (albedo, roughness, etc.) effects of the reconstructed character, such as Figure 8 The present invention can quickly reconstruct triangular meshes and materials from photos of people taken from sparse perspectives. The reconstructed geometry and materials can realistically restore the real geometry and appearance of the people ( Figures 6-8 ), the reconstruction results on human bodies of various clothing and genders prove the robustness of the present invention ( Figures 6-8 ).

[0106] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A method for reconstructing human body geometry and texture based on sparse flash images, characterized in that: The following steps are involved: Step 1: Obtain a publicly available human body scan dataset. For each human body model, normalize it to the unit cube, using the average coordinates of all vertices of the human body model as the origin. Use all vertices and normals of the normalized human body model as the geometric data, and use the human body model's texture as the material data. and use Camera parameters Rendering flash images at different perspectives around each mannequin The flash image refers to an image captured by placing a point light source at the camera position; Step 2: Construct and pre-train a geometry material prediction network, which includes an image feature extraction network, a geometry prediction network, and a material prediction network; wherein, The image feature extraction network is a U-Net architecture, and its input is The output is an image of feature block; where H represents the height of the input image and W represents the width of the input image; The geometric prediction network includes a geometric transformer and a geometric multilayer perceptron, wherein the geometric transformer includes two self-attention modules, each of which includes a multi-head attention network and a position-encoded forward network, the multi-head attention network includes three input linear layers, a Softmax layer and an output linear layer, and the position-encoded forward network includes a linear layer and an activation layer; the input of the geometric transformer is indivual The eigenvectors and The depth value Z of the camera space coordinates of the projection point is output as a fused 129-dimensional feature; the input of the geometric transformer is obtained as follows: the spatial coordinates of any point in the given unit cube, and The camera pose corresponding to the perspective image is The camera projects the point and obtains the camera space coordinates of the projection point. The depth value Z of the camera space coordinate of the projection point; then for The camera space coordinates of the projection points are obtained by sampling the feature blocks output by the image feature extraction network through bilinear interpolation. indivual The geometric multilayer perceptron includes five linear layers, the input is the fused 129-dimensional features output by the geometric transformer, and the output is the directed distance corresponding to each point, with a dimension of 1; The material prediction network includes a material transformer and a material multilayer perceptron. The structure, input dimension, and output dimension of the material transformer are the same as those of the geometric transformer. The input of the material multilayer perceptron is the fused 129-dimensional features output by the material transformer, and the output is the material coefficient corresponding to the position of each point, with a dimension of 7. Step 3: Capture an image of the target person and emit light from random pixels at each viewing angle. Use a spherical tracing algorithm and the geometry prediction network pre-trained in step 2 to determine the intersection of the light and the scene. The material of the intersection is determined using the material prediction network pre-trained in step 2. Based on the geometry and material properties of the intersection, physically based differentiable rendering is used to calculate the color. With the goal of approximating the actual pixel color, the geometry and material prediction network is further trained and optimized to output more accurate geometry and material. Step 4: At uniform grid points in the unit cube, the directed distance is calculated based on the optimized geometry prediction network obtained in step 3, and then the triangular mesh is extracted using the marching cube algorithm. For the vertices of the triangular mesh, the material information of the vertices is predicted using the optimized material prediction network obtained in step 3. Finally, the UV is automatically unfolded to obtain the material map.

2. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The image feature extraction network includes a 4-layer convolutional encoder, a 4-layer convolutional decoder and a convolutional output layer, and jump connections are used between the convolutional encoder and convolutional decoder at the same level.

3. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The output dimensions of the five linear layers of the geometric multilayer perceptron are 1024, 512, 256, 128, and 1, respectively, and the middle three linear layers and the input are skipped.

4. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The output dimensions of the five linear layers of the material multilayer perceptron are 1024, 512, 256, 128, and 7, respectively, and the middle three linear layers and the input are skipped.

5. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: The pre-trained loss function of step 2 includes geometric loss and material loss; the geometric loss is the loss between the actual directed distance information of a point in a given unit cube and the directed distance predicted by the geometric prediction network; the material loss is the loss between the material information of a point in a given unit cube and the material output by the material prediction network.

6. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 5, characterized in that: The expression of the loss function of the pre-training in step 2 is as follows: ； ； ； ； ； ； in, represents the signed distance loss function, represents the normal loss function, represents the regularized loss function, represents the alignment loss function, represents the material loss function, Respectively represent the weights of the corresponding loss functions; For sampling points predicted directed distance; is the label of the sampling point, 0 and 1 represent inside and outside the surface respectively; is a learnable parameter, BCE is the binary cross entropy function, is the sigmoid function; is the normal direction of the sampling point; for The projection point to the zero value surface, represents the gradient, is a hyperparameter.

7. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 6, characterized in that: The step three includes the following sub-steps: (3.1) Shooting data: Use a consumer-grade camera equipped with an LED light source to capture images of the target person from multiple different perspectives under low-light conditions. , solve the camera parameters , segment the person and background to get the segmentation map ; (3.2) Ray casting and intersection: For each camera view, randomly sample pixels from the image captured in step (3.1) , calculate the direction of the light and emit the ray to the scene; use the spherical tracing algorithm based on directed distance to iteratively solve the intersection of the ray and the human body surface. At each step in the iteration, query the geometric prediction network to obtain the directed distance of the step. If the directed distance is less than the set threshold, it is considered that the intersection is obtained. Otherwise, let the ray advance the directed distance. Iterate until the algorithm exits and obtain the intersection position of the ray and the human body, which is recorded as ; (3.3) Optimize by differentiable rendering: The intersection position obtained in step (3.2) Reparameterize it to propagate gradients to the geometry material prediction network: ； ； in, represents the light vector, represents the normal direction, express The directed distance at Obtained through the material prediction network query pre-trained in step 2 Material coefficient corresponding to the position; Through the rendering equation based on the microsurface model, the light direction, normal direction, and material information are input to calculate the rendering color of the parameterized intersection position : ； in, represents the incident light intensity, represents the bidirectional reflectance distribution function, express From sight Rendering color for direction viewing; The image loss is obtained by calculating the difference between the color of the image pixels and the rendered color. At the same time, the geometric regularization loss function is used to constrain the output of the geometric prediction network and the position of the corresponding points to conform to the standard signed distance field, preventing the optimization from not conforming to the definition of the signed distance. The image loss and geometric regularization loss are minimized by the gradient descent method to obtain the gradient of the neural network parameters, thereby optimizing the weight parameters of the neural network so that the rendered color is close to the real color and the predicted distance conforms to the standard signed distance field.

8. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 7, characterized in that: In step 3, the loss function of the geometric material prediction network is further trained and optimized The expression is: ； ； ； ； ； in, represents the color loss function, represents the mask loss function, represents the sparsity loss function, represents the smooth loss function; Represent the weights of each loss function respectively.

9. The method for reconstructing human body geometry and texture based on sparse flash images according to claim 1, characterized in that: Step 4 includes the following sub-steps: (4.1) Divide the space evenly into Voxel, get the spatial position of the target voxel vertex; (4.2) Replace step (4.1) The vertex position of the voxel and the image captured in step 3 are input into the geometric prediction network optimized in step 3 to obtain the directed distance at the vertex position; (4.3) Based on the voxel vertex positions in step (4.1) and the directed distances at the vertex positions obtained in step (4.2), a triangular mesh is calculated using the marching cube algorithm. The material information of the vertices of the triangular mesh is then predicted using the optimized material prediction network. Finally, the open source software Blender is used to automatically unfold UVs and export the material map.

10. A 3D human body reconstruction system based on sparse perspective flash images, characterized in that: The system is used to implement the method for reconstructing human body geometry and texture based on sparse flash images as described in any one of claims 1 to 9, and the system comprises: a rendering module, for rendering a data set including flash RGB images using the acquired three-dimensional human body model; A pre-training module, configured to use the dataset as training data and pre-train the geometric material prediction network using a gradient descent method; The fine-tuning optimization module is used to collect the flash RGB image of the person to be reconstructed and further optimize the pre-trained geometric material prediction network using the differentiable rendering framework and gradient descent method; The geometry and material export module is used to export the geometry and material of the reconstructed character output by the optimized geometry and material prediction network into a three-dimensional human body model with textured material.

Images