A method for fast reconstruction of textured 3D models based on neural shaders

Abstract

The application discloses a kind of based on neural shader's fast reconstruction textured three-dimensional model method, comprising the following steps: step S1, a group of images are photographed at different angles around the object to be reconstructed, and the foreground image, camera pose and internal parameter are obtained by pre-processing image as training data, step S2, establish initial triangular grid, step S3, establish differentiable renderer, it contains rasterization and neural shader two parts vertex and texture of the model of the application are divided into two optimizers, model is successively differentiable rasterization and differentiable neural shader, obtain rendering chart, calculate loss with original chart and back propagation, compared with the ray step of implicit function of other reconstruction methods, the time of rasterization is significantly shorter, since the application of neural shader, the model obtained can be used to paste texture with trained neural shader, so that fast reconstruction textured three-dimensional model is realized.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology in computer graphics, specifically a method for rapidly reconstructing textured 3D models based on neural shaders. Background Technology

[0002] 3D reconstruction based on multi-view images has been a long-standing problem in the field of computer graphics. In recent years, NeRF and NeuS and their various variants have made great strides in the field of 3D reconstruction. The former uses neural networks to learn density fields and color fields to synthesize images from new perspectives, and can render high-quality images even for highly complex scenes. However, this method of implicitly representing 3D scenes is difficult to extract high-quality surfaces. The latter uses the signed distance function (SDF) to describe the surface and uses a new volume rendering method to train the SDF representation, showing excellent performance in multi-view surface reconstruction tasks. Its main drawback is that the training time is long and the reconstructed model does not have texture.

[0003] Many existing 3D reconstruction techniques can extract high-quality 3D models, but few of them can simultaneously reconstruct textures. The few methods that can reconstruct textured 3D models are too time-consuming, limiting their practical value. Summary of the Invention

[0004] This invention provides a method for rapidly reconstructing textured 3D models based on neural shaders, which can effectively solve the problem mentioned in the background art that few existing 3D reconstruction techniques can simultaneously reconstruct textures, and the few existing methods for reconstructing textured 3D models are limited in their practical value due to excessive time consumption.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for rapidly reconstructing textured 3D models based on neural shaders, comprising the following steps:

[0006] Step S1: Take a set of images around the object to be reconstructed from different angles, and preprocess the images to obtain the foreground image, camera pose and intrinsic parameters as training data.

[0007] Step S2: Establish the initial triangular mesh;

[0008] Step S3: Create a differentiable renderer, which consists of two parts: rasterization and neural shaders.

[0009] Step S4: Deferred shading is used. First, the triangular mesh is rasterized at a given viewpoint. Then, the neural shader processes the overlap of mesh, material and light to render the image at that viewpoint.

[0010] Step S5 iterates over step S4 to optimize the triangular mesh and neural shader;

[0011] Step S6: After training, establish the mapping between the vertices of the triangular mesh and the texture map, and increase the number of mapping logs by interpolation. Calculate the color value of each point using the trained neural shader, and save it to the texture map through the mapping relationship between the spatial points of the mesh surface and the pixels of the texture map.

[0012] In step S3, the neural shader employs a multilayer perceptron (MLP), which is based on the following formula:

[0013] c = h(f(x),n,d);

[0014] In the formula, x, n, d, and c represent the position of each pixel in the given image corresponding to the triangular mesh, the surface normal vector, the view direction, and the returned color value, respectively.

[0015] First, position x is encoded and converted into high-dimensional position features by three fully connected layers. Then, it is input into the subsequent hidden layer along with the normal vector n and the view direction d. Finally, the color value c is returned through the output layer.

[0016] In step S5, the mesh optimization adopts a coarse-to-fine optimization scheme, starting from a coarse custom triangular mesh and gradually reducing the triangle scale during the optimization process to finally obtain a fine triangular mesh. The loss function is divided into two parts: mesh loss and color loss. The color loss function and mesh loss function adopt L1 loss and L2 loss respectively. The mesh loss is further subdivided into contour loss, Laplacian loss and surface normal consistency loss. The weighted average of the four is taken as the loss during backpropagation. After the triangular mesh is rasterized, 80% of the pixels are randomly selected to participate in subsequent calculations.

[0017] Contour loss (L mask The calculation formula used is as follows:

[0018]

[0019] Contour loss is achieved using the L2 norm of the rendered image mask and the original foreground image mask;

[0020] Laplace loss (L laplacian The calculation formula used is as follows:

[0021]

[0022] The Laplace loss uses the L2 norm of the difference coordinates of each vertex;

[0023] Surface normal uniformity loss (L normal The calculation formula used is as follows:

[0024]

[0025] The surface normal consistency loss is calculated using the L2 norm of the cosine similarity of the normal vectors of each triangle.

[0026] Color loss (L) color The calculation methods used for the four losses are as follows:

[0027]

[0028] Color loss is calculated using the L1 norm of the rendered image and the original foreground image.

[0029] According to the above technical solution, in step S4, a foreground image is first randomly selected from the training data. Under the viewpoint given by the camera pose, the triangular mesh is rasterized to obtain the coordinates of each visible triangle vertex in the image. Then, the position, surface normal vector and viewpoint direction of each pixel in the foreground image corresponding to the triangular mesh are calculated using interpolation. These are then input into the neural shader described in S3 to obtain the color value of each pixel. Finally, the image under the given viewpoint is rendered.

[0030] According to the above technical solution, in step S5, the triangle scale is gradually reduced during the optimization process. Specifically, when the number of iterations reaches 20%, 40%, and 60% of the total number of iterations, the triangle mesh is reorganized and refined.

[0031] The specific method of reorganization and refinement is as follows: edges longer than 4 / 3 of the target edge length are subdivided, and the midpoint of the newly added edge is used as the new grid vertex. Edges shorter than 3 / 5 of the target edge length are folded, and the midpoint of the edge replaces the two endpoints as the new grid point. In each reorganization and refinement, the target edge length is taken as half of the average of all edge lengths of the current triangle grid.

[0032] Specifically, when three mesh reorganizations and subdivisions have been completed, and the mesh loss term no longer decreases after falling below a certain value for a long time, the triangular network is fixed while the neural shader continues to be optimized.

[0033] According to the above technical solution, in step S6, the color value of each point is calculated using the trained neural shader. The position of each point on the mesh surface is calculated by interpolating the position of the mesh vertex. The surface normal vector is calculated by cross product of the edge vectors of the triangles where each point is located. For points located on edges or vertices, the surface normal vector is the average of the normal vectors of all triangles where the point is located. The viewing direction is selected from the surface normal vector and its surrounding directions, and invisible directions are eliminated. The color value of each direction is calculated and the average value is taken as the final color value of the point.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] The model's vertices and textures are optimized using two separate optimizers. The model is then processed by differentiable rasterization and differentiable neural shaders to obtain a rendered image. The loss is calculated with the original image and backpropagated. Compared to the implicit function ray stepping of other reconstruction methods, the rasterization time is significantly shorter. Due to the application of neural shaders, the obtained model can be textured using trained neural shaders, thus achieving fast reconstruction of textured 3D models. Attached Figure Description

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0037] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the network structure of the neural shader MLP of this invention;

[0039] Figure 3 It is a textureless model image obtained through training in the embodiments of the present invention;

[0040] Figure 4 It is a textured model image obtained through training in an embodiment of the present invention. Detailed Implementation

[0041] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0042] Example: Figure 1 As shown, this invention provides a technical solution: a method for rapidly reconstructing textured 3D models based on neural shaders, which can reconstruct textured 3D models while maintaining both performance and time efficiency. The method includes the following steps:

[0043] Step S1: Take a set of images around the object to be reconstructed from different angles, and use COLMAP and PaddleSeg tools to preprocess the images to obtain the foreground image, camera pose and intrinsic parameters as training data.

[0044] Step S2: Create an initial triangular mesh to obtain a textureless model;

[0045] Step S3: Establish a differentiable renderer and optimizer, which consists of two parts: rasterization and neural shader. The entire process of both is differentiable. The optimizer uses the Adam optimizer. The former optimizes the vertex positions of the triangle mesh, and the latter optimizes the parameters of the neural shader network.

[0046] like Figure 2 As shown, the differentiability of rasterization is implemented using the torch package in the nvdiffrast library. The neural shader uses an MLP, and in its network structure diagram, x, n, d, and c represent the position of each pixel in the given image corresponding to the triangular mesh, the surface normal vector, the view direction, and the returned color value, respectively. This network is based on the following formula:

[0047] c = h(f(x),n,d)

[0048] First, position x is encoded and converted into high-dimensional position features by three fully connected layers. Then, it is input into the subsequent hidden layer along with the surface normal vector n and the view direction d. Finally, the color value c is returned through the output layer. The hidden layer uses the ReLU activation function and the output layer uses the sigmoid activation function. The optimizer used in all cases is the Adam optimizer, which aims to make the optimization process converge faster by taking advantage of its adaptability and momentum characteristics.

[0049] Position encoding is performed on position x to convert a three-dimensional point into a high-dimensional position feature. The purpose is to learn the similar position features of regions with similar materials, and at the same time, to learn the differences in regions with the same material caused by different lighting conditions.

[0050] like Figure 3-4 As shown, in step S4, deferred shading is used. First, the triangular mesh is rasterized at a given viewpoint, and then the neural shader processes the overlap of mesh, material and light to render the image at that viewpoint.

[0051] First, a foreground image is randomly selected from the training data. Under the view given by the camera pose, the rasterize function in the torch package of the nvdiffrast library is used to rasterize the current triangular mesh to obtain the coordinates of each visible triangle vertex in the image. Then, interpolation is used to calculate the position, surface normal vector and view direction of each pixel in the foreground image corresponding to the triangular mesh. These are then input into the neural shader described in S3 to obtain the color value of each pixel. Finally, the image under the given view is rendered.

[0052] The position, surface normal vector, and viewing direction of each pixel in the foreground image corresponding to the triangular mesh are calculated. Based on the correspondence between the vertices of the triangular mesh and the pixels in the foreground image, the correspondence between each pixel and the three-dimensional points on the mesh surface is obtained through interpolation. The surface normal vector is obtained by the cross product of the two edge vectors of the triangle where the point is located on the mesh surface. For points located on edges and vertices, the surface normal vector is the average of the normal vectors of all triangles where the point is located. The viewing direction is obtained from the camera pose, specifically the vector from the camera position to each point on the mesh surface.

[0053] Step S5 iterates over step S4 to optimize the triangular mesh and neural shader;

[0054] Mesh optimization adopts a coarse-to-fine optimization scheme, starting with a coarse custom triangular mesh and gradually reducing the triangle scale during the optimization process to finally obtain a fine triangular mesh. The loss function consists of two parts: mesh loss and color loss. The mesh loss is further subdivided into contour loss, Laplacian loss, and surface normal consistency loss. The weighted average of the four is taken as the loss during backpropagation. To avoid overfitting, after the triangular mesh is rasterized, only 80% of the pixels are randomly selected to participate in subsequent calculations.

[0055] During the optimization process, the triangle scale is gradually reduced. Specifically, when the number of iterations reaches 20%, 40%, and 60% of the total number of iterations, the triangle mesh is reorganized and refined. Specifically, edges longer than 4 / 3 of the target side length are subdivided, and the midpoint of the newly added edge is used as the new mesh vertex. Edges shorter than 3 / 5 of the target side length are folded, and the midpoint of the edge replaces the two endpoints as the new mesh grid point. In each reorganization and refinement, the target side length is taken as half of the average of all side lengths of the current triangle mesh.

[0056] The calculation methods used for the four losses are as follows:

[0057]

[0058]

[0059]

[0060]

[0061] Contour loss uses the L2 norm of the rendered image mask and the original foreground image mask; Laplacian loss uses the L2 norm of the difference coordinates of each vertex; surface normal consistency loss uses the L2 norm of the cosine similarity of the normal vectors of each triangle; color loss uses the L1 norm of the rendered image and the original foreground image; mesh loss uses L2 loss to accelerate the convergence of mesh vertices; and color loss uses L1 loss to enhance robustness.

[0062] In particular, when three mesh reorganizations and subdivisions have been completed, and the mesh loss term no longer decreases after falling below a certain value for a long time, the triangular network is fixed while the neural shader is further optimized in order to improve the subsequent texture effect;

[0063] Step S6: After training, use the xatlax toolkit to establish the mapping between the vertices of the triangle mesh and the texture map, and use interpolation to increase the number of mapping logarithms to ensure that the surface texture of the model is clear enough. Use the trained neural shader to calculate the color value of each point, and save it to the texture map through the mapping relationship between the spatial points of the mesh surface and the pixels of the texture map.

[0064] The trained neural shader is used to calculate the color value of each point. The position of each point on the mesh surface is calculated by interpolating the position of the mesh vertex. The surface normal vector is calculated by cross product of the edge vectors of the triangle where each point is located. For points located on edges and vertices, the surface normal vector is the average of the normal vectors of all triangles where the point is located. The view direction is selected from the surface normal vector and its surrounding direction, and invisible directions are eliminated. The color value of each direction is calculated and the average value is taken as the final color value of the point.

[0065] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fast reconstruction of textured 3D models based on neural shaders, characterized in that: Includes the following steps: Step S1: Take a set of images around the object to be reconstructed from different angles, and preprocess the images to obtain the foreground image, camera pose and intrinsic parameters as training data. Step S2: Establish the initial triangular mesh; Step S3: Create a differentiable renderer, which consists of two parts: rasterization and neural shaders. Step S4: Deferred shading is used. First, the triangular mesh is rasterized at a given viewpoint. Then, the neural shader processes the overlap of mesh, material and light to render the image at that viewpoint. Step S5 iterates over step S4 to optimize the triangular mesh and neural shader; Step S6: After training, establish the mapping between the vertices of the triangular mesh and the texture map, and increase the number of mapping logs by interpolation. Calculate the color value of each point using the trained neural shader, and save it to the texture map through the mapping relationship between the spatial points of the mesh surface and the pixels of the texture map. In step S3, the neural shader employs a multilayer perceptron (MLP), which is based on the following formula: c = h(f(x),n,d); In the formula, x, n, d, and c represent the position of each pixel in the given image corresponding to the triangular mesh, the surface normal vector, the view direction, and the returned color value, respectively. First, position x is encoded and converted into high-dimensional position features by three fully connected layers. Then, it is input into the subsequent hidden layer along with the normal vector n and the view direction d. Finally, the color value c is returned through the output layer. In step S5, the mesh optimization adopts a coarse-to-fine optimization scheme, starting from a coarse custom triangular mesh and gradually reducing the triangle scale during the optimization process to finally obtain a fine triangular mesh. The loss function is divided into two parts: mesh loss and color loss. The color loss function and mesh loss function adopt L1 loss and L2 loss respectively. The mesh loss is further subdivided into contour loss, Laplacian loss and surface normal consistency loss. The weighted average of the four is taken as the loss during backpropagation. After the triangular mesh is rasterized, 80% of the pixels are randomly selected to participate in subsequent calculations. Contour loss (L mask The calculation formula used is as follows: Contour loss is achieved using the L2 norm of the rendered image mask and the original foreground image mask; Laplace loss (L laplacian The calculation formula used is as follows: The Laplace loss uses the L2 norm of the difference coordinates of each vertex; Surface normal uniformity loss (L normal The calculation formula used is as follows: The surface normal consistency loss is calculated using the L2 norm of the cosine similarity of the normal vectors of each triangle. Color loss (L) color The calculation methods used for the four losses are as follows: Color loss is calculated using the L1 norm of the rendered image and the original foreground image.

2. The method for fast reconstruction of textured 3D models based on neural shaders according to claim 1, characterized in that, In step S4, a foreground image is randomly selected from the training data. Under the viewpoint given by the camera pose, the triangular mesh is rasterized to obtain the coordinates of each visible triangle vertex in the image. Then, the position, surface normal vector and viewpoint direction of each pixel in the foreground image corresponding to the triangular mesh are calculated using interpolation. These are then input into the neural shader described in S3 to obtain the color value of each pixel. Finally, the image under the given viewpoint is rendered.

3. The method for fast reconstruction of textured 3D models based on neural shaders according to claim 1, characterized in that, In step S5, the triangle scale is gradually reduced during the optimization process. Specifically, the triangle mesh is reorganized and refined when the number of iterations reaches 20%, 40%, and 60% of the total number of iterations. The specific method of reorganization and refinement is as follows: edges longer than 4 / 3 of the target edge length are subdivided, and the midpoint of the newly added edge is used as the new grid vertex. Edges shorter than 3 / 5 of the target edge length are folded, and the midpoint of the edge replaces the two endpoints as the new grid point. In each reorganization and refinement, the target edge length is taken as half of the average of all edge lengths of the current triangle grid. Specifically, when three mesh reorganizations and subdivisions have been completed, and the mesh loss term no longer decreases after falling below a certain value for a long time, the triangular network is fixed while the neural shader continues to be optimized.

4. The method for fast reconstruction of textured 3D models based on neural shaders according to claim 1, characterized in that, In step S6, the color value of each point is calculated using the trained neural shader. The position of each point on the mesh surface is calculated by interpolating the position of the mesh vertex. The surface normal vector is calculated by cross product of the edge vectors of the triangles where each point is located. For points located on edges or vertices, the surface normal vector is the average of the normal vectors of all triangles where the point is located. The viewing direction is selected from the surface normal vector and its surrounding directions, and invisible directions are eliminated. The color value of each direction is calculated and the average value is taken as the final color value of the point.

Images