Method and device for reconstructing three-dimensional scene

Abstract

The invention discloses a method and equipment for reconstructing a three-dimensional scene, and relates to the field of computer graphics. The method comprises the following steps: simulating the process of light from the optical center of a camera to a three-dimensional scene through a light advancing module, and determining space coordinates of a plurality of sampling points in the three-dimensional scene; decomposing the color feature space and the density feature space, and determining a matrix and a vector pair corresponding to the space coordinates of each sampling point; interpolating each matrix and vector pair, and determining a color feature vector and density of each sampling point; determining a spherical harmonic feature vector of each sampling point according to the color feature vector of each sampling point and a spherical harmonic code of surface modeling; determining an anti-aliasing feature of each sampling point according to the color feature vector and the cone code of each sampling point; splicing the spherical harmonic feature vectors and the anti-aliasing features, and inputting a predicted color of each sampling point output by a target multi-layer perceptron; and carrying out volume rendering on the predicted color and density to determine the reconstructed three-dimensional scene, and reconstructing the three-dimensional scene with high efficiency and high quality.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics, and in particular to a method and device for reconstructing a three-dimensional scene. Background Art

[0002] In recent years, with the development of computer graphics and machine learning technology, Neural Radiance Fields (NeRF), as an emerging 3D scene reconstruction technology, has received extensive attention and application in the fields of virtual reality, augmented reality, and computer vision. On the one hand, traditional NeRF methods usually rely on complex optimization processes and a large number of training images to restore the geometric structure and appearance properties of the scene. Although this method can produce more realistic rendering effects, it is often accompanied by problems such as high computational cost and large memory usage. On the other hand, aliasing is prone to occur when processing high-frequency textures and details, resulting in low image rendering quality and rendering efficiency, which in turn leads to low reconstruction quality and reconstruction efficiency of 3D scenes.

[0003] Therefore, developing a method that can effectively reduce memory usage, improve rendering quality and efficiency, and thus improve the reconstruction quality and efficiency of three-dimensional scenes has become an important direction of current research. Summary of the invention

[0004] The purpose of the present application is to provide a three-dimensional scene reconstruction method and device, which can solve the problems of low quality and efficiency of three-dimensional scene reconstruction in the prior art.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for reconstructing a three-dimensional scene, comprising:

[0007] The ray marching module is used to simulate the process of light from the optical center of the camera to the three-dimensional scene, and determine the spatial coordinates of multiple sampling points in the three-dimensional scene; the three-dimensional scene is a three-dimensional voxel grid space composed of voxel grids; the ray marching module is used to simulate the process of light from the optical center of the camera to the three-dimensional scene by using the camera posture and the camera intrinsic parameters;

[0008] Converting the three-dimensional voxel grid space into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space;

[0009] Decomposing the color feature space and the density feature space to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point; the matrix and vector pair include a color matrix and vector pair, and a density matrix and vector pair;

[0010] Interpolating each of the matrix and vector pairs to determine the color feature vector and density of each sampling point;

[0011] Determine the spherical harmonic eigenvector of each sampling point according to the color eigenvector of each sampling point and the spherical harmonic coding of the surface modeling;

[0012] Determine the anti-aliasing feature of each sampling point according to the color feature vector and the cone encoding of each sampling point;

[0013] splicing the spherical harmonic feature vector and the anti-aliasing feature, inputting the result into a target multi-layer perceptron, and outputting the predicted color of each sampling point;

[0014] Performing volume rendering on the predicted color and the density of each sampling point to obtain a rendered image;

[0015] A reconstructed three-dimensional scene is determined according to the rendered image.

[0016] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene reconstruction method described above.

[0017] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0018] The present application provides a method and device for reconstructing a three-dimensional scene. First, a light marching module is used to simulate the process of light from the optical center of a camera to a three-dimensional scene, and the spatial coordinates of multiple sampling points in the three-dimensional scene are determined. Then, the color feature space and density feature space converted from the three-dimensional voxel grid space are decomposed to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point, thereby reducing memory consumption and achieving memory optimization. Then, the color feature space and density feature space are decomposed to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point, and each matrix and vector pair is interpolated to determine the color feature vector and density of each sampling point; further, according to the obtained color feature vector of each sampling point, the spherical harmonic coding and cone coding of surface modeling are used to obtain the spherical harmonic feature vector and anti-aliasing feature, so that not only the high quality of the rendered image is guaranteed, but also the anti-aliasing ability of the image in the rendering process is improved. Finally, the rendering efficiency and image quality of the image are improved by optimizing memory management, spherical harmonic coding and cone coding of surface modeling. Therefore, based on the high-quality rendered image, the reconstructed three-dimensional scene is obtained efficiently and with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a method for reconstructing a three-dimensional scene provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of tensor decomposition provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a ray marching module and a tensor decomposition module provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a cone encoding provided in an embodiment of the present application;

[0024] Figure 5 This is a general architecture diagram of a three-dimensional scene reconstruction method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a three-dimensional scene reconstruction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] like Figure 1 As shown, the present application provides a method for reconstructing a three-dimensional scene, comprising:

[0029] Step 101: The ray marching module is used to simulate the process of light from the optical center of the camera to the three-dimensional scene, and the spatial coordinates of multiple sampling points in the three-dimensional scene are determined; the three-dimensional scene is a three-dimensional voxel grid space composed of voxel grids; the ray marching module is used to simulate the process of light from the optical center of the camera to the three-dimensional scene by using the camera posture and camera intrinsic parameters.

[0030] In some embodiments, step 101 specifically includes: based on the ray marching module, using the camera pose and the camera intrinsic parameters, simulating the process of light from the optical center of the camera to the three-dimensional scene; in the process of light from the optical center of the camera to the three-dimensional scene, determining the spatial coordinates of multiple sampling points in the three-dimensional scene according to the propagation path of the light and the sampling strategy.

[0031] Among them, the Ray Marching Module is a technology used in computer graphics to render complex scenes.

[0032] The specific process of obtaining multiple sampling points according to the light path and sampling strategy is as follows:

[0033] The camera pose and camera intrinsic parameters are input into the ray marching module, and multiple light direction vectors emitted into the three-dimensional scene through the camera optical center are calculated based on these parameters.

[0034] Specifically, for each pixel in the image, first determine the coordinates (u, v) of the pixel on the image plane, then use the camera intrinsic parameters to convert the pixel coordinates to normalize to [-1, 1]; then, according to the camera pose matrix, transform the light vector to the world coordinate system to obtain the direction vector d of the light; then combine the position vector o of the camera optical center as the starting point of the light, and finally obtain the starting point and direction of each light, that is, the light direction vector o+d. Finally, sample the light direction vector to obtain the spatial coordinates of multiple sampling points, and use the spatial coordinates of multiple sampling points as training data points.

[0035] It is worth noting that in the early stages of training, for example when the number of iterations is less than 8000, an occupancy mask is calculated for each 1000 iterations of the voxel grid to indicate which grids are occupied, and then the bounding box is reduced according to the mask and the number of voxels is increased, so that the resolution of the voxels increases, and the value of the new voxel is determined by interpolating the previous lower resolution voxels. As the new resolution is determined, the ray travels in the bounding box and its sampling step is updated accordingly. Based on the new resolution and step ratio, the number of sampling points for each ray is calculated.

[0036] Exemplarily, there is the following camera intrinsic parameter matrix K:

[0037]

[0038] Among them, f x and f y is the focal length of the camera along the x and y axes, c x and c y is the offset of the image center relative to the pixel coordinate origin.

[0039] The camera pose matrix contains the rotation matrix R and the translation vector t, which are used to transform the points in the world coordinate system to the camera coordinate system. The general form is:

[0040]

[0041] Where T is the rotation and translation matrix.

[0042] Suppose you want to calculate the coordinates of the light direction corresponding to a pixel in the image, you first need to normalize it to [-1,1]. For the case of no distortion, this step can be completed in the following way:

[0043]

[0044]

[0045] Among them, (u, v) is the coordinate of the pixel point, u' is the X component of the light vector, and v' is the Y component of the light vector.

[0046] That is to say, calculate the distance from this pixel to the center of the image, and then divide it by the focal length to get the normalized coordinates as the XY components of the light. The Z component is set to -1 to ensure that the light is emitted from the camera toward the scene. This step obtains the direction vector r of the light in the camera coordinate system. c

[0047] r c =o c +d c .

[0048] Among them, c is (0,0,0,1), d c is (u,v,-1,0).

[0049] Furthermore, r c Multiply P on the left to get the direction vector r of the light in the world coordinate system w .

[0050] r w =o w +d w .

[0051] Step 102: converting the three-dimensional voxel grid space into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space.

[0052] Step 103: Decompose the color feature space and the density feature space to determine the matrix and vector pairs corresponding to the spatial coordinates of each sampling point; the matrix and vector pairs include color matrix and vector pairs, and density matrix and vector pairs.

[0053] Among them, step 103 is performed in the tensor decomposition module, and the tensor decomposition module performs interpolation operations in the matrix and vector pairs represented by the tensor according to the spatial coordinates of the sampling points to obtain the color feature vector and density. It helps to significantly reduce the required graphics processing unit memory consumption. Specifically, in order to efficiently represent and process the three-dimensional voxel grid space, the four-dimensional feature space is decomposed into a series of matrix-vector pairs to reduce memory usage.

[0054] like Figure 2 , assuming there is a 300 3 A three-dimensional voxel grid space, each voxel contains a feature vector of length 8. Storing the entire voxel space requires 300 3 ×8×4B=824MB of memory. However, using the above method to store the 16 matrix-vector pairs after decomposition, only (300 2 +300)×3×8×4B×16=149MB of memory space, where sparse voxel storage is a technology for efficiently storing and processing 3D voxel data. In the fields of 3D reconstruction, computer graphics, virtual reality, and augmented reality, sparse voxel storage technology can significantly reduce memory usage and improve computing efficiency.

[0055] Specifically, the three-dimensional voxel grid space is usually represented as a 4D tensor τ, where (X, Y, Z) are the spatial coordinates of the sampling points and F is the features of these sampling points. Consider decomposing the four-dimensional feature space (4D tensor) into a combination of planes and vectors. That is, decomposing the 4D tensor into a combination of multiple 1D tensors (vectors) and 2D tensors (matrices), represented as (v x ,v y ,v z ) and (M YZ ,M XZ ,M XY ), see Figure 2 .

[0056]

[0057] where τ refers to the three-dimensional voxel grid space, is the decomposed vector, is the decomposed matrix, For example, Include and Decomposition tensors corresponding to density and color, respectively, refers to the outer product, and R is the logarithm of the matrix-vector after decomposition.

[0058] Step 104: interpolate each of the matrix and vector pairs to determine the color feature vector and density of each sampling point.

[0059] In some embodiments, step 104 specifically includes steps 201 to 204:

[0060] Step 201: for the color feature space, interpolate the color matrix and vector pair corresponding to the space coordinates of each sampling point to determine each first interpolation result.

[0061] Step 202: Add the first interpolation results to determine the color feature vector of each sampling point.

[0062] Step 203: for the density feature space, interpolate the density matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each second interpolation result.

[0063] Step 204: Add the second interpolation results to determine the density of each sampling point.

[0064] In some embodiments, step 201 specifically includes steps 301 to 304:

[0065] Step 301: For any set of color matrices and vectors in the color matrix and vector pair, bilinear interpolation is performed on the matrices in the color matrix and vector to obtain matrix results, and linear interpolation is performed on the vectors in the color matrix and vector to obtain vector results; the color matrix and vector pair includes three sets of color matrices and vectors, each set of color matrices and vectors uses a different coordinate plane as a matrix and the remaining coordinate axes as vectors.

[0066] Step 302: taking the product of the first matrix result and the first vector result as the first interpolation result.

[0067] In some embodiments, step 203 specifically includes steps 401 to 404:

[0068] Step 401: For any set of density matrices and vectors in the density matrix and vector pairs, bilinear interpolation is performed on the matrices in the density matrix and vectors to obtain a second matrix result, and linear interpolation is performed on the vectors in the density matrix and vectors to obtain a second vector result; the density matrix and vectors include three sets of density matrix and vector pairs, each set of density matrix and vector uses a different coordinate plane as a matrix and the remaining coordinate axes as vectors.

[0069] Step 402: taking the product of the second matrix result and the second vector result as the second interpolation result.

[0070] like Figure 3As shown in the figure, at each sampling point, the features of the sampling point can be obtained by interpolation. For color information, linear interpolation is performed on each pair of matrices and vectors to obtain a set of color features. The color features obtained by interpolation of each pair of matrices and vectors are multiplied and then concatenated to obtain a color feature vector. For density information, interpolation is also performed. After summing the density features, the density value is directly obtained through the sigmoid function. For example, suppose is a two-dimensional tensor, is a one-dimensional tensor, in which each element is a feature vector, the projection coordinates of the sampling point (x, y, z) on the YZ plane are (y, z), and the projection coordinates on the X axis are x, and the following interpolation is performed:

[0071]

[0072]

[0073] in, is the feature of the tensor corresponding to the sampling point in the YZ plane, is the feature of the tensor corresponding to the X-axis; interp and interp2 represent linear and bilinear interpolation respectively.

[0074] for and Perform similar operations to obtain That is, the features on the tensor corresponding to the Y axis, That is, the characteristics of the tensor corresponding to the XZ plane, That is, the characteristics and That is, the features on the tensor corresponding to the XZ plane. Multiply them separately and then sum them to get the color feature vector f of the sampling point a .

[0075]

[0076] The density is obtained in the same way as the color feature vector. That is, the characteristics of the tensor corresponding to the YZ plane, That is, the features on the tensor corresponding to the X-axis, That is, the features on the tensor corresponding to the Y axis, That is, the characteristics of the tensor corresponding to the XZ plane, That is, the characteristics and That is, the features on the tensor corresponding to the XY plane.

[0077]

[0078] Among them, R is the total number of vector pairs after decomposition, and d is the density of sampling points.

[0079] Step 105: Determine the spherical harmonic feature vector of each sampling point according to the color feature vector of each sampling point and the spherical harmonic coding of the surface modeling.

[0080] In some embodiments, step 105 specifically includes steps 501 to 504:

[0081] Step 501: For any sampling point, the spatial coordinates of the sampling point, as well as the color feature vector and density of the sampling point, are input into a first multi-layer perceptron to determine the surface roughness and surface normal vector of the sampling point; wherein the surface roughness is the concentrated parameter of the von Mises-Fisher distribution of the surface normal vector.

[0082] Among them, the tensor decomposition module includes a first multi-layer perceptron, the input of the first multi-layer perceptron is the spatial coordinates of the sampling points, and the output is the density and color feature vector of the sampling points. The color feature vector is also used as the input of the cone encoding module and the spherical harmonic encoding based on surface modeling.

[0083] Step 502: Calculate the reflection direction of the sampling point using the surface normal vector and the observation direction of the sampling point; the observation direction is determined based on the camera posture.

[0084] Step 503: Encode the reflection direction of the sampling point using a spherical harmonic function, and determine an expected value of the spherical harmonic function encoding of the reflection direction.

[0085] Step 504: taking the expected value of the spherical harmonic function encoding of the reflection direction of each sampling point as the spherical harmonic feature vector of each sampling point.

[0086] Among them, the two parts of surface modeling and coding and cone coding can be regarded as feature coding modules, and surface modeling and coding and cone coding mainly operate on color feature vectors. In this embodiment, it is considered that the scene surface is composed of multiple tiny planes, each of which is a perfect plane. For surfaces with high roughness, the directions of these tiny planes are very different, indicating that the variance of tiny normal vectors is large. The appearance of the rough surface remains consistent no matter from which viewing direction. On the contrary, for surfaces with low roughness, the directions of the tiny planes are roughly similar. The visual properties of rough materials remain relatively consistent regardless of how the viewing direction changes, while smoother materials show significant changes when the viewing direction changes.

[0087] This application uses spherical harmonic coding to encode the reflection direction, and then calculates the expectation of its spherical harmonic coding under the von Mises-Fisher (vMF) distribution. First, the surface roughness and surface normal vector of the sampling point are predicted according to the first multi-layer perceptron, and then the reflection direction is calculated according to the surface normal vector and the light direction corresponding to the sampling point:

[0088]

[0089] in, is the predicted surface normal vector, w 0 is the light direction, w r is the reflection direction.

[0090] After obtaining the reflection direction, the expected encoding of the spherical harmonic function of the reflection direction under the vMF distribution is calculated. The specific formula is:

[0091]

[0092] M L =(I,m):I=1,...,2 L ,m=0,...I.

[0093] in is a spherical harmonic function, l and m are the order and sequence of the spherical harmonic function, is the reflection direction; κ is the surface roughness; m is the order of the spherical harmonic function, M L is the order matrix, Encode functions for spherical harmonics.

[0094] It is expected that this can be calculated using a simple closed-form expression, which is obtained by multiplying the attenuation coefficient of the vMF distribution by multiple basis functions of the spherical harmonics, as follows:

[0095]

[0096] Among them, A L (κ) is the decay function.

[0097] Specifically, the attenuation function A L (κ) can be approximated as:

[0098]

[0099] Where κ is the surface roughness predicted by MLP. Therefore:

[0100]

[0101] Step 106: Determine the anti-aliasing feature of each sampling point according to the color feature vector and the viewing cone code of each sampling point.

[0102] The cone encoding method is expected to be very similar to the orientation encoding used in NeRF. The specific steps are as follows: First, convert dxdydz to cylindrical coordinates rt 2 drdtdθ, and derive the first-order moment (mean) and second-order moment (variance) of t and r, where E(x) is equal to 0.

[0103]

[0104] Where f is the focal length, is the shortest distance from the point to the center line of the cone; V is the volume of the intercepted cone; t 0 is the near clipping plane distance of the intercepted view cone; t 1 The far clipping plane distance of the intercepted viewing cone.

[0105] like Figure 4 , t represents the distance from the cone to the optical center and the first and second order moments of r are given as follows:

[0106]

[0107]

[0108]

[0109] Var(t)=E(t 2 )-E 2 (t).

[0110] Where E(t) is the expectation of t and Var(r) is the variance of r.

[0111] Due to symmetry, the mean of r is zero, and the variance of r is equal to the variance of the corresponding cone in the x or y direction.

[0112] The first and second order moments of t and u are encoded into frequencies.

[0113]

[0114] Where: P is the frequency matrix of position encoding; L is the number of frequencies, x is the eigenvector; F(x) is the encoding function.

[0115] In some embodiments, step 106 specifically includes: obtaining a viewing cone projected from the optical center of the camera along the direction of the pixel point in the three-dimensional scene; intercepting the viewing cone according to the spatial coordinates of each sampling point to determine the intercepted viewing cone; calculating the uniform distribution of the color feature vector of each sampling point in the intercepted viewing cone; and performing high-frequency encoding on the variance and expectation of the uniform distribution to obtain the anti-aliasing feature of each sampling point.

[0116] This application directly multiplies the expected E by the matrix P to obtain the encoded high-frequency mean feature F(E)=PE. For σ, due to the property of the covariance of the variable under linear transformation, that is, the linear transformation of the covariance of the variable is equal to the linear transformation of the covariance of the variable, the encoded variance is F(σ 2 )=Pσ 2 P T . Let F(E) and F(σ 2 ) can be spliced ​​together to obtain the anti-aliasing feature.

[0117] Step 107: concatenate the spherical harmonic feature vector and the anti-aliasing feature, input them into a target multi-layer perceptron, and output the predicted color of each sampling point.

[0118] Step 108: Perform volume rendering on the predicted color and density of each sampling point to obtain a rendered image.

[0119] Among them, after using the color and density of the sampling points on each of the above rays, the present application performs the following integration along the rays to obtain the color of the pixel points corresponding to the rays, that is, to obtain the rendered image. The specific formula is as follows:

[0120]

[0121]

[0122] Among them, σ i represents the opacity of the i-th sample, c i represents the predicted color of the i-th sampling point, and δ j Represents the distance to the next sampling point; is the pixel color; i, j are the serial numbers of the sampling points, N is the number of sampling points, T i is the weight; c i The color of the sampling point.

[0123] In some embodiments, after step 108, it also includes: comparing the loss between the rendered image and the real image; if the loss is greater than a preset threshold, returning to the step of "decomposing the color feature space and the density feature space, and determining the matrix and vector pairs corresponding to the spatial coordinates of each sampling point" until the loss is less than or equal to the preset threshold.

[0124] The loss function loss represents the mean square error of the image. Based on the loss function loss, the loss between the rendered image and the real image is determined. The present application applies the total variation (TV) loss to the decomposed tensor to promote smoothness. The expression of the loss function loss is:

[0125]

[0126] L tv =∑|T i -T i-1 |.

[0127] Among them, I render For the rendered image, I gt is the real image, λ is the control L tv The hyperparameter of influence, L tv is the total variational loss, which is calculated by multiplying the adjacent tensors T i and T i-1 The smoothness of the tensor is measured by summing the absolute values ​​of the differences between .

[0128] Step 109: Determine a reconstructed three-dimensional scene according to the rendered image.

[0129] See also Figure 5 and Figure 6 , the present application consists of a ray marching module, a tensor decomposition module, a feature encoding module (spherical harmonic encoding module and view cone encoding module for surface modeling) and a volume rendering module. Given several camera poses and a camera intrinsic parameter, the ray marching module simulates the light emitted from the camera optical center along the direction of the pixel point into the three-dimensional scene, and then samples dozens to hundreds of sampling points on each ray. After obtaining multiple sampling points, the tensor decomposition module interpolates the color feature vector and density according to the spatial coordinates of the sampling points, and then the spherical harmonic encoding module based on surface modeling and the view cone encoding (view cone uniform encoding) module process the sampling points in parallel: the spatial coordinates of the sampling points, the light direction vector and the color feature vector are input into the spherical harmonic encoding module based on surface modeling to obtain the spherical harmonic encoding feature vector; after the spatial coordinates of the sampling points and the camera optical center position are input into the view cone encoding module, the uniform distribution of the view cone corresponding to the sampling points is obtained by processing, and then the anti-aliasing features of the sampling points are obtained by high-frequency encoding. The anti-aliasing features and the spherical harmonic encoding features are spliced ​​and input into a deeper MLP (target multi-layer perceptron) to obtain the predicted color of the sampling points. The volume rendering module integrates the density and color of the sampling points on each ray to obtain the color of the pixel corresponding to each ray and completes the rendering of the image.

[0130] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that: The three-dimensional scene reconstruction method comprises: Simulating the process of light from the optical center of the camera to the three-dimensional scene by a light marching module, and determining the spatial coordinates of multiple sampling points in the three-dimensional scene; the three-dimensional scene is a three-dimensional voxel grid space composed of voxel grids; Converting the three-dimensional voxel grid space into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space; Decomposing the color feature space and the density feature space to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point; the matrix and vector pair include a color matrix and vector pair, and a density matrix and vector pair; Interpolating each of the matrix and vector pairs to determine the color feature vector and density of each sampling point; Determine the spherical harmonic eigenvector of each sampling point according to the color eigenvector of each sampling point and the spherical harmonic coding of the surface modeling; Determine the anti-aliasing feature of each sampling point according to the color feature vector and the cone encoding of each sampling point; splicing the spherical harmonic feature vector and the anti-aliasing feature, inputting the result into a target multi-layer perceptron, and outputting the predicted color of each sampling point; Performing volume rendering on the predicted color and the density of each sampling point to obtain a rendered image; A reconstructed three-dimensional scene is determined according to the rendered image.

2. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that: The process of simulating the light from the optical center of the camera to the three-dimensional scene by the light marching module is used to determine the spatial coordinates of multiple sampling points in the three-dimensional scene, specifically including: Based on the ray marching module, the camera position and camera intrinsic parameters are used to simulate the process of light traveling from the camera optical center to the three-dimensional scene. In the process of the light from the optical center of the camera to the three-dimensional scene, the spatial coordinates of multiple sampling points in the three-dimensional scene are determined according to the propagation path of the light and the sampling strategy.

3. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that: Interpolating each of the matrix and vector pairs to determine the color feature vector and density of each sampling point specifically includes: For the color feature space, interpolate the color matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each first interpolation result; Adding the first interpolation results to determine the color feature vector of each sampling point; For the density feature space, interpolate the density matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each second interpolation result; The second interpolation results are added together to determine the density of each sampling point.

4. The method for reconstructing a three-dimensional scene according to claim 3, characterized in that: For the color feature space, interpolating the color matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each first interpolation result specifically includes: For any set of color matrices and vectors in the color matrix and vector pair, bilinear interpolation is performed on the matrices in the color matrix and vector to obtain a matrix result, and linear interpolation is performed on the vectors in the color matrix and vector to obtain a vector result; the color matrix and vector pair includes three sets of color matrices and vectors, each set of color matrices and vectors uses a different coordinate plane as a matrix, and the remaining coordinate axes are used as vectors; The product of the first matrix result and the first vector result is used as the first interpolation result.

5. The method for reconstructing a three-dimensional scene according to claim 3, characterized in that: For the density feature space, interpolating the density matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each second interpolation result specifically includes: For any set of density matrices and vectors in the density matrix and vector pairs, bilinear interpolation is performed on the matrix in the density matrix and vector to obtain a second matrix result, and linear interpolation is performed on the vector in the density matrix and vector to obtain a second vector result; the density matrix and vector include three sets of density matrix and vector pairs, each set of density matrix and vector uses a different coordinate plane as a matrix and the remaining coordinate axes as vectors; The product of the second matrix result and the second vector result is used as the second interpolation result.

6. The method for reconstructing a three-dimensional scene according to claim 2, characterized in that: According to the color feature vector of each sampling point and the spherical harmonic coding of surface modeling, the spherical harmonic feature vector of each sampling point is determined, specifically including: For any sampling point, the spatial coordinates of the sampling point, as well as the color feature vector and density of the sampling point, are input into a first multi-layer perceptron to determine the surface roughness and surface normal vector of the sampling point; wherein the surface roughness is a concentrated parameter of the von Mises-Fisher distribution of the surface normal vector; Calculating the reflection direction of the sampling point using the surface normal vector and the observation direction; the observation direction is determined based on the camera pose; Encoding the reflection direction of the sampling point by using a spherical harmonic function, and determining an expected value of the spherical harmonic function encoding of the reflection direction; The expected value of the spherical harmonic function encoding the reflection direction of each sampling point is used as the spherical harmonic eigenvector of each sampling point.

7. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that: According to the color feature vector and cone encoding of each sampling point, the anti-aliasing feature of each sampling point is determined, including: Get a frustum projected from the camera's optical center along the direction of the pixel point in the three-dimensional scene; According to the spatial coordinates of each sampling point, the viewing cone is intercepted to determine the intercepted viewing cone; Calculating the uniform distribution of the color feature vector of each sampling point in the truncated viewing cone; The variance and expectation of the uniform distribution are encoded at high frequency to obtain the anti-aliasing feature of each sampling point.

8. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that: After performing volume rendering on the predicted color and the density of each sampling point to obtain a rendered image, the method further includes: comparing the loss between the rendered image and the real image; If the loss is greater than a preset threshold, return to step "decomposing the color feature space and the density feature space, and determining the matrix and vector pair corresponding to the spatial coordinates of each sampling point" until the loss is less than or equal to the preset threshold.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene reconstruction method described in any one of claims 1 to 8.

Images