A method and apparatus for reconstructing a three-dimensional scene

Abstract

The application discloses a reconstruction method and device of a three-dimensional scene, and relates to the field of computer graphics. The method simulates the process of light rays from the camera optical center to the three-dimensional scene through a light ray travel module to determine the spatial coordinates of a plurality of sampling points in the three-dimensional scene; decomposes a color feature space and a density feature space to determine a matrix and a vector pair corresponding to the spatial coordinates of each sampling point; interpolates each matrix and vector pair to determine the color feature vector and the density of each sampling point; determines the spherical harmonic feature vector of each sampling point according to the color feature vector of each sampling point and the spherical harmonic coding of surface modeling; determines the anti-aliasing feature of each sampling point according to the color feature vector of each sampling point and the view cone coding; splices the spherical harmonic feature vector and the anti-aliasing feature to input the predicted color of each sampling point output by a target multi-layer perception; and performs volume rendering on the predicted color and the density to determine the reconstructed three-dimensional scene, thereby efficiently and high-quality reconstructing the three-dimensional scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer graphics, in particular to a three-dimensional scene reconstruction method and device. BACKGROUND

[0002] In recent years, with the development of computer graphics and machine learning technology, Neural Radiance Fields (NeRF) as a new three-dimensional scene reconstruction technology has been widely concerned and applied in virtual reality, augmented reality and computer vision fields. On the one hand, the traditional NeRF method usually relies on a complex optimization process and a large number of training images to recover the geometric structure and appearance attributes of the scene. Although this method can produce a more realistic rendering effect, it is often accompanied by high computational cost and large memory occupation. On the other hand, when dealing with high-frequency textures and details, aliasing phenomenon is easy to occur, resulting in low image rendering quality and rendering efficiency, and further resulting in low three-dimensional scene reconstruction quality and reconstruction efficiency.

[0003] Therefore, developing a method that can effectively reduce memory occupation, improve rendering quality and rendering efficiency, and further improve three-dimensional scene reconstruction quality and reconstruction efficiency, has become an important direction of current research. SUMMARY

[0004] The purpose of the present application is to provide a three-dimensional scene reconstruction method and device, which can solve the problem of low three-dimensional scene reconstruction quality and reconstruction efficiency in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a three-dimensional scene reconstruction method, comprising:

[0007] Simulating the process of light rays from the camera optical center to the three-dimensional scene by a light ray marching module to determine the spatial coordinates of a plurality of sampling points in the three-dimensional scene; the three-dimensional scene is a three-dimensional voxel grid space composed of a voxel grid; the light ray marching module is used to simulate the process of light rays from the camera optical center to the three-dimensional scene by using camera pose and camera intrinsic parameters;

[0008] Converting the three-dimensional voxel grid space into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space;

[0009] Decomposing the color feature space and the density feature space to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point; the matrix and vector pair includes a color matrix and vector pair, and a density matrix and vector pair;

[0010] interpolating each matrix and vector pair to determine a color feature vector and a density of each sampling point;

[0011] determining a spherical harmonic feature vector of each sampling point according to the color feature vector of each sampling point and a spherical harmonic encoding of surface modeling;

[0012] determining an anti-aliasing feature of each sampling point according to the color feature vector of each sampling point and a view frustum encoding;

[0013] concatenating the spherical harmonic feature vector and the anti-aliasing feature and inputting a target multi-layer perceptron to output a predicted color of each sampling point;

[0014] performing volume rendering on the predicted color and the density of each sampling point to obtain a rendered image;

[0015] determining a reconstructed three-dimensional scene according to the rendered image.

[0016] In a second aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene reconstruction method described in the above.

[0017] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0018] The present application provides a three-dimensional scene reconstruction method and device. Firstly, a light ray marching module is used to simulate the process of light rays from the camera optical center to the three-dimensional scene to determine the spatial coordinates of a plurality of sampling points in the three-dimensional scene. Then, the color feature space and the density feature space of the three-dimensional voxel grid space are decomposed to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point, thereby reducing the memory consumption and optimizing the memory. Then, the color feature space and the density feature space are decomposed to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point, and each matrix and vector pair is interpolated to determine the color feature vector and the density of each sampling point. Further, the spherical harmonic feature vector and the anti-aliasing feature are obtained using the spherical harmonic encoding and the view frustum encoding of surface modeling according to the obtained color feature vector of each sampling point. In this way, not only the high quality of the rendered image is ensured, but also the anti-aliasing ability of the image in the rendering process is improved. Finally, the rendering efficiency and the image quality are improved by optimizing the memory management, the spherical harmonic encoding and the view frustum encoding of surface modeling. Therefore, based on the high-quality rendered image, the reconstructed three-dimensional scene is obtained efficiently and with high quality. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0020] Figure 1 A flowchart of a three-dimensional scene reconstruction method provided in an embodiment of the present application;

[0021] Figure 2 A tensor decomposition diagram provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a light ray marching module and a tensor decomposition module provided in an embodiment of the present application;

[0023] Figure 4 A view frustum encoding diagram provided in an embodiment of the present application;

[0024] Figure 5 A general architecture diagram of a three-dimensional scene reconstruction method provided in an embodiment of the present application;

[0025] Figure 6 A structural schematic diagram of a three-dimensional scene reconstruction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0028] As shown in Figure 1 The present application provides a three-dimensional scene reconstruction method, comprising:

[0029] Step 101: determining the spatial coordinates of a plurality of sampling points in the three-dimensional scene by a light ray marching module simulating the process of light rays from the camera optical center to the three-dimensional scene; the three-dimensional scene is a three-dimensional voxel grid space composed of a voxel grid; the light ray marching module is used to simulate the process of light rays from the camera optical center to the three-dimensional scene by using camera pose and camera intrinsic parameters.

[0030] In some embodiments, step 101 specifically comprises: based on the ray marching module, simulating the process of light rays from the camera optical center to the three-dimensional scene by using the camera pose and the camera intrinsic parameters; and determining the spatial coordinates of the plurality of sampling points in the three-dimensional scene according to the propagation path and the sampling strategy of the light rays in the process of the light rays from the camera optical center to the three-dimensional scene.

[0031] wherein the ray marching module is a technique used in computer graphics for rendering complex scenes.

[0032] wherein the specific process of obtaining the plurality of sampling points according to the propagation path and the sampling strategy of the light rays is as follows:

[0033] The camera pose and the camera intrinsic parameters are input into the ray marching module, and a plurality of light ray direction vectors emitted through the camera optical center to the three-dimensional scene are calculated according to these parameters.

[0034] Specifically, for each pixel point in the image, first, the coordinates (u, v) of the pixel point on the image plane are determined, and then the pixel coordinates are converted into normalized coordinates to [-1, 1] by using the camera intrinsic parameters; then, the light ray vector is transformed into the world coordinate system according to the camera pose matrix to obtain the direction vector d of the light ray; and finally, the position vector o of the camera optical center is combined as the starting point of the light ray, and the starting point and direction of each light ray, i.e. the light ray direction vector o+d, are obtained. Finally, sampling is performed on the light ray direction vector to obtain the spatial coordinates of the plurality of sampling points, and the spatial coordinates of the plurality of sampling points are taken as the training data points.

[0035] It is worth noting that in the early stage of training, for example, when the number of iterations is less than 8000, an occupancy mask is calculated for the voxel grid every 1000 iterations to indicate which grids are occupied, and then the range of the bounding box is reduced and the number of voxels is increased according to the mask, so that the resolution of the voxels is increased, and the value of the new voxels is determined by interpolation of the voxels with lower resolution. With the determination of the new resolution, the marching of the light rays will be performed in the bounding box, and the sampling step will be updated accordingly. Based on the new resolution and step ratio, the number of sampling points for each light ray is calculated.

[0036] For example, the camera intrinsic parameter matrix K is as follows:

[0037]

[0038] wherein f x and f y are the focal lengths of the camera along the x and y axes, c x and c y are the offsets of the image center relative to the pixel coordinate origin.

[0039] The camera pose matrix contains a rotation matrix R and a translation vector t, which is used to convert a point in the world coordinate system to the camera coordinate system, and the general form is:

[0040]

[0041] wherein T is a rotation translation matrix.

[0042] Suppose the coordinates of the light direction corresponding to a pixel point in the image are to be calculated, which first needs to be normalized to [-1, 1], and for the case of no distortion, this step can be completed by the following way:

[0043]

[0044]

[0045] wherein (u, v) is the coordinate of the pixel point, u' is the X component of the light vector, and v' is the Y component of the light vector.

[0046] That is, the distance from this pixel point to the center of the image is calculated, and then divided by the focal length to obtain the normalized coordinates as the XY components of the light, and the Z component is set to -1, that is, the light is emitted from the camera to the scene. Through this step, the direction vector r of the light in the camera coordinate system is obtained c

[0047] r c =o c +d c .

[0048] wherein o c is (0, 0, 0, 1), and d c is (u, v, -1, 0).

[0049] Further, left-multiplying r c by P, the direction vector r w of the light in the world coordinate system is obtained.

[0050] r w =o w +d w .

[0051] Step 102: converting the three-dimensional voxel grid space into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space.

[0052] Step 103: decomposing the color feature space and the density feature space to determine the matrix and vector pair corresponding to the spatial coordinates of each sampling point; the matrix and vector pair includes a color matrix and vector pair, and a density matrix and vector pair.

[0053] where step 103 is performed in a tensor decomposition module, which performs interpolation operations in the matrix and vector pairs of the tensor representation to obtain the color feature vector and density according to the spatial coordinates of the sampling points. This helps to significantly reduce the required graphics processing unit memory consumption. Specifically, to efficiently represent and process the three-dimensional voxel grid space, the four-dimensional feature space is decomposed into a series of matrix-vector pairs, thereby reducing the memory footprint.

[0054] As Figure 2 , suppose there is a 300 3 three-dimensional voxel grid space, each voxel contains a feature vector of length 8. Storing the entire voxel space requires 300 3 × 8 × 4B = 824MB of memory. However, using the above method to store the decomposed 16 matrix-vector pairs, only (300 2 + 300) × 3 × 8 × 4B × 16 = 149MB of memory space is required, where Sparse Voxel Storage is a technique for efficiently storing and processing three-dimensional voxel data. In the fields of three-dimensional reconstruction, computer graphics, virtual reality and augmented reality, sparse voxel storage technology can significantly reduce memory footprint and improve computing efficiency.

[0055] Specifically, a three-dimensional voxel grid space is usually represented as a 4D tensor τ, where (X, Y, Z) is the spatial coordinate of the sampling point, and F is the feature of these sampling points. Consider decomposing the four-dimensional feature space (4D tensor) into a combination of planes and vectors. That is, decompose the 4D tensor into a combination of 1D tensors (vectors) and 2D tensors (matrices), represented as (v x ,v y ,v z ) and (M YZ ,M XZ ,M XY ), respectively, see Figure 2 .

[0056]

[0057] where τ refers to the three-dimensional voxel grid space, is the decomposed vector, is the decomposed matrix, and , for example, contains and correspond to the decomposed tensors of density and color, respectively, denotes the outer product, and R is the number of decomposed matrix-vector pairs.

[0058] Step 104: interpolating each of the matrix and vector pairs to determine the color feature vector and the density of each sample point.

[0059] In some embodiments, step 104 specifically comprises steps 201-204:

[0060] Step 201: interpolating, for the color feature space, the color matrix and vector pair corresponding to the spatial coordinates of each sample point to determine first interpolation results.

[0061] Step 202: summing each of the first interpolation results to determine the color feature vector of each sample point.

[0062] Step 203: interpolating, for the density feature space, the density matrix and vector pair corresponding to the spatial coordinates of each sample point to determine second interpolation results.

[0063] Step 204: summing each of the second interpolation results to determine the density of each sample point.

[0064] In some embodiments, step 201 specifically comprises steps 301-304:

[0065] Step 301: bilinearly interpolating the matrix in the color matrix and vector pair to obtain a first matrix result and linearly interpolating the vector in the color matrix and vector pair to obtain a first vector result; the color matrix and vector pair comprises three color matrix and vector pairs, each of which uses a different coordinate plane as the matrix and the remaining coordinate axis as the vector.

[0066] Step 302: taking the product of the first matrix result and the first vector result as the first interpolation result.

[0067] In some embodiments, step 203 specifically comprises steps 401-404:

[0068] Step 401: bilinearly interpolating the matrix in the density matrix and vector pair to obtain a second matrix result and linearly interpolating the vector in the density matrix and vector pair to obtain a second vector result; the density matrix and vector pair comprises three density matrix and vector pairs, each of which uses a different coordinate plane as the matrix and the remaining coordinate axis as the vector.

[0069] Step 402: taking the product of the second matrix result and the second vector result as the second interpolation result.

[0070] As Figure 3As shown, the feature of the sampling point can be obtained by interpolation at each sampling point. For color information, linear interpolation is performed on each pair of matrix and vector to obtain a set of color features, the color features obtained by interpolating each pair of matrix and vector are multiplied, and then spliced to obtain a color feature vector. For density information, interpolation is also performed, and the density features are summed to obtain the density value through a sigmoid function. For example, assuming is a two-dimensional tensor, is a one-dimensional tensor, where each element is a feature vector, the projection coordinates of the sampling point (x, y, z) on the YZ plane are (y, z), and the projection coordinates on the X axis are x, interpolation is performed as follows:

[0071]

[0072]

[0073] wherein, is the feature of the sampling point on the YZ plane corresponding to the tensor, is the feature on the X axis corresponding to the tensor; interp and interp2 represent linear and bilinear interpolation, respectively.

[0074] For and similar operations are also performed to obtain the feature on the Y axis corresponding to the tensor, the feature on the XZ plane corresponding to the tensor, the feature on the Z axis corresponding to the tensor, and the feature on the XZ plane corresponding to the tensor. They are respectively dot multiplied, and then summed to obtain the color feature vector f a of the sampling point.

[0075]

[0076] wherein, the method for obtaining the density is basically the same as that of the color feature vector. After interpolation, the following is obtained the feature on the YZ plane corresponding to the tensor, the feature on the X axis corresponding to the tensor, the feature on the Y axis corresponding to the tensor, the feature on the XZ plane corresponding to the tensor, the feature on the Z axis corresponding to the tensor, and the feature on the XY plane corresponding to the tensor.

[0077]

[0078] wherein R is the total number of decomposed vector pairs, and d is the density of the sampling points.

[0079] Step 105: determining the spherical harmonic feature vector of each sampling point according to the color feature vector of each sampling point and the spherical harmonic coding of surface modeling.

[0080] In some embodiments, step 105 specifically comprises steps 501-504:

[0081] Step 501: inputting the spatial coordinates of any sampling point, and the color feature vector and density of the sampling point into a first multi-layer perception, to determine the surface roughness and surface normal vector of the sampling point; wherein the surface roughness is the concentration parameter of the von Mises-Fisher distribution of the surface normal vector.

[0082] wherein the tensor decomposition module comprises a first multi-layer perception, the input of the first multi-layer perception is the spatial coordinates of the sampling point, and the output is the density and color feature vector of the sampling point, and the color feature vector is simultaneously input into the view frustum coding module and the spherical harmonic coding based on surface modeling.

[0083] Step 502: calculating the reflection direction of the sampling point using the surface normal vector of the sampling point and the observation direction; the observation direction is determined based on the camera pose.

[0084] Step 503: encoding the reflection direction of the sampling point using spherical harmonics to determine the expected value of the spherical harmonic coding of the reflection direction.

[0085] Step 504: taking the expected value of the spherical harmonic coding of the reflection direction of each sampling point as the spherical harmonic feature vector of each sampling point.

[0086] wherein the surface modeling and coding and the view frustum coding can be regarded as a feature coding module, and the surface modeling and coding and the view frustum coding mainly operate on the color feature vector. In this embodiment, it is considered that the scene surface is composed of a plurality of micro planes, each of which is a perfect plane. For a surface with high roughness, the directions of these micro planes are quite different, indicating that the variance of the micro normal vector is large. The appearance of the rough surface remains consistent regardless of the observation direction. On the contrary, for a surface with low roughness, the directions of the micro planes are roughly similar. The visual characteristics of the rough material remain relatively consistent regardless of the change in the observation direction, while the relatively smooth material exhibits significant changes when the observation direction changes.

[0087] The present application uses spherical harmonic coding to encode the reflection direction, and then calculates the expectation of its spherical harmonic coding under the von Mises-Fisher (vMF) distribution. First, the surface roughness and surface normal vector of the sampling point are predicted according to the first multi-layer perceptron, and then the reflection direction is calculated according to the surface normal vector and the light direction corresponding to the sampling point:

[0088]

[0089] wherein, is the predicted surface normal vector, w0 is the light direction, w r is the reflection direction.

[0090] After obtaining the reflection direction, the expectation of the spherical harmonic function coding of the reflection direction under the vMF distribution is calculated, and the specific formula is:

[0091]

[0092] M L =(I, m): I = 1,..., 2 L , m = 0,..., I.

[0093] wherein is the spherical harmonic function, l and m are the order and degree of the spherical harmonic function, is the reflection direction; κ is the surface roughness; m is the degree of the spherical harmonic function, M L is the degree matrix, is the spherical harmonic function coding function.

[0094] The expectation can be calculated by a simple closed-form expression, that is, by multiplying the spherical harmonic multiple basis functions by the attenuation coefficient of the vMF distribution, as follows:

[0095]

[0096] wherein, A L (κ) is the attenuation function.

[0097] Specifically, the attenuation function A L (κ) can be approximated as:

[0098]

[0099] wherein, κ is the surface roughness predicted by the MLP. Therefore:

[0100]

[0101] Step 106: Determine the anti-aliasing feature of each sampling point according to the color feature vector and the view frustum coding of each sampling point.

[0102] Where, the frustum encoding method is expected to be very similar to the orientation encoding used in NeRF. The specific steps are as follows: first, convert dxdydz into cylindrical coordinates rt 2 drdtdtheta, and derive the first moment (mean) and second moment (variance) of t and r, where E(x) is equal to 0.

[0103]

[0104] Where, f is the focal length, is the shortest distance from the point to the center line of the cone; V is the volume of the truncated frustum; t0 is the near clipping plane distance of the truncated frustum; t1 is the far clipping plane distance of the truncated frustum.

[0105] As Figure 4 , t represents the distance from the cone to the optical center, and the first moment and the second moment of r are given as follows:

[0106]

[0107]

[0108]

[0109] Var(t)=E(t 2 )-E 2 (t).

[0110] Where E(t) is the expectation of t, and Var(r) is the variance of r.

[0111] Due to symmetry, the mean of r is zero, and the variance of r is equal to the variance of the corresponding cone in the x or y direction.

[0112] The first moment and the second moment of t and u are encoded into frequencies.

[0113]

[0114] Where: P is the frequency matrix of position encoding; L is the number of frequencies, x is the feature vector; F(x) is the encoding function.

[0115] In some embodiments, step 106 specifically comprises: obtaining a frustum projected from the camera optical center along the direction of the pixel points in the three-dimensional scene; according to the spatial coordinates of each sampling point, the frustum is truncated to determine the truncated frustum; calculate the uniform distribution of the color feature vector of each sampling point in the truncated frustum; high-frequency encode the variance and expectation of the uniform distribution to obtain the anti-aliasing feature of each sampling point.

[0116] The application directly multiplies the desired E with the matrix P to obtain the encoded high-frequency mean feature F(E)=PE. For σ, since the covariance of variables has the property under linear transformation, i.e., the linear transformation of the covariance of variables is equal to the linear transformation of the covariance of variables, the encoded variance is F(σ 2 )=Pσ 2 P T . The F(E) and F(σ 2 ) are spliced to obtain the anti-aliasing feature.

[0117] Step 107: Splice the spherical harmonic feature vector and the anti-aliasing feature, and input the target multi-layer perception machine to output the predicted color of each sampling point.

[0118] Step 108: Perform volume rendering on the predicted color and the density of each sampling point to obtain a rendered image.

[0119] Wherein, after utilizing the color and the density of the sampling points on each light ray, the application performs the following integration along the light ray to obtain the color of the pixel point corresponding to the light ray, i.e., obtains the rendered image. The specific formula is as follows:

[0120]

[0121]

[0122] Wherein, σ i represents the opacity of the i-th sampling, c i represents the predicted color of the i-th sampling point, and δ j represents the distance to the next sampling point; is the pixel point color; i and j are the serial numbers of the sampling points, N is the number of sampling points, T i is the weight; c i is the color of the sampling point.

[0123] In some embodiments, after step 108, the method further comprises: comparing the loss between the rendered image and the real image; if the loss is greater than a preset threshold, returning to the step of “decomposing the color feature space and the density feature space, determining the matrix and the vector pair corresponding to the spatial coordinates of each sampling point”, until the loss is less than or equal to the preset threshold.

[0124] Wherein, the loss function loss represents the mean square error of the image. Based on the loss function loss, the loss between the rendered image and the real image is determined, and the application applies the total variation (TV) loss to the decomposed tensor to promote smoothness. The expression of the loss function loss is:

[0125]

[0126] L tv =∑|T i -T i-1 |.

[0127] where I render is the rendered image, I gt is the real image, λ is a hyper-parameter controlling the degree of L tv , L tv is the total variation loss, which measures the smoothness of tensors by summing the absolute value of the difference between adjacent tensors T i and T i-1 .

[0128] Step 109: determining a reconstructed three-dimensional scene according to the rendered image.

[0129] Referring to Figure 5 and Figure 6 , the present application consists of a ray marching module, a tensor decomposition module, a feature encoding module (a surface modeling based spherical harmonic encoding module and a view frustum encoding module) and a volume rendering module. Given a number of camera poses and an intrinsic parameter of the camera, the ray marching module simulates the emission of light rays from the camera optical center along the direction of the pixel points into the three-dimensional scene, and then samples tens to hundreds of sampling points on each light ray. After obtaining a plurality of sampling points, the tensor decomposition module interpolates the color feature vector and the density according to the spatial coordinates of the sampling points. Then, the surface modeling based spherical harmonic encoding module and the view frustum encoding (uniform encoding of the view frustum) module process the sampling points in parallel: the spatial coordinates of the sampling points and the light direction vector and the color feature vector are input into the surface modeling based spherical harmonic encoding module to obtain a spherical harmonic encoding feature vector; after the spatial coordinates of the sampling points and the position of the camera optical center are input into the view frustum encoding module, the uniform distribution of the view frustum corresponding to the sampling points is obtained, and then the anti-aliasing feature of the sampling points is obtained through high-frequency encoding. The anti-aliasing feature and the spherical harmonic encoding feature are spliced and input into a deeper MLP (target multi-layer perceptron) to obtain the predicted color of the sampling points. The volume rendering module integrates the density and color of the sampling points on each light ray to obtain the color of the pixel points corresponding to each light ray, and completes the rendering of the image.

[0130] In one exemplary embodiment, a computer device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above method.

[0131] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0132] The principles and implementations of the present application are described in detail with specific examples in this paper, and the above examples are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method of reconstruction of a three-dimensional scene, characterized in that, The method for reconstructing the three-dimensional scene includes: The process of light traveling from the camera's optical center to the 3D scene is simulated using a light travel module to determine the spatial coordinates of multiple sampling points in the 3D scene; the 3D scene is a 3D voxel grid space composed of voxel meshes. The three-dimensional voxel mesh space is transformed into a four-dimensional feature space; the four-dimensional feature space includes a color feature space and a density feature space. The color feature space and the density feature space are decomposed to determine the matrix and vector pairs corresponding to the spatial coordinates of each sampling point; the matrix and vector pairs include color matrix and vector pairs, and density matrix and vector pairs. Interpolate each of the matrix and vector pairs to determine the color feature vector and density of each sampling point; The spherical harmonic feature vector of each sampling point is determined based on the color feature vector of each sampling point and the spherical harmonic encoding of the surface modeling. Based on the color feature vector and view frustum encoding of each sampling point, the anti-aliasing feature of each sampling point is determined; The spherical harmonic feature vector and the anti-aliasing feature are concatenated and input into the target multilayer perceptron, which outputs the predicted color for each sampling point. Volume rendering is performed on the predicted color and density of each sampling point to obtain a rendered image; Based on the rendered image, the reconstructed 3D scene is determined.

2. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that, The process of light traveling from the camera's optical center to the 3D scene is simulated using a light-traveling module to determine the spatial coordinates of multiple sampling points in the 3D scene, specifically including: Based on the ray travel module, the process of light rays traveling from the camera's optical center to the 3D scene is simulated using the camera pose and camera intrinsic parameters. During the process of light rays traveling from the camera's optical center to the 3D scene, the spatial coordinates of multiple sampling points in the 3D scene are determined based on the light ray propagation path and sampling strategy.

3. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that, Interpolation is performed on each of the matrix and vector pairs to determine the color feature vector and density of each sampling point, specifically including: For the color feature space, interpolation is performed on the color matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each first interpolation result; The first interpolation results are summed to determine the color feature vector of each sampling point; For the density feature space, interpolation is performed on the density matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each second interpolation result; The density of each sampling point is determined by summing the results of the second interpolation.

4. The method for reconstructing a three-dimensional scene according to claim 3, characterized in that, For the color feature space, interpolation is performed on the color matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each first interpolation result, specifically including: For any set of color matrices and vectors in the color matrix and vector pair, bilinear interpolation is performed on the matrices in the color matrix and vector pair to obtain a first matrix result, and linear interpolation is performed on the vectors in the color matrix and vector pair to obtain a first vector result; the color matrix and vector pair includes three sets of color matrices and vectors, each set of color matrices and vectors uses a different coordinate plane as the matrix, and the remaining coordinate axes as the vectors; The product of the first matrix result and the first vector result is used as the first interpolation result.

5. The method for reconstructing a three-dimensional scene according to claim 3, characterized in that, For the density feature space, interpolation is performed on the density matrix and vector pair corresponding to the spatial coordinates of each sampling point to determine each second interpolation result, specifically including: For any pair of density matrices and vectors in the density matrix and vector pair, bilinear interpolation is performed on the matrices in the density matrix and vector pair to obtain a second matrix result, and linear interpolation is performed on the vectors in the density matrix and vector pair to obtain a second vector result; the density matrix and vector pair includes three pairs of density matrix and vector pairs, each pair of density matrix and vector pairs uses a different coordinate plane as the matrix, and the remaining coordinate axes as the vectors; The product of the second matrix result and the second vector result is used as the second interpolation result.

6. The method for reconstructing a three-dimensional scene according to claim 2, characterized in that, Based on the color feature vector of each sampling point and the spherical harmonic encoding of the surface model, the spherical harmonic feature vector of each sampling point is determined, specifically including: For any sampling point, the spatial coordinates of the sampling point, as well as the color feature vector and density of the sampling point, are input into the first multilayer perceptron to determine the surface roughness and surface normal vector of the sampling point; wherein, the surface roughness is the lumped parameter of the von Mises-Fisher distribution of the surface normal vector; The reflection direction of the sampling point is calculated using the surface normal vector and the viewing direction; the viewing direction is determined based on the camera pose. The reflection direction of the sampling point is encoded using a spherical harmonic function, and the expected value of the spherical harmonic function encoding of the reflection direction is determined. The expected value of the spherical harmonic function encoding the reflection direction of each sampling point is used as the spherical harmonic feature vector of each sampling point.

7. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that, Based on the color feature vector and view frustum encoding of each sampling point, the anti-aliasing features of each sampling point are determined, specifically including: Obtain a view frustum projected from the camera's optical center along the direction of pixels in the 3D scene; Based on the spatial coordinates of each sampling point, the view frustum is truncated to determine the truncated view frustum; Calculate the uniform distribution of the color feature vector of each sampling point in the truncated view frustum; The variance and expectation of the uniform distribution are encoded at high frequency to obtain the anti-aliasing feature of each sampling point.

8. The method for reconstructing a three-dimensional scene according to claim 1, characterized in that, After performing volume rendering on the predicted color and density of each sampling point to obtain the rendered image, the method further includes: Compare the loss between the rendered image and the real image; If the loss is greater than a preset threshold, return to the step "decompose the color feature space and the density feature space, and determine the matrix and vector pairs corresponding to the spatial coordinates of each sampling point" until the loss is less than or equal to the preset threshold.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method for reconstructing a three-dimensional scene according to any one of claims 1-8.

Images