Dynamic scene three-dimensional reconstruction method and system based on multi-scale space-time coding

Abstract

The invention provides a dynamic scene three-dimensional reconstruction method and system based on multi-scale space-time coding, and the method comprises the steps: enabling one-dimensional time information of a training video to be mapped to a high-dimensional space through frequency coding through a space-time coding module, and obtaining a frequency coding result; performing multi-scale and multi-resolution three-dimensional space feature coding on the training video, and splicing a three-dimensional space feature coding result and a frequency coding result to obtain space-time coding features; the space-time coding features are input into a multi-layer sensing module, color and density information of space coordinates in a three-dimensional space are output through multi-layer sensing processing, volume rendering is carried out, predicted colors of corresponding pixel points are obtained, and a three-dimensional reconstruction model is obtained according to mean square error training of the predicted colors and target colors; and inputting a video to be reconstructed into the three-dimensional reconstruction model to obtain the color and density of the space coordinates in the three-dimensional space. According to the invention, the spatial voxels are coded through the spatial coding technology, so that the model training and rendering time is shortened.

Description

Dynamic scene 3D reconstruction method and system based on multi-scale spatiotemporal coding Technical Field

[0001] The present invention relates to the technical field of image processing and three-dimensional reconstruction, and in particular to a method and system for three-dimensional reconstruction of dynamic scenes based on multi-scale spatiotemporal coding. Background Art

[0002] With the rise of 3D scene modeling technologies like NeRF, static scene modeling has become a well-established problem. However, static scene presentations cannot satisfy human visual needs. We live in a three-dimensional world, with both static and dynamic objects. Modeling dynamic characters in current 3D games and movie special effects requires significant time and labor costs. Furthermore, current computer-modeled characters lack realism and fail to provide a highly immersive visual experience. To address these issues, 3D reconstruction of real-world dynamic scenes is needed to provide a more realistic 3D video viewing experience.

[0003] Neural Radiance Field (NeRF) is an implicit representation of 3D scenes. A trained NeRF can generate voxel density values ​​at any coordinate in space, as well as color values ​​along the direction of a ray. Using this information, volume rendering techniques can be used to generate scene images from any viewing angle. For dynamic 3D scenes, simply adding time as a variable to the NeRF input yields a 3D scene representation at different times. Existing methods often use multi-layer perceptrons (MLPs) to model NeRF, but this approach is computationally intensive and inefficient, making it unsuitable for large-scale application deployment.

[0004] While using multi-layer MLPs to model implicit light fields can achieve relatively high accuracy, training time is very long, and rendering a single image can take tens of seconds or even longer. With current computing power, real-time rendering is not possible. To address this issue, a more efficient modeling approach is needed to shorten training and rendering times.

[0005] Invention Disclosure

[0006] To address the shortcomings of the existing technology, the present invention proposes a multi-scale spatial coding technology to encode spatial voxels to shorten model training and rendering time, and at the same time perform total variation regularization on the encoded features to eliminate artifacts in the space, thereby improving the model training efficiency and the quality of generated images.

[0007] Specifically, the present invention proposes a method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding, which is characterized by comprising:

[0008] Step 1: Obtain a training video shot in a specified scene and a real photo of the specified scene, and obtain a 3D reconstruction model including a spatiotemporal encoding module and a multi-layer perception module;

[0009] Step 2: Use the spatiotemporal coding module to map the one-dimensional time information of the training video to a high-dimensional space through frequency coding to obtain a frequency coding result; and use the spatiotemporal coding module to perform multi-scale and multi-resolution three-dimensional spatial feature coding on the training video, and splice the normalized three-dimensional spatial feature coding result with the frequency coding result to obtain a spatiotemporal coding feature;

[0010] Step 3: Input the spatiotemporal encoding features into the multi-layer perception module. After multi-layer perception processing, the color and density information of the spatial coordinates in the three-dimensional space are output respectively for volume rendering, and the RGB three-channel predicted color of the corresponding pixel point is obtained. The mean square error between the predicted color and the color of the real photo is used as the loss function to train the 3D reconstruction model.

[0011] Step 4: Input the video to be reconstructed into the trained 3D reconstruction model to obtain the color and density information of the spatial coordinates in the 3D space as the 3D scene reconstruction result.

[0012] The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding is characterized in that the spatiotemporal coding module of the 3D reconstruction model caches the frequency coding results and the 3D spatial feature coding results to the GPU during operation to accelerate the image rendering speed.

[0013] The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding is characterized in that the training of the 3D reconstruction model in step 3 includes subtracting the features of each adjacent voxel in the 3D space during the training phase and then summing them to obtain a total variation value L TV , L TV The value of is added to the loss function for optimization to suppress artifacts in three-dimensional space.

[0014] The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding is characterized in that the frequency coding process in step 2 includes: mapping the input time t to a higher dimensional space using a high-frequency function:

[0015] f(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 l-1 πt))

[0016] Where L is the coding level.

[0017] The present invention also proposes a dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding, which is characterized by comprising:

[0018] a 3D reconstruction module consisting of a spatiotemporal encoding module and a multi-layer perception module in series, and a 3D reconstruction module;

[0019] The spatiotemporal coding module is used to obtain a training video shot in a specified scene and a real photo of the specified scene, map the one-dimensional time information of the training video to a high-dimensional space through frequency coding, and obtain a frequency coding result; and perform multi-scale and multi-resolution three-dimensional spatial feature coding on the training video, and splice the normalized three-dimensional spatial feature coding result with the frequency coding result to obtain a spatiotemporal coding feature;

[0020] The multi-layer perception module is used to obtain the spatiotemporal coding features, and after multi-layer perception processing, output the color and density information of the spatial coordinates in the three-dimensional space for volume rendering, thereby obtaining the RGB three-channel predicted color of the corresponding pixel point. The mean square error between the predicted color and the color of the real photo is used as a loss function to train the 3D reconstruction model;

[0021] The 3D reconstruction module is used to input the video to be reconstructed into the trained 3D reconstruction model, and obtain the color and density information of the spatial coordinates in the 3D space as the 3D scene reconstruction result.

[0022] The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding is characterized in that the spatiotemporal coding module of the 3D reconstruction model caches the frequency coding results and the 3D spatial feature coding results to the GPU during operation to accelerate the image rendering speed.

[0023] The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding is characterized in that the 3D reconstruction module is used to perform a subtraction on the features of each adjacent voxel in the 3D space during the training phase, and then sum them up to obtain a total variation value L TV , L TV The value of is added to the loss function for optimization to suppress artifacts in three-dimensional space.

[0024] The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding is characterized in that the multi-layer perception module is used to map the input time t to a higher dimensional space using a high-frequency function:

[0025] f(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 l-1 πt))

[0026] Where L is the coding level.

[0027] The present invention also proposes a storage medium for storing a program for executing any one of the methods for three-dimensional reconstruction of dynamic scenes based on multi-scale spatiotemporal coding.

[0028] The present invention also proposes a client for use in any one of the dynamic scene three-dimensional reconstruction systems based on multi-scale spatiotemporal coding.

[0029] From the above scheme, it can be seen that the advantages of the present invention are:

[0030] 1. With the same computing resources, the training efficiency of the proposed model is about 100 times higher than that of the original NeRF in the existing technology.

[0031] 2. The model of the present invention can support real-time rendering, while the original NeRF takes 3-5 seconds to render an image;

[0032] 3. The peak signal-to-noise ratio (PSNR) of our model is 2dB higher than the original NeRF method. PSNR is the logarithm of the mean square error between the original and processed images relative to (2^n-1)^2 (the square of the maximum signal value, where n is the number of bits per sample), and is measured in dB.

[0033] BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is an overall framework diagram of the three-dimensional reconstruction model of the present invention.

[0035] Best Mode for Carrying Out the Invention

[0036] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0037] Currently, dynamic 3D scene reconstruction based on NeRF generally encodes time and inputs it into the neural network together with the camera pose to obtain the density and color information of each voxel in the 3D space under the corresponding viewing angle. This method requires a multi-layer, multi-node multi-layer perceptron (MLP), which is very slow in training and inference. To overcome this problem, the present invention proposes a new multi-scale dynamic 3D scene encoding method, which reduces the number of MLP layers and greatly increases the training and inference speed of the model. Figure 1 is the overall framework diagram of the model. Below I will give a detailed introduction to each module of the method:

[0038] First, the spatiotemporal encoding module. Unlike directly inputting a time series of video frames, we frequency-encode the one-dimensional time information and map it to a higher-dimensional space, thereby increasing the model's ability to express time. Using a high-frequency function to map the input time t to a higher-dimensional space can better fit data with high-frequency variations. The specific function form is:

[0039] f(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 l-1 πt))

[0040] Where L is the coding level.

[0041] For the three-dimensional scene to be reconstructed, we first determine the spatial range of the scene, which is generally a rectangular parallelepiped (length, width and height). The center of the space is set to (0, 0, 0), and other positions are represented by a three-dimensional Cartesian coordinate system (x, y, z). For the encoding of three-dimensional space, multi-scale and multi-resolution feature encoding is adopted. For the features of the (x, y, z) position in space, the features at each scale and resolution are subjected to three-dimensional linear interpolation. The obtained features are spliced ​​with the features at other scales, and then spliced ​​with the time coding features to obtain the final feature representation of spatiotemporal coding. Each voxel in the three-dimensional volumetric video has up to 4 degrees of freedom (x, y, z, t). The present invention adopts the above-mentioned spatiotemporal coding method to improve the expression ability of the features after spatiotemporal coding. Since multi-scale spatial coding improves the expression ability of the model, the number of MLP layers here will be greatly reduced compared to the original NeRF. After 3 layers of MLP, the color and density information of the corresponding spatial coordinates are output respectively.

[0042] During model training, we perform volume rendering on the color and density information of the space under the corresponding viewing angle to obtain the predicted RGB color of the corresponding pixel. The mean squared error between the predicted color and the real photo is then used as the loss function for training. Since the spatiotemporal encoding module and the multi-layer perception module we model are both differentiable, we can use gradient descent to train the network. During inference, we can cache the spatiotemporal encoding feature information on the GPU to accelerate image rendering.

[0043] Total variation regularization (TV Total Variation Regularization): The total variation model is an anisotropic model that relies on gradient descent flow to smooth the image. It is hoped that the image can be smoothed as much as possible within the image (the difference between adjacent pixels is small). In this invention, the eigenvalues ​​of each adjacent voxel in the three-dimensional space in the training phase are subtracted and then summed to obtain the total variation value LTV , L TV The value of is added to the loss function for optimization, which can effectively suppress the appearance of artifacts in the space.

[0044] The loss function of the model is: L total =L c +γL TV . Where γ is the weight of total variation loss. Because L TV and L c They are all differentiable, so the gradient descent technique is used for back propagation to optimize and remove noise in the space.

[0045] Industrial Applicability

[0046] The present invention proposes a method and system for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding, comprising: obtaining a 3D reconstruction model including a spatiotemporal coding module and a multi-layer perception module; using the spatiotemporal coding module to map the one-dimensional temporal information of a training video to a high-dimensional space through frequency coding to obtain a frequency coding result; and using the spatiotemporal coding module to perform multi-scale and multi-resolution 3D spatial feature encoding on the training video, and concatenating the normalized 3D spatial feature encoding result with the frequency coding result to obtain spatiotemporal coding features; inputting the spatiotemporal coding features into the multi-layer perception module, and after multi-layer perception processing, outputting the color and density information of the spatial coordinates in the 3D space for volume rendering, obtaining the RGB three-channel predicted color of the corresponding pixel point, and training the 3D reconstruction model based on the mean square error between the predicted color and the color of the real photo as a loss function; and inputting the video to be reconstructed into the trained 3D reconstruction model to obtain the color and density information of the spatial coordinates in the 3D space as the 3D scene reconstruction result. The present invention uses the above-mentioned spatial coding technology to encode spatial voxels to shorten model training and rendering time.

Claims

1. A method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding, characterized in that: include: Step 1: Obtain a training video shot in a specified scene and a real photo of the specified scene, and obtain a 3D reconstruction model including a spatiotemporal encoding module and a multi-layer perception module; Step 2: Use the spatiotemporal coding module to map the one-dimensional time information of the training video to a high-dimensional space through frequency coding to obtain a frequency coding result; and use the spatiotemporal coding module to perform multi-scale and multi-resolution three-dimensional spatial feature coding on the training video, and splice the normalized three-dimensional spatial feature coding result with the frequency coding result to obtain a spatiotemporal coding feature; Step 3: Input the spatiotemporal encoding features into the multi-layer perception module. After multi-layer perception processing, the color and density information of the spatial coordinates in the three-dimensional space are output respectively for volume rendering, and the RGB three-channel predicted color of the corresponding pixel point is obtained. The mean square error between the predicted color and the color of the real photo is used as the loss function to train the 3D reconstruction model. Step 4: Input the video to be reconstructed into the trained 3D reconstruction model to obtain the color and density information of the spatial coordinates in the 3D space as the 3D scene reconstruction result.

2. The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding according to claim 1, wherein: The spatiotemporal coding module of the 3D reconstruction model caches the frequency coding results and 3D spatial feature coding results to the GPU during runtime to accelerate image rendering.

3. The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding according to claim 1, wherein: The training of the 3D reconstruction model in step 3 includes subtracting the features of each adjacent voxel in the 3D space during the training phase and then summing them to obtain the total variation value L. TV , L TV The value of is added to the loss function for optimization to suppress artifacts in three-dimensional space.

4. The method for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding according to claim 1, wherein: The frequency encoding process in step 2 includes mapping the input time t to a higher dimensional space using a high frequency function: f(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 l-1 πt)) Where L is the coding level.

5. A dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding, characterized in that: include: a 3D reconstruction module consisting of a spatiotemporal encoding module and a multi-layer perception module in series, and a 3D reconstruction module; The spatiotemporal coding module is used to obtain a training video shot in a specified scene and a real photo of the specified scene, map the one-dimensional time information of the training video to a high-dimensional space through frequency coding, and obtain a frequency coding result; and perform multi-scale and multi-resolution three-dimensional spatial feature coding on the training video, and splice the normalized three-dimensional spatial feature coding result with the frequency coding result to obtain a spatiotemporal coding feature; The multi-layer perception module is used to obtain the spatiotemporal coding features, and after multi-layer perception processing, output the color and density information of the spatial coordinates in the three-dimensional space for volume rendering, thereby obtaining the RGB three-channel predicted color of the corresponding pixel point. The mean square error between the predicted color and the color of the real photo is used as a loss function to train the 3D reconstruction model; The 3D reconstruction module is used to input the video to be reconstructed into the trained 3D reconstruction model, and obtain the color and density information of the spatial coordinates in the 3D space as the 3D scene reconstruction result.

6. The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding according to claim 5, characterized in that: The spatiotemporal coding module of the 3D reconstruction model caches the frequency coding results and 3D spatial feature coding results to the GPU during runtime to accelerate image rendering.

7. The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding according to claim 5, characterized in that: The 3D reconstruction module is used to subtract the features of each adjacent voxel in the 3D space during the training phase and then sum them up to obtain the total variation value L TV , L TV The value of is added to the loss function for optimization to suppress artifacts in three-dimensional space.

8. The dynamic scene 3D reconstruction system based on multi-scale spatiotemporal coding according to claim 5, characterized in that: The multi-layer perception module is used to map the input time t to a higher dimensional space using a high-frequency function: f(t)=(sin(2 0 πt),cos(2 0 πt),…,sin(2 L-1 πt),cos(2 l-1 πt)) Where L is the coding level.

9. A storage medium for storing a program for executing any one of the methods for 3D reconstruction of dynamic scenes based on multi-scale spatiotemporal coding as claimed in claims 1 to 4.

10. A client, used in any one of the dynamic scene three-dimensional reconstruction systems based on multi-scale spatiotemporal coding according to claims 5 to 8.

Images