Efficient smoke reconstruction framework based on nerve transmission field and application

Abstract

The invention provides an efficient smoke reconstruction framework based on a nerve transmission field and application, and belongs to the technical field of computer graphics. The method comprises the following steps: obtaining sampling points in a three-dimensional space by using the pose of a camera; inputting the three-dimensional coordinates of the sampling points, the view angle direction vector and the time information into a voxel neural network to obtain implicit neural representation of smoke; and obtaining the smoke appearance and density with high space-time consistency through volume rendering. And then, inputting the rendering image and the rendering depth into a convolutional neural network to obtain a high-fidelity smoke appearance with rich details. And finally, through loop optimization, taking the output of the convolutional neural network as the input of the voxel neural network, and obtaining better smoke appearance and density. Compared with other methods, the frame can realize smog scene reconstruction results which are shorter in training time and higher in precision in smog scenes shot by a sparse multi-view camera and a single-view camera.

Description

Technical Field

[0001] The invention belongs to the technical field of computer graphics, and in particular relates to a framework and application for reconstructing smoke scenes. Background Art

[0002] Understanding fluids is important for weather forecasting, mechanical design and manufacturing, and various visual effects. Fluid systems have more complex dynamic shapes and motion patterns than solid or soft bodies. Reconstructing fluid systems from observations is a challenging task.

[0003] Traditional fluid simulation mainly involves solving partial differential equations (PDEs). Physics-based deep learning techniques use neural networks to learn through a problem-specific learning process. There are related methods to solve challenging physical problems. They integrate discretized numerical solvers into the training process and then obtain physically reasonable solutions for different dynamical systems. Previous models used several multi-layer perceptron (MLP) hidden layers to solve PDEs. Automatic differentiation-based training strategies are used to enhance the representation power of MLP. Other works focus on more efficient activity functions to estimate accurate derivatives. PINN combines physical constraints and observation data with MLP to achieve continuous prediction. Recent PINN-based methods add more physical parameters as input to obtain solutions to dynamical systems.

[0004] Fluid simulation is a long-studied task that focuses on simulating real-world smoke or liquids in virtual space. The simulation and rendering processes are always separate and consume a lot of computational resources. Reconstructing physically correct fluids from RGB images addresses these limitations and has the potential to extend fluid systems to real-world spaces. Previous studies reconstructed temporally coherent velocity fields from captured low-resolution images through physical constraints. Flow-based methods jointly reconstruct velocity and density fields from sparse tomography or linear image formation. Global transport methods use generative adversarial techniques and differentiable rendering to improve the quality of smoke reconstruction. Gas phenomenon methods train generative models from a set of orthogonal views.

[0005] Recent methods add physical priors learned from RGB images to Neural Radiance Fields (NeRF) to render smoke in new perspectives. NeRF has become a fundamental tool for various challenging computer graphics and computer vision tasks. The original NeRF was built on static scenes. Extending radiance fields to dynamic scenes is very important for upgrading NeRF to handle more real-world scenes. One branch of dynamic NeRF-related methods is to take the temporal dimension as input to model scene deformation. Another branch is to map point coordinates into a canonical space based on the predicted point offsets. Other methods enhance dynamic neural rendering with additional depth information and detect keyframes. TiNeuVox demonstrates that temporal information encoding is essential for dynamic scene rendering. It explicitly optimizes voxel features and enhances temporal information through tiny MLPs to reduce training time. MixVoxels explicitly separates the static and dynamic parts of dynamic scenes through an effective variation field and merges them to accelerate the training process. PINF and HyFluid adopt the strategy of PINN and combine physical constraints to learn the velocity field and density field of smoke from videos.

[0006] However, the common disadvantages of these methods are the long training time for each scene and the low fidelity of the reconstruction results of the rendered image and volume density. Therefore, both academia and industry are in urgent need of a framework that can efficiently and accurately reconstruct smoke scenes under the condition of RGB images as input. Summary of the invention

[0007] The purpose of this invention is to solve the problem that the existing framework cannot efficiently reconstruct smoke scenes under the condition of video as input, and propose an efficient smoke scene reconstruction framework based on neural transmission field that does not require additional supervision information. Compared with the previous framework based on neural radiation field, the framework of this invention obtains an implicit representation of the smoke scene with high spatiotemporal consistency through self-supervision learning of physical neural transmission field, and then reconstructs the smoke scene with high fidelity through convolutional neural network.

[0008] The technical solution of the present invention is:

[0009] An efficient smoke scene reconstruction framework based on neural transmission fields, including:

[0010] a. Introduce the transmission equation, constrain the neural deformable field and radiation field under the guidance of the multi-time step strategy, and obtain the density field and rendering results (rendered image and depth) with high spatiotemporal consistency.

[0011] b. Input the rendering result into the convolutional neural network to obtain a detailed smoke rendering image.

[0012] c. Import the model parameters trained in step a, supervise the rendered image of step a with the new perspective rendered image obtained in step b, fine-tune the model parameters of step a, and then retrain the model of step b with the rendering results of the model after fine-tuning the parameters of step a.

[0013] The smoke scene reconstruction framework based on neural transmission field provided by the present invention can use multi-view video as input to efficiently reconstruct the smoke scene in three dimensions. The framework can present the smoke in the real scene in the VR / AR scene, so that users can get a more realistic experience.

[0014] Advantages and beneficial effects of the present invention:

[0015] The present invention implements a smoke scene reconstruction framework based on neural transmission fields through modules. The framework obtains smoke scene rendering results with high spatiotemporal consistency through neural transmission fields, and then refines the details of the rendered image through convolutional neural networks, obtaining high-fidelity smoke scene reconstruction results in a shorter training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the smoke scene reconstruction framework based on neural transmission field in the present invention.

[0017] Figure 2 It is a schematic diagram of the effect of applying the present invention to a synthetic data set.

[0018] Figure 3 It is a schematic diagram of the effect of applying the present invention to a real data set. DETAILED DESCRIPTION

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0020] Reference Figure 1 ,The framework of smoke scene reconstruction based on neural transmission field is as follows:

[0021] The entire framework of the present invention is divided into two parts: neural transmission field and detail optimization based on convolutional neural network. It combines 3D neural transmission field and 2D convolutional neural network detail optimization to efficiently reconstruct high-quality smoke scenes from multi-view videos, and proposes an advanced physical constraint to jointly optimize neural radiation field and deformable field. Among them, the 3D neural transmission field uses neural voxel representation to learn deformable 3D points in the dynamic neural radiation field, and the 2D convolutional neural network uses the rendered image and depth information of the 3D neural transmission field as input to enhance the details of the rendered image. The physical constraint introduces the commonly used transmission equations in fluids to constrain the density of the deformable field and the neural radiation field to enhance the spatiotemporal consistency of the rendered image.

[0022] Based on the above implementations, the present invention proposes an advanced multi-time-step strategy to adaptively embed multiple timestamps to associate global time information.

[0023] The time information t is positionally encoded γ(t) and then input into a small neural network The characteristics of time information obtained in Then, the camera ray h is parsed from the camera parameters, the camera ray is sampled to obtain the three-dimensional space sampling points (x, y, z), and their positions are encoded as γ(x, y, z). and γ(x, y, z) are input into a merged MLP network to obtain the deformable field feature b(x, y, z, t), and γ(x, y, z) is input into another rigid body MLP to obtain the rigid body weight w(x, y, z); the deformable field φ p It can be expressed as:

[0024] φ p (x,y,z,t)=(x,y,z)+w⊙b;

[0025] The density field σ of the present invention depends on the spatial sampling points and the multi-time step strategy:

[0026]

[0027] n is the number of associated specified time points, Ω i is the time difference, is a set of real numbers. The density field σ can be expressed as follows:

[0028] σ=σ(x,y,z,t1(t),t2(t),…,t n (t));

[0029] Reference transmission equation:

[0030]

[0031] In the above formula, u is the velocity field. The chain rule of , and combined with the multi-time step strategy, we can get:

[0032]

[0033] Since the variables of the expression are consistent, the velocity field can be expressed as the deformable field φ p Replace it with a physically inspired spatiotemporal consistency loss:

[0034]

[0035] For convolutional neural network-based detail optimization, the rendered image I and depth D of the neural radiance field under the constraints of physically inspired spatiotemporal consistency loss are input to the encoder ε r In, through the decoder You can get the optimized rendered image I r The relevant formula is as follows:

[0036]

[0037] The present invention constructs a loop optimization system of 3D neural transmission field and 2D convolutional neural network, uses RGB images taken in smoke scenes for supervision, uses the new perspective smoke output by the 2D convolutional neural network to fine-tune the model parameters of the 3D neural transmission field, and the obtained rendering results are further used for training the 2D convolutional neural network, thereby achieving loop optimization.

[0038] To prove the effectiveness of the design of the present invention, the current advanced frameworks are compared with the framework of the present invention on synthetic and real datasets. These advanced frameworks include TiNeuVox, PINF and HyFluid. The indicators used for comparison include PSNR, SSIM, LPIPS, dense SSIM and training time. The results are shown in Table 1. Compared with other frameworks, the present framework has a shorter training time and performs better in other indicators.

[0039] Table 1

[0040]

[0041] Reference Figure 2 ,According to the reference images, compared with other methods on the synthetic dataset, ,the proposed framework achieves new perspective smoke rendering with higher fidelity.

[0042] Reference Figure 3 ,This framework can also obtain more accurate results than other methods in real shooting scenes.

[0043] Based on the above implementation methods, the present invention first provides an efficient smoke scene reconstruction framework based on neural transmission fields to achieve efficient smoke scene reconstruction. First, sampling points in three-dimensional space are obtained; then the three-dimensional coordinates of the sampling points, the viewing direction vector and the time information are input into the voxel neural network to obtain an implicit neural representation of the smoke; under the constraints of the transmission equation based on the multi-time step strategy, the spatiotemporal consistency of this implicit neural representation can be significantly improved, so that the smoke appearance and density with high spatiotemporal consistency can be obtained through volume rendering. Then, the rendered image and the rendering depth are input into a convolutional neural network to obtain a high-fidelity smoke appearance with rich details. Finally, through loop optimization, the output of the convolutional neural network is used as the input of the voxel neural network to obtain a better smoke appearance and density. It has been verified that compared with the existing framework, the present invention has the effects of short training time and better performance of various indicators.

[0044] It should be further explained that the above implementation modes are only used to understand the technical solutions of the present invention, and are not used to limit the protection scope of the present invention. Any obvious changes or adjustments made to the technical solutions of the present invention that belong to the above technical concepts of the present invention shall fall within the protection scope of the present invention.

Claims

1. An efficient smoke reconstruction framework based on neural transmission fields, characterized by: a. Combining 3D neural transmission fields and 2D convolutional neural network detail optimization to efficiently reconstruct high-quality smoke scenes from multi-view videos; b. An advanced physical constraint is proposed to jointly optimize the neural radiation field and the deformable field; c. An advanced multi-time-step strategy is proposed to adaptively embed multiple timestamps to associate global temporal information; d. A recurrent optimization system of 3D neural transmission fields and 2D convolutional neural networks was constructed, supervised by RGB images taken in smoke scenes.

2. The efficient smoke reconstruction framework based on neural transmission fields according to claim 1 is characterized in that: The 3D neural transmission field uses neural voxel representation to learn deformable 3D points in the dynamic neural radiation field. The 2D convolutional neural network takes the rendered image and depth information of the 3D neural transmission field as input to enhance the details of the rendered image.

3. The surface reconstruction framework according to claim 1, characterized in that: This physical constraint introduces the transport equations commonly used in fluids to constrain the density of the deformable field and the neural radiation field to enhance the spatiotemporal consistency of the rendered image.

4. The efficient smoke reconstruction framework based on neural transmission fields according to claim 1, characterized in that: The new perspective smoke output by the 2D convolutional neural network is used to fine-tune the model parameters of the 3D neural transmission field, and the resulting rendering results are further used to train the 2D convolutional neural network, thereby achieving loop optimization.

5. The efficient smoke reconstruction framework based on neural transmission fields according to claim 1, characterized in that: The multi-time step strategy is to position encode the time information t into a small neural network The characteristics of time information obtained in Then, the resolved light h is sampled to obtain three-dimensional space sampling points (x, y, z), which are encoded into position γ(x, y, z), and then and γ(x, y, z) are input into a merged MLP network to obtain the deformable field feature b(x, y, z, t), and γ(x, y, z) is input into another rigid body MLP to obtain the rigid body weight w(x, y, z); the deformable field φ p It can be expressed as: φ p (x,y,z,t)=(x,y,z)+w⊙b\ The density field σ depends on the spatial sampling points and the multi-time step strategy: n is the number of associated specified time points, Ω i is the time difference, is the set of real numbers; The density field σ is expressed as follows: σ=σ(x,y,z,t1(t),t2(t),…,t n (t))。 6. The efficient smoke reconstruction framework based on neural transmission fields according to claim 3 is characterized in that: The transmission equation is Where u is the velocity field, through The chain rule of , and combined with the multi-time step strategy, we get: The velocity field is transformed into a deformable field φ p Replace it with a physically inspired spatiotemporal consistency loss:

7. The efficient smoke reconstruction framework based on neural transmission fields according to claim 4, characterized in that: Based on the detail optimization of convolutional neural networks, the rendered image I and depth D of the neural radiance field under the constraint of physically inspired spatiotemporal consistency loss are input to the encoder ε r In, through the decoder You can get the optimized rendered image I r , the formula is as follows:

8. The application of the efficient smoke reconstruction framework based on neural transmission field according to any one of claims 1 to 7, characterized in that: This framework can be used to reconstruct realistic smoke scenes, thereby enhancing the immersive experience of applications such as AR / VR.

Images