A Highly Efficient Smoke Reconstruction Framework and Application Based on Neural Transmission Fields

Abstract

This invention provides an efficient smoke reconstruction framework and application based on neural transmission fields, belonging to the field of computer graphics technology. The method first uses the camera pose to obtain sampling points in 3D space; then, the 3D coordinates, viewpoint direction vector, and temporal information of the sampling points are input into a voxel neural network to obtain an implicit neural representation of the smoke; volume rendering is then used to obtain a smoke appearance and density with high spatiotemporal consistency. Next, the rendered image and rendering depth are input into a convolutional neural network to obtain a high-fidelity smoke appearance with rich details. Finally, through iterative optimization, the output of the convolutional neural network is used as the input of the voxel neural network to obtain an even better smoke appearance and density. This framework achieves smoke scene reconstruction results with shorter training time and higher accuracy in sparse multi-camera and monocular camera-captured smoke scenes, compared to other methods.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, and specifically relates to a framework and application for reconstructing smoke scenes. Background Technology

[0002] Understanding fluids plays a crucial role in weather forecasting, mechanical design and manufacturing, and various visual effects. Compared to solids or soft bodies, fluid systems possess more complex dynamic shapes and motion patterns. Reconstructing fluid systems from observations is a challenging task.

[0003] Traditional fluid simulations primarily involve solving partial differential equations (PDEs). Physics-based deep learning techniques, leveraging neural networks, learn through a problem-specific learning process. Existing methods address challenging physics problems. They integrate discretized numerical solvers into the training process, then obtain physically plausible solutions for different dynamical systems. Previous models used several multilayer perceptron (MLP) hidden layers to solve PDEs. Automatic differentiation-based training strategies enhance the representational power of MLPs. Other work focuses on more efficient activity functions to estimate accurate derivatives. PINN combines physical constraints and observational data with MLPs to achieve continuous prediction. Recent PINN-based methods add more physical parameters as input to obtain solutions for dynamical systems.

[0004] Fluid simulation is a long-studied task, focusing on simulating real-world smoke or liquids in virtual space. The simulation and rendering processes have always been separate and computationally intensive. Reconstructing physically accurate fluids from RGB images overcomes these limitations and has the potential to extend fluid systems to real-world spaces. Previous research has reconstructed temporally coherent velocity fields from captured low-resolution images using physical constraints. Flow-based methods jointly reconstruct velocity and density fields from sparse tomography or linear image formation. Global transport methods use generative adversarial techniques and differentiable rendering to improve the quality of smoke reconstruction. Gas phenomenon methods train generative models from a set of orthogonal views.

[0005] Recent methods add physically learned priors from RGB images to the Neural Radiation Field (NeRF) to render smoke from new perspectives. NeRF has become a fundamental tool for a variety of challenging computer graphics and computer vision tasks. The original NeRF is built on static scenes. Extending the radiation field to dynamic scenes is crucial for upgrading NeRF to handle more real-world scenarios. One branch of dynamic NeRF-related methods uses the temporal dimension as input to simulate scene deformation. Another branch maps point coordinates to canonical space based on predicted point offsets. Other methods enhance dynamic neural rendering and detect keyframes with additional depth information. TiNeuVox demonstrates that temporal information encoding is essential for dynamic scene rendering. It explicitly optimizes voxel features and enhances temporal information with a tiny MLP to reduce training time. MixVoxels explicitly separates and merges the static and dynamic parts of a dynamic scene using an efficient change field to accelerate the training process. PINF and HyFluid adopt the PINN strategy, incorporating physical constraints to learn the velocity and density fields of smoke from videos.

[0006] However, a common drawback of these methods is the long training time per scene and the low fidelity of the reconstructed images and volume density. Therefore, both academia and industry urgently need a framework that can efficiently and accurately reconstruct smoke scenes using RGB images as input. Summary of the Invention

[0007] The purpose of this invention is to address the problem that existing frameworks cannot efficiently reconstruct smoke scenes when video is used as input. It proposes a high-efficiency smoke scene reconstruction framework based on neural transmission fields that does not require additional supervision. Compared to previous frameworks based on neural radiation fields, this invention's framework learns the physical neural transmission field through self-supervision, thereby obtaining an implicit representation of the smoke scene with high spatiotemporal consistency. Then, a convolutional neural network is used to reconstruct a high-fidelity smoke scene.

[0008] The technical solution of this invention is:

[0009] An efficient smoke scene reconstruction framework based on neural transmission fields includes:

[0010] a. By introducing the transport equation and constraining the neural deformable field and radiation field under the guidance of a multi-time step strategy, a density field and rendering results with high spatiotemporal consistency (rendered image and depth) are obtained.

[0011] b. Input the rendering result into a convolutional neural network to obtain a smoke rendering image with rich details.

[0012] c. Import the model parameters trained in step a, supervise the rendering of the image from step a using the new perspective image obtained in step b, fine-tune the model parameters from step a, and then retrain the model from step b using the rendering result of the model with fine-tuned parameters from step a.

[0013] The smoke scene reconstruction framework based on neural transmission fields provided by this invention can efficiently reconstruct 3D smoke scenes using multi-view video as input. This framework can present smoke from real-world scenes in VR / AR environments, thereby providing users with a more realistic experience.

[0014] Advantages and beneficial effects of the present invention:

[0015] This invention implements a smoke scene reconstruction framework based on neural transmission fields through modules. This framework obtains high spatiotemporal consistency smoke scene rendering results through neural transmission fields, and further refines the details of the rendered image using a convolutional neural network, achieving high-fidelity smoke scene reconstruction results with a relatively short training time. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the smoke scene reconstruction framework based on neural transmission fields in this invention.

[0017] Figure 2 This is a schematic diagram illustrating the effect of applying the present invention to a synthetic dataset.

[0018] Figure 3 This is a schematic diagram illustrating the effect of applying the present invention to a real-world dataset. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Reference Figure 1 The framework for reconstructing smoke scenes based on neural transmission fields is as follows:

[0021] The overall framework of this invention is divided into two parts: a neural transport field and a detail optimization based on a convolutional neural network. It combines a 3D neural transport field and 2D convolutional neural network detail optimization to efficiently reconstruct high-quality smoke scenes from multi-view videos. Simultaneously, it proposes an advanced physical constraint to jointly optimize the neural radiation field and the deformable field. Specifically, the 3D neural transport field uses neural voxel representation to learn deformable 3D points in the dynamic neural radiation field. The 2D convolutional neural network uses the rendered image and depth information from the 3D neural transport field as input to enhance the details of the rendered image. This physical constraint introduces a transport equation commonly used in fluid dynamics to constrain the density of the deformable field and the neural radiation field, thereby enhancing the spatiotemporal consistency of the rendered image.

[0022] Based on the above implementation methods, this invention proposes an advanced multi-timestep strategy to adaptively embed multiple timestamps to associate global time information.

[0023] After the time information t is positionally encoded as γ(t), it is input into a small neural network. Features of obtaining time information Then, the camera ray h is parsed from the camera parameters, and the camera ray is sampled to obtain 3D spatial sampling points (x, y, z), which are then positionally encoded as γ(x, y, z). Then... The deformable field features b(x,y,z,t) are obtained by inputting γ(x,y,z) into a merged MLP network, and the rigid body weights w(x,y,z) are obtained by inputting γ(x,y,z) into another rigid body MLP; the deformable field φ p It can be represented as:

[0024] φ p (x,y,z,t)=(x,y,z)+w⊙b;

[0025] The density field σ of this invention depends on spatial sampling points and a multi-time-step strategy:

[0026]

[0027] n is the number of associated specified time points, Ω i For the time difference, Let be the set of real numbers. The density field σ can be expressed as follows:

[0028] σ=σ(x,y,z,t1(t),t2(t),…,t n (t));

[0029] Reference transmission equation:

[0030]

[0031] In the above equation, u represents the velocity field. (Through...) By applying the chain rule and combining it with a multi-timestep strategy, we can obtain:

[0032]

[0033] Since the variables in the expression are consistent, the velocity field can be represented by a deformable field φ. p Replacement, resulting in a physically inspired spatiotemporal consistency loss:

[0034]

[0035] For detail optimization based on convolutional neural networks, the image I and depth D after neural radiation field rendering under the constraint of physically inspired spatiotemporal consistency loss are input into encoder ε. r In the middle, through the decoder The optimized rendered image I can be obtained r The relevant formulas are as follows:

[0036]

[0037] This invention constructs a cyclic optimization system of a 3D neural transmission field and a 2D convolutional neural network. It uses RGB images captured in a smoke scene for supervision, and uses the new perspective smoke output by the 2D convolutional neural network to fine-tune the model parameters of the 3D neural transmission field. The resulting rendering results are then used to train the 2D convolutional neural network, thereby achieving cyclic optimization.

[0038] To demonstrate the effectiveness of this invention, it was compared with state-of-the-art frameworks on synthetic and real datasets. These state-of-the-art frameworks include TiNeuVox, PINF, and HyFluid. The metrics used for comparison included PSNR, SSIM, LPIPS, dense SSIM, and training time. The results are shown in Table 1. This framework has a shorter training time compared to other frameworks and performs better in all other metrics.

[0039] Table 1

[0040]

[0041] Reference Figure 2 Based on reference images, this framework achieves higher fidelity smoke rendering from a new perspective on synthetic datasets compared to other methods.

[0042] Reference Figure 3 This framework can also achieve more accurate results than other methods in real shooting scenarios.

[0043] Based on the above implementation methods, this invention first provides an efficient smoke scene reconstruction framework based on neural transmission fields to achieve efficient smoke scene reconstruction. First, sampling points in three-dimensional space are obtained. Then, the three-dimensional coordinates, viewpoint direction vector, and temporal information of the sampling points are input into a voxel neural network to obtain an implicit neural representation of the smoke. Under the constraint of a transmission equation based on a multi-timestep strategy, the spatiotemporal consistency of this implicit neural representation can be significantly improved, thereby obtaining a smoke appearance and density with high spatiotemporal consistency through volume rendering. Next, the rendered image and rendering depth are input into a convolutional neural network to obtain a high-fidelity smoke appearance with rich details. Finally, through iterative optimization, the output of the convolutional neural network is used as the input of the voxel neural network to obtain an even better smoke appearance and density. Verification shows that, compared with existing frameworks, this invention has shorter training time and better performance across various metrics.

[0044] It should be further noted that the above embodiments are merely for understanding the technical solution of the present invention and are not intended to limit the scope of protection of the present invention. Any obvious modifications or adjustments made to the technical solution of the present invention that fall within the above technical concept should be within the scope of protection of the present invention.

Claims

1. A highly efficient smoke reconstruction system based on neural transmission fields, characterized in that: a. Combines 3D neural transport fields and 2D convolutional neural network detail optimization to efficiently reconstruct high-quality smoke scenes from multi-view videos; b. An advanced physical constraint is proposed to jointly optimize neural radiation fields and deformable fields; c. An advanced multi-timestep strategy is proposed to adaptively embed multiple timestamps to correlate global time information; d. A recurrent optimization system of 3D neural transmission field and 2D convolutional neural network was constructed and supervised by RGB images captured in a smoke scene; In this system, sampling points in three-dimensional space are first obtained. Then, the three-dimensional coordinates, view direction vector, and time information of the sampling points are input into a voxel neural network to obtain the implicit neural representation of smoke. Under the constraint of the transfer equation based on a multi-time step strategy, a smoke appearance and density with high spatiotemporal consistency are obtained through volume rendering. Next, the rendered image and rendering depth are input into a convolutional neural network to obtain a high-fidelity smoke appearance with rich details. Finally, through iterative optimization, the output of the convolutional neural network is used as the input of the voxel neural network to obtain a better smoke appearance and density. The multi-time-step strategy involves encoding the time information t into a position γ(t) and then inputting it into a small neural network. Features of obtaining time information Then, the resolved ray h is sampled to obtain three-dimensional spatial sampling points (x, y, z), and these points are position-encoded as γ(x, y, z). The deformable field features b(x,y,z,t) are obtained by inputting γ(x,y,z) into a merged MLP network, and the rigid body weights w(x,y,z) are obtained by inputting γ(x,y,z) into another rigid body MLP; the deformable field φ p Expressed as: φ p (x,y,z,t)=(x,y,z)+w⊙b; The density field σ depends on the spatial sampling points and the multi-time-step strategy: n is the number of associated specified time points, Ω i For the time difference, It is the set of real numbers; The density field σ is represented as follows: σ=σ(x,y,z,t1(t),t2(t),…,t n (t)); The transmission equation is as follows: In the formula, u represents the velocity field, which is obtained through... The chain rule, combined with a multi-timestep strategy, yields: The velocity field is represented by a deformable field φ p Replacement, resulting in a physically inspired spatiotemporal consistency loss:

2. The efficient smoke reconstruction system based on neural transmission fields according to claim 1, characterized in that: The 3D neural transmission field uses neural voxel representation to learn deformable 3D points in the dynamic neural radiation field. The 2D convolutional neural network uses the rendered image and depth information of the 3D neural transmission field as input to enhance the details of the rendered image.

3. The efficient smoke reconstruction system based on neural transmission fields according to claim 1, characterized in that: This physical constraint introduces transport equations commonly used in fluids to constrain the density of deformable and neural radiation fields, thereby enhancing the spatiotemporal consistency of the rendered image.

4. The efficient smoke reconstruction system based on neural transmission field according to claim 1, characterized in that: The new perspective smoke output by the 2D convolutional neural network is used to fine-tune the model parameters of the 3D neural transmission field. The resulting rendering results are then used to train the 2D convolutional neural network, thus achieving iterative optimization.

5. The efficient smoke reconstruction system based on neural transmission fields according to claim 4, characterized in that: Based on detail optimization using convolutional neural networks, the image I and depth D rendered by neural radiation field under a physically inspired spatiotemporal consistency loss constraint are input into encoder ε. r In the middle, through the decoder The optimized rendered image I is obtained r The formula is as follows:

6. The application method of the efficient smoke reconstruction system based on neural transmission fields according to any one of claims 1-5, characterized in that: This system enables the reconstruction of realistic smoke scenes, thereby enhancing the immersive experience of AR / VR applications.

Images