Method and system for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding

Abstract

The present invention discloses a three-dimensional reconstruction method and system for smoke scenes based on dual-branch physical knowledge embedding. First, data is collected, fluid perception parameters under various observation angles are calculated, and a state estimation value of each pixel is calculated. The state estimation value is used as a priori supervision to pre-train a state estimation module to distinguish dynamic objects from static objects in the smoke scene. Static objects are sent to the static branch, and dynamic objects are sent to the dynamic branch for separate processing. The volume density and appearance color of static objects are estimated, and the volume density, appearance color and velocity field of dynamic objects are estimated. The velocity field and volume density of dynamic objects are updated using the fluid solution module. The output results of the static branch and the dynamic branch are rendered separately and synthesized. Multi-perspective image information and physical knowledge priors are used as supervision constraints to continuously iteratively optimize module parameters, and ultimately reconstruct a high-fidelity three-dimensional dynamic smoke scene. Image generation from any perspective in the scene is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and mainly relates to a three-dimensional reconstruction method and system for smoke scenes based on dual-branch physical knowledge embedding. Background Art

[0002] Smoke, the most common dynamic fluid in everyday life, typically exhibits irregular morphology and complex dynamic changes. Reconstructing dynamic fluids such as smoke from real-world observational data can provide important technical support and theoretical basis for fields such as fire warning and emergency rescue, climate and environmental monitoring, scientific experiments and engineering simulations, and film and game special effects, effectively promoting efficiency improvements in scientific decision-making and practical applications in related fields. However, unlike rigid objects with a defined shape and simple motion patterns, smoke fluids not only exhibit non-rigid and complex variations but also often exhibit significant translucency and highly irregular appearance in image modalities. This results in a lack of global consistency in the reconstruction results of dynamic smoke scenes, loss of details of non-smoke objects in the scene, and deviations in the density distribution or motion trajectory of smoke objects in local areas. This fails to truly reflect the motion state of smoke in real environments, thus limiting its promotion and application in related fields. Therefore, how to achieve high-fidelity 3D reconstruction of dynamic smoke scenes based on image observation data is a major challenge currently faced.

[0003] To address this challenge, given that fluid motion strictly adheres to the laws of physics, the core focus of related research can be summarized as how to resolve multiple flow field quantities—namely, velocity, pressure, and density—from real-world observational data. Early approaches to dynamic smoke scene reconstruction primarily relied on computational fluid dynamics (CFD)-based fluid reconstruction methods. However, these methods require extensive computational resources and specialized knowledge to set the boundary conditions. Different boundary conditions and meshing can significantly affect the accuracy of the results, limiting their applicability to real-world scenarios. Recently, physics-aware neural network-based fluid reconstruction methods have emerged. By leveraging underlying physical constraints and integrating physics knowledge into deep learning techniques, they offer a more flexible and efficient solution and achieve significant performance improvements. This approach introduces a new paradigm in computer vision, exploring how to leverage physical laws to enhance visual tasks. This approach primarily takes observable image modalities as input and models fluid behavior by learning the fluid motion patterns in the data. Leveraging underlying physical constraints, this approach achieves an accurate solution for dynamic flow, freeing itself from the constraints of precise boundary conditions and environmental requirements. Furthermore, by integrating this approach with a visual reconstruction framework, an end-to-end dynamic smoke scene reconstruction solution has been realized.

[0004] However, the fluid reconstruction method based on physical perception neural network still faces severe challenges, mainly manifested in the following aspects: (1) The reconstruction time is long. Existing studies directly use physical conservation laws as constraints to directly solve the pressure field and velocity field and use them to update the density field. However, the pressure field and velocity field in actual scenes are tightly coupled, and direct solution usually leads to numerical instability and difficulty in convergence; (2) The accuracy is limited in complex scenes. Dynamic smoke scenes have a large number of multi-scale features, especially small-scale structures in turbulence, which makes the imposed physical rule constraints unable to correctly handle the energy dissipation effect. In particular, under small-scale features, energy may be over-concentrated or dissipated, forming a non-physical degenerate solution; (3) The non-fluid reconstructed objects have ambiguous appearance. Smoke fluids often cause atmospheric absorption and scattering, which easily blur the boundaries and color distribution of non-fluid objects surrounded by smoke, resulting in poor fidelity of the reconstructed non-fluid objects, such as missing texture details and shape ambiguity. Summary of the Invention

[0005] The present invention is aimed at the problems existing in the prior art and proposes a 3D reconstruction method and system for smoke scene based on dual-branch physical knowledge embedding. First, video images of the same smoke scene from different observation angles are collected, and the fluid perception parameters under each observation angle are calculated. The fluid perception parameters are constructed together with the corresponding smoke scene video to form a reconstruction data set. The state estimation value of each pixel in the image is calculated using the fluid perception parameters, and the state estimation value is used as a priori supervision to pre-train a state estimation module in the reconstruction data set. The trained state estimation module is used to distinguish dynamic objects from static objects in the smoke scene. The static objects are sent to the static branch for processing, and a static target encoding and decoding module with time invariance is used to guide the feature learning of the 3D representation of static objects in the smoke scene, which is used to estimate the volume density and appearance color of static objects in the smoke scene. The dynamic objects are sent to the dynamic branch for processing. The dynamic branch is processed, and a dynamic target encoding and decoding module with time-varying characteristics is used to guide the feature learning of the three-dimensional representation of dynamic objects in the smoke scene, which is used to estimate the volume density, appearance color and velocity field of dynamic objects in the smoke scene; the fluid solution module is used to obtain a velocity field that conforms to physical rule constraints from the volume density of static objects and the intermediate velocity field of dynamic objects, and is used to update the velocity field and volume density of dynamic objects; the output results of the static branch and the dynamic branch are used to render static object images and dynamic object images of the smoke scene from different perspectives, and the reconstructed images of the smoke scene from different perspectives are obtained through the synthetic rendering module. The image information and physical knowledge priors of the multi-perspective smoke scene video are used as supervision constraints, and the module parameters are continuously iteratively optimized, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation from any perspective in the scene is supported. The method of the present invention can simultaneously realize the three-dimensional reconstruction and physical state estimation of dynamic smoke scenes, and reconstruct a high-fidelity three-dimensional scene that conforms to physical characteristics from multi-view observation images.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a 3D reconstruction method of smoke scene based on dual-branch physical knowledge embedding, comprising the following steps:

[0007] S1. Data Collection and Processing: Collect video images from different observation angles of the same smoke scene, calculate the fluid perception parameters at each observation angle, and construct a reconstructed dataset together with the corresponding smoke scene video; the fluid perception parameters include at least atmospheric transmittance and time-varying variance;

[0008] S2. Pre-training the state estimation module: Calculate the state estimation value of each pixel in the image based on the fluid perception parameters obtained in step S1. Use the state estimation value as a priori supervision to pre-train the state estimation module in the reconstructed dataset to obtain a trained state estimation module to distinguish dynamic objects from static objects in the smoke scene. For static objects, proceed to step S3; for dynamic objects, proceed to step S4.

[0009] S3, static branch processing: The static objects obtained after the differentiation in step S2 are sent to the static branch for processing. The static object encoding and decoding module with time invariance guides the feature learning of the three-dimensional representation of static objects in the smoke scene to estimate the volume density and appearance color of the static objects in the smoke scene.

[0010] S4, dynamic branch processing: The dynamic objects obtained after the differentiation in step S2 are sent to the dynamic branch for processing. The dynamic target encoding and decoding module with time-varying characteristics guides the feature learning of the three-dimensional representation of dynamic objects in the smoke scene to estimate the volume density, appearance color, and velocity field of the dynamic objects in the smoke scene;

[0011] S5. Update dynamic object features: Using the fluid solution module, a velocity field that complies with physical constraints is obtained from the static object volume density obtained in step S3 and the dynamic object intermediate velocity field obtained in step S4, and the velocity field and volume density of the dynamic object are updated accordingly.

[0012] S6, synthetic rendering: rendering static object images and dynamic object images of the smoke scene at different viewing angles according to the output results of the static branch of step S3 and the dynamic branch of step S5, and reconstructing images at different viewing angles through a synthetic rendering module;

[0013] S7. Iterative optimization: Using the image information and physical knowledge priors of multi-perspective smoke scene videos as supervisory constraints, the parameters of the static target encoding and decoding module, the dynamic target encoding and decoding module, and the fluid solution module are continuously iteratively optimized to obtain an optimized three-dimensional dynamic smoke scene, realizing image generation from any perspective in the scene.

[0014] As an improvement of the present invention, the atmospheric transmittance in the fluid sensing parameter of step S1 is obtained by dark channel prior estimation, specifically:

[0015]

[0016] Where p u is the pixel coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, ε is p u The local block centered on I c is the brightness value corresponding to RGB channel c;

[0017] The time-varying variance is calculated as follows:

[0018]

[0019] Where D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channels.

[0020] As an improvement of the present invention, the state estimation value in step S2 sets a transmittance threshold and a variance threshold to screen image pixels that simultaneously meet the requirements of atmospheric transmittance lower than a preset transmittance and time-varying variance greater than the variance threshold. The state estimation value of the image pixel is set to 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; the remaining image pixels are set to 0, indicating that all the light sampling spaces corresponding to the pixels are static objects.

[0021] As another improvement of the present invention, in step S3, the volume density and appearance color calculation process of the static object in the smoke scene is specifically as follows:

[0022]

[0023] Where, σ s is the volume density of the static object, p v is the three-dimensional position coordinate of the static object in the static branch, c s is the appearance color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

[0024] As another improvement of the present invention, in step S4, the volume density, appearance color, and velocity field calculation process of the dynamic object in the smoke scene are specifically as follows:

[0025]

[0026] Where, σ d is the volume density of the dynamic object, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, d is the observation direction, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

[0027] As another improvement of the present invention, in step S5, the velocity field of the dynamic object is updated to a velocity field that conforms to the incompressible fluid motion constraint based on the pressure field calculated by the pressure Poisson equation, and the volume density of the dynamic object is calculated and updated using the semi-Lagrangian method using the updated velocity field. The specific formula is as follows:

[0028]

[0029] Where u * is the velocity field before updating, u is the velocity field after updating, p is the pressure field, σ is the volume density before updating, is the updated volume density, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, t is the current time step, Δt is the time step, is the gradient operator.

[0030] As a further improvement of the present invention, in the synthesis rendering module of step S5, the dynamic and static object images are synthesized to obtain a reconstructed image of the smoke scene under each observation angle. The specific calculation formula is as follows:

[0031]

[0032] Where, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, N is the number of light sampling points, T i is the cumulative transmittance at the current position, δ i is the sampling interval of the current position point, σ s and c s are the volume density and appearance color of static objects, and c d They are the volume density and appearance color of dynamic objects respectively.

[0033] In order to achieve the above-mentioned purpose, the present invention also adopts a technical solution: a smoke scene 3D reconstruction method system based on dual-branch physical knowledge embedding, which at least includes a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthetic rendering module.

[0034] The state estimation module is pre-trained using a binary cross entropy loss function to distinguish dynamic objects from static objects in a smoke scene. The pre-training must meet the following requirements: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should all be 0, indicating that all objects in the light sampling space corresponding to the pixel are static objects; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel;

[0035] The static branch is used to process static objects in the smoke scene and includes at least a static target encoding and decoding module composed of three axial feature grids (XY / YZ / XZ) and an appearance decoder. The static target encoding and decoding module takes the three-dimensional position coordinates as input and calculates the volume density of the corresponding position through trilinear interpolation. The appearance decoder is composed of a multi-layer perceptron containing multiple fully connected layers and calculates the appearance color of the corresponding position.

[0036] The dynamic branch is used to process dynamic objects in the smoke scene and includes at least a dynamic target encoding and decoding module composed of six axial feature grids (XY / ZT / XZ / YT / TZ / XT), an appearance decoder, and a velocity decoder. The dynamic target encoding and decoding module takes three-dimensional position coordinates and corresponding time information as input and calculates the corresponding volume density by projecting the coordinate points onto each feature plane. The appearance decoder and velocity decoder are both composed of a multi-layer perceptron containing multiple fully connected layers, and respectively calculate the appearance color and velocity field of the corresponding position.

[0037] The fluid solving module is used to obtain a velocity field that complies with physical constraints and to update the velocity field and volume density of dynamic objects, and includes at least a pressure field solver and a flow solver;

[0038] The pressure field solver adopts a multi-scale convolutional architecture and uses the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene;

[0039] The flow solver takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to physical characteristics according to the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method;

[0040] The synthetic rendering module uses a volume rendering method to obtain static object images and dynamic object images of the smoke scene based on the volume density and appearance color of static objects and dynamic objects, calculates the transmittance of each position point in three-dimensional space based on the volume density, and renders to obtain a reconstructed image of the smoke scene under various observation angles.

[0041] As an improvement of the present invention, the loss function of the state estimation model is:

[0042]

[0043] Where p v is the three-dimensional position coordinate, p u is the two-dimensional pixel plane coordinate, S(p u ) is the state estimation module in the two-dimensional pixel plane coordinate p u The output value of The three-dimensional position coordinate p of the state estimation module v The output value of N is the number of light sampling points, L s Estimate the loss for the state.

[0044] As another improvement of the present invention, a projection loss function is used for training the pressure field solver. The loss function uses the divergence of the velocity field under the pressure projection to satisfy the incompressible physical property of the fluid. The specific formula is as follows:

[0045]

[0046] Where u is the velocity field after being updated by the fluid solution module, u * is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The method of the present invention proposes to use a state estimation module to distinguish dynamic objects from static objects in a smoke scene, and to use fluid perception parameters with physical priors to accurately perceive the motion states of different objects in the smoke scene, thereby achieving effective decoupling of dynamic and static objects and significantly reducing the computational complexity of the reconstruction algorithm.

[0049] (2) The method of the present invention utilizes a dual-branch structure to process dynamic and static objects, and introduces a fluid solution module embedded with physical knowledge into the dynamic branch. The module adopts a multi-scale convolutional architecture, which can effectively capture the global information and local details of dynamic objects and eliminate the degradation problem of visual ambiguity in smoke scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a working principle diagram of the smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding of the present invention;

[0051] Figure 2 These are example images of two smoke scenes at different observation angles in Example 2 of the present invention;

[0052] Figure 3 This is a flowchart of the steps of the method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding of the present invention;

[0053] Figure 4 This is a diagram showing the effect of Example 2 of the present invention after passing through the state estimation module;

[0054] Figure 5 Schematic diagram of the model structure of the pressure field solver in the system of the present invention;

[0055] Figure 6 This is a diagram showing the effect of velocity field estimation in a smoke scene in Example 2 of the present invention;

[0056] Figure 7 This is a diagram showing the effects of static branches and dynamic branches in a smoke scene in Example 2 of the present invention;

[0057] Figure 8 A streamline diagram of the forward reasoning and iterative optimization process of the entire model of the present invention;

[0058] Figure 9 This is a comparison chart of reconstruction results of different models in the test example of the present invention, where:

[0059] (A) is a schematic diagram of the results of NVFi, (B) is a schematic diagram of the results of PINF, (C) is a schematic diagram of the results of HyFluid, and (D) is a schematic diagram of the results of the method of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0061] Example 1

[0062] A smoke scene three-dimensional reconstruction system based on dual-branch physical knowledge embedding includes at least a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthetic rendering module.

[0063] The state estimation module is pre-trained using a binary cross-entropy loss function, which must satisfy the following assumptions: when the state estimate of an image pixel is 0, the state estimates of all 3D position coordinates obtained from light sampling of that pixel should also be 0; when the state estimate of an image pixel is 1, at least one of the state estimates of the 3D position coordinates obtained from light sampling of that pixel should be 1. The trained state estimation module receives the corresponding state estimate from the 3D position coordinates and classifies dynamic and static objects in the smoke scene based on the state estimate.

[0064] The static branch is used to process static objects in the smoke scene and includes at least a static target encoding and decoding module. The static target encoding and decoding module consists of three axial feature grids (XY / YZ / XZ) and an appearance decoder, which respectively contain the geometric features and appearance features of the static objects in the smoke scene. The appearance decoder consists of a multi-layer perceptron containing multiple fully connected layers. This module takes three-dimensional position coordinates as input, calculates the volume density of the corresponding position through trilinear interpolation, and calculates the appearance color of the corresponding position through the appearance decoder.

[0065] The dynamic branch is used to process dynamic objects in the smoke scene, and includes at least a dynamic target encoding and decoding module. The dynamic target encoding and decoding module contains six axial feature grids (XY / ZT / XZ / YT / TZ / XT), an appearance decoder, and a velocity decoder, which respectively contain the geometric features, appearance features, and velocity features of the dynamic objects in the smoke scene. The appearance decoder and the velocity decoder are both composed of a multi-layer perceptron containing multiple fully connected layers. The module takes four-dimensional spatial coordinates (three-dimensional position coordinates and corresponding time information) as input, calculates the corresponding volume density by projecting the coordinate points onto each feature plane, calculates the appearance color of the corresponding position through the appearance decoder, and calculates the velocity field of the corresponding position through the velocity decoder.

[0066] The fluid solving module includes a pressure field solver and a flow solver. The pressure field solver takes the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene. It adopts a multi-scale convolution architecture and captures the global information and local details of dynamic objects through multiple cascaded spatial downsampling encoding blocks and upsampling decoding blocks to eliminate the degradation problem of visual ambiguity in smoke scenes. The loss function uses the divergence of the velocity field under the pressure projection to meet the incompressible physical properties of the fluid; the flow solver takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to the physical properties based on the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method.

[0067] The synthetic rendering module uses volume rendering to obtain static and dynamic object images of the smoke scene based on the volume density and appearance color of static and dynamic objects. It also calculates the transmittance of each position point in three-dimensional space based on the volume density, and renders the reconstructed image of the smoke scene from each observation angle.

[0068] The working principle of this system is as follows Figure 1 As shown in the figure, the fluid perception parameters are first extracted and the state estimation module is used to distinguish dynamic objects from static objects in the smoke scene. Then, a dual-branch structure for separating dynamic and static objects is used to realize feature learning and state estimation of dynamic and static objects respectively. The module parameters are continuously iteratively optimized through supervised constraints, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, which supports image generation from any perspective in the scene.

[0069] Example 2

[0070] Using the system in Example 1, the input image of the model in this embodiment is obtained by capturing the same dynamic smoke scene from different observation angles using 6 fixed-position cameras. The image size is 800×800 pixels, the total number of frames exceeds 130, and the frame rate is 25FPS. The images contained in the scene are as follows: Figure 2 As shown in the figure, there are both static obstacles and dynamic smoke. By extracting image information at different observation angles and different time stamps, the image information of 4 angles is used for reconstruction, and the image information of the remaining 2 angles is used for testing. Therefore, a 3D reconstruction method of smoke scene based on dual-branch physical knowledge embedding is proposed. Figure 3 As shown, the specific steps include:

[0071] Step S1: Collect video images from different observation angles in the same smoke scene, calculate the fluid perception parameters under each observation angle, and construct a reconstruction dataset together with the corresponding smoke scene video.

[0072] This example includes observation images from six viewing angles, with an image size of 800×800 pixels, a total of over 130 frames, and a frame rate of 25 FPS. Fluid perception parameters are calculated for each viewing angle and combined with the corresponding smoke scene video to form a reconstructed dataset. The fluid perception parameters include the atmospheric transmittance and time-varying variance of each pixel. Atmospheric transmittance is calculated as follows:

[0073]

[0074] Where p u is the pixel coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, which is mainly obtained by selecting the pixel with the highest intensity through the statistical results of the dark channel, and ε is the value of p u The local block centered on (15×15 pixels in this example), I c is the brightness value corresponding to RGB channel c. The time-varying variance is calculated as follows:

[0075]

[0076] Where D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channels.

[0077] Step S2: Calculate the state estimation value of each pixel in the image using the fluid perception parameters under different observation perspectives, and use the state estimation value as a priori supervision to pre-train the state estimation module in the reconstructed dataset. The trained state estimation module is used to distinguish dynamic objects from static objects in the smoke scene.

[0078] The state estimation value is calculated based on the fluid perception parameters. The state estimation value is a Boolean variable. By setting the transmittance threshold and variance threshold, the image pixels that meet the fluid perception parameters and have an atmospheric transmittance lower than the preset transmittance and a time-varying variance greater than the variance threshold are screened. The state estimation value is set to 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel, and the rest are set to 0, indicating that the light sampling space corresponding to the pixel is all static objects. The state estimation value of the pixel is calculated as follows:

[0079]

[0080] Where S is the estimated value of the state, θ d is the variance threshold, θ t is the transmittance threshold.

[0081] In this example, θ d Set to 15, θ t Set to 0.75, the effect is as follows Figure 4The three-dimensional position coordinates of the pixel corresponding to the light sampling are calculated as follows:

[0082] p v =K[R,T]p u

[0083] Where p v is the three-dimensional position coordinate, K is the camera intrinsic parameter, and [R, T] is the camera extrinsic parameter including the rotation matrix and translation matrix.

[0084] The state estimation module uses a binary cross entropy loss function for pre-training, which must meet the following assumptions: when the state estimate value of an image pixel is 0, the state estimate values ​​of all 3D position coordinates obtained from the pixel through light sampling should be 0; when the state estimate value of an image pixel is 1, at least one of the state estimate values ​​of the 3D position coordinates obtained from the pixel through light sampling should be 1. The loss function is as follows:

[0085]

[0086] Where, The three-dimensional position coordinate p of the state estimation module v The output value of , N is the number of light sampling points.

[0087] In this embodiment, the state estimation module is trained for 50 cycles using the above loss function, with a batch size of 8, using an adaptive momentum estimation optimizer for optimization, and setting the learning rate to 5e-3. The trained state estimation module is used to distinguish between dynamic and static objects in the smoke scene.

[0088] Step S3: Static objects in the smoke scene are sent to the static branch for processing. A time-invariant static object encoding and decoding module is used to guide feature learning of the 3D representation of static objects in the smoke scene, which is used to estimate the volume density and appearance color of static objects in the smoke scene. The calculation process is as follows:

[0089]

[0090] Where, σ s is the volume density of the static object, c s is the appearance color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

[0091] Step S4: Dynamic objects in the smoke scene are sent to the dynamic branch for processing. A dynamic target encoding and decoding module with time-varying characteristics is used to guide feature learning of the 3D representation of dynamic objects in the smoke scene, which is used to estimate the volume density, appearance color, and velocity field of dynamic objects in the smoke scene. The calculation process is as follows:

[0092]

[0093] Where, σ d is the volume density of the dynamic object, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

[0094] Step S5: Utilize the fluid solving module to obtain a velocity field that complies with physical constraints from the volume density of the static object and the intermediate velocity field of the dynamic object, and use it to update the velocity field and volume density of the dynamic object.

[0095] The fluid solving module includes a pressure field solver and a flow solver. The pressure field solver takes the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene. It adopts a multi-scale convolution architecture, such as Figure 5 As shown in the figure, multiple cascaded spatial downsampling encoding blocks and upsampling decoding blocks are used to capture the global information and local details of dynamic objects, effectively eliminating the degradation problem of visual ambiguity in smoke scenes. Specifically, the solver uses three scales of 1 / 4, 1 / 2, and 1, and the calculation formula is as follows:

[0096]

[0097] Where Y i (i=2) represents the pressure field output by the pressure field solver, U represents the upsampling operation, D represents the downsampling operation, and C represents the convolution and ReLU activation operations.

[0098] The projection loss function is used for training the pressure solver. The loss function uses the divergence of the velocity field under the pressure projection to satisfy the incompressible physical properties of the fluid. The specific formula is as follows:

[0099]

[0100] Where u *is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

[0101] The flow solver uses the velocity field of the dynamic object and the pressure field output by the pressure field solver as input. It calculates and updates the velocity field that conforms to the physical characteristics based on the pressure Poisson equation. It then uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method. The specific formula is as follows:

[0102]

[0103] Where u is the updated velocity field, is the updated volume density, t is the current time step, Δt is the time step, is the gradient operator. The extracted velocity field visualization effect is as follows Figure 6 shown.

[0104] Step S6: using the static object encoding module and the dynamic object encoding module to render the static object images and dynamic object images of the smoke scene at different viewing angles respectively, and obtaining the reconstructed images of the smoke scene at different viewing angles through the synthesis rendering module.

[0105] In the forward reasoning process, firstly, according to the output results of the static branch and the dynamic branch, the volume rendering method is used to obtain the static object images and dynamic object images of the smoke scene under different perspectives. Figure 7 Then, the synthetic rendering module is used to synthesize the dynamic and static object images and obtain the reconstructed image of the smoke scene under each observation angle. The specific calculation formula is as follows:

[0106]

[0107] Where, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, T is the cumulative transmittance of the current position, and N is the number of light sampling points.

[0108] Step S7: Using the image information and physical knowledge priors of the multi-view smoke scene video as supervision constraints, the module parameters are continuously iteratively optimized, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation from any perspective in the scene is supported.

[0109] The process of steps S6 and S7 is as follows Figure 8 As shown in the figure, the blue solid line represents the forward reasoning process, and the red dotted line represents the iterative optimization process of the module parameters based on the supervision constraints.

[0110] During the optimization process, the module parameters are iteratively optimized using image information and physical knowledge priors from multi-view smoke scene videos as supervisory constraints. The supervisory constraints consist of two parts: image difference and physical knowledge priors. The image difference is used to minimize the difference between the real observed image and the reconstructed image obtained by the synthetic rendering module. The physical knowledge prior is used to constrain the reconstructed velocity field and the volume density of dynamic objects to comply with the laws of conservation of mass and momentum in fluid motion, thereby reconstructing a high-fidelity dynamic smoke scene that follows the laws of fluid physics. The specific calculation is as follows:

[0111] L=β1L c +β2L vel +β3L dens

[0112]

[0113] Where, L c is the color loss function between the reconstructed image and the real image, L vel L is the velocity loss function that complies with the law of conservation of momentum in fluid motion. dens is the density loss function that complies with the mass conservation in fluid motion. β is a hyperparameter.

[0114] In this example, the hyperparameters are set to β1 = 100, β2 = 0.1, and β3 = 0.1. The Adam optimizer is used for training for 15,000 rounds, the batch size is 2048, and the spatial resolution is set to 128. 3 , and the learning rate is set to 1e-4. After training, a high-fidelity 3D dynamic smoke scene can be reconstructed, and images of any perspective in the scene can be generated.

[0115] Figure 7 The reconstruction effect diagram of static branches and dynamic branches in a smoke scene is displayed, indicating that the present invention can accurately reconstruct static objects and dynamic objects from a smoke scene.

[0116] Test Case

[0117] Two different dynamic smoke scenes were selected for testing. Both scenes contained dynamic smoke and static obstacles and had six fixed-position cameras, numbered 1 through 6. The captured images were 800×800 pixels in size, with a total of 150 frames per second and a frame rate of 25 FPS. For each model, the image data from cameras 1 through 5 was used as the reconstruction dataset, and the image data from camera 6 was used as the test dataset. The different models used the same reconstruction dataset to reconstruct the scene. When all models fully converged, the reconstructed images of each model at camera 6 were rendered and visually compared with the real image captured by camera 6.

[0118] Figure 9 Comparison of visualization results reconstructed by different models, where: Figure 9 (A) is a schematic diagram of NVFi results. Figure 9 (B) is a schematic diagram of the PINF results. Figure 9 (C) is a schematic diagram of the results of HyFluid. Figure 9 (D) is a schematic diagram of the results of the method of the present invention. As can be seen from the comparison in the figure, the density and appearance of the smoke presented by the present invention are more consistent with the real view. The appearance reconstruction of static obstacles under the smoke is clearer and more accurate. Compared with other models, the reconstruction results of the present invention have higher fidelity, demonstrating that the present invention can reconstruct high-fidelity three-dimensional dynamic smoke scenes and support image generation from any perspective within the scene.

[0119] In summary, this paper proposes an innovative 3D reconstruction method that achieves high-fidelity 3D reconstruction of dynamic smoke scenes while accurately estimating the physical state. First, a state estimation module is used to distinguish between dynamic and static objects in the smoke scene. Then, a dual-branch structure that separates dynamic and static objects is used to perform feature learning and state estimation for each of these objects, ultimately reconstructing a high-fidelity 3D dynamic smoke scene. This method supports image generation from any perspective within the scene. Multi-view smoke scene videos are used to achieve 3D reconstruction and physical state estimation of dynamic smoke scenes, reconstructing a high-fidelity 3D scene that conforms to physical properties from multi-view observation images.

[0120] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A 3D reconstruction method for smoke scenes based on dual-branch physical knowledge embedding, characterized by: The steps include: S1. Data Collection and Processing: Collect video images from different observation angles of the same smoke scene, calculate the fluid perception parameters at each observation angle, and construct a reconstructed dataset together with the corresponding smoke scene video; the fluid perception parameters include at least atmospheric transmittance and time-varying variance; S2. Pre-training a state estimation module: Calculate a state estimation value for each pixel in the image based on the fluid perception parameters obtained in step S1. Use the state estimation value as a priori supervision to pre-train the state estimation module in the reconstructed dataset to obtain a trained state estimation module to distinguish between dynamic and static objects in the smoke scene. For static objects, go to step S3; For dynamic objects, go to step S4; S3, static branch processing: The static objects obtained after the differentiation in step S2 are sent to the static branch for processing. The static object encoding and decoding module with time invariance guides the feature learning of the three-dimensional representation of static objects in the smoke scene to estimate the volume density and appearance color of the static objects in the smoke scene. S4, dynamic branch processing: The dynamic objects obtained after the differentiation in step S2 are sent to the dynamic branch for processing. The dynamic target encoding and decoding module with time-varying characteristics guides the feature learning of the three-dimensional representation of dynamic objects in the smoke scene to estimate the volume density, appearance color, and velocity field of the dynamic objects in the smoke scene; S5. Update dynamic object features: Using the fluid solution module, a velocity field that conforms to the incompressible fluid motion constraint is obtained from the static object volume density obtained in step S3 and the dynamic object intermediate velocity field obtained in step S4, and the velocity field and volume density of the dynamic object are updated accordingly. S6, synthetic rendering: rendering static object images and dynamic object images of the smoke scene at different viewing angles according to the output results of the static branch of step S3 and the dynamic branch of step S5, and reconstructing images at different viewing angles through a synthetic rendering module; S7. Iterative optimization: Using the image information and physical knowledge priors of multi-perspective smoke scene videos as supervisory constraints, the parameters of the static target encoding and decoding module, the dynamic target encoding and decoding module, and the fluid solution module are continuously iteratively optimized to obtain the optimized three-dimensional dynamic smoke scene, realizing image generation from any perspective in the scene.

2. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: The atmospheric transmittance in the fluid sensing parameter of step S1 is obtained by dark channel prior estimation, specifically: Where p u is the two-dimensional pixel plane coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, ε is p u The local block centered on I c is the brightness value corresponding to RGB channel c; The time-varying variance is calculated as follows: Where p u is the two-dimensional pixel plane coordinate, D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channel.

3. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 2, characterized in that: The state estimation value in step S2 is obtained by setting a transmittance threshold and a variance threshold to screen image pixels that simultaneously meet the conditions of atmospheric transmittance lower than a preset transmittance and time-varying variance greater than the variance threshold. The state estimation value of the image pixel is set to 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; the state estimation value of the remaining image pixels is set to 0, indicating that all the light sampling spaces corresponding to the pixels are static objects.

4. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In step S3, the volume density and appearance color calculation process of the static object in the smoke scene is as follows: Where, σ s is the volume density of the static object, p v is the three-dimensional position coordinate of the static object in the static branch, c s is the appearance color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

5. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In step S4, the volume density, appearance color, and velocity field calculation process of the dynamic object in the smoke scene are specifically as follows: Where, σ d is the volume density of the dynamic object, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, d is the observation direction, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

6. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 4 or 5, characterized in that: In step S5, the velocity field of the dynamic object is updated to a velocity field that conforms to the incompressible fluid motion constraint based on the pressure field calculated by the pressure Poisson equation. The updated velocity field is then used to calculate the volume density of the dynamic object using the semi-Lagrangian method. The specific formula is as follows: Where u * is the velocity field before updating, u is the velocity field after updating, p is the pressure field, σ is the volume density before updating, is the updated volume density, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, t is the current time step, Δt is the time step, is the gradient operator.

7. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In the synthesis rendering module of step S5, the dynamic and static object images are synthesized to obtain a reconstructed image of the smoke scene under each observation angle. The specific calculation formula is as follows: Where, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, N is the number of light sampling points, T i is the cumulative transmittance at the current position, δ i is the sampling interval of the current position point, σ s and c s are the volume density and appearance color of static objects, and c d They are the volume density and appearance color of dynamic objects respectively.

8. A method system for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding using the method of claim 1, characterized in that: At least includes a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthetic rendering module, The state estimation module is pre-trained using a binary cross entropy loss function to distinguish dynamic objects from static objects in a smoke scene. The pre-training must meet the following requirements: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should all be 0, indicating that all objects in the light sampling space corresponding to the pixel are static objects; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; The static branch is used to process static objects in the smoke scene. It includes at least a static target encoding and decoding module composed of three axial feature grids (XY / YZ / XZ) and an appearance decoder. The static target encoding and decoding module takes three-dimensional position coordinates as input and calculates the volume density of the corresponding position through trilinear interpolation. The appearance decoder is composed of a multi-layer perceptron containing multiple fully connected layers, and calculates the appearance color of the corresponding position; The dynamic branch is used to process dynamic objects in the smoke scene and includes at least a dynamic target encoding and decoding module composed of six axial feature grids (XY / ZT / XZ / YT / TZ / XT), an appearance decoder, and a velocity decoder. The dynamic target encoding and decoding module takes three-dimensional position coordinates and corresponding time information as input and calculates the corresponding volume density by projecting the coordinate points onto each feature plane. The appearance decoder and velocity decoder are both composed of a multi-layer perceptron containing multiple fully connected layers, and respectively calculate the appearance color and velocity field of the corresponding position. The fluid solving module is used to obtain a velocity field that complies with physical constraints and to update the velocity field and volume density of dynamic objects, and includes at least a pressure field solver and a flow solver; The pressure field solver adopts a multi-scale convolutional architecture and uses the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene; The flow solver takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to physical characteristics according to the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method; The synthetic rendering module uses a volume rendering method to obtain static object images and dynamic object images of the smoke scene based on the volume density and appearance color of static objects and dynamic objects, calculates the transmittance of each position point in three-dimensional space based on the volume density, and renders to obtain a reconstructed image of the smoke scene under various observation angles.

9. The smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding according to claim 8, characterized in that: The loss function of the state estimation model is: Where p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, p u is the two-dimensional pixel plane coordinate, S(p u ) is the state estimation module in the two-dimensional pixel plane coordinate p u The output value of is the three-dimensional position coordinate p of the dynamic object in the dynamic branch of the state estimation module v The output value of N is the number of light sampling points, L s Estimate the loss for the state.

10. The smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding according to claim 8, characterized in that: The projection loss function is used for training the pressure field solver. The loss function uses the divergence of the velocity field under the pressure projection to satisfy the incompressible physical properties of the fluid. The specific formula is as follows: Where u is the velocity field after being updated by the fluid solution module, u * is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

Images