Smoke scene three-dimensional reconstruction method and system based on double-branch physical knowledge embedding

Abstract

The invention discloses a smoke scene three-dimensional reconstruction method and system based on double-branch physical knowledge embedding, and the method comprises the steps: firstly collecting data, calculating a fluid perception parameter under each observation view angle, calculating a state estimation value of each pixel, taking the state estimation value as priori supervision, pre-training a state estimation module, and carrying out the reconstruction of a smoke scene through the pre-training state estimation module; therefore, dynamic objects and static objects in the smoke scene are distinguished. The static object is sent to a static branch, the dynamic object is sent to a dynamic branch to be processed respectively, the volume density and appearance color of the static object are estimated, and the volume density, appearance color and velocity field of the dynamic object are estimated; updating the velocity field and the volume density of the dynamic object by using a fluid solving module; output results of the static branches and the dynamic branches are respectively rendered and synthesized, multi-view image information and physical knowledge priori are used as supervision constraints, module parameters are continuously iteratively optimized, finally, a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation of any view angle in the scene is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and mainly relates to a three-dimensional reconstruction method and system for smoke scenes based on dual-branch physical knowledge embedding. Background Art

[0002] As the most common dynamic fluid in daily life, smoke usually presents irregular appearance and complex dynamic changes. Reconstructing dynamic fluids such as smoke from real-world observation data can provide important technical support and theoretical basis for fire warning and emergency rescue, climate and environmental monitoring, scientific experiments and engineering simulations, film and game special effects, and effectively promote the efficiency of related fields in scientific decision-making and practical applications. However, unlike rigid objects with a certain shape and simple motion mode, smoke fluids not only have non-rigid complex change characteristics, but also often show obvious translucency and highly irregular appearance in image modality, resulting in the lack of global consistency in the reconstruction results of dynamic smoke scenes, missing details of non-smoke objects in the scene, and deviations in the density distribution or motion trajectory of smoke objects in local areas. It is impossible to truly reflect the motion state of smoke in the real environment, which in turn limits its promotion and use in related field practices. Therefore, how to perform high-fidelity three-dimensional reconstruction of dynamic smoke scenes based on image observation data is the main challenge currently faced.

[0003] In response to this challenge, considering that fluid motion strictly follows the laws of physics, the core focus of related research can be summarized as how to solve multiple physical quantities of the flow field, namely, velocity field, pressure field and density field, from real-world observation data. Early dynamic smoke scene reconstruction was mainly based on the fluid reconstruction method of computational fluid dynamics (CFD). However, this method requires a lot of computing resources and professional knowledge to assist in setting the boundary conditions of the calculation. Different boundary conditions and grid division may significantly affect the accuracy of the results. The above limitations restrict this type of method when dealing with actual scenes. Recently, the fluid reconstruction method based on physical perception neural network has provided a more flexible and efficient solution and achieved significant performance improvement by utilizing the potential physical law constraints and integrating physical knowledge into deep learning technology. It has introduced a new paradigm in the field of computer vision, namely, exploring how to use physical laws to enhance visual tasks. This method mainly takes observable image modalities as input, fits fluid behavior by learning the fluid motion pattern in the data, and uses the potential physical law constraints to achieve accurate solutions for dynamic fluids. It is no longer limited by precise boundary conditions and environmental requirements, and is combined with the visual reconstruction framework to achieve an end-to-end dynamic smoke scene reconstruction solution.

[0004] However, the fluid reconstruction method based on physical perception neural network still faces severe challenges, mainly manifested in the following aspects: (1) The reconstruction time is long. Existing studies directly use physical conservation laws as constraints to directly solve the pressure field and velocity field and use them to update the density field. However, the pressure field and velocity field in actual scenes are tightly coupled, and direct solution usually leads to numerical instability and difficulty in convergence; (2) The accuracy is limited in complex scenes. There are a large number of multi-scale features in dynamic smoke scenes, especially small-scale structures in turbulence, which makes the imposed physical rule constraints unable to correctly handle the energy dissipation effect. In particular, small-scale features may cause excessive concentration or loss of energy, forming non-physical degenerate solutions; (3) There is ambiguity in the appearance of non-fluid reconstructed objects. Smoky fluids often cause atmospheric absorption and scattering, which easily blurs the boundaries and color distribution of non-fluid objects surrounded by smoke, resulting in poor fidelity of the reconstructed non-fluid objects, such as missing texture details and shape ambiguity. Summary of the invention

[0005] The present invention aims at the problems existing in the prior art and proposes a method and system for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding. First, video images of the same smoke scene from different observation angles are collected, and the fluid perception parameters at each observation angle are calculated, which are constructed together with the corresponding smoke scene video as a reconstruction data set. The state estimation value of each pixel in the image is calculated using the fluid perception parameters, and the state estimation value is used as a priori supervision to pre-train a state estimation module in the reconstruction data set, and the trained state estimation module is used to distinguish between dynamic objects and static objects in the smoke scene. The static objects are sent to the static branch for processing, and a static target encoding and decoding module with time invariance is used to guide the feature learning of the 3D representation of the static objects in the smoke scene, which is used to estimate the volume density and appearance color of the static objects in the smoke scene. The dynamic objects are sent to the dynamic branch for processing. The dynamic branch is processed, and a dynamic target encoding and decoding module with time-varying characteristics is used to guide the feature learning of the three-dimensional representation of dynamic objects in the smoke scene, which is used to estimate the volume density, appearance color and velocity field of dynamic objects in the smoke scene; the fluid solution module is used to obtain the velocity field that conforms to the physical rule constraints from the volume density of static objects and the intermediate velocity field of dynamic objects, and is used to update the velocity field and volume density of dynamic objects; the output results of the static branch and the dynamic branch are used to render the static object images and dynamic object images of the smoke scene under different viewing angles respectively, and the reconstructed images of the smoke scene under different viewing angles are obtained through the synthetic rendering module, and the image information and physical knowledge priors of the multi-view smoke scene video are used as supervisory constraints, and the module parameters are continuously iterated and optimized, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation from any viewing angle in the scene is supported. The method of the present invention can simultaneously realize the three-dimensional reconstruction and physical state estimation of the dynamic smoke scene, and reconstruct a high-fidelity three-dimensional scene that conforms to physical characteristics from multi-view observation images.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a three-dimensional reconstruction method of smoke scene based on dual-branch physical knowledge embedding, comprising the following steps:

[0007] S1. Data collection and processing: Collect video images from different observation angles in the same smoke scene, calculate the fluid perception parameters at each observation angle, and construct a reconstructed data set together with the corresponding smoke scene video; the fluid perception parameters at least include atmospheric transmittance and time-varying variance;

[0008] S2, pre-training of state estimation module: calculating the state estimation value of each pixel in the image according to the fluid perception parameters obtained in step S1, using the state estimation value as a priori supervision, pre-training the state estimation module in the reconstructed data set, and obtaining a trained state estimation module to distinguish dynamic objects from static objects in the smoke scene; for static objects, proceed to step S3; for dynamic objects, proceed to step S4;

[0009] S3, static branch processing: The static objects obtained after the differentiation in step S2 are sent to the static branch for processing, and the feature learning of the three-dimensional representation of the static objects in the smoke scene is guided by the static target encoding and decoding module with time invariance to estimate the volume density and appearance color of the static objects in the smoke scene;

[0010] S4, dynamic branch processing: the dynamic objects obtained after the differentiation in step S2 are sent to the dynamic branch for processing, and the feature learning of the three-dimensional representation of the dynamic objects in the smoke scene is guided by the dynamic target encoding and decoding module with time-varying characteristics to estimate the volume density, appearance color and velocity field of the dynamic objects in the smoke scene;

[0011] S5, dynamic object feature update: through the fluid solution module, the velocity field that meets the physical rule constraints is obtained from the static object volume density obtained in step S3 and the dynamic object intermediate velocity field obtained in step S4, so as to update the velocity field and volume density of the dynamic object;

[0012] S6, synthetic rendering: rendering static object images and dynamic object images of the smoke scene at different viewing angles respectively according to the output results of the static branch of step S3 and the dynamic branch of step S5, and reconstructing images at different viewing angles through a synthetic rendering module;

[0013] S7. Iterative optimization: Using the image information and physical knowledge priors of multi-perspective smoke scene videos as supervisory constraints, the parameters of the static target encoding and decoding module, the dynamic target encoding and decoding module, and the fluid solution module are continuously iteratively optimized to obtain an optimized three-dimensional dynamic smoke scene and realize image generation from any perspective in the scene.

[0014] As an improvement of the present invention, the atmospheric transmittance in the fluid sensing parameter in step S1 is obtained by dark channel prior estimation, specifically:

[0015]

[0016] In the formula, p u is the pixel coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, ε is p u The local block centered on I c is the brightness value corresponding to RGB channel c;

[0017] The time-varying variance is calculated as follows:

[0018]

[0019] Where D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channels.

[0020] As an improvement of the present invention, the state estimation value in step S2 sets a transmittance threshold and a variance threshold to screen image pixels that simultaneously satisfy the atmospheric transmittance lower than a preset transmittance and the time-varying variance greater than the variance threshold. The state estimation value of the image pixel is set to 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; the remaining image pixels are set to 0, indicating that all the light sampling space corresponding to the pixel are static objects.

[0021] As another improvement of the present invention, in step S3, the volume density and appearance color calculation process of the static object in the smoke scene is specifically as follows:

[0022]

[0023] In the formula, σ s is the volume density of the static object, p v is the three-dimensional position coordinate of the static object in the static branch, c s is the apparent color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

[0024] As another improvement of the present invention, in step S4, the volume density, appearance color and velocity field calculation process of the dynamic object in the smoke scene are specifically as follows:

[0025]

[0026] In the formula, σ d is the volume density of the dynamic object, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, d is the observation direction, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

[0027] As another improvement of the present invention, in step S5, according to the pressure field calculated by the pressure Poisson equation, the velocity field of the dynamic object is updated to a velocity field that meets the incompressible fluid motion constraint, and the updated velocity field is used to calculate the volume density of the updated dynamic object using the semi-Lagrangian method. The specific formula is as follows:

[0028]

[0029] In the formula, u * is the velocity field before updating, u is the velocity field after updating, p is the pressure field, σ is the volume density before updating, is the updated volume density, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, t is the current time step, Δt is the time step, is the gradient operator.

[0030] As a further improvement of the present invention, in the synthesis rendering module of step S5, the dynamic and static object images are synthesized to obtain a reconstructed image of the smoke scene under each observation angle. The specific calculation formula is as follows:

[0031]

[0032] In the formula, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, N is the number of light sampling points, T i is the cumulative transmittance at the current position, δ i is the sampling interval of the current position point, σ s and c s are the volume density and appearance color of static objects, respectively. and c d They are the volume density and appearance color of dynamic objects, respectively.

[0033] In order to achieve the above object, the present invention also adopts a technical solution: a smoke scene three-dimensional reconstruction method system based on dual-branch physical knowledge embedding, which at least includes a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthesis rendering module.

[0034] The state estimation module: uses a binary cross entropy loss function for pre-training to distinguish dynamic objects from static objects in a smoke scene; the pre-training must meet the following requirements: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should all be 0, indicating that all objects in the light sampling space corresponding to the pixel are static objects; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel;

[0035] The static branch is used to process static objects in the smoke scene, and at least includes a static target encoding and decoding module composed of three axial feature grids XY / YZ / XZ and an appearance decoder. The static target encoding and decoding module takes the three-dimensional position coordinates as input and obtains the volume density of the corresponding position through trilinear interpolation calculation; the appearance decoder is composed of a multi-layer perceptron containing multiple fully connected layers, and calculates the appearance color of the corresponding position;

[0036] The dynamic branch is used to process dynamic objects in the smoke scene, and at least includes a dynamic target encoding and decoding module composed of six axial feature grids XY / ZT / XZ / YT / TZ / XT, an appearance decoder and a speed decoder. The dynamic target encoding and decoding module takes the three-dimensional position coordinates and the corresponding time information as input, and calculates the corresponding volume density by projecting the coordinate points onto each feature plane respectively; the appearance decoder and the speed decoder are both composed of a multi-layer perceptron containing multiple fully connected layers, and respectively calculate the appearance color and velocity field of the corresponding position;

[0037] The fluid solving module is used to obtain a velocity field that complies with physical rules and constraints, and to update the velocity field and volume density of the dynamic object, and at least includes a pressure field solver and a flow solver;

[0038] The pressure field solver: adopts a multi-scale convolutional architecture, takes the volume density of static objects and the velocity field of dynamic objects as input, and solves the pressure field of the smoke scene;

[0039] The flow solver: takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to the physical characteristics according to the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method;

[0040] The synthetic rendering module uses a volume rendering method to obtain static object images and dynamic object images of the smoke scene according to the volume density and appearance color of the static object and the dynamic object, calculates the transmittance of each position point in the three-dimensional space according to the volume density, and renders to obtain a reconstructed image of the smoke scene under each observation angle.

[0041] As an improvement of the present invention, the loss function of the state estimation model is:

[0042]

[0043] In the formula, p v is the three-dimensional position coordinate, p u is the two-dimensional pixel plane coordinate, S(p u ) is the state estimation module in the two-dimensional pixel plane coordinate p u The output value of is the three-dimensional position coordinate p of the state estimation module v The output value of N is the number of light sampling points, L s Estimate the loss for the state.

[0044] As another improvement of the present invention, a projection loss function is used for training the pressure field solver. The loss function uses the divergence of the velocity field under the pressure projection to satisfy the incompressible physical property of the fluid. The specific formula is as follows:

[0045]

[0046] Where u is the velocity field updated by the fluid solution module, u * is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The method of the present invention proposes to use a state estimation module to distinguish between dynamic objects and static objects in a smoke scene, and to use fluid perception parameters with physical priors to accurately perceive the motion states of different objects in the smoke scene, thereby achieving effective decoupling of dynamic and static objects and greatly reducing the computational complexity of the reconstruction algorithm.

[0049] (2) The method of the present invention utilizes a dual-branch structure to process dynamic and static objects, and introduces a fluid solution module embedded with physical knowledge into the dynamic branch. The module adopts a multi-scale convolutional architecture, which can effectively capture the global information and local details of dynamic objects and eliminate the degradation problem of visual ambiguity in smoke scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a working principle diagram of the smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding of the present invention;

[0051] Figure 2 These are example images of different observation angles in two smoke scenes in Example 2 of the present invention;

[0052] Figure 3 It is a flowchart of the steps of the method for three-dimensional reconstruction of smoke scenes based on dual-branch physical knowledge embedding of the present invention;

[0053] Figure 4 This is a diagram showing the effect of Embodiment 2 of the present invention after passing through the state estimation module;

[0054] Figure 5 Schematic diagram of the model structure of the pressure field solver in the system of the present invention;

[0055] Figure 6 This is a diagram showing the effect of velocity field estimation in a smoke scene in Example 2 of the present invention;

[0056] Figure 7 This is a diagram showing the effects of static branches and dynamic branches in a smoke scene in Example 2 of the present invention;

[0057] Figure 8 A streamline diagram of the forward reasoning and iterative optimization process of the overall model of the present invention;

[0058] Fig. 9 This is a comparison chart of reconstruction results of different models in the test example of the present invention, where:

[0059] (A) is a schematic diagram of the results of NVFi, (B) is a schematic diagram of the results of PINF, (C) is a schematic diagram of the results of HyFluid, and (D) is a schematic diagram of the results of the method of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0061] Example 1

[0062] A smoke scene three-dimensional reconstruction system based on dual-branch physical knowledge embedding at least includes a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthesis rendering module.

[0063] The state estimation module is pre-trained using a binary cross entropy loss function, which must meet the following assumptions: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should be 0; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1. The trained state estimation module obtains its corresponding state estimation value by inputting the three-dimensional position coordinates, and divides the dynamic objects and static objects in the smoke scene according to the state estimation value.

[0064] The static branch is used to process static objects in the smoke scene, and at least includes a static target encoding and decoding module. The static target encoding and decoding module consists of three axial feature grids XY / YZ / XZ and an appearance decoder, which respectively contain the geometric features and appearance features of the static objects in the smoke scene. The appearance decoder is composed of a multi-layer perceptron containing multiple fully connected layers. The module takes the three-dimensional position coordinates as input, calculates the volume density of the corresponding position through trilinear interpolation, and calculates the appearance color of the corresponding position through the appearance decoder.

[0065] The dynamic branch is used to process dynamic objects in the smoke scene, and at least includes a dynamic target encoding and decoding module. The dynamic target encoding and decoding module contains six axial feature grids of XY / ZT / XZ / YT / TZ / XT, an appearance decoder and a velocity decoder, which respectively contain the geometric features, appearance features and velocity features of the dynamic objects in the smoke scene, wherein the appearance decoder and the velocity decoder are both composed of a multi-layer perceptron containing multiple fully connected layers; the module takes four-dimensional space coordinates (three-dimensional position coordinates and corresponding time information) as input, calculates the corresponding volume density by projecting the coordinate points onto each feature plane respectively, calculates the appearance color of the corresponding position through the appearance decoder, and calculates the velocity field of the corresponding position through the velocity decoder.

[0066] The fluid solving module includes a pressure field solver and a flow solver. The pressure field solver takes the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene. It adopts a multi-scale convolution architecture and captures the global information and local details of dynamic objects through multiple cascaded spatial downsampling encoding blocks and upsampling decoding blocks to eliminate the degradation problem of visual ambiguity in smoke scenes. The loss function uses the divergence of the velocity field under the pressure projection to meet the incompressible physical properties of the fluid; the flow solver takes the velocity field of dynamic objects and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to the physical properties based on the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of dynamic objects using the semi-Lagrangian method.

[0067] The synthetic rendering module uses volume rendering method to obtain static object images and dynamic object images of the smoke scene according to the volume density and appearance color of static objects and dynamic objects, and calculates the transmittance of each position point in the three-dimensional space according to the volume density, and renders the reconstructed image of the smoke scene under each observation angle.

[0068] The working principle of this system is as follows Figure 1 As shown in the figure, firstly, the fluid perception parameters are extracted and the state estimation module is used to distinguish the dynamic objects and static objects in the smoke scene. Then, the dual-branch structure of dynamic and static object separation is used to realize the feature learning and state estimation of dynamic and static objects respectively. The module parameters are continuously iteratively optimized through supervised constraints, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation from any perspective in the scene is supported.

[0069] Example 2

[0070] Using the system in Example 1, the input image of the model in this embodiment is obtained by collecting the same dynamic smoke scene from different observation angles by 6 cameras at fixed positions. The image size is 800×800 pixels, the total number of frames exceeds 130, the frame rate is 25FPS, and the images contained in the scene are as follows: Figure 2 As shown in the figure, there are both static obstacles and dynamic smoke. By extracting image information from different observation angles at different timestamps, image information from four angles is used for reconstruction, and image information from the remaining two angles is used for testing. Therefore, a three-dimensional reconstruction method for smoke scenes based on dual-branch physical knowledge embedding is proposed. Figure 3 As shown, the specific steps include:

[0071] Step S1: Collect video images from different observation angles in the same smoke scene, calculate the fluid perception parameters at each observation angle, and construct a reconstruction dataset together with the corresponding smoke scene video.

[0072] This embodiment includes observation images from 6 viewing angles, with an image size of 800×800 pixels, a total number of frames exceeding 130, and a frame rate of 25FPS. The fluid perception parameters at each observation angle are calculated and constructed together with the corresponding smoke scene video as a reconstructed data set. Among them, the fluid perception parameters include the atmospheric transmittance and time-varying variance of each pixel. The atmospheric transmittance is calculated as follows:

[0073]

[0074] In the formula, p u is the pixel coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, which is mainly obtained by selecting the pixel with the highest intensity through the statistical results of the dark channel, and ε is the value of p u The local block centered on (15×15 pixels in this example), I c is the brightness value corresponding to RGB channel c. The time-varying variance is calculated as follows:

[0075]

[0076] Where D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channels.

[0077] Step S2: Calculate the state estimation value of each pixel in the image using the fluid perception parameters under different observation angles, and use the state estimation value as a priori supervision to pre-train the state estimation module in the reconstructed data set. The trained state estimation module is used to distinguish dynamic objects from static objects in the smoke scene.

[0078] The state estimation value is calculated based on the fluid perception parameters. The state estimation value is a Boolean variable. By setting the transmittance threshold and variance threshold, the image pixels that simultaneously meet the fluid perception parameters and whose atmospheric transmittance is lower than the preset transmittance and whose time-varying variance is greater than the variance threshold are screened. The state estimation value is set to 1, which means that there is at least one dynamic object in the light sampling space corresponding to the pixel, and the rest are set to 0, which means that all the light sampling space corresponding to the pixel is static objects. The state estimation value of the pixel is calculated as follows:

[0079]

[0080] In the formula, S is the state estimate, θ d is the variance threshold, θ t is the transmittance threshold.

[0081] In this example, θ d Set to 15, θ t Set to 0.75, the effect is as follows Figure 4The three-dimensional position coordinates of the pixel corresponding to the light sampling are calculated as follows:

[0082] p v =K[R,T]p u

[0083] In the formula, p v is the three-dimensional position coordinate, K is the camera intrinsic parameter, and [R, T] is the camera extrinsic parameter including the rotation matrix and translation matrix.

[0084] The state estimation module uses a binary cross entropy loss function for pre-training, which needs to meet the following assumptions: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should be 0; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1. The loss function is as follows:

[0085]

[0086] In the formula, is the three-dimensional position coordinate p of the state estimation module v The output value of , N is the number of light sampling points.

[0087] In this embodiment, the state estimation module is trained for 50 cycles using the above loss function, with a batch size of 8, optimized using an adaptive momentum estimation optimizer, and the learning rate is set to 5e-3. The trained state estimation module is used to distinguish dynamic objects from static objects in the smoke scene.

[0088] Step S3: The static objects in the smoke scene are sent to the static branch for processing, and the static target encoding and decoding module with time invariance is used to guide the feature learning of the three-dimensional representation of the static objects in the smoke scene, which is used to estimate the volume density and appearance color of the static objects in the smoke scene. The calculation process is as follows:

[0089]

[0090] In the formula, σ s is the volume density of the static object, c s is the apparent color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

[0091] Step S4: The dynamic objects in the smoke scene are sent to the dynamic branch for processing, and a dynamic target encoding and decoding module with time-varying characteristics is used to guide the feature learning of the three-dimensional representation of the dynamic objects in the smoke scene, which is used to estimate the volume density, appearance color and velocity field of the dynamic objects in the smoke scene. The calculation process is as follows:

[0092]

[0093] In the formula, σ d is the volume density of the dynamic object, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

[0094] Step S5: Utilize the fluid solution module to obtain a velocity field that complies with physical rules from the volume density of the static object and the intermediate velocity field of the dynamic object, and use it to update the velocity field and volume density of the dynamic object.

[0095] The fluid solving module includes a pressure field solver and a flow solver. The pressure field solver uses the volume density of static objects and the velocity field of dynamic objects as input to solve the pressure field of the smoke scene. It adopts a multi-scale convolution architecture, such as Figure 5 As shown in the figure, multiple cascaded spatial downsampling encoding blocks and upsampling decoding blocks are used to capture the global information and local details of dynamic objects, effectively eliminating the degradation problem of visual ambiguity in smoke scenes. Specifically, the solver uses three scales of 1 / 4, 1 / 2 and 1, and the calculation formula is as follows:

[0096]

[0097] Where Y i (i=2) represents the pressure field output by the pressure field solver, U represents an upsampling operation, D represents a downsampling operation, and C represents a convolution and ReLU activation operation.

[0098] The projection loss function is used for training the pressure solver. The loss function uses the divergence of the velocity field under the pressure projection to satisfy the incompressible physical properties of the fluid. The specific formula is as follows:

[0099]

[0100] In the formula, u *is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

[0101] The flow solver takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to the physical characteristics according to the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method. The specific formula is as follows:

[0102]

[0103] Where u is the updated velocity field, is the updated volume density, t is the current time step, Δt is the time step, is the gradient operator. The extracted velocity field visualization effect is as follows Figure 6 shown.

[0104] Step S6: using the static object encoding module and the dynamic object encoding module to render the static object images and the dynamic object images of the smoke scene at different viewing angles respectively, and obtaining the reconstructed images of the smoke scene at different viewing angles through the synthesis rendering module.

[0105] In the forward reasoning process, firstly, according to the output results of the static branch and the dynamic branch, the volume rendering method is used to obtain the static object images and dynamic object images of the smoke scene under different perspectives. Figure 7 Then, the synthetic rendering module is used to synthesize the dynamic and static object images, and the reconstructed image of the smoke scene under each observation angle is obtained. The specific calculation formula is as follows:

[0106]

[0107] In the formula, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, T is the cumulative transmittance of the current position, and N is the number of light sampling points.

[0108] Step S7: Using the image information and physical knowledge priors of the multi-perspective smoke scene video as supervision constraints, the module parameters are continuously iteratively optimized, and finally a high-fidelity three-dimensional dynamic smoke scene is reconstructed, and image generation from any perspective in the scene is supported.

[0109] The process of steps S6 and S7 is as follows Figure 8 As shown in the figure, the blue solid line represents the forward reasoning process, and the red dotted line represents the iterative optimization process of the module parameters based on the supervision constraints.

[0110] During the optimization process, the image information and physical knowledge priors of the multi-view smoke scene video are used as supervisory constraints to continuously iterate and optimize the module parameters. The supervisory constraints consist of two parts: image difference and physical knowledge priors. The image difference is used to minimize the difference between the real observed image and the reconstructed image obtained by the synthetic rendering module. The physical knowledge prior is used to constrain the reconstructed velocity field and the volume density of dynamic objects to comply with the laws of conservation of mass and momentum in fluid motion, so as to reconstruct a high-fidelity dynamic smoke scene that follows the laws of fluid physics. The specific calculation is as follows:

[0111] L=β1L c +β2L vel +β3L dens

[0112]

[0113] Where, L c is the color loss function between the reconstructed image and the real image, L vel is the velocity loss function that complies with the law of conservation of momentum in fluid motion, L dens is the density loss function that complies with the mass conservation in fluid motion. β is a hyperparameter.

[0114] In this example, the hyperparameters are set to β1=100, β2=0.1, and β3=0.1. The Adam optimizer is used for training for 15,000 rounds, the batch size is 2048, and the spatial resolution is set to 128. 3 , the learning rate is set to 1e-4. After training, a high-fidelity three-dimensional dynamic smoke scene can be reconstructed, and image generation from any perspective in the scene is supported.

[0115] Figure 7 The reconstruction effect diagram of the static branch and the dynamic branch in the smoke scene is displayed, indicating that the present invention can accurately reconstruct the static objects and the dynamic objects from the smoke scene.

[0116] Test Case

[0117] Two different dynamic smoke scenes were selected for testing. Both scenes contained dynamic smoke and static obstacles, and had 6 fixed-position shooting cameras, numbered 1 to 6, with a shooting image size of 800×800 pixels, a total number of frames of 150, and a frame rate of 25FPS. For different models, the image data of cameras 1 to 5 were selected as the reconstruction data set, and the image data of camera 6 was selected as the test data set. Different models used the same reconstruction data set for scene reconstruction. When different models were fully converged, the reconstructed images of different models at the position of camera 6 were rendered and generated, and visually compared with the images actually taken by camera 6.

[0118] Fig. 9 Comparison of visualization results reconstructed by different models, where: Fig. 9 (A) is a schematic diagram of NVFi results. Fig. 9 (B) is a schematic diagram of the PINF results. Fig. 9 (C) is a schematic diagram of the results of HyFluid. Fig. 9 (D) is a schematic diagram of the result of the method of the present invention. From the comparison in the figure, it can be seen that the density and appearance of the smoke of the present invention are more consistent with the real view, and the appearance reconstruction of the static obstacle under the smoke is clearer and more accurate. Compared with other models, the reconstruction result of the present invention has higher fidelity, which shows that the present invention can reconstruct a high-fidelity three-dimensional dynamic smoke scene and support image generation from any perspective in the scene.

[0119] In summary, the present invention proposes an innovative 3D reconstruction method to achieve high-fidelity 3D reconstruction of dynamic smoke scenes while accurately estimating the physical state. First, the state estimation module is used to distinguish dynamic objects from static objects in the smoke scene. Then, the dual-branch structure of dynamic and static object separation is used to respectively achieve feature learning and state estimation of dynamic and static objects. Finally, a high-fidelity 3D dynamic smoke scene is reconstructed, and image generation from any perspective in the scene is supported. Multi-perspective smoke scene videos are used to achieve 3D reconstruction and physical state estimation of dynamic smoke scenes, and high-fidelity 3D scenes that conform to physical characteristics are reconstructed from multi-view observation images.

[0120] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A 3D reconstruction method for smoke scenes based on dual-branch physical knowledge embedding, characterized in that: The steps include: S1. Data collection and processing: Collect video images from different observation angles in the same smoke scene, calculate the fluid perception parameters at each observation angle, and construct a reconstructed data set together with the corresponding smoke scene video; the fluid perception parameters at least include atmospheric transmittance and time-varying variance; S2, pre-training of state estimation module: calculating the state estimation value of each pixel in the image according to the fluid perception parameters obtained in step S1, using the state estimation value as a priori supervision, pre-training the state estimation module in the reconstructed data set, and obtaining a trained state estimation module to distinguish dynamic objects from static objects in the smoke scene; For static objects, proceed to step S3; For dynamic objects, go to step S4; S3, static branch processing: The static objects obtained after the differentiation in step S2 are sent to the static branch for processing, and the feature learning of the three-dimensional representation of the static objects in the smoke scene is guided by the static target encoding and decoding module with time invariance to estimate the volume density and appearance color of the static objects in the smoke scene; S4, dynamic branch processing: the dynamic objects obtained after the differentiation in step S2 are sent to the dynamic branch for processing, and the feature learning of the three-dimensional representation of the dynamic objects in the smoke scene is guided by the dynamic target encoding and decoding module with time-varying characteristics to estimate the volume density, appearance color and velocity field of the dynamic objects in the smoke scene; S5, dynamic object feature update: through the fluid solution module, the velocity field that meets the incompressible fluid motion constraint is obtained from the static object volume density obtained in step S3 and the dynamic object intermediate velocity field obtained in step S4, so as to update the velocity field and volume density of the dynamic object; S6, synthetic rendering: rendering static object images and dynamic object images of the smoke scene at different viewing angles respectively according to the output results of the static branch of step S3 and the dynamic branch of step S5, and reconstructing images at different viewing angles through a synthetic rendering module; S7. Iterative optimization: Using the image information and physical knowledge priors of multi-perspective smoke scene videos as supervisory constraints, the parameters of the static target encoding and decoding module, the dynamic target encoding and decoding module, and the fluid solution module are continuously iteratively optimized to obtain the optimized three-dimensional dynamic smoke scene, realizing image generation from any perspective in the scene.

2. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: The atmospheric transmittance in the fluid sensing parameter of step S1 is obtained by dark channel prior estimation, specifically: In the formula, p u is the two-dimensional pixel plane coordinate, is the atmospheric transmittance, J dark is the dark channel prior, A c is atmospheric light, ε is p u The local block centered on I c is the brightness value corresponding to RGB channel c; The time-varying variance is calculated as follows: In the formula, p u is the two-dimensional pixel plane coordinate, D is the time-varying variance, T is the time length, and I is the average brightness value of the RGB channel.

3. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 2, characterized in that: The state estimation value in step S2 sets a transmittance threshold and a variance threshold to screen image pixels that simultaneously satisfy the atmospheric transmittance lower than a preset transmittance and a time-varying variance greater than the variance threshold. The state estimation value of the image pixel is set to 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; the remaining image pixels are set to 0, indicating that all the light sampling spaces corresponding to the pixels are static objects.

4. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In step S3, the volume density and appearance color calculation process of the static object in the smoke scene is as follows: In the formula, σ s is the volume density of the static object, p v is the three-dimensional position coordinate of the static object in the static branch, c s is the apparent color of the volume object, d is the observation direction, θ s,σ,r is the density feature grid of the three axes XY / YZ / XZ, θ s,c,r is the appearance feature grid of the three axes XY / YZ / XZ, F s For the appearance decoder.

5. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In step S4, the volume density, appearance color and velocity field calculation process of the dynamic object in the smoke scene are specifically as follows: In the formula, σ d is the volume density of the dynamic object, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, c d is the appearance color of the dynamic object, u * is the velocity of the dynamic object, d is the observation direction, θ d,σ,r is the density feature grid of the six axes XY / ZT / XZ / YT / TZ / XT, θ d,c,r is the appearance feature grid, θ d,u,r is the speed feature network, F d is the appearance decoder, F u For the speed decoder.

6. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 4 or 5, characterized in that: In step S5, according to the pressure field calculated by the pressure Poisson equation, the velocity field of the dynamic object is updated to a velocity field that meets the incompressible fluid motion constraint, and the updated velocity field is used to calculate the volume density of the dynamic object using the semi-Lagrangian method. The specific formula is as follows: In the formula, u * is the velocity field before updating, u is the velocity field after updating, p is the pressure field, σ is the volume density before updating, is the updated volume density, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, t is the current time step, Δt is the time step, is the gradient operator.

7. The method for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding according to claim 1, characterized in that: In the synthesis rendering module of step S5, the dynamic and static object images are synthesized to obtain a reconstructed image of the smoke scene at each observation angle. The specific calculation formula is as follows: In the formula, I s and I d are static object images and dynamic object images respectively, I f is the reconstructed image after synthesis, N is the number of light sampling points, T i is the cumulative transmittance at the current position, δ i is the sampling interval of the current position point, σ s and c s are the volume density and appearance color of static objects, respectively. and c d They are the volume density and appearance color of dynamic objects, respectively.

8. A method system for 3D reconstruction of smoke scenes based on dual-branch physical knowledge embedding using the method as claimed in claim 1, characterized in that: At least includes a state estimation module, a static branch, a dynamic branch, a fluid solution module and a synthetic rendering module, The state estimation module: uses a binary cross entropy loss function for pre-training to distinguish dynamic objects from static objects in a smoke scene; the pre-training must meet the following requirements: when the state estimation value of an image pixel is 0, the state estimation values ​​of all three-dimensional position coordinates obtained from the pixel through light sampling should all be 0, indicating that all objects in the light sampling space corresponding to the pixel are static objects; when the state estimation value of an image pixel is 1, at least one of the state estimation values ​​of the three-dimensional position coordinates obtained from the pixel through light sampling should be 1, indicating that there is at least one dynamic object in the light sampling space corresponding to the pixel; The static branch is used to process static objects in the smoke scene, and at least includes a static target encoding and decoding module composed of three axial feature grids of XY / YZ / XZ and an appearance decoder. The static target encoding and decoding module takes the three-dimensional position coordinates as input and obtains the volume density of the corresponding position through trilinear interpolation calculation; The appearance decoder is composed of a multi-layer perceptron including multiple fully connected layers, and calculates the appearance color of the corresponding position; The dynamic branch is used to process dynamic objects in the smoke scene, and at least includes a dynamic target encoding and decoding module composed of six axial feature grids XY / ZT / XZ / YT / TZ / XT, an appearance decoder and a speed decoder. The dynamic target encoding and decoding module takes the three-dimensional position coordinates and the corresponding time information as input, and calculates the corresponding volume density by projecting the coordinate points onto each feature plane respectively; the appearance decoder and the speed decoder are both composed of a multi-layer perceptron containing multiple fully connected layers, and respectively calculate the appearance color and velocity field of the corresponding position; The fluid solving module is used to obtain a velocity field that complies with physical rules and constraints, and to update the velocity field and volume density of the dynamic object, and at least includes a pressure field solver and a flow solver; The pressure field solver: adopts a multi-scale convolutional architecture, takes the volume density of static objects and the velocity field of dynamic objects as input, and solves the pressure field of the smoke scene; The flow solver: takes the velocity field of the dynamic object and the pressure field output by the pressure field solver as input, calculates and updates the velocity field that conforms to the physical characteristics according to the pressure Poisson equation, and uses the updated velocity field to calculate and update the volume density of the dynamic object using the semi-Lagrangian method; The synthetic rendering module uses a volume rendering method to obtain static object images and dynamic object images of the smoke scene according to the volume density and appearance color of the static object and the dynamic object, calculates the transmittance of each position point in the three-dimensional space according to the volume density, and renders to obtain a reconstructed image of the smoke scene under each observation angle.

9. The smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding as claimed in claim 8, characterized in that: The loss function of the state estimation model is: In the formula, p v is the three-dimensional position coordinate of the dynamic object in the dynamic branch, p u is the two-dimensional pixel plane coordinate, S(p u ) is the state estimation module in the two-dimensional pixel plane coordinate p u The output value of is the three-dimensional position coordinate p of the dynamic object in the dynamic branch of the state estimation module v The output value of N is the number of light sampling points, L s Estimate the loss for the state.

10. The smoke scene 3D reconstruction system based on dual-branch physical knowledge embedding as claimed in claim 8, characterized in that: The projection loss function is used for training the pressure field solver. The loss function uses the divergence of the velocity field under the pressure projection to meet the incompressible physical properties of the fluid. The specific formula is as follows: Where u is the velocity field updated by the fluid solution module, u * is the velocity field output by the dynamic target encoding and decoding module, p is the pressure field output by the pressure field solver, represents the divergence operator and Δt is the time step.

Images