Real fluid reconstruction evolution method and system based on three-dimensional Gaussian

Abstract

The invention provides a real fluid reconstruction evolution method and system based on three-dimensional Gaussian. The system comprises a fluid video reconstruction module, a connection module and a fluid simulation module. The fluid video reconstruction module collects and makes fluid smoke data, reconstructs the fluid smoke data by using a method based on three-dimensional Gaussian and physical constraint, and obtains a smoke dynamic scene; the connection module learns a Gaussian particle vorticity by using a micro-simulation method to obtain a simulation initial vorticity; and the fluid simulation module is used for filling Gaussian particles in the reconstructed smoke by using a grid method to obtain reconstructed Gaussian particles, performing physical simulation according to the simulation initial vorticity to obtain a simulation picture, and performing large model correction to obtain a smoke fluid reconstruction result. According to the method, the dynamic fluid can be quickly and completely reconstructed, the capability of further physical evolution is provided, the reconstruction result has better physical fidelity, and the method has high industrial value.

Description

Technical Field

[0001] The present invention belongs to the field of computer graphics, and specifically, relates to a real fluid reconstruction and evolution method and system based on three-dimensional Gaussian, especially a real fluid rapid reconstruction and evolution method based on three-dimensional Gaussian, physics and differentiable simulation. Background Art

[0002] In the field of scientific research, fluid reconstruction technology is crucial for a deep understanding of fluid mechanics phenomena. For example, fluid reconstruction can more accurately simulate and analyze complex fluid flows, providing more detailed data support for the study of turbulence, shock waves and other phenomena. In the field of engineering, industries such as aerospace, energy and power have extremely high requirements for the accuracy and efficiency of fluid mechanics. In addition, in the field of environmental monitoring, fluid reconstruction can help monitor changes in water flow, air flow, etc., and provide data support for environmental protection and disaster warning. In industries such as automobiles and ships, fluid reconstruction technology can optimize fluid dynamics performance, reduce energy consumption and improve operating efficiency. In the field of medical diagnosis, fluid reconstruction can be used to analyze blood flow, etc., to provide a reference for disease diagnosis and treatment.

[0003] At the same time, deep learning-based technologies have shown great application potential in fluid reconstruction. Deep learning models are used to solve basic equations of fluid mechanics, such as the Navier-Stokes equations, bringing new possibilities for the numerical solution of fluid mechanics equations. In terms of turbulence simulation, deep learning technology is used to improve turbulence models and improve the accuracy and efficiency of turbulence simulation. In addition, significant results have been achieved in flow feature recognition and classification, flow control and optimization, etc. Integrating learning frameworks into traditional computational fluid dynamics software to improve computational efficiency and accuracy has become one of the current research hotspots. On the other hand, developing neural network models that satisfy physical conservation laws is important for ensuring the physical consistency of simulation results. For example, deep learning can be used to predict and analyze shock waves in fluids and transition phenomena in boundary layers, which is of great significance in fields such as hypersonic vehicle design.

[0004] In the field of reconstruction, the competition between different technical schools is gradually emerging. With its strong learning ability and adaptability, neural network fluid reconstruction technology has advantages in dealing with complex scenes and even complex fluid flow problems. It can learn the characteristics and laws of fluids from a large amount of data to achieve high-precision scene and fluid reconstruction. In addition, the fusion of physical neural networks has also made certain progress in the field of fluid reconstruction. It can fuse data and equations, and does not need to divide fine grids during the solution process, realizing efficient fluid reconstruction with sparse space. However, neural network fluid reconstruction technology also faces problems such as high computational cost and the need to improve the generalization ability of the model. Three-dimensional Gaussian reconstruction technology performs well in some specific application scenarios with its efficient real-time rendering capability and explicit geometric structure. For example, in an environment with limited computing resources, 3DGS can render high-resolution images with low latency.

[0005] 3D Gaussian Splatting (3DGS) technology is a 3D scene construction method based on explicit representation, which has unique advantages in the field of fluid reconstruction. It does not rely on neural networks but uses Gaussian functions to represent 3D points, so that high-quality images can be generated in real time when performing multi-view synthesis. Compared with the common neural radiance field (NeRF) and neural graphics primitive (Instant-NGP) technologies, 3DGS shows higher efficiency and flexibility in real-time rendering and computing resource utilization. Its basic principle is to represent the 3D world with a set of 3D Gaussian ellipsoids, each of which is defined by attributes such as mean, transparency and color, which enables it to capture the details in the scene more accurately. By utilizing CUDA parallel computing, it not only speeds up the rendering speed, but also reduces the computing overhead, significantly improving the real-time processing capability. At present, 3DGS has shown far-reaching application potential in many fields such as autonomous driving, robot navigation and augmented reality. Although 3D Gaussian reconstruction technology has many advantages, its application in the field of fluid is very rare.

[0006] As a promising method in the field of fluid reconstruction, three-dimensional Gaussian fluid reconstruction technology inevitably faces many challenges in practical application scenarios. On the one hand, the difficulty in maintaining physical fidelity is a more prominent problem. In actual fluid systems, the movement and changes of fluids follow complex physical laws, such as the principles of conservation of mass, conservation of momentum, and conservation of energy. However, when reconstructing fluids, although the three-dimensional Gaussian model can capture the morphology and general flow trend of the fluid to a certain extent, it is not easy to accurately restore these physical properties. Due to the limitations of the model itself and factors such as approximate processing in the data processing process, the physical properties of the reconstructed fluid often deviate from those of the real fluid, making it difficult to achieve highly accurate physical fidelity. This will have a greater impact on the final results in some application scenarios that require extremely high physical accuracy, such as precise fluid simulation experiments in scientific research and key design links involving fluid mechanics principles in the engineering field. On the other hand, it is difficult for the reconstructed fluid to evolve further. After obtaining the fluid state at a certain moment or stage through three-dimensional Gaussian fluid reconstruction technology, if you want to predict and evolve the future fluid state based on the existing reconstruction results, you will encounter many obstacles. This is because the movement of the fluid is highly complex and uncertain, and its subsequent development is affected by a combination of factors, including but not limited to slight changes in initial conditions, interactions within the fluid, etc. In the process of constructing the three-dimensional Gaussian reconstruction model, although it can reconstruct the current fluid more reasonably based on the existing data, it is insufficient in the comprehensive consideration and accurate modeling of these complex factors, making it difficult to accurately infer the state changes of the fluid at subsequent moments based on the existing reconstruction results, thereby limiting the effective application of this technology in application scenarios that require fluid state prediction and dynamic analysis.

[0007] Patent document "A fluid simulation method based on video reconstruction and Euler model coupling" (CN107085629A) discloses a method that can reconstruct the density field and velocity field of the fluid with high accuracy, tightly couple the reconstructed data with the fluid geometric model, obtain a fluid animation effect that is closer to the real situation, and add controllable fluid details. However, the method is based on dynamic video for simulation and does not provide a method for further evolving the simulation results.

[0008] Patent document "Dynamic reverse gas stacking model of a portable chemical detection device for locating threats and point sources from effluent streams" (US201562243530) discloses a method for triangulating the source of one or more target molecules or the possible location of the effluent based on a dynamic reverse gas stacking model of one or more target molecules.

[0009] Patent document "System and method for detecting the source of molecules, and computer readable medium" (CN108473305B) uses traditional methods to predict the source position by assuming that the target molecule follows a Gaussian distribution after flowing from the source position. However, the Gaussian distribution used is a macroscopic concept in a statistical sense, a reverse reasoning of fluid motion, and does not involve Gaussian reconstruction, differentiable simulation, and physical simulation technologies.

[0010] Therefore, how to combine the advantages of learning-based neural network methods and three-dimensional Gaussian methods to achieve reconstruction and evolution of fluid phenomena has become an urgent problem to be solved. The present invention provides a real fluid reconstruction and evolution method and system based on three-dimensional Gaussian to solve the problems of low physical fidelity of fluid reconstruction and poor further evolution effect after reconstruction in the prior art. Summary of the invention

[0011] In view of the defects in the prior art, the object of the present invention is to provide a method and system for reconstructing the evolution of real fluid based on three-dimensional Gaussian.

[0012] A three-dimensional Gaussian-based real fluid reconstruction evolution method provided by the present invention comprises:

[0013] Step S1: collecting and producing fluid smoke data;

[0014] Step S2: reconstruct the fluid smoke data using a method based on three-dimensional Gaussian and physical constraints to obtain a smoke dynamic scene;

[0015] Step S3: using the smoke dynamic scene to learn the Gaussian particle vorticity using a differentiable simulation method to obtain the simulated initial vorticity;

[0016] Step S4: using a grid method to fill the Gaussian particles in the reconstructed smoke to obtain reconstructed Gaussian particles;

[0017] Step S5: Performing physical simulation on the reconstructed Gaussian particles according to the simulated initial vorticity to obtain a simulation image;

[0018] Step S6: Use the large model to correct the simulation image to obtain the smoke fluid reconstruction result.

[0019] Preferably, the step S1 comprises:

[0020] Step S1.1: Select the real smoke physical data captured by photography technology provided by ScalarFlow smoke dataset, including the velocity field and density field of smoke. Use OpenVDB library to convert the density field of NPZ format data into VDB format smoke.

[0021] Step S1.2: Use Blender to build a dynamic scene to process smoke in VDB format, and set up light sources and cameras;

[0022] Step S1.3: Rendering images from different angles in a spherical area using dynamic smoke in a spherical surround manner, while outputting the camera's position to obtain the three-dimensional coordinates of the Gaussian particle position;

[0023] Step S1.4: Set a corresponding timestamp for each image, with the timestamp uniformly sampled in the range of 0 to 1, and output the smoke data.

[0024] The step S2 comprises:

[0025] Step S2.1: read in smoke data, encode the three-dimensional coordinates and timestamp using the 6-plane spatiotemporal encoder in the four-dimensional Gaussian, and divide the four-dimensional data into six two-dimensional planes;

[0026] Step S2.2: Use a multi-head deformation decoder to output variable correction values ​​of position deformation and scale deformation, and obtain variable values ​​of final particle position and time according to the three-dimensional coordinates and timestamp variables in the smoke data;

[0027] Step S2.3: Use the physical constraint based on velocity field as the image L1 loss and total variational loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion and the multi-head deformation decoder;

[0028] Step S2.4: After the training is completed, the image smoke data is obtained through the GS rendering method to form a video and obtain a dynamic smoke scene.

[0029] Preferably, step S3 comprises:

[0030] Step S3.1: Set the initial vorticity variable of the Gaussian particles in the Nth frame from the end.

[0031] Step S3.2: Calculate the velocity from the vorticity in the next N-1 frames using the Biot-Savart formula using the vortex method

[0032]

[0033] Among them, u(x) represents the velocity field at the position x;

[0034] ω i represents the vorticity of the i-th Gaussian particle;

[0035] Γ i represents the vorticity intensity of the i-th Gaussian particle;

[0036] (xx i) represents the distance between the x position and the i-th Gaussian particle.

[0037] Step S3.3: Use the calculated velocity to convect the Gaussian particle using the RK3 formula, k1 = u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=ftn+Δt,xn-k1+2k2Δt, xn+1=xn+16(k1+4k2+k3);

[0038] Among them, t n represents the current time step;

[0039] Δt represents the time step;

[0040] x n Indicates a known location;

[0041] f() represents a function of time t and physical quantity x;

[0042] k1 represents the slope at the beginning of the time period;

[0043] k2 and k3 represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period, respectively;

[0044] x n+1 Represents the position of the next time step.

[0045] Step S3.4: Render the subsequent N-1 frames of Gaussian particles after convection to obtain subsequent rendered images for supervised training, and optimize the initial vorticity through differentiable simulation to obtain the optimized vorticity.

[0046] Step S3.5: Through supervised training, the optimized vorticity and subsequent rendered images are used to optimize the vorticity of the Nth last frame to obtain the simulation initial vorticity.

[0047] Preferably, step S4 comprises:

[0048] Step S4.1: Discretize the entire three-dimensional space using a regular grid and calculate the density of the grid based on the number and opacity of the Gaussian particles at the center of the grid.

[0049] Among them, δ p Indicates the opacity of Gaussian particle p;

[0050] x represents the position of the grid center;

[0051] x p represents the position of each Gaussian particle p;

[0052] Σ p represents the covariance of Gaussian particle p;

[0053] Step S4.2: Emit rays along the Cartesian coordinate direction or a random direction, and determine the number of intersections between the ray and the boundary of the object. If the number of intersections is greater than a threshold, it is considered to be inside, and a Gaussian particle with the same color attribute as the nearest Gaussian particle and vorticity set to 0 is added. If the number of intersections is less than or equal to the threshold, it is considered to be outside and no Gaussian particle is filled.

[0054] Traverse all grid centers to obtain reconstructed Gaussian particles;

[0055] The boundary of the object is a dividing line between a grid below a set threshold and a grid above the set threshold.

[0056] The step S5 comprises:

[0057] Step S5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles;

[0058] Step S5.2: Calculate the velocity of the vortex particles and the new vorticity according to the vorticity using the Biot-Savart formula;

[0059] Step S5.3: Convection of the vortex particles using the RK3 formula according to the velocity to obtain the new position of the vortex particles;

[0060] Repeat steps S5.2 to S5.3, and judge the vorticity and the set maximum vorticity threshold every N frames. If the absolute value of the vorticity is greater than the maximum vorticity threshold, the vortex particle splits into two vortex particles to the offset position. The two vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum vorticity threshold, no operation is performed until all frames are traversed to obtain a simulated image.

[0061] Preferably, step S6 comprises:

[0062] Step S6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as optimizable parameters, use the vortex method for simulation calculation, and render it into a 2D picture.

[0063] Step S6.2: Input the scene statement and call the StableDiffusion model of the open platform Huggingface to generate a series of scene pictures at different perspectives and times.

[0064] Step S6.3: Call the SDS tool, use fractional distillation sampling to extract 3D knowledge from 2D images and series scene images, calculate the loss function and compare it with the simulation image, and obtain the gradient optimized initial vorticity.

[0065] Step S6.5: Optimize the initial vorticity according to the gradient to perform physical simulation on the simulation image to obtain the smoke fluid reconstruction result.

[0066] A three-dimensional Gaussian-based real fluid reconstruction and evolution system provided by the present invention includes: a fluid video reconstruction module, a connection module and a fluid simulation module.

[0067] The fluid video reconstruction module collects and produces fluid smoke data, and reconstructs the fluid smoke data using a method based on three-dimensional Gaussian and physical constraints to obtain a dynamic smoke scene;

[0068] The connection module uses the smoke dynamic scene to learn the Gaussian particle vorticity using the differentiable simulation method to obtain the simulation initial vorticity;

[0069] The fluid simulation module uses a grid method to fill the Gaussian particles in the reconstructed smoke to obtain the reconstructed Gaussian particles. It performs physical simulation based on the initial vorticity of the simulation to obtain a simulation image and corrects it through a large model to obtain the smoke fluid reconstruction result.

[0070] Preferably, the fluid video reconstruction module comprises:

[0071] Module M1.1: Select the real smoke physical data captured by photography technology provided by ScalarFlow smoke dataset, including the velocity field and density field of smoke. Use OpenVDB library to convert the density field of NPZ format data into VDB format smoke.

[0072] Module M1.2: Use Blender to build a dynamic scene, process smoke in VDB format, and set up light sources and cameras;

[0073] Module M1.3: Render images from different angles in a spherical area using dynamic smoke in a spherical surround mode, and output the camera's position to obtain the three-dimensional coordinates of the Gaussian particle position;

[0074] Module M1.4: Set the corresponding timestamp for each picture, the timestamp is uniformly sampled between 0 and 1, and output the smoke data.

[0075] Module M2.1: Read smoke data, encode 3D coordinates and timestamps using a 6-plane spatiotemporal encoder in a 4D Gaussian, and divide the 4D data into 6 2D planes;

[0076] Module M2.2: Use a multi-head deformation decoder to output variable correction values ​​of position deformation and scale deformation, and obtain the variable values ​​of final particle position and time according to the three-dimensional coordinates and timestamp variables in the smoke data;

[0077] Module M2.3: Use velocity field-based physical constraints as image L1 loss and total variational loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion, and the multi-head deformation decoder;

[0078] Module M2.4: After training, the image smoke data is obtained through the GS rendering method to form a video and obtain a dynamic smoke scene.

[0079] Preferably, the connection module comprises:

[0080] Module M3.1: Set the initial vorticity variables of the Gaussian particles in the Nth frame from the end.

[0081] Module M3.2: Using the vortex method, the Biot-Savart formula is used to calculate the velocity from the vorticity in the next N-1 frames.

[0082]

[0083] Among them, u(x) represents the velocity field at the position x;

[0084] ω i represents the vorticity of the i-th Gaussian particle;

[0085] Γ i represents the vorticity intensity of the i-th Gaussian particle;

[0086] (xx i ) represents the distance between the x position and the i-th Gaussian particle.

[0087] Module M3.3: Convection of Gaussian particles using the RK3 formula using the calculated velocity, k1 = u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=ftn+Δt,xn-k1+2k2Δt, xn+1=xn+16(k1+4k2+k3);

[0088] Among them, t n represents the current time step;

[0089] Δt represents the time step;

[0090] x n Indicates a known location;

[0091] f() represents a function of time t and physical quantity x;

[0092] k1 represents the slope at the beginning of the time period;

[0093] k2 and k3 represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period, respectively;

[0094] x n+1 Represents the position of the next time step.

[0095] Module M3.4: Based on the subsequent N-1 frames of Gaussian particle rendering after convection, the subsequent rendered images are obtained for supervised training, and the initial vorticity can be optimized by differentiable simulation to obtain the optimized vorticity.

[0096] Module M3.5: Through supervised training, the optimized vorticity and subsequent rendered images are used to optimize the vorticity of the Nth last frame to obtain the initial vorticity of the simulation.

[0097] Preferably, the fluid simulation module comprises:

[0098] Module M4.1: Discretize the entire 3D space using a regular grid and calculate the density of the grid based on the number and opacity of Gaussian particles at the center of the grid.

[0099] Among them, δ p Indicates the opacity of Gaussian particle p;

[0100] x represents the position of the grid center;

[0101] x p represents the position of each Gaussian particle p;

[0102] Σ p represents the covariance of Gaussian particle p;

[0103] Module M4.2: Emit rays along the Cartesian coordinate direction or a random direction, determine the number of intersections between the ray and the boundary of the object, if the number of intersections is greater than the threshold, it is considered to be inside, and a new Gaussian particle with the same color attribute as the nearest Gaussian particle and vorticity set to 0 is added; if the number of intersections is less than or equal to the threshold, it is considered to be outside and no Gaussian particle is filled;

[0104] Traverse all grid centers to obtain reconstructed Gaussian particles;

[0105] The boundary of the object is a dividing line between a grid below a set threshold and a grid above the set threshold.

[0106] Module M5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles;

[0107] Module M5.2: Use the Biot-Savart formula to calculate the velocity of vortex particles and the new vorticity based on the vorticity;

[0108] Module M5.3: Use the RK3 formula to convect the vortex particles according to the velocity and obtain the new position of the vortex particles;

[0109] Repeat the triggering of modules M5.2 and M5.3, and judge the vorticity and the set maximum vorticity threshold every N frames. If the absolute value of the vorticity is greater than the maximum vorticity threshold, the vortex particle splits into two vortex particles at the offset position. The two vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum vorticity threshold, no operation is performed until all frames are traversed to obtain a simulation image.

[0110] Preferably, the fluid simulation module comprises:

[0111] Module M6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as optimizable parameters, use the vortex method to simulate and calculate, and render it into a 2D picture.

[0112] Module M6.2: Input the scene statement and call the StableDiffusion model of the open platform Huggingface to generate a series of scene pictures at different perspectives and times.

[0113] Module M6.3: Call the SDS tool, use fractional distillation sampling to extract 3D knowledge from 2D images and series scene images, calculate the loss function and compare it with the simulation images, and obtain the gradient optimized initial vorticity.

[0114] Module M6.4: Based on the gradient optimization of the initial vorticity, the physical simulation of the simulation image is performed to obtain the smoke fluid reconstruction result.

[0115] Compared with the prior art, the present invention has the following beneficial effects:

[0116] 1. The present invention combines promising three-dimensional Gaussian reconstruction, differentiable simulation, and large model technologies to provide a new framework that can quickly and completely reconstruct dynamic fluids and provide the ability to further physical evolution compared to existing reconstruction and simulation technologies, and has high industrial value.

[0117] 2. The present invention adopts a method based on three-dimensional Gaussian video reconstruction and a data set captured from real smoke, uses physical constraints as losses to improve learning effects, and efficiently reconstructs dynamic fluid scenes, so that the reconstruction results have better physical fidelity.

[0118] 3. The present invention adopts a learning method of differentiable physics to convert the velocity field and the vorticity field, learn the initial vorticity before the simulation starts, effectively connect the reconstruction module and the simulation module, optimize the initial vorticity, and lay the foundation for further driving subsequent simulations using the vortex method.

[0119] 4. The present invention adopts a fluid simulation method based on the vortex method, uses the vortex method to drive the Gaussian particle flow, considers Gaussian particle filling and vortex particle splitting, and then uses a large model to correct the physical properties of the fluid, effectively solving the problem of further evolution of the fluid after reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0121] Figure 1 It is a schematic diagram of the overall process of the real fluid reconstruction evolution method based on three-dimensional Gaussian;

[0122] Figure 2 Schematic diagram of the architecture of the real fluid reconstruction evolution system based on three-dimensional Gaussian;

[0123] Figure 3 To collect and produce fluid smoke data scenes and effect diagrams;

[0124] Figure 4 A schematic diagram of Gaussian particles and rendering effects for reconstructing smoke using a method based on 3D Gaussian and physical constraints;

[0125] Figure 5 A schematic diagram illustrating the sparse distribution of particles inside the Gaussian particles filled using the grid method;

[0126] Figure 6 This is a schematic diagram showing the comparison of the effect before and after filling Gaussian particles using the grid method;

[0127] Figure 7 A schematic diagram showing the comparison of vortex particle splitting before and after the physics-based smoke simulation;

[0128] Figure 8 This is a schematic diagram of the effect of the fluid video reconstruction module;

[0129] Fig. 9 Schematic diagram of the fluid simulation module effect. DETAILED DESCRIPTION

[0130] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0131] According to a three-dimensional Gaussian-based real fluid reconstruction evolution method provided by the present invention, three-dimensional Gaussian particles are used to dynamically reconstruct a three-dimensional fluid scene in stages according to a fluid video, and an evolution scheme after the video is cut off is provided. Figure 1 For example, it includes:

[0132] Step S1: collecting and producing fluid smoke data;

[0133] Step S1.1: Reconstructing fluid smoke requires reference smoke data, which is the basis for fluid reconstruction. Use a dataset of real smoke samples, and select and process the dataset into VDB format;

[0134] The process is to convert the density field of ScalarFlow's NPZ format data into VDB format using OpenVDB.

[0135] Step S1.2: Use Blender to build a dynamic scene to process VDB smoke, set up the light source camera and render.

[0136] Step S1.3: The spherical area samples dynamic smoke and outputs the camera pose, and renders the dynamic smoke into pictures from different angles by spherical surround, and outputs the camera pose at the same time.

[0137] Step S1.4: Set the timestamp, and set the timestamps between 0 and 1 uniformly sampled according to the number of frames for the data at different times.

[0138] In more preferred examples, the dynamic reconstruction method based on 3D Gaussian has certain requirements for the data used, so dynamic smoke images from multiple angles and corresponding camera poses are required. The current reconstruction field focuses on the reconstruction of solids rather than fluids. Fluid data is relatively scarce, and existing data cannot directly meet the reconstruction requirements. Therefore, fluid data for reconstruction is produced by ourselves.

[0139] The ScalarFlow smoke dataset was initially used, which provided real smoke physical data captured by photography technology. The dataset provided the velocity field and density field of the smoke, but the pictures it provided were only smoke pictures taken from a sparse perspective and could not meet the requirements of 3D Gaussian reconstruction.

[0140] Use the OpenVDB library to process the density field of ScalarFlow into a VDB file, further adjust the smoke in VDB format in Blender, set the light source and camera, and generate rendered images, camera poses and timestamps through spherical surround, such as Figure 3 As shown in Figure 2, the final dataset meets the training requirements of 3D Gaussian.

[0141] Step S2: reconstructing smoke using a method based on three-dimensional Gaussian and physical constraints;

[0142] The three-dimensional Gaussian function (elliptical Gaussian kernel) is used as the basic unit of the fluid, and this basic unit can be optimized and learned through techniques such as gradient descent. The dynamic video reconstruction method adopted is the optimized 4DGS scheme. The main difference between this scheme and 3DGS is that it has an additional time dimension. Previously, 3D Gaussians were mainly used for static scene reconstruction. Although it is crucial in the field of three-dimensional vision, dynamic scene rendering faces the challenge of complex motion modeling, and it is difficult to balance efficiency and accuracy. Therefore, 4DGS proposes the concept of time dimension in the field of dynamic scene reconstruction, combining 3D Gaussians and 4D neural voxels, and converting canonical 3D Gaussians into new positions and shapes through a Gaussian deformation field network, and using spatiotemporal structure encoders and multi-head Gaussian deformation decoders to achieve efficient rendering.

[0143] Specifically, dynamic reconstruction is performed based on three-dimensional Gaussian, and a method based on four-dimensional Gaussian (4DGS) is adopted to introduce the time dimension to reconstruct the dynamic flow field according to the video data. The specific steps include:

[0144] Step S2.1: Add the time dimension as input;

[0145] Use a 6-plane space-time encoder to encode 3D coordinates and timestamps. Read in data, encode Gaussian particle position, time and other variables using the space-time encoder, divide the 4D data into 6 2D planes, reduce storage consumption, and provide physical correlation between adjacent areas;

[0146] Step S2.2: Use a multi-head deformation decoder to output the deformation result. The multi-head decoder outputs the correction values ​​of variables such as position deformation and scale deformation, and obtains the final variable values ​​of particle position based on the initial variable values ​​of Gaussian particle position;

[0147] Step S2.3: Use physical constraints as losses to give physical meaning to Gaussian particles, and include image L1 loss and total variation loss to jointly train variables such as Gaussian position, HexPlane feature plane, MLP feature fusion and multi-head deformation decoder. Introduce physical constraints based on speed, add speed as a physical constraint to guide the learning of Gaussian particles, so that it has stronger physical meaning.

[0148] Step S2.4: After the training is completed, the image is obtained by the conventional GS rendering method. Figure 4 As an example, and compose a video.

[0149] In more preferred examples, compared with the implementation structure of 3DGS, the two most important differences are the spatiotemporal structure encoder and the multi-head Gaussian deformation decoder. The spatiotemporal structure encoder refers to the use of multi-resolution HexPlane and tiny MLP to decompose 4D neural voxels into 6 planes to effectively encode the spatiotemporal features of 3D Gaussians. The specific method is to encode the three-dimensional coordinates x, y, z and the time dimension t into 6 two-dimensional planes R i,j (xy, xz, yz, xt, yt, zt), the features stored on the plane can be obtained through learning. Then, according to the position and timestamp of each Gaussian particle, the interpolation of features is obtained from the 6 planes and fused together through a small MLP. The role of the spatiotemporal structure encoder is to effectively encode the spatiotemporal features of 3D Gaussians. Nearby 3D Gaussians always have similar spatiotemporal information. In this way, the features of 3D Gaussians can be effectively modeled, while reducing storage consumption compared to pure explicit methods. The formula for calculating the features of each voxel is:

[0150] f h =∪interp(R i,j )

[0151] At the same time, the multi-head Gaussian deformation decoder is similar to the multi-head attention mechanism, which uses the characteristics of the spatiotemporal encoder to use multiple MLPs to output the position, rotation and scale deformation of the 3D Gaussian respectively, and obtains the deformed 3D Gaussian physical quantity based on the initial physical quantity for rendering.

[0152] Introduce physical constraints to assist training. The spatiotemporal features learned by the three-dimensional Gaussian itself do not contain physical meanings, but are merely the mapping of the position and other features of the three-dimensional Gaussian particles in space and time. Use the velocity field as a physical quantity to constrain its reconstruction and add it to the training loss. Through supervised training of the velocity field, the Gaussian particles have C1 continuity in position, which has a clear physical meaning.

[0153] Step S3: learning Gaussian particle vorticity using a differentiable simulation method;

[0154] Differentiable simulation is an important research direction in the field of graphics. It means that in the simulation process, the dynamics and kinematics models of the system can be differentiated. The two key parts are differentiable models and automatic differentiation technology. Common differentiable models include Newton's mechanics equations, elasticity equations, etc., while automatic differentiation technology is provided by default in many deep learning or graphics frameworks. Due to the above characteristics, differentiable simulation can often be used in conjunction with deep learning technology.

[0155] Steps S1-S2 obtain a complete reconstruction of the smoke dynamic scene, but the reconstructed smoke asset can only end at the last frame of the video and cannot be further simulated. The vortex method is needed to further drive subsequent simulations. The vortex method requires a relatively good initialization vorticity. Therefore, first, the various properties of Gaussian particles are obtained through learning, and the initialization vorticity is obtained through differentiable simulation.

[0156] Specifically, the reconstructed dynamic Gaussian particles are used to learn the vorticity field of the last frame by differentiable simulation, while taking into account the conversion between vorticity and velocity. The Biot-Savart law is used to connect the Gaussian particle velocity of the reconstruction module and the vorticity of the simulation module.

[0157] In more preferred examples, video dynamic scene reconstruction mainly involves learning a series of frames, setting a physical quantity of vorticity for the Gaussian particles in the Nth frame from the end, and using the idea of ​​the vortex method for the subsequent N-1 frames, converting vorticity into velocity through the Biot-Savart formula, and using velocity for Runge-Kutta 3 (RK3) convection. This is equivalent to no longer using the spatiotemporal feature deformation mapping solution for the subsequent N frames, but using a differentiable simulation solution, and also obtaining a rendered image for each frame. Specifically, it includes:

[0158] Step S3.1: Set the initial vorticity variables of the Nth last frame for optimization.

[0159] Step S3.2: Calculate the velocity from the vorticity using the Biot-Savart formula;

[0160] The formula for the Biot-Savart vorticity to velocity conversion is as follows:

[0161]

[0162] Where u(x) represents the velocity field at position x, ω i represents the vorticity of the vortex particles, Γ i represents the vorticity intensity of vortex particles, (xx i ) represents the distance between the current position and the vortex particle. It should be noted that Gaussian particles carry vorticity properties, and the vortex particles in this step are actually Gaussian particles directly used as vortex particles. Therefore, in the content of the present invention, "vortex particles" can be equivalent to "Gaussian particles".

[0163] The physical meaning of this formula is that each vortex particle will generate an induced velocity field in space. By superimposing the induced velocities of all vortex particles, the velocity of the entire flow field at position x can be obtained. The existence of the vortex vector ω determines the direction and magnitude of the induced velocity, which is related to the position vector (x i) makes the direction of the velocity perpendicular to the plane formed by the vorticity and position difference, and the velocity magnitude is related to factors such as vorticity intensity and distance.

[0164] At the same time, in order to prevent the velocity field calculated by the learned Gaussian particles from interfering with the velocity field obtained by the vortex method, a small value of N is selected as much as possible, such as N=2, so that a reasonable context velocity relationship can be obtained.

[0165] Step S3.3: using the calculated velocity to convect the Gaussian particles using the RK3 formula;

[0166] The Gaussian particle is the Gaussian particle of the Nth frame from the last step S3.1. Meanwhile, the Runge-Kutta 3 method is often used to update the changes of relevant physical quantities of the fluid (such as velocity, position, etc.) over time. To update the change of position, assume that the current time step is t n , Δt is the time step, and the known position is x n , u(x) is the velocity calculated by the above Biot-Savart formula at position x, f is a function of time t and physical quantity x, and three slopes k1, k2, k3 need to be calculated, and finally the position x of the next time step is calculated n+1 , the specific formula is as follows:

[0167] k1=u(x n )Δt=f(t n ,x n )Δt

[0168]

[0169] k3=u(x n -k1+2k2)Δt=f(t n +Δt,x n -k1+2k2)Δt

[0170]

[0171] Among them, k1 is the slope at the beginning of the time period; k2 is the slope at the midpoint of the time period. The slope k1 is used by the Euler method to determine f(t,x) at point k3 is also the slope of the midpoint, but this time the slope k2 is used to determine the value of f(t,x).

[0172] The application of the RK3 method in convection problems can accurately update physical quantities to a certain extent. Its accuracy is between low-order (such as Euler method) and high-order (such as RK4) methods, providing a balance between computational efficiency and accuracy. It is suitable for some convection simulation scenarios that do not require extremely high accuracy but hope to be more accurate than low-order methods.

[0173] Step S3.4: The subsequent N-1 frames of rendered images participate in the supervised training of reconstruction, and the initial vorticity of the simulation module can be optimized by differentiable simulation to obtain the optimized vorticity.

[0174] Step S3.5: After the image is obtained by vorticity calculation and subsequent rendering, the vorticity of the Nth last frame of the initial setting is also optimized through supervision. This vorticity is used as the initial vorticity of the subsequent step S5 simulation, and is directly used as the initial value of the vortex particle at the beginning of the simulation.

[0175] The reconstructed position and the vorticity of the differentiable simulation are used as initial conditions, and the vortex method is used for further physical simulation. The grid method is used to fill Gaussian particles, and the particles are dynamically split based on the vorticity. At the same time, a large model is used to correct the unreasonable vorticity that does not conform to the fluid flow.

[0176] Step S4: Filling Gaussian particles using a grid method;

[0177] There is an inherent defect in the 3D Gaussian reconstruction scene, that is, the Gaussian particles reconstructed are mostly focused on the surface of the object, and the Gaussian particles inside the object are very sparse, which is also true for fluids. The sparse interior is not a problem for static scenes, but for dynamic scenes, especially fluid scenes, the Gaussian particles inside will flow to the surface through the movement of the fluid. If there are not enough particles inside, there will be voids, such as Figure 5 As shown, it is necessary to further fill the existing Gaussian particles through appropriate methods, including:

[0178] Step S4.1: Gridding the three-dimensional space, based on 3D Gaussian, maintaining an opacity attribute, discretizing the entire space using regular grids, and calculating the density of the grids based on the number and opacity of Gaussian particles at the centers of these grids.

[0179] Specifically, the fluid domain is first divided into 128 3 The grid space of each grid is weighted by the opacity of all Gaussian particles inside it to calculate the density of the grid center. The calculation formula is as follows:

[0180]

[0181] Among them, δ p is the opacity of the Gaussian particles, x is the center position of the grid, and x p is the position of each Gaussian particle, Σ p is the covariance of Gaussian particles. By calculating the opacity attribute of each grid, an opaque object with a boundary is constructed.

[0182] Step S4.2: Set a threshold to determine whether the grid density is greater than the threshold, determine whether the grid center is inside the object based on the grid density, insert particles at the grid center based on the determination result, and obtain the particle attributes from the nearest Gaussian particle.

[0183] Specifically, after obtaining the center density of each grid, a custom threshold can be set, so that the boundary of the object can be considered to be grids below the threshold and grids above the threshold on both sides. Then traverse all grid centers, emit rays along the Cartesian coordinate direction or a random direction, and judge whether the grid center is inside the object by the number of intersections between the rays and the boundary of the object. If it is inside, a new Gaussian particle is added, and its color and other attributes are the same as those of the nearest Gaussian particle, and the vorticity is set to 0. If it is outside, no new Gaussian particles are filled in the center of this grid. Without considering special cases, when emitting rays along the Cartesian coordinate direction, 6 intersections indicate that it is inside the object, that is, the threshold is set to 6, and the rest indicate that it is outside the object; when emitting rays in random directions, only one intersection is judged to be inside the object, and the rest indicate that it is outside the object. The effect after filling is completed is as follows Figure 6 For example.

[0184] Step S5: Perform smoke simulation based on a physical method;

[0185] Specifically, step S5.1: using Gaussian particles as vortex particles and using a vortex method to control the movement of the Gaussian particles, specifically including initializing the position and vorticity of the vortex particles;

[0186] Step S5.2: Calculate the velocity of the vortex particles and the new vorticity from the vorticity using the Biot-Savart law;

[0187] Step S5.3: Convection of the vortex particles using Runge-Kutta 3 according to the velocity to obtain the new position of the vortex particles;

[0188] The velocity, vorticity and position are calculated iteratively. According to the vorticity, the splitting of vortex particles is calculated every N frames according to the threshold.

[0189] In more preferred examples, the vortex method is mainly used to drive the reconstructed Gaussian particles to achieve the purpose of further evolution. The reconstructed Gaussian particles are regarded as vortex particles, and the vortex method is used as the basic physical method to drive the simulation. The vortex particle method is a numerical method for simulating fluid flow. It discretizes the vortex distribution of the fluid into a series of vortex particles, and simulates the behavior of the fluid by tracking the movement and interaction of vortex particles. In the vortex particle method, the vortex field of the fluid is discretized into a group of vortex particles, each of which has a certain vortex intensity, position and possible other properties. The source of the vortex particle method is the vortex quantification of the Navier-Stokes equations, which converts the original equation with velocity as the basic physical quantity into vortex. The specific formula is as follows:

[0190]

[0191] The first formula mainly indicates that the velocity is affected by factors such as pressure and gravity, and evolves with the velocity itself; the second formula indicates the non-divergence condition that the velocity needs to meet, which also corresponds to the incompressibility condition of the fluid; the third formula is the result of multiplying both sides of the Navier-Stokes equation by the curl operator, with the incompressibility condition embedded, indicating the evolution of vorticity. Compared with the traditional velocity-based numerical solution, the vortex method is essentially a discretization of vorticity, which allows vorticity to convect from a Lagrangian perspective; at the same time, because the vorticity is well maintained, it can retain more fluid details, thereby realizing more complex fluid phenomena.

[0192] The core of the vortex method is the Biot-Savart law, which assumes that vortex particles will generate induced velocity in space. Its main function is to convert the vorticity carried by vortex particles into velocity, thereby facilitating subsequent convection calculations. The formula is as follows:

[0193]

[0194] The numerical calculation steps of the vortex method are first to initialize the vortex particle vorticity, and the initial position and initial vorticity of the vortex particles need to be set; then a certain number of vortex particles are considered, and the velocity is calculated by the Biot-Savart law. It should be noted that due to the Lagrangian discrete perspective, the velocity only needs to be calculated at sparse positions, such as the position of vortex particles. After completing the velocity calculation, the RK3 convection method is used to calculate the convection of vortex particles with velocity, and the velocity is recalculated when it reaches the new position.

[0195] In addition to the traditional vortex particle method, the present invention also adds a vortex particle splitting method based on vorticity value. A maximum vorticity threshold is set. When the absolute value of the evolving vorticity exceeds the maximum threshold, the vortex particle will split into two vortex particles at a random and very small offset position. The two new vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl, scaling and other properties. This splitting method is executed once every N frames. N can be set according to user needs. In this embodiment, N=30 is set. The schematic diagram of the splitting is as follows: Figure 7 shown.

[0196] Step S6: Use the large model to correct simulation details.

[0197] The large model is one of the technologies with the most research and application value at present. Most of them are based on the Transformer structure, and the underlying structure follows the Scaling law, which means that the more model parameters and the deeper the model, the more accurate the model results. At the same time, it has demonstrated strong capabilities in multi-modal fusion. Therefore, the iterative optimization and feedback correction capabilities of the large model are used to modify the simulation details in an iterative manner.

[0198] Specifically, they include:

[0199] Step S6.1: vortex method simulation to obtain simulation images;

[0200] First, the vorticity of the initial vortex particles and the vorticity of the newly added particles at the beginning of the simulation are set as optimizable parameters, and the preliminary vortex method simulation calculation in step S5 is performed, and then rendered into a 2D image.

[0201] Step S6.2: Input a scene statement and call the big model of the open platform to generate a series of scene pictures;

[0202] In the absence of a 3D dataset, you can input text (such as "smoke rising") and call the StableDiffusion model of the open platform Huggingface to generate images from different perspectives and times.

[0203] Step S6.3: Use the vortex method to simulate and obtain a series of result images;

[0204] The Score Distillation Sampling (SDS) method is used to extract 3D knowledge from images. It is a technique used in machine learning, especially in the field of generative models, to generate samples from complex probability distributions. Converting text to 3D dynamic video requires accurately converting the semantic information in the text (including the movement of objects, changes in scenes, time sequence, etc.) into the feature representation of 3D dynamic video. Score distillation sampling can help optimize this mapping process. By learning a score function that can reflect the distribution of 3D dynamic video features, the model can more accurately adjust the 3D structure, object movement speed and direction and other dynamic features of each frame in the video according to the semantics of the text, so that the generated video is more in line with the intention of the text description.

[0205] The knowledge extraction of the pictures obtained by vortex method simulation and the knowledge extraction of the pictures generated by the large model using text prompts are calculated, and the loss functions of the two kinds of knowledge are calculated to achieve the purpose of optimization and correction. The optimization target is the initial vorticity of the vortex particles (including the original vortex particles and the newly added vortex particles in step S4). The gradient optimized initial vorticity is obtained.

[0206] Step S6.4: Obtain smoke fluid reconstruction results based on gradient optimization of initial vorticity physical simulation.

[0207] The forward reasoning of fluid motion completely reconstructs the forward flow and further forward evolution of the fluid. The video reconstruction results of the smoke fluid are as follows: Figure 8 As shown in the figure, the further evolution results are as follows Fig. 9 The present invention can be integrated into any application to generate reusable smoke assets based on the provided smoke video, and can further evolve reasonably beyond the scope of the video to present rich fluid phenomena.

[0208] The present invention also provides a three-dimensional Gaussian-based real fluid reconstruction and evolution system, which can be implemented by executing the process steps of the three-dimensional Gaussian-based real fluid reconstruction and evolution method, that is, those skilled in the art can understand the three-dimensional Gaussian-based real fluid reconstruction and evolution method as a preferred implementation of the three-dimensional Gaussian-based real fluid reconstruction and evolution system.

[0209] According to a three-dimensional Gaussian real fluid reconstruction evolution system provided by the present invention, Figure 2 For example, it specifically includes: a fluid video reconstruction module, a connection module and a fluid simulation module.

[0210] These three modules perform their respective functions and work together to solve the problems that arise in 3DGS fluid reconstruction and subsequent processing from different angles and levels, thereby improving the overall reconstruction effect and evolution capability.

[0211] The fluid video reconstruction module collects and produces fluid smoke data, and reconstructs smoke using a method based on 3D Gaussian and physical constraints. This module is mainly based on 3D Gaussian, and quickly and accurately reconstructs dynamic fluids by adding physical constraints.

[0212] Specifically, we use a real smoke dataset captured by ScalarFlow; a dynamic fluid reconstruction framework using a four-dimensional Gaussian; and a video learning scheme using physical constraints.

[0213] The connection module uses a differentiable simulation method to learn the vorticity of Gaussian particles. The purpose of this module is to connect the velocity of the reconstructed Gaussian particles and the vorticity of the subsequent simulation, and provide reasonable initialization conditions.

[0214] Specifically, it includes a method for converting a velocity field into a vorticity field; and uses a vorticity initialization method based on a differentiable fluid simulation.

[0215] The fluid simulation module uses a grid method to fill Gaussian particles, a physics-based method to simulate smoke, and a large model to correct simulation details. This module provides further physical evolution results of the fluid in addition to video reconstruction.

[0216] Specifically, it includes fluid simulation based on vortex method; grid-based Gaussian particle filling method; particle splitting method based on vortex control; and fluid detail correction based on large model.

[0217] In more preferred embodiments, the fluid video reconstruction module includes:

[0218] Module M1.1: Select the real smoke physical data captured by photography technology provided by ScalarFlow smoke dataset, including the velocity field and density field of smoke. Use OpenVDB library to convert the density field of NPZ format data into VDB format smoke.

[0219] Module M1.2: Use Blender to build a dynamic scene, process smoke in VDB format, and set up light sources and cameras;

[0220] Module M1.3: Render images from different angles in a spherical area using dynamic smoke in a spherical surround mode, output the camera's position, and obtain the three-dimensional coordinates of the Gaussian particle position;

[0221] Module M1.4: Set the corresponding timestamp for each picture, the timestamp is uniformly sampled between 0 and 1, and output the smoke data.

[0222] Module M2.1: Read smoke data, encode 3D coordinates and timestamps using a 6-plane spatiotemporal encoder in a 4D Gaussian, and divide the 4D data into 6 2D planes;

[0223] Module M2.2: Use a multi-head deformation decoder to output variable correction values ​​of position deformation and scale deformation, and obtain the variable values ​​of final particle position and time according to the three-dimensional coordinates and timestamp variables in the smoke data;

[0224] Module M2.3: Use velocity field-based physical constraints as image L1 loss and total variational loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion, and the multi-head deformation decoder;

[0225] Module M2.4: After training, the image smoke data is obtained through the GS rendering method to form a video and obtain a dynamic smoke scene.

[0226] In more preferred embodiments, the connection module includes:

[0227] Module M3.1: Set the initial vorticity variables of the Gaussian particles in the Nth frame from the end.

[0228] Module M3.2: Using the vortex method, the Biot-Savart formula is used to calculate the velocity from the vorticity in the next N-1 frames.

[0229]

[0230] Among them, u(x) represents the velocity field at the position x;

[0231] ω i represents the vorticity of the i-th Gaussian particle;

[0232] Γ i represents the vorticity intensity of the i-th Gaussian particle;

[0233] (xx i ) represents the distance between the x position and the i-th Gaussian particle.

[0234] Module M3.3: Convection of Gaussian particles using the RK3 formula using the calculated velocity, k1 = u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=ftn+Δt,xn-k1+2k2Δt, xn+1=xn+16(k1+4k2+k3);

[0235] Among them, t n represents the current time step;

[0236] Δt represents the time step;

[0237] x nIndicates a known location;

[0238] f() represents a function of time t and physical quantity x;

[0239] k1 represents the slope at the beginning of the time period;

[0240] k2 and k3 represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period, respectively;

[0241] x n+1 Represents the position of the next time step.

[0242] Module M3.4: Based on the subsequent N-1 frames of Gaussian particle rendering after convection, the subsequent rendered images are obtained for supervised training, and the initial vorticity can be optimized by differentiable simulation to obtain the optimized vorticity.

[0243] Module M3.5: Through supervised training, the optimized vorticity and subsequent rendered images are used to optimize the vorticity of the Nth last frame to obtain the initial vorticity of the simulation.

[0244] In more preferred embodiments, the fluid simulation module includes:

[0245] Module M4.1: Discretize the entire 3D space using a regular grid and calculate the density of the grid based on the number and opacity of Gaussian particles at the center of the grid.

[0246] Among them, δ p Indicates the opacity of Gaussian particle p;

[0247] x represents the position of the grid center;

[0248] x p represents the position of each Gaussian particle p;

[0249] Σ p represents the covariance of Gaussian particle p;

[0250] Module M4.2: Emit rays along the Cartesian coordinate direction or a random direction, determine the number of intersections between the ray and the boundary of the object, if the number of intersections is greater than the threshold, it is considered to be inside, and a new Gaussian particle with the same color attribute as the nearest Gaussian particle and vorticity set to 0 is added; if the number of intersections is less than or equal to the threshold, it is considered to be outside and no Gaussian particle is filled;

[0251] Traverse all grid centers to obtain reconstructed Gaussian particles;

[0252] The boundary of the object is a dividing line between a grid below a set threshold and a grid above the set threshold.

[0253] Module M5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles;

[0254] Module M5.2: Use the Biot-Savart formula to calculate the velocity of vortex particles and the new vorticity based on the vorticity;

[0255] Module M5.3: Use the RK3 formula to convect the vortex particles according to the velocity and obtain the new position of the vortex particles;

[0256] Repeat the triggering of modules M5.2 and M5.3, and judge the vorticity and the set maximum vorticity threshold every N frames. If the absolute value of the vorticity is greater than the maximum vorticity threshold, the vortex particle splits into two vortex particles at the offset position. The two vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum vorticity threshold, no operation is performed until all frames are traversed to obtain a simulation image.

[0257] In more preferred embodiments, the fluid simulation module includes:

[0258] Module M6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as optimizable parameters, use the vortex method to simulate and calculate, and render it into a 2D picture.

[0259] Module M6.2: Input the scene statement and call the StableDiffusion model of the open platform Huggingface to generate a series of scene pictures at different perspectives and times.

[0260] Module M6.3: Call the SDS tool, use fractional distillation sampling to extract 3D knowledge from 2D images and series scene images, calculate the loss function and compare it with the simulation images, and obtain the gradient optimized initial vorticity.

[0261] Module M6.4: Based on the gradient optimization of the initial vorticity, the physical simulation of the simulation image is performed to obtain the smoke fluid reconstruction result.

[0262] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0263] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A real fluid reconstruction evolution method based on three-dimensional Gaussian, characterized in that: include: Step S1: collecting and producing fluid smoke data; Step S2: reconstruct the fluid smoke data using a method based on three-dimensional Gaussian and physical constraints to obtain a smoke dynamic scene; Step S3: using the smoke dynamic scene to learn the Gaussian particle vorticity using a differentiable simulation method to obtain the simulated initial vorticity; Step S4: using a grid method to fill the Gaussian particles in the reconstructed smoke to obtain reconstructed Gaussian particles; Step S5: Performing physical simulation on the reconstructed Gaussian particles according to the simulated initial vorticity to obtain a simulation image; Step S6: Use the large model to correct the simulation image to obtain the smoke fluid reconstruction result.

2. The three-dimensional Gaussian-based real fluid reconstruction evolution method according to claim 1, characterized in that: The step S1 comprises: Step S1.1: Select the real smoke physical data captured by photography technology provided by ScalarFlow smoke dataset, including the velocity field and density field of smoke. Use OpenVDB library to convert the density field of NPZ format data into VDB format smoke. Step S1.2: Use Blender to build a dynamic scene to process smoke in VDB format, and set up light sources and cameras; Step S1.3: Rendering images from different angles in a spherical area using dynamic smoke in a spherical surround manner, while outputting the camera's position to obtain the three-dimensional coordinates of the Gaussian particle position; Step S1.4: Set a corresponding timestamp for each image, the timestamp is uniformly sampled between 0 and 1, and output smoke data; The step S2 comprises: Step S2.1: read in smoke data, encode the three-dimensional coordinates and timestamp using the 6-plane spatiotemporal encoder in the four-dimensional Gaussian, and divide the four-dimensional data into six two-dimensional planes; Step S2.2: Use a multi-head deformation decoder to output variable correction values ​​of position deformation and scale deformation, and obtain variable values ​​of final particle position and time according to the three-dimensional coordinates and timestamp variables in the smoke data; Step S2.3: Use the physical constraint based on velocity field as the image L1 loss and total variational loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion and the multi-head deformation decoder; Step S2.4: After the training is completed, the image smoke data is obtained through the GS rendering method to form a video and obtain a dynamic smoke scene.

3. The real fluid reconstruction evolution method based on three-dimensional Gaussian according to claim 1, characterized in that: The step S3 comprises: Step S3.1: setting the initial vorticity variable of the Gaussian particle in the Nth frame from the end; Step S3.2: Calculate the velocity from the vorticity in the next N-1 frames using the Biot-Savart formula using the vortex method Among them, u(x) represents the velocity field at the position x; ω i represents the vorticity of the i-th Gaussian particle; Γ i represents the vorticity intensity of the i-th Gaussian particle; (xx i ) represents the distance between the x position and the i-th Gaussian particle; Step S3.3: Use the calculated velocity to convect the Gaussian particle using the RK3 formula, k1 = u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=ftn+Δt,xn-k1+2k2Δt, xn+1=xn+16(k1+4k2+k3); Among them, t n represents the current time step; Δt represents the time step; x n Indicates a known location; f() represents a function of time t and physical quantity x; k1 represents the slope at the beginning of the time period; k2 and k3 represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period, respectively; x n+1 Indicates the position of the next time step; Step S3.4: Rendering the subsequent N-1 frames of Gaussian particles after convection to obtain subsequent rendered images for supervised training, and optimizing the initial vorticity through differentiable simulation to obtain the optimized vorticity; Step S3.5: Through supervised training, the optimized vorticity and subsequent rendered images are used to optimize the vorticity of the Nth last frame to obtain the initial vorticity of the simulation.

4. The three-dimensional Gaussian-based real fluid reconstruction evolution method according to claim 1, characterized in that: The step S4 comprises: Step S4.1: Discretize the entire three-dimensional space using a regular grid and calculate the density of the grid based on the number and opacity of the Gaussian particles at the center of the grid. Among them, δ p Indicates the opacity of Gaussian particle p; x represents the position of the grid center; x p represents the position of each Gaussian particle p; Σ p represents the covariance of Gaussian particle p; Step S4.2: Emit rays along the Cartesian coordinate direction or a random direction, and determine the number of intersections between the ray and the boundary of the object. If the number of intersections is greater than a threshold, it is considered to be inside, and a Gaussian particle with the same color attribute as the nearest Gaussian particle and vorticity set to 0 is added. If the number of intersections is less than or equal to the threshold, it is considered to be outside and no Gaussian particle is filled. Traverse all grid centers to obtain reconstructed Gaussian particles; The boundary of the object is a dividing line between a grid below a set threshold and a grid above the set threshold; The step S5 comprises: Step S5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles; Step S5.2: Calculate the velocity of the vortex particles and the new vorticity according to the vorticity using the Biot-Savart formula; Step S5.3: Convection of the vortex particles using the RK3 formula according to the velocity to obtain the new position of the vortex particles; Repeat steps S5.2 to S5.3, and judge the vorticity and the set maximum vorticity threshold every N frames. If the absolute value of the vorticity is greater than the maximum vorticity threshold, the vortex particle splits into two vortex particles to the offset position. The two vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum vorticity threshold, no operation is performed until all frames are traversed to obtain a simulated image.

5. The real fluid reconstruction evolution method based on three-dimensional Gaussian according to claim 1, characterized in that: The step S6 comprises: Step S6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as the optimizable parameters, use the vortex method to simulate and calculate, and render it into a 2D picture; Step S6.2: Input the scene statement and call the StableDiffusion model of the open platform Huggingface to generate a series of scene pictures at different perspectives and times; Step S6.3: Call the SDS tool, use fractional distillation sampling to extract 3D knowledge from 2D images and series scene images, calculate the loss function and compare it with the simulation image, and obtain the gradient optimized initial vorticity; Step S6.4: Optimize the initial vorticity according to the gradient to perform physical simulation on the simulation image to obtain the smoke fluid reconstruction result.

6. A real fluid reconstruction evolution system based on three-dimensional Gaussian, characterized in that: include: Fluid video reconstruction module, connection module and fluid simulation module; The fluid video reconstruction module collects and produces fluid smoke data, and reconstructs the fluid smoke data using a method based on three-dimensional Gaussian and physical constraints to obtain a dynamic smoke scene; The connection module uses the smoke dynamic scene to learn the Gaussian particle vorticity using the differentiable simulation method to obtain the simulation initial vorticity; The fluid simulation module uses a grid method to fill the Gaussian particles in the reconstructed smoke to obtain the reconstructed Gaussian particles. It performs physical simulation based on the initial vorticity of the simulation to obtain a simulation image and corrects it through a large model to obtain the smoke fluid reconstruction result.

7. The three-dimensional Gaussian-based real fluid reconstruction evolution system according to claim 6, characterized in that: The fluid video reconstruction module comprises: Module M1.1: Select the real smoke physical data captured by photography technology provided by ScalarFlow smoke dataset, including the velocity field and density field of smoke. Use OpenVDB library to convert the density field of NPZ format data into VDB format smoke. Module M1.2: Use Blender to build a dynamic scene, process smoke in VDB format, and set up light sources and cameras; Module M1.3: Render images from different angles in a spherical area using dynamic smoke in a spherical surround mode, output the camera's position, and obtain the three-dimensional coordinates of the Gaussian particle position; Module M1.4: Set the corresponding timestamp for each picture, the timestamp is uniformly sampled between 0 and 1, and output the smoke data; Module M2.1: Read smoke data, encode 3D coordinates and timestamps using a 6-plane spatiotemporal encoder in a 4D Gaussian, and divide the 4D data into 6 2D planes; Module M2.2: Use a multi-head deformation decoder to output variable correction values ​​of position deformation and scale deformation, and obtain the variable values ​​of final particle position and time according to the three-dimensional coordinates and timestamp variables in the smoke data; Module M2.3: Use velocity field-based physical constraints as image L1 loss and total variational loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion, and the multi-head deformation decoder; Module M2.4: After training, the image smoke data is obtained through the GS rendering method to form a video and obtain a dynamic smoke scene.

8. The three-dimensional Gaussian-based real fluid reconstruction evolution system according to claim 6, characterized in that: The connection module comprises: Module M3.1: Set the initial vorticity variable of Gaussian particles in the Nth frame from the end; Module M3.2: Using the vortex method, the Biot-Savart formula is used to calculate the velocity from the vorticity in the next N-1 frames. Among them, u(x) represents the velocity field at the position x; ω i represents the vorticity of the i-th Gaussian particle; Γ i represents the vorticity intensity of the i-th Gaussian particle; (xx i ) represents the distance between the x position and the i-th Gaussian particle; Module M3.3: Convection of Gaussian particles using the RK3 formula using the calculated velocity, k1 = u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=ftn+Δt,xn-k1+2k2Δt, xn+1=xn+16(k1+4k2+k3); Among them, t n represents the current time step; Δt represents the time step; x n Indicates a known location; f() represents a function of time t and physical quantity x; k1 represents the slope at the beginning of the time period; k2 and k3 represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period, respectively; x n+1 Indicates the position of the next time step; Module M3.4: According to the Gaussian particle rendering after convection, the subsequent rendering images are obtained for supervised training, and the initial vorticity can be optimized by differentiable simulation to obtain the optimized vorticity; Module M3.5: Through supervised training, the optimized vorticity and subsequent rendered images are used to optimize the vorticity of the Nth last frame to obtain the initial vorticity of the simulation.

9. The three-dimensional Gaussian-based real fluid reconstruction and evolution system according to claim 6, characterized in that: The fluid simulation module includes: Module M4.1: Discretize the entire 3D space using a regular grid and calculate the density of the grid based on the number and opacity of Gaussian particles at the center of the grid. Among them, δ p Indicates the opacity of Gaussian particle p; x represents the position of the grid center; x p represents the position of each Gaussian particle p; Σ p represents the covariance of Gaussian particle p; Module M4.2: Emit rays along the Cartesian coordinate direction or a random direction, determine the number of intersections between the ray and the boundary of the object, if the number of intersections is greater than the threshold, it is considered to be inside, and a new Gaussian particle with the same color attribute as the nearest Gaussian particle and vorticity set to 0 is added; if the number of intersections is less than or equal to the threshold, it is considered to be outside and no Gaussian particle is filled; Traverse all grid centers to obtain reconstructed Gaussian particles; The boundary of the object is a dividing line between a grid below a set threshold and a grid above the set threshold; Module M5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles; Module M5.2: Use the Biot-Savart formula to calculate the velocity of vortex particles and the new vorticity based on the vorticity; Module M5.3: Use the RK3 formula to convect the vortex particles according to the velocity and obtain the new position of the vortex particles; Repeat the triggering of modules M5.2 and M5.3, and judge the vorticity and the set maximum vorticity threshold every N frames. If the absolute value of the vorticity is greater than the maximum vorticity threshold, the vortex particle splits into two vortex particles at the offset position. The two vortex particles each carry half the vorticity and opacity of the original vortex particle, and inherit the original curl and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum vorticity threshold, no operation is performed until all frames are traversed to obtain a simulation image.

10. The three-dimensional Gaussian-based real fluid reconstruction evolution system according to claim 6, characterized in that: The fluid simulation module includes: Module M6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as the optimizable parameters, use the vortex method to simulate and calculate, and render it into a 2D picture; Module M6.2: Input the scene statement and call the StableDiffusion model of the open platform Huggingface to generate a series of scene pictures at different perspectives and times; Module M6.3: Call the SDS tool, use fractional distillation sampling to extract 3D knowledge from 2D images and series scene images, calculate the loss function and compare it with the simulation images, and obtain the gradient optimized initial vorticity; Module M6.4: Based on the gradient optimization of the initial vorticity, the physical simulation of the simulation image is performed to obtain the smoke fluid reconstruction result.

Images