Real fluid reconstruction evolution method and system based on three-dimensional gauss

Abstract

The application provides a three-dimensional Gaussian-based real fluid reconstruction evolution method and system, comprising a fluid video reconstruction module, a connection module and a fluid simulation module; the fluid video reconstruction module collects and produces fluid smoke data, and uses a three-dimensional Gaussian and physical constraint-based method to reconstruct the fluid smoke data to obtain a smoke dynamic scene; the connection module uses a differentiable simulation method to learn a Gaussian particle vorticity to obtain a simulation initial vorticity; the fluid simulation module uses a grid method to fill the reconstructed smoke with Gaussian particles to obtain reconstructed Gaussian particles, and performs physical simulation according to the simulation initial vorticity to obtain simulation pictures and corrects the simulation pictures through a large model to obtain a smoke fluid reconstruction result. The application can quickly and completely reconstruct a dynamic fluid and provide further physical evolution capability, so that the reconstruction result has better physical fidelity, and has high industrial value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer graphics, and specifically relates to a real fluid reconstruction evolution method and system based on three-dimensional Gauss, in particular to a real fluid rapid reconstruction and evolution method based on three-dimensional Gauss, physics and differentiable simulation. BACKGROUND

[0002] In the field of scientific research, fluid reconstruction technology is crucial for in-depth understanding of fluid mechanics phenomena. For example, through fluid reconstruction, complex fluid flow can be more accurately simulated and analyzed, providing more detailed data support for the study of phenomena such as turbulence and shock waves. In the engineering field, the accuracy and efficiency of fluid mechanics are extremely high in the aerospace, energy and power industries. In addition, in the field of environmental monitoring, fluid reconstruction can help monitor changes in water flow, air flow, etc., providing data support for environmental protection and disaster warning. In the industry, such as the design of automobiles, ships, etc., fluid reconstruction technology can optimize fluid dynamics performance, reduce energy consumption and improve operating efficiency. In the field of medical diagnosis, fluid reconstruction can be used to analyze blood flow, etc., providing a reference for disease diagnosis and treatment.

[0003] At the same time, deep learning-based technology has shown great application potential in fluid reconstruction. Deep learning models are used to solve the basic equations of fluid mechanics, such as the Navier-Stokes equation, bringing new possibilities for numerical solutions of fluid mechanics equations. In the aspect of turbulence simulation, deep learning technology is applied to improve the turbulence model, improving the accuracy and efficiency of turbulence simulation. In addition, significant achievements have been made in flow feature recognition and classification, flow control and optimization, etc. Integrating learning frameworks into traditional computational fluid dynamics software to improve computational efficiency and accuracy has become one of the current research hotspots. On the other hand, developing neural network models that meet the physical conservation laws is important to ensure the physical consistency of simulation results. For example, deep learning can be used to predict and analyze shock waves and boundary layer transition phenomena in fluids, which is of great significance in the field of hypersonic vehicle design.

[0004] In the field of reconstruction, different technical schools are gradually emerging. Neural network fluid reconstruction technology has advantages in dealing with complex scenes and even complex fluid flow problems due to its strong learning ability and adaptability. It can learn the characteristics and laws of fluids from a large amount of data, achieving high-precision scene and fluid reconstruction. In addition, the fusion of physical neural networks has made some progress in fluid reconstruction. It can fuse data and equations, and does not need to divide fine grids in the solving process, achieving efficient fluid reconstruction with spatial sparsity. However, neural network fluid reconstruction technology also faces problems such as high computational cost and the need to improve model generalization ability. Three-dimensional Gaussian reconstruction technology is excellent in some specific application scenarios due to its efficient real-time rendering capability and explicit geometric structure. For example, in environments with limited computing resources, 3DGS can achieve high-resolution image rendering with low latency.

[0005] Three-dimensional Gaussian (3D Gaussian Splatt ing, 3DGS) technology is a three-dimensional scene construction method based on explicit representation, which has unique advantages in fluid reconstruction. It does not rely on neural networks but uses Gaussian functions to represent three-dimensional points, enabling real-time generation of high-quality images when performing multi-view synthesis. Compared with common neural radiance field (NeRF) and neural graphics primitive (Instant-NGP) technology, 3DGS exhibits higher efficiency and flexibility in real-time rendering and computing resource utilization. Its basic principle is to represent the three-dimensional world with a set of three-dimensional Gaussian ellipsoids, each defined by mean, transparency, and color attributes, which allows it to capture scene details more accurately. By utilizing CUDA parallel computing, it not only speeds up rendering but also reduces computational overhead, significantly improving real-time processing capabilities. Currently, 3DGS has shown profound application potential in autonomous driving, robot navigation, and augmented reality. Although three-dimensional Gaussian reconstruction technology has many advantages, its application in the fluid field is very rare.

[0006] As a promising method in the field of fluid reconstruction, three-dimensional Gaussian fluid reconstruction technology inevitably faces many challenges in practical application scenarios. On the one hand, maintaining physical fidelity is a prominent problem. In actual fluid systems, the movement and change of fluid follow complex physical laws, such as the principles of conservation of mass, momentum, and energy. However, when reconstructing fluid, the three-dimensional Gaussian model can capture the shape and general flow trend of fluid to some extent, but it is not easy to accurately restore these physical properties. Due to the limitations of the model itself and the approximate processing in the data processing process, the reconstructed fluid often deviates from the real fluid in terms of physical properties, and it is difficult to achieve high-precision physical fidelity, which will have a great impact on the final results in some applications that require high physical accuracy, such as precise fluid simulation experiments in scientific research and key design steps involving fluid mechanics principles in engineering. On the other hand, it is difficult to further evolve the reconstructed fluid. When the fluid state at a certain time or stage is obtained through three-dimensional Gaussian fluid reconstruction technology, it is difficult to predict and evolve the future state of the fluid based on the existing reconstruction results. This is because the movement of fluid has high complexity and uncertainty, and its subsequent development is influenced by many factors, including but not limited to small changes in initial conditions and interactions within the fluid. Although the three-dimensional Gaussian reconstruction model can reasonably reconstruct the current fluid based on existing data, it lacks comprehensive consideration and accurate modeling of these complex factors, making it difficult to accurately predict the state change of fluid at subsequent times based on the existing reconstruction results, thereby limiting the effective application of the technology in applications that require fluid state prediction and dynamic analysis.

[0007] The patent document "Fluid simulation method based on coupling of video reconstruction and Euler model" (CN107085629A) discloses a method that can reconstruct the density field and velocity field of fluid with high precision, tightly couple reconstruction data with fluid geometric model, obtain more realistic fluid animation effects, and add controllable fluid details. However, it is based on dynamic video simulation, and does not provide a method for further evolution of the simulation results.

[0008] The patent document "Dynamic reverse gas stack model for portable chemical detection device to locate threats and point sources from effluent streams" (US201562243530) discloses a method for triangulating the possible location of the source or effluent of one or more target molecules based on a dynamic reverse gas stack model of the one or more target molecules.

[0009] Patent document 'System and method for detecting the source of molecules, and computer readable medium' (CN108473305B) uses traditional methods to predict the source location by assuming that the target molecule meets the Gaussian distribution after flowing from the source location. However, the Gaussian distribution used is a statistical macroscopic concept, which is a reverse reasoning of fluid motion, and does not involve Gaussian reconstruction, differentiable simulation, and physical simulation technologies.

[0010] Therefore, how to combine the advantages of learning-based neural network methods and three-dimensional Gaussian methods to realize the reconstruction and evolution of fluid phenomena has become a problem to be solved. The present application provides a real fluid reconstruction and evolution method and system based on three-dimensional Gaussian, which solves the problems of low physical fidelity of fluid reconstruction and poor further evolution effect after reconstruction in the prior art. SUMMARY

[0011] In view of the defects in the prior art, the purpose of the present application is to provide a real fluid reconstruction and evolution method and system based on three-dimensional Gaussian.

[0012] According to the real fluid reconstruction and evolution method based on three-dimensional Gaussian provided by the present application, the following steps are included:

[0013] Step S1: collecting and preparing fluid smoke data;

[0014] Step S2: using a method based on three-dimensional Gaussian and physical constraints to reconstruct the fluid smoke data, and obtaining a smoke dynamic scene;

[0015] Step S3: using the smoke dynamic scene to learn the Gaussian particle vorticity using a differentiable simulation method, and obtaining a simulation initial vorticity;

[0016] Step S4: using a grid method to fill the Gaussian particles in the reconstructed smoke, and obtaining reconstructed Gaussian particles;

[0017] Step S5: performing physical simulation on the reconstructed Gaussian particles according to the simulation initial vorticity, and obtaining a simulation picture;

[0018] Step S6: using a large model to correct the simulation picture, and obtaining a smoke fluid reconstruction result.

[0019] Preferably, the step S1 includes:

[0020] Step S1.1: selecting real smoke physical data captured by using photography technology provided by the ScalarFlow smoke dataset, including the velocity field and density field of the smoke, and using the OpenVDB library to convert the density field of the NPZ format data into the VDB format smoke;

[0021] Step S1.2: Blender is used to build a dynamic scene to process VDB format smoke, set light source and camera;

[0022] Step S1.3: render pictures by dynamic smoke from different angles in spherical area in a spherical ring way, output the pose of camera and get the three-dimensional coordinates of Gaussian particle position;

[0023] Step S1.4: set the corresponding timestamp for each picture, the timestamp is uniformly sampled in the range of 0 to 1, and output the smoke data.

[0024] The step S2 comprises:

[0025] Step S2.1: read in the smoke data, encode the three-dimensional coordinates and timestamp using the six-plane spatiotemporal encoder in four dimensions, and divide the four-dimensional data into six two-dimensional planes;

[0026] Step S2.2: output the variable correction value of position deformation and scaling deformation using the multi-head deformation decoder, and get the final particle position and time variable value according to the three-dimensional coordinates and time stamp variable value in the smoke data;

[0027] Step S2.3: use the physical constraint based on velocity field as the image L1 loss and total variation loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion and the multi-head deformation decoder;

[0028] Step S2.4: after training, obtain the video composed of image smoke data by GS rendering method, and obtain the smoke dynamic scene.

[0029] Preferably, the step S3 comprises:

[0030] Step S3.1: set the initial vorticity variable of the last Nth Gaussian particle.

[0031] Step S3.2: use the Biot-Savart formula to calculate the velocity of the subsequent N-1 frames from the vorticity using the vortex method

[0032]

[0033] Wherein, u(x) represents the velocity field at x position;

[0034] ω i represents the vorticity of the i-th Gaussian particle;

[0035] Γ i represents the vorticity intensity of the i-th Gaussian particle;

[0036] (x-x irepresents the distance between the x position and the i-th Gaussian particle.

[0037] Step S3.3: Adopt RK3 formula to the convection Gaussian particle using the calculated velocity, k1=u(x n )Δt=f(t n ,x n )Δt, k3=u(x n -k1+2k2Δt=f(t +Δt,xn-k1+2k2Δt,xn+1=xn+16(k1+4k2+k3);

[0038] wherein t n represents the current time step;

[0039] Δt represents the step of time;

[0040] x n represents the known position;

[0041] f() represents a function about time t and physical quantity x;

[0042] k1 represents the slope at the beginning of the time period;

[0043] k2, k3 respectively represent the slope determined by the slope k1, k2 at the midpoint of the time period;

[0044] x n+1 represents the position of the next time step.

[0045] Step S3.4: According to the convection Gaussian particle, render the subsequent N-1 frames to obtain the subsequent rendered image for supervised training, and micro-simulate and optimize the initial vorticity to obtain the optimized vorticity.

[0046] Step S3.5: Through supervised training, adopt the optimized vorticity and the subsequent rendered image to optimize the vorticity of the last N-th frame, and obtain the simulated initial vorticity.

[0047] Preferably, the step S4 comprises:

[0048] Step S4.1: Discretize the entire three-dimensional space using a regular grid, and calculate the density of the grid at the center of the grid according to the number and opacity of the Gaussian particles

[0049] wherein δ p represents the opacity of the Gaussian particle p;

[0050] x represents the position of the center of the grid;

[0051] x p represents the position of each Gaussian particle p;

[0052] Σ p covariance of the Gaussian particle p;

[0053] Step S4.2: Emit a ray along the Cartesian coordinate direction or a random direction, judge the number of intersection points of the ray and the boundary of the object, if the number of intersection points is greater than a threshold, it is determined to be inside, a new color attribute is added to the Gaussian particle which is the same as the nearest Gaussian particle and the vorticity is set to 0, if the number of intersection points is less than or equal to the threshold, it is determined to be outside, and no Gaussian particle is filled;

[0054] Step S4.3: Traverse all the grid centers to obtain the reconstructed Gaussian particles;

[0055] The boundary of the object is the boundary line between the grid below the set threshold and the grid above the set threshold.

[0056] The step S5 includes:

[0057] Step S5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position and vorticity of the vortex particles;

[0058] Step S5.2: Calculate the velocity and new vorticity of the vortex particles according to the vorticity using the Biot-Savart formula;

[0059] Step S5.3: Use the RK3 formula to perform convection on the vortex particles according to the velocity to obtain the new position of the vortex particles;

[0060] Repeat steps S5.2 to S5.3 every N frames to judge the vorticity and set the maximum threshold of vorticity, if the absolute value of the vorticity is greater than the maximum threshold of vorticity, the vortex particle is split into two vortex particles at the offset position, each carrying half of the vorticity and opacity of the original vortex particle, inheriting the original vorticity and scaling attribute, if the absolute value of the vorticity is less than or equal to the maximum threshold of vorticity, no operation is performed, until all frames are traversed, and a simulation picture is obtained.

[0061] Preferably, the step S6 includes:

[0062] Step S6.1: Input the simulation picture into a large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as an optimizable parameter, and use the vortex method simulation calculation to render a 2D picture.

[0063] Step S6.2: Input the scene statement, call the StableDiffusion large model of the open platform Huggingface to generate a series of scene pictures of different perspectives and times.

[0064] Step S6.3: Call the SDS tool, use score distillation sampling to extract 3D knowledge from the 2D picture and the series of scene pictures, calculate the loss function compared with the simulation picture, and obtain the gradient optimization initial vorticity.

[0065] Step S6.5: According to the gradient optimization, the initial vorticity is simulated to obtain the smoke fluid reconstruction result.

[0066] According to the application, a three-dimensional Gaussian-based real fluid reconstruction evolution system is provided, which comprises a fluid video reconstruction module, a connection module and a fluid simulation module.

[0067] The fluid video reconstruction module collects and produces fluid smoke data, and uses a three-dimensional Gaussian and physical constraint-based method to reconstruct the fluid smoke data to obtain a smoke dynamic scene;

[0068] The connection module uses a smoke dynamic scene to learn a Gaussian particle vorticity using a differentiable simulation method to obtain a simulation initial vorticity;

[0069] The fluid simulation module fills the reconstructed smoke with Gaussian particles using a grid method to obtain reconstructed Gaussian particles, and performs physical simulation according to the simulation initial vorticity to obtain a simulation picture and correct it through a large model to obtain a smoke fluid reconstruction result.

[0070] Preferably, the fluid video reconstruction module comprises:

[0071] Module M1.1: Selecting the real smoke physical data captured by the photographic technology provided by the ScalarFlow smoke dataset, including the velocity field and density field of the smoke, and using the OpenVDB library to convert the NPZ format data density field into the VDB format smoke;

[0072] Module M1.2: Using Blender to build a dynamic scene to process the VDB format smoke, and setting up a light source and a camera;

[0073] Module M1.3: Using dynamic smoke rendering in a spherical region from different angles in a spherical ring around manner to obtain a picture, and outputting the pose of the camera to obtain the three-dimensional coordinates of the Gaussian particle position;

[0074] Module M1.4: Setting a corresponding timestamp for each picture, and uniformly sampling the timestamp range between 0 and 1 to output the smoke data.

[0075] Module M2.1: Reading in the smoke data, and using a six-plane spatiotemporal encoder in a four-dimensional Gaussian to encode the three-dimensional coordinates and the timestamp, and dividing the four-dimensional data into six two-dimensional planes;

[0076] Module M2.2: Using a multi-head deformation decoder to output variable correction values of position deformation and scaling deformation, and obtaining the final particle position and time variable values according to the three-dimensional coordinates and the timestamp variable values in the smoke data;

[0077] Module M2.3: jointly train the final particle position and time, the HexPlane feature plane in the spatio-temporal encoder, the MLP feature fusion and the multi-head morphing decoder with the physical constraint based on the velocity field as the image L1 loss and the total variation loss;

[0078] Module M2.4: after training, obtain the video by rendering the image smoke data set with the GS method, and obtain the smoke dynamic scene.

[0079] Preferably, the connection module comprises:

[0080] Module M3.1: set the initial vorticity variable of the Nth frame of Gaussian particles.

[0081] Module M3.2: use the Biot-Savart formula to calculate the velocity of the subsequent N-1 frames from the vorticity using the vortex method

[0082]

[0083] wherein u(x) represents the velocity field at the x position;

[0084] ω i represents the vorticity of the i th Gaussian particle;

[0085] Γ i represents the vorticity intensity of the i th Gaussian particle;

[0086] (x-x i ) represents the distance between the x position and the i th Gaussian particle.

[0087] Module M3.3: use the RK3 formula to convect the Gaussian particles using the calculated velocity n k1 = u(x n , xn) Δt = f(t n , xn) Δt, k3 = u(x n -k1 + 2k2 Δt = ftn + Δt, xn-k1 + 2k2 Δt, xn+1 = xn + 16(k1 + 4k2 + k3) ;

[0088] wherein t n represents the current time step;

[0089] Δt represents the step size of time;

[0090] x n represents the known position;

[0091] f() represents a function about time t and physical quantity x;

[0092] k1 represents the slope at the beginning of the time period;

[0093] k2, k3 represent the slope determined by the slope k1, k2 at the middle of the time period, respectively;

[0094] x n+1 represents the position of the next time step.

[0095] Module M3.4: Supervised training is performed on the subsequent rendered images obtained by rendering the subsequent N-1 frames according to the advection of the Gaussian particles, and the initial vorticity can be optimized by micro-simulation, and the optimized vorticity is obtained.

[0096] Module M3.5: The vorticity of the last Nth frame is optimized by using the optimized vorticity and the subsequent rendered images through supervised training, and the simulation initial vorticity is obtained.

[0097] Preferably, the fluid simulation module comprises:

[0098] Module M4.1: Discretize the entire three-dimensional space using a regular grid, and calculate the density of the grid at the center of the grid according to the number and opacity of the Gaussian particles

[0099] where δ p represents the opacity of the Gaussian particle p;

[0100] x represents the position of the center of the grid;

[0101] x p represents the position of each Gaussian particle p;

[0102] Σ p represents the covariance of the Gaussian particle p;

[0103] Module M4.2: Emit a ray along the Cartesian coordinate direction or a random direction, determine the number of intersection points of the ray and the boundary of the object, if the number of intersection points is greater than a threshold, it is determined to be inside, and a new Gaussian particle with the same color attribute as the nearest Gaussian particle and the vorticity set to 0 is added, if the number of intersection points is less than or equal to the threshold, it is determined to be outside, and no Gaussian particle is filled;

[0104] Traverse all grid centers to obtain reconstructed Gaussian particles;

[0105] The boundary of the object is the demarcation line between the grid below the set threshold and the grid above the set threshold.

[0106] Module M5.1: Use the reconstructed Gaussian particles as vortex particles, initialize the position and vorticity of the vortex particles;

[0107] Module M5.2: Calculate the velocity and new vorticity of the vortex particles using the Biot-Savart formula according to the vorticity;

[0108] Module M5.3: Convection of vortex particles using RK3 formula according to speed, to get the new position of the vortex particle;

[0109] Repeat the triggering modules M5.2 and M5.3, judge the vorticity and set the maximum threshold of vorticity every N frames, if the absolute value of vorticity is greater than the maximum threshold of vorticity, the vortex particle is split into two vortex particles at the offset position, each of the two vortex particles carries half of the original vortex particle's vorticity and opacity, inherits the original spin and scaling properties, if the absolute value of vorticity is less than or equal to the maximum threshold of vorticity, no operation is performed, until all frames are traversed, to get the simulation picture.

[0110] Preferably, the fluid simulation module comprises:

[0111] Module M6.1: input the simulation picture into the large model, set the vorticity of the vortex particle and the newly added vortex particle at the beginning of the simulation as an optimizable parameter, use vortex method simulation calculation, and render into a 2D picture.

[0112] Module M6.2: input scene sentences, call the StableDiffusion large model of the open platform Huggingface to generate a series of scene pictures of different perspectives and times.

[0113] Module M6.3: call SDS tool, use score distillation sampling to extract 3D knowledge from 2D pictures and a series of scene pictures, calculate the loss function compared with the simulation picture, and get the gradient optimization initial vorticity.

[0114] Module M6.4: according to the gradient optimization initial vorticity, the simulation picture is physically simulated to get the smoke fluid reconstruction result.

[0115] Compared with the prior art, the present application has the following beneficial effects:

[0116] 1. The present application combines the promising three-dimensional Gaussian reconstruction, differentiable simulation and large model technology, and provides a new framework, which can quickly and completely reconstruct dynamic fluid and provide further physical evolution capability compared with the existing reconstruction and simulation technology, and has high industrial value.

[0117] 2. The present application uses the method based on three-dimensional Gaussian video reconstruction and the data set captured from real smoke, uses physical constraints as loss to improve learning effect, and efficiently reconstructs the dynamic fluid scene, so that the reconstruction result has better physical fidelity.

[0118] 3. The present application uses a differentiable physical learning method to convert the velocity field and the vorticity field, learns the initial vorticity before simulation starts, effectively connects the reconstruction module and the simulation module, optimizes the initial vorticity, and lays a foundation for further driving subsequent simulation using the vortex method.

[0119] 4. The application adopts a fluid simulation method based on a vortex method, uses the vortex method to drive Gaussian particle flow, considers Gaussian particle filling and vortex particle splitting, and further solves the problem of further evolution after fluid reconstruction by using a large model to correct fluid physical properties. BRIEF DESCRIPTION OF DRAWINGS

[0120] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, when read in conjunction with the accompanying drawings:

[0121] Figure 1 A schematic diagram of the overall process of a three-dimensional Gaussian-based real fluid reconstruction evolution method;

[0122] Figure 2 A schematic diagram of the system architecture of a three-dimensional Gaussian-based real fluid reconstruction evolution method;

[0123] Figure 3 A schematic diagram of a fluid smoke data scene and effect acquisition and production;

[0124] Figure 4 A schematic diagram of Gaussian particles and rendering effects of smoke reconstructed using a three-dimensional Gaussian and physical constraint-based method;

[0125] Figure 5 A schematic diagram of sparse distribution of particles inside a Gaussian particle filled using a grid method;

[0126] Figure 6 A schematic diagram of the comparison of filling before and after a Gaussian particle filled using a grid method;

[0127] Figure 7 A schematic diagram of the comparison of vortex particle splitting before and after smoke simulation based on a physical method;

[0128] Figure 8 A schematic diagram of the effect of a fluid video reconstruction module;

[0129] Figure 9 A schematic diagram of the effect of a fluid simulation module. DETAILED DESCRIPTION

[0130] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0131] The application provides a three-dimensional Gaussian-based real fluid reconstruction evolution method, which dynamically reconstructs a three-dimensional fluid scene according to a fluid video in stages by using three-dimensional Gaussian particles, and provides an evolution scheme after the video is cut off, so that Figure 1 For example, specifically includes:

[0132] Step S1: collecting and making fluid smoke data;

[0133] Step S1.1: reconstructing fluid smoke needs to provide smoke data for reference, which is the basis of fluid reconstruction. A data set is used for real smoke sampling, and the data set is selected and processed into a VDB format;

[0134] The processing is to convert the density field of the NPZ format data of ScalarFlow into the VDB format by using OpenVDB.

[0135] Step S1.2: using Blender to build a dynamic scene to process VDB smoke, setting up a light source camera and rendering.

[0136] Step S1.3: sampling dynamic smoke in a spherical region and outputting camera poses, rendering dynamic smoke into pictures from different angles in a spherical surrounding manner, and outputting camera poses.

[0137] Step S1.4: setting a timestamp, setting a timestamp between 0 and 1 for data at different times according to uniform sampling of the number of frames.

[0138] In more preferred examples, the three-dimensional Gaussian-based dynamic reconstruction method has certain requirements for the data used, so dynamic smoke pictures from multiple angles and corresponding camera poses are needed. At present, the reconstruction field is mostly focused on the reconstruction of solids rather than fluids, fluid data is relatively scarce, and existing data cannot directly meet the reconstruction requirements, so fluid data for reconstruction is made by the applicant.

[0139] Initially, the real smoke physical data captured by using a photography technology is provided by using a ScalarFlow smoke data set, the data set provides a velocity field and a density field of smoke, but the pictures provided by the data set are only sparse-view smoke pictures, which cannot meet the reconstruction requirements of the three-dimensional Gaussian.

[0140] By using an OpenVDB library, the density field of ScalarFlow is processed into a VDB file, and the VDB format smoke is further adjusted in Blender, a light source and a camera are set, and pictures, camera poses and timestamps are generated by using a spherical surrounding manner, as shown in Figure 3 The finally made data set meets the training requirements of the three-dimensional Gaussian.

[0141] Step S2: reconstructing smoke by using a three-dimensional Gaussian and physical constraint-based method;

[0142] The three-dimensional Gaussian function (elliptical Gaussian kernel) is taken as the basic unit of fluid, and the basic unit can be optimized and learned by gradient descent and other techniques. The dynamic video reconstruction method adopted is the optimized 4DGS scheme, and the main difference between the scheme and the 3DGS is that there is an additional time dimension. The 3D Gaussian is mainly used for static scene reconstruction, and although it is crucial in the field of three-dimensional vision, dynamic scene rendering faces the challenge of complex motion modeling, and it is difficult to balance efficiency and accuracy. Therefore, the 4DGS proposes the concept of time dimension in the field of dynamic scene reconstruction, combines 3D Gaussian and 4D neural voxel, converts the standard 3D Gaussian to a new position and shape through the Gaussian deformation field network, and realizes efficient rendering by using the spatiotemporal structure encoder and the multi-head Gaussian deformation decoder.

[0143] Specifically, based on the three-dimensional Gaussian for dynamic reconstruction, a method based on four-dimensional Gaussian (4DGS) is adopted, and a time dimension is introduced to reconstruct a dynamic flow field according to video data. The specific steps include:

[0144] Step S2.1: Increase the time dimension as input;

[0145] A 6-plane spatiotemporal encoder is used to encode three-dimensional coordinates and timestamps. The data is read in, and the Gaussian particle position, time and other variables are encoded using the spatiotemporal encoder. The four-dimensional data is divided into six two-dimensional planes to reduce storage consumption and provide physical correlation in the adjacent area.

[0146] Step S2.2: A multi-head deformation decoder is used to output the results of deformation. The multi-head decoder outputs the correction values of variables such as position deformation and scaling deformation, and the final particle position and other variable values are obtained according to the initial Gaussian particle position and other variable values.

[0147] Step S2.3: Physical constraints are used as losses to give the Gaussian particles physical meaning, and the Gaussian position and other variables, HexPlane feature planes, MLP feature fusion and multi-head deformation decoder are trained together with the image L1 loss and total variation loss. The velocity-based physical constraint is introduced, and the velocity is increased as a physical constraint to guide the learning of the Gaussian particles, so that they have stronger physical meaning.

[0148] Step S2.4: After training, the image is obtained by the conventional GS rendering method. Figure 4 For example, and compose a video.

[0149] In more preferred examples, the two most important differences compared to the implementation structure of 3DGS are the spatio-temporal structure encoder and the multi-head Gaussian deformation decoder. The spatio-temporal structure encoder refers to the decomposition of 4D neural voxel into 6 planes using multi-resolution HexPlane and tiny MLP, which effectively encodes the spatio-temporal features of 3D Gaussian. The specific method is to encode the three-dimensional coordinates x, y, z and the time dimension t into 6 two-dimensional planes R i,j (xy, xz, yz, xt, yt, zt), and the features stored on the plane can be obtained through learning. Then, according to the position and timestamp of each Gaussian particle, the interpolation of the features is obtained from the 6 planes and fused together through a small MLP. The role of the spatio-temporal structure encoder is to effectively encode the spatio-temporal features of 3D Gaussian. The nearby 3D Gaussian always has similar spatio-temporal information, which can effectively model the features of 3D Gaussian in this way, while reducing the storage consumption compared to the pure explicit method. The formula for calculating the features of each voxel is:

[0150] f h = ∪ interp(R i,j )

[0151] At the same time, the multi-head Gaussian deformation decoder is similar to the multi-head attention mechanism, which refers to using the features of the spatio-temporal encoder to output the deformation of the position, rotation and scaling of the 3D Gaussian with multiple MLPs, and obtaining the physical quantity of the deformed 3D Gaussian according to the initial physical quantity for rendering.

[0152] Introduce physical constraint auxiliary training. The spatio-temporal features learned by the three-dimensional Gaussian itself do not contain physical meaning, but only the mapping of the position and other features of the three-dimensional Gaussian particle in space and time. The velocity field is used as a constraint on the reconstructed physical quantity added to the loss of training, and the C1 continuity of the Gaussian particle position is obtained through the supervised training of the velocity field, which has a clear physical meaning.

[0153] Step S3: learning the vorticity of Gaussian particles using differentiable simulation methods;

[0154] Differentiable simulation is an important research direction in the field of graphics, which refers to the process of simulation in which the dynamics and kinematics models of the system can be differentiated. The two key parts are differentiable models and automatic differentiation techniques. Common differentiable models include Newtonian mechanics equations and elastic mechanics equations, while automatic differentiation techniques are provided by default in many deep learning or graphics frameworks. Due to the above characteristics, differentiable simulation can often be used in conjunction with deep learning technology.

[0155] The steps S1-S2 obtain a complete smoke dynamic scene reconstruction, but the reconstructed smoke asset can only be cut off to the last frame of the video and cannot be further simulated, and a vortex method needs to be used to further drive subsequent simulation, and the vortex method needs to have a good initial vortex, therefore, first, the attributes of the Gaussian particles are obtained by learning, and the initial vortex is obtained by differentiable simulation.

[0156] Specifically, the reconstructed dynamic Gaussian particles are used to learn the vortex field of the last frame in a differentiable simulation manner, while the conversion between the vortex and the speed is taken into account. The Biot-Savart law is used to connect the Gaussian particle speed of the reconstruction module and the vortex of the simulation module.

[0157] In more preferred examples, the video dynamic scene reconstruction mainly learns a series of frames to set a vortex physical quantity for the Gaussian particles of the last Nth frame, and the subsequent N-1 frames use the vortex method to convert the vortex into the speed by using the Biot-Savart formula, and use the speed to perform Runge-Kutta 3 (RK3) convection. It is equivalent to that the subsequent N frames no longer use the scheme of spatiotemporal feature deformation mapping, but use the differentiable simulation scheme, and the same rendering image of each frame is obtained. Specifically, it includes:

[0158] Step S3.1: setting an initial vortex variable for the last Nth frame for optimization.

[0159] Step S3.2: calculating the speed from the vortex by using the Biot-Savart formula;

[0160] The formula for converting the Biot-Savart vortex into the speed is as follows:

[0161]

[0162] where u(x) represents the speed field at the position x, ω i represents the vortex of the vortex particle, Γ i represents the vortex intensity of the vortex particle, (x-x i ) represents the distance between the current position and the vortex particle. It should be noted that the Gaussian particles carry the vortex attribute, and the vortex particles in this step are actually directly used as the vortex particles, and therefore, in the content of the present application, the "vortex particle" can be equivalent to the "Gaussian particle".

[0163] The physical meaning of the formula is that each vortex particle will generate an induced speed field in space, and by superimposing the induced speeds of all vortex particles, the speed of the entire flow field at position x can be obtained. The existence of the vortex vector ω determines the direction and size of the induced speed, and it is related to the position vector (x-x iThe cross-multiplication relationship of the Biot-Savart formula makes the direction of the velocity perpendicular to the plane formed by the vorticity and the position difference, and the velocity magnitude is related to the vorticity intensity, distance, etc.

[0164] At the same time, in order to prevent the learned velocity field calculated by the Gaussian particle from interfering with the velocity field obtained by the vortex method, a small value is selected for N, such as N=2, so that a more reasonable context velocity relationship can be obtained.

[0165] Step S3.3: using the calculated velocity to update the Gaussian particle by RK3 formula;

[0166] The Gaussian particle is the Gaussian particle of the last Nth frame of step S3.1. At the same time, the Runge-Kutta 3 method is commonly used to update the change of the relevant physical quantity (such as velocity, position, etc.) of the fluid over time. In order to update the change of the position, it is assumed that the current time step is t n , Δt is the time step, the known position is x n , u(x) is the velocity calculated by the above Biot-Savart formula at position x, f is a function of time t and physical quantity x, three slopes k1, k2, k3 need to be calculated, and finally the position x n+1 of the next time step is calculated, and the specific formula is as follows:

[0167] k1=u(x n )Δt=f(t n ,x n )Δt

[0168]

[0169] k3=u(x n -k1+2k2)Δt=f(t n +Δt,x n -k1+2k2)Δt

[0170]

[0171] Wherein, k1 is the slope at the beginning of the time period; k2 is the slope at the midpoint of the time period, which is determined by the Euler method using the slope k1 to determine the value of f(t,x) at point , and k3 is also the slope at the midpoint, but this time the slope k2 is used to determine the value of f(t,x).

[0172] The application of RK3 method in convection problem can accurately update the physical quantity to a certain extent, and its precision is between low order (such as Euler method) and high order (such as RK4, etc.), which provides a balance between calculation efficiency and precision, and is suitable for some convection simulation scenarios which do not require extremely high precision but want to be more accurate than low order method.

[0173] Step S3.4: The subsequent N-1 frame rendering image participates in the reconstruction of the supervised training, and the initial vorticity of the micro-simulation optimization simulation module can be simulated and optimized to obtain the optimized vorticity.

[0174] Step S3.5: After using the vorticity calculation and the subsequent rendering to obtain the image, the initial Nth frame vorticity is also optimized by supervision, and this vorticity is used as the initial vorticity of the subsequent step S5 simulation, which is directly used as the initial value of the vortex particle at the beginning of the simulation.

[0175] Using the reconstructed position and the micro-simulatable vorticity as the initial condition, the vortex method is further physically simulated, the grid method is used to fill the Gaussian particles, and the particles are dynamically split based on the vorticity, and the large model is used to correct unreasonable vorticity that does not conform to fluid flow.

[0176] Step S4: Fill the Gaussian particles using the grid method;

[0177] There is an inherent defect in the three-dimensional Gaussian reconstruction scene, that is, most of the reconstructed Gaussian particles are focused on the surface of the object, and the Gaussian particles in the interior of the object are very sparse, which is also true for fluids. Sparse internal is not a problem for static scenes, but for dynamic scenes, especially fluid scenes, internal Gaussian particles will flow to the surface through fluid motion, and if the internal particles are insufficient, it will cause a void phenomenon, as shown in Figure 5 Therefore, the existing Gaussian particles need to be further filled by a suitable method, including:

[0178] Step S4.1: Grid the three-dimensional space, maintain an opacity attribute based on 3D Gaussian, and use a regular grid to discretize the entire space. The density of the grid is calculated according to the number of Gaussian particles and the opacity at the center of the grid.

[0179] Specifically, first divide the fluid domain into 128 3 grid spaces, and calculate the density of the grid center according to the opacity of all Gaussian particles inside each grid. The calculation formula is as follows:

[0180]

[0181] Where δ p is the opacity of the Gaussian particle, x is the grid center position, x p is the position of each Gaussian particle, and Σ p is the covariance of the Gaussian particle. By calculating the opacity attribute of each grid, an opaque object with boundaries is constructed.

[0182] Step S4.2: Set a threshold value to determine whether the grid density is greater than the threshold value, determine whether the grid center is inside the object according to the grid density, insert a particle at the grid center according to the determination result, and obtain the particle attribute from the nearest Gaussian particle.

[0183] Specifically, after obtaining the center density of each grid, a custom threshold value can be set, so that the boundary of the object can be considered to be on both sides of the grid below the threshold value and the grid above the threshold value. Then, traverse all grid centers, emit a ray along the Cartesian coordinate direction or a random direction, and determine whether the grid center is inside the object by the number of intersection points of the ray and the boundary of the object. If it is inside, a new Gaussian particle is added, and its color and other attributes are the same as those of the nearest Gaussian particle, and the vorticity is set to 0. If it is outside, no new Gaussian particle is filled at the grid center. Without considering special cases, when the ray is emitted along the Cartesian coordinate direction, 6 intersection points indicate that the object is inside, i.e. the threshold value is set to 6, and the remaining cases indicate that the object is outside; when the ray is emitted in a random direction, only one intersection point indicates that the object is inside, and the remaining cases indicate that the object is outside. The effect after filling is taken as an example. Figure 6

[0184] Step S5: Perform smoke simulation based on a physical method;

[0185] Specifically, step S5.1: use Gaussian particles as vortex particles, and use the vortex method to control the motion of the Gaussian particles, which specifically includes initializing the position of the vortex particle and the vorticity;

[0186] Step S5.2: use the Biot-Savart law to calculate the velocity and new vorticity of the vortex particle from the vorticity;

[0187] Step S5.3: use Runge-Kutta 3 to convect the vortex particle according to the velocity to obtain the new position of the vortex particle;

[0188] Iteratively calculate the velocity, vorticity and position, and according to the vorticity, calculate the splitting of the vortex particle once every N frames according to the threshold value.

[0189] ​In more preferred examples, the reconstructed Gaussian particles are driven using a vortex method for the purpose of further evolution. The reconstructed Gaussian particles are treated as vortex particles, and the vortex method is used as the underlying physical method for driving the simulation. The vortex particle method is a numerical method for simulating fluid flow, which discretizes the vorticity distribution of a fluid into a series of vortex particles, and simulates the behavior of the fluid by tracking the motion and interaction of the vortex particles. In the vortex particle method, the vorticity field of a fluid is discretized into a set of vortex particles, each with a certain vorticity strength, position, and possibly other attributes. The vortex particle method is derived from the vorticity formulation of the Navier-Stokes equations, which transforms the original equations based on velocity as the underlying physical quantity into vorticity. The specific formulas are as follows:

[0190]

[0191] The first formula mainly represents that the velocity is affected by pressure, gravity, and other factors, and evolves with the velocity itself. The second formula represents the divergence-free condition that the velocity needs to satisfy, which also corresponds to the incompressibility condition of the fluid. The third formula is the result of multiplying the curl operator on both sides of the Navier-Stokes equation, which embeds the incompressibility condition and represents the evolution of vorticity. Compared to traditional numerical solutions based on velocity, the vortex method essentially discretizes vorticity, allowing vorticity to be tracked in a Lagrangian perspective. At the same time, due to the good maintenance of vorticity, it can preserve more details of the fluid, thus realizing more complex fluid phenomena.

[0192] The core of the vortex method is the Biot-Savart law, which assumes that vortex particles will generate induced velocities in space. The main role is to convert the vorticity carried by the vortex particles into velocity, thus facilitating subsequent calculations such as convection. The formula is as follows:

[0193]

[0194] The numerical calculation steps of the vortex method first initialize the vorticity of the vortex particles, which need to set the initial position and initial vorticity of the vortex particles. Then, considering a certain number of vortex particles, the velocity is calculated through the Biot-Savart law. It needs to be noted that, due to the Lagrangian discrete perspective, the velocity only needs to be calculated at sparse positions, such as the positions of the vortex particles. After completing the velocity calculation, the RK3 convection method is used to calculate the convection of the vortex particles with the velocity, and the new position is calculated again.

[0195] In addition to the traditional vortex particle method, the present application also adds a vortex particle splitting method based on the value of vorticity. A maximum threshold of vorticity is set, when the absolute value of vorticity exceeds the maximum threshold during evolution, the vortex particle will split into two vortex particles at a small random offset position, the new two vortex particles each carry half of the vorticity and opacity of the original vortex particle, inherit the original spin, scaling and other attributes. This splitting method is executed every N frames, the setting of N can be according to user demand, in this embodiment, N = 30 is set, and the schematic diagram of splitting is as shown in Figure 7

[0196] Step S6: using a large model to correct simulation details.

[0197] Large models are one of the most valuable technologies for research and application at present, most of which are based on the Transformer structure and follow the Scaling law at the bottom, meaning that the more model parameters and the deeper the model, the more accurate the model results. At the same time, it has shown strong ability in the fusion of multi-modal. Therefore, by using the iterative optimization and feedback correction ability of the large model, the simulation detail correction is carried out in an iterative way.

[0198] Specifically, it includes:

[0199] Step S6.1: obtain a simulation picture by vortex method simulation;

[0200] First, set the vorticity of the initial vortex particle and the vorticity of the newly added particle at the beginning of simulation as the optimizable parameters, perform the preliminary vortex method simulation calculation in step S5, and then render into a 2D picture.

[0201] Step S6.2: input scene sentences, call large models of open platform to generate a series of scene pictures;

[0202] Under the premise of no 3D data set, different view and time pictures can be generated by inputting text (such as "smoke rises upward") and calling the StableDiffusion model of the open platform Huggingface.

[0203] Step S6.3: obtain a series of result pictures by using vortex method simulation;

[0204] ​Score Distillation Sampling (SDS) is a technique used in the field of machine learning, particularly in the generation of models, to generate samples from complex probability distributions. The process of converting text to 3D dynamic video involves accurately translating the semantic information in the text, including object movements, scene changes, and temporal sequences, into feature representations of the 3D dynamic video. SDS can aid in optimizing this mapping process by learning a score function that reflects the distribution of 3D dynamic video features. This allows the model to adjust the dynamic features of each frame in the video, such as 3D structure, object movement speed, and direction, more accurately based on the text semantics, resulting in a video that better aligns with the intended meaning of the text.

[0205] The knowledge extraction from the pictures obtained by vortex method simulation and the knowledge extraction from the pictures generated by the large model using text prompts are used to calculate the loss function of the two types of knowledge to achieve the purpose of optimization and correction. The optimization target is the initial vorticity of the vortex particles (including the original vortex particles and the newly added vortex particles in step S4). The gradient optimization initial vorticity is obtained.

[0206] Step S6.4: Physically simulating the gradient optimization initial vorticity to obtain the smoke fluid reconstruction result.

[0207] The forward reasoning of fluid motion completely reconstructs the forward flow and further forward evolution of the fluid. The video reconstruction result of the smoke fluid is shown in Figure 8 , and the further evolution result is shown in Figure 9 . The present application can be integrated in any application to generate reusable smoke assets from the provided smoke video, and can further evolve beyond the video category to present rich fluid phenomena.

[0208] The present application also provides a real fluid reconstruction evolution system based on three-dimensional Gaussian, which can be realized by executing the process steps of the real fluid reconstruction evolution method based on three-dimensional Gaussian, that is, the real fluid reconstruction evolution method based on three-dimensional Gaussian can be understood by those skilled in the art as the preferred embodiment of the real fluid reconstruction evolution system based on three-dimensional Gaussian.

[0209] According to the real fluid reconstruction evolution system based on three-dimensional Gaussian provided by the present application, taking Figure 2 as an example, it specifically includes a fluid video reconstruction module, a connection module, and a fluid simulation module.

[0210] These three modules have their own functions and cooperate with each other, aiming to solve the problems in the process of 3DGS reconstruction and subsequent processing from different angles and levels, thereby improving the overall reconstruction effect and evolution ability.

[0211] The fluid video reconstruction module collects and prepares fluid smoke data, and reconstructs smoke using a method based on three-dimensional Gaussian and physical constraints. This module mainly uses three-dimensional Gaussian to quickly and accurately reconstruct dynamic fluid by adding physical constraints.

[0212] Specifically, a real smoke dataset captured using ScalarFlow is used; a dynamic fluid reconstruction framework using four-dimensional Gaussian is used; a video learning scheme using physical constraints is used.

[0213] The adaption module uses a differentiable simulation method to learn the vorticity of Gaussian particles. The purpose of this module is to adapt the velocity of the reconstructed Gaussian particles and the subsequent simulated vorticity to provide reasonable initialization conditions.

[0214] Specifically, a conversion method including velocity field and vorticity field is used; a vorticity initialization method based on differentiable fluid simulation is used.

[0215] The fluid simulation module uses a grid method to fill Gaussian particles, a physics-based method for smoke simulation, and a large model to correct simulation details. This module provides the results of further physical evolution of fluid outside video reconstruction.

[0216] Specifically, a fluid simulation based on vortex method is included; a grid-based Gaussian particle filling method is used; a particle splitting method based on vorticity control is used; a fluid detail correction based on a large model is used.

[0217] In more preferred examples, the fluid video reconstruction module comprises:

[0218] Module M1.1: Select real smoke physical data captured using photography technology provided by the ScalarFlow smoke dataset, including the velocity field and density field of the smoke, and convert the density field of the NPZ format data to the VDB format smoke using the OpenVDB library;

[0219] Module M1.2: Process the VDB format smoke using Blender to build a dynamic scene, and set up light sources and cameras;

[0220] Module M1.3: Render pictures using dynamic smoke from different angles in a spherical region in a spherical ring around manner, and output the pose of the camera and the three-dimensional coordinates of the Gaussian particle positions at the same time;

[0221] Module M1.4: Set a corresponding timestamp for each picture, and the timestamp is uniformly sampled in the range of 0 to 1, and output the smoke data.

[0222] Module M2.1: Read in the smoke data, and use the six-plane spatiotemporal encoder in the four-dimensional Gaussian to encode the three-dimensional coordinates and timestamps, and divide the four-dimensional data into six two-dimensional planes;

[0223] Module M2.2: Adopting variable correction values of position morphing and scaling morphing of multi-head morphing decoder, obtaining final particle position and time variable values according to three-dimensional coordinates and time stamp variable values in smoke data;

[0224] Module M2.3: Adopting physical constraints based on velocity field as image L1 loss and total variation loss, jointly training final particle position and time, HexPlane feature plane in space-time encoder, MLP feature fusion and multi-head morphing decoder;

[0225] Module M2.4: After training, obtaining video composed of image smoke data by GS rendering method, and obtaining smoke dynamic scene.

[0226] In more preferred examples, the adapter module comprises:

[0227] Module M3.1: Setting initial vorticity variable of the Nth frame from the end Gaussian particle.

[0228] Module M3.2: Using Biot-Savart formula to calculate velocity of subsequent N-1 frames from vorticity using vortex method

[0229]

[0230] Wherein, u(x) represents the velocity field at x position;

[0231] ω i represents the vorticity of the i th Gaussian particle;

[0232] Γ i represents the vorticity intensity of the i th Gaussian particle;

[0233] (x-x i ) represents the distance between x position and the i th Gaussian particle.

[0234] Module M3.3: Using RK3 formula to convect Gaussian particles using the calculated velocity n k1 = u(x n , xn) Δt = f(t n , xn) Δt, k3 = u(x n -k1 + 2k2 Δt = ftn + Δt, xn-k1 + 2k2 Δt, xn+1 = xn + 16(k1 + 4k2 + k3);

[0235] Wherein, t n represents the current time step;

[0236] Δt represents the step length of time;

[0237] x nrepresents a known position;

[0238] f() represents a function of time t and physical quantity x;

[0239] k1 represents a slope at the beginning of a time period;

[0240] k2, k3 respectively represent a slope determined by k1, k2 at the midpoint of a time period;

[0241] x n+1 represents a position of the next time step.

[0242] Module M3.4: Supervised training is performed on the subsequent rendered images obtained by rendering the subsequent N-1 frames according to the advection of the Gaussian particles, and the initial vorticity can be optimized by micro-simulation, and the optimized vorticity is obtained.

[0243] Module M3.5: Through supervised training, the vorticity of the last Nth frame is optimized using the optimized vorticity and the subsequent rendered images, and the simulation initial vorticity is obtained.

[0244] In more preferred examples, the fluid simulation module comprises:

[0245] Module M4.1: Discretize the entire three-dimensional space using a regular grid, and calculate the density of the grid at the center of the grid according to the number and opacity of the Gaussian particles

[0246] where δ p represents the opacity of the Gaussian particle p;

[0247] x represents the position of the center of the grid;

[0248] x p represents the position of each Gaussian particle p;

[0249] Σ p represents the covariance of the Gaussian particle p;

[0250] Module M4.2: Emit a ray along a Cartesian coordinate direction or a random direction, determine the number of intersection points of the ray with the boundary of the object, if the number of intersection points is greater than a threshold, it is determined to be inside, and a new Gaussian particle with the same color attribute as the nearest Gaussian particle and the vorticity set to 0 is added, if the number of intersection points is less than or equal to the threshold, it is determined to be outside, and no Gaussian particle is filled;

[0251] Traverse all grid centers to obtain reconstructed Gaussian particles;

[0252] The boundary of the object is the boundary line between the grid below the set threshold and the grid above the set threshold.

[0253] Module M5.1: initialize the position and vorticity of the vortex particle using the reconstructed Gaussian particle as the vortex particle;

[0254] Module M5.2: calculate the velocity and new vorticity of the vortex particle according to the vorticity using the Biot-Savart formula;

[0255] Module M5.3: convect the vortex particle according to the velocity using the RK3 formula to obtain the new position of the vortex particle;

[0256] Repeat modules M5.2 and M5.3 every N frames to determine the vorticity and set the maximum threshold of vorticity. If the absolute value of the vorticity is greater than the maximum threshold of vorticity, the vortex particle splits into two vortex particles at the offset position, each carrying half of the original vortex particle's vorticity and opacity, inheriting the original spin and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum threshold of vorticity, no operation is performed. Until all frames are traversed, the simulation picture is obtained.

[0257] In more preferred examples, the fluid simulation module comprises:

[0258] Module M6.1: input the simulation picture into the large model, set the vorticity of the vortex particle and the newly added vortex particle at the beginning of the simulation as an optimizable parameter, use the vortex method simulation calculation, and render it into a 2D picture.

[0259] Module M6.2: input the scene statement, call the StableDiffusion large model of the open platform Huggingface to generate a series of scene pictures of different perspectives and times.

[0260] Module M6.3: call the SDS tool, use score distillation sampling to extract 3D knowledge from the 2D picture and the series of scene pictures, calculate the loss function compared with the simulation picture, and obtain the gradient optimization initial vorticity.

[0261] Module M6.4: according to the gradient optimization initial vorticity, the simulation picture is physically simulated to obtain the smoke fluid reconstruction result.

[0262] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0263] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other in any manner without conflict.

Claims

1. A three-dimensional Gaussian-based true fluid reconstruction evolution method, characterized in that, The method comprises the following steps: Step S1: collecting and preparing fluid smoke data; Step S2: reconstructing the fluid smoke data using a method based on three-dimensional Gauss and physical constraints to obtain a smoke dynamic scene; Step S3: learning the Gauss particle vorticity using a differentiable simulation method with the smoke dynamic scene to obtain simulation initial vorticity; Step S4: filling the reconstructed smoke with Gauss particles using a grid method to obtain reconstructed Gauss particles; Step S5: performing physical simulation on the reconstructed Gauss particles according to the simulation initial vorticity to obtain simulation pictures; Step S6: correcting the simulation pictures using a large model to obtain a smoke fluid reconstruction result; The step S1 comprises: Step S1.1: selecting real smoke physical data captured by photography technology provided by a ScalarFlow smoke dataset, including a smoke velocity field and a density field, and converting the NPZ format data density field into a VDB format smoke using an OpenVDB library; Step S1.2: processing the VDB format smoke by setting up a dynamic scene using Blender, setting a light source and a camera; Step S1.3: rendering the smoke into pictures from different angles in a spherical region in a spherical ring manner, and outputting the poses of the cameras and the three-dimensional coordinates of the Gauss particle positions; Step S1.4: setting a corresponding timestamp for each picture, uniformly sampling the timestamp in a range of 0 to 1, and outputting the smoke data; The step S2 comprises: Step S2.1: reading in the smoke data, encoding the three-dimensional coordinates and the timestamp into a six-plane spatiotemporal encoder in a four-dimensional Gauss, and dividing the four-dimensional data into six two-dimensional planes; Step S2.2: outputting the variable correction values of the position deformation and the scaling deformation using a multi-head deformation decoder, and obtaining the final particle position and time variable values according to the three-dimensional coordinates and the timestamp variable values in the smoke data; Step S2.3: using the physical constraints based on the velocity field as the image L1 loss and the total variation loss to jointly train the final particle position and time, the HexPlane feature plane in the spatiotemporal encoder, the MLP feature fusion and the multi-head deformation decoder; Step S2.4: obtaining a video composed of image smoke data by a GS rendering method after the training is completed, and obtaining a smoke dynamic scene; The step S3 comprises: Step S3.1: setting the initial vorticity variable of the last N-th frame of Gauss particles; Step S3.2: Calculate velocity for the next N-1 frames from vorticity using Biot-Savart formula using vortex method Wherein, u(x) represents the velocity field at the position x; ω i represents the vorticity of the i-th Gaussian particle; Γ i represents the vorticity intensity of the i-th Gaussian particle; (x-x i ) represents the distance between the x position and the i-th Gaussian particle; Step S3.3: Advect the Gaussian particles using the computed velocity with the RK3 formula, k1 = u(x n )Δt = f(t n , x n )Δt, k3 = u(x n -k1 + 2k2)Δt = f(t n +Δt, x n -k1 + 2k2)Δt, where t n denotes the current time step; Δt represents the step length of time; x n represents a known position; f() represents a function with respect to time t and physical quantity x; k1 represents the slope at the beginning of the time period; k2 and k3 respectively represent the slopes determined by the slopes k1 and k2 at the midpoint of the time period; x n+1 represents the position of the next time step; Step S3.4: supervising the training according to the subsequent N-1 frames of rendering images rendered according to the convection of the Gauss particles, and optimizing the initial vorticity to obtain the optimized vorticity; Step S3.5: optimizing the vorticity of the last N-th frame by using the optimized vorticity and the subsequent rendering images through supervised training to obtain the simulation initial vorticity.

2. The three-dimensional Gaussian-based true fluid reconstruction evolution method of claim 1, wherein, The step S4 comprises: Step S4.1: Discretize the whole three-dimensional space using a regular grid, compute the density of the grid at the center of the grid from the number of Gaussian particles and opacity where δ p represents the opacity of the Gaussian particle p; x represents the position of the grid center; x p represents the position of each Gaussian particle p; ∑ p denotes the covariance of the Gaussian particle p; Step S4.2: Emit a ray along a Cartesian coordinate direction or a random direction, determine the number of intersection points of the ray with the boundary of the object, if the number of intersection points is greater than a threshold, it is determined to be inside, a new color attribute is added to the nearest Gaussian particle with the same Gaussian particle and the vorticity is set to 0, if the number of intersection points is less than or equal to the threshold, it is determined to be outside, and no Gaussian particle is filled; Traverse all grid centers to obtain reconstructed Gaussian particles; The boundary of the object is the boundary line between the grid below the set threshold and the grid above the set threshold; The step S5 includes: Step S5.1: Use the reconstructed Gaussian particles as vortex particles to initialize the position of the vortex particles and the vorticity; Step S5.2: Calculate the velocity and new vorticity of the vortex particles using the Biot-Savart formula according to the vorticity; Step S5.3: Use the RK3 formula to convect the vortex particles according to the velocity to obtain the new position of the vortex particles; Repeat steps S5.2 to S5.3 every N frames to determine the vorticity and set the maximum threshold of vorticity, if the absolute value of the vorticity is greater than the maximum threshold of vorticity, the vortex particle is split into two vortex particles at the offset position, each carrying half of the vorticity and opacity of the original vortex particle, inheriting the original spin and scaling attributes, if the absolute value of the vorticity is less than or equal to the maximum threshold of vorticity, no operation is performed, until all frames are traversed, and a simulation picture is obtained.

3. The three-dimensional Gaussian-based true fluid reconstruction evolution method of claim 1, wherein, The step S6 includes: Step S6.1: Input the simulation picture into the large model, set the vorticity of the vortex particles and the newly added vortex particles at the beginning of the simulation as the optimizable parameters, use the vortex method simulation calculation, and render into a 2D picture; Step S6.2: Input the scene statement, call the StableDiffusion large model of the open platform Huggingface to generate a series of scene pictures of different perspectives and times; Step S6.3: Call the SDS tool, use score distillation sampling to extract 3D knowledge from the 2D picture and the series of scene pictures, calculate the loss function compared with the simulation picture, and obtain the gradient optimization initial vorticity; Step S6.4: According to the gradient optimization initial vorticity, the simulation picture is physically simulated to obtain the smoke fluid reconstruction result.

4. A three-dimensional Gaussian-based real fluid reconstruction evolution system, characterized in that, It includes: A fluid video reconstruction module, a connection module, and a fluid simulation module; The fluid video reconstruction module collects and produces fluid smoke data, and uses a method based on three-dimensional Gaussian and physical constraints to reconstruct the fluid smoke data to obtain a smoke dynamic scene; The connection module uses the smoke dynamic scene to learn the vorticity of the Gaussian particles using a differentiable simulation method to obtain the simulation initial vorticity; The fluid simulation module fills the reconstructed Gaussian particles in the smoke using a grid method to obtain reconstructed Gaussian particles, and performs physical simulation according to the simulation initial vorticity to obtain a simulation picture and correct it through a large model to obtain a smoke fluid reconstruction result; The fluid video reconstruction module includes: Module M1.1: Select the real smoke physical data captured by the photography technology provided by the ScalarFlow smoke dataset, including the velocity field and density field of the smoke, and use the OpenVDB library to convert the NPZ format data density field into a VDB format smoke; Module M1.2: Blender is used to build a dynamic scene, smoke in VDB format is processed, light sources and cameras are set; Module M1.3: dynamic smoke is rendered from different angles in a spherical region in a spherical ring mode to obtain pictures, and the poses of the cameras and the three-dimensional coordinates of the Gaussian particle positions are outputted; Module M1.4: a corresponding timestamp is set for each picture, the timestamp is uniformly sampled in the range of 0 to 1, and smoke data is outputted; Module M2.1: smoke data is read in, a four-dimensional Gaussian six-plane space-time encoder is used to encode three-dimensional coordinates and timestamps, and four-dimensional data is divided into six two-dimensional planes; Module M2.2: a multi-head deformation decoder is used to output variable correction values of position deformation and scaling deformation, and variable values of the final particle positions and time are obtained according to the three-dimensional coordinates and timestamp variable values in the smoke data; Module M2.3: a physical constraint based on a velocity field is used as an image L1 loss and a total variation loss to jointly train the final particle positions and time, HexPlane feature planes in the space-time encoder, MLP feature fusion and the multi-head deformation decoder; Module M2.4: after training, image smoke data is obtained by a GS rendering method to form a video, and a dynamic smoke scene is obtained; The connection module comprises: Module M3.1: an initial vorticity variable of the Nth last Gaussian particle is set; Module M3.2: Use the Biot-Savart formula to calculate the velocity from the vorticity for the next N-1 frames using the vortex method Wherein, u(x) represents a velocity field at an x position; ω i represents the vorticity of the i-th Gaussian particle; Γ i represents the vorticity intensity of the i-th Gaussian particle; (x-x i ) represents the distance between the x position and the i-th Gaussian particle; Module M3.3: Advect Lagrangian particles using RK3 formula with computed velocities, k1 = u(x n )Δt = f(t n , x n )Δt, k3 = u(x n -k1 + 2k2)Δt = f(t n +Δt, x n -k1 + 2k2)Δt, where t n denotes the current time step; Δt represents a time step; x n represents a known position; f() represents a function related to time t and physical quantity x; k1 represents a slope at the beginning of a time period; k2 and k3 respectively represent slopes determined by the slopes k1 and k2 at the middle of the time period; x n+1 represents the position of the next time step; Module M3.4: supervised training is performed according to subsequent N-1 frames of rendered images obtained by rendering the Gaussian particles after convection, the initial vorticity is optimized by micro-simulation, and the optimized vorticity is obtained; Module M3.5: through supervised training, the vorticity of the Nth last frame is optimized by using the optimized vorticity and the subsequent rendered images, and a simulated initial vorticity is obtained.

5. The three-dimensional Gaussian-based true fluid reconstruction evolution system of claim 4, wherein, The fluid simulation module comprises: Module M4.1 : discretize the whole 3D space using a regular grid, compute the density of the grid at the center of the grid from the number of Gaussian particles and opacity where δ p represents the opacity of the Gaussian particle p; x represents the position of the center of the grid; x p represents the position of each Gaussian particle p; ∑ p denotes the covariance of the Gaussian particle p; Module M4.2: a ray is emitted along a Cartesian coordinate direction or a random direction, the number of intersection points of the ray and the boundary of the object is judged, if the number of intersection points is greater than a threshold, it is determined to be inside, a new Gaussian particle with the same color attribute as the nearest Gaussian particle and a vorticity of 0 is added, if the number of intersection points is less than or equal to the threshold, it is determined to be outside, and no Gaussian particle is filled; All grid centers are traversed to obtain reconstructed Gaussian particles; The boundary of the object is a dividing line between a grid lower than a set threshold and a grid higher than the set threshold; Module M5.1: the reconstructed Gaussian particles are used as vortex particles, and the positions and vorticity of the vortex particles are initialized; Module M5.2: the Biot-Savart formula is used to calculate the velocity and new vorticity of the vortex particles according to the vorticity; Module M5.3: the RK3 formula is used to convect the vortex particles according to the velocity to obtain new positions of the vortex particles; The vortex trigger modules M5.2 and M5.3 are repeated every N frames to determine the vorticity and set a maximum threshold of vorticity. If the absolute value of the vorticity is greater than the maximum threshold of vorticity, the vortex particle splits into two vortex particles at the offset position, and each of the two vortex particles carries half of the vorticity and opacity of the original vortex particle, inherits the original spin and scaling properties. If the absolute value of the vorticity is less than or equal to the maximum threshold of vorticity, no operation is performed until all frames are traversed, and a simulation picture is obtained.

6. The three-dimensional Gaussian-based true fluid reconstruction evolution system of claim 4, wherein, The fluid simulation module comprises: Module M6.1: input the simulation picture into the large model, set the vorticity of the vortex particle and the newly added vortex particle at the beginning of the simulation as an optimizable parameter, use the vortex method for simulation calculation, and render it into a 2D picture; Module M6.2: input the scene statement, call the StableDiffusion large model of the open platform Huggingface to generate a series of scene pictures of different perspectives and times; Module M6.3: call the SDS tool, use score distillation sampling to extract 3D knowledge from the 2D picture and the series of scene pictures, calculate the loss function compared with the simulation picture, and obtain the gradient optimization initial vorticity; Module M6.4: according to the gradient optimization initial vorticity, the simulation picture is physically simulated to obtain the smoke fluid reconstruction result.

Images