Method, device and equipment for reconstructing fluid-solid mixed scene based on sparse view angle video
Through implicit neural network, the neural feature track field model and neural radiation field model are constructed, combined with the neural distance field, the accuracy problem of fluid solid mixed scene reconstruction in sparse view video is solved, and physically accurate fluid and obstacle reconstruction is achieved to generate realistic rendered image.
Patent Information
- Application Number
- CN202410922918.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-29
AI Technical Summary
In the prior art, when processing sparse viewing videos, it is difficult to support the reconstruction of streamlined mixed scenarios, and the reconstruction accuracy is low, especially when dynamic fluid information is complex and obstacles exist, it is difficult to achieve physically accurate reconstruction.
Using the neural feature track field model, neural radiation field model and neural distance field model based on implicit neural network, the density distribution, velocity field and geometric information of solid obstacles of the fluid are reconstructed through the joint optimization of the preset loss function to satisfy the fluid The long-term physical conservation and instantaneous physical laws of motion.
The physical accurate reconstruction of convective solid hybrid scenes is realized, the physical rationality and temporal consistency of the reconstruction results are ensured, and the realistic new perspective rendering images are generated, which improves the accuracy and efficiency of dynamic fluid scene reconstruction.
Smart Images

Figure CN119048665A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of computer graphics, and in particular to a method, device and equipment for reconstructing a fluid-solid hybrid scene based on sparse perspective video. Background Art
[0002] Fluid reconstruction technology is an important research direction at the intersection of computer graphics and fluid dynamics. This technology aims to recover the three-dimensional structure and dynamic properties of the fluid from the observed data. Traditional fluid reconstruction methods rely on specific hardware and lighting settings, such as particle imaging velocimetry, which is achieved by tracking passive markers in the fluid. With the development of technology, reconstruction methods based on RGB video have gradually reduced their dependence on professional equipment and improved the flexibility and efficiency of reconstruction. How to reconstruct fluids as physically accurate as possible based on limited RGB videos has become the focus of research in this field.
[0003] The current scheme, when processing sparse view videos, is very difficult to reconstruct dynamic fluid information due to factors such as rapid fluid dynamics, occlusion, and uncertainty in illumination color, especially in scenes with obstacles. In addition, existing neural network methods based on physical information focus on applying short-term physical constraints to reduce instantaneous momentum loss, but are insufficient in maintaining long-term energy, mass, and momentum conservation, which limits the effectiveness and physical accuracy of their reconstruction results. Summary of the invention
[0004] The embodiments of the present application provide a method, device and equipment for reconstructing a fluid-solid mixed scene based on sparse perspective video, so as to at least solve the technical problems in the related art that it is difficult to support the reconstruction of the fluid-solid mixed scene and the reconstruction accuracy is low.
[0005] According to one aspect of an embodiment of the present application, a method for reconstructing a fluid-solid hybrid scene based on a sparse perspective video is provided, comprising:
[0006] Obtaining a dynamic video image of a scene and camera parameters corresponding to the video image as input;
[0007] Constructing a neural feature trajectory field model, a neural radiation field model and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property;
[0008] The preset loss function is used to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model to achieve the reconstruction of solid obstacles in the video image, the density distribution of the fluid, the velocity field and the trajectory that changes over time.
[0009] In an optional embodiment, constructing a neural feature trajectory field model includes:
[0010] Constructing a neural feature trajectory field model based on implicit neural network;
[0011] Constructing a trajectory encoder and a trajectory decoder of the neural feature trajectory field model based on a multi-layer perceptron;
[0012] The neural feature trajectory field model is in the form of: f(x, t) = z;
[0013] Among them, x represents the three-dimensional space coordinate, t represents the time coordinate, and z represents the trajectory of the particle corresponding to the space-time coordinate.
[0014] In an optional embodiment, before jointly optimizing the neural feature trajectory field model, the neural radiation field model and the neural distance field model using a preset loss function, the method further includes:
[0015] Based on the physical characteristics of fluid motion, an essential constraint function and a physical constraint function are established, wherein the essential constraint function and the physical constraint function include an instantaneous constraint and a long-term constraint respectively;
[0016] Constructing a fine density model based on a neural radiation field and a Lagrangian rough density model based on a multi-layer perceptron, and constructing a density constraint function based on the fine density model and the rough density model;
[0017] Extracting geometric information of solid obstacles based on the neural distance field, and constructing a boundary constraint function according to the geometric information;
[0018] The essential constraint function, physical constraint function, density constraint function and boundary constraint function are used as the preset loss function.
[0019] In an optional embodiment, based on the physical characteristics of fluid motion, an essential constraint function and a physical constraint function are established, including:
[0020] Based on the trajectory path encoding and decoding consistency constraints, the first essential constraint function is constructed;
[0021] Based on the constraint that the path encoding of the same particle remains unchanged on its trajectory, a second essential constraint function is constructed;
[0022] Based on the constraint that the body derivative of the particle path encoding is 0, the third essential constraint function is constructed;
[0023] Based on the constraints of the fluid transport equation, a first physical constraint function is constructed;
[0024] Based on the constraints of the fluid Navier-Stokes equations, a second physical constraint function is constructed;
[0025] A third physical constraint function is constructed based on the constraints limiting fluid convection derived from co-vector advection.
[0026] In an optional embodiment, a fine density model based on a neural radiation field and a Lagrangian rough density model based on a multi-layer perceptron are constructed, and a density constraint function is constructed based on the fine density model and the rough density model, including:
[0027] Based on the constraints of the video image, construct a first density constraint function;
[0028] Based on the density mapping constraint, a second density constraint function is constructed;
[0029] Based on the color mapping constraint, a third density constraint function is constructed;
[0030] Based on the coarse and fine density field consistency constraints, the fourth density constraint function is constructed;
[0031] Based on the transmission constraints of density in the fine density model and the coarse density model, a fifth density constraint function is constructed.
[0032] In an optional embodiment, extracting geometric information of solid obstacles based on the neural distance field, and constructing a boundary constraint function according to the geometric information includes:
[0033] Based on the density constraint at the boundary, construct a first boundary constraint function;
[0034] Based on the velocity constraint at the boundary, a second boundary constraint function is constructed.
[0035] In an optional embodiment, a preset loss function is used to jointly optimize the neural feature trajectory field model, the neural radiation field model, and the neural distance field model to achieve reconstruction of solid obstacles in video images, density distribution of fluids, velocity fields, and time-varying trajectories, including:
[0036] The neural feature trajectory field model is jointly optimized using a preset loss function to obtain a velocity field of the fluid and a trajectory that changes over time;
[0037] The neural radiation field model is jointly optimized using a preset loss function to obtain a density distribution of the fluid;
[0038] The neural distance field model is jointly optimized using a preset loss function to obtain a reconstructed solid obstacle.
[0039] According to another aspect of an embodiment of the present application, a fluid-solid hybrid scene reconstruction device based on sparse perspective video is also provided, comprising:
[0040] An acquisition module, used to acquire a dynamic video image of a scene and camera parameters corresponding to the video image as input;
[0041] A construction module is used to construct a neural feature trajectory field model, a neural radiation field model and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property;
[0042] The reconstruction module is used to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model using a preset loss function to achieve reconstruction of solid obstacles, fluid density distribution, velocity field and trajectory that changes over time.
[0043] In an optional embodiment, it also includes:
[0044] The loss function construction module is used to establish an essential constraint function and a physical constraint function based on the physical characteristics of fluid motion, wherein the essential constraint function and the physical constraint function respectively include an instantaneous constraint and a long-term constraint; construct a fine density model based on a neural radiation field and a Lagrangian rough density model based on a multi-layer perceptron, and construct a density constraint function based on the fine density model and the rough density model; extract geometric information of solid obstacles based on the neural distance field, and construct a boundary constraint function according to the geometric information; and use the essential constraint function, the physical constraint function, the density constraint function and the boundary constraint function as the preset loss function.
[0045] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned fluid-solid hybrid scene reconstruction method based on sparse perspective video through the computer program.
[0046] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0047] The reconstruction method of the embodiment of the present application proposes a neural feature trajectory field technology, which uses the Euler space-time field constructed by an implicit neural network to simulate fluid trajectories, implement the constraints of long-term physical conservation laws, and satisfy the instantaneous physical laws of the Navier-Stokes equations. A more reliable constraint relationship can be established to provide physically accurate reconstruction. The neural radiation field and the neural distance field are integrated to finely reconstruct the three-dimensional radiosity distribution of the dynamic scene through the neural radiation field, and the geometric information of obstacles is extracted using the neural distance field to achieve physically accurate reconstruction of the fluid-solid mixed scene. Through the comprehensive application of the above-mentioned technologies, accurate reconstruction of the density, velocity, and trajectory of dynamic fluids such as smoke can be achieved, while ensuring the physical rationality and temporal consistency of the reconstruction results, and generating realistic new perspective rendered images. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0049] Figure 1 is a flowchart of an optional method for reconstructing a fluid-solid hybrid scene based on sparse perspective video according to an embodiment of the present application;
[0050] Figure 2 is a schematic diagram of a neural feature trajectory field according to an embodiment of the present application;
[0051] Figure 3 is a schematic diagram of a smoke reconstruction effect according to an embodiment of the present application;
[0052] Figure 4 is a structural diagram of a fluid-solid hybrid scene reconstruction device based on sparse perspective video according to an embodiment of the present application;
[0053] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0056] From aircraft navigation to ocean circulation, physical phenomena such as fluids have a profound impact on our lives. Reconstructing and predicting dynamic fluid phenomena in real life such as smoke and hurricanes from video and other observational data, and accurately capturing the interaction between fluids and solids, are crucial for practical tasks such as climate forecasting, aerodynamic design of vehicles, and drug metabolism in the human body, and have broad application prospects. Using video as input, collecting and reconstructing the three-dimensional density and velocity field of fluids that change over time can help people better understand and verify various complex physical phenomena in the real world.
[0057] However, it is very difficult to reconstruct the complex dynamic fluid information based on the video content and camera pose using only sparse multi-view video information, without relying on information such as lighting and complex geometric composition of the scene. The fluid dynamics in the video are very fast, and there are difficulties such as occlusion and lighting color uncertainty in videos with limited viewing angles. The challenge lies in how to use prior knowledge such as physical principles to establish reliable constraint relationships, thereby solving the information gap between limited information input and complex multi-degree-of-freedom reconstruction results, and providing physically accurate reconstruction results. Existing neural network methods based on physical information often focus on applying short-term physical constraints to reduce instantaneous momentum loss, but are insufficient in maintaining long-term energy, mass, and momentum conservation. This limits the effectiveness and accuracy of these methods in complex dynamic scenes.
[0058] In the face of the above practical needs and challenges, this solution proposes a method for reconstructing fluid mixing scenes based on characteristic trajectories. Specifically, a neural characteristic trajectory field is proposed, which is a new representation that uses implicit neural networks to form an Euler space-time field to simulate fluid trajectories that conform to Lagrangian properties. Based on the neural characteristic trajectory field, "long-term" constraints can be constructed based on the laws of physical conservation, that is, physical properties such as mass do not change along the trajectory of material movement. In addition, the neural characteristic trajectory field can provide fluid velocity information at any point in space and time through the automatic differentiation technology of neural networks. The velocity provided by the trajectory field and its time and space differentials can be further constrained to satisfy the underlying physical laws of fluid motion, namely the Navier-Stokes equations, thereby constructing "instantaneous" constraints.
[0059] In the task of reconstructing dynamic scenes, the above neural feature trajectory field is combined with the neural radiation field commonly used in current three-dimensional scene reconstruction. Therefore, the neural radiation field is used to reconstruct the distribution of radiance in time and space in the scene from multi-view videos. Our feature trajectory field is used to add "long-term" and "instantaneous" physical constraints at the same time, so that the reconstructed radiance distribution and trajectory field can satisfy the image information in the video, as well as the instantaneous physical laws of fluid motion and the long-term law of conservation of matter, thereby reconstructing the physically accurate neural radiance spatiotemporal distribution and trajectory expression to complete the reconstruction of dynamic scenes. It is also possible to further integrate the implicit distance field based on neural expression to support the reconstruction of mixed scenes containing static solids and dynamic fluids. This method can automatically segment fluids and solids and physically accurately reconstruct the interaction of boundaries.
[0060] The following is combined with Figure 1 The fluid-solid hybrid scene reconstruction method based on sparse perspective video in the embodiment of the present application is described in detail. Figure 1 As shown, the method mainly includes the following steps:
[0061] S101 obtains a dynamic video image of a scene and camera parameters corresponding to the video image as input.
[0062] The reconstruction method provided in the embodiment of the present application can be reconstructed based on images with sparse perspectives, for example, by using only five cameras for shooting.
[0063] In an exemplary scenario, multiple cameras are set up to capture smoke flow scenes from different angles, treating smoke as a fluid. Each camera shoots a two-minute video, and the captured video images and the corresponding camera parameters are used as input.
[0064] S102 constructs a neural feature trajectory field model, a neural radiation field model and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property.
[0065] The embodiment of the present application proposes a neural feature trajectory field model to construct a neural feature trajectory field model, including: constructing a neural feature trajectory field model based on an implicit neural network; constructing a trajectory encoder and a trajectory decoder of the neural feature trajectory field model based on a multi-layer perceptron; the form of the neural feature trajectory field model is: f(x, t) = z.
[0066] Among them, x represents the three-dimensional space coordinate, t represents the time coordinate, and z represents the trajectory of the particle corresponding to the space-time coordinate.
[0067] From the Lagrangian perspective, the motion of a fluid is composed of the motion of countless particles. Assume that the motion trajectory of each particle uses a high-dimensional characteristic attribute To describe, among them, represents the vector space, and D represents the spatial dimension. Then the neural feature trajectory field aims to use implicit neural networks to describe the spatiotemporal continuous feature field formed by countless particles, that is, f(x, t) = z, where x is the three-dimensional space coordinate, t is the time coordinate, and the corresponding z describes the trajectory of the particle corresponding to the spatiotemporal coordinate.
[0068] like Figure 2 As shown, the neural feature trajectory field consists of a trajectory encoder that extracts consistent features z along the trajectory and a trajectory decoder. The velocity can be extracted from the decoder through automatic differentiation.
[0069] In one embodiment, a multi-layer perceptron (MLP) ε is used to learn the encoding of this neural trace feature field, performing the following operations: The network ε establishes a connection between the Euler coordinates and the Lagrangian feature representation. In order to map the high-dimensional feature attribute z back to the particle coordinate x' corresponding to a certain time t', another multi-layer perceptron (MLP) is trained while training ε Responsible for learning the decoding of z, that is In practice, this application uses
[0070] In the optimization of a specific scenario, ε and The time evolution of any point can be quickly predicted. Given a 4D point (x, t) and another time t′, its corresponding position x′ at time t′ can be directly obtained through a single network prediction: Therefore, for the space-time velocity field u(x, t): The method of this application simplifies the integral calculation to: Compared with the previous work that convections each frame to achieve velocity integration and establish time correspondence, the representation of this application only requires a single network prediction and can quickly support long-term mapping of more than 50 frames. This enables this application to impose long-term mapping constraints that are critical for high-fidelity reconstruction, which is difficult to do in the past.
[0071] Furthermore, the use of the above-mentioned neural characteristic trajectory field to describe the trajectory of fluid motion requires the guidance of physics awareness to ensure that the trajectory conforms to the laws of physics and the physical characteristics of fluid motion.
[0072] Therefore, before using the preset loss function to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model, it also includes: based on the physical characteristics of fluid motion, establishing an essential constraint function and a physical constraint function, the essential constraint function and the physical constraint function respectively include instantaneous constraints and long-term constraints;
[0073] First, a set of "essential constraints" are introduced, which are based on the Lagrangian properties of the object's motion path.
[0074] Specifically, based on the physical characteristics of fluid motion, the essential constraint functions are established including:
[0075] Based on the path encoding and decoding consistency constraints of the trajectory, the first essential constraint function is constructed.
[0076]
[0077] in, Represents the path encoding and decoding consistency constraint of the trajectory, which is the first essential constraint function.
[0078] Based on the constraint that the path encoding of the same particle remains unchanged on its trajectory, the second essential constraint function is constructed.
[0079]
[0080] in, It means that the path encoding of the same particle on its trajectory remains unchanged, which is the second essential constraint function. This is the Lagrangian property of the particle trajectory and is a long-term constraint.
[0081] Based on the constraint that the body derivative encoded by the particle path is 0, the third essential constraint function is constructed.
[0082]
[0083] in, The constraint that the body derivative of the particle path encoding is 0, that is, the third essential constraint function, ensures that the body derivative of the path encoding at the current moment does not change from the Euler perspective. This is an instantaneous constraint. The maintenance of the above essential constraints ensures that the path encoding obtained by learning corresponds to a physically reliable path.
[0084] Furthermore, this application introduces "physical constraints" based on fluid dynamics, such as the Navier-Stokes equations and the mass conservation equation. The former is an instantaneous constraint, while the latter is a long-term constraint. Among these constraints, the velocity u can be obtained by automatically differentiating the neural feature trajectory field: where t 1 , t 2 is ε and time inputs, consistently share the same time value. 2 The reason for taking partial derivatives is that when a particle moves along its trajectory, its characteristic z = ε (x, t 1 ) should be unchanged. Similarly, you can use The second-order automatic differentiation of Calculate divergence, curl and other physical quantities.
[0085] Specifically, based on the constraints of the fluid transport equation, a first physical constraint function is constructed;
[0086]
[0087] in, represents the constraint of the fluid transport equation, which is also the first physical constraint function, and σ represents the fluid density. The instantaneous constraint ensures that the body derivative of the fluid mass does not change at the current moment.
[0088] Based on the constraints of the fluid Navier-Stokes equations, a second physical constraint function is constructed.
[0089]
[0090] in, represents the constraint based on the Navier-Stokes equations of the fluid, which is the second physical constraint function. Represents the objective function that constrains the divergence of the fluid velocity field to zero. The constraint is an instantaneous constraint that ensures that the fluid velocity field is divergent and incompressible.
[0091] A third physical constraint function is constructed based on the constraints limiting fluid convection derived from co-vector advection.
[0092] Among them, z a =ε(x a , t a ),
[0093] M a→b
[0094] in, Represents the constraint that limits fluid convection, which is the third physical constraint function, u a Indicates that the fluid is at x a Position a The speed at the moment, u b Indicates that the fluid is at x b Position b The speed of the moment, M a→b represents the deformation matrix corresponding to the co-vector convection,
[0095] This constraint, derived from covector advection, is a long-term constraint that facilitates globally reliable and non-dissipative velocity extraction.
[0096] Through the joint optimization of the above essential constraints and physical constraints, and by using long-term and short-term constraint functions, a fluid trajectory extraction with more physical accuracy and more complete global information can be obtained.
[0097] Furthermore, a neural radiance field model is constructed. Based on the neural feature trajectory field, the present application further combines the neural radiance field (NeRF) technology to achieve a fine reconstruction of the fluid scene.
[0098] Although the neural radiation field model is a neural network known in the prior art, the present application innovatively proposes a dual-density strategy.
[0099] Although the shape is F N (x, t) = (σ N , c) time-varying neural radiance field networks, which can represent dynamic scenes, but a large body of work has found that training them with sparse views leads to high underdetermination. This underdetermination σ N Further through Pollution speed.
[0100] To alleviate this underdeterminism, this application proposes a novel dual-density strategy, which is supplemented by the long-term temporal supervision of the trajectory field of this application. The dual-density strategy of this application includes a fine model F N (x, t) is used as the Euler representation of the density field, and a Lagrangian rough density model σ based on a multi-layer perceptron (MLP) is proposed L =F L (ε(x,t)). and σ N In contrast, the Lagrangian density σ L Taking our trajectory features as prior input, a Lagrangian rough density model σ based on a multi-layer perceptron (MLP) L =F L (ε(x,t)). and σ N In contrast, the Lagrangian density σ L The trajectory characteristics of the present application are used as a priori input, which has limited degrees of freedom and is subject to strong physical constraints, and has the advantage of being more consistent along the particle trajectory.
[0101] This application constructs a fine density model based on neural radiation field and a Lagrangian rough density model based on multi-layer perceptron, and constructs a density constraint function based on the fine density model and the rough density model.
[0102] Specifically, we use the image and mapping constraints to train F N (x, t), based on the constraints of the video image, construct the first density constraint function.
[0103]
[0104] in, Represents the constraint of the video image, that is, the first density constraint function, and C represents the color value corresponding to the light r.
[0105] Based on the density mapping constraint, a second density constraint function is constructed.
[0106]
[0107] in, represents the density mapping constraint, which is the second density constraint function, σ a Represents the refined model F N The fluid obtained in x a Position a The fluid density at the time, σ b Represents the refined model F N The fluid obtained in x b Position b The density of the fluid at that moment.
[0108] Based on the color mapping constraint, a third density constraint function is constructed.
[0109] Where (σ a , c a )=F N (x a , t a ), (σ b , c b )=F N (x b , t b )
[0110] in, Represents the color mapping constraint, which is the third density constraint function, c a Represents the refined model F N The scene obtained in x a Position a The object color at time c b Represents the refined model F N The scene obtained in x b Position b The color of the object at that moment.
[0111] Based on the coarse and fine density field consistency constraints, the fourth density constraint function is constructed.
[0112]
[0113] in, represents the consistency constraint of the coarse and fine density fields, that is, the fourth density constraint function, σ N Represents the refined model F N The scene density obtained in L Represents the rough model F L The scene density obtained in .
[0114] Lagrangian density σ L The gradient shows a higher degree of structure and is very suitable for use Training speed. Neural radiation field density σ N Effectively capture more high-frequency details, which is crucial for reconstructing high-fidelity density and speed results. Therefore, the dual-density strategy of this application converts the original NeRF density σ with a larger degree of freedom into N As a scene representation, while using constraints that better L As a speed guide.
[0115] Therefore, based on the transport constraints of density in the fine density model and the coarse density model, a fifth density constraint function is constructed.
[0116]
[0117] in, Represents transport constraints on density in fine density models and coarse density models.
[0118] By combining the neural feature trajectory field with the neural radiation field, the proposed Loss function, and neural radiance field The loss function is optimized to simultaneously reconstruct the density distribution, velocity field, and time-varying trajectory of the smoke, thereby reproducing a smoke scene that is both physically consistent and realistic.
[0119] Furthermore, in order to process mixed scenes containing smoke and obstacles (such as buildings, trees, etc.), the present application also proposes a fluid-solid mixed scene reconstruction method combined with a neural distance field.
[0120] This application uses an additional neural distance field, the NeuS model, to extract geometric information of obstacles from RGB video. Based on the neural distance field, the geometric information of solid obstacles is extracted, and the boundary constraint function is constructed according to the geometric information; these geometric information are used as boundary constraints to further guide the reconstruction process of smoke and ensure that the interaction between smoke and obstacles complies with physical laws. The following constraints are used to restrict the movement of fluid at the boundary to comply with physical laws:
[0121] Based on the density constraint at the boundary, a first boundary constraint function is constructed; based on the speed constraint at the boundary, a second boundary constraint function is constructed.
[0122]
[0123] in, represents the density constraint at the boundary, which is also the first boundary constraint function, Represents the velocity constraint at the boundary, which is the second boundary constraint function. Mask represents the mask operation, SDF represents the neural distance field, x represents a position in space, u represents the spatiotemporal velocity field, n represents the distance normal direction calculated from the neural distance field, and t represents a moment in time.
[0124] Among them, SDF(x)≤0 represents the internal area of the obstacle, The fluid density should be 0; SDF(x)≤∈ represents the boundary area. It is required that the fluid velocity at the boundary is tangent to the obstacle surface and that there is no fluid velocity inside the solid.
[0125] The essential constraint function, physical constraint function, density constraint function and boundary constraint function are used as preset loss functions.
[0126] S103 uses a preset loss function to jointly optimize the neural feature trajectory field model, the neural radiation field model, and the neural distance field model to achieve reconstruction of solid obstacles in video images, density distribution of fluids, velocity fields, and time-varying trajectories.
[0127] In an optional embodiment, a preset loss function is used to jointly optimize the neural feature trajectory field model to obtain the velocity field of the fluid and its trajectory changing over time; a preset loss function is used to jointly optimize the neural radiation field model to obtain the density distribution of the fluid; and a preset loss function is used to jointly optimize the neural distance field model to obtain a reconstructed solid obstacle.
[0128] Specifically, the proposed neural feature trajectory field is combined with Loss function, and neural radiance field proposed Loss function, and boundary function proposed by neural distance field After optimization, it is possible to simultaneously reconstruct mixed scenes with obstacles and fluids, realize the reconstruction of obstacles, the density distribution, velocity field and time-varying trajectory of the fluid, and reproduce realistic mixed scenes that conform to both physical laws and geometric boundary conditions.
[0129] like Figure 3 The figure shows a reconstruction effect of obstacles and smoke fluids. The dynamic mixed scene containing obstacles and smoke in the figure is shot, and multi-angle videos are input. Then, the obstacle boundary, smoke speed, density, and trajectory are reconstructed based on the neural feature trajectory field, neural radiation field, and neural distance field. A new perspective rendering image is obtained.
[0130] This application uses implicit neural networks to simulate fluid trajectories, and introduces essential constraints and physical constraints through physically informed deep learning to ensure the physical accuracy of fluid trajectories. Combining neural radiation fields with neural distance fields, this application can accurately reconstruct the three-dimensional radiance distribution of dynamic scenes, and extract the geometric information of obstacles to achieve physically accurate reconstruction of fluid-solid mixed scenes. Ultimately, this technology can accurately reconstruct the density, velocity, and trajectory of fluids such as smoke, ensure the physical rationality and temporal consistency of the reconstruction results, and generate realistic new perspective rendering images, providing strong technical support for applications such as aerodynamic design and smoke leak detection, and significantly improving the accuracy and efficiency of dynamic fluid scene reconstruction.
[0131] This application can be applied to the field of observing and studying aerodynamics: This method can reconstruct fluid dynamics from video data, including smoke and obstacles in complex scenes, without relying on specific lighting conditions or detailed geometric information of the scene. This provides a new tool for aerodynamic research, which can be used to record and analyze air flow characteristics, so as to deeply understand the physical properties and dynamic behavior of fluids. It can be applied to fluid reconstruction tasks in medical imaging, and this technology also has great potential in the medical field. By analyzing medical imaging data such as magnetic resonance imaging of hemangiomas, blood flow characteristics can be reconstructed and studied. This is of great significance for the formulation of diagnosis and treatment plans, and can help doctors assess the condition more accurately. It can be applied to the field of aerodynamic design. When designing transportation tools such as vehicles and drones, this technology can assist designers in predicting and verifying the impact of air flow on these devices through observation, recording and reconstruction analysis, which can optimize product performance, reduce resistance and improve energy efficiency. It can be applied to scientific research and education, and this technology also plays an important role in scientific education and research. It provides an intuitive tool to help researchers and students better understand the complex concepts of fluid dynamics.
[0132] According to another aspect of the embodiment of the present application, a fluid-solid hybrid scene reconstruction device based on sparse perspective video is provided for implementing the fluid-solid hybrid scene reconstruction method based on sparse perspective video. Figure 4 As shown, the device comprises:
[0133] An acquisition module 401 is used to acquire a dynamic video image of a scene and camera parameters corresponding to the video image as input;
[0134] A construction module 402 is used to construct a neural feature trajectory field model, a neural radiation field model, and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property;
[0135] The reconstruction module 403 is used to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model using a preset loss function to achieve reconstruction of solid obstacles, density distribution of fluids, velocity fields and time-varying trajectories.
[0136] In an optional embodiment, it also includes:
[0137] The loss function construction module is used to establish the essential constraint function and the physical constraint function based on the physical characteristics of fluid motion. The essential constraint function and the physical constraint function include instantaneous constraints and long-term constraints respectively; construct a fine density model based on the neural radiation field and a Lagrangian rough density model based on the multi-layer perceptron, and construct a density constraint function based on the fine density model and the rough density model; extract the geometric information of solid obstacles based on the neural distance field, and construct a boundary constraint function according to the geometric information; use the essential constraint function, physical constraint function, density constraint function and boundary constraint function as the preset loss function.
[0138] It should be noted that the fluid-solid hybrid scene reconstruction device based on sparse perspective video provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the fluid-solid hybrid scene reconstruction method based on sparse perspective video. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the fluid-solid hybrid scene reconstruction device based on sparse perspective video provided in the above embodiment and the fluid-solid hybrid scene reconstruction method based on sparse perspective video belong to the same concept. The implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0139] According to another aspect of an embodiment of the present application, an electronic device corresponding to the fluid-solid hybrid scene reconstruction method based on sparse perspective video provided in the aforementioned embodiment is also provided to execute the above-mentioned fluid-solid hybrid scene reconstruction method based on sparse perspective video.
[0140] Please refer to Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, the electronic device includes: a processor 500, a memory 501, a bus 502 and a communication interface 503, and the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can be run on the processor 500, and when the processor 500 runs the computer program, it executes the fluid-solid hybrid scene reconstruction method based on sparse perspective video provided in any of the aforementioned embodiments of the present application.
[0141] The memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 503 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0142] The bus 502 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs, and the processor 500 executes the program after receiving the execution instruction. The fluid-solid hybrid scene reconstruction method based on sparse perspective video disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 500, or implemented by the processor 500.
[0143] The processor 500 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 500. The above processor 500 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.
[0144] The electronic device provided in the embodiment of the present application and the fluid-solid hybrid scene reconstruction method based on sparse perspective video provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.
[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0146] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.
Claims
1. A fluid-solid hybrid scene reconstruction method based on sparse view video, characterized in that: include: Obtaining a dynamic video image of a scene and camera parameters corresponding to the video image as input; Constructing a neural feature trajectory field model, a neural radiation field model and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property; The preset loss function is used to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model to achieve the reconstruction of solid obstacles in the video image, the density distribution of the fluid, the velocity field and the trajectory that changes over time.
2. The method according to claim 1, characterized in that Construct a neural feature trajectory field model, including: Constructing a neural feature trajectory field model based on implicit neural network; Constructing a trajectory encoder and a trajectory decoder of the neural feature trajectory field model based on a multi-layer perceptron; The form of the neural feature trajectory field model is: f(x, t) = z; Among them, x represents the three-dimensional space coordinate, t represents the time coordinate, and z represents the trajectory of the particle corresponding to the space-time coordinate.
3. The method according to claim 1, characterized in that Before jointly optimizing the neural feature trajectory field model, the neural radiation field model and the neural distance field model using a preset loss function, the method further includes: Based on the physical characteristics of fluid motion, an essential constraint function and a physical constraint function are established, wherein the essential constraint function and the physical constraint function include an instantaneous constraint and a long-term constraint respectively; Constructing a fine density model based on a neural radiation field and a Lagrangian rough density model based on a multi-layer perceptron, and constructing a density constraint function based on the fine density model and the rough density model; Extracting geometric information of solid obstacles based on the neural distance field, and constructing a boundary constraint function according to the geometric information; The essential constraint function, physical constraint function, density constraint function and boundary constraint function are used as the preset loss function.
4. The method according to claim 3, characterized in that Based on the physical characteristics of fluid motion, essential constraint functions and physical constraint functions are established, including: Based on the trajectory path encoding and decoding consistency constraints, the first essential constraint function is constructed; Based on the constraint that the path encoding of the same particle remains unchanged on its trajectory, a second essential constraint function is constructed; Based on the constraint that the body derivative of the particle path encoding is 0, the third essential constraint function is constructed; Based on the constraints of the fluid transport equation, a first physical constraint function is constructed; Based on the constraints of the fluid Navier-Stokes equations, a second physical constraint function is constructed; A third physical constraint function is constructed based on the constraints limiting fluid convection derived from co-vector advection.
5. The method according to claim 3, characterized in that: Constructing a fine density model based on neural radiation field and a Lagrangian rough density model based on multi-layer perceptron, and constructing a density constraint function based on the fine density model and the rough density model, including: Based on the constraints of the video image, construct a first density constraint function; Based on the density mapping constraint, a second density constraint function is constructed; Based on the color mapping constraint, a third density constraint function is constructed; Based on the coarse and fine density field consistency constraints, the fourth density constraint function is constructed; Based on the transmission constraints of density in the fine density model and the coarse density model, a fifth density constraint function is constructed.
6. The method according to claim 3, characterized in that Extracting geometric information of solid obstacles based on the neural distance field, and constructing a boundary constraint function according to the geometric information, including: Based on the density constraint at the boundary, construct a first boundary constraint function; Based on the velocity constraint at the boundary, a second boundary constraint function is constructed.
7. The method according to claim 1, characterized in that The neural feature trajectory field model, the neural radiation field model and the neural distance field model are jointly optimized using a preset loss function to achieve reconstruction of solid obstacles in video images, density distribution of fluids, velocity fields and time-varying trajectories, including: The neural feature trajectory field model is jointly optimized using a preset loss function to obtain a velocity field of the fluid and a trajectory that changes over time; The neural radiation field model is jointly optimized using a preset loss function to obtain a density distribution of the fluid; The neural distance field model is jointly optimized using a preset loss function to obtain a reconstructed solid obstacle.
8. A fluid-solid hybrid scene reconstruction device based on sparse perspective video, characterized in that: include: An acquisition module, used to acquire a dynamic video image of a scene and camera parameters corresponding to the video image as input; A construction module is used to construct a neural feature trajectory field model, a neural radiation field model and a neural distance field model; wherein the neural feature trajectory field model is a model that uses an implicit neural network to simulate a fluid trajectory that conforms to the Lagrangian property; The reconstruction module is used to jointly optimize the neural feature trajectory field model, the neural radiation field model and the neural distance field model using a preset loss function to achieve reconstruction of solid obstacles, fluid density distribution, velocity field and trajectory that changes over time.
9. The device according to claim 8, characterized in that Also includes: The loss function construction module is used to establish an essential constraint function and a physical constraint function based on the physical characteristics of fluid motion, wherein the essential constraint function and the physical constraint function respectively include an instantaneous constraint and a long-term constraint; construct a fine density model based on a neural radiation field and a Lagrangian rough density model based on a multi-layer perceptron, and construct a density constraint function based on the fine density model and the rough density model; extract geometric information of solid obstacles based on the neural distance field, and construct a boundary constraint function according to the geometric information; and use the essential constraint function, the physical constraint function, the density constraint function and the boundary constraint function as the preset loss function.
10. An electronic device, characterized in that: It comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the fluid-solid hybrid scene reconstruction method based on sparse perspective video as described in any one of claims 1 to 7 when executing the program instructions.
Citation Information
Cited By
VOCs fluid three-dimensional reconstruction method and device based on neural fluid field
CN119669601A
Method and device for 3d reconstruction of vocs fluid based on neural fluid field
CN119669601B
