A micromechanically-based monocular video reconstruction and extrapolation simulation method
By combining deep learning and spring-mass physics, the problem of dynamic object reconstruction and extrapolation simulation in monocular video is solved, achieving high-quality dynamic object reconstruction and physical parameter identification, and supporting real-time rendering under new perspectives and physical states.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 陈永强
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies rely on multi-view video, which cannot accurately reconstruct dynamic objects from monocular video and predict their future dynamics, and have limited adaptability to real objects.
By employing data induction priors based on deep learning neural networks and empirical induction priors based on the physical mechanics and physical optics of springs and particles, combined with depth estimation and camera pose prediction, a 3D Gaussian model is used for initialization and physical parameter identification. Differentiable physical rendering and backpropagation optimization are then used to achieve the reconstruction and extrapolation simulation of monocular video.
It enables accurate reconstruction of the physical properties of dynamic objects from monocular videos, and supports extrapolation simulation of user-specified external forces and real-time rendering from new perspectives, thereby improving reconstruction quality and prediction accuracy.
Smart Images
Figure CN122134935A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and computer graphics, and in particular to a method for monocular video reconstruction and extrapolation simulation based on differentiable physics. Background Technology
[0002] Reconstructing and simulating dynamic objects from videos is a fundamental challenge in computer vision. Accurately modeling the physical properties of dynamic objects is crucial for creating immersive AR / VR experiences and enabling robots to understand and interact with common moving objects in daily life. Furthermore, this method can be used to construct experimental apparatus for physics teaching, allowing students to record videos with their mobile phones and obtain the motion state and physical parameters of the recorded objects, learning physical principles. For example, inputting correct physical formulas is essential for accurately modeling physical states, and the intuitive impact of incorrect physical formulas can be visually experienced through rendered videos.
[0003] Existing dynamic scene reconstruction methods, such as 3D Gaussian and its dynamically extended 4D Gaussian, have made significant progress in capturing temporal changes in the appearance and geometry of objects. However, these methods rely on dense multi-view video as input, making it impossible to accurately reconstruct dynamic objects from monocular video and predict their future dynamics. While some recent methods, such as PAC-NeRF, attempt to integrate physically based priors into 3D object representations using MPM material point methods, their ability to handle real-world objects, especially heterogeneous ones, is limited because they assume a known material model and assign only a single global physical parameter to the entire object, restricting their adaptability to real-world objects. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a monocular video reconstruction and extrapolation simulation method based on differentiable physics. This method solves the problems of existing technologies that rely on multiple synchronized multi-view videos for dynamic reconstruction and cannot perform physical extrapolation. This method introduces a data induction prior based on deep learning neural networks to simultaneously initialize the reconstruction of the input monocular video. It also incorporates an empirical induction prior based on spring-mass physics and physical optics to identify physical parameters. By accurately modeling the physical properties of dynamic objects through physical constraints, the reconstruction quality is improved. Furthermore, it enables extrapolation simulation of user-specified external forces and supports real-time rendering under new perspectives and physical states.
[0005] The specific technical solution is as follows: A method for monocular video reconstruction and extrapolation simulation based on differentiable physics includes the following steps: S1. Preprocessing: Acquire 2D video sequences and use neural networks to estimate their corresponding depth and camera intrinsic and extrinsic parameters; S2. Initialization: 3D Gaussian reconstruction is performed based on depth and camera intrinsic and extrinsic parameters. The anchor point connection relationship and initial values of physical parameters of the spring mass 3D Gaussian representation model are obtained through the initial values of the 3D Gaussian model and the nearest neighbor relationship of the point cloud. S3, Forward Propagation, Predicting 3D Gaussian Reconstruction and Physical Parameters: Utilizing differentiable physical simulation to calculate the physical state of the 3D Gaussian at each moment, and obtaining the rendered 2D video of the 3D Gaussian at the corresponding position in the input camera sequence through differentiable physical rendering. S4. Backpropagation to optimize 3D Gaussian reconstruction and physical parameters: Construct a loss function by comparing the difference between the rendered 2D video and the input 2D video, obtain the gradient using the backpropagation algorithm, update the physical parameters of the 3D Gaussian using the stochastic gradient descent algorithm, repeat S3-S4 until the loss function converges and obtains an accurate 3D Gaussian reconstruction result. S5, Extrapolation Simulation and Rendering: Performs physical simulation on the 3D Gaussian reconstruction results, calculates the physical motion of the 3D Gaussian under the external force conditions specified by the user, and renders the corresponding 2D rendered video under the specified viewpoint.
[0006] Furthermore, in S1, a depth estimation neural network is used to predict the depth D corresponding to the video, and a camera pose prediction neural network is used to predict the camera pose E and the camera intrinsic parameter K.
[0007] Furthermore, S2 is specifically implemented through the following sub-steps: (S2.1) Based on the depth and the combination of camera internal and external parameters, the corresponding initial 3D point cloud position is calculated by using the perspective camera inverse projection model; (S2.2) Use the position of the initial 3D point cloud as the initial mean of the 3D Gaussian; use the pixel color corresponding to the initial 3D point cloud as the initial color of the 3D Gaussian; initialize the covariance of the 3D Gaussian according to the nearest neighbor relationship of the initial 3D point cloud. Achieving 3D Gaussian reconstruction using the initial mean, initial color, and initial covariance of a 3D Gaussian; the reconstructed 3D Gaussian includes the following parameters: 3D Gaussian mean, covariance direction, covariance magnitude, opacity, and 3D Gaussian color; (S2.3) Using the initial values of the 3D Gaussian model and the nearest neighbor relationship of the point cloud, the anchor point connection relationship and the initial values of the physical parameters in the 3D Gaussian representation model of the spring mass are obtained; the physical parameters include: the distance between anchor points, the elastic stiffness parameter of the spring and the damping coefficient.
[0008] Furthermore, the calculation process of the 3D Gaussian representation model of the spring mass is as follows: (S2.3.1) Introduce volume sampling to generate a set of N data points. A anchor point Let the i-th anchor point x be set. i Having mass m i and initial velocity v i The anchor point is connected by a spring. Each with its n k Connect the nearest neighbor anchor points, using Represents the i-th anchor point x i Its j-th neighboring anchor point x i,j The connection relationships between them, i=1,2,3,…,N A j=1,2,3,…,n k ; (S2.3.2) Measure the center of each 3D Gaussian and its n b The distance between the nearest neighbor anchor points, i.e., the distance between anchor points ; Calculate the resultant force on each anchor point at each time step t. The resultant force includes: spring force, damping force, and gravity. The spring force is calculated using the elastic stiffness parameter of the spring, and the damping force is calculated using the damping coefficient.
[0009] Furthermore, the number n of the nearest neighbor anchor points of the anchor point k Through soft vectors The soft vector is modulated by learnable parameters. Control; if the current neighboring anchor point number j is less than or equal to the empirical threshold n c ,but ;like ,but .
[0010] Furthermore, at time step t, the i-th anchor point experiences a spring force from its j-th neighboring anchor point, the magnitude of which is determined by the spring's elastic stiffness parameter and x. i With x i,j distance relative to The degree of deviation is determined and, after nonlinear scaling by hyperparameters, is passed through a soft vector. Modulation is performed; the direction of the spring force points towards the i-th anchor point; The i-th anchor point is subjected to a damping force from its j-th neighboring anchor point. The magnitude of the damping force is determined by the damping coefficient and the relative velocity component of the two anchor points along the line connecting them, and the direction is opposite to the direction of the relative velocity.
[0011] Furthermore, in S3, the physical state of the 3D Gaussian at each moment is calculated using differentiable physical simulation, and the update rule is as follows: (S3.1.1) The position and velocity of the anchor point are updated by forward propagation using semi-implicit Euler integrals; (S3.1.2) Apply boundary conditions to the anchor points to simulate the interaction process with the environment; (S3.1.3) The position of the 3D Gaussian is updated by inverse distance weighted interpolation to accurately reflect the dynamic changes.
[0012] Furthermore, in S3, the 3D Gaussian image at the corresponding position in the input camera sequence is rendered as a 2D video using differentiable physical rendering, specifically achieved through the following operations: (S3.2.1) The projection of the three-dimensional Gaussian onto the image plane is represented as a two-dimensional Gaussian; (S3.2.2) The two-dimensional Gaussian image is effectively rendered into an RGB image and a depth map by alpha synthesis.
[0013] Furthermore, in S4, the loss function is... Norm and structural similarity index The weighted combination.
[0014] Furthermore, in S5, the calculated information includes: the future motion state of the object, the new force applied to the object, the new 3D Gaussian position and physical parameters after changing physical parameters or the material of the object, etc., as specified by the user.
[0015] The beneficial effects of this invention are: This invention introduces data induction priors based on deep learning neural networks and empirical induction priors based on spring-mass physics and physical optics, enabling simultaneous reconstruction and physical parameter identification of input monocular videos. It accurately models the physical properties of dynamic objects through physical constraints, improving reconstruction quality and enabling extrapolation simulation of user-specified external forces. It also supports real-time rendering under new perspectives and physical states. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the monocular video reconstruction and extrapolation simulation method based on differentiable physics in an embodiment of the present invention.
[0017] Figure 2 This is a schematic diagram of the reconstruction output of monocular video in an embodiment of the present invention.
[0018] Figure 3 This is a schematic diagram of the physical simulation output under different physical parameters in an embodiment of the present invention.
[0019] Figure 4 This is a schematic diagram of the physical simulation output when the material of the object is changed in an embodiment of the present invention.
[0020] Figure 5 This is a schematic diagram of the physical simulation output and extrapolation simulation results of complex materials in an embodiment of the present invention.
[0021] Figure 6This is a comparison chart showing the results of reconstruction and prediction of the first monocular video using the method of this invention and existing methods.
[0022] Figure 7 This is a comparison chart showing the results of reconstruction and prediction of the second monocular video using the method of this invention and existing methods. Detailed Implementation
[0023] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0024] like Figure 1 As shown, a monocular video reconstruction and extrapolation simulation method based on differentiable physics includes the following steps: S1. Preprocessing: Obtain a 2D video sequence, input the 2D video sequence into the neural network, and estimate the corresponding depth of the video and the intrinsic and extrinsic parameters of the camera.
[0025] The 2D video sequence is a video shot by the user.
[0026] Specifically, a depth estimation neural network (in this embodiment, the MoGe depth estimation neural network) is used to predict the depth D corresponding to the video, and a camera pose prediction neural network (in this embodiment, the MegaSAM camera pose prediction neural network) is used to predict the camera pose. And the camera's internal parameters K.
[0027] S2: Initialization, obtaining initial values for 3D Gaussian reconstruction and physical parameters. This is achieved through the following sub-steps: (S2.1) Combine the obtained depth and camera intrinsic and extrinsic parameters, and calculate the corresponding initial 3D point cloud position using the perspective camera inverse projection model. The expression is as follows: In the formula, X is the set of point cloud locations, U represents the vertical coordinate of a pixel, and V is the horizontal coordinate of a pixel.
[0028] (S2.2) Use the position of the initial 3D point cloud as the initial mean of the 3D Gaussian, and use the pixel color corresponding to the initial 3D point cloud as the initial color of the 3D Gaussian; initialize the covariance of the 3D Gaussian according to the nearest neighbor relationship of the initial 3D point cloud.
[0029] 3D Gaussian reconstruction is achieved by using the initial mean, initial color, and initial covariance of the 3D Gaussian. The reconstructed 3D Gaussian is represented as follows: In the formula, This is the 3D Gaussian mean, used to represent the position of the 3D Gaussian in the world coordinate system, initialized to... x represents the position of a single point cloud; Indicates the direction of covariance, initialized to the identity matrix. The magnitude of the covariance is represented by the distance between the nearest neighbors. The direction and magnitude of the covariance are used together to represent the geometry of the 3D Gaussian. This represents the opacity, initialized to 0.1. This represents the 3D Gaussian color, initialized to the pixel color. Opacity and color are used together to represent the appearance of the 3D Gaussian. These five parameters are all persistent properties shared across time.
[0030] (S2.3) Using the initial values of the 3D Gaussian model and the nearest neighbor relationship of the point cloud, the initial values of the anchor point connection relationship and physical parameters (including the distance between anchor points, the elastic stiffness parameter of the spring and the damping coefficient) in the 3D Gaussian representation model of the spring mass are obtained.
[0031] The 3D Gaussian representation model of a spring-mass point represents the dynamics of elastic objects using a learnable 3D Gaussian representation model of a spring-mass point. This model does not assume any material type and can efficiently simulate realistic elastic objects. The specific calculation process of the 3D Gaussian representation model of a spring-mass point is as follows: (S2.3.1) Introduce volume sampling to generate a set of anchor points: In the formula, N A It is the total number of anchor points. It is a sampling function. In this embodiment, 2048 anchor points are generated.
[0032] Set the i-th anchor point x i Having mass m i and initial velocity v i Through spring L and its nearest neighbor anchor points The connection is represented by spring L, and its expression is as follows: In the formula, This represents the connection relationship between the i-th anchor point and its j-th neighboring anchor point. This indicates the total number of springs connected to each anchor point. Let k represent the k-nearest neighbor function. In this embodiment, we take... The initial mass of each anchor point is unified as .
[0033] It is a hyperparameter whose value will significantly affect the results of physical simulations: a larger value indicates a higher value. It makes objects appear more rigid, while smaller ones... This results in a significantly softer point cloud behavior. To adapt... To mitigate the impact on the physics simulation results, a mitigation strategy was introduced, namely, by introducing soft vectors. To regulate; the vector is subject to a learnable parameter. Control (shared across all anchor points) is used to adjust the number of connecting springs, thereby enabling adaptive responses to different objects. An empirical threshold is given. The components of the soft vector η are calculated as follows: If ,but ;like ,but .
[0034] (S2.3.2) To update the position of the 3D Gaussians later, at the start of the dynamic simulation, the center of each 3D Gaussian (i.e., the point cloud) is measured relative to its position. The distance between the nearest neighbor anchor points is expressed as follows: In the formula, N is the total number of point clouds.
[0035] For each time step t, each anchor point The resultant force Calculate as follows: In the formula, For spring force, The damping force is g, and g represents the acceleration due to gravity.
[0036] For each spring Its spring force and damping force The calculation is as follows: In the formula, It is the elastic stiffness parameter between the i-th anchor point and its j-th neighboring anchor point. Let be the elastic stiffness parameter of the i-th anchor point; It is the damping coefficient between the i-th anchor point and its j-th neighboring anchor point. This is the initial damping coefficient; It is a hyperparameter used to control the nonlinearity of the spring force. When the expression for spring force is given, it degenerates into Hooke's Law; when At this time, the spring force becomes a non-linear function of the spring length. In this embodiment, it is set that... Initial damping coefficient .
[0037] Further obtain each anchor point The net force acting on the body is expressed as follows: S3. Forward Propagation: Predicting 3D Gaussian Reconstruction and Physical Parameters: Utilizing differentiable physical simulation, the physical state of the 3D Gaussian at each time step is calculated. Differentiable physical rendering is then used to obtain the corresponding position of the 3D Gaussian in the input camera sequence, resulting in a rendered 2D video. This is achieved through the following sub-steps: (S3.1) Using differentiable physics simulation, calculate the physical state of the 3D Gaussian at each time step. There is a time interval between every two keyframes. Each time step, and the anchor point of each time step. The position from the previous moment The position is calculated according to the update rules. In this embodiment, it is only calculated at the keyframe time (i.e., Based on visual observations of the current frame, physical parameters are optimized, and as the optimization converges, the parameters are gradually increased. The values are adjusted to strike a balance between computational cost and recognition accuracy. The update rules are as follows: (S3.1.1) The position and velocity of the anchor point are updated by forward propagation using semi-implicit Euler integrals: (S3.1.2) Apply boundary conditions to the anchor points To simulate the interaction process with the environment: (S3.1.3) Positioning of the 3D Gaussian using inverse distance weighted interpolation Update to accurately reflect dynamic changes: In the formula, This is a positive real number used to control the degree of attenuation of the distance's influence on interpolation. In this embodiment, it is set... .
[0038] (S3.2) In order to render an image at a specific moment and from a specific camera angle and frame (hereinafter referred to as the input image) Using the rendering equation combined with the updated 3D Gaussian position, the updated 3D Gaussian is rendered to obtain a rendered 2D video (hereinafter referred to as the rendered image) of the 3D Gaussian at the corresponding position in the input camera sequence. Specifically, this is achieved through the following operations: (S3.2.1) To render a set of 3D Gaussians from a camera with pose E and intrinsic parameters K, the projection of the 3D Gaussians onto the image plane is represented as a 2D Gaussian, with the following parameters: , : In the formula, Π represents perspective projection. Σ is the Jacobian matrix of the perspective projection at point μ with respect to the camera pose E and the camera intrinsic parameter K, where Σ represents the 3D covariance. This represents the covariance of 2D.
[0039] (S3.2.2) The two-dimensional Gaussian image is effectively rendered into an RGB image and a depth map through alpha synthesis, as shown in the following expression: In the formula, This represents the opacity weight of the a-th Gaussian at pixel p. This represents the opacity at the a-th Gaussian point; It represents the visibility ratio of the current Gaussian over the cumulative transparency of all preceding Gaussians. This represents the opacity weight of the b-th Gaussian at pixel p; Let represent the depth of the a-th Gaussian, and let H(p) represent the set of all Gaussians that intersect with the ray emanating from pixel p. The entire rendering process is fully differentiable, therefore the parameters of the 3D Gaussian can be directly optimized.
[0040] This invention sets the mass of each anchor point to a constant. and using a single parameter All damping factors are controlled; these two parameters are fixed constants and do not participate in the optimization. Furthermore, for each anchor point... Introducing a unified elastic stiffness parameter This is used to control the spring constant (and thus the spring force) of the connected spring, thereby simplifying the model without sacrificing functional integrity. Spring stiffness and damping coefficient Defined as , This setting simplifies the optimization process: only optimization is needed. A stiffness coefficient, rather than an optimization order of magnitude. All spring parameters were obtained. Through the above simplification, the model was simplified, the number of learnable parameters was reduced, and computational efficiency and optimizability were improved without affecting the physical and dynamic expression capability of the 3D Gaussian representation model of the spring mass.
[0041] After simplification, the learnable parameters in the 3D Gaussian model and the 3D Gaussian representation of the spring-mass model include: ① Geometric parameters: mean μ, used to represent the position of the 3D Gaussian in the world coordinate system; covariance direction θ and covariance magnitude s, used to represent the geometry of the 3D Gaussian; ② Appearance parameters: opacity α and color c, used to represent the appearance of the 3D Gaussian; ③ Physical parameters: the velocity vector at the initial moment of the anchor point. This is used to provide the initial motion pattern for the physical simulation; the elastic stiffness parameter of each anchor point. This enables the control of local elasticity; soft vector parameters This enables precise control of spring dynamics; learnable parameters in boundary conditions (such as the coefficient of friction) This affects the physical realism of the simulation. Specifically, a small number of frames captured before the object interacts with the environment are used to independently optimize the initial velocity vector. .
[0042] S4. Backpropagation to optimize 3D Gaussian reconstruction and physical parameters: Construct a loss function by comparing the differences between the rendered 2D video and the input 2D video, obtain the gradient using the backpropagation algorithm, update the physical parameters of the 3D Gaussian using the stochastic gradient descent algorithm, and repeat S3-S4 until the loss function converges to obtain an accurate 3D Gaussian reconstruction result.
[0043] The loss function is defined as Norm and structural similarity index The weighted combination of the loss function applied to the input image With rendering images Between them, its expression is as follows: In the formula, These are weighting factors used for balancing. Item and The contribution of each item to the overall loss. In this embodiment, we take... .
[0044] Since the physical optics rendering process and the physical mechanics simulation process are completely differentiable, they can be determined through the loss function. Jointly optimize all 3D Gaussian and physical parameters.
[0045] S5, Extrapolation Simulation and Rendering: Further physical simulation is performed on the reconstruction results to calculate the physical motion of 3D Gaussian under the external force conditions specified by the user, and render the 2D rendered video under the external force conditions and the specified viewpoint.
[0046] Given the 3D Gaussian and physical parameter reconstruction results for the input video, the physical model (i.e., the spring-mass 3D Gaussian representation model) can predict events that have not yet occurred by further updating the physical parameters. The predicted information includes: the future motion state of the object, new forces applied to the object, the new 3D Gaussian position and physical parameters after changing the physical parameters or the material of the object, etc., as specified by the user, and the 2D rendered video from the specified viewpoint.
[0047] Figure 1 This paper presents a schematic diagram of the overall process of a monocular video reconstruction and extrapolation simulation method based on differentiable physics proposed in this invention. As shown in the diagram, the process begins with an input monocular video, followed by video preprocessing to obtain the camera pose. Geometric and physical initialization is performed using 3D Gaussian and spring-mass models. Physical state deduction is then conducted through forward propagation, and backpropagation is performed using the error generated by comparing differentiable rendering with the input video, thereby jointly optimizing the geometric appearance and physical parameters. This demonstrates that this invention can combine physical and mechanical priors with data inductive priors to achieve accurate reconstruction and parameter identification of dynamic objects.
[0048] Figure 2 A schematic diagram illustrating the reconstruction output of the method of this invention under monocular video input is shown. As can be seen from the diagram, the system incorporating the method of this invention can reconstruct a 3D object (such as a toy pig) with a continuous dynamic trajectory from a user-captured video sequence, combined with the estimated camera viewpoint. This demonstrates that the present invention possesses high-quality geometric reconstruction capabilities and viewpoint consistency when processing monocular video of real-world scenes.
[0049] Figure 3 The diagram illustrates the physical simulation output under different physical parameters. As can be seen from the diagram, adjusting the initial velocity based on the original video reconstruction results... Parameters such as object stiffness (softer or harder) or environmental gravity (weightlessness or hypergravity) cause objects to exhibit drastically different motion trajectories and deformation states. This demonstrates that the physical parameters resolved by this invention are editable and can support physical extrapolation simulations for events that have not yet occurred.
[0050] Figure 4 The diagram illustrates the physical simulation output when the object's material is changed. As can be seen from the diagram, the original hot dog model was simulated as a fluid (liquefaction effect) under user commands (from left to right, showing different moments of fluid state). This demonstrates that the spring-mass 3D Gaussian representation model of this invention does not assume a fixed material type, possesses strong versatility, and can efficiently simulate the behavior of various physical materials, from elastic bodies to fluids.
[0051] Figure 5The diagrams show the physical simulation outputs for complex materials such as bread, cakes, aluminum cans, and sand piles composed of discrete particles. As can be seen from the diagrams, objects with different structures exhibit realistic deformations consistent with their physical properties when subjected to force, such as the soft cracking of a cake, the metallic wrinkling of an aluminum can, and the flowing accumulation of sand. This demonstrates that the proposed method can adapt to and accurately extrapolate the simulation of diverse objects with complex geometries and physical properties in real life.
[0052] Figure 6 and Figure 7 A comparative diagram of the method of this invention with existing technologies (such as PAC-NeRF) and ground truth values is shown. As can be seen from the diagram, in the prediction (extrapolation) stage after reconstruction, the images generated by the existing technology PAC-NeRF exhibit significant artifacts, blurring, and structural collapse, while the prediction results of this invention remain highly consistent with the ground truth over long time series. This demonstrates that the core forward propagation process of this invention significantly improves the prediction accuracy and realism of dynamic objects over long time spans.
[0053] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.
Claims
1. A method for monocular video reconstruction and extrapolation simulation based on differentiable physics, characterized in that, Includes the following steps: S1. Preprocessing: Acquire 2D video sequences and use neural networks to estimate their corresponding depth and camera intrinsic and extrinsic parameters; S2. Initialization: 3D Gaussian reconstruction is performed based on depth and camera intrinsic and extrinsic parameters. The anchor point connection relationship and initial values of physical parameters of the spring mass 3D Gaussian representation model are obtained through the initial values of the 3D Gaussian model and the nearest neighbor relationship of the point cloud. S3, Forward Propagation, Predicting 3D Gaussian Reconstruction and Physical Parameters: Utilizing differentiable physical simulation to calculate the physical state of the 3D Gaussian at each moment, and obtaining the rendered 2D video of the 3D Gaussian at the corresponding position in the input camera sequence through differentiable physical rendering. S4. Backpropagation to optimize 3D Gaussian reconstruction and physical parameters: Construct a loss function by comparing the difference between the rendered 2D video and the input 2D video, obtain the gradient using the backpropagation algorithm, update the physical parameters of the 3D Gaussian using the stochastic gradient descent algorithm, repeat S3-S4 until the loss function converges and obtains an accurate 3D Gaussian reconstruction result. S5, Extrapolation Simulation and Rendering: Performs physical simulation on the 3D Gaussian reconstruction results, calculates the physical motion of the 3D Gaussian under the external force conditions specified by the user, and renders the corresponding 2D rendered video under the specified viewpoint.
2. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, In S1, a depth estimation neural network is used to predict the depth D corresponding to the video, and a camera pose prediction neural network is used to predict the camera pose E and the camera intrinsic parameter K.
3. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, S2 is specifically implemented through the following sub-steps: (S2.1) Based on the depth and the combination of camera internal and external parameters, the corresponding initial 3D point cloud position is calculated by using the perspective camera inverse projection model; (S2.2) Use the position of the initial 3D point cloud as the initial mean of the 3D Gaussian; use the pixel color corresponding to the initial 3D point cloud as the initial color of the 3D Gaussian; initialize the covariance of the 3D Gaussian according to the nearest neighbor relationship of the initial 3D point cloud. Achieving 3D Gaussian reconstruction using the initial mean, initial color, and initial covariance of a 3D Gaussian; the reconstructed 3D Gaussian includes the following parameters: 3D Gaussian mean, covariance direction, covariance magnitude, opacity, and 3D Gaussian color; (S2.3) Using the initial values of the 3D Gaussian model and the nearest neighbor relationship of the point cloud, the anchor point connection relationship and the initial values of the physical parameters in the 3D Gaussian representation model of the spring mass are obtained; the physical parameters include: the distance between anchor points, the elastic stiffness parameter of the spring and the damping coefficient.
4. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 3, characterized in that, The calculation process of the 3D Gaussian representation model of the spring mass is as follows: (S2.3.1) Introduce volume sampling to generate a set of N data points. A anchor point Let the i-th anchor point x be set. i Having mass m i and initial velocity v i The anchor point is connected by a spring. Each with its n k Connect the nearest neighbor anchor points, using Represents the i-th anchor point x i Its j-th neighboring anchor point x i,j The connection relationships between them, i=1,2,3,…,N A j=1,2,3,…,n k ; (S2.3.2) Measure the center of each 3D Gaussian and its n b The distance between the nearest neighbor anchor points, i.e., the distance between anchor points ; Calculate the resultant force on each anchor point at each time step t. The resultant force includes: spring force, damping force, and gravity. The spring force is calculated using the spring's elastic stiffness parameter, and the damping force is calculated using the damping coefficient.
5. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 4, characterized in that, The number of nearest neighbor anchor points n of the anchor point k Through soft vectors The soft vector is modulated by learnable parameters. Control; if the current neighboring anchor point number j is less than or equal to the empirical threshold n c ,but ;like ,but .
6. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 5, characterized in that, At time step t, the i-th anchor point experiences a spring force from its j-th neighboring anchor point, the magnitude of which is determined by the spring's elastic stiffness parameter and x. i With x i,j distance relative to The degree of deviation is determined and, after nonlinear scaling by hyperparameters, is passed through a soft vector. Modulation is performed; the direction of the spring force points towards the i-th anchor point; The i-th anchor point is subjected to a damping force from its j-th neighboring anchor point. The magnitude of the damping force is determined by the damping coefficient and the relative velocity component of the two anchor points along the line connecting them, and the direction is opposite to the direction of the relative velocity.
7. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, In step S3, the physical state of the 3D Gaussian at each moment is calculated using a differentiable physical simulation. The update rule is as follows: (S3.1.1) The position and velocity of the anchor point are updated by forward propagation using semi-implicit Euler integrals; (S3.1.2) Apply boundary conditions to the anchor points to simulate the interaction process with the environment; (S3.1.3) The position of the 3D Gaussian is updated by inverse distance weighted interpolation to accurately reflect the dynamic changes.
8. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, In step S3, the 3D Gaussian image at the corresponding position in the input camera sequence is rendered as a 2D video using differentiable physical rendering. This is achieved through the following operations: (S3.2.1) The projection of the three-dimensional Gaussian onto the image plane is represented as a two-dimensional Gaussian; (S3.2.2) The two-dimensional Gaussian image is effectively rendered into an RGB image and a depth map by alpha synthesis.
9. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, In S4, the loss function is: Norm and structural similarity index The weighted combination.
10. The monocular video reconstruction and extrapolation simulation method based on differentiable physics according to claim 1, characterized in that, In S5, the calculated information includes: the future motion state of the object, the new force applied to the object, the new 3D Gaussian position and physical parameters after changing physical parameters or the material of the object, etc., as specified by the user.