A deformable 3D Gaussian spatio-temporal motion prediction method

Abstract

The application provides a deformable three-dimensional Gaussian space-time motion prediction method, which can be applied to the technical field of computer vision and computer graphics. The method comprises the following steps: fusing a background sparse point cloud corresponding to a static background region in an image sequence with a dynamic region point cloud to obtain an initial three-dimensional Gaussian particle set; performing future state prediction on a key point graph structure with key three-dimensional Gaussian particles as nodes based on a graph attention mechanism to obtain future space-time state information of the key three-dimensional Gaussian particles; driving other three-dimensional Gaussian particles in the initial three-dimensional Gaussian particle set by using the future space-time state information of the key three-dimensional Gaussian particles to obtain future space-time motion information of the initial three-dimensional Gaussian particle set; and performing rendering reconstruction on the image sequence by using the future space-time motion information of the initial three-dimensional Gaussian particle set to obtain a dynamic image sequence corresponding to a preset camera pose at a target future time.

Description

Technical Field

[0001] This application relates to the fields of computer vision and computer graphics, specifically to a deformable 3D Gaussian spatiotemporal motion prediction method and a training method for a deformable 3D Gaussian spatiotemporal motion prediction model. Background Technology

[0002] With the development of 3D perception, spatial intelligence, and novel view synthesis technologies, dynamic scene reconstruction and future motion prediction based on monocular video or multi-view image sequences have become important research directions in computer vision and computer graphics. In recent years, the 3D Gaussian splashing method, by employing explicit 3D Gaussian particles to represent the scene, has shown significant advantages in rendering efficiency and differentiability optimization capabilities, and has gradually been extended to dynamic scene modeling tasks.

[0003] Existing physically deformable Gaussian splashing schemes are limited to scene reconstruction tasks and lack explicit future state prediction capabilities. Furthermore, they cannot effectively model long-term temporal dependencies and local motion coupling relationships in dynamic scenes, lack efficient prediction mechanisms, and struggle to extrapolate future states of dynamic scenes. Existing Gaussian prediction schemes rely entirely on data-driven modeling without physical prior constraints, leading to problems such as trajectory drift, result instability, and mode collapse in out-of-domain predictions, resulting in weak generalization ability. Moreover, existing Gaussian prediction schemes predict all Gaussian particles as a whole, failing to balance prediction efficiency, detailed representation of local motion, and accuracy in modeling complex spatiotemporal dependencies.

[0004] In addition, existing technical solutions also have problems such as insufficient dynamic initialization capabilities, shortcomings in framework design, weak spatiotemporal modeling capabilities, and poor prediction stability. Summary of the Invention

[0005] In view of the above problems, this application provides a deformable three-dimensional Gaussian spatiotemporal motion prediction method and a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model, which is used to solve at least one of the existing problems.

[0006] The first aspect of this application provides a deformable 3D Gaussian spatiotemporal motion prediction method, comprising: completing the dynamic region in an image sequence with 3D information based on inter-frame relationships to obtain a dynamic region point cloud; fusing the background sparse point cloud corresponding to the static background region in the image sequence with the dynamic region point cloud to obtain an initial 3D Gaussian particle set, wherein the image sequence is a multi-frame image sequence generated under a preset camera pose; performing future state prediction on a key point graph structure with key 3D Gaussian particles as nodes based on a graph attention mechanism to obtain the future spatiotemporal state information of the key 3D Gaussian particles, wherein the key 3D Gaussian particles are selected from the initial 3D Gaussian particle set based on preset screening conditions; using the future spatiotemporal state information of the key 3D Gaussian particles to drive other 3D Gaussian particles in the initial 3D Gaussian particle set to obtain the future spatiotemporal motion information of the initial 3D Gaussian particle set; and using the future spatiotemporal motion information of the initial 3D Gaussian particle set to render and reconstruct the image sequence to obtain a dynamic image sequence corresponding to the preset camera pose at the target's future time.

[0007] According to embodiments of this application, the above-mentioned fusion of the background sparse point cloud corresponding to the static background region in the image sequence with the dynamic region point cloud to obtain an initial three-dimensional Gaussian particle set includes: performing inter-frame modeling on the dynamic region to obtain inter-frame modeling results; using the inter-frame modeling results to perform predictive three-dimensional information completion on the three-dimensional point positions and / or local geometric structures of the dynamic region to obtain an initial point cloud of the dynamic region; filtering, denoising, and / or confidence filtering on the initial point cloud of the dynamic region to obtain a dynamic region point cloud; performing multi-view-based geometric structure restoration on the static background region to obtain a background sparse point cloud; fusing the background sparse point cloud and the dynamic region point cloud to obtain an initial three-dimensional point cloud representation; and initializing the parameters of multiple three-dimensional Gaussian particles using the initial three-dimensional point cloud representation to obtain an initial three-dimensional Gaussian particle set, wherein the parameters of the three-dimensional Gaussian particles include center position parameters, rotation parameters, scale parameters, opacity parameters, color parameters, appearance feature parameters, and motion feature parameters.

[0008] According to embodiments of this application, the above-described prediction of the future state of key three-dimensional Gaussian particles based on graph attention mechanism using a key point graph structure with key three-dimensional Gaussian particles as nodes to obtain the future spatiotemporal state information of key three-dimensional Gaussian particles includes: selecting key three-dimensional Gaussian particles that meet preset selection conditions from an initial set of three-dimensional Gaussian particles; constructing a key point graph structure with key three-dimensional Gaussian particles as nodes and the association relationship between key three-dimensional Gaussian particles and other three-dimensional Gaussian particles as edges, wherein the association relationship includes spatial proximity, similarity of motion features and / or correlation of topological structure; and performing future state prediction of the key point graph structure based on graph attention mechanism using a sliding time window to obtain the future spatiotemporal state information representing the motion state of key three-dimensional Gaussian particles at multiple future moments.

[0009] According to an embodiment of this application, the above-mentioned method of using the future spatiotemporal state information of key three-dimensional Gaussian particles to drive other three-dimensional Gaussian particles in an initial set of three-dimensional Gaussian particles to obtain the future spatiotemporal motion information of the initial set of three-dimensional Gaussian particles includes: constructing multiple mapping relationships between multiple associated key three-dimensional Gaussian particles that are related to other three-dimensional Gaussian particles and other three-dimensional Gaussian particles; performing corresponding weighted operations on the future spatiotemporal state information of multiple associated key three-dimensional Gaussian particles and the control weights of the multiple mapping relationships to obtain the future spatiotemporal state information of other three-dimensional Gaussian particles; and performing multiple mapping relationship construction operations and corresponding weighted operation operations on each other three-dimensional Gaussian particle in the initial set of three-dimensional Gaussian particles to obtain the future spatiotemporal motion information of the initial set of three-dimensional Gaussian particles.

[0010] The second aspect of this application provides a training method for a deformable 3D Gaussian spatiotemporal motion prediction model. The deformable 3D Gaussian spatiotemporal motion prediction model includes a deformation network, a physical information network, a graph structure prediction network, and a differentiable rendering module. The training method includes: using the deformation network to perform deformation prediction on a standard 3D Gaussian sample particle set generated from image sequence samples, obtaining the motion state information of the standard 3D Gaussian particle sample set at multiple prediction times; and using the physical information network to apply physical constraints to the deformation prediction operation of the standard 3D Gaussian particle sample set, obtaining a physical loss value; based on the motion state information of the standard 3D Gaussian particle sample set at multiple prediction target times, using the graph structure prediction network to perform a future state prediction operation based on a graph attention mechanism on the graph structure of key point samples, obtaining the future spatiotemporal state information of the key 3D Gaussian sample particles, and obtaining the prediction loss value generated by performing the future state prediction operation. The key 3D Gaussian sample particles... The key point sample graph structure is constructed by selecting key 3D Gaussian sample particles as nodes through screening a standard 3D Gaussian particle sample set. Based on the future spatiotemporal motion information of the standard 3D Gaussian particle sample set, the image reconstruction operation is performed on the image sequence samples using a differentiable rendering module to obtain the dynamic reconstructed image of the image sequence samples at future time points, and the image reconstruction loss value generated by the image reconstruction operation is obtained. The future spatiotemporal motion information of the standard 3D Gaussian particle sample set is obtained by driving the future spatiotemporal state information of the key 3D Gaussian sample particles to obtain other 3D Gaussian sample particles in the standard 3D Gaussian particle sample set. The parameters of the deformable 3D Gaussian spatiotemporal motion prediction model are optimized using the physical loss value, prediction loss value, and image reconstruction loss value to obtain the trained deformable 3D Gaussian spatiotemporal motion prediction model. The trained deformable 3D Gaussian spatiotemporal motion prediction model is used to execute the deformable 3D Gaussian spatiotemporal motion prediction method.

[0011] According to an embodiment of this application, the above-mentioned standard three-dimensional Gaussian sample particle set is obtained through the following operations: using a three-dimensional Gaussian particle modeling network of a deformable three-dimensional Gaussian spatiotemporal motion prediction model, the background sparse point cloud representation and dynamic region point cloud representation of the image sequence sample are fused to obtain an initial three-dimensional Gaussian particle sample set; the initial three-dimensional Gaussian particle sample set is mapped to a normalized space to obtain a standard three-dimensional Gaussian sample particle set.

[0012] According to embodiments of this application, the above-mentioned physical information constraint on the deformation prediction operation of a standard three-dimensional Gaussian sample particle set using a physical information network to obtain a physical loss value includes: spatiotemporally encoding the motion state information of the standard three-dimensional Gaussian sample particles at the current prediction time to obtain the spatiotemporal characteristics of the standard three-dimensional Gaussian sample particle set; processing the spatiotemporal characteristics of the standard three-dimensional Gaussian sample particle set using a physical information network to obtain the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set; constructing physical residuals using the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set according to preset physical conservation constraints, wherein the preset physical conservation constraints include momentum conservation constraints, mass conservation constraints, or energy conservation constraints based on continuous medium dynamics; and applying physical information constraints to the deformation prediction operation of the standard three-dimensional Gaussian sample particle set using the physical residuals to obtain a physical loss value.

[0013] According to embodiments of this application, the aforementioned key 3D Gaussian sample particles are obtained by screening a standard 3D Gaussian particle sample set, including: screening the standard 3D Gaussian particle sample set using the spatial distribution and motion characteristics of the 3D Gaussian particles; or screening the standard 3D Gaussian particle sample set using local gradient response information characterizing the geometric texture of an image sequence; or clustering the standard 3D Gaussian particle sample set and screening the standard 3D Gaussian particle sample set based on the clustering results; or screening the standard 3D Gaussian particle sample set using a farthest point sampling algorithm.

[0014] According to the embodiments of this application, the parameters of each three-dimensional Gaussian sample particle in the above-mentioned standard three-dimensional Gaussian sample particle set include center position parameters, rotation parameters, scale parameters, opacity parameters, color parameters, appearance feature parameters, and motion feature parameters.

[0015] According to embodiments of this application, the graph structure prediction network described above includes a graph attention network, a graph convolutional network, or a spatiotemporal graph neural network.

[0016] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0018] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] This application provides a deformable 3D Gaussian spatiotemporal motion prediction method that addresses the shortcomings of 3D structure loss and incomplete Gaussian particle distribution in dynamic regions under sparse scenes and large-scale motion. It completes the 3D information of dynamic regions through inter-frame relationship modeling and integrates sparse background point clouds with dynamic region point clouds to construct a complete initial 3D Gaussian particle set, effectively improving the initial 3D representation of dynamic scenes and enhancing the completeness and accuracy of subsequent deformation modeling and temporal prediction from the source. Compared to physically deformable Gaussian methods that only focus on reconstruction and lack future extrapolation capabilities, this application constructs a key point graph structure and relies on graph attention mechanisms to model the long-term temporal dependencies and local particle coupling relationships of dynamic scenes, achieving explicit and accurate prediction of the future state of Gaussian particles. It possesses complete spatiotemporal extrapolation capabilities for dynamic scenes, thus overcoming the limitations of traditional methods. This application addresses the shortcomings of traditional methods, such as the lack of prediction mechanisms and weak spatiotemporal correlation modeling. The method proposed in this application combines deformable Gaussian modeling with a temporal prediction mechanism, balancing the rationality of scene deformation with the accuracy of temporal extrapolation. It avoids the shortcomings of pure data-driven prediction lacking physical prior constraints, out-of-domain prediction prone to trajectory drift, mode collapse, and poor generalization, significantly improving the physical consistency and temporal stability of long-term predictions. This application abandons the inefficient method of directly predicting all Gaussian particles, instead selecting key 3D Gaussian particles as graph nodes for state prediction, and then using these key particles to drive the global particle motion state. This accurately captures the core dynamic changes of the scene, preserves the fine expression of local motion, and significantly reduces the computational load, effectively balancing the ability to model complex spatiotemporal dependencies, the ability to express local motion, and prediction efficiency. Furthermore, the method proposed in this application integrates the entire process of 3D structure completion, particle deformation modeling, spatiotemporal state prediction, and image rendering and reconstruction, forming a unified technical framework that balances physical consistency, structural integrity, and temporal predictability, solving the core shortcoming of existing technologies that cannot simultaneously guarantee the rationality of reconstruction and the accuracy of future predictions. Attached Figure Description

[0020] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 The diagram illustrates an application scenario of the deformable 3D Gaussian spatiotemporal motion prediction method and the training method of the deformable 3D Gaussian spatiotemporal motion prediction model according to embodiments of this application.

[0022] Figure 2 A flowchart of a deformable three-dimensional Gaussian spatiotemporal motion prediction method according to an embodiment of this application is shown.

[0023] Figure 3 A flowchart is shown of a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model according to an embodiment of this application.

[0024] Figure 4 The diagram illustrates the training and inference process of a deformable Gaussian motion prediction model that incorporates physical information according to an embodiment of this application.

[0025] Figure 5 The diagram shows the effect of inter-frame reconstruction and global reconstruction based on feedforward Transformer for real-world datasets and physics simulation datasets according to embodiments of this application.

[0026] Figure 6 The diagram illustrates a canonical space deformable Gaussian model and a physical information deformation field structure diagram according to an embodiment of this application.

[0027] Figure 7 A diagram illustrating the keypoint motion distillation and graph attention prediction process according to an embodiment of this application is shown.

[0028] Figure 8 A block diagram of an electronic device suitable for implementing a deformable three-dimensional Gaussian spatiotemporal motion prediction method and a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model, according to an embodiment of this application, is shown. Detailed Implementation

[0029] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] Existing 3D Gaussian splashing methods include physically deformable Gaussian splashing methods and Gaussian prediction methods.

[0034] Among them, the physically information-deformable Gaussian splash method models the dynamic deformation of Gaussian particles in gauge space and introduces physical information constraints to improve physical consistency and temporal stability during dynamic reconstruction. Compared with purely data-driven deformation methods, this type of method can alleviate the problems of non-physical motion and local structural distortion to some extent. However, this type of method focuses more on dynamic scene reconstruction and has relatively insufficient explicit prediction ability for future states. Furthermore, its modeling of long-term dependencies, local coupling relationships, and efficient prediction mechanisms in dynamic scenes is still inadequate.

[0035] Gaussian prediction methods typically extrapolate and predict future scene states based on deformable 3D Gaussian representations, thereby generating dynamic future views of the scene. These methods represent a significant advancement over traditional dynamic reconstruction methods in future prediction tasks, but they generally rely heavily on data-driven temporal modeling and lack explicit physical constraints. Consequently, they are prone to issues such as trajectory drift, prediction instability, and pattern collapse outside the training timeframe. Furthermore, these methods often directly predict a large number of Gaussian particles, making it difficult to balance prediction efficiency, local motion representation capabilities, and the ability to model complex spatiotemporal dependencies.

[0036] Furthermore, when the input to a dynamic scene is sparse or involves significant motion, existing methods often struggle to fully recover the 3D structure of the dynamic region during the initialization phase. This results in an incomplete distribution of dynamic Gaussian particles in the subsequent normalized space, further impacting dynamic modeling and future prediction performance. Therefore, existing technologies still suffer from the following shortcomings: First, the initial representation of the dynamic region is insufficient; second, there is a lack of a unified framework that balances physical consistency and future prediction capabilities; third, there is insufficient modeling of key spatiotemporal dependencies in complex dynamic scenes; and fourth, the stability and generalization ability in future prediction tasks still need improvement.

[0037] To at least address one of the problems in the prior art, this application provides a deformable three-dimensional Gaussian spatiotemporal motion prediction method and a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model.

[0038] Figure 1 The diagram illustrates an application scenario of the deformable 3D Gaussian spatiotemporal motion prediction method and the training method of the deformable 3D Gaussian spatiotemporal motion prediction model according to embodiments of this application.

[0039] like Figure 1 As shown, application scenario 100 according to this embodiment may include application scenarios such as computer vision and computer graphics. Network 104 is used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0040] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0042] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0043] It should be noted that the deformable 3D Gaussian spatiotemporal motion prediction method and the training method of the deformable 3D Gaussian spatiotemporal motion prediction model provided in this application embodiment can generally be executed by server 105. The deformable 3D Gaussian spatiotemporal motion prediction method and the training method of the deformable 3D Gaussian spatiotemporal motion prediction model provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0045] The following will be based on Figure 1 The described scenario, with reference to accompanying drawings and multiple specific implementation paradigms, provides a detailed description of the indoor scene construction method of the disclosed embodiments.

[0046] Figure 2 A flowchart of a deformable three-dimensional Gaussian spatiotemporal motion prediction method according to an embodiment of this application is shown.

[0047] like Figure 2 As shown, the deformable three-dimensional Gaussian spatiotemporal motion prediction method of this embodiment includes operations S210 to S240.

[0048] In operation S210, the dynamic region in the image sequence is completed with three-dimensional information based on the inter-frame relationship to obtain the dynamic region point cloud. The background sparse point cloud corresponding to the static background region in the image sequence is fused with the dynamic region point cloud to obtain the initial three-dimensional Gaussian particle set.

[0049] The image sequence is a multi-frame image sequence generated under a preset camera pose. The image sequence can be a monocular video sequence, a multi-view image sequence, or a timestamped video stream. The preset camera pose can be used to obtain camera intrinsic and extrinsic parameters, providing relevant information for subsequent 3D structure restoration, rendering, and reconstruction of the image sequence.

[0050] Three-dimensional Gaussian Particle: refers to the basic unit that makes up the three-dimensional Gaussian splash model. Each particle is described by parameters such as center position, scale parameter, rotation parameter, opacity and appearance characteristics.

[0051] Three-dimensional information completion can be performed on dynamic regions in an image sequence based on inter-frame relationships. For example, a dynamic region structure restoration mechanism based on Visual Geometry Grounded Transformer (VGGT) feedforward reconstruction can be used to complete the three-dimensional information of dynamic regions.

[0052] In addition, Deformable 3D Gaussians refer to three-dimensional Gaussian particles whose spatial position, shape, or appearance parameters change over time, and are used to represent spatiotemporal changes in dynamic scenes.

[0053] In operation S220, the future state prediction based on graph attention mechanism is performed on the key point graph structure with key three-dimensional Gaussian particles as nodes to obtain the future spatiotemporal state information of key three-dimensional Gaussian particles.

[0054] Among them, the key three-dimensional Gaussian particles are selected from the initial set of three-dimensional Gaussian particles based on preset screening conditions.

[0055] Key 3D Gaussian particles (or key points for short) can be determined based on spatial distribution, motion saliency, local gradient response, or clustering results.

[0056] The keypoint graph structure uses key 3D Gaussian particles as graph nodes and the relationships (or mapping relationships) between key 3D Gaussian particles and other 3D Gaussian particles as edges of the keypoint graph structure.

[0057] Optionally, the initial 3D Gaussian particle set is mapped to a canonical space to obtain a standard 3D Gaussian particle set, from which key 3D Gaussian particles are selected. In the field of 3D Gaussian Splatting (3DGS) technology, the canonical space refers to a reference space to which dynamic scenes at different time frames are uniformly mapped. Gaussian particles are uniformly modeled in this space to describe their changes over time. 3DGS refers to an explicit 3D representation method that represents a 3D scene as multiple learnable 3D Gaussian distributions and renders them using a differentiable rasterization method.

[0058] In operation S230, the future spatiotemporal state information of the key three-dimensional Gaussian particle is used to drive other three-dimensional Gaussian particles in the initial three-dimensional Gaussian particle set, thereby obtaining the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set.

[0059] Using the future spatiotemporal state information of a key 3D Gaussian particle to drive other 3D Gaussian particles in the initial set of 3D Gaussian particles means: based on the existing mapping relationship (or binding relationship) between the key and other 3D Gaussian particles, synchronously assigning and driving the predicted motion / state information of the key 3D Gaussian particle to each corresponding other 3D Gaussian particle.

[0060] In operation S240, the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set is used to render and reconstruct the image sequence, thereby obtaining a dynamic image sequence corresponding to the preset camera pose at the target's future time.

[0061] This application provides a deformable 3D Gaussian spatiotemporal motion prediction method that addresses the shortcomings of 3D structure loss and incomplete Gaussian particle distribution in dynamic regions under sparse scenes and large-scale motion. It completes the 3D information of dynamic regions through inter-frame relationship modeling and integrates sparse background point clouds with dynamic region point clouds to construct a complete initial 3D Gaussian particle set, effectively improving the initial 3D representation of dynamic scenes and enhancing the completeness and accuracy of subsequent deformation modeling and temporal prediction from the source. Compared to physically deformable Gaussian methods that only focus on reconstruction and lack future extrapolation capabilities, this application constructs a key point graph structure and relies on graph attention mechanisms to model the long-term temporal dependencies and local particle coupling relationships of dynamic scenes, achieving explicit and accurate prediction of the future state of Gaussian particles. It possesses complete spatiotemporal extrapolation capabilities for dynamic scenes, thus overcoming the limitations of traditional methods. This application addresses the shortcomings of traditional methods, such as the lack of prediction mechanisms and weak spatiotemporal correlation modeling. The method proposed in this application combines deformable Gaussian modeling with a temporal prediction mechanism, balancing the rationality of scene deformation with the accuracy of temporal extrapolation. It avoids the shortcomings of pure data-driven prediction lacking physical prior constraints, out-of-domain prediction prone to trajectory drift, mode collapse, and poor generalization, significantly improving the physical consistency and temporal stability of long-term predictions. This application abandons the inefficient method of directly predicting all Gaussian particles, instead selecting key 3D Gaussian particles as graph nodes for state prediction, and then using these key particles to drive the global particle motion state. This accurately captures the core dynamic changes of the scene, preserves the fine expression of local motion, and significantly reduces the computational load, effectively balancing the ability to model complex spatiotemporal dependencies, the ability to express local motion, and prediction efficiency. Furthermore, the method proposed in this application integrates the entire process of 3D structure completion, particle deformation modeling, spatiotemporal state prediction, and image rendering and reconstruction, forming a unified technical framework that balances physical consistency, structural integrity, and temporal predictability, solving the core shortcoming of existing technologies that cannot simultaneously guarantee the rationality of reconstruction and the accuracy of future predictions.

[0062] According to embodiments of this application, the above-mentioned fusion of the background sparse point cloud corresponding to the static background region in the image sequence with the dynamic region point cloud to obtain an initial three-dimensional Gaussian particle set includes: performing inter-frame modeling on the dynamic region to obtain inter-frame modeling results; using the inter-frame modeling results to perform predictive three-dimensional information completion on the three-dimensional point positions and / or local geometric structures of the dynamic region to obtain an initial point cloud of the dynamic region; filtering, denoising, and / or confidence filtering on the initial point cloud of the dynamic region to obtain a dynamic region point cloud; performing multi-view-based geometric structure restoration on the static background region to obtain a background sparse point cloud; fusing the background sparse point cloud and the dynamic region point cloud to obtain an initial three-dimensional point cloud representation; and initializing the parameters of multiple three-dimensional Gaussian particles using the initial three-dimensional point cloud representation to obtain an initial three-dimensional Gaussian particle set, wherein the parameters of the three-dimensional Gaussian particles include center position parameters, rotation parameters, scale parameters, opacity parameters, color parameters, appearance feature parameters, and motion feature parameters.

[0063] The embodiments described above relate to feedforward initialization and deformable 3D Gaussian joint representation for dynamic scenes. Addressing the issues of insufficient initialization of the 3D structure of dynamic regions and incomplete distribution of foreground Gaussian particles in existing dynamic 3D Gaussian methods when input observations are sparse or involve significant motion, this application introduces a dynamic region structure recovery mechanism based on VGGT feedforward reconstruction in the initial modeling stage. This mechanism completes the 3D information of the dynamic region and, combined with a deformable 3D Gaussian representation in a unified normed space, models the temporal state of each Gaussian particle in the scene, enabling each Gaussian particle to explicitly represent its position, rotation, scale, and appearance changes over time. Compared to implicit fitting methods that rely solely on the overall deformation field, the embodiments provided in this application not only improve the initialization quality of dynamic regions but also express the local motion differences of different regions in a more granular manner, enhancing the completeness, interpretability, and expressiveness of spatiotemporal modeling.

[0064] The process of obtaining an initial three-dimensional Gaussian particle set provided in this application will be further described in detail below through specific implementation methods.

[0065] The process involves acquiring image sequences and corresponding timestamps of the dynamic scene to be processed; these image sequences can be monocular video sequences or multi-view image sequences. When the camera pose is known or can be estimated, camera intrinsic and extrinsic parameters are further acquired for subsequent 3D structure restoration and differentiable rendering. For static background regions in the scene, a method based on multi-view geometry or structure restoration is used to obtain sparse background point clouds. For dynamic regions in the scene, addressing the problem that traditional structure restoration methods struggle to establish stable matching under conditions of large-scale movement, occlusion, or viewpoint changes, leading to missing 3D structures in the dynamic foreground, a feedforward 3D structure prediction method based on inter-frame relationship modeling is used to complete the 3D information of the dynamic regions, resulting in a point cloud representation of the dynamic regions. This feedforward 3D structure prediction method can directly predict the 3D point positions or local geometric structures of the corresponding dynamic regions based on two or more frames of image input using a deep neural network. The sparse background point cloud and the dynamic region point cloud are then fused to construct an initial 3D point cloud representation of the scene, which is further initialized as a set of 3D Gaussian particles. .in, This represents the total number of Gaussian particles, or a single three-dimensional Gaussian particle (or simply a Gaussian particle). At least the following parameters should be included: The center position of the Gaussian particle; : The rotational parameters of the Gaussian particle; : The scale parameter of the Gaussian particle; Opacity parameter of Gaussian particles; The color or appearance characteristics of Gaussian particles; : Motion characteristic parameters of Gaussian particles.

[0066] In the specific implementation process, the point cloud completion results of the dynamic region are screened, denoised, or filtered by confidence level before participating in the initial Gaussian particle construction to improve the stability of subsequent gauge space modeling. The final initial 3D Gaussian particle set is used for subsequent deformable 3D Gaussian spatiotemporal modeling in gauge space.

[0067] The above specific implementation involves dynamic-enhanced initialization technology, which means that in the initial 3D modeling stage, by combining traditional structure restoration methods with feedforward 3D reconstruction methods, the dynamic region is supplemented with modeling, thereby improving the initial distribution integrity of Gaussian particles in the dynamic scene.

[0068] According to embodiments of this application, the above-described prediction of the future state of key three-dimensional Gaussian particles based on graph attention mechanism using a key point graph structure with key three-dimensional Gaussian particles as nodes to obtain the future spatiotemporal state information of key three-dimensional Gaussian particles includes: selecting key three-dimensional Gaussian particles that meet preset selection conditions from an initial set of three-dimensional Gaussian particles; constructing a key point graph structure with key three-dimensional Gaussian particles as nodes and the association relationship between key three-dimensional Gaussian particles and other three-dimensional Gaussian particles as edges, wherein the association relationship includes spatial proximity, similarity of motion features and / or correlation of topological structure; and performing future state prediction of the key point graph structure based on graph attention mechanism using a sliding time window to obtain the future spatiotemporal state information representing the motion state of key three-dimensional Gaussian particles at multiple future moments.

[0069] The above embodiments of this application involve a key-point driven Gaussian particle motion distillation mechanism: to address the problem of large computational load and complex spatiotemporal dependencies in directly predicting the future state of large-scale Gaussian particles, this application extracts representative motion key points (i.e., key three-dimensional Gaussian particles) from a three-dimensional Gaussian particle set, uses fewer key points to represent the overall motion trend, and establishes a mapping relationship between key points and ordinary three-dimensional Gaussian particles (i.e., other three-dimensional Gaussian particles). The key points drive the future state changes of the entire particle, thereby reducing the prediction dimensionality and improving the future prediction efficiency while retaining the main motion characteristics.

[0070] Furthermore, the above embodiments of this application also involve key point spatiotemporal relationship prediction based on a graph attention mechanism: using key points as graph nodes, a graph structure relationship is constructed between key points, and a graph attention network is used to model the state of key points at several past moments to predict the motion state of key points at future moments. Compared with methods that rely solely on point-by-point regression or simple temporal extrapolation, this approach can better model the local coupling relationships and global cooperative motion relationships in dynamic scenes, improving the accuracy and stability of future predictions.

[0071] According to an embodiment of this application, the above-mentioned method of using the future spatiotemporal state information of key three-dimensional Gaussian particles to drive other three-dimensional Gaussian particles in an initial set of three-dimensional Gaussian particles to obtain the future spatiotemporal motion information of the initial set of three-dimensional Gaussian particles includes: constructing multiple mapping relationships between multiple associated key three-dimensional Gaussian particles that are related to other three-dimensional Gaussian particles and other three-dimensional Gaussian particles; performing corresponding weighted operations on the future spatiotemporal state information of multiple associated key three-dimensional Gaussian particles and the control weights of the multiple mapping relationships to obtain the future spatiotemporal state information of other three-dimensional Gaussian particles; and performing multiple mapping relationship construction operations and corresponding weighted operation operations on each other three-dimensional Gaussian particle in the initial set of three-dimensional Gaussian particles to obtain the future spatiotemporal motion information of the initial set of three-dimensional Gaussian particles.

[0072] The embodiments described above in this application fully characterize the complex coupling relationships between particles, adapting to the motion characteristics of complex dynamic scenes. Traditional technical solutions often employ one-to-one single associations or global overall driving methods, which cannot characterize the complex motion relationships of local multi-particle interactions. The embodiments described above in this application associate each ordinary particle with multiple key particles and establish multiple mapping relationships, enabling accurate modeling of the spatiotemporal correlation characteristics of multi-source, local, and coupled elements in dynamic scenes. This aligns with the motion laws of multi-element linkage in real scenes, solving the problem of insufficient modeling of complex dynamic spatiotemporal dependencies in existing technologies. Simultaneously, by adaptively allocating weights based on different spatial locations and motion correlation strengths, the future spatiotemporal states of each ordinary particle are solved differentially, avoiding the loss of local motion details and the single, fixed motion expression caused by traditional uniform mapping and indiscriminate driving. This significantly improves the refinement and accuracy of the local motion expression of 3D particles. Furthermore, relying on the driving mechanism of joint weighted constraints of multiple key particles, the motion state of a single particle no longer depends on a single prediction result. It can be mutually constrained and complementaryly corrected through multi-source correlation information, effectively alleviating problems such as trajectory deviation, local motion anomalies, and temporal jumps that easily occur in pure data-driven predictions. This significantly improves the temporal stability and physical rationality of the future motion prediction of the global particle ensemble.

[0073] The embodiments described above in this application perform high-precision predictions only on a small number of key particles, and then complete global particle inference through a multi-mapping weighting mechanism, eliminating the need to predict massive numbers of particles individually. This significantly improves the modeling capabilities and prediction accuracy of complex dynamic scenes while effectively controlling computational overhead, balancing prediction accuracy and inference efficiency.

[0074] Figure 3 A flowchart is shown of a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model according to an embodiment of this application.

[0075] like Figure 3 As shown, the deformable three-dimensional Gaussian spatiotemporal motion prediction model provided in this application embodiment includes a deformation network, a physical information network, a graph structure prediction network, and a differentiable rendering module. The training method includes operations S310 to S340.

[0076] In operation S310, the deformation network is used to predict the deformation of the standard three-dimensional Gaussian sample particle set generated from the image sequence samples, so as to obtain the motion state information of the standard three-dimensional Gaussian particle sample set at multiple prediction times. The physical information network is used to impose physical constraints on the deformation prediction operation of the standard three-dimensional Gaussian particle sample set to obtain the physical loss value.

[0077] According to the embodiments of this application, the parameters of each three-dimensional Gaussian sample particle in the above-mentioned standard three-dimensional Gaussian sample particle set include center position parameters, rotation parameters, scale parameters, opacity parameters, color parameters, appearance feature parameters, and motion feature parameters.

[0078] According to embodiments of this application, the graph structure prediction network described above includes a graph attention network, a graph convolutional network, or a spatiotemporal graph neural network.

[0079] In operation S320, based on the motion state information of the standard three-dimensional Gaussian particle sample set at multiple prediction target times, a graph structure prediction network is used to perform a future state prediction operation based on the graph attention mechanism on the graph structure of key point samples, so as to obtain the future spatiotemporal state information of key three-dimensional Gaussian sample particles and obtain the prediction loss value generated by performing the future state prediction operation.

[0080] Among them, the key 3D Gaussian sample particles are obtained by screening the standard 3D Gaussian particle sample set, and the key point sample graph structure is constructed using the key 3D Gaussian sample particles as nodes.

[0081] The following detailed explanation of the future state prediction operation based on the graph attention mechanism is provided through specific implementation methods.

[0082] Using key sample points (i.e., key 3D Gaussian sample particles) as graph nodes, a key point sample graph structure is constructed based on spatial proximity, motion similarity, or topological correlation. This involves past continuous... The input graph attention prediction network takes the state of key sample points at each time step as input and outputs the state of key sample points at future time steps.

[0083] Let the graph attention prediction network be... ,but: in: Indicates the past Frame key sample point state sequence; This represents the predicted state of key sample points at the next time step. Graph Attention Network (GAT) is a neural network model based on a graph structure that models relationships between nodes by assigning adaptive weights to connections between them.

[0084] In a preferred embodiment, the state of key sample points at multiple future moments is continuously predicted using a sliding time window method. Then, through the mapping relationship between key sample points and Gaussian particles, the prediction results of key sample points are transmitted to all Gaussian particles, thereby obtaining the dynamic Gaussian particle set at future moments.

[0085] In operation S330, based on the future spatiotemporal motion information of the standard three-dimensional Gaussian particle sample set, the image reconstruction operation is performed on the image sequence sample using the differentiable rendering module to obtain the dynamic reconstructed image of the image sequence sample at future time, and the image reconstruction loss value generated by the image reconstruction operation is obtained.

[0086] Among them, the future spatiotemporal motion information of the standard three-dimensional Gaussian particle sample set is obtained by driving the future spatiotemporal state information of the key three-dimensional Gaussian sample particles to obtain the other three-dimensional Gaussian sample particles in the standard three-dimensional Gaussian particle sample set.

[0087] Input the dynamic Gaussian particle set into the differentiable rendering module to generate rendered images at the target viewpoint and target time. and with real images Comparisons are made to construct the image reconstruction loss: .

[0088] In the S340 operation, the parameters of the deformable 3D Gaussian spatiotemporal motion prediction model are optimized using physical loss, prediction loss, and image reconstruction loss, resulting in a trained deformable 3D Gaussian spatiotemporal motion prediction model.

[0089] Among them, the trained deformable 3D Gaussian spatiotemporal motion prediction model is used to execute the deformable 3D Gaussian spatiotemporal motion prediction method.

[0090] The image reconstruction loss, physical loss, and future prediction loss are combined to form the total loss function: .in: These are the loss weighting coefficients; This represents the loss for predicting the future state of key sample points. Through joint optimization, the model simultaneously possesses good dynamic representation capabilities, physical consistency, and future prediction capabilities.

[0091] The training method for the deformable 3D Gaussian spatiotemporal motion prediction model provided in the above embodiments of this application applies dynamic physical constraints to the Gaussian particle deformation prediction process through a physical information network, and constructs physical loss values ​​to participate in the model's back-optimization, breaking the limitations of traditional pure data-driven training without physical prior constraints. This ensures that the particle deformation and motion evolution laws learned by the model closely match real physical characteristics, effectively preventing particle trajectory drift, motion anomalies, structural distortions, and mode collapse problems that occur in out-of-training predictions, significantly improving the physical consistency and temporal stability of dynamic deformation modeling. Simultaneously, based on selected key Gaussian particles, a graph structure is constructed. Utilizing the attention mechanism of the graph structure prediction network, it adaptively learns the local coupling relationships and long-term temporal dependencies between particles, enabling accurate modeling of nonlinear spatiotemporal motion laws in complex dynamic scenes. This solves the shortcomings of traditional methods in insufficient modeling of long-term dependencies in dynamic scenes and local particle coupling relationships, significantly improving the model's ability to perceive and predict complex dynamic motions.

[0092] According to an embodiment of this application, the above-mentioned standard three-dimensional Gaussian sample particle set is obtained through the following operations: using a three-dimensional Gaussian particle modeling network of a deformable three-dimensional Gaussian spatiotemporal motion prediction model, the background sparse point cloud representation and dynamic region point cloud representation of the image sequence sample are fused to obtain an initial three-dimensional Gaussian particle sample set; the initial three-dimensional Gaussian particle sample set is mapped to a normalized space to obtain a standard three-dimensional Gaussian sample particle set.

[0093] According to embodiments of this application, the above-mentioned physical information constraint on the deformation prediction operation of a standard three-dimensional Gaussian sample particle set using a physical information network to obtain a physical loss value includes: spatiotemporally encoding the motion state information of the standard three-dimensional Gaussian sample particles at the current prediction time to obtain the spatiotemporal characteristics of the standard three-dimensional Gaussian sample particle set; processing the spatiotemporal characteristics of the standard three-dimensional Gaussian sample particle set using a physical information network to obtain the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set; constructing physical residuals using the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set according to preset physical conservation constraints, wherein the preset physical conservation constraints include momentum conservation constraints, mass conservation constraints, or energy conservation constraints based on continuous medium dynamics; and applying physical information constraints to the deformation prediction operation of the standard three-dimensional Gaussian sample particle set using the physical residuals to obtain a physical loss value.

[0094] The deformation prediction operation and physical information constraint operation involved in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0095] First, physical-informed constraints refer to the introduction of physical constraints (such as dynamic relationships) into the model training or prediction process to limit the model output results and make them conform to physical consistency.

[0096] Deformable 3D Gaussian Modeling in Standard Space: Initial 3D Gaussian sample particles are mapped to a unified standard space, where each 3D Gaussian sample particle is treated as an independently trackable spatiotemporal motion unit. The spatial position, temporal information, and motion characteristics of each Gaussian sample particle are encoded and input into a deformation network to predict the state change of the 3D Gaussian sample particle at the target time.

[0097] Let the deformable network be Then the first A three-dimensional Gaussian sample particle at time... The deformation output is shown in formula (1):

[0098] (1).

[0099] in, Represents the encoding function. Indicates the displacement increment. Indicates the rotation increment. Indicates the scale increment. This indicates the increment in opacity.

[0100] The time is obtained from this. The dynamic Gaussian state is shown in equation (2):

[0101] (2).

[0102] in, This indicates a rotation combination operation.

[0103] The above operations output the dynamic three-dimensional Gaussian sample particle states at each time step, providing a basic representation for subsequent physical constraints and future predictions.

[0104] Physical information constraint operation: In order to improve the physical consistency of Gaussian particle spatiotemporal motion prediction, this application introduces physical information constraints in the deformation modeling process of three-dimensional Gaussian sample particles to jointly model the velocity field and internal physical state of three-dimensional Gaussian sample particles.

[0105] Let the physical information network be... Then for spacetime points The output is shown in formula (3):

[0106] (3).

[0107] in, Represents spatiotemporal feature encoding, Represents the velocity field. It represents stress-related physical quantities or internal state quantities.

[0108] Based on the momentum balance relationship in continuous medium dynamics, the physical residual is constructed as shown in formula (4):

[0109] (4).

[0110] in, Indicates the density of the medium. The mass derivative of velocity with respect to time. The divergence term represents the internal state quantity. Indicates the external force term. Represents physical residuals.

[0111] The physical loss term is further defined as shown in formula (5):

[0112] (5).

[0113] in, Indicates the number of sampling points.

[0114] By introducing this physical loss term into the model training process, the predicted 3D Gaussian sample particle motion not only meets the image supervision requirements but also the physical consistency requirements, thereby improving the stability of future prediction stages.

[0115] The embodiments described above provide a deformable Gaussian motion constraint mechanism that integrates physical information for future prediction. Addressing the problems of existing Gaussian prediction methods that primarily rely on data-driven extrapolation, are prone to trajectory drift outside the training timeframe, exhibit non-physical deformation, and suffer from prediction instability, this application introduces physical information constraints based on continuum dynamics during deformable 3D Gaussian spatiotemporal modeling. This imposes physical consistency constraints on the motion states of 3D Gaussian sample particles (or 3D Gaussian particles in the inference process), ensuring that the particle velocity evolution and related internal state changes satisfy preset physical laws. By combining physical constraints with the future prediction task, this approach can improve the physical rationality and temporal stability of future state inferences while maintaining the ability to express dynamic scenes, thereby enhancing the model's generalization prediction capability in complex dynamic scenarios.

[0116] According to embodiments of this application, the aforementioned key 3D Gaussian sample particles are obtained by screening a standard 3D Gaussian particle sample set, including: screening the standard 3D Gaussian particle sample set using the spatial distribution and motion characteristics of the 3D Gaussian particles; or screening the standard 3D Gaussian particle sample set using local gradient response information characterizing the geometric texture of an image sequence; or clustering the standard 3D Gaussian particle sample set and screening the standard 3D Gaussian particle sample set based on the clustering results; or screening the standard 3D Gaussian particle sample set using a farthest point sampling algorithm.

[0117] The following detailed description of the screening process for the key three-dimensional Gaussian sample particles provided in this application is provided through specific implementation methods.

[0118] To address the problem of the large number of Gaussian particles and the high cost of direct future prediction, a key sample point set is extracted from the standard three-dimensional Gaussian sample particle set in the gauge space. ,in This represents the number of key sample points. Key sample points can be determined based on spatial distribution, motion significance, local gradient response, or clustering results.

[0119] For each ordinary 3D Gaussian sample particle (i.e., other 3D Gaussian sample particles), establish the mapping relationship between it and several key sample points, and calculate the control weights. The future motion state of an ordinary Gaussian particle is obtained by weighting the state changes of relevant key sample points. This step enables a low-dimensional compressed representation of the motion of large-scale Gaussian particles.

[0120] The following describes specific implementation methods in conjunction with appendices. Figures 4-7 The present application provides a more detailed explanation of the deformable 3D Gaussian spatiotemporal motion prediction method and the training method of the deformable 3D Gaussian spatiotemporal motion prediction model.

[0121] Figure 4 The diagram illustrates the training and inference process of a deformable Gaussian motion prediction model that incorporates physical information according to an embodiment of this application.

[0122] To address the shortcomings of existing dynamic 3D Gaussian modeling methods, such as insufficient physical consistency, inadequate modeling of local motion relationships, and poor prediction stability during future prediction, this application provides a training and inference method for a deformable 3D Gaussian spatiotemporal motion prediction model that integrates physical information and graph attention mechanisms. This method is applicable to dynamic scene modeling, future motion prediction, and new view synthesis in scenarios with monocular video sequences or multi-view image sequences as input, and belongs to the interdisciplinary field of 3D reconstruction, spatiotemporal modeling, physical information neural networks, and graph neural networks. The overall technical solution is as follows: Figure 4 As shown, this approach includes stages such as normalized space initialization, deformable Gaussian representation of physical information, and future prediction, forming an integrated technical route of "dynamic enhanced initialization + deformable 3D Gaussian representation + physical constraints + keypoint distillation + graph attention prediction". This application does not merely improve a single aspect of dynamic reconstruction or future prediction, but organically combines dynamic region initialization and completion, deformable 3D Gaussian particle representation, physical information constraints, keypoint motion distillation, and graph structure future prediction to form a unified framework for dynamic scene spatiotemporal motion prediction, thus balancing dynamic representation integrity, physical consistency, prediction stability, and future extrapolation capability. Specifically, keypoint motion distillation refers to a dimensionality reduction modeling method that extracts a small number of representative keypoints from a large number of Gaussian particles and uses the motion state of these keypoints to drive the overall particle motion.

[0123] like Figure 4 As shown, the process involves several steps. First, the normalized space is initialized by using a feedforward Transformer network to complete the 3D information of dynamic regions in the image sequence, generating a dynamically enhanced 3D point cloud to achieve an initial 3D structural representation. Then, the sparse background point cloud is fused with the dynamic region point cloud to construct a dynamic Gaussian particle set in the normalized space, resulting in the initial 3D Gaussian representation. Finally, the parameters of each Gaussian particle are output, including position, rotation, scale, color, opacity, momentum, and velocity.

[0124] like Figure 4 As shown, the physically deformable Gaussian is constructed based on the Lagrangian perspective (a modeling approach that uses a single particle as the tracking object to describe its time-varying trajectory and state changes). This constructs a physically deformable field to impose continuous medium dynamic constraints on the Gaussian particle in gauge space. Using a physically-informed neural network, physical residual terms are constructed based on the Cauchy momentum equation to calculate the velocity and stress fields, where the velocity field is a three-dimensional vector and the stress field is a six-dimensional symmetric tensor. A physical loss is constructed using the sum of squared residuals to constrain the deformation process of the Gaussian particle and update its motion state parameters.

[0125] like Figure 4 As shown, Future Prediction (the process of inferring and generating scene states for future moments outside the training timeframe based on historical time-series data) employs a concentric motion distillation method to filter motion keypoints from a dynamic Gaussian particle set. A keypoint graph structure is constructed using these keypoints as nodes. A graph attention network is then used to predict the future states of the keypoint graph structure based on a graph attention mechanism, obtaining the future spatiotemporal state information of key 3D Gaussian particles. Based on the future state information of the key particles, other 3D Gaussian particles are driven to obtain the future spatiotemporal motion information of the overall scene, achieving future view rendering and reconstruction.

[0126] The training and inference method for the deformable Gaussian motion prediction model that integrates physical information provided in this application can effectively solve the problems of motion divergence, local structural distortion, lack of physical consistency, and insufficient modeling of complex spatiotemporal dependencies that are prone to occur in the time extrapolation process of existing dynamic 3D Gaussian modeling methods.

[0127] Figure 5 The diagram shows the effect of inter-frame reconstruction and global reconstruction based on feedforward Transformer for real-world datasets and physics simulation datasets according to embodiments of this application.

[0128] Figure 5 The paper demonstrates the inter-frame 3D reconstruction process based on feedforward Transformer in both real-world and physically simulated scenarios, and compares the effects of using 1 frame, 2 frames, 8 frames of input images, and global reconstruction.

[0129] The real-world scene reconstruction process is as follows: With a 1-frame input, only sparse and incomplete scene point clouds can be reconstructed. The camera pose is uniform, and the 3D structure of dynamic regions (such as brooms) is significantly missing. With a 2-frame input, the completeness of the point cloud is improved based on two-view geometric constraints, and the outlines of dynamic objects begin to appear, but the details are still relatively coarse. With a 8-frame input, as information from multiple frames accumulates, the scene structure and details of dynamic objects become significantly richer, and the camera trajectory (colored wireframe) begins to show a multi-view distribution. Global reconstruction fuses all frame information to obtain a complete scene point cloud and camera trajectory. The 3D structure of dynamic objects is completed, achieving a complete reconstruction of the real dynamic scene.

[0130] The physical simulation scene reconstruction involves inputting 1 / 2 / 8 frames of images. As the number of input frames increases, the 3D point clouds of static objects such as the chessboard and trophy gradually become complete, and the multi-view distribution of the camera trajectory becomes clearer. Global reconstruction obtains complete point clouds of the chessboard and trophy scenes, as well as continuous camera motion trajectories, achieving a complete reconstruction of the simulated dynamic scene.

[0131] Figure 6 The diagram illustrates a canonical space deformable Gaussian model and a physical information deformation field structure diagram according to an embodiment of this application.

[0132] Figure 6 The network architecture of the physical information deformation field from a Lagrange perspective is shown: where the input is the spatiotemporal coordinate point. The system sequentially passes through a six-plane feature embedding module, Fourier time coding, and a joint prediction framework for velocity and stress fields, ultimately outputting the velocity field corresponding to that point. With stress field Among them, the six-plane feature embedding module will input four-dimensional spatiotemporal coordinates. The system is decomposed into six orthogonal planes: XY, XZ, YZ, XT, ZT, and YT. Two-dimensional feature maps are extracted independently from each plane. These planar features, after projection and fusion, generate a unified high-dimensional spatiotemporal feature representation, providing multi-view spatial and temporal information for subsequent predictions. Among these, the joint prediction framework of velocity and stress fields is the core of the physical information (i.e., physically constrained) neural network, including a multilayer perceptron (receiving spatiotemporal features and outputting the velocity field). With stress field The module includes a physical constraint module (used to construct physical residuals based on partial differential equations (such as the Cauchy momentum equation) and a measurement / analytical loss module (which uses the output of the data loss supervision network to achieve joint optimization of data-driven and physical constraints).

[0133] Figure 7 A diagram illustrating the keypoint motion distillation and graph attention prediction process according to an embodiment of this application is shown.

[0134] like Figure 7 As shown, with motion key points , , , Construct a graph structure for the nodes, with edges representing relationships between them. Each node carries a feature vector C. Calculate attention weights for each node and its neighboring nodes. The weights are obtained by performing a non-linear transformation on the node features using the Tanh activation function. Based on the attention weights, the features of neighboring nodes are weighted and aggregated to update the feature representation of each node, modeling the local dependencies between nodes. After multi-layer stacking, the node features gradually incorporate a wider range of neighborhood information, forming a higher-order relational representation. The node features updated through multiple attention layers are then input into a multilayer perceptron to obtain the updated keypoint features. , , , That is, the future spatiotemporal state information of key points.

[0135] In the specific implementation, the input is a set of monocular video sequences. First, a feedforward 3D structure prediction model is used to estimate 3D point clouds from any two frames of images and initialize them as several 3D Gaussian particles. Then, the Gaussian particles are mapped to a gauge space, and the displacement and rotation of each particle at different time steps are predicted through a deformation network. Based on this, a physical information deformation field network is constructed to jointly predict the velocity and stress fields of the Gaussian particles, and the residual of the Cauchy momentum equation is used as a physical loss term in the training. Then, several keypoints are distilled from the Gaussian particles, and a keypoint graph structure model is constructed. The future state of the keypoints is predicted using a graph attention mechanism. Finally, the deformation of the Gaussian particles is driven by the predicted keypoint states to achieve future motion prediction of dynamic scenes and synthesis of new views.

[0136] As can be seen from the above embodiments, specific implementation methods, and accompanying drawings, the method provided in this application has the following advantages:

[0137] (1) By adopting a deformable 3D Gaussian particle-level spatiotemporal representation, the ability to model local motion in dynamic scenes is improved. Existing methods often treat dynamic scenes as a whole deformation field, which makes it difficult to express the local motion differences between different regions in detail. This application represents the scene as multiple independently trackable Gaussian particles and predicts their temporal state changes separately. Therefore, it can more accurately describe the local non-rigid motion in complex dynamic scenes and improve the spatiotemporal representation accuracy.

[0138] (2) By introducing physical information constraints, the physical consistency and stability of the future prediction process are improved. Purely data-driven spatiotemporal extrapolation is prone to trajectory drift, local deformation distortion and prediction collapse outside the training time range. This application imposes physical constraints on the velocity field and internal state of Gaussian particles, so that the prediction process is restricted by physical laws, thus reducing non-physical motion results and improving the stability of future predictions.

[0139] (3) By adopting a key point motion distillation mechanism, the computational complexity of future prediction for large-scale particle systems is reduced. This application does not directly predict the future state of all Gaussian particles one by one, but first extracts key points, and then uses the key points to drive the overall particle motion. Since the number of key points is much smaller than the number of all Gaussian particles, it is possible to reduce the prediction dimension while maintaining the main motion characteristics, thereby improving training efficiency and inference efficiency.

[0140] (4) By using a graph attention mechanism to model the spatiotemporal dependencies between key points, the accuracy of future motion prediction is improved. Compared with methods that only use point-by-point regression or ordinary time series networks, this application adaptively models the local associations and global dependencies between key points through a graph attention mechanism, enabling the prediction model to better handle coupled motion and cooperative changes in complex dynamic scenes, and thus has better future prediction capabilities.

[0141] (5) Through the synergistic cooperation of "physical constraints + graph attention prediction", the integration of dynamic representation and future prediction is realized. This application does not improve the reconstruction module or the prediction module separately, but unifies Gaussian particle spatiotemporal representation, physical information constraints, key point distillation and graph structure prediction under the same framework, so that the modules cooperate with each other to improve the quality of dynamic scene modeling, the stability of future prediction and the interpretability of results.

[0142] Figure 8 A block diagram of an electronic device suitable for implementing a deformable three-dimensional Gaussian spatiotemporal motion prediction method and a training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model, according to an embodiment of this application, is shown.

[0143] like Figure 8As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0144] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0145] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0146] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0147] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0148] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0149] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0151] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0155] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A deformable three-dimensional Gaussian spatiotemporal motion prediction method, characterized in that, The method includes: The dynamic regions in the image sequence are completed with three-dimensional information based on inter-frame relationships to obtain dynamic region point clouds. The background sparse point clouds corresponding to the static background regions in the image sequence are fused with the dynamic region point clouds to obtain an initial three-dimensional Gaussian particle set. The image sequence is a multi-frame image sequence generated under a preset camera pose. The future state prediction of the key point graph structure with key three-dimensional Gaussian particles as nodes is performed based on the graph attention mechanism to obtain the future spatiotemporal state information of the key three-dimensional Gaussian particles. The key three-dimensional Gaussian particles are selected from the initial set of three-dimensional Gaussian particles based on preset screening conditions. Using the future spatiotemporal state information of the key three-dimensional Gaussian particle, other three-dimensional Gaussian particles in the initial three-dimensional Gaussian particle set are driven to obtain the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set; The image sequence is rendered and reconstructed using the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set to obtain a dynamic image sequence corresponding to the preset camera pose at the target's future time.

2. The method according to claim 1, characterized in that, The initial three-dimensional Gaussian particle set is obtained by fusing the sparse point cloud of the static background region in the image sequence with the point cloud of the dynamic region, including: Inter-frame modeling is performed on the dynamic region to obtain the inter-frame modeling results; The inter-frame modeling results are used to predictively complete the 3D point positions and / or local geometry of the dynamic region to obtain the initial point cloud of the dynamic region. The initial point cloud of the dynamic region is filtered, denoised, and / or confidence filtered to obtain the dynamic region point cloud. The static background region is subjected to geometric structure restoration based on multiple views to obtain a sparse point cloud of the background. The background sparse point cloud and the dynamic region point cloud are fused to obtain an initial three-dimensional point cloud representation; The parameters of multiple three-dimensional Gaussian particles are initialized using the initial three-dimensional point cloud representation to obtain the initial three-dimensional Gaussian particle set. The parameters of the three-dimensional Gaussian particles include center position parameters, rotation parameters, scale parameters, opacity parameters, color parameters, appearance feature parameters, and motion feature parameters.

3. The method according to claim 1, characterized in that, The future state prediction of the key point graph structure with key 3D Gaussian particles as nodes is performed based on a graph attention mechanism, and the future spatiotemporal state information of the key 3D Gaussian particles is obtained as follows: Select key three-dimensional Gaussian particles that meet the preset selection criteria from the initial set of three-dimensional Gaussian particles; The key point graph structure is constructed using the key three-dimensional Gaussian particle as a node and the association relationship between the key three-dimensional Gaussian particle and the other three-dimensional Gaussian particles as an edge, wherein the association relationship includes spatial proximity, similarity of motion features and / or correlation of topological structure. By employing a sliding time window approach, future state prediction based on a graph attention mechanism is performed on the key point graph structure to obtain future spatiotemporal state information characterizing the motion state of the key three-dimensional Gaussian particle at multiple future moments.

4. The method according to claim 1, characterized in that, Using the future spatiotemporal state information of the key three-dimensional Gaussian particle to drive other three-dimensional Gaussian particles in the initial three-dimensional Gaussian particle set, the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set is obtained, including: Construct multiple mapping relationships between multiple key three-dimensional Gaussian particles that are associated with the other three-dimensional Gaussian particles and the other three-dimensional Gaussian particles; The future spatiotemporal state information of the multiple associated key three-dimensional Gaussian particles is weighted and calculated with the control weights of the multiple mapping relationships to obtain the future spatiotemporal state information of the other three-dimensional Gaussian particles. Perform multiple mapping relationship construction operations and corresponding weighted operation operations on each of the other three-dimensional Gaussian particles in the initial three-dimensional Gaussian particle set to obtain the future spatiotemporal motion information of the initial three-dimensional Gaussian particle set.

5. A training method for a deformable three-dimensional Gaussian spatiotemporal motion prediction model, characterized in that, The deformable 3D Gaussian spatiotemporal motion prediction model includes a deformation network, a physical information network, a graph structure prediction network, and a differentiable rendering module. The training method includes: The deformation network is used to predict the deformation of a standard three-dimensional Gaussian sample particle set generated from image sequence samples, so as to obtain the motion state information of the standard three-dimensional Gaussian particle sample set at multiple prediction times. The physical information network is used to impose physical constraints on the deformation prediction operation of the standard three-dimensional Gaussian particle sample set to obtain the physical loss value. Based on the motion state information of the standard three-dimensional Gaussian particle sample set at multiple prediction target times, the graph structure prediction network is used to perform a future state prediction operation based on a graph attention mechanism on the key point sample graph structure to obtain the future spatiotemporal state information of the key three-dimensional Gaussian sample particles, and the prediction loss value generated by performing the future state prediction operation is obtained. The key three-dimensional Gaussian sample particles are obtained by filtering the standard three-dimensional Gaussian particle sample set, and the key point sample graph structure is constructed with the key three-dimensional Gaussian sample particles as nodes. Based on the future spatiotemporal motion information of the standard three-dimensional Gaussian particle sample set, the image reconstruction operation is performed on the image sequence sample using the differentiable rendering module to obtain the dynamic reconstructed image of the image sequence sample at a future time, and the image reconstruction loss value generated by the image reconstruction operation is obtained. The future spatiotemporal motion information of the standard three-dimensional Gaussian particle sample set is obtained by driving the other three-dimensional Gaussian sample particles in the standard three-dimensional Gaussian particle sample set with the future spatiotemporal state information of the key three-dimensional Gaussian sample particles. The parameters of the deformable 3D Gaussian spatiotemporal motion prediction model are optimized using the physical loss value, the prediction loss value, and the image reconstruction loss value to obtain a trained deformable 3D Gaussian spatiotemporal motion prediction model, wherein the trained deformable 3D Gaussian spatiotemporal motion prediction model is used to perform the method described in any one of claims 1 to 4.

6. The training method according to claim 5, characterized in that, The standard three-dimensional Gaussian sample particle set is obtained through the following operation: The background sparse point cloud representation and dynamic region point cloud representation of the image sequence samples are fused using the three-dimensional Gaussian particle modeling network of the deformable three-dimensional Gaussian spatiotemporal motion prediction model to obtain an initial three-dimensional Gaussian particle sample set. The initial three-dimensional Gaussian particle sample set is mapped to the gauge space to obtain the standard three-dimensional Gaussian sample particle set.

7. The training method according to claim 5, characterized in that, Using the physical information network to perform physical information constraints on the deformation prediction operation of the standard three-dimensional Gaussian sample particle set, the physical loss values ​​obtained include: The motion state information of the standard three-dimensional Gaussian sample particles at the current prediction time is spatiotemporally encoded to obtain the spatiotemporal features of the standard three-dimensional Gaussian sample particle set. The spatiotemporal characteristics of the standard three-dimensional Gaussian sample particle set are processed using the physical information network to obtain the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set. Based on preset physical conservation constraints, physical residuals are constructed using the velocity field and internal stress state quantities of the standard three-dimensional Gaussian sample particle set. The preset physical conservation constraints include momentum conservation constraints, mass conservation constraints, or energy conservation constraints based on continuous medium dynamics. The physical loss value is obtained by applying physical information constraints to the deformation prediction operation of the standard three-dimensional Gaussian sample particle set using the physical residual.

8. The training method according to claim 5, characterized in that, The key three-dimensional Gaussian sample particles are obtained by screening the standard three-dimensional Gaussian particle sample set, including: The key three-dimensional Gaussian sample particles are obtained by screening the standard three-dimensional Gaussian particle sample set using the spatial distribution and motion characteristics of the three-dimensional Gaussian particles; or The key three-dimensional Gaussian sample particles are obtained by filtering the standard three-dimensional Gaussian particle sample set using local gradient response information characterizing the geometric texture of the image sequence; or The key three-dimensional Gaussian sample particles are obtained by clustering the standard three-dimensional Gaussian particle sample set and filtering the sample set based on the clustering results; or The key three-dimensional Gaussian sample particles are obtained by filtering the standard three-dimensional Gaussian particle sample set using the farthest point sampling algorithm.

9. The training method according to any one of claims 5 to 8, characterized in that, The parameters of each three-dimensional Gaussian sample particle in the standard three-dimensional Gaussian sample particle set include center position parameter, rotation parameter, scale parameter, opacity parameter, color parameter, appearance feature parameter, and motion feature parameter.

10. The training method according to any one of claims 5 to 8, characterized in that, The graph structure prediction network includes graph attention networks, graph convolutional networks, or spatiotemporal graph neural networks.

Images