Gaussian splashing method and device without annotation guidance and medium

Abstract

The invention relates to a Gaussian splashing method and device without annotation guidance and a medium, and the method comprises the following steps: dynamic Gaussian stream decomposition: obtaining a group of video streams in an interaction scene, and capturing the motion trail of a Gaussian ball through reconstructing a 4D Gaussian field; dynamic Gaussian clustering and tracking: positioning interactive object areas through a flow decomposition and clustering algorithm, and tracking a motion track of a Gaussian ball in each area; and self-guiding control: the reconstruction and control of the interactive scene are realized through the motion direction and speed of a three-dimensional spherical vector modeling Gaussian ball, and the three-dimensional spherical vector is directly obtained from a dynamic Gaussian tracking trajectory. Compared with the prior art, the method has the advantages that additional labeling is not needed, and self-supervision optimization of Gaussian motion can be achieved.

Description

Technical Field

[0001] The present invention relates to reconstruction and control technology in monocular video, and in particular to a Gaussian splashing method, device and medium without labeling guidance. Background Art

[0002] 4D New View Synthesis: Neural Radiance Fields (NeRF) have made great progress in dynamic scene reconstruction. Existing methods can be divided into three categories: time-varying, deformable canonical, and hybrid representation methods. Time-varying methods directly model the time-varying radiation field and enhance the temporal information through temporal embedding, scene flow, etc. The deformable canonical method decouples the 4D field into a dynamic deformable field and a static canonical space, and queries the canonical features by warping coordinates. In contrast, the hybrid representation method achieves high-quality reconstruction and fast rendering by exploiting spatiotemporal feature planes, dynamic voxels, and 4D hash coding.

[0003] 3D Gaussian Splashing: Compared with using MLP to fit complex dynamic scenes, 3D Gaussian Splashing has become a popular choice recently due to its superior training efficiency and ultra-high-quality rendering speed. Related progress usually directly learns dense Gaussian motion, utilizes feature planes or learnable motion basis to obtain better rendering quality, or introduces different paradigms of flow loss to enhance dynamic 3DGS. Recently, S4D introduced a generalized flow pipeline to reconstruct 4D real-world scenes using Gaussian and 3D control points.

[0004] Controllable scene representation: Separating color, transparency, geometry from time provides greater flexibility for 4D reconstruction, which is of great significance for digital humans and simulators. CoNeRF pioneered this field by extending HyperNeRF and regressing attributes and masks to achieve few-shot attribute control. CoNFies proposed a controllable facial self-portrait representation using AU intensity and facial landmarks. EditableNeRF introduced a mechanism for detecting key points and joint weight optimization. In contrast, CoGS exploits 3D Gaussians to achieve real-time control of dynamic scenes without explicit control signals. Recently, LiveScene has pushed the progress to the scene level and introduced an efficient decomposition to decompose the interaction space. Despite the breakthroughs of these methods, they either require dense manual interaction variable annotations or mask supervision, which limits their applicability in the real world.

[0005] In summary, the shortcomings of the prior art mainly include the following aspects:

[0006] 1. Over-reliance on manual annotation: If there is no supervision of masks or control signals in the training data, the model will collapse and fail to correctly decode color information from features, thus losing control of the scene.

[0007] 2. Complex high-dimensional space modeling: As the number of interactive objects in the scene increases, the complexity of the interactive space continues to increase. Traditional methods are usually difficult to effectively capture and model such high-dimensional interactive spaces, resulting in insufficient modeling accuracy when dealing with complex interactive scenes. This not only significantly increases computing time and memory overhead, but also prolongs the training time of the model. Summary of the invention

[0008] The purpose of the present invention is to overcome the limitations of the above-mentioned prior art and to provide a Gaussian splatting method, device and medium that do not require annotation guidance, so as to achieve zero-annotation, low-cost modeling in complex scenes. It uses flow priors to automatically explore interactive scene structures and restore scene interactivity without any manual annotations, while having high rendering quality and fast rendering speed.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] This invention aims to use 3D Gaussian splash technology to solve the reconstruction and control problems in monocular video. By deriving the dynamic Gaussian motion trajectory from the optical flow and camera motion, and using the trajectory as a constraint, the connection between 2D optical flow and 3D Gaussian dynamic control is established to achieve self-supervised optimization of Gaussian motion. This technological breakthrough greatly improves the operational convenience and controllability in practical applications, and provides important support for research and application in the fields of robot control and augmented reality.

[0011] According to a first aspect of the present invention, a Gaussian splashing method without labeling guidance is provided, comprising the following steps:

[0012] S1, dynamic Gaussian flow decomposition: obtain a set of video streams in the interactive scene and capture the motion trajectory of the Gaussian ball by reconstructing the 4D Gaussian field;

[0013] S2, dynamic Gaussian clustering and tracking: locate the interactive object area through flow decomposition and clustering algorithm, and track the movement trajectory of the Gaussian ball in each area;

[0014] S3, self-guided control: The movement direction and speed of the Gaussian ball are modeled by three-dimensional spherical vectors to achieve reconstruction and control of the interactive scene. The three-dimensional spherical vectors are directly obtained from the dynamic Gaussian tracking trajectory.

[0015] In S1, the dynamic Gaussian flow under instantaneous motion is analytically separated from the optical flow and camera motion by differential analysis of alpha synthesis.

[0016] Consider a dynamic scene with interactive objects. The camera and 3D Gaussian maintain different speeds in consecutive frames 0 and t. Assume that the dynamic 3D Gaussian G i The speed is v G, which is projected as image measurement g with translation velocity v and rotation velocity ω under constant camera instantaneous motion i , at time t, follows a two-dimensional Gaussian distribution g i The pixel m i,t The optical flow u caused by (v, ω) is expressed as:

[0017]

[0018] Among them, u Cam represents the camera-induced scene flow, represents the dynamic Gaussian flow on the image plane, Δ represents the flow residual term, and f x ,f y ,c x ,c y is the camera internal parameter, M means the Gaussian depth Z i The number of sorted Gaussian projections that intersect pixel m, T i is the transmittance, α i is the opacity learned by the Gaussian model, pixel m i,t satisfy μ i,t represents the two-dimensional average value, ∑ i i ,t represents the covariance, B i,t Represents the decomposition of the covariance matrix, dynamic Gaussian flow Use the 2D Gaussian mean displacement μ i,t -μ i,0 To accumulate.

[0019] In a static Gaussian scene, the optical flow caused by a pixel is expressed as u = u Cam .

[0020] Specifically, S2 is: using a set of camera-induced scene flows to pre-train a deformable 3D Gaussian G ′ , extract dynamic Gaussian flow frame by frame The flow map is then binarized to obtain a flow map, and the dynamic 3D Gaussians are identified by back-projecting the flow map, and Gaussians with sharp dynamics are highlighted. Among them, g i represents the Gaussian ball in the scene, Q represents the number of Gaussian balls; the unsupervised clustering algorithm DBSCAN is used to cluster the dynamic Gaussian balls into multiple clusters. Where K is the number of interacting objects, c i Represents the cluster center; the cluster center evolves over time, generating a continuous trajectory where k indicates the object the trajectory belongs to.

[0021] In S3, the Gaussian G and the network Θ for predicting the dynamic Gaussian offset are simultaneously optimized using a rasterization-based loss function, which measures the difference between the rendered image and the input image and the dynamic Gaussian flow.

[0022] During the training phase, cluster trajectory coordinates are used represents the Gaussian dynamical state and is related to the Gaussian center X i connect; use encode the coordinates and jointly train the network Θ to recover the Gaussian dynamics < ΔX i ,Δ∑ i >:

[0023]

[0024] Afterwards, splat rasterization is performed, combining the Gaussian with the predicted dynamic trajectory.

[0025] In the control phase, the interactive 3D vector v′ is input c , through f Θ (X i ,v′ c ) retrieves the Gaussian dynamics from the optimized network.

[0026] In summary, the key points of the present invention include:

[0027] 1. No annotation guidance: The method does not require manual annotation, so it is a more efficient and automated method.

[0028] 2. Dynamic Gaussian flow constraint: This method uses differential analysis with alpha synthesis to link optical flow, camera motion and dynamic Gaussian flow, and derives the dynamic Gaussian flow constraint analytically.

[0029] 3. Unsupervised scene structure positioning: Use flow constraints to achieve unsupervised interactive scene structure positioning.

[0030] 4. Continuous Gaussian Motion Variation Training: This method allows continuous Gaussian Motion Variation training, resulting in more realistic scene reconstruction.

[0031] These key points and points to be protected together constitute the core technology and innovation of the present invention, and provide a new solution for solving the problem of unsupervised interactive reconstruction.

[0032] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method described above when executing the program.

[0033] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method as described above when executed by a processor.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Higher rendering quality: By using dynamic 3D Gaussian for self-guiding control mechanism, the present invention can more accurately capture the motion trajectory of objects in complex interactive scenes, thereby improving the rendering quality.

[0036] (2) Mask-free training: Using optical flow as a priori, the present invention can adaptively perceive operable objects in the interaction field, saving a lot of manual labeling costs.

[0037] (3) Lower model parameters and faster rendering speed: The present invention separates static fields and dynamic objects, and reconstructs and controls controllable objects separately, saving a large amount of storage space and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 It is a schematic diagram of the dynamic Gaussian flow of the present invention;

[0040] Figure 3 Schematic diagram of dynamic Gaussian flow graph in static and dynamic scenes;

[0041] Figure 4 A comparison diagram of visualization results of new view synthesis on an OmniSim dataset between the present invention and the prior art in an embodiment;

[0042] Figure 5 A comparison diagram of visualization results of new view synthesis on the InterReal dataset in an embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] The present invention proposes a new technical solution, which is a Gaussian splashing method that does not require annotation guidance and can reconstruct controllable scenes. Given a set of video streams in an interactive scene, this method first captures the motion trajectory of the Gaussian ball by reconstructing a 4D Gaussian field, then locates the interactive object area through the decomposition and clustering algorithm of the flow, and tracks the Gaussian ball trajectory in each area, and finally models the movement direction and speed of the Gaussian ball through a 3D control vector, thereby realizing the reconstruction and control of the interactive scene. The innovation of the method lies in that, through differential analysis, the dynamic Gaussian flow can be analytically derived from the optical flow and camera motion. This makes it possible to track dynamic Gaussian motion relying only on the camera view during training, without any manual annotation, to locate the controllable structure and provide continuous optimization constraints, and to restore the controllable scene from continuous frames.

[0045] This innovation simplifies existing controllable view synthesis methods by introducing prior information based on optical flow, eliminating the dependence on annotations while improving the applicability of the method in the real world.

[0046] Figure 1 The complete pipeline of the present invention is presented. Given a set of video streams {P(t), I(t)}, a controllable 3D Gaussian G is recovered in two stages. * First, a deformable 3DGS is pre-trained and the dynamic Gaussian flow u is calculated from the optical and camera flows. G . Then, the dynamic Gaussian flow graph is reprojected and the Gaussian spheres are clustered using the DBSCAN algorithm, and trajectory computation is performed. During the controllable Gaussian training phase, the Gaussian G and the network Θ are optimized using a rasterization-based loss function that measures the difference between the rendered image and the input image, as well as the dynamic Gaussian flow. This method exploits the connection between dynamic Gaussian flow, optical flow, and camera motion to restore scene interactivity without any manual annotation. Dynamic Gaussian flow derivatives facilitate 3DGS trajectory clustering and enable a flexible 3D spherical vector control pipeline, which simplifies and accelerates interactive Gaussian modeling schemes.

[0047] Specifically, Figure 1 As shown, it includes the following steps:

[0048] S1, dynamic Gaussian flow decomposition: obtain a set of video streams in the interactive scene and capture the motion trajectory of the Gaussian ball by reconstructing the 4D Gaussian field.

[0049] Dynamic Gaussian flow under temporal motion is analytically separated from optical flow and camera motion via differential analysis of alpha synthesis.

[0050] Consider a dynamic scene with interactive objects such as Figure 2 As shown, the camera and the 3D Gaussian maintain different speeds in consecutive frames 0 and t. Assuming the dynamic 3D Gaussian Gi The speed is v G , which is projected as image measurement g with translation velocity v and rotation velocity ω under constant camera instantaneous motion i , at time t, follows a two-dimensional Gaussian distribution g i The pixel m i,t The optical flow u caused by (v,ω) can be obtained by the following Lemma 1.

[0051] Lemma 1: Dynamic Gaussian flow u under instantaneous motion GS It can be obtained by optical flow u and camera flow u Cam It is derived by the following transformation:

[0052] u=u Cam +u GS +Δ

[0053]

[0054] Among them, u Cam represents the camera-induced scene flow, Δ represents the flow residual term, and f x ,f y ,c x ,c y is the camera internal parameter, M means the Gaussian depth Z i The number of sorted Gaussian projections that intersect pixel m, T i is the transmittance, α i is the opacity learned by the Gaussian model. The flow residual terms Δ are retained to ensure accuracy, even though they approach zero after fine-tuning.

[0055] Lemma 1 illustrates the ternary relationship, but Gaussian flow is not suitable for joint 3DGS training. For flexibility, the present invention considers following a two-dimensional Gaussian distribution g at time t. i The pixel m i,t , and obtain Among them, μ i,t represents the two-dimensional average value, ∑ i ′ ,t represents the covariance, B i,t represents the decomposition of the covariance matrix. The following corollary describes the dynamic Gaussian flow using a two-dimensional Gaussian mean:

[0056] Corollary 1: Dynamic Gaussian Flow on the Image Plane The two-dimensional Gaussian mean displacement μ i,t -μ i,0 To accumulate:

[0057]

[0058] The expressions in Lemma 1 and Corollary 1 show that the dynamic Gaussian flow can be directly derived from the 2D image flow u and the camera-induced scene flow u Cam , and accumulate the 2DGS projection displacement μ i,t -μ i,0 This naturally aligns with the 3D Gaussian rasterization process, providing a continuous motion constraint for dynamic Gaussian optimization. Moreover, in a static Gaussian scene, this equation degenerates to the scene flow, where u = u Cam . Therefore, the generated dynamic Gaussian flow plot will highlight the interactive 3D Gaussians, such as Figure 3 As shown. Figure 3 (a), in a static scene with only camera motion, Corollary 1 degenerates into pure scene flow, leading to zero-dynamic Gaussian flow. Figure 3 (b), On the contrary, for dynamic scenes with interactive objects, the dynamic Gaussian flow map will highlight the interactive 3D Gaussians.

[0059] S2, dynamic Gaussian clustering and tracking: The interactive object areas are located through flow decomposition and clustering algorithms, and the movement trajectory of the Gaussian ball in each area is tracked.

[0060] Based on the above analysis, the dynamic Gaussian flow constraint provides a continuous Gaussian constraint and reveals the location of the interaction area. The changing topological structure of these interaction areas in dynamic scenes is reflected in the changing Gaussian. In order to overcome the heavy reliance of existing methods on mask annotations, the present invention uses dynamic Gaussian flow to explore the dynamic Gaussian of the interaction object and extract its trajectory for joint training.

[0061] Using the formula in Corollary 1, a set of camera-induced scene flows are used to pre-train a deformable 3D Gaussian G ′ , extract dynamic Gaussian flow frame by frame The flow map is then binarized to obtain a flow map, and the dynamic 3D Gaussians are identified by back-projecting the flow map, and Gaussians with sharp dynamics are highlighted. like Figure 1 As shown, where g i represents the Gaussian ball in the scene, and Q represents the number of Gaussian balls. Then, the unsupervised clustering algorithm DBSCAN is used to cluster the dynamic Gaussian into multiple clusters. Where K is the number of interacting objects, c i Represents the cluster center; the cluster center evolves over time, generating a continuous trajectory where k indicates the object the trajectory belongs to.

[0062] S3, self-guided control: The movement direction and speed of the Gaussian ball are modeled by three-dimensional spherical vectors to achieve reconstruction and control of the interactive scene. The three-dimensional spherical vectors are directly obtained from the dynamic Gaussian tracking trajectory.

[0063] Traditional methods of describing object state changes using 1D state variables are limited by their reliance on prior knowledge or Gaussian trajectory fitting and their inability to accurately capture dynamic changes. The present invention overcomes these limitations by representing Gaussian states with 3D spherical vectors that can be directly obtained from dynamic Gaussian tracking trajectories. This technique eliminates the requirements for control signals and curve fitting while increasing control flexibility.

[0064] In this embodiment, the Gaussian G and the network Θ that predicts the dynamic Gaussian offset are simultaneously optimized using a rasterization-based loss function, which measures the difference between the rendered image and the input image and the dynamic Gaussian flow.

[0065] During the training phase, cluster trajectory coordinates are used represents the Gaussian dynamical state and is related to the Gaussian center X i connect; use encode the coordinates and jointly train the network Θ to recover the Gaussian dynamics < ΔX i ,Δ∑ i >:

[0066]

[0067] Afterwards, splat rasterization is performed, combining the Gaussian with the predicted dynamic trajectory.

[0068] In the control phase, the interactive 3D vector v′ is input c , through f Θ (X i ,v′ c ) retrieves the Gaussian dynamics from the optimized network.

[0069] The technology proposed in this invention has been extensively verified and evaluated on CoNeRF-Synthetic, CoNeRF-Controllable, OmniSim and InterReal, and has achieved significant improvements in new view synthesis and operating efficiency.

[0070] Table 1 shows the results of various methods on the CoNeRF synthetic and controllable datasets. It can be seen that the performance of the present invention ranks among the best in the synthetic scene and achieves the best PSNR on the controllable dataset.

[0071] Table 1 Quantitative results on CoNeRF synthetic and controlled datasets

[0072]

[0073] Table 2 shows the quantitative results of various methods on the OmniSim dataset, indicating that the proposed method surpasses the previous methods in most indicators and achieves the highest average score in both the medium difficulty subset and the entire dataset.

[0074] Table 2 Quantitative results on the OmniSim dataset

[0075]

[0076]

[0077] Table 3 Quantitative results on the InterReal dataset

[0078]

[0079] Table 3 shows the quantitative results of various methods on the InterReal dataset, indicating that the proposed method consistently outperforms other methods in various settings and achieves the highest SSIM scores in all scenarios.

[0080] Table 4. Performance comparison of models in terms of size and speed

[0081] method Batch size Ray Samples Frame rate Parameter size (MB) Video Memory(GB) CoNeRF 1024 256 0.22 149.58 71.93 MK-Planes 4096 48 2.07 154.19 12.48 MK-Planes* 4096 48 0.61 152.35 11.90 LiveScene 4096 48 0.62 144.80 8.24 CoGS 1 - 215.93 189.70 25.50 The present invention 1 - 123.88 49.84 5.43

[0082] Table 4 shows the performance of various methods in terms of size and speed. This embodiment shows the comparison of model performance in terms of the number of parameters, rendering speed, and runtime memory. It can be seen that the present invention has great advantages.

[0083] Figure 4 The results of view synthesis visualization of various methods on the OmniSim dataset are shown. Figure 4 It can be seen that our method achieves more realistic and detailed rendering quality compared to other methods, while other methods suffer from ghosting. Figure 5 The synthetic visualization results of various methods on the InterReal dataset are shown, and the present invention is compared with the SOTA method on RGB rendering in real scenes. Figure 5 It can be seen that the present invention has a more detailed and accurate representation of the object, while other methods cannot capture the shape of the object and cause severe artifacts.

[0084] In summary, the present invention has significant advantages in rendering quality, operating efficiency and practical usage conditions, and can provide certain technical support for interactive reconstruction in real scenes.

[0085] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0086] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0087] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 in any other appropriate manner (e.g., by means of firmware).

[0088] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A Gaussian splashing method without labeling guidance, characterized in that: The following steps are involved: S1, dynamic Gaussian flow decomposition: obtain a set of video streams in the interactive scene and capture the motion trajectory of the Gaussian ball by reconstructing the 4D Gaussian field; S2, dynamic Gaussian clustering and tracking: locate the interactive object area through flow decomposition and clustering algorithm, and track the movement trajectory of the Gaussian ball in each area; S3, self-guided control: The movement direction and speed of the Gaussian ball are modeled by three-dimensional spherical vectors to achieve reconstruction and control of the interactive scene. The three-dimensional spherical vectors are directly obtained from the dynamic Gaussian tracking trajectory.

2. A Gaussian splashing method without labeling guidance according to claim 1, characterized in that: In S1, the dynamic Gaussian flow under instantaneous motion is analytically separated from the optical flow and camera motion by differential analysis of alpha synthesis.

3. A Gaussian splashing method without labeling guidance according to claim 2, characterized in that: Consider a dynamic scene with interactive objects. The camera and 3D Gaussian maintain different speeds in consecutive frames 0 and t. Assume that the dynamic 3D Gaussian G i The speed is v G , which is projected as image measurement g with translation velocity v and rotation velocity ω under constant camera instantaneous motion i , at time t, follows a two-dimensional Gaussian distribution g i The pixel m i,t The optical flow u caused by (v, ω) is expressed as: Among them, u Cam represents the camera-induced scene flow, represents the dynamic Gaussian flow on the image plane, Δ represents the flow residual term, and f x ,f y ,c x ,c y is the camera internal parameter, M means the Gaussian depth Z i The number of sorted Gaussian projections that intersect pixel m, T i is the transmittance, α i is the opacity learned by the Gaussian model, pixel m i,t satisfy μ i,t represents the two-dimensional average value, ∑ i i ,t represents the covariance, B i,t Represents the decomposition of the covariance matrix, dynamic Gaussian flow Use the 2D Gaussian mean displacement μ i,t -μ i,0 To accumulate.

4. A Gaussian splatting method without labeling guidance according to claim 3, characterized in that: In a static Gaussian scene, the optical flow caused by a pixel is expressed as u = u Cam .

5. The Gaussian splatting method without labeling guidance according to claim 3, characterized in that: Specifically, S2 is: using a set of camera-induced scene flows to pre-train a deformable 3D Gaussian G ′ , extract dynamic Gaussian flow frame by frame The flow map is then binarized to obtain a flow map, and the dynamic 3D Gaussians are identified by back-projecting the flow map, and Gaussians with sharp dynamics are highlighted. Among them, g i represents the Gaussian ball in the scene, Q represents the number of Gaussian balls; the unsupervised clustering algorithm DBSCAN is used to cluster the dynamic Gaussian balls into multiple clusters. Where K is the number of interacting objects, c i Represents the cluster center; the cluster center evolves over time, generating a continuous trajectory where k indicates the object the trajectory belongs to.

6. A Gaussian splashing method without labeling guidance according to claim 5, characterized in that: In S3, the Gaussian G and the network Θ for predicting the dynamic Gaussian offset are simultaneously optimized using a rasterization-based loss function, which measures the difference between the rendered image and the input image and the dynamic Gaussian flow.

7. A Gaussian splashing method without labeling guidance according to claim 6, characterized in that: During the training phase, cluster trajectory coordinates are used represents the Gaussian dynamical state and is related to the Gaussian center X i connect; use encode the coordinates and jointly train the network Θ to recover the Gaussian dynamics < ΔX i ,Δ∑ i >: Afterwards, splat rasterization is performed, combining the Gaussian with the predicted dynamic trajectory.

8. A Gaussian splatting method without labeling guidance according to claim 7, characterized in that: In the control phase, the interactive 3D vector v′ is input c , through f Θ (X i ,v′ c ) retrieves the Gaussian dynamics from the optimized network.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images