A dynamic scene reconstruction method based on markov chain and gaussian splatting

Abstract

The application provides a dynamic scene reconstruction method based on Markov chain and Gaussian splash, comprising the following steps: S1. Markov point initialization; S2. joint encoding of position and rotation; S3. construction of a historical state sliding window; S4. Markov chain deformation prediction: using a multi-head time attention mechanism to adaptively weight the historical state sequence, then integrating into a unified feature representation through an output projection matrix, and then inputting the attention output into a Markov predictor to output displacement offset and rotation offset; S5. noise injection in the training phase; S6. double-channel weighted fusion. The method models the motion process of three-dimensional Gaussian in the dynamic scene as a state transition process of Markov chain with memory, predicts the deformation at the next moment by learning the historical motion state of limited length, and improves the timing consistency and robustness of motion prediction by combining a multi-head time attention mechanism and a noise injection strategy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a dynamic scene reconstruction method based on Markov chains and Gaussian splashing. Background Technology

[0002] High-quality 3D models and scenes are crucial and widely used in many fields today, such as virtual reality, game development, and video production. 3D Gaussian Splash (3DGS), proposed in 2023, is one of the most revolutionary innovations in the field of 3D reconstruction in recent times. Compared to the implicit representation of previous NeRF-based methods, 3DGS leverages the explicit representation of point cloud-based 3D reconstruction methods. Unlike NeRF, which requires querying neural networks to calculate scenes in real time, 3DGS boasts faster rendering speeds and shorter training times.

[0003] In the field of novel perspective synthesis for dynamic scenes, existing 3DGS-based methods mainly employ deformation field strategies to predict the position and rotation of 3D Gaussians. D-3DGS was the first to introduce deformation fields into the 3DGS framework, using a multilayer perceptron (MLP) with the position and time of the 3D Gaussian as input to directly predict the deformation of each Gaussian point. 4D-GS further introduces multi-resolution planar encoding, combining spatial coordinates and time pairwise into six sets of planar features, which are then processed by a lightweight MLP and a multi-head decoder to obtain deformation predictions. 4DRotorGS utilizes four-dimensional spinters to describe 3D spatial rotation and spatiotemporal rotation. However, all of the above methods apply deformation independently to each 3D Gaussian, ignoring the spatial correlation between Gaussian points, resulting in high computational costs.

[0004] To address the aforementioned issues, SC-GS and SP-GS introduce a sparse control point mechanism within the deformation field framework. SC-GS defines a set of sparse control points and uses K-Nearest Neighbors (KNN) to associate neighboring 3D Gaussian points with their corresponding control points. It then uses an MLP to predict the deformation of each control point and propagates the deformation to neighboring Gaussian points, achieving high-quality reconstruction under local rigid constraints. However, SC-GS and SP-GS only consider the spatial association between local 3D Gaussian points at the same time step, without modeling the temporal dependencies between adjacent time steps. This leads to a significant degradation in rendering quality when moving objects are occluded in dynamic scenes, specifically manifested as motion artifacts and temporal discontinuities when the occluded object reappears. Furthermore, existing methods generally suffer from insufficient temporal consistency in monocular dynamic scene reconstruction. When an object in the scene is temporarily occluded (i.e., the object is temporarily unobservable from the camera's perspective), the lack of photometric supervision signals for that time period makes it difficult for the deformation network to provide accurate motion predictions for the occluded area, easily leading to jumps and artifacts in the reconstruction results before and after occlusion. Existing dynamic scene reconstruction methods based on 3D Gaussian splashing only consider spatial information at a single moment when modeling motion, failing to fully utilize the temporal context. This results in motion artifacts and temporal discontinuities when dealing with scenes involving occlusion and complex motion patterns, leading to poor rendering quality.

[0005] The challenges in addressing these issues and shortcomings are as follows: First, the trajectory distribution of moving objects in dynamic scenes is complex, exhibiting both smooth continuity (such as physically reasonable acceleration changes) and high nonlinearity (such as interactions between objects and non-rigid deformation). Modeling both characteristics simultaneously within a unified framework presents a significant challenge. Second, under monocular video input conditions, photometric supervision signals are extremely sparse, and occluded areas lack observation during the occlusion period. The network needs to provide reasonable motion predictions even during periods without supervision signals, which places high demands on the model's generalization ability. Third, there is an inherent coupling relationship between positional and rotational changes in 3D Gaussians. Modeling them independently would result in the loss of crucial motion correlation information, while joint modeling requires simultaneously representing two heterogeneous geometric quantities in a unified latent space. Fourth, the effective utilization of temporal historical information requires an attention mechanism to adaptively identify the contribution weights of different historical moments within a finite sliding window. Improper design can easily lead to the neglect of long-term historical information or the introduction of irrelevant noise.

[0006] The significance of addressing the above problems and shortcomings is as follows: At the academic level, this invention is the first to introduce Markov chains with memory into a dynamic scene reconstruction framework based on 3D Gaussian splashing, providing a new theoretical perspective for temporal consistency modeling and expanding the application scope of Markov processes in the field of 3D visual representation. At the technical level, the dual-channel weighted fusion architecture and adaptive noise injection strategy proposed in this invention provide an effective solution for motion prediction under occlusion conditions, and have certain methodological reference value for the field of new perspective synthesis of dynamic scenes. Summary of the Invention

[0007] This invention provides a dynamic scene reconstruction method based on Markov chains and Gaussian splashing. The motion process of a 3D Gaussian in a dynamic scene is modeled as a state transition process of a Markov chain with memory. By learning the historical motion states of a finite length, the deformation at the next moment is predicted. Combined with a multi-head temporal attention mechanism and a noise injection strategy, the temporal consistency and robustness of motion prediction are improved.

[0008] The technical solution of the present invention is as follows: The dynamic scene reconstruction method based on Markov chains and Gaussian splashing of the present invention includes the following steps: S1. Initializing Markov points: Randomly sampling a set of sparse points from the normalized space as a Markov point set, and establishing a correlation between each Markov point and its neighboring 3D Gaussian points to drive the deformation of the neighboring Gaussian points; S2. Jointly encoding position and rotation: For each Markov point at time... Location and rotation quaternions Using a unified encoder, it is jointly mapped to Hidden state encoding at any time This captures the intrinsic relationship between position and rotation changes; S3. Construct a history state sliding window: maintain a window of length... A sliding window to store past The hidden states at each time step are encoded to form a historical state sequence, which serves as the state space of the Markov chain with memory; S4. Markov chain deformation prediction: The historical state sequence is adaptively weighted using a multi-head temporal attention mechanism. The output projection matrix is ​​then integrated into a unified feature representation. The attention output is then fed into the Markov predictor, and the output displacement and rotation offsets are used as the deformation outputs of the Markov chain deformation prediction. S5. Training phase noise injection: Controlled Gaussian noise is introduced into the deformation outputs of the Markov chain deformation prediction to obtain the displacement and rotation offsets predicted by the Markov deformation network with noise. S6. Dual-channel weighted fusion: The displacement and rotation offsets predicted by the Markov deformation network with noise obtained in step S5 are weighted and fused with the output of the direct prediction branch of the multilayer perceptron (MLP) to obtain the final displacement and rotation deformations. The final displacement and rotation deformations are applied to the state update of the Markov points and propagated to the neighboring 3D Gaussians through K-nearest neighbor relationships. Finally, Gaussian splash rendering is performed to obtain the target image.

[0009] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S1, a set of sparse points is first randomly sampled from the normal space as a Markov point set, represented as: , in, The number of random samples, taken as a positive integer. The i-th Markov point at time i is represented as: , in, For at any time The The location of each Markov point In order to be in The first moment Rotational quaternions of Markov points For the first A scaling vector of Markov points , , And the K-nearest neighbor algorithm is used to establish a relationship between each Markov point and its neighboring 3D Gaussian points.

[0010] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S2, for a Markov point of time Using a unified encoder, a two-layer multilayer perceptron (MLP) is applied to it. Location at any moment and rotation quaternions Joint encoding is dimensional latent vector ,Right now Hidden state encoding at any given time: , in, This represents a learnable mapping function consisting of multiple linear transformation layers, layer normalization layers, and nonlinear activation functions. For at any time The The location of each Markov point In order to be in The first moment Rotational quaternions of Markov points.

[0011] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S3, the historical state sequence is represented as... , ,in for Hidden state encoding at any time. In a Markov chain with memory, the conditional probability distribution of future states depends on the past. There are several states, namely: = ， when It degenerates into a traditional memoryless Markov chain.

[0012] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S4, for Hidden state encoding at any time The calculation method for multi-head attention is as follows: , Attention weight Attention is calculated by scaling the dot product: , Where K is the number of attention heads, and each attention head has an independent transformation matrix. , , , To output the projection matrix, the distributed outputs of each attention head are integrated into a unified feature representation. This represents the normalized exponential function; For each attention head, the feature dimensions, Feed the attention output into the Markov predictor Historical information processed by time attention is converted into deformation prediction: , In the formula, This indicates the multilayer perceptron operation corresponding to the Markov predictor, which consists of three fully connected layers cascaded with ReLU activation functions. The subscript... This indicates that it can learn network parameters. This represents a Markov deformation prediction function based on a historical state sequence. It is a historical state sequence. and These are the displacement and rotation offsets predicted by the Markov deformation network, respectively.

[0013] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S5, additive Gaussian noise is added to the displacement offset: , For the rotational offset, noise is injected using the spherical linear interpolation SLERP method: , SLERP Represents the spherical linear interpolation operation for quaternions. and For noise weights, It is Gaussian noise, and its variance is... The annealing schedule gradually decreases during training. The displacement shift predicted by a Markov deformation network that introduces noise. The rotational offset predicted by a Markov deformation network that introduces noise. This represents the result after applying a Gaussian perturbation to the rotation offset and normalizing it to a unit quaternion.

[0014] Optionally, in the above dynamic scene reconstruction method based on Markov chains and Gaussian splashing, in step S6, for the direct prediction branch of the multilayer perceptron (MLP), position encoding (PE) is used to encode the position, rotation quaternion, and time of the Markov points to capture the geometric structure and static mapping relationships in the scene. Then, the encoded features are fed into a multilayer perceptron containing eight fully connected layers. Directly predict displacement and rotation deformation: , , in, This indicates the operation corresponding to the multilayer perceptron of the direct prediction branch, which consists of 8 cascaded fully connected layers, each containing 256 neurons. The subscript... This indicates that it can learn network parameters. This represents a sinusoidal positional coding function, used to map input to high-dimensional frequency features. , , They represent respectively to The position of the Markov point at time Rotation Quaternions as well as The features obtained by performing the position encoding at each time step. Then, through adaptive weight coefficients and The outputs of the two branches are weighted and fused to obtain the final displacement and rotation deformations for each Markov point: , in and During training, the value is gradually increased from a small value to achieve a progressive training strategy. This allows the model to first learn the basic scene geometry and then gradually incorporate temporal information to optimize motion dynamics; where... Represents the spherical linear interpolation operation for quaternions, with The interpolation coefficients are the rotational deformations of the output of the direct prediction branch. Rotational offset predicted by Markov deformation network Between them, interpolation is performed along the shortest path on the quaternion unit hypersphere at a constant angular velocity, thereby achieving a weighted fusion of the two-branch rotation predictions; Apply the final deformation to the state update of the Markov point: , , in This represents quaternion multiplication, used for composition by rotation.

[0015] The beneficial effects of the technical solution of the present invention are as follows: This invention achieves higher-quality dynamic scene reconstruction by encoding the attributes of 3D Gaussian points as states of Markov chains. By modeling the motion process of 3D Gaussian points as state transitions of Markov chains with memory, and using finite historical states to predict future deformations, this invention effectively captures the temporal dependencies of motion, significantly improving the temporal consistency problem in dynamic scene reconstruction and eliminating motion artifacts when occluded objects reappear. An adaptive weighting of historical states through a multi-head temporal attention mechanism allows different attention heads to focus on different aspects of temporal relationships, enhancing the modeling ability for complex motion patterns. By introducing adaptive noise injection and employing an annealing strategy during the training phase, observation uncertainties in real-world scenarios such as occlusion are simulated, acting as a regularization mechanism and improving the model's robustness and generalization ability. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0017] Figure 1 This is a flowchart of the dynamic scene reconstruction method based on Markov chain and Gaussian splashing of the present invention; Figure 2 This is a comparison chart of the performance of the present invention and existing methods on the D-NeRF dataset; Figure 3 This is a comparison chart of the performance of the present invention and existing methods on the HyperNeRF dataset. Figure 4 This is a comparison image of the reconstruction details of the present invention and existing methods in a 3D printer scenario. Detailed Implementation

[0018] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the scope of the invention.

[0019] The dynamic scene reconstruction method based on Markov chains and Gaussian splashing of this invention utilizes Markov chains with memory to perform temporal modeling of the motion process of control points in 3D Gaussian splashing, combines multi-head temporal attention mechanism to adaptively weight historical states, and improves the temporal consistency and detail quality of dynamic scene reconstruction through a dual-channel weighted fusion architecture of Markov chain prediction and multilayer perceptron (MLP) direct prediction. This achieves high-fidelity reconstruction of complex motion patterns (including occluded scenes) in dynamic scenes, and realizes real-time rendering speed while maintaining high rendering quality, thus improving the temporal consistency and detail quality of dynamic scene reconstruction.

[0020] The principles of this invention are as follows: 1) By modeling the motion process of a 3D Gaussian as a state transition of a Markov chain with memory, future deformations are predicted using finite historical states, effectively capturing the temporal dependencies of motion. 2) Adaptive weighting of historical states is achieved through a multi-head temporal attention mechanism, allowing different attention heads to focus on different aspects of the temporal relationship, thus enhancing the ability to model complex motion patterns. 3) Adaptive noise injection and annealing strategies are introduced during the training phase to simulate observation uncertainties in real-world scenarios such as occlusion. 4) A dual-channel weighted fusion architecture combining Markov chain prediction and multilayer perceptron direct prediction balances the advantages of both temporal dependency modeling and static geometric mapping learning. 5) A strategy of using Markov points (sparse control points) to drive neighboring Gaussian points significantly reduces computational complexity while maintaining high reconstruction quality, compared to independent deformation point-by-point Gaussian, enabling real-time rendering.

[0021] like Figure 1 As shown, the dynamic scene reconstruction method based on Markov chains and Gaussian splashing of the present invention includes the following steps from start to finish: S1. Initialize Markov points First, a set of sparse points is randomly sampled from the normal space to form the Markov point set, denoted as: , in, The number of random samples, taken as a positive integer.

[0022] Each Markov point is itself a three-dimensional Gaussian with properties such as position, rotation quaternion, scaling, opacity, and spherical harmonic coefficients, among which: The first moment A Markov point is represented as: , in, For at any time The The location of each Markov point In order to be in The first moment Rotational quaternions of Markov points For the first A scaling vector of Markov points , Used for color rendering.

[0023] The K-nearest neighbor algorithm is used to establish a relationship between each Markov point and its neighboring 3D Gaussian points, thereby driving the deformation of the neighboring Gaussian points.

[0024] S2. Joint Encoding Position and Rotation: For each Markov point at time... Location and rotation quaternions Using a unified encoder, it is jointly mapped to dimensional latent vector ,Right now Hidden state encoding at any time This allows us to capture the intrinsic relationship between positional and rotational changes.

[0025] Specifically, for a Markov point of time This invention uses a unified encoder (comprising a two-layer multilayer perceptron (MLP) with layer normalization and ReLU activation, whose corresponding nonlinear mapping function is denoted as... , and put it in Location at any moment and rotation quaternions Joint encoding is dimensional latent vector, i.e. Hidden state encoding at any time : , in, This represents a learnable mapping function consisting of multiple linear transformation layers, layer normalization layers, and nonlinear activation functions, the parameters of which are optimized through the training process. This joint encoding enables the model of this invention to learn the interdependencies between position and rotational changes that might be ignored when treated individually.

[0026] S3. Construct a sliding window for historical states: Maintain a sliding window of length... A sliding window to store past The hidden state encoding at each time step forms a historical state sequence. As the state space of a Markov chain with memory, , In a Markov chain with memory, the conditional probability distribution of future states depends on the past. There are several states, namely: = when It degenerates into a traditional memoryless Markov chain. This invention utilizes a sliding window length... Implement memory order A Markov chain with memory allows for a sequence of historical states. Simultaneously capture temporal dynamics of position and rotation in a unified latent representation.

[0027] S4. Markov Chain Deformation Prediction: This method utilizes a multi-head temporal attention mechanism to adaptively weight the historical state sequence, resulting in... Each attention head calculates its own attention weight and aggregates the historical encodings using a weighted average. The weighted average weights are then integrated into a unified feature representation via the output projection matrix. The attention output is fed into a Markov predictor consisting of three fully connected layers and ReLU activation. The output displacement and rotation offsets are used as the deformation outputs for Markov chain deformation prediction.

[0028] The present invention outputs the historical state sequence through step S3. As the state space of a Markov chain, predict the position offset and rotation offset of Markov points.

[0029] Specifically, firstly, a multi-head temporal attention mechanism is used to adaptively weight the historical state sequence for the current state. Hidden state encoding at any time The calculation method for multi-head attention is as follows: , Attention weight Attention is calculated by scaling the dot product: , Where K is the number of attention heads, and each attention head has an independent transformation matrix. , , , To output the projection matrix, the distributed outputs of each attention head are integrated into a unified feature representation. This represents the normalized exponential function; Each attention head has a feature dimension. Different attention heads can focus on different aspects of temporal relationships and learn how to optimally combine different temporal feature patterns.

[0030] Then, the attention output is fed into the Markov predictor. The predictor consists of three fully connected layers and a ReLU activation function, which converts historical information processed by temporal attention into deformation predictions. , In the formula, This represents the multilayer perceptron operation corresponding to the Markov predictor. The Markov predictor consists of three fully connected layers cascaded with ReLU activation functions. The subscript... This indicates its learnable network parameters; the Markov predictor is compared with the multilayer perceptron used in step S3 for encoding position and rotation, and the multilayer perceptron used for directly predicting deformation. For different networks with independent parameters, This represents a Markov deformation prediction function based on a historical state sequence, used to predict the future deformation parameters of the Gaussian point at the current moment based on multiple historical state information prior to the current moment. The output is a sequence of historical states. and These are the displacement and rotation offsets predicted by the Markov deformation network, respectively.

[0031] This step is the core inventive point of this invention: Markov chain prediction branches are adept at capturing temporal dependencies. When there are occlusions or incomplete observations in the scene, the Markov chain with memory can provide more reliable deformation predictions based on historical motion states. S5. Noise Injection During Training: Controlled Gaussian noise is introduced into the deformation output of the Markov chain deformation prediction, resulting in Markov chain prediction branches (the displacement offset predicted by the noisy Markov deformation network and the rotation offset predicted by the noisy Markov deformation network). Additive Gaussian noise is added to the displacement offset, and noise is injected into the rotation offset using spherical linear interpolation. The noise variance is gradually reduced during training using annealing scheduling.

[0032] Specifically, during the training phase, controlled Gaussian noise is introduced into the deformation output of the Markov chain deformation prediction to enhance the model's robustness to occlusion and incomplete observations.

[0033] For the displacement, add additive Gaussian noise: , For the rotational offset, noise is injected using spherical linear interpolation (SLERP). , SLERP Represents the spherical linear interpolation operation for quaternions. and As a noise weight, a larger value is taken in areas that may be obscured (determined through visibility analysis). It is Gaussian noise, and its variance is... The annealing schedule gradually decreases during training. The displacement shift predicted by a Markov deformation network that introduces noise. The rotational offset predicted by a Markov deformation network that introduces noise. This represents the result after applying a Gaussian perturbation to the rotation offset and normalizing it to a unit quaternion.

[0034] This invention employs SLERP to interpolate the rotational quaternions, enabling a smooth transition along the shortest path on the quaternion hypersphere. This avoids the non-uniform velocity problem caused by linear interpolation and improves the robustness of the rotational dynamic representation. Noise injection serves two purposes: 1) to simulate the observation uncertainty under occlusion conditions in real-world scenarios; and 2) to act as a regularization method to prevent the model from overfitting to specific motion patterns and to improve generalization ability.

[0035] S6. Dual-channel weighted fusion: Using adaptive weight coefficients, the displacement and rotation offsets predicted by the noisy Markov deformation network obtained in step S5 are weighted and fused with the output of the MLP direct prediction branch to obtain the final displacement and rotation deformation variables. These final deformation variables are applied to the state update of the Markov points and propagated to neighboring 3D Gaussians through KNN relationships. Finally, Gaussian splash rendering is performed to obtain the target image. The weight coefficients are gradually increased from small values ​​during training to achieve a progressive training strategy.

[0036] This invention designs a dual-channel weighted deformation prediction structure, and the final deformation prediction is jointly completed by the Markov chain prediction branch and the MLP direct prediction branch.

[0037] For the direct prediction branch of MLP, positional encoding (PE) is used to encode the position and time of Markov points separately to capture the geometric structure and static mapping relationships in the scene. The encoded features are then fed into a multilayer perceptron containing 8 fully connected layers (256 neurons per layer). Its corresponding emission function is denoted as Directly predict displacement and rotation deformation: , , This indicates the operation corresponding to the direct prediction branch of the multilayer perceptron, which consists of 8 cascaded fully connected layers, each containing 256 neurons. The subscript... This indicates that it can learn network parameters, similar to a Markov predictor. These are different networks with independent parameters. This represents a sinusoidal positional coding function, used to map input to high-dimensional frequency features. , , They represent respectively to The position of the Markov point at time Rotation Quaternions and time Features obtained by position encoding.

[0038] Then, through adaptive weight coefficients and The outputs of the two branches are weighted and fused to obtain the final displacement and rotation deformations for each Markov point: , , in and During training, the value is gradually increased from a small value to achieve a progressive training strategy. This allows the model to first learn the basic scene geometry and then gradually incorporate temporal information to optimize motion dynamics; where... Represents the spherical linear interpolation operation of quaternions, that is, using The interpolation coefficients are the rotational deformations of the output of the direct prediction branch. Rotational offset predicted by Markov deformation network Between them, interpolation is performed along the shortest path on the quaternion unit hypersphere at a constant angular velocity, thereby achieving a weighted fusion of the rotation predictions of the two branches.

[0039] Apply the final deformation to the state update of the Markov point: , , in This represents quaternion multiplication used for composition by rotation. The updated Markov point deformation is propagated to neighboring 3D Gaussians through K-nearest neighbor relationships, and finally, 3D Gaussian splash rendering is performed to obtain the target view image.

[0040] This invention uses a joint loss function for end-to-end optimization: , in For pixel-level L1 loss, The two are used together to measure rendering quality, with structural similarity loss as the primary metric. The local rigidity regularization loss is used to maintain the local rigidity constraint between neighboring Gaussians. The loss is due to physical constraints and includes an acceleration penalty term. and smoothing terms : , The acceleration penalty term suppresses unreasonable acceleration changes: , The smoothing term encourages smooth changes in position and rotation over time. , in This is a time gradient operator used to calculate the rate of change of physical quantities with respect to time. Physical constraint loss is particularly important for generating plausible motion in regions with limited observations or occlusion, compensating for the shortcomings of relying solely on photometric supervision signals.

[0041] This invention implements a dynamic scene reconstruction method based on Markov chains and Gaussian splashing. By modeling the motion process of a 3D Gaussian object as a state transition of a Markov chain with memory, and using finite historical states to predict future deformations, it effectively captures the temporal dependencies of motion, significantly improving the temporal consistency problem in dynamic scene reconstruction and eliminating motion artifacts when occluded objects reappear. An adaptive weighting of historical states is applied through a multi-head temporal attention mechanism, allowing different attention heads to focus on different aspects of the temporal relationship, enhancing the modeling ability for complex motion patterns. By introducing adaptive noise injection and employing an annealing strategy during the training phase, observation uncertainties in real-world scenes such as occlusion are simulated, acting as a regularization mechanism and improving the model's robustness and generalization ability.

[0042] like Figure 2 As shown, the visual effects of this invention are compared with existing dynamic 3D reconstruction methods. During the movement of dynamic objects, traditional methods, lacking effective modeling of historical motion states, are prone to problems such as 3D Gaussian point drift, edge blurring, and local structural discontinuities. For example, in fast-moving and deformed regions, some methods produce reconstruction results with texture ghosting, boundary misalignment, and missing details. In contrast, this invention constructs a Markov deformation prediction module based on historical state sequences, using position and rotation state information within a continuous time window to predict the future deformation of Gaussian points, thus maintaining the temporal continuity of dynamic Gaussian points. Therefore, as... Figure 2 As shown, the motion region in the reconstruction result of this invention has a clearer contour boundary, reduces the Gaussian distribution misalignment caused by motion estimation error, and can maintain the continuity and stability of the object surface texture.

[0043] The above results show that the present invention effectively improves the motion consistency of the 3D Gaussian representation in dynamic scenes and reduces the generation of artifacts in the dynamic reconstruction process by jointly modeling position and rotation changes and combining historical state information for deformation prediction.

[0044] like Figure 3 As shown, to further verify the reconstruction capability of this invention in real dynamic environments, it was tested on the HyperNeRF dataset. Compared to synthetic datasets, real dynamic scenes typically contain more complex factors, such as non-rigid motion, local occlusion, and texture variations, thus placing higher demands on dynamic scene modeling capabilities. Figure 3The visual results show that existing methods are prone to problems such as local structural blurring, insufficient texture restoration, and edge distortion in complex motion areas, while the present invention can more accurately restore the detailed information in the scene. For example, in the area of ​​human movement, the present invention can maintain the integrity of the human hand contour structure; in the area of ​​fine structure, the present invention can reduce local blurring and deformation errors caused by time changes.

[0045] The 3D printer scene in the HyperNeRF dataset has continuously moving occluders, making it particularly suitable for verifying the ability of the method of this invention to handle temporal occlusion, such as... Figure 4 As shown, the present invention clearly reconstructs the graffiti on the wall sticker and the connection line of the printer.

[0046] To quantitatively evaluate the dynamic 3D scene reconstruction effect of this invention, for the synthetic dataset D-NeRF, the following metrics are used: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). For the real-world HyperNeRF dataset, PSNR, Multi-Scale SSIM (MS-SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) are used as evaluation metrics. PSNR measures the pixel-level error between the reconstructed image and the real image; a higher value indicates lower distortion. SSIM and MS-SSIM evaluate the consistency of structural information between the reconstructed result and the real image; higher values ​​indicate stronger structural preservation. LPIPS measures perceptual differences through the feature space of the deep network; lower values ​​indicate smaller differences in human visual perception between the reconstructed result and the real image.

[0047] The quantitative results for the D-NeRF and HyperNeRF datasets are shown in Tables 1 and 2, where the best values ​​are marked in bold and the second-best values ​​are marked with underlines.

[0048] As shown in Table 1, on the D-NeRF dataset, this invention achieved optimal results in all three evaluation metrics: PSNR, SSIM, and LPIPS. Specifically, the PSNR of this invention reached 42.74, an improvement of 2.44 dB compared to the existing dynamic scene reconstruction method D-3DGS (40.30) and 4.11 dB compared to SC-GS (38.63), indicating that this invention achieves lower pixel reconstruction errors and higher image fidelity. Simultaneously, the SSIM of this invention reached 0.998, higher than other comparative methods, indicating that this invention can more accurately preserve structural information in dynamic scenes. Furthermore, the LPIPS of this invention reached 0.0074, the lowest among all comparative methods, a reduction of approximately 36% compared to D-3DGS (0.0116), indicating that the new perspective images generated by this invention have better consistency in visual perception quality. These results demonstrate that this invention, through a Markov deformation prediction mechanism based on historical state modeling, can more accurately describe the motion law of dynamic Gaussian points changing over time, thereby improving the quality of dynamic scene reconstruction.

[0049] As shown in Table 2, on the real-world HyperNeRF dataset, this invention still achieves superior reconstruction results compared to existing methods. Specifically, the PSNR of this invention reaches 25.76, an improvement of 0.54 dB compared to SP-GS's 25.22 and 0.74 dB compared to 4D-GS, indicating that this invention can obtain more accurate color and geometry restoration results in complex and realistic dynamic scenes. Furthermore, the SSIM of this invention reaches 0.852, the highest among all compared methods, indicating that this invention can better maintain structural consistency in real-world scenes. Although some methods have similar results on the LPIPS metric, this invention achieves both the highest PSNR and the highest SSIM, demonstrating its comprehensive advantages in pixel accuracy and structure preservation.

[0050] To verify the effectiveness of each key module in this invention, an ablation experiment was further conducted to remove the time attention mechanism and the noise injection module, and the results were compared with the complete model. The results are shown in Table 3.

[0051] First, comparing the results of the "no attention" model and the complete model reveals that after removing the temporal attention mechanism, the PSNR decreased from 42.74 to 40.45, a reduction of 2.29 dB. This result indicates that the temporal attention mechanism can adaptively weight the model according to the importance of different time steps within the historical state window, enabling the model to more effectively utilize historical motion information and improve the accuracy of dynamic deformation prediction.

[0052] Secondly, comparing the results of the "noise-free" and complete models reveals that after removing noise injection during the training phase, the PSNR decreased from 42.74 to 41.59, a reduction of 1.15 dB. Simultaneously, the number of rendered frames decreased from 174.75 to 170, indicating that noise injection enhances the model's adaptability to dynamic changes and incomplete observations, thereby improving its robustness.

[0053] Finally, the complete model achieved the highest PSNR (42.74) and a relatively high rendering speed (174.75 FPS) while maintaining high training efficiency, showing significant advantages over both SP-GS and SC-GS. Specifically, compared to SP-GS, this invention improved PSNR by 2.21 dB, while reducing training time from 32:53 to 15:54, demonstrating that this invention improved reconstruction quality while reducing computational overhead.

[0054] In summary, the ablation experiment results demonstrate that the temporal attention mechanism can effectively enhance the modeling ability of historical motion information, and noise injection can improve the robustness of the model. The combined effect of the two enables the present invention to achieve higher quality and more stable dynamic scene reconstruction results.

[0055] Removing temporal attention results in a significant performance drop, with a 2.29 dB decrease in PSNR. This indicates that attention plays a crucial role in capturing temporal dependencies through adaptive weighting of historical states, thereby enhancing the model's understanding of motion patterns and maintaining temporal consistency in generated results. Removing noise injection leads to a slight performance decrease. This suggests that noise injection contributes to improved robustness and generalization ability. Noise acts as a regularizer during training, preventing the model from overfitting to the training data and thus improving performance.

[0056] Table 1: Comparison results of the present invention and existing models on the D-NeRF dataset. Table 2: Comparison results of the present invention and existing models on the HyperNeRF dataset Table 3: Ablation experimental results of the present invention and existing models on the D-NeRF dataset. This invention presents a dynamic scene reconstruction method based on Markov chains and Gaussian splashing. It utilizes a Markov chain with memory to temporally model the motion of control points in a 3D Gaussian splash, combines a multi-head temporal attention mechanism with adaptive weighted historical states, and employs a dual-channel weighted fusion architecture of Markov chain prediction and MLP direct prediction to achieve high-fidelity reconstruction of complex motion patterns (including occluded scenes) in dynamic scenes. It captures the inherent coupling relationship of deformation through joint encoding of position and rotation, uses adaptive noise injection as a training regularization method to enhance robustness, uses physical constraint loss to ensure motion rationality, and achieves real-time rendering speed while maintaining high rendering quality by driving neighboring 3D Gaussians through sparse Markov points, thus improving the temporal consistency and detail quality of dynamic scene reconstruction.

[0057] This invention presents a dynamic scene reconstruction method based on Markov chains and Gaussian splashing. By modeling the motion process of a 3D Gaussian object as a state transition of a Markov chain with memory, it predicts future deformation using finite historical states, effectively capturing the temporal dependencies of motion. This significantly improves the temporal consistency problem in dynamic scene reconstruction and eliminates motion artifacts when occluded objects reappear. An adaptive weighting of historical states is achieved through a multi-head temporal attention mechanism, allowing different attention heads to focus on different aspects of the temporal relationship, enhancing the modeling ability for complex motion patterns. Adaptive noise injection and annealing strategies are introduced during the training phase to simulate observation uncertainties in real-world scenarios such as occlusion, acting as a regularization mechanism and improving the model's robustness and generalization ability. The dual-channel weighted fusion architecture, combining Markov chain prediction and multilayer perceptron direct prediction, leverages the advantages of both temporal dependency modeling and static geometric mapping learning.

[0058] The above description represents the preferred embodiment based on the inventive concept and working principle. The above embodiments should not be construed as limiting the scope of protection of these claims; other embodiments and combinations of implementations based on the inventive concept are all within the scope of protection of this invention.

Claims

1. A dynamic scene reconstruction method based on Markov chains and Gaussian splashing, characterized in that, Includes the following steps: S1. Initialize Markov points: Randomly sample a set of sparse points from the normal space as a Markov point set, and establish a relationship between each Markov point and the neighboring 3D Gaussian points to drive the deformation of the neighboring Gaussian points; S2. Joint encoding position and rotation: For each Markov point at time... Location and rotation quaternions Using a unified encoder, it is jointly mapped to Hidden state encoding at any time This allows us to capture the intrinsic relationship between positional and rotational changes; S3. Construct a sliding window for historical states: Maintain a sliding window of length... A sliding window to store past The hidden state encoding of each time step forms a historical state sequence, which serves as the state space of the Markov chain with memory. S4. Markov chain deformation prediction: The historical state sequence is adaptively weighted using a multi-head temporal attention mechanism, and then integrated into a unified feature representation by the output projection matrix. The attention output is then fed into the Markov predictor, and the output displacement offset and rotation offset are used as the deformation output of the Markov chain deformation prediction. S5. Training phase noise injection: Controlled Gaussian noise is introduced into the deformation output predicted by the Markov chain deformation to obtain the displacement offset and rotation offset predicted by the Markov deformation network with noise. as well as S6. Dual-channel weighted fusion: The displacement offset and rotation offset predicted by the noisy Markov deformation network obtained in step S5 are weighted and fused with the output of the direct prediction branch of the multilayer perceptron (MLP) to obtain the final displacement and rotation variables. The final displacement and rotation variables are applied to the state update of the Markov points and propagated to the neighboring 3D Gaussians through K-nearest neighbor relations. Finally, Gaussian splash rendering is performed to obtain the target image.

2. The dynamic scene reconstruction method based on Markov chains and Gaussian splashing according to claim 1, characterized in that, In step S1, a set of sparse points is first randomly sampled from the canonical space as a Markov point set, denoted as: ， in, The number of random samples, taken as a positive integer. The i-th Markov point at time i is represented as: ， in, For at any time The The location of each Markov point In order to be in The first moment Rotational quaternions of Markov points For the first A scaling vector of Markov points , , And the K-nearest neighbor algorithm is used to establish a relationship between each Markov point and its neighboring 3D Gaussian points.

3. The dynamic scene reconstruction method based on Markov chains and Gaussian splashing according to claim 1, characterized in that, In step S2, for one Markov point of time Using a unified encoder, a two-layer multilayer perceptron (MLP) is applied to it. Location at any moment and rotation quaternions Joint encoding is dimensional latent vector ,Right now Hidden state encoding at any given time: ， in, This represents a learnable mapping function consisting of multiple linear transformation layers, layer normalization layers, and nonlinear activation functions. For at any time The The location of each Markov point In order to be in The first moment Rotational quaternions of Markov points.

4. The dynamic scene reconstruction method based on Markov chains and Gaussian splashing according to claim 1, characterized in that, In step S3, the historical state sequence is represented as follows: , ,in for Hidden state encoding at any time. In a Markov chain with memory, the conditional probability distribution of future states depends on the past. There are several states, namely: = ， when It degenerates into a traditional memoryless Markov chain.

5. The dynamic scene reconstruction method based on Markov chains and Gaussian splashing according to claim 1, characterized in that, In step S4, for Hidden state encoding at any time The calculation method for multi-head attention is as follows: ， Attention weight Attention is calculated by scaling the dot product: ， Where K is the number of attention heads, and each attention head has an independent transformation matrix. , , , To output the projection matrix, the distributed outputs of each attention head are integrated into a unified feature representation. This represents the normalized exponential function; For each attention head, the feature dimensions, Feed the attention output into the Markov predictor The historical information processed by time attention is converted into deformation prediction: ， In the formula, This indicates the multilayer perceptron operation corresponding to the Markov predictor, which consists of three fully connected layers cascaded with ReLU activation functions. The subscript... This indicates that it can learn network parameters. This represents a Markov deformation prediction function based on a historical state sequence. It is a historical state sequence. and These are the displacement and rotation offsets predicted by the Markov deformation network, respectively.

6. The dynamic scene reconstruction method based on Markov chains and Gaussian splashing according to claim 1, characterized in that, In step S5, additive Gaussian noise is added to the displacement offset: ， For the rotational offset, noise is injected using the spherical linear interpolation SLERP method: ， SLERP Represents the spherical linear interpolation operation for quaternions. and For noise weights, It is Gaussian noise, and its variance is... The annealing schedule gradually decreases during training. The displacement shift predicted by a Markov deformation network that introduces noise. The rotational offset predicted by a Markov deformation network that introduces noise. This represents the result after applying a Gaussian perturbation to the rotation offset and normalizing it to a unit quaternion.

7. The dynamic scene reconstruction method based on Markov chain and Gaussian splashing according to claim 1, characterized in that, In step S6, for the direct prediction branch of the multilayer perceptron (MLP), position encoding (PE) is used to encode the position, rotation quaternion, and time of the Markov point to capture the geometric structure and static mapping relationships in the scene. Then, the encoded features are fed into the multilayer perceptron containing 8 fully connected layers. Directly predict displacement and rotation deformation: ， ， in, This indicates the operation corresponding to the multilayer perceptron of the direct prediction branch, which consists of 8 cascaded fully connected layers, each containing 256 neurons. The subscript... This indicates that it can learn network parameters. This represents a sinusoidal positional coding function, used to map input to high-dimensional frequency features. , , They represent respectively to The position of the Markov point at time Rotation Quaternions as well as The features obtained by performing the position encoding at each time step. Then, through adaptive weight coefficients and The outputs of the two branches are weighted and fused to obtain the final displacement and rotation deformations for each Markov point: ， in and During training, the value is gradually increased from a small value to achieve a progressive training strategy. This allows the model to first learn the basic scene geometry and then gradually incorporate temporal information to optimize motion dynamics; where... Represents the spherical linear interpolation operation for quaternions, with The interpolation coefficients are the rotational deformations of the output of the direct prediction branch. Rotational offset predicted by Markov deformation network Between them, interpolation is performed along the shortest path on the quaternion unit hypersphere at a constant angular velocity, thereby achieving a weighted fusion of the two-branch rotation predictions; Apply the final deformation to the state update of the Markov point: ， ， in This represents quaternion multiplication, used for composition by rotation.

Images