Dynamic scene repairing method based on four-dimensional Gaussian sputtering

Abstract

The invention discloses a dynamic scene repairing method based on four-dimensional Gaussian sputtering, which is characterized in that efficient dynamic scene space repairing is realized based on a repairing framework of a four-dimensional Gaussian sputtering monocular video reconstruction algorithm and a video repairing model, and by counting the time frame number of Gaussian primitives in a mask area and removing Gaussian points exceeding a threshold value, the dynamic scene space repairing efficiency is improved. According to the method, high-efficiency dynamic object removal is realized, and a mask weighting loss function is introduced to guide the repair area to converge to real texture information in a subsequent time frame, so that the repair quality is improved. The invention provides a dynamic scene repair method based on four-dimensional Gaussian sputtering, and provides a repair framework based on a four-dimensional Gaussian sputtering monocular video reconstruction algorithm and a video repair model for removing and repairing a target object in a dynamic scene to realize three-dimensional repair of the dynamic scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration processing, and in particular to a dynamic scene restoration method based on four-dimensional Gaussian sputtering. Background Art

[0002] To date, large 2D image restoration models combined with neural radiance fields or 3D Gaussian sputtering have achieved high-quality restoration of static scenes. Specifically, 3D Gaussian sputtering is an example. 3D Gaussian sputtering is a 3D reconstruction algorithm that reconstructs a 3D scene from 2D images of multiple views of the scene. It can render 2D images of any view.

[0003] Static scene restoration methods based on 3D Gaussian sputtering generally first use the initial training set to train the 3D Gaussian point cloud representation of the initial scene. Then, SAM is used to obtain a mask of the object to be removed. The mask is then used to erase the object information in the depth map rendered from the initial training set image and the 3D Gaussian point cloud representation of the initial scene, replacing it with a black background. The image without the target object is then input into a large 2D image restoration model based on a diffusion model to obtain the restored image and depth map. Afterwards, the Gaussian point cloud representation of the target object is removed from the 3D Gaussian point cloud representation of the initial scene. Finally, the restored 2D image and depth map are used to fine-tune the 3D Gaussian point cloud representation.

[0004] However, current approaches for dynamic scenes are limited to video inpainting, a technique designed to restore or fill in missing or damaged portions of a video. This technique can be applied to a variety of applications, such as object removal, visual restoration, and completion. With the advent of deep learning, visual inpainting models typically employ convolutional neural networks and generative adversarial networks. Recent research has also applied visual transformers to enhance global interactions between visual features. However, existing research primarily focuses on two-dimensional video inpainting frameworks, and an effective three-dimensional dynamic scene inpainting paradigm has yet to be established. This approach lacks the ability to synthesize new views with spatiotemporal consistency. Summary of the Invention

[0005] The purpose of the present invention is to provide a dynamic scene restoration method based on four-dimensional Gaussian sputtering. Aiming at the target object removal and restoration tasks in dynamic scenes, a restoration framework based on a four-dimensional Gaussian sputtering monocular video reconstruction algorithm and a video restoration model is proposed to achieve three-dimensional restoration of dynamic scenes.

[0006] The technical solution adopted by the dynamic scene restoration method based on four-dimensional Gaussian sputtering disclosed in the present invention is:

[0007] A dynamic scene restoration method based on four-dimensional Gaussian sputtering, comprising:

[0008] Reconstruct the initial scene. A monocular dynamic scene reconstruction algorithm is used to reconstruct the initial scene. The algorithm is based on the representation of point clouds and realizes dynamic scene modeling by decoupling 4DGS into 3DGS and deformation field.

[0009] Dynamic object removal: In dynamic scenes, if the object to be removed is in motion or the camera is moving, then when removing the object mask at each given time frame, only the object is always within the mask. As the object moves, the background area behind the object in the current time frame may be revealed in subsequent time frames after the object moves to other areas. In this way, the background area is only within the mask area for a period of time. By counting the number of time frames in which Gaussian primitives are within the mask area and removing Gaussian primitives that exceed the threshold, efficient dynamic object removal can be achieved.

[0010] Scene restoration and optimization: remove the target object in the monocular video and restore the video based on this to obtain the two-dimensional restoration information required for dynamic scene restoration. First, use a mask to block the target object in the video image, and then input the data in batches to extract the two-dimensional restoration prior information.

[0011] As a preferred solution, the monocular dynamic scene reconstruction algorithm is specifically as follows: input a set of monocular dynamic scene images with time tags and the corresponding camera poses, and use sparse point clouds reconstructed by random generation or motion recovery methods to construct a set of three-dimensional Gaussian distributions H(p, r, s, α), where p represents the center position and α represents the opacity. The three-dimensional covariance matrix ∑ is calculated by the quaternion r and the scaling factor s. Its mathematical expression is:

[0012]

[0013] For optimization, the covariance matrix ∑ is decomposed into ∑ = RSSTST, where R is the rotation matrix represented by the quaternion q∈SO(3), and S is the scaling matrix represented by the three-dimensional vector s;

[0014] Each Gaussian voxel also contains an opacity value α for adjusting its influence in rendering, and is associated with a spherical harmonic coefficient sh to achieve a perspective-dependent appearance. The entire scene is parameterized by a set of Gaussian point clouds, namely: G = {Gj:pj,qj,sj,αj,shj}. The rendering process projects these three-dimensional Gaussian point clouds onto the two-dimensional image plane and aggregates them through fast α blending. By using the view transformation matrix W and the Jacobian matrix J of the projection transformation affine approximation, the covariance matrix ∑′ can be calculated in the camera coordinate system:

[0015]

[0016] The color C of the pixel in the image is rendered by the alpha blending method of Gaussian points, and the formula is as follows:

[0017]

[0018] SH is the spherical harmonic function, vi is the viewing direction, and αi′ refers to the actual opacity of the corresponding projected Gaussian basis Gi at the pixel:

[0019]

[0020] Among them, μ is the position of the pixel on the image plane, μi′ and ∑i′ are the projection and covariance matrices of the Gaussian voxel Gi, respectively. 3DGS uses stochastic gradient descent technology for optimization, fully utilizes the standard GPU acceleration framework, and implements some operations through custom CUDA kernels, realizing that the fast rasterization module plays a key role in optimizing efficiency and optimizing the main computational bottlenecks in the process.

[0021] As a preferred solution, the monocular dynamic scene reconstruction algorithm uses the image of the monocular video to train a rough static three-dimensional Gaussian point cloud representation of a scene in the previous iteration, and then uses a deformation network MLP to learn the change of 3DGS over time. It takes the initial position of the three-dimensional Gaussian and the current time t as input, and outputs deformation parameters Δp, Δr, Δs. The deformed three-dimensional Gaussian G(p+Δp,r+Δr,s+Δs,α) is then input into the efficient differentiable Gaussian rasterization pipeline for rendering. This process uses stochastic gradient descent technology to jointly optimize the three-dimensional Gaussian point cloud and the MLP deformation network, thereby training the Gaussian point cloud for modeling the initial scene.

[0022] As a preferred solution, for each time frame in the monocular video, the timestamp ti, the mask Mi for marking the removed object, and the camera projection matrix at the timestamp ti are known. The present invention first calculates the spatial position p(ti)=p+Δp(ti) after deformation of the Gaussian basis element, and then uses the camera projection matrix to calculate the coordinate μ(t) of the deformed Gaussian basis element projected onto the two-dimensional image plane. The number of markers of the Gaussian basis element whose image plane coordinate μ(t) is within the mask Mi is increased by 1. After traversing all the timestamps, the Gaussian basis elements whose marker number is greater than the number of time frames T×0.8 are removed.

[0023] As a preferred solution, in the subsequent optimization process, densification is performed every 100 iterations, and those transparent Gaussian points, that is, Gaussian points with α less than the threshold ∈α, are removed. The goal of adaptive control is to fill the blank areas, focusing on the under-reconstruction areas where geometric features are missing and the over-reconstruction areas where the Gaussian coverage area is too large. These areas have large view space position gradients, indicating that they have not been fully reconstructed. The optimization process attempts to correct them by moving Gaussian points. By densifying the Gaussian points whose average amplitude of the view space position gradient exceeds the threshold τpos=0.0002, the small Gaussian in the under-reconstruction area is cloned and moved along the position gradient direction. For large Gaussian points, they are split into two new Gaussians and their scales are reduced by the experimentally determined factor φ=1.6. At the same time, the original Gaussian is used as the probability density function for sampling to initialize their positions. In addition, the value α is approached to zero every 3000 iterations so that the optimization only increases α on the required Gaussian points, and the Gaussian points with α less than ∈α are removed by the above-mentioned elimination method.

[0024] As a preferred solution, a mask of the object to be removed is first used to erase the target object information in all images of the monocular video and replace it with a black background area. These masked monocular video images are then input into the video inpainting model in batches and prompted to repair the black background area through language. The video inpainting model then inputs the repaired images and uses the repaired monocular video images to fine-tune the Gaussian basis set remaining after removing the target object.

[0025] In dynamic scenes, after the object to be removed moves, the originally occluded background area will appear in subsequent time frames. These time frames contain the true texture information of the repaired area. To this end, a mask-weighted loss function is used to supervise the fine-tuning process:

[0026]

[0027] Where Ci is the image rendered by the monocular dynamic scene reconstruction algorithm, Ii is the repaired image generated by the video repair model based on the initial video frame and its corresponding mask, Mi is the mask that marks the removed object, and the area with a value of 0 is where the object is located. λ1 and λ2 are hyperparameters to weight the loss inside and outside the mask area.

[0028] The beneficial effects of a dynamic scene restoration method based on four-dimensional Gaussian sputtering disclosed by the present invention are: based on the restoration framework of the four-dimensional Gaussian sputtering monocular video reconstruction algorithm and the video restoration model, efficient dynamic scene spatial restoration is achieved, and efficient dynamic object removal is achieved by counting the number of time frames of Gaussian primitives in the mask area and removing Gaussian points exceeding the threshold. A mask weighted loss function is introduced to guide the restoration area to converge to the real texture information in subsequent time frames, thereby improving the restoration quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a dynamic scene repair method based on four-dimensional Gaussian sputtering according to the present invention.

[0030] Figure 2 This is a visualization result diagram of an embodiment of a dynamic scene restoration method based on four-dimensional Gaussian sputtering of the present invention.

[0031] Figure 3 This is a measurement value diagram of an image within a mask area of a dynamic scene restoration method based on four-dimensional Gaussian sputtering according to the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described and explained below in conjunction with specific embodiments and accompanying drawings:

[0033] Please refer to Figure 1 , a dynamic scene restoration method based on four-dimensional Gaussian sputtering, comprising:

[0034] Rebuilding the initial scene

[0035] The monocular dynamic scene reconstruction algorithm (Deformable 3D gaussians, D3DGS) is used to reconstruct the initial scene. The algorithm is based on the representation of point clouds and realizes dynamic scene modeling by decoupling 4DGS into 3DGS and deformation field.

[0036] The monocular dynamic scene reconstruction algorithm is specifically as follows: a set of monocular dynamic scene images with time tags and the corresponding camera poses are input, and a set of three-dimensional Gaussian distributions G(p, r, s, α) is constructed using sparse point clouds reconstructed by random generation or motion recovery structure method (Structure from Motion, SFM). p represents the center position and α represents the opacity. The three-dimensional covariance matrix ∑ is calculated by the quaternion r and the scaling factor s. Its mathematical expression is:

[0037]

[0038] For optimization, the covariance matrix ∑ is decomposed into ∑ = RSSTST, where R is the rotation matrix represented by the quaternion q∈SO(3), and S is the scaling matrix represented by the three-dimensional vector s;

[0039] Each Gaussian voxel also contains an opacity value α for adjusting its influence in rendering, and is associated with a spherical harmonic coefficient sh to achieve a perspective-dependent appearance. The entire scene is parameterized by a set of Gaussian point clouds, namely: G = {Gj:pj,qj,sj,αj,shj}. The rendering process projects these three-dimensional Gaussian point clouds onto the two-dimensional image plane and aggregates them through fast α blending. By using the view transformation matrix W and the Jacobian matrix J of the projection transformation affine approximation, the covariance matrix ∑′ can be calculated in the camera coordinate system:

[0040]

[0041] The color C of the pixel in the image is rendered by the alpha blending method of Gaussian points, and the formula is as follows:

[0042]

[0043] SH is the spherical harmonic function, vi is the viewing direction, and αi′ refers to the actual opacity of the corresponding projected Gaussian basis Gi at the pixel:

[0044]

[0045] Among them, μ is the position of the pixel on the image plane, μi′ and ∑i′ are the projection and covariance matrices of the Gaussian voxel Gi, respectively. 3DGS uses stochastic gradient descent technology for optimization, fully utilizes the standard GPU acceleration framework, and implements some operations through custom CUDA kernels, realizing that the fast rasterization module plays a key role in optimizing efficiency and optimizing the main computational bottlenecks in the process.

[0046] In the previous iteration, the monocular dynamic scene reconstruction algorithm uses the image of the monocular video to train a rough static three-dimensional Gaussian point cloud representation of a scene, and then uses a deformation network MLP to learn the change of 3DGS over time. It takes the initial position of the three-dimensional Gaussian and the current time t as input, and outputs deformation parameters Δp, Δr, Δs. The deformed three-dimensional Gaussian G(p+Δp,r+Δr,s+Δs,α) is then input into the efficient differentiable Gaussian rasterization pipeline for rendering. This process uses stochastic gradient descent technology to jointly optimize the three-dimensional Gaussian point cloud and the MLP deformation network to train the Gaussian point cloud for modeling the initial scene.

[0047] Dynamic object removal

[0048] In dynamic scenes, if the object to be removed is in motion or the camera is moving, then when removing the object mask at each time frame, only the object is always within the mask. As the object moves, the background area behind the object in the current time frame may be revealed in subsequent time frames after the object moves to other areas. In this way, the background area is only within the mask area for a period of time. By counting the number of time frames in which the Gaussian primitives are within the mask area and removing the Gaussian primitives that exceed the threshold, efficient dynamic object removal can be achieved.

[0049] For each time frame in the monocular video, the timestamp ti, the mask Mi for marking the removed object, and the camera projection matrix at the timestamp ti are known. The present invention first calculates the spatial position p(ti)=p+Δp(ti) after deformation of the Gaussian basis element, and then uses the camera projection matrix to calculate the coordinate μ(t) of the deformed Gaussian basis element projected onto the two-dimensional image plane. The number of markers of the Gaussian basis element whose image plane coordinate μ(t) is within the mask Mi is increased by 1. After traversing all timestamps, the Gaussian basis elements whose marker number is greater than the number of time frames T×0.8 are removed.

[0050] In the subsequent optimization process, densification is performed every 100 iterations, and those transparent Gaussian points, that is, Gaussian points with α less than the threshold ∈α, are removed. The goal of adaptive control is to fill the blank areas, focusing on the under-reconstruction areas where geometric features are missing and the over-reconstruction areas where the Gaussian coverage area is too large. These areas have large view space position gradients, indicating that they have not been fully reconstructed. The optimization process attempts to correct them by moving Gaussian points. By densifying the Gaussian points whose average amplitude of the view space position gradient exceeds the threshold τpos=0.0002, the small Gaussian in the under-reconstruction area is cloned and moved along the position gradient direction. For large Gaussian points, they are split into two new Gaussians and their scales are reduced by the experimentally determined factor φ=1.6. At the same time, the original Gaussian is used as the probability density function for sampling to initialize their positions. In addition, the value α is approached to zero every 3000 iterations so that the optimization only increases α on the required Gaussian points, and the Gaussian points with α less than ∈α are removed by the above elimination method.

[0051] Scene repair and optimization

[0052] Remove the target object in the monocular video and repair the video based on it to obtain the two-dimensional repair information required for dynamic scene repair. First, use a mask to block the target object in the video image, and then input data in batches to extract the two-dimensional repair prior information.

[0053] First, a mask of the object to be removed is used to erase the target object information in all images of the monocular video and replace it with a black background area. These masked monocular video images are then fed into the video inpainting model in batches and prompted to repair the black background area through language. The video inpainting model then inputs the repaired images and uses the repaired monocular video images to fine-tune the remaining Gaussian basis points after removing the target object.

[0054] In dynamic scenes, after the object to be removed moves, the originally occluded background area will appear in subsequent time frames. These time frames contain the true texture information of the repaired area. To this end, a mask-weighted loss function is used to supervise the fine-tuning process:

[0055]

[0056] in It is the image rendered by the monocular dynamic scene reconstruction algorithm, Ii is the repaired image generated by the video repair model based on the initial video frame and its corresponding mask, Mi is the mask that marks the removed object, and the area with a value of 0 is where the object is located. λ1 and λ2 are hyperparameters to weight the loss inside and outside the mask area.

[0057] Performance Analysis

[0058] The above method was experimentally validated on the NeRF-DS dataset. The NeRF-DS dataset is a real-world monocular video reconstruction dataset acquired using a dual-camera configuration: videos captured by the left camera serve as the training set, and videos captured by the right camera serve as the test set. Each video contains hundreds of frames with a resolution of 480×270 pixels and provides a precise mask of the target object. In this example, the target object within the masked area was first removed, and then the scene was inpainted to verify the effectiveness of the proposed method.

[0059] In this example, 20,000 iterations were performed for both initial scene training and inpainting fine-tuning. The weight coefficients λ1 and λ2 in the mask-weighted loss function were set to 3 and 1, respectively. Furthermore, during object removal, a frequency threshold of 0.8 was set, meaning that Gaussian points projected into the masked region in more than 80% of the time frames were removed. This setting ensures effective object removal in dynamic scenes while preserving the scene's geometric consistency and detail.

[0060] Figure 2 The visualization results of the proposed dynamic scene spatial inpainting framework are presented. Experimental results show that the proposed method can not only effectively inpaint scenes, but also synthesize new views that are consistent in time and space.

[0061] At the same time, in order to verify the superiority of the framework invented in this chapter over the direct use of the video inpainting model to inpaint video frames trained on Gaussian point cloud representation, as well as the effectiveness of Masked Weighted Loss (MWL), an ablation experiment was conducted, and the image quality of the inpainted area was evaluated using the no-reference image evaluation metrics MUSIQ and CLIPIQA+

[75] . Figure 3 In English, MUSIQ-K and MUSIQ-S stand for 'musiq-koniq' and 'musiq-spaq' respectively.

[0062] CLIPIQA+RN50. All results in the table are measurements of images within the mask area. Figure 3 In the figure, the first row shows the results of directly using the video restoration model to restore the video frames and train the monocular dynamic scene reconstruction algorithm. The second row shows the results of removing the target object and then fine-tuning the pre-trained monocular dynamic scene reconstruction algorithm model. The last row shows the results of introducing the mask-weighted loss in the framework of the present invention. Experiments show that the restoration effect of fine-tuning on the pre-trained model is better than direct training. This is due to the fact that the Gaussian primitive removal method proposed in this invention effectively preserves the geometric information of the scene. In addition, although the results of introducing the mask-weighted loss are comparable to those without the introduction in terms of CLIPIQA+ indicators, there is a certain improvement in the MUSIQ indicator.

[0063] The present invention provides a dynamic scene restoration method based on four-dimensional Gaussian sputtering. The restoration framework based on the four-dimensional Gaussian sputtering monocular video reconstruction algorithm and the video restoration model realizes efficient dynamic scene spatial restoration. By counting the number of time frames in which Gaussian basis units are within the mask area and removing Gaussian points exceeding the threshold, efficient dynamic object removal is achieved. A mask-weighted loss function is introduced to guide the restoration area to converge to the real texture information in subsequent time frames, thereby improving the restoration quality.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A dynamic scene restoration method based on four-dimensional Gaussian sputtering, characterized in that: include: Reconstruct the initial scene. A monocular dynamic scene reconstruction algorithm is used to reconstruct the initial scene. The algorithm is based on the representation of point clouds and realizes dynamic scene modeling by decoupling 4DGS into 3DGS and deformation field. Dynamic object removal: In dynamic scenes, if the object to be removed is in motion or the camera is moving, then when removing the object mask at each given time frame, only the object is always within the mask. As the object moves, the background area behind the object in the current time frame may be revealed in subsequent time frames after the object moves to other areas. In this way, the background area is only within the mask area for a period of time. By counting the number of time frames in which Gaussian primitives are within the mask area and removing Gaussian primitives that exceed the threshold, efficient dynamic object removal can be achieved. Scene restoration and optimization: remove the target object in the monocular video and restore the video based on this to obtain the two-dimensional restoration information required for dynamic scene restoration. First, use a mask to block the target object in the video image, and then input the data in batches to extract the two-dimensional restoration prior information.

2. A dynamic scene restoration method based on four-dimensional Gaussian sputtering according to claim 1, characterized in that: The monocular dynamic scene reconstruction algorithm is specifically as follows: input a set of monocular dynamic scene images containing time tags and the corresponding camera poses, and use the sparse point cloud reconstructed by random generation or motion recovery method to construct a set of three-dimensional Gaussian distribution G (p, r, s, α), where p represents the center position, α represents the opacity, and the three SO(3) represents the rotation matrix, S is the scaling matrix represented by the three-dimensional vector s; Each Gaussian voxel also contains an opacity value α for adjusting its influence in rendering, and is associated with a spherical harmonic coefficient sh to achieve a perspective-dependent appearance. The entire scene is parameterized by a set of Gaussian point clouds, namely: G = {Gj:pj,qj,sj,αj,shj}. The rendering process projects these three-dimensional Gaussian point clouds onto the two-dimensional image plane and aggregates them through fast α blending. By using the view transformation matrix W and the Jacobian matrix J of the projection transformation affine approximation, the covariance matrix ∑′ can be calculated in the camera coordinate system: The color C of the pixel in the image is rendered by the alpha blending method of Gaussian points, and the formula is as follows: SH is the spherical harmonic function, vi is the viewing angle, and αi′ refers to the corresponding projection Gaussian basis element Gi at the pixel. To neutralize the covariance matrix, 3DGS uses stochastic gradient descent technology for optimization, fully utilizing the standard GPU acceleration framework and implementing some operations through custom CUDA kernels, thus achieving a fast rasterization module that plays a key role in optimizing efficiency and optimizing the main computational bottlenecks in the process.

3. A dynamic scene restoration method based on four-dimensional Gaussian sputtering according to claim 2, characterized in that: The monocular dynamic scene reconstruction algorithm uses the monocular video image to train a rough static 3D Gaussian point cloud representation of the scene in the previous iteration, and then uses a deformation network MLP to learn the change of 3DGS over time. It takes the initial position of the 3D Gaussian and the current time t as input and outputs the deformation parameter Δp. Δs, and then the deformed three-dimensional Gaussian G(p+Δp,r+Δr,s+Δs,α) is input into the efficient differentiable Gaussian rasterization pipeline for rendering. This process uses the stochastic gradient descent technique to jointly optimize the three-dimensional Gaussian point cloud and the MLP deformation network to train the Gaussian point cloud for modeling the initial scene.

4. The dynamic scene restoration method based on four-dimensional Gaussian sputtering according to claim 1, characterized in that: For each time frame in the monocular video, given the timestamp ti, the mask Mi that marks the removed object, and the camera projection matrix at the timestamp ti, the present invention first calculates the spatial position of the Gaussian primitive after deformation. (ti) = p + Δp(ti), and then use the camera projection matrix to calculate the coordinate μ(t) of the deformed Gaussian primitive projected onto the two-dimensional image plane. For the Gaussian primitives whose image plane coordinate μ(t) is within the mask Mi, the number of markers is increased by 1. After traversing all the timestamps, remove the Gaussian primitives whose markers are greater than the number of time frames T×0.

8.

5. The dynamic scene restoration method based on four-dimensional Gaussian sputtering according to claim 4, characterized in that: In the subsequent optimization process, densification is performed every 100 iterations, and those transparent Gaussian points, that is, Gaussian points with α less than the threshold ∈α, are removed. The goal of adaptive control is to fill the blank areas, focusing on the under-reconstruction areas where geometric features are missing and the over-reconstruction areas where the Gaussian coverage area is too large. These areas have large view space position gradients, indicating that they have not been fully reconstructed. The optimization process attempts to correct them by moving Gaussian points. By densifying the Gaussian points whose average amplitude of the view space position gradient exceeds the threshold τpos=0.0002, the small Gaussian in the under-reconstruction area is cloned and moved along the position gradient direction. For large Gaussian points, they are split into two new Gaussians and their scales are reduced by the experimentally determined factor φ=1.

6. At the same time, the original Gaussian is used as the probability density function for sampling to initialize their positions. In addition, the value α is approached to zero every 3000 iterations so that the optimization only increases α on the required Gaussian points, and the Gaussian points with α less than ∈α are removed by the above elimination method.

6. The method for dynamic scene restoration based on four-dimensional Gaussian sputtering according to claim 1, characterized in that: First, a mask of the object to be removed is used to erase the target object information in all images of the monocular video and replace it with a black background area. These masked monocular video images are then fed into the video inpainting model in batches and prompted to repair the black background area through language. The video inpainting model then inputs the repaired images and uses the repaired monocular video images to fine-tune the remaining Gaussian basis points after removing the target object. In dynamic scenes, after the object to be removed moves, the originally occluded background area will appear in subsequent time frames. These time frames contain the true texture information of the repaired area. To this end, a mask-weighted loss function is used to supervise the fine-tuning process: Where Ci is the image rendered by the monocular dynamic scene reconstruction algorithm, Ii is the repaired image generated by the video repair model based on the initial video frame and its corresponding mask, Mi is the mask that marks the removed object, and the area with a value of 0 is where the object is located. λ1 and λ2 are hyperparameters to weight the loss inside and outside the mask area.

Images