Scene completion method based on Gaussian sputtering

Abstract

The invention discloses a scene completion method based on Gaussian sputtering, and relates to the technical field of scene reconstruction. Obtaining an original video of a target scene collected by a camera device; performing Gaussian sputtering conversion on pixel points of each frame of image in the original video to obtain Gaussian distribution representation; the Gaussian distribution representation comprises spatial position and color information of a plurality of three-dimensional Gaussian points; inputting the spatial positions of all the three-dimensional Gaussian points into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing features extracted by a contrast language image pre-training CLIP model and a self-distillation visual converter DINO model as supervision information into a point cloud completion network PCN model; and according to the Gaussian distribution representation, performing color rendering on the complete Gaussian point cloud to obtain optimized Gaussian point cloud distribution. According to the method, a scene with a relatively vivid effect can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of scene reconstruction, and in particular to a scene completion method based on Gaussian sputtering. Background Art

[0002] In actual scenarios (such as complex indoor environments, urban streets, or power transmission and distribution line scenarios), occlusion between objects is common, especially when vegetation, buildings, or equipment overlap with each other in complex scenes, which often leads to incomplete three-dimensional reconstruction.

[0003] Existing technologies for completing missing regions in traditional point clouds primarily rely on point cloud completion methods based on geometric priors or deep learning, such as the Point Completion Network (PCN). These methods typically infer missing regions by analyzing the local structure of the point cloud, but they often struggle to capture global geometric relationships when dealing with complex scenes. Deep learning-based point cloud completion methods, on the other hand, utilize neural networks to model the global structure of the point cloud, generating more expressive completion results. However, these methods struggle to produce detailed completion results in noisy or complex scenes.

[0004] Therefore, there is an urgent need in the prior art for a scene completion method that can generate a more realistic effect. Summary of the Invention

[0005] Based on this, it is necessary to provide a scene completion method based on Gaussian sputtering to address the above technical problems, which can generate scenes with more realistic effects.

[0006] The present invention adopts the following technical solutions: The present invention provides a scene completion method based on Gaussian sputtering, comprising: Obtaining the original video of the target scene captured by the camera device; Perform Gaussian sputtering transformation on each pixel point of each frame in the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of multiple three-dimensional Gaussian points; The spatial positions of all 3D Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene. The improved point cloud completion model is constructed by introducing the features extracted from the contrastive language image pre-training CLIP model and the self-distilled visual transformer DINO model as supervision information into the point cloud completion network PCN model. According to the Gaussian distribution representation, the complete Gaussian point cloud is rendered in color to obtain the optimized Gaussian point cloud distribution.

[0007] Optionally, the pixels of each frame in the original video are subjected to Gaussian sputtering transformation to obtain Gaussian distribution representation, including: Determine the essential matrix based on any two matching points between images in the original video; Decompose the essential matrix to obtain the rotation matrix and translation vector; Convert the two-dimensional pixel points of each image into initial three-dimensional points according to the intrinsic parameter matrix, rotation matrix and translation vector of the camera device; Adjust the position of the initial 3D point according to minimizing the reprojection error on all images to obtain the candidate 3D point; The candidate 3D points are added with Gaussian attributes to obtain the Gaussian distribution representation corresponding to the candidate 3D points.

[0008] Optionally, the reprojection error Minimize to: ; in, is the camera's internal parameter matrix, representing the camera hardware parameters, Indicates the k The rotation matrix of the pixel points, Indicates the k The translation vector of pixels, Indicates the k Two-dimensional pixels, Indicates the k An initial 3D point.

[0009] Alternatively, the Gaussian distribution is represented as: ; in, is the spatial position of the three-dimensional Gaussian point; is the covariance matrix of the three-dimensional Gaussian points, Is an RGB vector, representing the color or texture information of a three-dimensional Gaussian point; is opacity.

[0010] Optionally, the improved point cloud completion model includes a PCN point cloud completion model using a CLIP model, a DINO model, and a Transformer model; the spatial positions of all three-dimensional Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene, including: Input the spatial positions of all 3D Gaussian points into the encoder of the PCN model to obtain point cloud features; The image segmentation results corresponding to all 3D Gaussian points are input into the CLIP model and DINO model respectively to obtain high-level semantic features and pixel-level image features; Obtain the average features of high-level semantic features and pixel-level image features, concatenate the average features with the point cloud features, and input them into the cross-attention module of the Transformer model to obtain the completed point cloud features; The completed point cloud features are input into the decoder of the PCN model to obtain the complete Gaussian point cloud of the target scene.

[0011] Optionally, the complete Gaussian point cloud is rendered in color according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution, including: Project the complete Gaussian point cloud onto a two-dimensional plane and detect whether there is an occluded area; In the absence of occlusion areas, the Gaussian points on the two-dimensional plane are rendered in color according to Gaussian sputtering to obtain the optimized Gaussian point cloud distribution; In the presence of occluded areas, the colors of the newly added Gaussian points in the complete Gaussian points are adjusted through image completion technology and Gaussian rendering to obtain the optimized Gaussian point cloud distribution.

[0012] Optionally, the colors of the newly added Gaussian points in the complete Gaussian point cloud are adjusted through image completion technology and Gaussian rendering to obtain an optimized Gaussian point cloud distribution, including: Calculate the mask of the occluded area, and based on the mask of the occluded area, complete the occluded area of ​​the three-dimensional Gaussian points to obtain the completed image; Render the newly added Gaussian points using Gaussian sputtering technology to obtain a rendered image; Determine the gradient of the loss function based on the rendered image and the completed image; The colors of the newly added Gaussian points are adjusted according to the gradient of the loss function to obtain the optimized Gaussian point cloud distribution.

[0013] Optionally, after obtaining the optimized Gaussian point cloud distribution, the method further includes: The Gaussian points in the Gaussian point cloud distribution are subjected to boundary removal and scene consistency retraining to obtain the target Gaussian point cloud distribution.

[0014] The present invention provides a scene completion device based on Gaussian sputtering, comprising: An acquisition module is used to acquire the original video of the target scene captured by the camera device; A conversion module is used to perform Gaussian sputtering conversion on the pixels of each frame of the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of multiple three-dimensional Gaussian points; The completion module is used to input the spatial positions of all 3D Gaussian points into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene. The improved point cloud completion model is constructed by introducing features extracted from the contrastive language image pre-training CLIP model and the self-distilled visual transformer DINO model as supervision information into the point cloud completion network PCN model. The rendering module is used to perform color rendering on the complete Gaussian point cloud according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution.

[0015] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method for scene completion based on Gaussian sputtering is implemented.

[0016] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned Gaussian sputtering-based scene completion method when executing the program.

[0017] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects: In the present invention, first, the pixel points of each frame of the original video are subjected to Gaussian sputtering transformation to obtain a Gaussian distribution representation containing the spatial position and color information of the three-dimensional Gaussian points. This transformation method can more flexibly express the geometric and color information in the scene. Each Gaussian point can be regarded as an approximate description of the local area of ​​the image. Compared with the traditional pixel representation, it can better capture the spatial structure and details of the scene, and provide a richer information basis for subsequent processing. Furthermore, when completing the point cloud, the supervision information of the contrast language image pre-training CLIP model and the self-distillation visual transformer DINO model is combined to further enhance the existing point cloud information and supervision features, so as to accurately generate a complete Gaussian point cloud. Finally, the complete Gaussian point cloud is color rendered according to the Gaussian distribution representation of the original Gaussian point cloud, so that the completed point cloud is not only closer to the real scene in terms of geometry, but also matches the information in the original video in color, so that the completed scene is visually more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 A schematic flow chart of a scene completion method based on Gaussian sputtering provided by the present invention; Figure 2A PCN provided by the present invention adopts an encoder-decoder architecture diagram; Figure 3 A schematic flow chart of another scene completion method based on Gaussian sputtering provided by the present invention; Figure 4 A schematic diagram of a computer device for implementing a scene completion method based on Gaussian sputtering provided by the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Gaussian Splatting (GS), a new 3D scene representation technology, enables efficient, flexible, and realistic scene restoration. However, point cloud completion is a poor substitute for Gaussian splatting, as the resulting point cloud model has large holes and cannot achieve the realistic 3D representation of Gaussian splatting. Image completion, or image restoration, aims to automatically fill in missing or damaged areas in an image, making them visually natural, reasonable, and consistent with the surrounding content. Its advantage is that it leverages prior knowledge from big data to make the filled areas natural, textured, and realistic. However, its disadvantage is that it only works within the image domain and the filling effect is somewhat random.

[0022] It is necessary to optimize the consistency of the Gaussian representation of the scene, otherwise it will easily lead to local distortion or global inconsistency problems.

[0023] Therefore, the present invention provides a scene completion method based on Gaussian sputtering. This method combines point cloud completion methods with image completion models. By designing a rational network framework and loss function to jointly optimize the Gaussian sputtering representation of the scene, it achieves intelligent reconstruction of the original scene. This method not only effectively completes missing areas but also generates a detailed 3D model while maintaining point cloud consistency, providing higher assurance for the authenticity of scene restoration and subsequent applications in various industries.

[0024] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 The figure is a flow chart of a scene completion method based on Gaussian sputtering in the present invention, which specifically includes the following steps: S101: Obtain an original video of a target scene captured by a camera device.

[0026] S102 , performing Gaussian sputtering transformation on the pixel points of each frame image in the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes spatial position and color information of multiple three-dimensional Gaussian points.

[0027] In one embodiment, the pixel points of each frame image in the original video are Gaussian sputtering transformed to obtain a Gaussian distribution representation, including: determining an essential matrix based on any two matching points between images in the original video; decomposing the essential matrix to obtain a rotation matrix and a translation vector; converting the two-dimensional pixel points of each image into initial three-dimensional points based on the intrinsic parameter matrix, rotation matrix and translation vector of the camera device; adjusting the position of the initial three-dimensional point based on minimizing the reprojection error on all images to obtain a candidate three-dimensional point; and adding Gaussian attributes to the candidate three-dimensional point to obtain a Gaussian distribution representation corresponding to the candidate three-dimensional point.

[0028] Adding Gaussian attributes to the candidate three-dimensional points includes supplementing each candidate three-dimensional point with an initialized Gaussian attribute value.

[0029] The Segment Anything Model (SAM) is used to generate a panoramic segmentation 2D mask for each frame of the original video, and the inter-frame object association is performed through each frame mask to obtain a temporally and spatially consistent segmentation result for all objects.

[0030] At the same time, the original video is filtered for key frames based on the ORB feature overlap rate and frame rate between frames, and the image pose is extracted using the Structure from Motion (SfM) method, such as the existing colmap tool. Since feature extraction and matching have been completed in SAM, the next step is to use the epipolar geometry method for preliminary 3D reconstruction. When the camera intrinsic parameter matrix K is known, the essential matrix E describes the matching points of the two images. and The corresponding relationship: .

[0031] Therefore, the essential matrix can be obtained by solving the equation using the eight-point method or nonlinear method. Then the rotation matrix is ​​obtained by SVD decomposition and translation vectors And restore the initial 3D points by triangulation After the 3D points and camera poses are initially restored, they need to be optimized. Projecting into the image to get 2D points , the camera poses and 3D point positions are optimized by minimizing the reprojection errors on all images through bundle adjustment (BA).

[0032] ; ; in, Indicates the k The rotation matrix of pixels represents the parameters of the camera hardware. Indicates the k The translation vector of pixels, Indicates the k Two-dimensional pixels, Indicates the k The initial three-dimensional point is converted into a two-dimensional pixel point through the camera's internal and external parameters. It maps three-dimensional points to two-dimensional points.

[0033] The output of the SFM method is a sparse 3D point cloud of the extracted camera pose and scene. This sparse point cloud is then used to initialize and train a Gaussian (GS) model to obtain a GS-based 3D representation of the scene.

[0034] Specifically, each three-dimensional point obtained by SFM is initialized to a Gaussian distribution in the form of: in, is the spatial position of the three-dimensional Gaussian point, initialized by the spatial position of the sparse point; It is the covariance matrix of the three-dimensional Gaussian points, which indicates the degree of expansion of the three-dimensional Gaussian points in space. It is initialized to 1 for subsequent adjustment. Is an RGB vector, representing the color or texture information of a three-dimensional Gaussian point, represented by the spherical harmonic function of Gauss, and initialized here to the color corresponding to the point cloud. is the opacity, initialized to 0.5.

[0035] Next, using the projection formula, the three-dimensional Gaussian points are projected into the camera coordinate system using the Jacobian matrix, and then mapped onto the two-dimensional image plane to render the image. During rendering, all Gaussian points are sorted by depth from far to near, and alpha blending is used to calculate the final pixel color:

[0036] in, Indicates the cumulative transparency of all previous Gaussian points (to achieve occlusion effect); represents the color of the 𝑖th Gaussian point; is the opacity of the Gaussian point.

[0037] The position, color, opacity and other parameters of the Gaussian point can be directly optimized using the gradient descent method. For the covariance matrix of the Gaussian distribution Gradient of , applying the chain rule, will be The gradient is converted into the scaling factor s and the rotation quaternion q The gradient of the parameters. The loss function is the image rendered by the model With real images The weighted sum of the L1 loss and D-SSIM loss between:

[0038] In one embodiment, a three-dimensional segmentation of a scene based on Gaussian sputtering may also be determined, which specifically includes the following steps:

[0039] This step passes the 2D mask and its label obtained by the Segment Everything Model (SAM) to each Gaussian point, so that each Gaussian point has a category label. This category label is used to merge 3D Gaussian points with the same semantic category. Specifically, the back-projection formula of 2D pixel coordinates to 3D points in space is used, which is:

[0040] in( are the coordinates of the 2D image plane; is the intrinsic parameter of the camera; d is the depth information of the camera; It is a 2D mask generated by the SAM large model for the key frame. Then initialize a Gaussian counter for each Gaussian point, that is, expand the attributes of the Gaussian point to . The Gaussian counter is used to calculate the label value of each Gaussian point after back-projection. Due to the limitations of the SAM algorithm, it is inevitable to assign an incorrect label value to a Gaussian point at a certain perspective. Therefore, a counter is used to save the labels of the spatial Gaussian point under multiple two-dimensional perspectives, and then the final object label l of the Gaussian point is obtained by filtering based on the maximum value. The label filtering formula is as follows (where g represents a three-dimensional Gaussian point; V is the set of all camera perspectives; ) is the label assigned by the sam model under the perspective v: .

[0041] S103, input the spatial positions of all three-dimensional Gaussian points into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing the features extracted by the contrastive language image pre-training CLIP model and the self-distillation visual transformer DINO model as supervision information into the point cloud completion network PCN model.

[0042] Because the scene may be sparse, missing or occluded, the point cloud data of some objects is incomplete. The task of point cloud completion is to identify individual objects one by one and generate dense and complete object point clouds through an improved point cloud completion model, and finally obtain a complete 3D point cloud representation of the scene. The point cloud completion model used in this invention is an improved PCN (Point Completion Network) method. PCN adopts an encoder-decoder architecture, such as Figure 2 shown.

[0043] The encoder compresses the point cloud data into a compact global feature vector. It extracts point cloud features, generates a feature matrix, and performs maximum pooling to obtain global features. These global features are then concatenated with the original features and encoded and pooled again to obtain the final global feature vector. The decoder, starting from this global feature vector, gradually recovers the complete point cloud through coarse and fine stages. The coarse stage uses a fully connected network to generate a coarse point cloud, while the fine stage refines the coarse point cloud through neighborhood search and shared multi-layer perception (MLP), ultimately generating refined point cloud data.

[0044] To improve the accuracy of GS point cloud completion, this paper proposes introducing features extracted by the Contrastive Language-Image Pre-Training (CLIP) model and the Self-Distilled Vision Transformer (DINO) model as additional supervisory information into the traditional PCN process. The CLIP model is a powerful multimodal model that can map images and text into the same embedding space. This embedding approach not only extracts semantic information from the image but also provides a strong semantic prior for subsequent point cloud completion and reconstruction tasks. The Self-Distilled Vision Transformer (DINO) model can extract deep visual features from images. It uses unsupervised learning methods to capture spatial information, object structure, and depth relationships in the image through contrastive learning. These deep visual features supplement the geometric information of the image and complement the semantic features.

[0045] Optionally, the improved point cloud completion model includes a PCN point cloud completion model using a CLIP model, a DINO model, and a Transformer model; the spatial positions of all three-dimensional Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene, including: inputting the spatial positions of all three-dimensional Gaussian points into the encoder of the PCN model to obtain point cloud features; inputting the segmentation results of the images corresponding to all three-dimensional Gaussian points into the CLIP model and the DINO model respectively to obtain high-level semantic features and pixel-level image features; obtaining the average features of the high-level semantic features and the pixel-level image features, and concatenating the average features with the point cloud features and inputting them into the cross-attention module of the Transformer model to obtain completed point cloud features; inputting the completed point cloud features into the decoder of the PCN model to obtain a complete Gaussian point cloud of the target scene.

[0046] Specifically, first convert the three-dimensional Gaussian points with the same semantic label into input point cloud (simply concatenate the coordinate positions and color vectors of the three-dimensional Gaussian points), and use the PCN encoder to extract point cloud features. ; At the same time, randomly select the corresponding n SAM model segmentation result images to extract CLIP and DINO features 、 . Then the high-level semantic features Pixel-level image features Average and stitching, and point cloud features The features are then fed into the cross-attention module of the Transformer model for feature fusion to obtain the completed point cloud features. Finally, the completed point cloud is obtained by passing it through the decoder of the PCN model. The formula is as follows:

[0047] ; ; Among them, the input Query of the Transformer model is , ensuring that the geometric structure dominates the attention calculation; the Key / Value source is the splicing feature , allowing image information to influence geometric features through attention weights. By performing cross-attention operations between queries and keys within the Transformer model, point cloud geometric features are obtained that fuse high-level semantics with local detail features. The fused features provide more contextual information for point cloud completion and 3D reconstruction. Semantic features can provide semantic constraints for missing regions, while geometric features can more accurately describe the structural relationships of the point cloud. This feature fusion approach significantly improves the accuracy and robustness of PCN in point cloud segmentation and 3D reconstruction tasks.

[0048] The improved PCN point cloud completion model is trained using a fully supervised training method guided by random masks. The datasets used are CO3D (containing 50 common objects from MS-COCO categories) and ShapeNet (containing 55 common indoor and outdoor object categories). 30%-90% of adjacent point clouds are randomly hidden to simulate partial point cloud loss caused by insufficient observation or occlusion. The original complete point cloud is used as the model supervision signal. After training, the improved PCN model is capable of completing point clouds with any degree of missing objects, making it plug-and-play.

[0049] S104 , performing color rendering on the complete Gaussian point cloud according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution.

[0050] Point cloud completion solves the missing parts in the Gaussian distribution point cloud, obtains a complete point cloud expression through completion, and initializes the newly supplemented point cloud as a new 3D Gaussian point. However, these supplemented point clouds only contain position information , other Gaussian properties (color, transparency, covariance, etc.) need to be optimized to make it more consistent and realistic in the scene.

[0051] In one embodiment, the complete Gaussian point cloud is color rendered according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution, including: projecting the complete Gaussian point cloud onto a two-dimensional plane and detecting whether there is an occlusion area; in the absence of an occlusion area, color rendering the Gaussian points under the two-dimensional plane according to Gaussian sputtering to obtain an optimized Gaussian point cloud distribution; in the presence of an occlusion area, adjusting the color of the newly added Gaussian points in the complete Gaussian point cloud through image completion technology and Gaussian rendering to obtain an optimized Gaussian point cloud distribution.

[0052] Among them, the color of the newly added Gaussian points in the complete Gaussian points is adjusted through image completion technology and Gaussian rendering to obtain the optimized Gaussian point cloud distribution, including: calculating the mask of the occluded area, and based on the mask of the occluded area, completing the occluded area of ​​the three-dimensional Gaussian points to obtain the completed image; rendering the newly added Gaussian points through Gaussian sputtering technology to obtain a rendered image; determining the gradient of the loss function based on the rendered image and the completed image; adjusting the color of the newly added Gaussian points according to the gradient of the loss function to obtain the optimized Gaussian point cloud distribution.

[0053] In order to detect missing or occluded areas of the point cloud, the newly supplemented 3D point cloud needs to be projected onto a 2D image. For the projected image, determine which pixels belong to the occluded area based on the depth map and obtain the corresponding mask. Then use the existing image completion model inpaint-anything to complete the occluded image and get the complete image Since the existing camera images do not contain occluded areas, it is necessary to perturb the original camera position, sample the new camera position, and determine whether the unobserved area M in the new pose image is large enough (for example, more than 10% of the object area) to decide whether to retain the image. .

[0054] For the Gaussian points corresponding to the blocked area, the above-mentioned Gaussian sputtering technology is used to render the image .

[0055] Compare the rendered image with the completed image at a certain perspective, and the loss function is: .

[0056] During training, the gradient of the loss function is transferred to the newly added GS point cloud. The position of the Gaussian point cloud is first fixed, and only its properties such as color, opacity, and covariance matrix are optimized. Finally, the positions of the Gaussian point cloud are jointly optimized. Ultimately, the present invention obtains a GS point cloud of the occluded area with complete scene representation capabilities.

[0057] In one embodiment, after obtaining the optimized Gaussian point cloud distribution, boundary removal and scene consistency retraining may be performed on the Gaussian points in the Gaussian point cloud distribution to obtain a target Gaussian point cloud distribution.

[0058] Specifically, during scene reconstruction, especially when multiple frames of video data are involved, the reconstructed scene may exhibit some boundary issues. PCN has already been used to address missing or discontinuous 3D point clouds caused by occlusion and changes in perspective. Now, we need to address the issue of the model inaccurately identifying certain objects or areas during the segmentation process, leading to mis-segmentation or missed segmentation. Common unsegmented points include noise points and redundant points. Noise points refer to isolated or invalid points that are unrelated to the rest of the scene. These points typically appear in areas that were misidentified during segmentation. Setting a threshold and removing points with too low a density can effectively remove noise points. For duplicate or redundant points (possibly due to multiple reconstructions during multi-frame synthesis), clustering algorithms such as DBSCAN or K-means can be used to cluster the point cloud and remove duplicate points.

[0059] After boundary removal, the reconstructed scene may still have some consistency issues, meaning that the representation of objects or scenes may be inconsistent across time points. To ensure global consistency of the scene, retraining is often necessary to optimize the scene representation. The original video or image data is processed to regenerate the segmentation results and the corresponding Gaussian point cloud representation. During training, consistency between the segmentation results and the Gaussian point cloud must be ensured. Specifically, the segmented objects should be consistent with the objects in the Gaussian point cloud representation to avoid deviations caused by training errors.

[0060] By removing boundaries and retraining for scene consistency, this method effectively eliminates noise, redundancy, missegmentation, and discontinuities in scene reconstruction, ensuring the final Gaussian point cloud representation is more accurate and consistent. The key to this process lies in using a variety of optimization techniques (such as point cloud completion, image inpainting, and joint optimization) to ensure that object segmentation and point cloud data can seamlessly transition across multiple viewpoints and time points, thereby achieving high-quality scene reconstruction.

[0061] In one embodiment, Figure 3 As shown, the present invention also provides a scene completion method based on Gaussian sputtering, and its complete processing flow is divided into four main steps, specifically: Step 1: Panoptic segmentation and 3D Gaussian initialization.

[0062] Specifically, the input video is processed frame by frame, and then panoramic segmentation is performed to generate a mask, and inter-frame correlation is performed. Then, the Gaussian is initialized, the two-dimensional mask is back-projected, and finally a three-dimensional Gaussian and the corresponding mask are output.

[0063] Step 2: Point cloud completion.

[0064] Specifically, a Gaussian distributed point cloud is input to determine whether the point cloud is correct. If there is any missing point, a complete point cloud is output. If there is a missing point, a point cloud completion model is used to fill the point cloud in the occluded area and output a complete point cloud.

[0065] Step 3: GS attribute optimization.

[0066] Input point cloud, initialize it as GS point and project it onto a two-dimensional plane. If the occluded area does not exist, update the GS attributes and finally output the optimized GS expression. If the occluded area exists, calculate the occluded area mask, perform image completion, compare the GS rendering with the completed image to update the GS attributes and finally output the optimized GS expression.

[0067] Step 4: Boundary removal and consistency retraining.

[0068] Input the optimized GS expression and detect whether boundary noise exists. If so, delete the noise points, update the GS expression, and retrain based on the original video data and the updated GS expression to output a consistent GS expression.

[0069] The present invention solves the problems of incomplete reconstruction and incomplete individual objects caused by using Gaussian sputtering to reconstruct scenes in the prior art, and facilitates downstream interactive applications such as the metaverse.

[0070] When applying the scene completion method based on Gaussian sputtering provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0071] The above is a scene completion method based on Gaussian sputtering provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding scene completion device based on Gaussian sputtering, which includes: An acquisition module is used to acquire the original video of the target scene captured by the camera device; A conversion module is used to perform Gaussian sputtering conversion on the pixels of each frame of the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of multiple three-dimensional Gaussian points; The completion module is used to input the spatial positions of all 3D Gaussian points into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene. The improved point cloud completion model is constructed by introducing features extracted from the contrastive language image pre-training CLIP model and the self-distilled visual transformer DINO model as supervision information into the point cloud completion network PCN model. The rendering module is used to perform color rendering on the complete Gaussian point cloud according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution.

[0072] For the specific definition of the scene completion device based on Gaussian sputtering, please refer to the definition of the scene completion method based on Gaussian sputtering above, which will not be repeated here. The various modules in the above-mentioned scene completion device based on Gaussian sputtering can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0073] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The provided scene completion method based on Gaussian sputtering.

[0074] The present invention also provides Figure 4 The structural diagram of the computer equipment shown in FIG. Figure 4 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The provided scene completion method based on Gaussian sputtering.

[0075] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0076] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A scene completion method based on Gaussian sputtering, characterized in that: include: Obtaining the original video of the target scene captured by the camera device; Perform Gaussian sputtering transformation on each pixel point of each frame in the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of multiple three-dimensional Gaussian points; The spatial positions of all 3D Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene. The improved point cloud completion model is constructed by introducing the features extracted from the contrastive language image pre-training CLIP model and the self-distilled visual transformer DINO model as supervision information into the point cloud completion network PCN model. According to the Gaussian distribution representation, the complete Gaussian point cloud is rendered in color to obtain the optimized Gaussian point cloud distribution.

2. The method according to claim 1, characterized in that Perform Gaussian sputtering transformation on each pixel of each frame in the original video to obtain Gaussian distribution representation, including: Determine the essential matrix based on any two matching points between images in the original video; Decompose the essential matrix to obtain the rotation matrix and translation vector; Convert the two-dimensional pixel points of each image into initial three-dimensional points according to the intrinsic parameter matrix, rotation matrix and translation vector of the camera device; Adjust the position of the initial 3D point according to minimizing the reprojection error on all images to obtain the candidate 3D point; The candidate 3D points are added with Gaussian attributes to obtain the Gaussian distribution representation corresponding to the candidate 3D points.

3. The method according to claim 2, characterized in that Reprojection error Minimize to: ； ； in, is the camera's intrinsic parameter matrix, Indicates the k The rotation matrix of the pixel points, Indicates the k The translation vector of pixels, Indicates the k Two-dimensional pixels, Indicates the k An initial 3D point.

4. The method according to claim 2, characterized in that The Gaussian distribution is expressed as: ； in, is the spatial position of the three-dimensional Gaussian point; is the covariance matrix of the three-dimensional Gaussian points, Is an RGB vector, representing the color or texture information of a three-dimensional Gaussian point; is opacity.

5. The method according to claim 1, wherein The improved point cloud completion model includes a PCN point cloud completion model using the CLIP model, the DINO model, and the Transformer model. The spatial positions of all 3D Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene, including: Input the spatial positions of all 3D Gaussian points into the encoder of the PCN model to obtain point cloud features; The image segmentation results corresponding to all 3D Gaussian points are input into the CLIP model and DINO model respectively to obtain high-level semantic features and pixel-level image features; Obtain the average features of high-level semantic features and pixel-level image features, concatenate the average features with the point cloud features, and input them into the cross-attention module of the Transformer model to obtain the completed point cloud features; The completed point cloud features are input into the decoder of the PCN model to obtain the complete Gaussian point cloud of the target scene.

6. The method according to claim 1, characterized in that According to the Gaussian distribution representation, the complete Gaussian point cloud is rendered in color to obtain the optimized Gaussian point cloud distribution, including: Project the complete Gaussian point cloud onto a two-dimensional plane and detect whether there is an occluded area; In the absence of occlusion areas, the Gaussian points on the two-dimensional plane are rendered in color according to Gaussian sputtering to obtain the optimized Gaussian point cloud distribution; In the presence of occluded areas, the colors of the newly added Gaussian points in the complete Gaussian points are adjusted through image completion technology and Gaussian rendering to obtain the optimized Gaussian point cloud distribution.

7. The method according to claim 6, characterized in that The colors of the newly added Gaussian points in the complete Gaussian points are adjusted through image completion technology and Gaussian rendering to obtain the optimized Gaussian point cloud distribution, including: Calculate the mask of the occluded area, and based on the mask of the occluded area, complete the occluded area of ​​the three-dimensional Gaussian points to obtain the completed image; Render the newly added Gaussian points using Gaussian sputtering technology to obtain a rendered image; Determine the gradient of the loss function based on the rendered image and the completed image; The colors of the newly added Gaussian points are adjusted according to the gradient of the loss function to obtain the optimized Gaussian point cloud distribution.

8. The method according to claim 1, characterized in that After obtaining the optimized Gaussian point cloud distribution, the method further includes: The Gaussian points in the Gaussian point cloud distribution are subjected to boundary removal and scene consistency retraining to obtain the target Gaussian point cloud distribution.

Images