A gaussian sputtering based scene completion method

Abstract

The application discloses a scene completion method based on Gaussian sputtering, and relates to the technical field of scene reconstruction. An original video of a target scene collected by a camera device is acquired; pixel points of each frame of image in the original video are respectively converted through Gaussian sputtering to obtain Gaussian distribution representation; the Gaussian distribution representation comprises spatial positions and color information of a plurality of three-dimensional Gaussian points; the spatial positions of all the three-dimensional Gaussian points are input into an improved point cloud completion model to obtain complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing features extracted from a contrast language image pre-training CLIP model and a self-distillation visual transformer DINO model into a point cloud completion network PCN model as supervision information; and the complete Gaussian point cloud is color rendered according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution. The method can generate a scene with a relatively realistic effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene reconstruction, in particular to a scene completion method based on Gaussian sputtering. BACKGROUND

[0002] In actual scenes (such as indoor complex environments, urban streets or power transmission and distribution line scenes), occlusion between objects is ubiquitous, especially in complex scenes where vegetation, buildings or equipment are superimposed on each other, often resulting in incomplete three-dimensional reconstruction.

[0003] In the prior art, for the completion of missing areas of traditional point clouds, point cloud completion methods based on geometric priors or deep learning are mainly relied on, such as Point Completion Network (PCN). The method based on geometric priors usually infers the missing area by analyzing the local structure of the point cloud, but these methods often have difficulty in capturing the global geometric relationship when dealing with complex scenes. The point cloud completion method based on deep learning uses neural networks to model the global structure of the point cloud, generating more expressive completion results, but this method has difficulty in generating detailed completion results in high-noise or complex scenes.

[0004] Therefore, there is an urgent need in the prior art for a completion method that can generate a more realistic scene. SUMMARY

[0005] Therefore, it is necessary to provide a scene completion method based on Gaussian sputtering, which can generate a more realistic scene.

[0006] The present application adopts the following technical solutions:

[0007] The present application provides a scene completion method based on Gaussian sputtering, comprising:

[0008] obtaining an original video of a target scene collected by a camera device;

[0009] performing Gaussian sputtering conversion on the pixel points of each image in the original video to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial positions and color information of a plurality of three-dimensional Gaussian points;

[0010] inputting the spatial positions of all three-dimensional Gaussian points into an improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing the features extracted from a contrastive language image pre-training CLIP model and a self-distillation visual transformer DINO model into a point cloud completion network PCN model as supervision information;

[0011] According to the Gaussian distribution representation, color rendering is performed on the complete Gaussian point cloud to obtain an optimized Gaussian point cloud distribution.

[0012] Optionally, the pixel points of each image frame in the original video are respectively converted by Gaussian sputtering to obtain a Gaussian distribution representation, including:

[0013] According to any two matching points between images in the original video, an essential matrix is determined.

[0014] The essential matrix is decomposed to obtain a rotation matrix and a translation vector.

[0015] According to the intrinsic matrix of the camera device, the rotation matrix and the translation vector, the two-dimensional pixel points of each image are converted into initial three-dimensional points.

[0016] According to the minimization of the re-projection error on all images, the position of the initial three-dimensional points is adjusted to obtain candidate three-dimensional points.

[0017] The candidate three-dimensional points are added with Gaussian attributes to obtain a Gaussian distribution representation corresponding to the candidate three-dimensional points.

[0018] Optionally, the re-projection error is minimized as:

[0019] ;

[0020] wherein, is an internal parameter matrix of the camera, representing the parameters of the camera hardware, represents the rotation matrix of the k th pixel point, represents the translation vector of the k th pixel point, represents the k th two-dimensional pixel point, represents the k th initial three-dimensional point.

[0021] Optionally, the Gaussian distribution representation is:

[0022] ;

[0023] wherein, is the spatial position of the three-dimensional Gaussian point; is the covariance matrix of the three-dimensional Gaussian point, is an RGB vector representing the color or texture information of the three-dimensional Gaussian point; is the opacity.

[0024] Optionally, the improved point cloud completion model includes a PCN point cloud completion model using a CLIP model and a DINO model and a Transformer model; the spatial positions of all three-dimensional Gaussian points are input into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene, including:

[0025] The spatial positions of all three-dimensional Gaussian points are input into the encoder of the PCN model to obtain point cloud features;

[0026] The segmentation results of the images corresponding to all three-dimensional Gaussian points are respectively input into the CLIP model and the DINO model to obtain high-level semantic features and pixel-level image features;

[0027] The average features of the high-level semantic features and the pixel-level image features are obtained, and the average features are concatenated with the point cloud features and input into the cross-attention module of the Transformer model to obtain completed point cloud features;

[0028] The completed point cloud features are input into the decoder of the PCN model to obtain a complete Gaussian point cloud of the target scene.

[0029] Optionally, according to the Gaussian distribution representation, the complete Gaussian point cloud is color rendered to obtain an optimized Gaussian point cloud distribution, including:

[0030] Project the complete Gaussian point cloud onto a two-dimensional plane and detect whether there is an occlusion area;

[0031] In the case where there is no occlusion area, the Gaussian points under the two-dimensional plane are color rendered according to Gaussian sputtering to obtain an optimized Gaussian point cloud distribution;

[0032] In the case where there is an occlusion area, the color of the new Gaussian points in the complete Gaussian points is adjusted through image completion technology and Gaussian rendering to obtain an optimized Gaussian point cloud distribution.

[0033] Optionally, the color of the new Gaussian points in the complete Gaussian points is adjusted through image completion technology and Gaussian rendering to obtain an optimized Gaussian point cloud distribution, including:

[0034] Calculate the mask of the occlusion area, and based on the mask of the occlusion area, complete the occlusion area of the three-dimensional Gaussian points to obtain a completed image;

[0035] Render the new Gaussian points through Gaussian sputtering technology to obtain a rendered image;

[0036] According to the rendered image and the completed image, determine the gradient of the loss function;

[0037] According to the gradient of the loss function, adjust the color of the new Gaussian points to obtain an optimized Gaussian point cloud distribution.

[0038] Optionally, after obtaining the optimized Gaussian point cloud distribution, the method further comprises:

[0039] Boundary removal and scene consistency retraining are performed on the Gaussian points in the Gaussian point cloud distribution to obtain a target Gaussian point cloud distribution.

[0040] The application provides a scene completion device based on Gaussian sputtering, comprising:

[0041] An acquisition module is configured to acquire an original video of a target scene captured by a camera device.

[0042] A conversion module is configured to perform Gaussian sputtering conversion on each pixel point of each image in the original video to obtain a Gaussian distribution representation, wherein the Gaussian distribution representation comprises spatial positions and color information of a plurality of three-dimensional Gaussian points.

[0043] A completion module is configured to input the spatial positions of all three-dimensional Gaussian points into an improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene, wherein the improved point cloud completion model is constructed by introducing features extracted from a contrastive language-image pre-training (CLIP) model and a self-distillation visual transformer (DINO) model as supervision information into a point cloud completion network (PCN) model.

[0044] A rendering module is configured to perform color rendering on the complete Gaussian point cloud according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution.

[0045] The application provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the scene completion method based on Gaussian sputtering.

[0046] The application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the scene completion method based on Gaussian sputtering when executing the program.

[0047] The above at least one technical solution adopted by the application can achieve the following beneficial effects:

[0048] In the present application, first, the pixel points of each frame image of the original video are converted by Gaussian sputtering conversion to obtain a Gaussian distribution representation containing three-dimensional Gaussian point space position and color information. This conversion method can more flexibly express the geometric and color information in the scene. Each Gaussian point can be regarded as an approximate description of a local area of the image. Compared with the traditional pixel representation, the spatial structure and details of the scene can be better captured, and more abundant information basis is provided for subsequent processing. Further, in the point cloud completion, the supervision information of the pre-trained CLIP model and the self-distillation visual transformer DINO model is combined to further enhance the existing point cloud information and supervision features, so as to accurately generate a complete Gaussian point cloud. Finally, the complete Gaussian point cloud is color rendered according to the Gaussian distribution representation of the original Gaussian point cloud, so that the completed point cloud is not only closer to the real scene in geometric structure, but also can match the information in the original video in color, so that the completed scene is more natural and real in vision. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0050] Figure 1 A Gaussian sputtering-based scene completion method provided by the present application is shown in the flowchart;

[0051] Figure 2 A PCN encoder-decoder architecture provided by the present application is shown in the diagram;

[0052] Figure 3 Another Gaussian sputtering-based scene completion method provided by the present application is shown in the flowchart;

[0053] Figure 4 A computer device for implementing the Gaussian sputtering-based scene completion method provided by the present application is shown in the diagram. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in conjunction with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] As a new three-dimensional scene representation technology, Gaussian Splatting (GS) can achieve efficient, flexible and realistic scene restoration effect. In addition, point cloud completion cannot replace Gaussian Splatting completion, and the generated point cloud model is large in void, which cannot realize the realistic representation effect of three-dimensional Gaussian Splatting. The image completion or image inpainting task aims to automatically fill the missing or damaged areas in the image, so that it is visually natural, reasonable and consistent with the surrounding content. Its advantage is to use the prior of big data to make the filled part natural, rich in texture and realistic, and the disadvantage is that it can only work in the image field, and the filling effect has randomness.

[0056] Consistency optimization needs to be carried out for scene Gaussian representation, otherwise it is easy to cause local distortion or global incoordination problems.

[0057] Therefore, the present application provides a scene completion method based on Gaussian Splatting, which combines point cloud completion method with image completion model, and realizes intelligent completion and reconstruction of the original scene by designing reasonable network framework and loss function to jointly optimize the Gaussian Splatting representation of the scene. This not only can effectively complete the missing area, but also can generate a three-dimensional model with rich details on the basis of maintaining the consistency of the point cloud, thereby providing higher protection for the reality of scene restoration and subsequent application in various industries.

[0058] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0059] Figure 1 A flowchart of a scene completion method based on Gaussian Splatting in the present application is shown, which specifically includes the following steps:

[0060] S101, obtaining an original video of a target scene collected by a camera device.

[0061] S102, performing Gaussian Splatting conversion on pixel points of each image in the original video respectively to obtain Gaussian distribution representation; the Gaussian distribution representation includes spatial positions and color information of a plurality of three-dimensional Gaussian points.

[0062] In one embodiment, Gaussian Splatting conversion is performed on pixel points of each image in the original video respectively to obtain Gaussian distribution representation, including: determining an essential matrix according to any two matching points between images in the original video; decomposing the essential matrix to obtain a rotation matrix and a translation vector; converting two-dimensional pixel points of each image into initial three-dimensional points according to an intrinsic matrix of the camera device, the rotation matrix and the translation vector; adjusting the positions of the initial three-dimensional points according to the minimization of the re-projection error on all images to obtain candidate three-dimensional points; adding Gaussian attributes to the candidate three-dimensional points to obtain Gaussian distribution representation corresponding to the candidate three-dimensional points.

[0063] Adding Gaussian properties to candidate 3D points involves supplementing each candidate 3D point with an initialized Gaussian property value.

[0064] The Segment Anything Model (SAM) is used to generate panoramic segmentation 2D masks for each frame of the original video. Inter-frame target association is then performed using each frame mask to obtain spatiotemporally consistent target segmentation results.

[0065] Simultaneously, keyframes are selected from the original video based on the ORB feature overlap rate and frame rate. The Structure from Motion (SfM) method, such as the existing colmap tool, is used to extract image poses. Since feature extraction and matching have already been completed in SAM, the next step is to perform preliminary 3D reconstruction using epipolar geometry. Given the camera intrinsic matrix K, the essential matrix E describes the matching points between the two images. and Correspondence: .

[0066] Therefore, the essential matrix can be obtained by solving this equation using the eight-point method or a nonlinear method. Then, the rotation matrix is ​​obtained through SVD decomposition. Translation vector And the preliminary three-dimensional points are recovered using triangulation. After initially recovering the 3D points and camera pose, optimization is required. This invention uses 3D points... Projecting onto the image yields 2D points The camera pose and the position of 3D points are optimized by minimizing the reprojection error on all images through bundle adjustment (BA).

[0067] ;

[0068] ;

[0069] in, Indicates the first k A rotation matrix of pixels represents the parameters of the camera hardware. Indicates the first k Translation vector of 1 pixel, Indicates the first k Two-dimensional pixels, Indicates the first k Two-dimensional pixels are generated from an initial 3D point after camera intrinsic and extrinsic parameter conversion. It maps three-dimensional points to two-dimensional points.

[0070] The output of the above SFM method is the extracted camera poses and a sparse 3D point cloud of the scene. Then the sparse point cloud is used to initialize and train a Gauss (GS) model to obtain a GS-based 3D representation of the scene.

[0071] Specifically, each spatial 3D point obtained by SFM is initialized as a Gaussian distribution in the form of:

[0072]

[0073] wherein, is the spatial position of the 3D Gaussian point, initialized by the spatial position of the sparse point; is the covariance matrix of the 3D Gaussian point, representing the extension degree of the 3D Gaussian point in space, initialized as 1 for subsequent adjustment. is an RGB vector representing the color or texture information of the 3D Gaussian point, represented by the spherical harmonic function of the Gaussian, which is initialized as the color corresponding to the point cloud, is the opacity, initialized as 0.5.

[0074] Then, using the projection formula, the above 3D Gaussian point is projected into the camera coordinate system through the Jacobian matrix, and then mapped to the two-dimensional image plane to render the image. When rendering, after sorting all Gaussian points from far to near according to the depth, the final pixel color is calculated using alpha blending:

[0075]

[0076] wherein, represents the transparency accumulation of all previous Gaussian points (to achieve the occlusion effect); represents the color of the i-th Gaussian point; is the opacity of the Gaussian point.

[0077] The position, color, opacity and other parameters of the Gaussian point can be directly optimized by using gradient descent. As for the gradient of the covariance matrix of the Gaussian distribution, the chain rule is applied to convert the gradient of into the gradient of the scaling factor s and the rotation quaternion q parameters. The loss function is the weighted sum of the L1 loss and the D-SSIM loss between the image rendered by the model and the real image :

[0078]

[0079] In an embodiment, a Gaussian-based scene 3D segmentation can also be determined. Specifically, the following steps are included:

[0080] This step passes the 2D mask and its label obtained by the segmentation all models (SAM) to each Gaussian point, so that each Gaussian point has a class label, which is used to merge three-dimensional Gaussian points with the same semantic category. Specifically, the back projection formula of two-dimensional pixel coordinates to spatial three-dimensional points is used, specifically:

[0081]

[0082] wherein (x, y) is the coordinate of the two-dimensional image plane; is the internal parameter of the camera; d is the depth information of the camera; is the 2D mask generated by the SAM large model for the key frame. Then a Gaussian counter is initialized for each Gaussian point, that is, the attributes of the Gaussian point are extended to . The Gaussian counter is used to calculate the label value of each Gaussian point after back projection. Due to the limitation of the SAM algorithm, it is inevitable to assign an incorrect label value to a Gaussian point at a certain view angle. Therefore, the counter form is adopted to save the label of the spatial Gaussian point under multiple two-dimensional views, and then the final object label l of the Gaussian point is obtained according to the maximum value. The screening label formula is as follows (where g represents a three-dimensional Gaussian point; V is the set of all camera views; ) is the label assigned by the sam model at the view v: . .

[0083] S103, input the spatial positions of all three-dimensional Gaussian points into the improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing the features extracted by the contrast language image pre-training CLIP model and the self-distillation visual transformer DINO model into the point cloud completion network PCN model as supervision information.

[0084] Due to the sparsity, loss or occlusion of the scene, the point cloud data of some objects may be incomplete. The task of point cloud completion identifies individual objects one by one, and generates dense and complete object point clouds through an improved point cloud completion model, finally obtaining a complete scene 3D point cloud representation. The point cloud completion model used in the present application is an improved PCN (Point Completion Network) method. PCN adopts an encoder-decoder architecture, as shown in Figure 2 .

[0085] Among them, the encoder is responsible for compressing the point cloud data into a compact global feature vector, obtaining the global feature by extracting the point cloud feature, generating the feature matrix, and performing maximum pooling, and splicing the global feature with the original feature to obtain the final global feature vector. The decoder starts from this global feature vector, gradually recovers the complete point cloud through the coarse stage and the fine stage, the coarse stage uses the full connection network to generate the rough point cloud, and the fine stage adjusts the rough point cloud through neighborhood search and shared multi-layer perception (MLP), and finally generates fine point cloud data.

[0086] In order to improve the accuracy of the GS point cloud completion, the application proposes to use the features extracted by the contrastive language-image pre-training (CLIP) model and the self-distilled vision transformer (DINO) model as additional supervision information and introduce them into the traditional PCN process. The CLIP model is a powerful multi-modal model that can map images and text to the same embedding space. This embedding method not only extracts semantic information of the image, but also provides strong semantic prior for subsequent point cloud completion and reconstruction tasks; the self-distilled vision transformer DINO model can extract deep visual features of the image, which uses an unsupervised learning method to capture spatial information, object structure and depth relationship in the image through a contrastive learning method. These deep visual features complement the geometric information of the image and form a complement with the semantic features.

[0087] Optionally, the improved point cloud completion model includes a PCN point cloud completion model using the CLIP model and the DINO model and the Transformer model; the spatial positions of all three-dimensional Gaussian points are input into the improved point cloud completion model to obtain the complete Gaussian point cloud of the target scene, including: inputting the spatial positions of all three-dimensional Gaussian points into the encoder of the PCN model to obtain point cloud features; inputting the segmentation results of the images corresponding to all three-dimensional Gaussian points into the CLIP model and the DINO model respectively to obtain high-level semantic features and pixel-level image features; obtaining the average features of the high-level semantic features and the pixel-level image features, and inputting the average features into the cross-attention module of the Transformer model after splicing with the point cloud features to obtain the completed point cloud features; inputting the completed point cloud features into the decoder of the PCN model to obtain the complete Gaussian point cloud of the target scene.

[0088] Specifically, first, the three-dimensional Gaussian points with the same semantic label are converted into input point clouds (the coordinate positions and color vectors of the three-dimensional Gaussian points can be simply spliced), and the point cloud features are extracted using the encoder of the PCN ; meanwhile, n corresponding SAM model segmentation result images are randomly selected to extract CLIP and DINO features 、 . Then, the high-level semantic features are averaged with the pixel-level image features , and the point cloud features are input into the cross-attention module of the Transformer model for feature fusion to obtain the completed point cloud features; finally, the completed point cloud is obtained through the decoder of the PCN model. The formula is as follows:

[0089] ;

[0090] ;

[0091] wherein, the input Query of the Transformer model is to ensure that the geometric structure dominates the attention calculation; the Key / Value source is the spliced features , so that the image information affects the geometric features through the attention weight. By performing the cross-attention operation of Query and Key in the Transformer model, the point cloud geometric features fused with high-level semantic and local detail features are obtained. The fused features provide more context information for point cloud completion and 3D reconstruction. The semantic features can provide semantic constraints for the missing areas, and the geometric features can more accurately describe the structural relationship of the point cloud. Through this feature fusion method, the accuracy and robustness of PCN in the point cloud segmentation and 3D reconstruction tasks can be significantly improved.

[0092] The training of the improved PCN point cloud completion model adopts a random mask guided fully supervised training method. The data set adopts CO3D (containing 50 MS-COCO categories of common objects) and ShapeNet (containing 55 categories of common indoor and outdoor objects), wherein 30%-90% of the adjacent point clouds are randomly hidden to simulate the partial point cloud missing caused by insufficient observation or occlusion, and the original complete point cloud is used as the model supervision signal. After training, the improved PCN model has the ability to complete the point cloud of any missing degree, that is, it can be used immediately.

[0093] In S104, the complete Gaussian point cloud is color rendered according to the Gaussian distribution to obtain an optimized Gaussian point cloud distribution.

[0094] The point cloud completion solves the missing part in the Gaussian distribution point cloud, and obtains a complete point cloud expression through completion, and initializes the newly supplemented point cloud as a new three-dimensional Gaussian point. However, these completed point clouds only contain position information , other Gaussian properties (color, transparency, covariance, etc.) need to be optimized to make them more consistent and realistic in the scene.

[0095] In one embodiment, according to the Gaussian distribution representation, the color rendering of the complete Gaussian point cloud is performed to obtain the optimized Gaussian point cloud distribution, including: projecting the complete Gaussian point cloud to a two-dimensional plane, and detecting whether there is an occlusion area; in the case where there is no occlusion area, performing color rendering on the Gaussian points under the two-dimensional plane according to Gaussian sputtering to obtain the optimized Gaussian point cloud distribution; in the case where there is an occlusion area, adjusting the color of the new Gaussian points in the complete Gaussian points through an image completion technology and Gaussian rendering to obtain the optimized Gaussian point cloud distribution.

[0096] Wherein, the color of the new Gaussian points in the complete Gaussian points is adjusted through the image completion technology and the Gaussian rendering to obtain the optimized Gaussian point cloud distribution, including: calculating a mask of the occlusion area, and based on the mask of the occlusion area, completing the occlusion area of the three-dimensional Gaussian points to obtain a completed image; rendering the new Gaussian points through a Gaussian sputtering technology to obtain a rendered image; determining the gradient of a loss function according to the rendered image and the completed image; adjusting the color of the new Gaussian points according to the gradient of the loss function to obtain the optimized Gaussian point cloud distribution.

[0097] In order to detect the missing or occlusion area of the point cloud, it is necessary to project the newly supplemented three-dimensional point cloud to the corresponding area in the two-dimensional image . According to the depth map, it is determined which pixel points belong to the occlusion area in the projected image to obtain the corresponding mask . Then, the existing image completion model inpaint-anything is used to complete the occluded image to obtain a complete image . Since the existing camera images do not contain the occlusion area, it is necessary to perturb the original camera position to sample new camera positions, and to determine whether the unobserved area M in the new pose image is large enough (such as more than 10% of the object area) to decide whether to keep the image .

[0098] For the Gaussian points corresponding to the occluded area, the Gaussian sputtering technology mentioned above is used to render the image .

[0099] The rendered image and the completed image under a certain view angle are compared, and the loss function is:

[0100] .

[0101] During the training process, the gradient of the loss function is passed to the newly added GS point cloud. First, fix the position of the Gaussian point cloud, only optimize its color, opacity, and covariance matrix attributes, and finally jointly optimize the position of the Gaussian point cloud. Finally, the present application obtains a GS point cloud of the occluded area with complete scene representation capability.

[0102] In one embodiment, after obtaining the optimized Gaussian point cloud distribution, boundary removal and scene consistency retraining of the Gaussian points in the Gaussian point cloud distribution can also be performed to obtain a target Gaussian point cloud distribution.

[0103] Specifically, in the scene reconstruction process, especially in the case of multiple video data, the reconstructed scene may have some boundary problems. The above has been processed by PCN for missing or discontinuous 3D point cloud due to occlusion and changes in viewing angle. Now the problem of inaccurate identification of some objects or areas in the segmentation process of the model, leading to missegmentation or missed segmentation, needs to be solved. Common points that are not segmented include noise points and redundant points. Noise points refer to isolated points or invalid points that are unrelated to other parts of the scene. These points usually appear in the misidentified area during segmentation. By setting a threshold to remove points with low density, noise points can be effectively removed. For repeated or redundant points (which may be caused by multiple reconstructions during multi-frame synthesis), clustering algorithms such as DBSCAN or K-means can be used to cluster the point cloud and delete the repeated points.

[0104] After boundary removal, the reconstructed scene may still have some consistency problems, i.e., the expression of objects or scenes between different time points is inconsistent. In order to ensure the global consistency of the scene, it is usually necessary to optimize the expression of the scene by retraining. By processing the original video or image data, the segmentation results and the corresponding Gaussian point cloud representation are regenerated. During the training process, the coordination between the segmentation results and the GS needs to be ensured. That is, the segmented objects should be consistent with the objects in the GS point cloud representation, avoiding deviations caused by training errors.

[0105] Through boundary removal and scene consistency retraining, the present application can effectively eliminate noise, redundancy, missegmentation, and discontinuity in scene reconstruction, ensuring that the final Gaussian point cloud representation has higher precision and consistency. The key to this process is to ensure that the segmentation of objects and point cloud data can seamlessly transition between multiple viewing angles and time points through various optimization methods (such as point cloud completion, image inpainting, and joint optimization), thereby achieving high-quality scene reconstruction.

[0106] In one embodiment, as Figure 3 shown, the present application also provides a Gaussian sputtering-based scene completion method, and its complete processing flow is divided into four main steps, specifically:

[0107] Step one: panoramic segmentation and three-dimensional Gaussian initialization.

[0108] Specifically, the input video is processed frame by frame, then panoramic segmentation is performed to generate a mask, inter-frame correlation is performed, and then Gaussian is initialized, two-dimensional mask back projection is performed, and finally three-dimensional Gaussian and the corresponding mask are output.

[0109] Step two: point cloud completion.

[0110] Specifically, the input Gaussian distribution point cloud is judged to be complete or not, if there is no missing, the complete point cloud is output, if there is missing, the point cloud completion model is used to fill the point cloud in the occluded area, and the complete point cloud is output.

[0111] Step three: GS attribute optimization.

[0112] The input point cloud is initialized as a GS point and projected onto a two-dimensional plane, if the occluded area does not exist, the GS attribute is updated, and finally the optimized GS expression is output; if the occluded area exists, the occluded area mask is calculated, image completion is performed, the GS rendering is compared with the completed image to update the GS attribute, and finally the optimized GS expression is output.

[0113] Step four: boundary removal and consistency retraining.

[0114] The input optimized GS expression is detected to determine whether the boundary noise exists, if the boundary noise exists, the noise points are deleted, the GS expression is updated, and the original video data and the updated GS expression are retrained to output the consistent GS expression.

[0115] The present application solves the problem of incomplete reconstruction and incomplete single object caused by Gaussian sputtering reconstruction of the scene in the prior art, and facilitates downstream interactive applications such as metaverse.

[0116] When the scene completion method based on Gaussian sputtering provided by the present application is applied, the original video data of the target scene can not be obtained according to the Gaussian sputtering of the target scene. Figure 1 The order of execution of each step shown in the figure can be determined as needed, and the present application does not limit the execution order of each step.

[0117] The above is the scene completion method based on Gaussian sputtering provided by one or more embodiments of the present application. Based on the same idea, the present application also provides a corresponding scene completion device based on Gaussian sputtering, which comprises:

[0118] The acquisition module is configured to acquire original video of a target scene collected by a camera device;

[0119] The conversion module is configured to perform Gaussian sputtering conversion on each pixel point of each image in the original video to obtain Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of a plurality of three-dimensional Gaussian points.

[0120] The completion module is configured to input the spatial positions of all three-dimensional Gaussian points into an improved point cloud completion model to obtain a complete Gaussian point cloud of the target scene; the improved point cloud completion model is constructed by introducing features extracted by a contrastive language-image pre-training CLIP model and a self-distillation visual transformer DINO model as supervision information into a point cloud completion network PCN model;

[0121] The rendering module is configured to perform color rendering on the complete Gaussian point cloud according to the Gaussian distribution representation to obtain an optimized Gaussian point cloud distribution.

[0122] The specific limitations of the Gaussian sputtering-based scene completion apparatus can be seen from the limitations of the Gaussian sputtering-based scene completion method described above, and will not be repeated here. Each module in the Gaussian sputtering-based scene completion apparatus described above can be realized by software, hardware, or a combination thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0123] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the above-mentioned Figure 1 The Gaussian sputtering-based scene completion method is provided.

[0124] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the above-mentioned Figure 4 The structure diagram of the computer device is shown in FIG. Figure 4 As shown in FIG. 1, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course can also include other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the above-mentioned Figure 1 The Gaussian sputtering-based scene completion method is provided.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of the methods. In the embodiments of the present application, any reference to memory, storage, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0126] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

Claims

1. A scene completion method based on Gaussian sputtering, characterized in that, include: Acquire the raw video of the target scene captured by the camera equipment; Each pixel in each frame of the original video is transformed by Gaussian sputtering to obtain a Gaussian distribution representation; the Gaussian distribution representation includes the spatial position and color information of multiple three-dimensional Gaussian points; The spatial locations of all three-dimensional Gaussian points are input into the improved point cloud completion model to obtain the complete Gaussian point cloud of the target scene. The improved point cloud completion model is constructed by introducing the features extracted by the contrastive language image pre-trained CLIP model and the self-distillation visual transformer DINO model as supervision information into the point cloud completion network PCN model. Based on the Gaussian distribution representation, the complete Gaussian point cloud is rendered with color to obtain the optimized Gaussian point cloud distribution. The improved point cloud completion model includes a PCN point cloud completion model using CLIP, DINO, and Transformer models. The spatial positions of all 3D Gaussian points are input into the improved point cloud completion model to obtain the complete Gaussian point cloud of the target scene, including: The spatial locations of all three-dimensional Gaussian points are input into the encoder of the PCN model to obtain point cloud features; The segmentation results of the images corresponding to all three-dimensional Gaussian points are input into the CLIP model and the DINO model respectively to obtain high-level semantic features and pixel-level image features; The average features of high-level semantic features and pixel-level image features are obtained, and the average features are concatenated with the point cloud features and then input into the cross-attention module of the Transformer model to obtain the completed point cloud features. The completed point cloud features are input into the decoder of the PCN model to obtain the complete Gaussian point cloud of the target scene.

2. The method according to claim 1, characterized in that, Each pixel in each frame of the original video is subjected to Gaussian sputtering transformation to obtain a Gaussian distribution representation, including: Determine the essential matrix based on any two matching points between images in the original video; The essential matrix is ​​decomposed to obtain the rotation matrix and translation vector; Based on the camera device's intrinsic parameter matrix, rotation matrix, and translation vector, convert each image's two-dimensional pixels into initial three-dimensional points; The positions of the initial 3D points are adjusted by minimizing the reprojection error on all images to obtain candidate 3D points; Adding Gaussian attributes to candidate 3D points yields a Gaussian distribution representation of the candidate 3D points.

3. The method according to claim 2, characterized in that, Reprojection error Minimize to: ； ； in, It is the camera's internal parameter matrix. Indicates the first k Rotation matrix of pixels, Indicates the first k Translation vector of 1 pixel, Indicates the first k Two-dimensional pixels, Indicates the first k An initial three-dimensional point.

4. The method according to claim 2, characterized in that, The Gaussian distribution is represented as: ； in, It is the spatial location of a three-dimensional Gaussian point; It is the covariance matrix of a three-dimensional Gaussian point. It is an RGB vector representing the color or texture information of a three-dimensional Gaussian point; Opacity.

5. The method according to claim 1, characterized in that, Based on the Gaussian distribution representation, color rendering is performed on the complete Gaussian point cloud to obtain the optimized Gaussian point cloud distribution, including: Project the complete Gaussian point cloud onto a two-dimensional plane and detect whether there are occluded regions; In the absence of occlusion areas, Gaussian points in a two-dimensional plane are color-rendered according to Gaussian sputtering to obtain an optimized Gaussian point cloud distribution. In the presence of occluded areas, the colors of newly added Gaussian points in the complete Gaussian points are adjusted by image completion technology and Gaussian rendering to obtain an optimized Gaussian point cloud distribution.

6. The method according to claim 5, characterized in that, By adjusting the colors of newly added Gaussian points within a complete Gaussian point cloud using image completion techniques and Gaussian rendering, an optimized Gaussian point cloud distribution is obtained, including: Calculate the mask of the occluded region, and based on the mask of the occluded region, perform occlusion region completion on the 3D Gaussian points to obtain the completed image; The newly added Gaussian points are rendered using Gaussian sputtering technology to obtain the rendered image; The gradient of the loss function is determined based on the rendered and completed images; The colors of newly added Gaussian points are adjusted based on the gradient of the loss function to obtain the optimized Gaussian point cloud distribution.

7. The method according to claim 1, characterized in that, After obtaining the optimized Gaussian point cloud distribution, the method further includes: The target Gaussian point cloud distribution is obtained by performing boundary removal and scene consistency retraining on the Gaussian points in the Gaussian point cloud distribution.

Images