A 3D scene target culling method and system based on 3DGS

Abstract

This invention discloses a 3D scene target hidden line removal method and system based on 3DGS, belonging to the field of computer vision processing. The method first performs 3D reconstruction using multi-view images to generate an original Gaussian point cloud scene. Based on target text information, the scene is divided into foreground and background point clouds, and the foreground is spatially expanded to generate a 3D soft mask. This mask is projected onto a 2D image space to generate 2D mask images for each viewpoint, and then these are compared with the original image input image hidden line removal model to generate a candidate hidden line removal image set. After the user selects a reference image from the initial viewpoint, the candidate image most similar to the preceding viewpoint is selected from non-initial viewpoints as the final hidden line removal result, and fidelity enhancement is performed. Finally, the parameters of the original Gaussian point cloud are optimized using the hidden line removal results from each viewpoint to obtain the hidden line removal target 3D scene. Based on this invention, users can perform texture reshaping on the structure of the original 3D scene while maintaining the consistency of the 3D scene across multiple views.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing, and in particular relates to a method and system for target cloaking in a 3D scene based on 3DGS. Background Technology

[0002] In fields such as 3D modeling, virtual reality, intelligent transportation, and film and television post-production, users are increasingly demanding precise hidden object removal and seamless editing of specific targets within scenes. Especially in large-scale 3D environments acquired from real-world data or automatically reconstructed, there are often redundant or sensitive elements such as pedestrians, vehicles, billboards, and obstructions. Directly using these elements for interactive displays, content creation, or downstream modeling tasks will negatively impact the visual experience and content expression. However, existing 3D target removal methods largely rely on 2D image restoration or point cloud cropping, making it difficult to accurately locate the target in space and maintain consistency across different viewpoints. Problems such as target residue, blurred edges, and background discontinuities often arise during the removal process, affecting the overall editing effect and the quality of 3D reconstruction.

[0003] In the process of target caching, the target to be caching is usually extracted in three-dimensional space first, and then the background of the caching target is repaired to finally obtain the scene after the target caching. However, this process faces three key technical challenges: (1) Accurate extraction of the target scene to be edited: The appearance of objects in real scenes is complex, the background interference information is rich, and the target may be occluded, scale changed or visually discontinuous from different perspectives, making it difficult to stably map the target detection at the traditional two-dimensional image level to three-dimensional space. How to achieve an efficient and stable target localization mechanism without sacrificing localization accuracy is the basis for achieving high-quality caching; (2) High-quality repair of images from different perspectives: The caching of the target area needs to generate content that is highly coordinated with the surrounding environment while maintaining the original structure and texture style. The background complexity varies greatly between different perspectives, and the texture information of the occluded area is seriously missing. It is necessary to design a repair strategy with semantic understanding and structural completion capabilities to achieve high-fidelity image completion; (3) Multi-view consistency: Traditional image repair methods generate results independently from each perspective, which often leads to problems such as texture discontinuity and inconsistent detail style. To ensure that the final hidden area is seamlessly integrated in three-dimensional space, it is necessary to ensure that the repaired content maintains visual coherence and structural consistency between adjacent viewpoints, thereby effectively improving the overall quality of three-dimensional reconstruction and the stability of user perception. Summary of the Invention

[0004] The purpose of this invention is to solve the problems existing in the prior art and to provide a 3D scene target scavenging method and system based on 3DGS (3DGaussian Splatting). It aims to achieve automated and seamless elimination of targets in 3D scenes by accurately locating the target area and repairing the background across different viewpoints, based on Gaussian point clouds. This provides an efficient and feasible technical path for controllable scavenging of large-scale real scenes.

[0005] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0006] In a first aspect, the present invention provides a method for hidden object removal in a three-dimensional scene based on 3DGS, which includes the following steps:

[0007] S1: Obtain the multi-view image to be processed, perform 3D scene reconstruction on the multi-view image, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm to perform semantic segmentation on the original scene according to the obtained text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden.

[0008] S2: Spatial expansion of the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions;

[0009] S3: Project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0010] S4: Input the 2D original image formed by the projection of the original scene under each viewpoint and its corresponding 2D mask image into the pre-trained image culling model, perform target culling operation on the mask region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is taken as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to finally form the final culling result under that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs.

[0011] S5: The final hidden surface removal results from each viewpoint are used as training data input. Training and updating are performed sequentially according to the preset continuous viewpoint path to optimize the parameters of the original 3D Gaussian point cloud. The trained and optimized 3D Gaussian scene is used as the target 3D scene after hidden surface removal to complete the target hidden surface removal of the 3D scene.

[0012] Based on the above scheme, each step can be implemented in the following preferred manner.

[0013] As a preferred embodiment of the first aspect above, the specific steps of S1 include: constructing an original scene using multi-view images and using it as the basic input for the target culling operation; receiving the target text information to be culled provided by the user through a command line interface; using a three-dimensional segmentation algorithm to decompose the original scene into a foreground Gaussian point cloud and a remaining background Gaussian point cloud based on the target text information to be culled; and finally storing the foreground Gaussian point cloud and the remaining background Gaussian point cloud separately.

[0014] As a preferred embodiment of the first aspect mentioned above, the specific process of generating the three-dimensional soft mask in step S2 is as follows:

[0015] S21: Construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the set of Gaussian point clouds that need to be hidden;

[0016] S22: The initial hard mask is expanded by a preset radius using a Gaussian neighborhood search strategy to form a three-dimensional soft mask.

[0017] As a preferred embodiment of the first aspect above, in step S22, the three-dimensional soft mask is composed of Gaussian points in the original scene that satisfy the mask generation conditions; the mask generation conditions are: for a Gaussian point in the original scene, there exists at least one Gaussian point in the initial hard mask, such that the positional distance between the two Gaussian points in the three-dimensional space is less than a preset radius.

[0018] Further, in step S22, the three-dimensional soft mask The generation process is specifically represented as follows:

[0019]

[0020] in, Indicates the initial hard mask; Represents the original scene; The preset radius size; This represents a Gaussian point in the original scene; This represents a Gaussian point in the initial hard mask; This represents the distance between two Gaussian points in three-dimensional space.

[0021] As a preferred embodiment of the first aspect mentioned above, the specific process of generating the two-dimensional mask image in step S3 is as follows:

[0022] S31: Remove the Gaussian points belonging to the 3D soft mask in the original scene to form a 3D Gaussian point cloud scene after removing the foreground. Project the 3D Gaussian point cloud scene after removing the foreground onto each viewpoint to obtain the image after removing the foreground under the corresponding viewpoint. Then project the original scene onto each viewpoint to obtain the 2D original image under the corresponding viewpoint.

[0023] S32: Subtract the original two-dimensional image and the image after removing the foreground from each other pixel by pixel from the same viewpoint to generate a difference map from that viewpoint. Then compare the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the pixel is set to 1. If the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the pixel is set to 0, thereby generating a two-dimensional mask map from that viewpoint.

[0024] Further, in step S32, the first Two-dimensional mask image from a different perspective The definition is as follows:

[0025]

[0026] in, This is a preset pixel difference threshold; Number of viewpoints; Indicates the first Two-dimensional original image from a single perspective; express The position in two-dimensional coordinates The pixel value at that pixel; Indicates the first Images after removing the foreground from each viewpoint; express The position in two-dimensional coordinates The pixel value at that pixel; This indicates taking the absolute value.

[0027] As a preferred embodiment of the first aspect above, in step S4, a diffusion model is used as the image hidden surface removal model.

[0028] As a preferred embodiment of the first aspect above, in step S4, the CLIP image similarity function is used to measure the similarity between the candidate hidden image from the non-initial viewpoint and the final hidden image from the preceding viewpoint.

[0029] As a preferred embodiment of the first aspect, in S4, the specific process of performing non-mask region fidelity enhancement processing on the final hidden image is as follows: the region in the original two-dimensional image under a viewpoint that is not covered by its corresponding two-dimensional mask image is directly copied to the final hidden image under that viewpoint to form the final hidden image under that viewpoint.

[0030] Further, in step S4, the first The final caching result from each perspective In two-dimensional coordinates pixel value at It can be represented as:

[0031]

[0032] in, Indicates the first A two-dimensional mask image from a single perspective in two-dimensional coordinates The mask value at that location; Indicates the first The final hidden image from each viewpoint in two-dimensional coordinates Pixel value at; Indicates the first A two-dimensional original image from a single perspective in two-dimensional coordinates The pixel value at that location.

[0033] Secondly, the present invention provides a 3D scene target hidden surface removal system based on 3DGS, which includes the following steps:

[0034] The scene extraction module is used to acquire multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, and use a 3D segmentation algorithm to semantically segment the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden.

[0035] The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions.

[0036] The mask generation module is used to project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0037] The consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene from each viewpoint and its corresponding two-dimensional mask image into the pre-trained image culling model, perform target culling operation on the masked region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is used as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to form the final culling result for that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs.

[0038] The result acquisition module is used to take the final hidden surface removal results from each viewpoint as training data input, and to perform training and updates sequentially according to the preset continuous viewpoint path. It optimizes the parameters of the original 3D Gaussian point cloud, and uses the trained and optimized 3D Gaussian scene as the target 3D scene after hidden surface removal, thus completing the target hidden surface removal of the 3D scene.

[0039] Thirdly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the 3DGS-based method for 3D scene target culling as described in any of the solutions in the first aspect above.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] This invention introduces a 3D scene target caching framework based on 3DGS, employing a 3D segmentation algorithm and a mask expansion strategy to obtain the objects requiring caching. These objects are then projected into a 2D space and fed into an image caching diffusion model for background restoration. Specifically, a 3D segmentation algorithm is first used to semantically divide the scene, extracting the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be cached. The extracted foreground Gaussian point cloud is spatially expanded to generate a 3D soft mask covering the foreground boundary and its adjacent areas, enhancing the integrity of the caching region. The expanded 3D soft mask is then projected into a multi-view 2D image space to obtain a 2D mask map, accurately locating the image region requiring target caching. Subsequently, the original 2D image and the 2D mask map from each viewpoint are input into the image caching model to perform target caching operations on the masked regions. A user-guided mechanism is introduced during the caching process to select a satisfactory caching result. Finally, following a preset continuous viewpoint path, the target caching results are replaced and updated viewpoint by viewpoint, and the caching-reduced 3D scene is obtained. Based on this invention, users can efficiently complete target caching in complex 3D scenes while maintaining high-quality background and multi-view consistency.

[0042] Therefore, the method of this invention fully leverages the structural advantages of explicit representation by 3D Gaussian Splatting (3DGS), integrating mask prediction, mask-guided image-level hidden surface removal, and a structure-aware similarity screening mechanism between viewpoints, achieving high consistency and naturalness in multi-view target hidden surface removal results. Through multi-stage process collaboration, the geometric alignment and texture continuity of target hidden surface removal in the 3D reconstruction process are systematically improved, effectively alleviating the problems of inconsistent hidden surface removal regions and significant artifacts in traditional methods, demonstrating excellent fidelity and structural restoration capabilities in 3D scene target removal tasks. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the target hiding process of the method of the present invention;

[0045] Figure 3 This is a comparison diagram of the editing results of this invention and other target closing methods from different perspectives;

[0046] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0047] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0048] In this embodiment, the technical background of the present invention will first be described, which can be summarized in the following aspects:

[0049] (1) Target caching in 3D space: Target caching aims to remove specified objects from an image or scene while maintaining the natural transition and visual rationality of the background. At the 2D image level, target caching is usually achieved through segmentation, masking, and region restoration. However, in 3D space, due to the geometric depth and multi-view characteristics of object distribution, the mask information from a single viewpoint is insufficient to support comprehensive caching, and joint modeling using multi-view data is necessary. 3D target caching needs to consider complex factors such as the geometric contour of the object in space, the consistency of projection in various viewpoints, and the occlusion relationship between viewpoints.

[0050] (2) Background Inpainting: After the target is masked, the background of the original target area needs to be filled in to ensure a natural transition in terms of texture details, structural trends, and lighting consistency. Background inpainting is essentially a conditional generation task, the goal of which is to synthesize the content of the missing area under the guidance of the image context. Traditional methods such as PatchMatch and image propagation algorithms perform well in simple backgrounds, but in scenes with complex textures or strong semantic structures, the generated results are prone to logical inconsistencies or visual abruptness. In recent years, diffusion models have shown great potential in the field of image inpainting due to their excellent generation performance. By introducing noise and progressively denoising in reverse, diffusion models can learn high-quality content generation paths in complex semantic spaces, making them particularly suitable for inpainting tasks with large missing areas. For background inpainting problems in 3D scenes, diffusion models can perform independent inpainting in images from different perspectives and combine multi-view context information to achieve high-fidelity background generation.

[0051] (3) Multi-view consistency in 3D tasks: In 3D scene modeling and editing tasks, multi-view consistency is a fundamental constraint, requiring that the image representation of the same 3D point under different viewpoints should maintain consistency in geometric position, color texture, and structural semantics. Lack of consistency between viewpoints will directly lead to problems such as drift, texture breakage, or structural distortion in the reconstructed scene. Traditional methods often use geometric priors (such as SFM or MVS) for multi-view registration and fuse information between different images based on manually designed feature comparison strategies, but it is difficult to maintain stable registration results in complex lighting, occlusion, and low-texture areas. For hidden scene removal, the background restoration results of each viewpoint must be consistent in content; otherwise, obvious visual artifacts will be generated at the fusion boundary when they are merged back into the 3D scene. In addition, the differences in detail style between viewpoints will also be significantly amplified, thereby destroying the immersiveness and realism of the entire 3D scene.

[0052] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned 3DGS-based three-dimensional scene target culling method includes the following steps S1 to S5. The specific implementation process of each step will be described in detail below.

[0053] S1: Acquire the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm (Segment Any 3D Gaussians, SAGA) to perform semantic segmentation on the original scene based on the obtained text information of the target to be hidden, extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden, and use them for subsequent target hidden removal operations.

[0054] It should be noted that, in this invention, the specific steps of S1 include: constructing an original scene using multi-view images and using it as the basic input for the target culling operation; receiving the target text information to be culled provided by the user through a command line interface; using a 3D segmentation algorithm to decompose the original scene into a foreground Gaussian point cloud and a remaining background Gaussian point cloud based on the target text information; and finally storing the foreground Gaussian point cloud and the remaining background Gaussian point cloud separately to provide input data for subsequent mask construction and rendering.

[0055] In this embodiment, as Figure 2 As shown, in the first step of extracting the target scene, the part selected by the dashed line represents the internal execution process of the 3D segmentation algorithm (i.e., Figure 2 The training original scene + training segmentation features, after rendering, can form a foreground Gaussian point cloud ( Figure 2 (foreground) and remaining background Gaussian point cloud ( Figure 2 (Background in the text). In addition, the implementation method of the 3D segmentation algorithm belongs to the existing technology. For details, please refer to the literature Cen J, Fang J, Yang C, et al. Segment any 3d gaussians[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(2): 1971-1979, which will not be elaborated here.

[0056] In this embodiment, the foreground Gaussian point cloud With the remaining background Gaussian point cloud At the same time, the intersection of the two sets is an empty set. The union of the two sets represents the original scene. The specific formula is as follows:

[0057]

[0058]

[0059] in, This represents the union operation; This indicates the intersection operation.

[0060] S2: Spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions.

[0061] It should be noted that in step S2 of the present invention, in order to enhance the integrity of the hidden surface region, the extracted foreground Gaussian point cloud is spatially expanded to generate a three-dimensional soft mask covering the foreground boundary and its adjacent regions, so as to alleviate the artifact problem caused by edge residue and improve the robustness of the hidden surface process.

[0062] In this invention, the specific process of generating the three-dimensional soft mask in step S2 is as follows:

[0063] S21: Construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the set of Gaussian point clouds that need to be hidden;

[0064] S22: The initial hard mask is expanded by a preset radius using a Gaussian neighborhood search strategy to form a three-dimensional soft mask.

[0065] In step S22 of this invention, the three-dimensional soft mask is composed of Gaussian points in the original scene that satisfy the mask generation condition; the mask generation condition is: for a Gaussian point in the original scene... There is at least one Gaussian point in the initial hard mask. This makes the two Gaussian points (i.e. and The position distance in three-dimensional space is less than the preset radius.

[0066] In step S22 of this embodiment, the three-dimensional soft mask The generation process is specifically represented as follows:

[0067]

[0068] in, Indicates the initial hard mask; Represents the original scene; The preset radius size; This represents a Gaussian point in the original scene; This represents a Gaussian point in the initial hard mask; This represents the distance between two Gaussian points in three-dimensional space.

[0069] The aforementioned 3D soft mask not only ensures a smoother connection between the foreground and background throughout the scene, but also mitigates errors in extracting the target scene to some extent. It prevents the entire editing scene from being paralyzed due to flaws in scene extraction, thus improving visual effects while maintaining high robustness.

[0070] S3: Project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0071] In this invention, in step S3, the generated two-dimensional mask image can accurately pinpoint the image region requiring target hidden area removal. The specific generation process is as follows:

[0072] S31: Remove the Gaussian points belonging to the 3D soft mask in the original scene to form a 3D Gaussian point cloud scene after removing the foreground. Project the 3D Gaussian point cloud scene after removing the foreground onto each viewpoint to obtain the image after removing the foreground under the corresponding viewpoint. Then project the original scene onto each viewpoint to obtain the 2D original image under the corresponding viewpoint.

[0073] S32: Subtract the original two-dimensional image and the image after removing the foreground from each other pixel by pixel from the same viewpoint to generate a difference map from that viewpoint. Then compare the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the pixel is set to 1. If the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the pixel is set to 0, thereby generating a two-dimensional mask map from that viewpoint.

[0074] In this embodiment, the first Two-dimensional mask image from a different perspective It is a binary image (taking values ​​of 0 or 1) used to distinguish between foreground and background. If the... Two-dimensional original image from a different perspective In two-dimensional coordinates The pixel value at that position, compared with the pixel value at the 1st position. Images after foreground removal from each viewpoint The absolute difference in pixel values ​​at the same location is greater than a preset pixel difference threshold. If the foreground is specified, the mask value at that location is set to 1; otherwise, the mask value is set to 0 (indicating the background). Simply put, this 2D mask image identifies regions of significant change (potentially foreground objects) by comparing the pixel differences between the original 2D image and the image after removing the foreground, and uses a pixel difference threshold to control the sensitivity. Therefore, the above 2D mask image... The definition is as follows:

[0075]

[0076] in This is a preset pixel difference threshold; Number of viewpoints; express The position in two-dimensional coordinates The pixel value at that pixel; express The position in two-dimensional coordinates The pixel value at that pixel; This indicates taking the absolute value.

[0077] S4: Input the 2D original image formed by the projection of the original scene under each viewpoint and its corresponding 2D mask image into the pre-trained image culling model, perform target culling operation on the mask region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is taken as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to finally form the final culling result under that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs.

[0078] It should be noted that the image hidden surface removal model in this invention can be selected by those skilled in the art according to actual needs, as long as it can complete the target hidden surface removal task. In this embodiment, a diffusion model is used as the above-mentioned image hidden surface removal model. The diffusion model includes a text encoder, an image encoder-decoder, and a U-shaped network (i.e., UNet). This model is pre-trained on a large amount of text image data. Given the input image to be edited and a mask, it can perform local image generation in the mask region. The specific implementation of the diffusion model is prior art and will not be described in detail here.

[0079] It should be noted that in step S4 of the present invention, the CLIP image similarity function is used to measure the similarity between the candidate hidden image of the non-initial view and the final hidden image of the preceding view.

[0080] In this embodiment, step S3 corresponds to Figure 2 The second step—expanding the 3D mask—corresponds to step S4. Figure 2 The third step in the process is multi-perspective consistency editing. During multi-perspective consistency editing, for the... Each perspective, the image from that perspective is the original two-dimensional image. and two-dimensional mask image mask Input the image hidden surface removal model, and then call the image hidden surface removal model to perform target hidden surface removal, for the first... Each viewpoint generates a set of candidate hidden image sets. . These are the first and second images in the candidate hidden image set. Candidate hidden image, This represents the number of candidate hidden-face images in the candidate hidden-face image set. Then, in this embodiment, a user-guided mechanism is introduced during the hidden-face removal process in step S4. Specifically, after the image hidden-face removal model generates multiple candidate hidden-face images for each viewpoint, the user first starts from a specified initial viewpoint. ( )of Choose the candidate hidden image that best meets your personalized needs from among the candidate hidden images as the reference image. This option can be used for guided similarity assessments of subsequent perspectives. For non-initial perspectives... ( Then, from the set of candidate hidden image generated by it... The system automatically selects the optimal candidate hidden image as the final hidden image. So that it is consistent with the initial perspective The result (i.e., the reference image) The highest similarity was found in CLIP image features. Similarly, for subsequent non-initial viewpoints... ( All of these can be derived from the candidate hidden image generated by it. The optimal candidate hidden image is automatically selected as the first hidden image. The final hidden image from each perspective , so that it is consistent with the first A perspective The final hidden image The highest similarity is achieved between them to improve the consistency and stability of cross-view hidden surface removal. The objective optimization function is as follows:

[0081]

[0082] in, Indicates the first Any candidate hidden image from any viewpoint; Indicates calculation and CLIP image similarity between them; This represents a candidate hidden image for finding the objective function that maximizes the objective function.

[0083] Therefore, the user guidance mechanism designed in this invention can maintain the quality of hidden surface removal while ensuring the semantic consistency of the target hidden surface removal area under adjacent viewpoints to the greatest extent, thereby improving the visual stability of the scene after 3D reconstruction.

[0084] It should be noted that in step S4, considering the noise disturbance and detail fluctuation in the diffusion model during image generation, in order to enhance the stability and detail preservation of the final hidden image, the present invention performs non-mask region fidelity enhancement processing on the final hidden image: that is, the region in the two-dimensional original image under a viewpoint that is not covered by its corresponding two-dimensional mask image is directly copied to the final hidden image under that viewpoint to form the final hidden image under that viewpoint.

[0085] In this embodiment, the first The final caching result from each perspective For example, when the two-dimensional mask image of this viewpoint is in two-dimensional coordinates mask value at When it is 1, the first The pixel value at this coordinate in the final hidden surface removal result from each viewpoint The final hidden surface image for this viewpoint in two-dimensional coordinates pixel value at When the first A two-dimensional mask image from a single perspective in two-dimensional coordinates When the mask value at point 0, the first The final hidden surface removal result at each viewpoint has pixel values ​​at that coordinate as the original 2D image of that viewpoint in 2D coordinates. pixel value at Specifically, it is expressed as follows:

[0086]

[0087] This operation can further enhance the realism and structural continuity of the background region without affecting the quality of the hidden area generation.

[0088] S5: The final hidden surface removal results from each viewpoint are used as training data input. Training and updating are performed sequentially according to the preset continuous viewpoint path to optimize the parameters of the original 3D Gaussian point cloud. The trained and optimized 3D Gaussian scene is used as the target 3D scene after hidden surface removal to complete the target hidden surface removal of the 3D scene.

[0089] It should be noted that in step S5, image hidden removal is performed on all viewpoints according to the aforementioned steps S1 to S4, and the final hidden removal results for all viewpoints are obtained. As input for training data, it is used to update the properties of the original 3D Gaussian point cloud. Following a preset continuous viewpoint path, the target hidden surface removal results are replaced and updated view-by-view, ensuring that changes between adjacent views are slow, allowing the target Gaussian point cloud to continuously adapt and update. The final output is the trained and optimized 3D Gaussian scene. This step ensures structural consistency and visual coherence across multiple viewpoints, avoiding artifacts or floating points caused by inconsistent caching.

[0090] The 3DGS-based 3D scene target hidden surface removal method described in steps S1-S5 of the above preferred implementation will be applied to a specific dataset for testing. The specific steps are as described in S1-S5 and will not be repeated here. The main focus is on demonstrating the specific parameters and technical effects.

[0091] Example

[0092] Following the implementation process of steps S1-S5 described above, a 3D scene dataset collected from the real world is first acquired for 3D scene reconstruction, followed by rendering of multi-view images. During target caching, a 3D segmentation algorithm is first used to semantically segment the original scene, extracting the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be cached. The extracted foreground Gaussian point cloud is spatially expanded to generate a 3D soft mask covering the foreground boundary and its adjacent areas, enhancing the integrity of the caching region. Then, the expanded 3D soft mask is projected into the multi-view 2D image space to obtain a 2D mask image, accurately calibrating the image region requiring target caching. Subsequently, the original 2D image and the 2D mask image from each viewpoint are input into the image caching model to perform target caching operations on the masked regions. A user-guided mechanism is introduced during the caching process to select a satisfactory caching result. Finally, following a preset continuous viewpoint path, the target caching results are replaced and updated view-by-view, and the caching-reduced 3D target scene is obtained.

[0093] To quantify the metrics, this invention is compared with two existing methods, IN2N(GS) and GaussianEditor. Two metrics are selected: CLIP text-image orientation similarity and CLIP orientation consistency. The test results of the above methods on the 360_v2 dataset are shown in Table 1. The editing effects of the method of this invention and the two existing methods are compared as follows: Figure 3 As shown.

[0094] Table 1. Quantitative Evaluation Table of Editing Effectiveness

[0095]

[0096] It should also be noted that the 3DGS-based 3D scene target cloaking method in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a 3DGS-based 3D scene target cloaking system corresponding to the 3DGS-based 3D scene target cloaking method provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0097] The scene extraction module is used to acquire multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, and use a 3D segmentation algorithm to semantically segment the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden.

[0098] The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions.

[0099] The mask generation module is used to project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0100] The consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene from each viewpoint and its corresponding two-dimensional mask image into the pre-trained image culling model, perform target culling operation on the masked region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is used as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to form the final culling result for that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs.

[0101] The result acquisition module is used to take the final hidden surface removal results from each viewpoint as training data input, and to perform training and updates sequentially according to the preset continuous viewpoint path. It optimizes the parameters of the original 3D Gaussian point cloud, and uses the trained and optimized 3D Gaussian scene as the target 3D scene after hidden surface removal, thus completing the target hidden surface removal of the 3D scene.

[0102] It is understood that the 3DGS-based target cloaking method described in S1-S5 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the 3DGS-based target cloaking method provided in the above embodiments, which includes a computer program / instructions. When executed by a processor, the computer program / instructions can implement the 3DGS-based target cloaking method as described in the above embodiments.

[0103] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the 3DGS-based three-dimensional scene target closing method provided in the above embodiments, which includes a memory and a processor;

[0104] The memory is used to store computer programs;

[0105] The processor is configured to implement the 3DGS-based three-dimensional scene target culling method in the above embodiments when executing the computer program.

[0106] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0107] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the 3DGS-based three-dimensional scene target culling method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can implement the 3DGS-based three-dimensional scene target culling method in the above embodiments.

[0108] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S5.

[0109] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0110] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0111] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for hidden target removal in a 3D scene based on 3DGS, characterized in that, Includes the following steps: S1: Obtain the multi-view image to be processed, perform 3D scene reconstruction on the multi-view image, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm to perform semantic segmentation on the original scene according to the obtained text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden. S2: Spatial expansion of the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions; S3: Project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground. S4: Input the 2D original image formed by the projection of the original scene under each viewpoint and its corresponding 2D mask image into the pre-trained image culling model, perform target culling operation on the mask region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is taken as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to finally form the final culling result under that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs. S5: Use the final hidden surface removal results from each viewpoint as training data input, and perform training and updates sequentially according to the preset continuous viewpoint path to optimize the parameters of the original 3D Gaussian point cloud. Use the trained and optimized 3D Gaussian scene as the target 3D scene after hidden surface removal to complete the target hidden surface removal of the 3D scene. In step S4, a diffusion model is used as the image hidden surface removal model.

2. The 3D scene target hidden surface removal method based on 3DGS as described in claim 1, characterized in that, The specific steps of S1 include: constructing the original scene using multi-view images and using it as the basic input for the target culling operation; receiving the target text information to be culled provided by the user through the command line interface; using a three-dimensional segmentation algorithm to decompose the original scene into foreground Gaussian point clouds and remaining background Gaussian point clouds based on the target text information to be culled; and finally storing the foreground Gaussian point clouds and the remaining background Gaussian point clouds respectively.

3. The method for hidden target removal in a 3D scene based on 3DGS as described in claim 1, characterized in that, The specific process of generating the three-dimensional soft mask in step S2 is as follows: S21: Construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the set of Gaussian point clouds that need to be hidden; S22: The initial hard mask is expanded by a preset radius using a Gaussian neighborhood search strategy to form a three-dimensional soft mask.

4. The 3DGS-based method for target hidden line removal in a three-dimensional scene as described in claim 3, characterized in that, In step S22, the three-dimensional soft mask is composed of Gaussian points in the original scene that satisfy the mask generation conditions; the mask generation conditions are: for a Gaussian point in the original scene, there exists at least one Gaussian point in the initial hard mask, such that the positional distance between the two Gaussian points in the three-dimensional space is less than a preset radius.

5. The method for hidden target removal in a 3D scene based on 3DGS as described in claim 1, characterized in that, The specific process of generating the two-dimensional mask image in step S3 is as follows: S31: Remove the Gaussian points belonging to the 3D soft mask in the original scene to form a 3D Gaussian point cloud scene after removing the foreground. Project the 3D Gaussian point cloud scene after removing the foreground onto each viewpoint to obtain the image after removing the foreground under the corresponding viewpoint. Then project the original scene onto each viewpoint to obtain the 2D original image under the corresponding viewpoint. S32: Subtract the original two-dimensional image and the image after removing the foreground from each other pixel by pixel from the same viewpoint to generate a difference map from that viewpoint. Then compare the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the pixel is set to 1. If the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the pixel is set to 0, thereby generating a two-dimensional mask map from that viewpoint.

6. The method for hidden target removal in a 3D scene based on 3DGS as described in claim 1, characterized in that, In step S4, the CLIP image similarity function is used to measure the similarity between the candidate hidden image from the non-initial viewpoint and the final hidden image from the preceding viewpoint.

7. The method for hidden target removal in a 3D scene based on 3DGS as described in claim 1, characterized in that, In S4, the specific process of performing non-mask region fidelity enhancement processing on the final hidden image is as follows: the region in the original two-dimensional image under a viewpoint that is not covered by its corresponding two-dimensional mask image is directly copied to the final hidden image under that viewpoint to form the final hidden image under that viewpoint.

8. A 3D scene target hidden surface removal system based on 3DGS, characterized in that, Includes the following steps: The scene extraction module is used to acquire multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, and use a 3D segmentation algorithm to semantically segment the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden. The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent regions. The mask generation module is used to project the extended 3D soft mask onto the multi-view 2D image space, and generate a 2D mask map corresponding to each view by utilizing the pixel differences between the images projected from the original scene and the 3D Gaussian point cloud scene after removing the foreground. The consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene from each viewpoint and its corresponding two-dimensional mask image into the pre-trained image culling model, perform target culling operation on the masked region, and generate a set of candidate culling images for each viewpoint. Each set of candidate culling images consists of multiple candidate culling images. After obtaining the reference image selected by the user for the initial viewpoint, for non-initial viewpoints, the candidate culling image with the highest similarity to the final culling image of its respective preceding viewpoint is used as the final culling image. The final culling image is then subjected to fidelity enhancement processing on the non-masked region to form the final culling result for that viewpoint. The reference image is the candidate culling image that the user selects from the set of candidate culling images for the initial viewpoint that best meets their personalized needs. The result acquisition module is used to take the final hidden surface removal results from each viewpoint as training data input, and to perform training and updates sequentially according to the preset continuous viewpoint path. It optimizes the parameters of the original 3D Gaussian point cloud, and uses the trained and optimized 3D Gaussian scene as the target 3D scene after hidden surface removal, thus completing the target hidden surface removal of the 3D scene. In the consistency editing module, the diffusion model is used as the image hidden surface removal model.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the 3DGS-based target closing method as described in any one of claims 1 to 7.

Images