Three-dimensional scene target blanking method and system based on 3DGS

Abstract

The invention discloses a three-dimensional scene target blanking method and system based on 3DGS, and belongs to the field of computer vision processing. The method comprises the following steps: firstly, performing three-dimensional reconstruction through a multi-view image to generate an original Gaussian point cloud scene; after a scene is divided into foreground and background point clouds based on target text information, spatial expansion is performed on the foreground to generate a three-dimensional soft mask. And projecting the mask to a two-dimensional image space to generate a two-dimensional mask image of each view angle, and inputting the two-dimensional mask image and an original image into an image blanking model to generate a candidate blanking image set. After a user selects a reference image from an initial view angle, a candidate image most similar to a preorder view angle is selected as a final blanking result from a non-initial view angle, and fidelity enhancement is carried out. And finally, optimizing the original Gaussian point cloud parameters by using the blanking result of each view angle to obtain a target three-dimensional scene after blanking. Based on the invention, a user can carry out texture remodeling on the structure of the original three-dimensional scene, and the multi-view consistency of the three-dimensional scene is maintained.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing, and in particular relates to a three-dimensional scene target hiding method and system based on 3DGS. Background Art

[0002] In the fields of 3D modeling, virtual reality, intelligent transportation, film and television post-production, etc., users have an increasing demand for accurate hiding and seamless editing of specific targets in the scene. Especially in large-scale 3D environments acquired through real-world acquisition or automatic reconstruction, they often contain redundant or sensitive elements such as pedestrians, vehicles, billboards, and obstructions. If used directly for interactive display, content creation, or downstream modeling tasks, it will affect the visual experience and content expression. However, existing 3D target hiding methods mostly rely on 2D image restoration or point cloud cropping, which makes it difficult to accurately locate the position of the hidden target in space, and even more difficult to maintain consistency under different perspectives. Problems such as target residue, blurred edges, and background discontinuity often occur during the hiding process, affecting the overall editing effect and 3D restoration quality.

[0003] In the process of target fading, the target to be fading is usually extracted in three-dimensional space first, and then the background of the fading target is repaired, and finally the scene after the target is fading is obtained. However, this process faces three key technical challenges: (1) Accurate extraction of the target scene to be edited: The appearance of objects in real scenes is complex, the background interference information is rich, and the targets may be occluded, scale-changed or visually discontinuous at different perspectives, making it difficult to stably map the target detection at the traditional two-dimensional image level to the three-dimensional space. How to achieve an efficient and stable target positioning mechanism without sacrificing positioning accuracy is the basis for achieving high-quality fading; (2) High-quality restoration of images from different perspectives: The fading of the target area requires generating content that is highly coordinated with the surrounding environment while maintaining the original structure and texture style. The background complexity varies greatly between different perspectives, and the texture information of the occluded area is seriously missing. It is necessary to design a restoration strategy with semantic understanding and structure completion capabilities to achieve high-fidelity image completion; (3) Multi-perspective consistency: Traditional image restoration methods generate results independently at each perspective, which often leads to problems such as texture discontinuity and inconsistent detail style. To ensure that the final blanked area is seamlessly integrated in the three-dimensional space, it is necessary to ensure that the repaired content maintains visual coherence and structural consistency between adjacent perspectives, thereby effectively improving the overall three-dimensional reconstruction quality and user perception stability. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a three-dimensional scene target elision method and system based on 3DGS (3D Gaussian Splatting). The method aims to realize the automatic and traceless elimination of targets in three-dimensional scenes by accurately positioning the target area and repairing the background across perspectives based on Gaussian point clouds, thus providing an efficient and feasible technical path for controllable elision in large-scale real scenes.

[0005] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for 3D scene object hiding based on 3DGS, which comprises the following steps:

[0007] S1: Acquire the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm to perform semantic segmentation on the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud corresponding to the target to be hidden and the remaining background Gaussian point cloud;

[0008] S2: spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent areas;

[0009] S3: Project the expanded 3D soft mask into the multi-view 2D image space, and use the pixel difference between the projected images of the original scene and the 3D Gaussian point cloud scene after removing the foreground to generate a 2D mask corresponding to each view.

[0010] S4: Input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into the pre-trained image hidden model, perform a target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, the non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at this viewing angle. The reference image is the candidate hidden image selected by the user from the candidate hidden image set for the initial viewing angle that best meets his or her own personalized needs.

[0011] S5: The final hidden point cloud fading results under each viewing angle are used as training data input. Training updates are performed sequentially according to the preset continuous viewing angle path to optimize the original 3D Gaussian point cloud parameters. The optimized 3D Gaussian scene is used as the target 3D scene after hidden point cloud fading to complete the target hidden point cloud fading of the 3D scene.

[0012] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0013] As a preferred embodiment of the above-mentioned first aspect, the specific steps of S1 include: using multi-view images to construct the original scene and using it as the basic input for the target hiding operation, receiving the target text information to be hidden provided by the user through the command line interface, using a three-dimensional segmentation algorithm to decompose the original scene into a foreground Gaussian point cloud and a remaining background Gaussian point cloud according to the target text information to be hidden, and finally storing the foreground Gaussian point cloud and the remaining background Gaussian point cloud separately.

[0014] As a preferred embodiment of the first aspect, the specific process of generating the three-dimensional soft mask in step S2 is as follows:

[0015] S21: construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the Gaussian point cloud set that needs to be blanked;

[0016] S22: Using a Gaussian neighborhood search strategy, the initial hard mask is expanded by a preset radius to form a three-dimensional soft mask.

[0017] As a preferred embodiment of the above-mentioned first aspect, in step S22, the three-dimensional soft mask is composed of Gaussian points in the original scene that meet the mask generation condition; the mask generation condition is: for a Gaussian point in the original scene, there is at least one Gaussian point in the initial hard mask, so that the position distance between the two Gaussian points in the three-dimensional space is less than a preset radius.

[0018] Furthermore, in step S22, the three-dimensional soft mask The generation process is specifically expressed as follows:

[0019]

[0020] in, represents the initial hard mask; Represents the original scene; is the preset radius size; Represents a Gaussian point in the original scene; represents a Gaussian point in the initial hard mask; Represents the position distance between two Gaussian points in three-dimensional space.

[0021] As a preferred embodiment of the first aspect, the specific process of generating the two-dimensional mask image in step S3 is as follows:

[0022] S31: removing Gaussian points belonging to the three-dimensional soft mask from the original scene to form a three-dimensional Gaussian point cloud scene after removing the foreground, projecting the three-dimensional Gaussian point cloud scene after removing the foreground to each viewpoint to obtain an image after removing the foreground at the corresponding viewpoint, and then projecting the original scene to each viewpoint to obtain a two-dimensional original image at the corresponding viewpoint;

[0023] S32: Subtracting the two-dimensional original image and the image after foreground removal pixel by pixel at the same viewing angle to generate a difference map at the viewing angle, and then comparing the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the position where the pixel is located is set to 1; if the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the position where the pixel is located is set to 0, thereby generating a two-dimensional mask map at the viewing angle.

[0024] Further, in step S32, Two-dimensional mask map under different viewing angles The definition is as follows:

[0025]

[0026] in, is the preset pixel difference threshold; is the number of viewing angles; Indicates the Two-dimensional original image under different viewing angles; express The center position in two-dimensional coordinates The pixel value of the pixel at Indicates the The image after removing the foreground from each perspective; express The center position in two-dimensional coordinates The pixel value of the pixel at Indicates taking the absolute value.

[0027] As a preferred embodiment of the first aspect, in S4, a diffusion model is used as the image hidden model.

[0028] As a preferred embodiment of the first aspect, in S4, a CLIP image similarity function is used to measure the similarity between the candidate blanking image of the non-initial viewing angle and the final blanking image of the previous viewing angle.

[0029] As a preferred embodiment of the above-mentioned first aspect, in S4, the specific process of performing non-mask area fidelity enhancement processing on the final hidden image is: directly copying the area of ​​the two-dimensional original image under a viewing angle that is not covered by its corresponding two-dimensional mask image to the final hidden image of the viewing angle to form the final hidden result under the viewing angle.

[0030] Further, in step S4, The final hidden result under different viewing angles In two-dimensional coordinates The pixel value at It can be expressed as:

[0031]

[0032] in, Indicates the The two-dimensional mask image of each view is in the two-dimensional coordinate The mask value at ; Indicates the The final hidden image of the viewing angle is in two-dimensional coordinates The pixel value at ; Indicates that the The original two-dimensional image of the perspective is in the two-dimensional coordinate The pixel value at .

[0033] In a second aspect, the present invention provides a three-dimensional scene object hiding system based on 3DGS, which comprises the following steps:

[0034] The scene extraction module is used to obtain the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene. The 3D segmentation algorithm is used to perform semantic segmentation on the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud corresponding to the target to be hidden and the remaining background Gaussian point cloud;

[0035] The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a three-dimensional soft mask covering the foreground boundary and its adjacent areas;

[0036] The mask generation module is used to project the expanded 3D soft mask into the multi-view 2D image space and generate a 2D mask image corresponding to each view using the pixel difference between the projected images of the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0037] A consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into a pre-trained image hidden model, perform a target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at that viewing angle. The reference image is the candidate hidden image selected by the user from the candidate hidden image set for the initial viewing angle that best meets their own personalized needs.

[0038] The result acquisition module is used to input the final hidden layer fading results under each perspective as training data, perform training updates in sequence according to the preset continuous perspective path, optimize the original 3D Gaussian point cloud parameters, and use the trained and optimized 3D Gaussian scene as the target 3D scene after hidden layer fading to complete the target hidden layer fading of the 3D scene.

[0039] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the three-dimensional scene target hiding method based on 3DGS as described in any one of the solutions of the first aspect above.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention introduces a 3DGS-based 3D scene object removal framework, employing a 3D segmentation algorithm and a mask expansion strategy to identify the objects to be removed. These objects are then projected into a 2D space and fed into an image removal diffusion model for background restoration. Specifically, a 3D segmentation algorithm is first used to semantically segment the scene, extracting the foreground Gaussian point cloud corresponding to the object to be removed and the remaining background Gaussian point cloud. The extracted foreground Gaussian point cloud is then spatially expanded to generate a 3D soft mask covering the foreground boundary and its adjacent areas, enhancing the integrity of the removed area. This expanded 3D soft mask is then projected into a multi-view 2D image space to generate a 2D mask map, precisely calibrating the image regions to be removed. The 2D original image and the 2D mask map at each viewpoint are then input into the image removal model, and the masked regions are removed. A user-guided mechanism is introduced during the removal process to select a satisfactory removal result. Finally, the target removal result is updated view by viewpoint along a pre-set continuous view path, and the removed target 3D scene is optimized. Based on this invention, users can efficiently remove objects in complex 3D scenes while maintaining high background quality and multi-view consistency.

[0042] This demonstrates that the proposed method leverages the structural advantages of explicit 3D Gaussian Splatting (3DGS) representation, integrating mask prediction, mask-guided image-level inpainting, and a structure-aware similarity screening mechanism across views to achieve highly consistent and natural multi-view object removal results. Through a multi-stage collaborative process, the proposed method systematically improves the geometric alignment and texture continuity of object removal during 3D reconstruction, effectively alleviating the issues of inconsistent inpainted regions and significant artifacts found in traditional methods. This method demonstrates excellent fidelity and structural restoration capabilities in 3D scene object removal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the steps of the method of the present invention;

[0044] Figure 2 A schematic diagram of the target blanking process of the method of the present invention;

[0045] Figure 3 Comparison diagram of the editing results of the present invention and other target blanking methods at different viewing angles;

[0046] Figure 4 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0048] In this embodiment, the technical background of the present invention is first described, which can be summarized into the following aspects:

[0049] (1) Object culling in three-dimensional space: Object culling aims to remove a specified object from an image or scene while maintaining the natural transition and visual plausibility of the background. At the two-dimensional image level, object culling is usually achieved through segmentation, masking, and region repair. However, in three-dimensional space, due to the geometric depth and multi-perspective characteristics of object distribution, the mask information of a single perspective is insufficient to support comprehensive culling, and joint modeling must be performed with the help of multi-perspective data. Three-dimensional object culling must simultaneously consider complex factors such as the geometric outline of the object in space, the consistency of its projection in each perspective, and the occlusion relationship between perspectives.

[0050] (2) Background restoration: After the target is masked, the background of the original target area needs to be filled in so that it has a natural transition in terms of texture details, structural trends, and lighting consistency. Background restoration is essentially a conditional generation task, whose goal is to synthesize the content of the missing area under the guidance of the image context. Traditional methods such as PatchMatch and image propagation algorithms perform well in simple backgrounds, but in scenes with complex textures or strong semantic structures, the generated results are prone to logical irrationality or visual abruptness. In recent years, the diffusion model has shown great potential in the field of image restoration due to its excellent generation performance. By introducing noise and reversely denoising it step by step, the diffusion model can learn high-quality content generation paths in complex semantic spaces, which is particularly suitable for the restoration task of large missing areas. For the background restoration problem in three-dimensional scenes, the diffusion model can independently repair images from different perspectives and combine multi-perspective context information to achieve high-fidelity background generation.

[0051] (3) Multi-view consistency in 3D tasks: In 3D scene modeling and editing tasks, multi-view consistency is a fundamental constraint, requiring that the image representation of the same 3D point under different viewpoints should maintain consistency in geometric position, color texture, and structural semantics. Lack of inter-view consistency will directly lead to problems such as drift, texture breakage, or structural distortion in the reconstructed scene. Traditional methods often use geometric priors (such as SFM or MVS) for multi-view registration and fuse information between different images based on manually designed feature comparison strategies, but it is difficult to maintain stable registration effects in complex lighting, occlusion, and low-texture areas. For hidden scenes, the background restoration results of each viewpoint must be consistent in content, otherwise obvious visual artifacts will be generated at the fusion boundary when they are merged back into the 3D scene. In addition, the differences in detailed styles between viewpoints will also be significantly amplified, thereby destroying the immersion and realism of the entire 3D scene.

[0052] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned 3D scene object hidden method based on 3DGS includes the following steps S1 to S5. The specific implementation process is described in detail below.

[0053] S1: Acquire the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm (Segment Any 3D Gaussians, SAGA) to semantically divide the original scene according to the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud and the remaining background Gaussian point cloud corresponding to the target to be hidden for subsequent target hiding operations.

[0054] It should be noted that, in the present invention, the specific steps of S1 include: using multi-view images to construct the original scene and using it as the basic input for the target hiding operation, receiving the target text information to be hidden provided by the user through the command line interface, and using a three-dimensional segmentation algorithm to decompose the original scene into a foreground Gaussian point cloud and a remaining background Gaussian point cloud according to the target text information to be hidden, and finally storing the foreground Gaussian point cloud and the remaining background Gaussian point cloud separately to provide input data for subsequent mask construction and rendering.

[0055] In this embodiment, if Figure 2 As shown in the figure, in the first step of extracting the target scene, the part selected by the dotted line is the process performed internally by the 3D segmentation algorithm (i.e. Figure 2 The original training scene + training segmentation features in the training scene are rendered to form the foreground Gaussian point cloud ( Figure 2 foreground in ) and the remaining background Gaussian point cloud ( Figure 2 In addition, the implementation of the three-dimensional segmentation algorithm belongs to the existing technology. For details, please refer to the literature Cen J, Fang J, Yang C, et al. Segment any 3d gaussians[C] / / Proceedings of the AAAI Conference on ArtificialIntelligence. 2025, 39(2): 1971-1979, which will not be repeated here.

[0056] In this embodiment, the foreground Gaussian point cloud and the remaining background Gaussian point cloud At the same time, the intersection of the two sets is an empty set. , the union of the two is the original scene , specifically as follows:

[0057]

[0058]

[0059] in, Indicates the union operation; Represents the intersection operation.

[0060] S2: Spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent areas.

[0061] It should be noted that in step S2 of the present invention, in order to enhance the integrity of the blanking area, the extracted foreground Gaussian point cloud is spatially expanded to generate a three-dimensional soft mask covering the foreground boundary and its adjacent areas, so as to alleviate the artifact problem caused by edge residue and improve the robustness of the blanking processing.

[0062] In the present invention, the specific process of generating the three-dimensional soft mask in step S2 is as follows:

[0063] S21: construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the Gaussian point cloud set that needs to be blanked;

[0064] S22: Using a Gaussian neighborhood search strategy, the initial hard mask is expanded by a preset radius to form a three-dimensional soft mask.

[0065] In step S22 of the present invention, the three-dimensional soft mask is composed of Gaussian points in the original scene that meet the mask generation condition; the mask generation condition is: for a Gaussian point in the original scene , there is at least one Gaussian point in the initial hard mask , so that the two Gaussian points (i.e. and ) in three-dimensional space is less than the preset radius.

[0066] In step S22 of this embodiment, the three-dimensional soft mask The generation process is specifically expressed as follows:

[0067]

[0068] in, represents the initial hard mask; Represents the original scene; is the preset radius size; Represents a Gaussian point in the original scene; represents a Gaussian point in the initial hard mask; Represents the position distance between two Gaussian points in three-dimensional space.

[0069] The above-mentioned three-dimensional soft mask can not only ensure that the foreground and background are connected more smoothly in the entire scene, but also alleviate the error when extracting the target scene to a certain extent, and will not cause the entire editing scene to be paralyzed due to defects in scene extraction. It improves the visual effect while ensuring high robustness.

[0070] S3: The expanded 3D soft mask is projected into the multi-view 2D image space, and the 2D mask corresponding to each view is generated using the pixel difference between the images projected from the original scene and the 3D Gaussian point cloud scene after foreground removal.

[0071] In the present invention, in step S3, the generated two-dimensional mask image can accurately mark the image area that needs to be blanked. The specific generation process is:

[0072] S31: removing Gaussian points belonging to the three-dimensional soft mask from the original scene to form a three-dimensional Gaussian point cloud scene after removing the foreground, projecting the three-dimensional Gaussian point cloud scene after removing the foreground to each viewpoint to obtain an image after removing the foreground at the corresponding viewpoint, and then projecting the original scene to each viewpoint to obtain a two-dimensional original image at the corresponding viewpoint;

[0073] S32: Subtracting the two-dimensional original image and the image after foreground removal pixel by pixel at the same viewing angle to generate a difference map at the viewing angle, and then comparing the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the position where the pixel is located is set to 1; if the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the position where the pixel is located is set to 0, thereby generating a two-dimensional mask map at the viewing angle.

[0074] In this embodiment, the Two-dimensional mask map under different viewing angles is a binary image (value 0 or 1) used to distinguish foreground from background. 2D original image under different viewing angles In two-dimensional coordinates The pixel value at Image after removing foreground from different viewing angles The absolute difference of pixel values ​​at the same position is greater than the preset pixel difference threshold , the mask value of that position is set to 1 (indicating it belongs to the foreground), otherwise, the mask value is set to 0 (indicating it belongs to the background). Simply put, this two-dimensional mask map compares the pixel differences between the two-dimensional original image and the image after removing the foreground, finds the area with significant changes (possibly foreground objects), and uses the pixel difference threshold to control the sensitivity. Therefore, the above two-dimensional mask map The definition is as follows:

[0075]

[0076] in is the preset pixel difference threshold; is the number of viewing angles; express The center position in two-dimensional coordinates The pixel value of the pixel at express The center position in two-dimensional coordinates The pixel value of the pixel at Indicates taking the absolute value.

[0077] S4: Input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into the pre-trained image hidden model, perform target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, the non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at this viewing angle; the reference image is the candidate hidden image selected by the user for the initial viewing angle in the candidate hidden image set that best meets his or her own personalized needs.

[0078] It should be noted that the image concealment model in the present invention can be selected by those skilled in the art based on practical needs, as long as it can achieve the target concealment task. In this embodiment, a diffusion model is used as the image concealment model. The diffusion model comprises a text encoder, an image encoder-decoder, and a U-type network (i.e., UNet). This model is pre-trained on a large amount of text image data. Taking the image to be edited and a mask as input, it can locally generate the image in the masked area. The specific implementation of the diffusion model is prior art and will not be detailed here.

[0079] It should be noted that, in step S4 of the present invention, the CLIP image similarity function is used to measure the similarity between the candidate blanking image of the non-initial viewing angle and the final blanking image of the previous viewing angle.

[0080] In this embodiment, step S3 corresponds to Figure 2 The second step in the expansion of the three-dimensional mask, step S4 corresponds to Figure 2 The third step in the multi-view consistency editing process is to A viewing angle, and the image under this viewing angle is a two-dimensional original image And the 2D mask map mask Input the image hidden model, and then call the image hidden model to perform target hidden, for the first A set of candidate hidden image sets corresponding to each view . are the first candidate blanking images, is the number of candidate hidden images in the candidate hidden image set. Then, this embodiment introduces a user guidance mechanism in the hidden process of step S4. Specifically, after the image hidden model generates multiple candidate hidden images for each perspective, the user first selects the candidate hidden image from the specified initial perspective. ( )of Select a candidate hidden image that best meets personalized needs from the candidate hidden images as the reference image , this selection can be used to guide similarity evaluation of subsequent perspectives. ( ), then the candidate hidden image set generated from it Automatically select the best candidate blanking image as the final blanking image , so that it is consistent with the initial perspective The result (i.e. the reference image ) has the highest similarity in CLIP image features. And so on, for subsequent non-initial perspectives ( ), can be generated from the candidate hidden image Automatically select the best candidate hidden image as the first The final blanked image at each viewing angle , so that it is consistent with the perspectives The final blanked image The similarity between them is the highest to improve the consistency and stability of cross-view culling. The target optimization function is as follows:

[0081]

[0082] in, Indicates the Any candidate blanking image under any viewing angle; Represents calculation and CLIP image similarity between them; It represents the search for candidate blanking images that maximize the target optimization function.

[0083] It can be seen that the user guidance mechanism designed in the present invention can maintain the quality of the hidden area while ensuring the semantic consistency of the target hidden area under adjacent viewing angles to the greatest extent, thereby improving the visual stability of the scene after 3D reconstruction.

[0084] It should be noted that in step S4, considering the existence of noise disturbance and detail fluctuation in the diffusion model during the image generation process, in order to enhance the stability and detail retention of the final blanked image, the present invention performs non-mask area fidelity enhancement processing on the final blanked image: that is, the area of ​​the two-dimensional original image under a viewing angle that is not covered by its corresponding two-dimensional mask image is directly copied to the final blanked image of the viewing angle to form the final blanked result under the viewing angle.

[0085] In this embodiment, the The final hidden result under different viewing angles For example, when the two-dimensional mask image of the perspective is in the two-dimensional coordinate The mask value at When it is 1, The pixel value at this coordinate of the final hidden result under the viewing angle The final hidden image of the viewing angle is in two-dimensional coordinates The pixel value at ; When the The two-dimensional mask image of each view is in the two-dimensional coordinate When the mask value at is 0, The pixel value of the final hidden result at the coordinate of the viewing angle is the pixel value of the two-dimensional original image at the two-dimensional coordinate of the viewing angle. The pixel value at The specific expressions are as follows:

[0086]

[0087] This operation can further enhance the authenticity and structural continuity of the background area without affecting the quality of the generated blanking area.

[0088] S5: The final hidden point cloud fading results under each viewing angle are used as training data input. Training updates are performed sequentially according to the preset continuous viewing angle path to optimize the original 3D Gaussian point cloud parameters. The optimized 3D Gaussian scene is used as the target 3D scene after hidden point cloud fading to complete the target hidden point cloud fading of the 3D scene.

[0089] It should be noted that in step S5, all viewing angles are subjected to image hiding according to the implementation process of the aforementioned steps S1 to S4, and the final hidden results of all viewing angles are As training data input, it is used to update the original 3D Gaussian point cloud attributes. According to the preset continuous view path, the target hidden result is replaced and updated view by view. In this process, the adjacent view changes slowly, so that the target Gaussian point cloud can continuously adapt to the update, and finally output the 3D Gaussian scene after training optimization. This step ensures structural consistency and visual coherence between multiple views, avoiding artifacts or floating points caused by inconsistent hidden tiling.

[0090] The following tests apply the 3DGS-based 3D scene object removal method, described in steps S1 through S5 of the preferred implementation above, to a specific dataset. The specific steps are as described in S1 through S5 and will not be repeated here. The focus will be on demonstrating the specific parameters and technical results.

[0091] Example

[0092] According to the implementation process of the aforementioned steps S1 to S5, a 3D scene dataset collected from the real world is first obtained for 3D scene reconstruction, and then a multi-view image is rendered. During the target concealment process, a 3D segmentation algorithm is first used to semantically divide the original scene, extracting the foreground Gaussian point cloud corresponding to the target to be concealed and the remaining background Gaussian point cloud. The extracted foreground Gaussian point cloud is spatially expanded to generate a 3D soft mask covering the foreground boundary and its adjacent areas to enhance the integrity of the concealed area. The expanded 3D soft mask is then projected into the multi-view 2D image space to obtain a 2D mask map, accurately calibrating the image area where the target is to be concealed. Subsequently, the 2D original image and the 2D mask map at each view are input into the image concealment model, and the target concealment operation is performed on the masked area. A user-guided mechanism is introduced during the concealment process to select a satisfactory concealment result. Finally, according to a preset continuous view path, the target concealment result is replaced and updated view by view, and the concealed target 3D scene is optimized.

[0093] As a quantitative indicator, the present invention is compared with two existing methods, IN2N (GS) and GaussianEditor, and two indicators, CLIP text-image direction similarity and CLIP direction consistency, are selected. The test results of the above methods on the 360_v2 dataset are shown in Table 1. The editing effects of the present invention and the above two existing methods are compared. Figure 3 shown.

[0094] Table 1. Quantitative evaluation table of editing effect

[0095] It should also be noted that the 3DGS-based 3D scene object hiding method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a 3DGS-based 3D scene object hiding system corresponding to the 3DGS-based 3D scene object hiding method provided in the above embodiment, such as Figure 4 As shown, it includes:

[0096] The scene extraction module is used to obtain the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene. The 3D segmentation algorithm is used to perform semantic segmentation on the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud corresponding to the target to be hidden and the remaining background Gaussian point cloud;

[0097] The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a three-dimensional soft mask covering the foreground boundary and its adjacent areas;

[0098] The mask generation module is used to project the expanded 3D soft mask into the multi-view 2D image space and generate a 2D mask image corresponding to each view using the pixel difference between the projected images of the original scene and the 3D Gaussian point cloud scene after removing the foreground.

[0099] A consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into a pre-trained image hidden model, perform a target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at that viewing angle. The reference image is the candidate hidden image selected by the user from the candidate hidden image set for the initial viewing angle that best meets their own personalized needs.

[0100] The result acquisition module is used to input the final hidden layer fading results under each perspective as training data, perform training updates in sequence according to the preset continuous perspective path, optimize the original 3D Gaussian point cloud parameters, and use the trained and optimized 3D Gaussian scene as the target 3D scene after hidden layer fading to complete the target hidden layer fading of the 3D scene.

[0101] It is understood that the 3DGS-based 3D scene object hiding method described in S1-S5 above can essentially be implemented via a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the 3DGS-based 3D scene object hiding method provided in the above embodiment. The computer program product includes a computer program / instructions. When executed by a processor, the computer program / instructions can implement the 3DGS-based 3D scene object hiding method described in the above embodiment.

[0102] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the 3DGS-based three-dimensional scene object hiding method provided in the above embodiment, which includes a memory and a processor;

[0103] The memory is used to store computer programs;

[0104] The processor is configured to implement the three-dimensional scene object hiding method based on 3DGS in the above embodiment when executing the computer program.

[0105] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0106] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the three-dimensional scene target hiding method based on 3DGS provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the three-dimensional scene target hiding method based on 3DGS in the above embodiment can be implemented.

[0107] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S5.

[0108] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0109] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0110] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0111] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A 3D scene object hiding method based on 3DGS, characterized in that: The following steps are involved: S1: Acquire the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene, use the 3D segmentation algorithm to perform semantic segmentation on the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud corresponding to the target to be hidden and the remaining background Gaussian point cloud; S2: spatially expand the extracted foreground Gaussian point cloud to generate a 3D soft mask covering the foreground boundary and its adjacent areas; S3: Project the expanded 3D soft mask into the multi-view 2D image space, and use the pixel difference between the projected images of the original scene and the 3D Gaussian point cloud scene after removing the foreground to generate a 2D mask corresponding to each view. S4: Input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into the pre-trained image hidden model, perform a target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, the non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at this viewing angle. The reference image is the candidate hidden image selected by the user from the candidate hidden image set for the initial viewing angle that best meets his or her own personalized needs. S5: The final hidden point cloud fading results under each viewing angle are used as training data input. Training updates are performed sequentially according to the preset continuous viewing angle path to optimize the original 3D Gaussian point cloud parameters. The optimized 3D Gaussian scene is used as the target 3D scene after hidden point cloud fading to complete the target hidden point cloud fading of the 3D scene.

2. A 3D scene object hiding method based on 3DGS according to claim 1, characterized in that: The specific steps of S1 include: using multi-view images to construct the original scene and using it as the basic input for the target hiding operation, receiving the target text information to be hidden provided by the user through the command line interface, using a three-dimensional segmentation algorithm to decompose the original scene into a foreground Gaussian point cloud and a remaining background Gaussian point cloud according to the target text information to be hidden, and finally storing the foreground Gaussian point cloud and the remaining background Gaussian point cloud separately.

3. The 3D scene object hiding method based on 3DGS according to claim 1, characterized in that: The specific process of generating a three-dimensional soft mask in step S2 is as follows: S21: construct an initial hard mask based on the extracted foreground Gaussian point cloud to mark the Gaussian point cloud set that needs to be blanked; S22: Using a Gaussian neighborhood search strategy, the initial hard mask is expanded by a preset radius to form a three-dimensional soft mask.

4. A 3D scene object hiding method based on 3DGS according to claim 3, characterized in that: In step S22, the three-dimensional soft mask is composed of Gaussian points in the original scene that meet the mask generation condition; the mask generation condition is: for a Gaussian point in the original scene, there is at least one Gaussian point in the initial hard mask, so that the position distance between the two Gaussian points in the three-dimensional space is less than a preset radius.

5. The 3D scene object hiding method based on 3DGS according to claim 1, characterized in that: The specific process of generating a two-dimensional mask image in step S3 is as follows: S31: removing Gaussian points belonging to the three-dimensional soft mask from the original scene to form a three-dimensional Gaussian point cloud scene after removing the foreground, projecting the three-dimensional Gaussian point cloud scene after removing the foreground to each viewpoint to obtain an image after removing the foreground at the corresponding viewpoint, and then projecting the original scene to each viewpoint to obtain a two-dimensional original image at the corresponding viewpoint; S32: Subtracting the two-dimensional original image and the image after foreground removal pixel by pixel at the same viewing angle to generate a difference map at the viewing angle, and then comparing the difference map with a preset pixel difference threshold pixel by pixel. If the pixel value of a pixel in the difference map is greater than the pixel difference threshold, the mask value of the position where the pixel is located is set to 1; if the pixel value of a pixel in the difference map is less than or equal to the pixel difference threshold, the mask value of the position where the pixel is located is set to 0, thereby generating a two-dimensional mask map at the viewing angle.

6. A 3D scene object hiding method based on 3DGS according to claim 1, characterized in that: In S4, a diffusion model is used as the image hidden model.

7. The method for 3D scene object hiding based on 3DGS according to claim 1, characterized in that: In S4, the CLIP image similarity function is used to measure the similarity between the candidate blanking image of the non-initial viewing angle and the final blanking image of the previous viewing angle.

8. The 3D scene object hiding method based on 3DGS according to claim 1, characterized in that: In S4, the specific process of performing the non-mask area fidelity enhancement processing on the final hidden image is: directly copying the area of ​​the two-dimensional original image at a certain viewing angle that is not covered by its corresponding two-dimensional mask image to the final hidden image at that viewing angle, thereby forming the final hidden result at that viewing angle.

9. A three-dimensional scene object hiding system based on 3DGS, characterized in that: The following steps are involved: The scene extraction module is used to obtain the multi-view images to be processed, perform 3D scene reconstruction on the multi-view images, generate the original 3D Gaussian point cloud scene and use it as the original scene. The 3D segmentation algorithm is used to perform semantic segmentation on the original scene based on the acquired text information of the target to be hidden, and extract the foreground Gaussian point cloud corresponding to the target to be hidden and the remaining background Gaussian point cloud; The spatial expansion module is used to spatially expand the extracted foreground Gaussian point cloud to generate a three-dimensional soft mask covering the foreground boundary and its adjacent areas; The mask generation module is used to project the expanded 3D soft mask into the multi-view 2D image space and generate a 2D mask image corresponding to each view using the pixel difference between the projected images of the original scene and the 3D Gaussian point cloud scene after removing the foreground. A consistency editing module is used to input the two-dimensional original image formed by the projection of the original scene at each viewing angle and its corresponding two-dimensional mask image into a pre-trained image hidden model, perform a target hidden operation on the mask area, and generate a set of candidate hidden image sets for each viewing angle. Each set of candidate hidden image sets is composed of multiple candidate hidden images. After obtaining the reference image selected by the user for the initial viewing angle, non-initial viewing angles will use the candidate hidden image with the highest similarity to the final hidden image of their respective previous viewing angles as the final hidden image, and perform fidelity enhancement processing on the non-mask area of ​​the final hidden image to finally form the final hidden result at that viewing angle. The reference image is the candidate hidden image selected by the user from the candidate hidden image set for the initial viewing angle that best meets their own personalized needs. The result acquisition module is used to input the final hidden layer fading results under each perspective as training data, perform training updates in sequence according to the preset continuous perspective path, optimize the original 3D Gaussian point cloud parameters, and use the trained and optimized 3D Gaussian scene as the target 3D scene after hidden layer fading to complete the target hidden layer fading of the 3D scene.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the 3D scene object hiding method based on 3DGS as described in any one of claims 1 to 8 can be implemented.

Images