3D GS instance segmentation method combining two-dimensional segmentation and multi-view voting

Abstract

The invention relates to a three-dimensional Gaussian splash (3DGS) instance segmentation method combining two-dimensional segmentation and multi-view voting, belongs to the field of three-dimensional scene analysis and processing, and aims to solve the problem of boundary blur in the existing three-dimensional Gaussian splash (3DGS) scene instance segmentation. Therefore, the method comprises the following steps: firstly, segmenting a three-dimensional scene multi-view image by using a two-dimensional image segmentation model to obtain a two-dimensional mask and an image of a target instance; a motion recovery structure algorithm is adopted to obtain sparse point clouds of the three-dimensional scene and camera parameters of all view angles; based on the sparse point cloud, the camera parameters and the target instance image, performing three-dimensional reconstruction by using a 3DGS algorithm to obtain an initial 3DGS model of the target instance; and finally, performing optimization processing on the initial 3DGS model by adopting a voting strategy based on multiple view angles to obtain a 3DGS instance model with clear boundary and clean surface.

Description

Technical Field

[0001] This invention relates to a method for instance segmentation of a 3D scene, and particularly to a 3DGS instance segmentation method that combines 2D segmentation and multi-view voting, belonging to the field of 3D scene analysis and processing. Background Technology

[0002] Building upon 3D scene reconstruction, instance segmentation of the 3D scene to obtain 3D instance models is a crucial foundation for analyzing and understanding 3D scenes. Furthermore, 3D instance models are fundamental elements in fields such as virtual reality and 3D games. Therefore, how to obtain realistic 3D instance models from real-world scenes has become a focal point of great interest in both academia and industry.

[0003] 3DGS (3D Gaussian Splatting) is a high-fidelity scene reconstruction and rendering method proposed by Kerbl et al. in 2023 (B.Kerbl et al., 3D Gaussian Splatting for Real-time RadianceField Rendering. ACM Transactions on Graphics, 2023, 42(4): 1-14.). This method uses a series of three-dimensional Gaussian ellipsoids to represent a 3D scene, and then continuously adjusts and optimizes the number, position, size, color, and opacity information of the Gaussian ellipsoids based on the 2D views of the 3D scene from different perspectives to obtain a 3DGS model that can represent the 3D scene with high fidelity. The 3DGS model uses three-dimensional Gaussian ellipsoids as primitives to explicitly encode the 3D scene, and on this basis, it realizes fast real-time rendering of the 3D scene. Therefore, 3DGS has become a prominent emerging technology in the field of 3D scene reconstruction.

[0004] However, 3DGS models a holistic 3D scene based on multiple viewpoints, without distinguishing between individual objects within the scene, making it impossible to directly extract information about individual object instances from the modeling results. To extract object instances from 3DGS scenes, researchers have proposed several 3DGS instance segmentation methods, such as the Gaussian Grouping method proposed by Ye et al. (M.Ye et al..Gaussian Grouping:Segment and Edit Anything in 3D Scenes.ECCV2024:18th European Conference,Milan,Italy,September 29–October 4,2024,pp.162-179.) and the SAGA method proposed by Cen et al. (J.Cen et al..Segment Any3D Gaussians.Proceedingsof the AAAI Conference on Artificial Intelligence,vol.39,no.2,pp.1971-1979,Apr.2025.). The methods described above are all based on two-dimensional instance segmentation. They integrate instance feature information obtained from two-dimensional multi-view image segmentation into the original three-dimensional Gaussian ellipsoid representation, and then use the extended Gaussian ellipsoid representation method to achieve the segmentation of a single instance in a 3DGS scene. However, due to the ambiguity of features, these 3DGS segmentation methods guided by two-dimensional instance features often classify Gaussian ellipsoids that are semantically different but visually similar and should belong to different instances into the same category, ultimately leading to blurred segmentation boundaries and the obtained three-dimensional instance surfaces having feather-like appendages.

[0005] In view of the above, this invention proposes a 3DGS instance segmentation method that combines two-dimensional segmentation and multi-view voting. This method combines two-dimensional instance segmentation with 3DGS technology to construct a three-dimensional instance model, and then employs a multi-view voting-based filtering strategy to remove redundant Gaussian ellipsoids from the instance model, resulting in a 3DGS instance model with clear boundaries and a clean surface. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of blurred boundaries in existing 3DGS scene instance segmentation, and to provide a 3DGS instance segmentation method that combines two-dimensional segmentation and multi-view voting, so as to realize 3D instance segmentation based on 3DGS and obtain 3DGS instance models with clear boundaries and clean surfaces.

[0007] The objective of this invention is achieved through the following technical solutions.

[0008] A 3DGS instance segmentation method combining 2D segmentation and multi-view voting includes the following steps:

[0009] Step 1: Obtain a multi-view image set of the 3D scene, including the target instance. Where V represents the number of viewpoints, I i This represents the image from the i-th viewpoint.

[0010] Step 2: Use a 2D image segmentation model to process multi-view image sets of 3D scenes. All images are segmented to obtain a set of two-dimensional masks for the target instance. and utilize from Extracting target instances from multi-view image sets

[0011] Step 3: Employ the SfM (Structured Motion Recovery) algorithm to analyze the multi-view image set. Estimate the intrinsic and extrinsic parameter set θ of each camera from different viewpoints, and simultaneously recover the sparse point cloud set Φ representing the 3D scene.

[0012] Step 4: Combine the set of intrinsic and extrinsic parameters θ from each camera viewpoint, the set of sparse point clouds Φ, and the set of multi-view images of the target instance. As input to the original 3DGS 3D reconstruction algorithm, 3D reconstruction is performed to obtain an initial 3DGS model of the target instance represented by a series of 3D Gaussian ellipsoids. Where N represents the number of Gaussian ellipsoids, g j Let j represent the j-th Gaussian ellipsoid.

[0013] Step 5: Use a multi-view voting strategy to create an initial 3DGS model of the target instance. The optimization process is as follows:

[0014] Step 5.1: Let o j =[x j ,y j ,z j ] T Represents the Gaussian ellipsoid g j The center point coordinates are obtained; for each viewpoint v, its projection matrix is ​​calculated using the intrinsic and extrinsic parameters of the camera at that viewpoint. Then, using the projection matrix P v The center point of the Gaussian ellipsoid is o j =[x j ,y j ,z j ] T Transformed into normalized device coordinates

[0015]

[0016] Among them [o j ;1]·P v =[x′ j ,y′ j ,z′ j ,w′ j ] T π(·) represents the conversion from homogeneous coordinates to Cartesian coordinates; then, based on the image size at viewpoint v, the center point o of the Gaussian ellipsoid is calculated. j Image coordinates

[0017]

[0018] in and They are The x and y coordinates are given, I is the identity matrix, and w and h are the image width and height at viewpoint v, respectively.

[0019] Step 5.2: For each viewpoint v, define a Gaussian ellipsoid g. j single-view state like If the image area extends beyond the viewing angle v, then... like Within the image region and simultaneously located at the target instance mask M v The foreground area that is marked makes like Mask M, located within the image region but not belonging to the target instance v The foreground area that is marked makes This yields the single-view state vectors of all Gaussian ellipsoids at viewpoint v.

[0020]

[0021] Step 5.3: Calculate g for each Gaussian ellipsoid using the following formula. j Multi-perspective voting ratio

[0022]

[0023] Among them 1 A (x) is an indicator function, which has a value of 1 when x∈A and 0 otherwise.

[0024] Step 5.4: Examine each Gaussian ellipsoid and adjust the multi-view voting ratio r. j Gaussian ellipsoid g below the preset threshold τ j From the initial 3DGS model of the target instance Delete, thus obtaining the optimized target instance model.

[0025] Step 6: Output the optimized target instance model

[0026] Beneficial effects

[0027] The 3DGS instance segmentation method combining two-dimensional segmentation and multi-view voting described in this invention has the following technical advantages and characteristics compared with existing three-dimensional instance segmentation methods for 3DGS scenes:

[0028] (1) This invention first uses a two-dimensional image segmentation model to segment multi-view images of a three-dimensional scene to obtain multi-view images of the target instance, and then uses the multi-view images of the target instance for 3DGS modeling. Compared with the three-dimensional instance segmentation method that directly uses the original three-dimensional scene multi-view images for 3DGS modeling, the method of this invention avoids the semantic ambiguity problem of the three-dimensional scene and the interference of complex background on the 3DGS modeling process, which helps to obtain a more accurate 3DGS instance model.

[0029] (2) The method of the present invention adopts a multi-view voting strategy to optimize the 3DGS instance model, which can effectively remove the feather-like artifacts generated by the original 3DGS modeling and obtain a 3DGS instance model with clear boundaries and a clean surface.

[0030] (3) The method of this invention directly utilizes the original 3DGS algorithm for 3D instance modeling without changing or expanding the 3DGS modeling framework, and without additional scene training overhead. Compared with methods such as Gaussian Grouping and SAGA that require expanding the 3DGS framework, the method of this invention has higher computational efficiency. Attached Figure Description

[0031] Figure 1 Flowchart of the method described in this invention;

[0032] Figure 2 Basic information about the garden scene;

[0033] Figure 3 The initial 3DGS model rendering result of the table instance;

[0034] Figure 4 Optimized rendering result of the table instance model;

[0035] Figure 5 The table instance model obtained using the SAGA method;

[0036] Figure 6 The table instance model obtained using the SAGD method. Specific implementation methods

[0037] The specific embodiments of the present invention are described below with reference to the accompanying drawings and examples.

[0038] Figure 1 A flowchart of a 3DGS instance segmentation method combining two-dimensional segmentation and multi-view voting, as described in this invention, is presented. Its main steps are as follows:

[0039] S1: Obtain a multi-view image set of the 3D scene, including the target instance. Where V represents the number of viewpoints, and i represents the image of the i-th viewpoint.

[0040] This embodiment selects a garden scene as the implementation object from the publicly available test dataset Mip-NeRF 360, and uses the table as the target instance. Therefore, this embodiment obtains a multi-view image set in this step. This is a multi-view image of a garden scene. The scene contains 185 different perspectives. Figure 2 It is an image from one particular perspective, showing the basic situation of the scene.

[0041] S2: Using a 2D image segmentation model to process multi-view image sets of 3D scenes All images are segmented to obtain a set of two-dimensional masks for the target instance. and utilize from Extracting target instances from multi-view image sets

[0042] This embodiment uses the basic image segmentation model SAM2 (N. Ravi et al..SAM 2:Segment Anything in Images and Videos.arXiv:2408.00714v2,Oct.2024.) as the two-dimensional image segmentation model for this step; at the same time, a multi-view image set is used. The image sequence is treated as an image sequence. Target instances are labeled on the first frame. Then, SAM2 is used to segment the image sequence, obtaining a set of two-dimensional masks of the target instances from different viewpoints. and multi-view image collection

[0043] It should be noted that the 2D image segmentation model used in this step includes, but is not limited to, SAM2. It is also feasible to use other models or methods that can successfully obtain target instance images and masks from multi-view image sets.

[0044] S3: Employs the SfM motion reconstruction algorithm from a multi-view image set. Estimate the intrinsic and extrinsic parameter set θ of each camera from different viewpoints, and simultaneously recover the sparse point cloud set Φ representing the 3D scene.

[0045] The SfM (Structure from Motion) algorithm, also known as the structure-from-motion algorithm, can recover sparse point clouds describing the three-dimensional structure of a scene and the camera's motion trajectory from a sequence of continuously captured two-dimensional images. This invention uses this algorithm to obtain camera parameters (including intrinsic and extrinsic parameters) from multiple viewpoints, and simultaneously recovers the sparse point cloud representing the three-dimensional scene; let θ be the set of obtained camera intrinsic and extrinsic parameters, and Φ be the set of sparse point clouds.

[0046] S4: Combine the set of intrinsic and extrinsic parameters θ from each camera viewpoint, the sparse point cloud set Φ, and the multi-view image set of the target instance. As input to the original 3DGS 3D reconstruction algorithm, 3D reconstruction is performed to obtain an initial 3DGS model of the target instance represented by a series of 3D Gaussian ellipsoids. Where N represents the number of Gaussian ellipsoids, g j Let j represent the j-th Gaussian ellipsoid.

[0047] This step mainly involves using the original 3DGS 3D reconstruction algorithm to perform 3D reconstruction of the target instance, obtaining a 3DGS model representing the target instance. The original 3DGS 3D reconstruction algorithm refers to the method initially proposed by Kerbl et al. in 2023 (B.Kerbl et al., 3D Gaussian Splatting for Real-time Radiance FieldRendering. ACM Transactions on Graphics, 2023, 42(4): 1-14.). Since the obtained target instance model is output by the original 3DGS 3D reconstruction algorithm, it is referred to here as the initial 3DGS model.

[0048] The target instance in this embodiment is a table in a garden scene; therefore, the result of this step is an initial 3DGS model of the table. This model contains 290,608 Gaussian ellipsoids. Figure 3This is the rendering result of the model from a certain perspective. Most of the background content unrelated to the table has been removed, but some white, feather-like floating artifacts of varying sizes still adhere to the edges and surface of the table. These artifacts, which do not belong to the table instance, severely affect the accuracy of the 3D instance modeling. The reasons behind this are mainly twofold. First, the 2D image segmentation model used in step S2 has certain errors, making it difficult for the generated instance mask to accurately match the actual boundaries of the object instances. This deviation leads to inconsistencies in instance boundaries across multiple perspectives, causing the 3DGS algorithm to generate redundant Gaussian ellipsoids around the instance boundaries. Second, the occlusion relationship between 3D scene instances results in discontinuities in the instance boundaries of the 2D image, causing the 3DGS algorithm to generate redundant Gaussian ellipsoids to fit the discontinuous boundaries. In either case, further measures are needed to remove these floating artifacts to obtain an accurate instance model. Therefore, this invention employs a multi-view voting strategy to optimize the initial 3DGS model in subsequent steps.

[0049] S5: Initial 3DGS model of the target instance using a multi-view voting strategy. The optimization process is as follows:

[0050] S5.1: Let o j =[x j ,y j ,z j ] T Represents the Gaussian ellipsoid g j The center point coordinates are obtained; for each viewpoint v, its projection matrix is ​​calculated using the intrinsic and extrinsic parameters of the camera at that viewpoint. Then, using the projection matrix P v The center point of the Gaussian ellipsoid is o j =[x j ,y j ,z j ] T Transformed into normalized device coordinates

[0051]

[0052] Among them [o j ;1]·P v =[x′ j ,y′ j ,z′ j ,w′ j ] T π(·) represents the conversion from homogeneous coordinates to Cartesian coordinates; then, based on the image size at viewpoint v, the center point o of the Gaussian ellipsoid is calculated. j Image coordinates

[0053]

[0054] in and They are The x and y coordinates are given, I is the identity matrix, and w and h are the image width and height at viewpoint v, respectively.

[0055] S5.2: Define the Gaussian ellipsoid g for each viewpoint v. j single-view state like If the image area extends beyond the viewing angle v, then... like Within the image region and simultaneously located at the target instance mask M v The foreground area that is marked makes like Mask M, located within the image region but not belonging to the target instance v The foreground area that is marked makes This yields the single-view state vectors of all Gaussian ellipsoids at viewpoint v.

[0056]

[0057] S5.3: Calculate g for each Gaussian ellipsoid using the following formula. j Multi-perspective voting ratio

[0058]

[0059] Among them 1 A (x) is an indicator function, which has a value of 1 when x∈A and 0 otherwise.

[0060] S5.4: Examine each Gaussian ellipsoid and calculate the multi-view voting ratio r. j Gaussian ellipsoid g below the preset threshold τ j From the initial 3DGS model of the target instance Delete, thus obtaining the optimized target instance model.

[0061] In this embodiment, the preset threshold τ = 0.9 is set for the table instance in step S5.4. Based on this threshold, the Gaussian ellipsoids in the initial 3DGS model of the table are filtered, and 16032 Gaussian ellipsoids are found to be unsatisfactory and therefore deleted. Finally, the optimized table instance model... It contains 274,576 valid Gaussian ellipsoids. Figure 4 This is the rendering result of the optimized table instance model from a certain viewpoint. Figure 3 In comparison, the optimized table instance model has clearer edges and surfaces, and most artifacts have been removed.

[0062] S6: Output the optimized target instance model

[0063] In this embodiment, the output at this step is the optimized table instance model. To facilitate subsequent processing and application of the model, this embodiment saves the model as a data file in .ply format.

[0064] To further illustrate the beneficial effects of the present invention, this embodiment compares the method of the present invention with two recently disclosed methods in the field. The comparison test results and related information are as follows.

[0065] Benchmark methods. Benchmark methods for comparison with the method of this invention include: 1) SAGA method (J.Cen et al., “Segment Any 3D Gaussians,” Proceedings of the AAAI Conference on Artificial Intelligence, vol.39, no.2, pp.1971–1979, Apr.2025); 2) SAGD method (X.Hu et al., “SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition,” arXiv:arXiv:2401.17857, Jan.19,2025.).

[0066] Evaluation indicators. The evaluation metrics used in the comparative test were: 1) IoU metric (J.Yu et al..UnitBox:An Advanced Object Detection Network.Proceedings of the 24th ACM international conference on Multimedia,pp.516–520,Oct.2016.); 2) BIoU metric (B.Cheng et al..Boundary IoU:Improving Object-Centric Image SegmentationEvaluation.2021IEEE / CVF Conference on Computer Vision and Pattern Recognition,pp.15329–15337,June 2021.); 3) PSNR metric (HRSheikh et al..AStatistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms.IEEE Transactions on Image Processing,15(11):3440-3451,2006.); 4) SSIM metric (Z.Wang et al.Image quality assessment:from error visibility to structural similarity. IEEE Transactions on Image Processing, vol.13, no.4, pp.600–612, Apr.2004.).

[0067] IoU (Intersection over Union) calculates the intersection-over-union ratio (IoU) of the predicted and ground truth masks, measuring the accuracy of the predicted mask. A higher IoU value indicates a more accurate segmentation result. BIoU (Intersection over Union) calculates the IoU of the predicted and ground truth masks in the boundary region, measuring the edge precision of the predicted mask. A higher BIoU value indicates more accurate edge precision in the segmentation result. PSNR (Simplified Mean Square Error) is based on the mean square error definition and measures the error between the ground truth image and the generated image. A higher PSNR value indicates a smaller error and higher generated image quality. SSIM (Simplified Mean Square Error) integrates features related to brightness, contrast, and structure, and measures the error between the ground truth image and the generated image. A higher SSIM value indicates a smaller error and higher generated image quality.

[0068] Experimental Results. For the garden scene in this embodiment, the table instance models obtained using the SAGA and SAGD segmentation methods are as follows: Figure 5 and Figure 6 As shown. With Figure 4 Comparing the results of the method of the present invention with those of the present invention, it is not difficult to find that... Figure 5 and Figure 6 The table instance models shown are not very complete, with many blurry or even incomplete edges. Figure 6 Besides severely blurred and incomplete edges, there are also numerous white, feather-like floating artifacts. In contrast, the table instance model obtained by the method of this invention has very clear edges and surfaces. This indicates that the method of this invention is superior to SAGA and SAGD, and can obtain higher-quality 3D instance models.

[0069] To further illustrate the beneficial effects of the present invention, comparative tests were conducted using test data from the publicly available NVOS dataset (Z. Rennet al.. Neural Volumetric Object Selection. 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 6123–6132, June 2022.). The mean values ​​of the evaluation metrics of SAGA, SAGD, and the method of the present invention in various test scenarios were statistically analyzed, and the results are shown in Table 1.

[0070] Table 1 Evaluation metrics of SAGA, SAGD, and the method of this invention on the NVOS dataset.

[0071]

[0072] As can be seen from the comparison results in Table 1, the method of the present invention outperforms the SAGA and SAGD methods in all evaluation metrics. This fully demonstrates that the method of the present invention can obtain higher quality 3D instance models. Figure 3 The table instance model shown also visually confirms this fact.

[0073] The above steps and embodiments illustrate the entire process of the 3DGS instance segmentation method combining two-dimensional segmentation and multi-view voting described in this invention.

[0074] It should be understood that this embodiment is merely a specific example of the implementation of the present invention and should not be considered a limitation on the scope of protection of the present invention. Any equivalent modifications or alterations to the above content without departing from the spirit and scope of the present invention should be included within the scope of protection claimed by the present invention.

Claims

1. A 3DGS instance segmentation method combining two-dimensional segmentation and multi-view voting, characterized in that, Includes the following steps: Step 1: Obtain a multi-view image set of the 3D scene, including the target instance. Where V represents the number of viewpoints, I i This represents the image from the i-th viewpoint; Step 2: Use a 2D image segmentation model to process multi-view image sets of 3D scenes. All images are segmented to obtain a set of two-dimensional masks for the target instance. and utilize from Extracting target instances from multi-view image sets Step 3: Employ the SfM (Structured Motion Recovery) algorithm to analyze the multi-view image set. Estimate the intrinsic and extrinsic parameter set θ of each camera from different viewpoints, and simultaneously recover the sparse point cloud set Φ representing the 3D scene; Step 4: Combine the set of intrinsic and extrinsic parameters θ from each camera viewpoint, the set of sparse point clouds Φ, and the set of multi-view images of the target instance. As input to the original 3DGS 3D reconstruction algorithm, 3D reconstruction is performed to obtain an initial 3DGS model of the target instance represented by a series of 3D Gaussian ellipsoids. Where N represents the number of Gaussian ellipsoids, g j Denotes the j-th Gaussian ellipsoid; Step 5: Use a multi-view voting strategy to create an initial 3DGS model of the target instance. The optimization process is as follows: Step 5.1: Let Represents the Gaussian ellipsoid g j The center point coordinates are obtained; for each viewpoint v, its projection matrix is ​​calculated using the intrinsic and extrinsic parameters of the camera at that viewpoint. Then, using the projection matrix P v The center point of the Gaussian ellipsoid Transformed into normalized device coordinates in π(·) represents the conversion from homogeneous coordinates to Cartesian coordinates; then, based on the image size at viewpoint v, the center point o of the Gaussian ellipsoid is calculated. j Image coordinates in and They are The x and y coordinates are given, I is the identity matrix, and w and h are the image width and height at viewpoint v, respectively. Step 5.2: For each viewpoint v, define a Gaussian ellipsoid g. j single-view state like If the image area extends beyond the viewing angle v, then... like Within the image region and simultaneously located at the target instance mask M v The foreground area that is marked makes like Mask M, located within the image region but not belonging to the target instance v The foreground area that is marked makes This yields the single-view state vectors of all Gaussian ellipsoids at viewpoint v. Step 5.3: Calculate g for each Gaussian ellipsoid using the following formula. j Multi-perspective voting ratio Among them 1 A (x) is an indicator function, which has a value of 1 when x∈A and 0 otherwise; Step 5.4: Examine each Gaussian ellipsoid and adjust the multi-view voting ratio r. j Gaussian ellipsoid g below the preset threshold τ j From the initial 3DGS model of the target instance Delete, thus obtaining the optimized target instance model. Step 6: Output the optimized target instance model 2. The 3DGS instance segmentation method combining two-dimensional segmentation and multi-view voting according to claim 1, characterized in that, The two-dimensional image segmentation model in step 2 includes, but is not limited to, the basic image segmentation model SAM2; other image segmentation models can also be used in this step, as long as the model can successfully obtain the image and mask of the target instance from the multi-view image set of the three-dimensional scene.

Images