3D segmentation method based on 3D Gaussian sputtering and SAM 2D segmentation model

Abstract

The present invention discloses a 3D segmentation method based on 3D Gaussian sputtering and SAM 2D segmentation model, which belongs to the field of 3D image processing and includes the following steps: obtaining RGB images of existing public data sets, training 3D Gaussian scenes through 3D Gaussian sputtering method; constructing a new perspective selector, and selecting the optimal segmentation perspective for 3D segmentation through the new perspective selector; calculating 2D segmentation mask with the help of SAM 2D segmentation model, using separation algorithm based on 3D Gaussian spatial information to initially segment 3D Gaussian scene, and obtaining a rough 3D segmentation mask; constructing a background filter combining 3D Gaussian depth and color information, filtering out background Gaussian points in the rough 3D segmentation mask, retaining foreground segmentation results, and then obtaining the final 3D Gaussian scene; using 3D Gaussian sputtering method for rendering, and obtaining a foreground image after background removal. The present invention can achieve accurate foreground-background segmentation.

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional image processing, and in particular relates to a three-dimensional segmentation method based on three-dimensional Gaussian sputtering and a SAM two-dimensional segmentation model. Background Art

[0002] The purpose of 3D scene segmentation is to segment independent objects in the 3D reconstructed scene. In the prior art, Neural Radiance Field (NeRF) is a 3D reconstruction method based on deep learning. It generates high-quality images and perspectives by learning the volume representation of the scene through a neural network. Neural Radiance Field uses a multi-layer perceptron (MLP) network to map the point coordinates and perspective information in 3D space to the color and transparency of the point, thereby achieving realistic light transmission simulation. By optimizing the network parameters, Neural Radiance Field can reconstruct a complete 3D scene from a set of sparse 2D images. The 3D Gaussian sputtering method is a 3D point cloud representation method based on Gaussian distribution. It achieves efficient 3D reconstruction and rendering by representing points in 3D space as Gaussian kernels with specific shapes and distributions; each "sputtering" point (i.e., Gaussian point) is regarded as a 3D Gaussian distribution with position, color, transparency, and covariance matrix, which can capture local geometric details and lighting information. By optimizing and fusing these Gaussian points, high-quality, continuous 3D object surfaces can be generated, which is particularly suitable for processing complex 3D scenes, point cloud data, and fast rendering.

[0003] However, existing methods inevitably require additional training of existing models when integrating 2D information into 3D space, whether through neural radiation fields or 3D Gaussian sputtering. Therefore, these methods cannot start 3D segmentation tasks immediately after obtaining a 3D scene, require a long time for scene training and feature embedding, and the foreground-background segmentation results are inaccurate. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a 3D segmentation method based on 3D Gaussian sputtering and SAM 2D segmentation model. On the basis of 3D Gaussian sputtering, a new perspective selector is designed to find the most suitable 2D image for prompting segmentation by comprehensively judging the camera matrix between 2D images and the projection entropy value of 3D Gaussian in 2D images. At the same time, the present invention also designs a background filter combining 3D Gaussian color and depth information to filter out background Gaussian points and obtain more refined foreground segmentation results.

[0005] The technical solution of the present invention is as follows:

[0006] A three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model comprises the following steps:

[0007] Step 1: Obtain RGB images of existing public data sets, and train a 3D Gaussian scene composed of a large number of 3D Gaussian distributions through a 3D Gaussian sputtering method;

[0008] Step 2: construct a new view selector through the uncertainty between images, the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and select the optimal segmentation view for 3D segmentation through the new view selector;

[0009] Step 3: Calculate the two-dimensional segmentation mask with the help of the SAM two-dimensional segmentation model, and use the separation algorithm based on three-dimensional Gaussian spatial information to initially segment the three-dimensional Gaussian scene to obtain a rough three-dimensional segmentation mask;

[0010] Step 4: construct a background filter that combines 3D Gaussian depth and color information, filter out background Gaussian points in the rough 3D segmentation mask, retain the foreground segmentation result, and then obtain the final 3D Gaussian scene;

[0011] Step 5: Render the final three-dimensional Gaussian scene using a three-dimensional Gaussian sputtering method to obtain a rendering result, which is a foreground image after removing the background.

[0012] Furthermore, in step 1, the public data sets used include the forward scene data set LLFF and the 360-degree scene data set Mip-NeRF360; wherein the forward scene data set LLFF includes eight scenes, namely, ferns, flowers, towers, leaves, triceratops skulls, orchids, rooms, and Tyrannosaurus Rex skeletons, and the number of images in each scene is different, all between 20 and 62; the 360-degree scene data set Mip-NeRF360 includes seven scenes, namely, bicycles, bonsai, room corners, gardens, kitchens, Lego in the room, and tree stumps, and the number of images in each scene is different, all between 180 and 230;

[0013] The 3D Gaussian sputtering method is used to reconstruct and render the public dataset, and a large number of 3D Gaussian distributions with position, color, transparency and covariance matrix are obtained.

[0014] Furthermore, in step 2, the working process of the new perspective selector is as follows:

[0015] Step 2.1, use the entropy calculation formula to calculate the entropy of each three-dimensional Gaussian point. The entropy calculation formula is as follows:

[0016] (1);

[0017] in, For the 3D Gaussian points Entropy of Indicates 3D Gaussian points The covariance matrix of

[0018] Step 2.2, according to the camera's intrinsic parameter matrix and extrinsic parameter matrix provided by the public data set, and based on the projection transformation formula, calculate the projection coordinates of each 3D Gaussian point in the 2D plane, and then compare the projection coordinates with the size of the image frame to determine whether to include the entropy of the current 3D Gaussian point in the 2D image entropy value calculation;

[0019] The projection transformation formula is as follows:

[0020] (2);

[0021] in, is the internal parameter matrix; is the external parameter matrix, is the rotation matrix, is the translation matrix; Representative The center coordinates of the three-dimensional Gaussian points, , , are the coordinate values ​​of the three-dimensional Gaussian point in the x-direction, y-direction, and z-direction in the world coordinate system respectively; Representative The coordinates of the three-dimensional Gaussian points projected onto the two-dimensional plane are: , are the horizontal and vertical coordinates in the two-dimensional plane respectively;

[0022] The projection coordinates of each 3D Gaussian point in the 2D plane can be calculated by formula (2), and the projection coordinates of each 3D Gaussian point are compared with the size of the image frame. Specifically, if the 2D projection coordinates of the current 3D Gaussian point are larger than the 2D range of the image, the entropy of the current 3D Gaussian point is included in the entropy value calculation of the 2D image.

[0023] Step 2.3: Use the maximum pooling method to perform entropy feature compression calculation on the entropy of all three-dimensional Gaussian points included in the two-dimensional image entropy calculation. The formula of the maximum pooling method is as follows:

[0024] (3);

[0025] in, It is the result of entropy feature compression; Represents the size of the unit pooling matrix, and the calculation formula is , Indicates the desired compression length. Represents the original entropy value array Length; For the The three-dimensional Gaussian points in the unit pooling matrix size is Entropy of time; For the The three-dimensional Gaussian points in the unit pooling matrix size is Entropy of time;

[0026] Step 2.4: Calculate the information entropy of the two-dimensional image; the specific process is:

[0027] First, a Bayesian neural network is used to generate the probability distribution of the predicted category. The input of the Bayesian neural network includes the view pixel value, the camera's intrinsic and extrinsic parameter matrices, and the entropy feature compression result. , the output is the probability distribution set of all predicted categories , Indicates The first two-dimensional image The probability distribution of the predicted categories, Represents the total number of categories, The number of prediction categories generated for the model;

[0028] After that, the variance of all predicted categories is calculated and normalized to obtain the normalized probability distribution. Finally, the information entropy of each two-dimensional image is calculated using the normalized probability distribution. The formula is as follows:

[0029] (4);

[0030] (5);

[0031] (6);

[0032] in, For the The variance of all predicted categories of a 2D image; is the variance calculation; For the Two-dimensional image Normalized probability of each category; A variable that traverses the categories in each two-dimensional image; For the The information entropy of a two-dimensional image; is the sequence number of the intermediate variable used for traversal;

[0033] Step 2.6, calculate the penalty information entropy of each two-dimensional image, and select the two-dimensional image with the largest reciprocal penalty information entropy as the optimal two-dimensional image most suitable for three-dimensional segmentation. The viewing angle corresponding to the optimal two-dimensional image is the optimal segmentation viewing angle.

[0034] Furthermore, the specific process of step 2.6 is as follows:

[0035] Step 2.6.1, calculate the first The cosine similarity between a view and all other views is calculated, and the maximum similarity is used as the coefficient of the penalty information entropy. The cosine similarity calculation formula is:

[0036] (7);

[0037] in, , are the serial numbers of different perspectives; , Respectively represent Perspective, The intrinsic and extrinsic matrix of the camera with each viewing angle; Representative Perspective and Cosine similarity of perspectives;

[0038] The maximum similarity formula is:

[0039] (8);

[0040] in, For the The maximum similarity between the intrinsic parameter matrix and the extrinsic parameter matrix of the camera with each view; Represents the maximum value in the array; For the The internal and external parameter matrices of the first view are The cosine similarity between the internal parameter matrix and the external parameter matrix of each view;

[0041] Step 2.6.2, calculate the penalty information entropy of each two-dimensional image; the formula is:

[0042] (9);

[0043] in, Representative The penalized information entropy of a two-dimensional image;

[0044] Step 2.6.3, compare the inverse of the penalty information entropy of all two-dimensional images, find the image with the largest inverse penalty information entropy value, and use it as the optimal two-dimensional image that is most suitable for prompting segmentation. The perspective corresponding to the optimal two-dimensional image is the optimal segmentation perspective.

[0045] Furthermore, the specific process of step 3 is as follows:

[0046] Step 3.1, mark the cue points in the optimal two-dimensional image, and automatically generate a two-dimensional mask with the help of the SAM two-dimensional segmentation model;

[0047] Step 3.2: Project the 2D mask into 3D space with the help of the camera's intrinsic and extrinsic matrix to obtain projected Gaussian points. If a 3D Gaussian point is detected within the projection range of the 2D mask, the 3D Gaussian point is included in the original 3D mask. The formula is as follows:

[0048] (10);

[0049] in, is the original 3D mask of 3D Gaussian points; is a two-dimensional mask;

[0050] Step 3.3: Use a separation algorithm based on three-dimensional Gaussian spatial information to reduce background interference and obtain a rough three-dimensional segmentation mask; the specific formula is as follows:

[0051] (11);

[0052] (12);

[0053] (13);

[0054] (14);

[0055] (15);

[0056] in, For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; Indicates that it is different from The three-dimensional Gaussian point number of ; For the 3D Gaussian points; For the 3D Gaussian points The nearest area 3D Gaussian points; is an exponential function with base e; Representative 3D Gaussian points and The Euclidean distance between three-dimensional Gaussian points; represents the Euclidean distance; It is Neighborhood distance metric of three-dimensional Gaussian points; For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; represents a rough 3D Gaussian point set comprehensively judged by using density and distance information, and the rough 3D Gaussian point set constitutes a rough 3D segmentation mask; is the neighborhood density threshold; is the domain distance threshold; It is the operation of selecting the median in an array; Represents Gaussian point adjacent points.

[0057] Furthermore, the specific process of step 4 is as follows:

[0058] Step 4.1, calculate the depth weight of the three-dimensional Gaussian point; calculate the depth weight of the three-dimensional Gaussian point through the camera's intrinsic matrix and extrinsic matrix, and the world coordinates of the three-dimensional Gaussian point The distance between the three-dimensional Gaussian point and the camera; then the first The depth weight of the 3D Gaussian points ;

[0059] Step 4.2, calculate the color weight of the three-dimensional Gaussian point; the specific process is:

[0060] Step 4.2.1, calculate the color distance of each three-dimensional Gaussian point, the formula is as follows:

[0061] (16);

[0062] in, For the The color distance of three-dimensional Gaussian points; For the Color information of three-dimensional Gaussian points; For adjacent The average color of three-dimensional Gaussian points;

[0063] Step 4.2.2: Calculate the color weight of each three-dimensional Gaussian point by maximum normalization. The formula is as follows:

[0064] (17);

[0065] in, For the The color weights of the three-dimensional Gaussian points; Represents adjacent The maximum color value among three-dimensional Gaussian points; is the color distance array;

[0066] Step 4.3: Combine the 3D Gaussian point depth weight and color weight to calculate the final weight. The calculation formula is as follows:

[0067] (18);

[0068] in, For the The final weights of the three-dimensional Gaussian points; and are different hyperparameters;

[0069] Step 4.4: Pre-set a threshold , filter out weight values ​​below the threshold All the background three-dimensional Gaussian points are obtained to obtain the finely segmented foreground result, and then the final three-dimensional Gaussian scene is obtained.

[0070] The beneficial technical effects brought about by the present invention are as follows: the present invention assists users in selecting the two-dimensional image that is most suitable for prompting segmentation from a large number of images through a new perspective selector that comprehensively evaluates camera matrix information and three-dimensional Gaussian projection entropy value information, thereby improving the efficiency of human-computer interaction; a background filter that combines three-dimensional Gaussian color and depth information is proposed to refine the foreground-background segmentation results, while avoiding secondary training, greatly improving the segmentation efficiency of the overall method. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 The flowchart of the three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model of the present invention.

[0072] Figure 2 This is an example flow chart of the three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0074] like Figure 1 and Figure 2 As shown, a three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model includes the following steps:

[0075] Step 1: Obtain RGB images of existing public data sets, and train a three-dimensional Gaussian scene composed of a large number of three-dimensional Gaussian distributions through a three-dimensional Gaussian sputtering method.

[0076] The public data sets used in the present invention include the forward scene data set LLFF and the 360-degree scene data set Mip-NeRF360. Among them, the forward scene data set LLFF includes eight scenes, namely, ferns, flowers, towers, leaves, Triceratops skulls, orchids, rooms, and Tyrannosaurus Rex skeletons. The number of images in each scene is different, ranging from 20 to 62. The 360-degree scene data set Mip-NeRF360 contains seven scenes, namely, bicycles, bonsai, room corners, gardens, kitchens, Lego in the room, and tree stumps. The number of images in each scene is different, ranging from 180 to 230.

[0077] A 3D Gaussian point is a mathematical model used to represent a point in 3D space, usually defined by its position (mean) and shape (covariance matrix). A 3D Gaussian point set can be rendered into a 2D image using the 3D Gaussian sputtering method.

[0078] The 3D Gaussian sputtering method is used to reconstruct and render the public dataset, and a large number of 3D Gaussian distributions with position, color, transparency and covariance matrix are obtained.

[0079] Step 2: construct a new view selector through the uncertainty between images, the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and select the optimal segmentation view most suitable for three-dimensional segmentation through the new view selector.

[0080] The present invention designs a method to evaluate the information content of an image by combining three-dimensional Gaussian points, so as to select the view with the largest information content in the image set as the next best view. In the design of the new perspective selector, the entropy of the three-dimensional Gaussian distribution is calculated using covariance. The entropy represents the uncertainty of the three-dimensional Gaussian distribution, and the uncertainty obtained from the projections at different angles is incorporated into the new perspective selection evaluation criteria. At the same time, the influence of camera pose differences on the segmentation perspective is also considered.

[0081] The new perspective selector works as follows:

[0082] Step 2.1, use the entropy calculation formula to calculate the entropy of each three-dimensional Gaussian point. The entropy calculation formula is as follows:

[0083] (1);

[0084] in, For the 3D Gaussian points Entropy of Indicates 3D Gaussian points The covariance matrix of .

[0085] Step 2.2, the camera's intrinsic parameter matrix and extrinsic parameter matrix are a mathematical tool used to describe how the camera projects points in three-dimensional space onto a two-dimensional image plane. It contains internal parameters (such as focal length and principal point position) and external parameters (such as the position and orientation of the camera). Through the camera's intrinsic parameter matrix and extrinsic parameter matrix, the world coordinates can be converted into image coordinates, which are widely used in computer vision, image processing, and three-dimensional reconstruction. The present invention calculates the projection coordinates of each three-dimensional Gaussian point in the two-dimensional plane based on the projection transformation formula according to the intrinsic parameter matrix and extrinsic parameter matrix of the camera provided by the LLFF dataset and the Mip-NeRF360 dataset, and then compares the projection coordinates with the size of the image frame to determine whether to include the entropy of the current three-dimensional Gaussian point in the two-dimensional image entropy value calculation;

[0086] The projection transformation formula is as follows:

[0087] (2);

[0088] in, is the internal parameter matrix; is the external parameter matrix, is the rotation matrix, is the translation matrix; Representative The center coordinates of the three-dimensional Gaussian points, , , are the coordinate values ​​of the three-dimensional Gaussian point in the x-direction, y-direction, and z-direction in the world coordinate system respectively; Representative The coordinates of the three-dimensional Gaussian points projected onto the two-dimensional plane are: , are the horizontal and vertical coordinates in the two-dimensional plane respectively.

[0089] The projection coordinates of each 3D Gaussian point in the 2D plane can be calculated by formula (2), and the projection coordinates of each 3D Gaussian point are compared with the size of the image frame. Specifically, if the 2D projection coordinates of the current 3D Gaussian point are larger than the 2D range of the image, the entropy of the current 3D Gaussian point is included in the entropy value calculation of the 2D image.

[0090] Step 2.3: Use the maximum pooling method to perform entropy feature compression calculation on the entropy of all three-dimensional Gaussian points included in the two-dimensional image entropy calculation. The formula of the maximum pooling method is as follows:

[0091] (3);

[0092] in, is the entropy feature compression result, which can be used to measure the uncertainty of three-dimensional Gaussian distribution; Represents the size of the unit pooling matrix, and the calculation formula is , Indicates the desired compression length. Represents the original entropy value array Length; For the The three-dimensional Gaussian points in the unit pooling matrix size is Entropy of time; For the The three-dimensional Gaussian points in the unit pooling matrix size is The entropy of time.

[0093] Step 2.4, calculate the information entropy of the two-dimensional image. Image information entropy is an indicator to measure the amount and complexity of information in an image. It reflects the distribution of image pixel values ​​and their uncertainty. The higher the entropy value, the richer the details and changes in the image, and the more complex the image; the lower the entropy value, the simpler the image is and may contain more uniform areas. Image information entropy is often used in image processing and analysis, such as image compression, feature extraction and quality assessment. The specific process is:

[0094] First, a Bayesian neural network is used to generate the probability distribution of the predicted category. The input of the Bayesian neural network includes the view pixel value (i.e. the value of the image pixel), the camera's intrinsic and extrinsic parameter matrices, and the entropy feature compression result. , the output is the probability distribution set of all predicted categories , Indicates The first two-dimensional image The probability distribution of the predicted categories, Represents the total number of categories, The number of predicted classes generated for the model.

[0095] After that, the variance of all predicted categories is calculated and normalized to obtain the normalized probability distribution. Finally, the information entropy of each two-dimensional image is calculated using the normalized probability distribution. The formula is as follows:

[0096] (4);

[0097] (5);

[0098] (6);

[0099] in, For the The variance of all predicted categories of a 2D image; is the variance calculation; For the Two-dimensional image Normalized probability of each category; A variable that traverses the categories in each two-dimensional image; For the The information entropy of a two-dimensional image; It is the sequence number of the intermediate variable used for traversal.

[0100] Step 2.6, calculate the penalty information entropy of image information evaluation by combining the view pixel value, the camera's intrinsic parameter matrix and the extrinsic parameter matrix. The larger the reciprocal of the penalty information entropy, the more suitable the image is for prompt segmentation. Therefore, this step calculates the penalty information entropy of each two-dimensional image, and selects the two-dimensional image with the largest reciprocal of the penalty information entropy as the optimal two-dimensional image most suitable for three-dimensional segmentation. The viewing angle corresponding to the optimal two-dimensional image is the optimal segmentation viewing angle; the specific process is:

[0101] Step 2.6.1, calculate the first The cosine similarity between a view and all other views is calculated, and the maximum similarity is used as the coefficient of the penalty information entropy. The cosine similarity calculation formula is:

[0102] (7);

[0103] in, , are the serial numbers of different perspectives; , Respectively represent Perspective, The intrinsic and extrinsic matrix of the camera with each viewing angle; Representative Perspective and Cosine similarity of perspectives;

[0104] The maximum similarity formula is:

[0105] (8);

[0106] in, For the The maximum similarity between the intrinsic parameter matrix and the extrinsic parameter matrix of the camera with each viewing angle; Represents the maximum value in the array; For the The internal and external parameter matrices of the first view are The cosine similarity between the internal parameter matrix and the external parameter matrix of each view;

[0107] Step 2.6.2: Because the closer the camera’s intrinsic parameter matrix and extrinsic parameter matrix are, the less cross-view information is obtained, so the penalty information entropy of each two-dimensional image is calculated by maximum similarity; the formula is:

[0108] (9);

[0109] in, Representative The penalized information entropy of a two-dimensional image;

[0110] Step 2.6.3, compare the inverse of the penalty information entropy of all two-dimensional images, find the image with the largest inverse penalty information entropy value, and use it as the optimal two-dimensional image that is most suitable for prompting segmentation. The perspective corresponding to the optimal two-dimensional image is the optimal segmentation perspective.

[0111] Step 3: Calculate the two-dimensional segmentation mask with the help of the SAM two-dimensional segmentation model, and use the separation algorithm based on three-dimensional Gaussian spatial information to initially segment the three-dimensional Gaussian scene to obtain a rough three-dimensional segmentation mask.

[0112] The present invention obtains the optimal segmentation perspective after passing through the new perspective selector and starts the 3D instance segmentation work. This step will introduce how to obtain a rough 3D segmentation mask. The steps are as follows:

[0113] Step 3.1, mark the prompt points in the optimal two-dimensional image, and obtain the two-dimensional mask with the help of the SAM two-dimensional segmentation model. The SAM (Segment Anything Model) segmentation model is an efficient image segmentation tool based on deep learning, which aims to achieve high-quality two-dimensional instance segmentation. It has a wide range of adaptability and can handle various image types and scenes, whether it is a natural environment, indoor images or specific objects. With the help of the SAM two-dimensional segmentation model, the present invention allows users to generate prompt points by clicking on the objects they want to segment in the optimal two-dimensional image, and the SAM two-dimensional segmentation model can automatically generate a two-dimensional mask based on the prompt points. Figure 2 Examples of 3 views are shown in , for example, the SAM 2D segmentation model can generate a 2D mask based on cue point 1.

[0114] Step 3.2: Project the 2D mask into 3D space with the help of the camera's intrinsic and extrinsic matrix to obtain projected Gaussian points. If a 3D Gaussian point is detected within the projection range of the 2D mask, the 3D Gaussian point is included in the original 3D mask. The formula is as follows:

[0115] (10);

[0116] in, is the original 3D mask of 3D Gaussian points; is a two-dimensional mask;

[0117] Step 3.3, use a separation algorithm based on 3D Gaussian spatial information to reduce background interference and obtain a rough 3D segmentation mask. If the result obtained by only using the projection method will have many 3D Gaussian points belonging to the background mixed in, the present invention uses the position information of the Gaussian points to treat the Gaussian points that are not close to the main instance in the spatial distribution as background noise. Specifically, by calculating the Gaussian point distance The comprehensive value of the distance between adjacent points determines whether to retain the three-dimensional Gaussian point, and the density information of the point set is considered jointly; the specific formula is as follows:

[0118] (11);

[0119] (12);

[0120] (13);

[0121] (14);

[0122] (15);

[0123] in, For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; Indicates that it is different from The three-dimensional Gaussian point number of ; For the 3D Gaussian points; For the 3D Gaussian points The nearest area 3D Gaussian points; is an exponential function with base e; Representative 3D Gaussian points and The Euclidean distance between three-dimensional Gaussian points; represents the Euclidean distance; It is Neighborhood distance metric of three-dimensional Gaussian points; For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; represents a rough 3D Gaussian point set comprehensively judged by using density and distance information, and the rough 3D Gaussian point set constitutes a rough 3D segmentation mask; is the neighborhood density threshold; is the domain distance threshold; It is the median operation in selecting an array.

[0124] Step 4: Construct a background filter that combines 3D Gaussian depth and color information, filter out the background Gaussian points in the rough 3D segmentation mask, retain the accurate foreground segmentation result, and then obtain the final 3D Gaussian scene. Studies have found that relying solely on spatial information cannot effectively eliminate the background 3D Gaussian near the foreground. In order to improve the accuracy of foreground-background segmentation, the present invention designs a background filter that jointly considers 3D Gaussian depth and color information. The specific process is as follows:

[0125] Step 4.1, calculate the depth weight of the three-dimensional Gaussian point. 3D Gaussian points If the distance from the camera is very far, its weight will be smaller than other 3D Gaussian points that are closer to the camera. Through the camera's intrinsic and extrinsic matrix and the world coordinates of the 3D Gaussian point, the first The distance between the three-dimensional Gaussian point and the camera. Then the first The depth weight of the 3D Gaussian points .

[0126] Step 4.2, calculate the color weight of the three-dimensional Gaussian point. If the hue of the three-dimensional Gaussian point does not match the surrounding hue, the color weight of the three-dimensional Gaussian point will be reduced. In consideration of the amount of calculation, the present invention uses the zero-order color information of the three-dimensional Gaussian point. As the basis for calculating color distance, color distance represents the color information of a three-dimensional Gaussian point and its adjacent The difference in the average color of three-dimensional Gaussian points, , , The zero-order color information of the three channels of R, G, and B of the three-dimensional Gaussian point respectively; the specific process is:

[0127] Step 4.2.1, calculate the color distance of each three-dimensional Gaussian point, the formula is as follows:

[0128] (16);

[0129] in, For the The color distance of three-dimensional Gaussian points; For the Color information of three-dimensional Gaussian points; For adjacent The average color of three-dimensional Gaussian points;

[0130] Step 4.2.2: Calculate the color weight of each three-dimensional Gaussian point by maximum normalization. The formula is as follows:

[0131] (17);

[0132] in, For the The color weights of the three-dimensional Gaussian points; Represents adjacent The maximum color value among three-dimensional Gaussian points; is the color distance array.

[0133] Step 4.3, combine the depth weight and color weight of the 3D Gaussian point to calculate the final weight. The depth and color joint filter of the Gaussian point will comprehensively consider the depth information and color information of the 3D Gaussian point to finally calculate a final weight. The probability of each 3D Gaussian point being included in the final 3D mask is evaluated based on the final weight. The calculation formula is as follows:

[0134] (18);

[0135] in, For the The final weights of the three-dimensional Gaussian points; and are different hyperparameters obtained through a large number of experiments. The method of the present invention can ensure that the target three-dimensional Gaussian points with color deviation will not be easily discarded.

[0136] Step 4.4: Pre-set a threshold , filter out weight values ​​below the threshold By using all the three-dimensional Gaussian points of the background, we can get the finely segmented foreground result and then get the final three-dimensional Gaussian scene.

[0137] Step 5: Render the final three-dimensional Gaussian scene using a three-dimensional Gaussian sputtering method to obtain a rendering result, which is a foreground image after removing the background.

[0138] In order to verify the feasibility and superiority of the present invention, the following experiment is given.

[0139] Experiment 1: The present invention was compared with four existing methods. The comparative experiment was conducted on the LLFF dataset. Eight scenes, including ferns, flowers, towers, leaves, Triceratops skulls, orchids, rooms, and Tyrannosaurus Rex skeletons, were selected for main object segmentation, and the segmentation results were compared with the segmentation results of the present invention. The comparison results are shown in Table 1. Among them, the Graph-Cut 3D method is a three-dimensional data segmentation method based on graph theory. It models three-dimensional data (such as voxels or point clouds) as a graph and divides the data into foreground and background through a minimum cut algorithm. Each three-dimensional data unit is a node in the graph, and the edges between nodes represent their similarities. The NVOS method is a benchmark dataset and method for visual object segmentation, which is mainly used in video object segmentation tasks. The goal of this task is to segment specific objects frame by frame based on video sequences, that is, in each frame in the video, accurately mark the area of ​​the target object. ISRF is an interactive segmentation method for radiation field data, which aims to accurately segment complex three-dimensional scenes through user input. This method combines radiation field technology with user interaction, allowing users to guide the segmentation process through simple annotations in three-dimensional space, and is suitable for complex three-dimensional reconstruction models such as neural radiation fields. Its main contribution is that it makes 3D segmentation more intuitive and efficient through interactive operations, while avoiding the errors in traditional fully automatic segmentation methods, especially when dealing with complex object shapes and details. The SGISRF method combines the powerful rendering capabilities of neural radiance fields with interactive user guidance, allowing users to quickly define the foreground and background in the scene through simple annotations and effectively handle diverse 3D environments; its main contribution is that it can adapt to different 3D scenes and object forms through a scene-universal framework, avoiding the limitations of traditional automatic segmentation methods. The intersection-over-union ratio refers to the ratio of the intersection to the union between the predicted bounding box and the true bounding box, indicating how much overlap there is. The average intersection-over-union ratio is the average of the intersection-over-union ratios of all categories. The higher the average intersection-over-union ratio, the more overlap there is between the actual segmentation result and the correct result. The higher the average accuracy, the more correctly segmented the actual segmentation result is.

[0140] Table 1 Comparison results of the present invention and other four methods on the LLFF dataset

[0141] .

[0142] As can be seen from Table 1, on the LFFF dataset, the method proposed in the present invention achieves the best results in terms of average intersection-over-union ratio and average accuracy, with an average intersection-over-union ratio of 86.9% and an average accuracy of 98%, effectively improving the accuracy of three-dimensional segmentation. In addition, based on the three-dimensional Gaussian sputtering technology, the present invention greatly improves the rendering speed and can achieve real-time interactive three-dimensional segmentation.

[0143] Experiment 2: Ablation experiment. In order to verify the effectiveness of the joint filter based on the depth and color information of three-dimensional Gaussian points, the present invention conducts an ablation experiment on the LLFF dataset. The experimental results are shown in Table 2:

[0144] Table 2 Ablation experiment results on LLFF dataset

[0145] .

[0146] It can be seen from Table 2 that after adopting the joint filter based on the depth and color information of three-dimensional Gaussian points of the present invention, the intersection-over-union ratio index of the overall segmentation is significantly improved, with an average increase of 2.2%, while the accuracy index is slightly improved by an average of 0.8% due to its high inherent value.

[0147] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model, characterized in that: The steps include: Step 1: Obtain RGB images of existing public data sets, and train a 3D Gaussian scene composed of a large number of 3D Gaussian distributions through a 3D Gaussian sputtering method; Step 2: construct a new view selector through the uncertainty between images, the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, and select the optimal segmentation view for 3D segmentation through the new view selector; Step 3: Calculate the two-dimensional segmentation mask with the help of the SAM two-dimensional segmentation model, and use the separation algorithm based on three-dimensional Gaussian spatial information to initially segment the three-dimensional Gaussian scene to obtain a rough three-dimensional segmentation mask; Step 4: construct a background filter that combines 3D Gaussian depth and color information, filter out background Gaussian points in the rough 3D segmentation mask, retain the foreground segmentation result, and then obtain the final 3D Gaussian scene; Step 5: Render the final three-dimensional Gaussian scene using a three-dimensional Gaussian sputtering method to obtain a rendering result, which is a foreground image after removing the background.

2. The three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model according to claim 1, characterized in that: In the step 1, the public data sets used include the forward scene data set LLFF and the 360-degree scene data set Mip-NeRF360; wherein the forward scene data set LLFF includes eight scenes, namely, ferns, flowers, towers, leaves, triceratops skulls, orchids, rooms, and Tyrannosaurus Rex skeletons, and the number of images in each scene is different, all between 20 and 62; the 360-degree scene data set Mip-NeRF360 includes seven scenes, namely, bicycles, bonsai, room corners, gardens, kitchens, Lego in the room, and tree stumps, and the number of images in each scene is different, all between 180 and 230; The 3D Gaussian sputtering method is used to reconstruct and render the public dataset, and a large number of 3D Gaussian distributions with position, color, transparency and covariance matrix are obtained.

3. The three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model according to claim 1, characterized in that: In step 2, the working process of the new perspective selector is as follows: Step 2.1, use the entropy calculation formula to calculate the entropy of each three-dimensional Gaussian point. The entropy calculation formula is as follows: (1)； in, For the 3D Gaussian points Entropy of Indicates 3D Gaussian points The covariance matrix of Step 2.2, according to the camera's intrinsic parameter matrix and extrinsic parameter matrix provided by the public data set, and based on the projection transformation formula, calculate the projection coordinates of each 3D Gaussian point in the 2D plane, and then compare the projection coordinates with the size of the image frame to determine whether to include the entropy of the current 3D Gaussian point in the 2D image entropy value calculation; The projection transformation formula is as follows: (2)； in, is the internal parameter matrix; is the external parameter matrix, is the rotation matrix, is the translation matrix; Representative The center coordinates of the three-dimensional Gaussian points, , , are the coordinate values ​​of the three-dimensional Gaussian point in the x-direction, y-direction, and z-direction in the world coordinate system respectively; Representative The coordinates of the three-dimensional Gaussian points projected onto the two-dimensional plane are: , are the horizontal and vertical coordinates in the two-dimensional plane respectively; The projection coordinates of each 3D Gaussian point in the 2D plane can be calculated by formula (2), and the projection coordinates of each 3D Gaussian point are compared with the size of the image frame. Specifically, if the 2D projection coordinates of the current 3D Gaussian point are larger than the 2D range of the image, the entropy of the current 3D Gaussian point is included in the entropy value calculation of the 2D image. Step 2.3: Use the maximum pooling method to perform entropy feature compression calculation on the entropy of all three-dimensional Gaussian points included in the two-dimensional image entropy calculation. The formula of the maximum pooling method is as follows: (3)； in, It is the result of entropy feature compression; Represents the size of the unit pooling matrix, and the calculation formula is , Indicates the desired compression length. Represents the original entropy value array Length; For the The three-dimensional Gaussian points in the unit pooling matrix size is Entropy of time; For the The three-dimensional Gaussian points in the unit pooling matrix size is Entropy of time; Step 2.4: Calculate the information entropy of the two-dimensional image; the specific process is: First, a Bayesian neural network is used to generate the probability distribution of the predicted category. The input of the Bayesian neural network includes the view pixel value, the camera's intrinsic and extrinsic parameter matrices, and the entropy feature compression result. , the output is the probability distribution set of all predicted categories , Indicates The first two-dimensional image The probability distribution of the predicted categories, Represents the total number of categories, The number of prediction categories generated for the model; After that, the variance of all predicted categories is calculated and normalized to obtain the normalized probability distribution. Finally, the information entropy of each two-dimensional image is calculated using the normalized probability distribution. The formula is as follows: (4)； (5)； (6)； in, For the The variance of all predicted categories of a two-dimensional image; is the variance calculation; For the Two-dimensional image Normalized probability of each category; A variable that traverses the categories in each two-dimensional image; For the The information entropy of a two-dimensional image; is the sequence number of the intermediate variable used for traversal; Step 2.6, calculate the penalty information entropy of each two-dimensional image, and select the two-dimensional image with the largest reciprocal penalty information entropy as the optimal two-dimensional image most suitable for three-dimensional segmentation. The viewing angle corresponding to the optimal two-dimensional image is the optimal segmentation viewing angle.

4. The three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model according to claim 3, characterized in that: The specific process of step 2.6 is as follows: Step 2.6.1, calculate the first The cosine similarity between a view and all other views is calculated, and the maximum similarity is used as the coefficient of the penalty information entropy. The cosine similarity calculation formula is: (7)； in, , are the serial numbers of different perspectives; , Respectively represent Perspective, The intrinsic and extrinsic matrix of the camera with each viewing angle; Representative Perspective and Cosine similarity of perspectives; The maximum similarity formula is: (8)； in, For the The maximum similarity between the intrinsic parameter matrix and the extrinsic parameter matrix of the camera with each view; Represents the maximum value in the array; For the The internal and external parameter matrices of the first view are The cosine similarity between the internal parameter matrix and the external parameter matrix of each view; Step 2.6.2, calculate the penalty information entropy of each two-dimensional image; the formula is: (9)； in, Representative The penalized information entropy of a two-dimensional image; Step 2.6.3, compare the inverse of the penalty information entropy of all two-dimensional images, find the image with the largest inverse penalty information entropy value, and use it as the optimal two-dimensional image that is most suitable for prompting segmentation. The perspective corresponding to the optimal two-dimensional image is the optimal segmentation perspective.

5. The three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model according to claim 4, characterized in that: The specific process of step 3 is as follows: Step 3.1, mark the cue points in the optimal two-dimensional image, and automatically generate a two-dimensional mask with the help of the SAM two-dimensional segmentation model; Step 3.2: Project the 2D mask into 3D space with the help of the camera's intrinsic and extrinsic matrix to obtain projected Gaussian points. If a 3D Gaussian point is detected within the projection range of the 2D mask, the 3D Gaussian point is included in the original 3D mask. The formula is as follows: (10)； in, is the original 3D mask of 3D Gaussian points; is a two-dimensional mask; Step 3.3: Use a separation algorithm based on three-dimensional Gaussian spatial information to reduce background interference and obtain a rough three-dimensional segmentation mask; the specific formula is as follows: (11)； (12)； (13)； (14)； (15)； in, For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; Indicates that it is different from The three-dimensional Gaussian point number of ; For the 3D Gaussian points; For the 3D Gaussian points The nearest area 3D Gaussian points; is an exponential function with base e; Representative 3D Gaussian points and The Euclidean distance between three-dimensional Gaussian points; represents the Euclidean distance; It is Neighborhood distance metric of three-dimensional Gaussian points; For the The 3D Gaussian neighborhood density estimate of 3D Gaussian points; represents a rough 3D Gaussian point set comprehensively judged by using density and distance information, and the rough 3D Gaussian point set constitutes a rough 3D segmentation mask; is the neighborhood density threshold; is the domain distance threshold; It is the operation of selecting the median in an array; Represents Gaussian point adjacent points.

6. The three-dimensional segmentation method based on three-dimensional Gaussian sputtering and SAM two-dimensional segmentation model according to claim 5, characterized in that: The specific process of step 4 is as follows: Step 4.1, calculate the depth weight of the three-dimensional Gaussian point; calculate the depth weight of the three-dimensional Gaussian point through the camera's intrinsic matrix and extrinsic matrix, and the world coordinates of the three-dimensional Gaussian point The distance between the three-dimensional Gaussian point and the camera; then the first The depth weight of the 3D Gaussian points ; Step 4.2, calculate the color weight of the three-dimensional Gaussian point; the specific process is: Step 4.2.1, calculate the color distance of each three-dimensional Gaussian point, the formula is as follows: (16)； in, For the The color distance of three-dimensional Gaussian points; For the Color information of three-dimensional Gaussian points; For adjacent The average color of three-dimensional Gaussian points; Step 4.2.2: Calculate the color weight of each three-dimensional Gaussian point by maximum normalization. The formula is as follows: (17)； in, For the The color weights of the three-dimensional Gaussian points; Represents adjacent The maximum color value among three-dimensional Gaussian points; is the color distance array; Step 4.3: Combine the 3D Gaussian point depth weight and color weight to calculate the final weight. The calculation formula is as follows: (18)； in, For the The final weights of the three-dimensional Gaussian points; and are different hyperparameters; Step 4.4: Set a threshold in advance , filter out weight values ​​below the threshold All the background three-dimensional Gaussian points are obtained to obtain the finely segmented foreground result, and then the final three-dimensional Gaussian scene is obtained.

Images