Three-dimensional segmentation method and system based on sparse visual angle three-dimensional reconstruction and dynamic alignment

Abstract

The invention discloses a three-dimensional segmentation method and system based on sparse visual angle three-dimensional reconstruction and dynamic alignment, and relates to the technical field of three-dimensional target segmentation. Inputting the preprocessed pictures into an input picture group by taking two view pictures as one group, and inputting the input picture group into a neural network MAST3R to obtain point cloud data of each input picture group in the same camera coordinate system; traversing the point cloud corresponding to each input picture group, and learning a transformation matrix of each view mapped to a world coordinate system based on a dynamic global alignment strategy; and two-dimensional segmentation masks of all view angles are obtained based on user prompts and SAM2, and the two-dimensional segmentation masks are aligned to a world coordinate system through the learned matrix. The feed-forward three-dimensional segmentation model based on the two-dimensional view is provided based on the neural network MAST3R, real-time interactive three-dimensional segmentation in any scene can be achieved, performance is guaranteed, and efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional target segmentation, and in particular to a three-dimensional segmentation method and system based on sparse viewpoint three-dimensional reconstruction and dynamic alignment. Background Art

[0002] In recent years, with the rapid improvement of computer performance and the rapid development of big data, the processing and application of visual information has received widespread attention in various fields, especially in video surveillance, virtual reality, autonomous driving and other scenarios. As a cutting-edge research topic in the field of computer vision and deep learning, 3D interactive segmentation technology has broad application prospects. Different from traditional 2D image segmentation methods, 3D interactive segmentation can accurately locate and segment target objects in 3D space, thus providing richer and more efficient information in a variety of practical applications.

[0003] Three-dimensional interactive segmentation is an important research direction in the field of computer vision and machine learning. It integrates cutting-edge knowledge from multiple cutting-edge fields such as image processing, artificial intelligence, three-dimensional reconstruction, and augmented reality. Compared with traditional two-dimensional segmentation, three-dimensional interactive segmentation can provide support for more complex real-world scene analysis. With the rapid development of virtual reality, augmented reality, autonomous driving, medical imaging and other fields, three-dimensional interactive segmentation technology has been widely used in these scenarios. For example, in autonomous driving, three-dimensional segmentation can accurately identify and separate different traffic participants and obstacles in real scenes, improving the vehicle's environmental perception ability; in virtual reality, three-dimensional segmentation can help users interact with target objects in the virtual environment and enhance the immersive experience. Through simple interactive prompts provided by users, three-dimensional interactive segmentation can quickly and accurately complete target recognition and segmentation in complex three-dimensional scenes, and has broad application potential and development prospects.

[0004] Two-dimensional visual foundation models (VFMs) are usually trained on large-scale datasets, have a large number of parameters, and show high adaptability in multiple downstream tasks. SAM (Segment Anything Model) proposed by Kirillov et al. is a prompt-based two-dimensional segmentation foundation model. SAM generates segmentation masks based on prompts to identify target objects in images, and can generalize to unseen categories, providing technical support for subsequent research on three-dimensional interactive segmentation. As an improved version of SAM, SAM2 (Segment Anything Model 2) introduces architectural improvements to achieve interactive segmentation of videos. In recent years, interactive three-dimensional segmentation technology has made significant progress. Among them, interactive three-dimensional segmentation based on neural radiance fields (NeRF) and 3D Gaussian splatting (3DGS) has attracted widespread attention from researchers due to its potential applications in multiple fields such as scene manipulation, automatic annotation, and virtual reality. For example, Cen et al. started with user prompts on a single view, generated a 2D initial mask through SAM, and obtained the 3D segmentation mask of the target through iteration: the first step was to use NeRF to perform reverse rendering of the mask and project the 2D mask into 3D space; the second step was to optimize the mask through a self-prompt mechanism across views, use NeRF to render the 2D masks of other views, and generate new prompts to improve the segmentation results. However, this method relies on prompts from the initial view, and different instances with similar semantics between views are prone to training ambiguity. Liu et al. embedded semantic features from multiple views into the neural radiance field based on SAM. When rendering a new view, the segmentation mask can be obtained only through the decoder of SAM, which improves the segmentation quality and efficiency. However, all NeRF-based models require time-consuming pre-training of the neural radiance field to obtain prior knowledge of the 3D scene. As the latest progress in radiance fields, 3DGS proposed by Kerbl et al. uses trainable 3D Gaussians to explicitly represent 3D scenes, and proposes an effective differentiable rasterization algorithm for rendering and training, which solves the problem of high training requirements of NeRF and accelerates the reconstruction of the scene. Cen et al. combined the 2D segmentation base model with 3DGS and, through contrast training, were able to efficiently embed the multi-granular 2D segmentation results from the segmentation base model into 3D Gaussian point features. However, this approach lacks geometric constraints and makes full use of local information, making the task of accurately segmenting a single object through feature matching complicated. Xie et al. simplified the 3D Gaussian point cloud rendering process through linearization, converting the 2D to 3D segmentation task into a linear integer optimization problem, significantly improving the optimization speed.The idea of ​​the above method is to obtain a two-dimensional segmentation mask or feature based on a two-dimensional segmentation basic model (such as SAM), and map the feature to an implicitly represented neural radiance field (NeRF) or a three-dimensional Gaussian volume to obtain a segmentation mask in three-dimensional space. Among them, the three-dimensional scene representation based on NeRF and 3DGS requires pre-training of dense views for specific scene inputs. Although the trained scene can achieve interactive three-dimensional segmentation, the timeliness problem brought about by the pre-training of three-dimensional scene representation greatly limits its application in real-time scenarios. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide a three-dimensional segmentation method based on sparse perspective three-dimensional reconstruction and dynamic alignment, comprising the following steps:

[0007] Image preprocessing step, preprocessing multi-view images of specific scenes;

[0008] In the point cloud generation step, the preprocessed images are input into the neural network MASt3R with two view images as a group, and the point cloud data of each input image group in the same camera coordinate system is obtained;

[0009] In the global alignment training step, the point cloud corresponding to each group of input images is traversed, and the transformation matrix that maps each view to the world coordinate system is learned based on the dynamic global alignment strategy;

[0010] In the interactive segmentation step, based on user prompts and SAM2, two-dimensional segmentation masks of all views are obtained, and the two-dimensional segmentation masks are aligned to the world coordinate system through the transformation matrix.

[0011] Preferably, the neural network MASt3R is expressed as:

[0012] : ;

[0013] in, represents the neural network MASt3R, represents a set of input images that are fed into the neural network MASt3R each time, Respectively indicate the corresponding Point cloud data, confidence maps and dense feature maps; Respectively indicate the corresponding Point cloud data, confidence maps and dense feature maps; , indicating that and Combination of pictures, and All located in In the camera coordinate system of the viewing angle; W represents the image width of the input image group, and H represents the image height of the input image group.

[0014] Preferably, the global alignment training step comprises the following steps:

[0015] Receive the preprocessed multi-view image, perform panoramic segmentation on the first view based on SAM2, and extract the segmentation masks of all objects as prompt information;

[0016] Input the segmentation masks of all targets into SAM2 to obtain the segmentation masks corresponding to each target under all viewing angles. ;

[0017] The confidence map corresponding to each view And the corresponding soft segmentation mask Multiply them together to obtain the weighted confidence, and then map the weighted confidence to interval, get the confidence score , expressed as:

[0018] ;

[0019] Among them, v represents the view, , N represents the total number of views; i represents the pixel index, Represents the sigmoid function;

[0020] Initialize a transformation matrix for each view to achieve the projection transformation from the two-dimensional pixel coordinates in the corresponding view to the world coordinates;

[0021] For each set of input images, the dense feature map generated by the neural network MASt3R is used Compute a pair of matching graphs ;

[0022] The confidence score is dynamically adjusted using the matching graph to obtain the dynamically adjusted confidence score, which is expressed as:

[0023] ;

[0024] ;

[0025] in, represents the dynamically adjusted confidence score, is the adjustment factor, and Respectively represent the adjustment factors corresponding to the matched pixels and the unmatched pixels; Represents a matching graph The i-th pixel in;

[0026] Map the dynamically adjusted confidence score back to the original value range, expressed as:

[0027] ;

[0028] in, Represents the inverse function of sigmoid;

[0029] Calculate the point cloud corresponding to all input image groups and the corresponding world coordinate system of Loss, the final dynamic global alignment calculation method is:

[0030] ;

[0031] in, Indicates gradient descent; Represents the point in the world coordinate system obtained by calculating the transformation matrix of the point cloud data; represents the scale factor, represents the external parameter matrix, Represents the coordinates of the world coordinate system calculated by the optimized transformation matrix of the point cloud data; represents the set consisting of photo combinations e;

[0032] By traversing the point clouds corresponding to all input image groups, an optimized transformation matrix can be learned for each perspective.

[0033] Preferably, the transformation matrix includes the camera intrinsic matrix, extrinsic matrix and depth information. For a two-dimensional point coordinate from a single view , whose corresponding coordinates in the world coordinate system are It is expressed as:

[0034] ;

[0035] in, represents the depth map, represents the camera intrinsic parameter matrix, Represents the camera extrinsic matrix.

[0036] Preferably, the interactive segmentation step is expressed as:

[0037] ;

[0038] in, Represents a collection of all views. Indicates that the user prompt is in the view The segmentation mask under It represents the point cloud set of the target corresponding to the prompt in the world coordinate system, that is, the three-dimensional segmentation result.

[0039] The present invention further provides a 3D segmentation system based on sparse view 3D reconstruction and dynamic alignment, which is used to implement any of the above 3D segmentation methods based on sparse view 3D reconstruction and dynamic alignment, comprising:

[0040] Image preprocessing module, which preprocesses multi-view images of specific scenes;

[0041] The point cloud generation module inputs the preprocessed images into the neural network MASt3R with two view images as a group, and obtains the point cloud data of each input image group in the same camera coordinate system;

[0042] The global alignment training module traverses the point cloud corresponding to each group of input images and learns the transformation matrix that maps each view to the world coordinate system based on the dynamic global alignment strategy;

[0043] The interactive segmentation module obtains the 2D segmentation masks of all views based on user prompts and SAM2, and aligns the 2D segmentation masks to the world coordinate system through the transformation matrix.

[0044] The present invention has the following beneficial effects:

[0045] (1) The present invention proposes a feed-forward 3D segmentation model that operates based on 2D views. It does not require pre-training for specific scenes and can immediately segment any 3D objects in different environments.

[0046] (2) The present invention introduces a dynamic global alignment method, which effectively solves the three-dimensional alignment error and improves the segmentation accuracy by iteratively learning the transformation matrix;

[0047] (3) The present invention obtains segmentation masks under multiple perspectives based on the visual basic model SAM2, and realizes real-time interactive 3D segmentation by combining with the optimized transformation matrix;

[0048] (4) Extensive experiments on NVOS and SPIn-NeRF datasets show that compared with other competitive methods, the proposed WildSeg3D achieves significant efficiency improvement while ensuring performance.

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A method step diagram of an embodiment of the present invention;

[0051] Figure 2A schematic diagram of a flow chart of an embodiment of the present invention;

[0052] Figure 3 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0053] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0054] S101, a picture preprocessing step, preprocessing multi-view pictures of a specific scene;

[0055] S102, point cloud generation step, inputting the preprocessed images into a group of two view images into the neural network MASt3R to obtain point cloud data of each group of input images in the same camera coordinate system;

[0056] S103, a global alignment training step, traversing the point cloud corresponding to each group of input images, and learning the transformation matrix of each view mapped to the world coordinate system based on a dynamic global alignment strategy;

[0057] S104, an interactive segmentation step, obtains two-dimensional segmentation masks of all views based on user prompts and SAM2, and aligns the two-dimensional segmentation masks to the world coordinate system through a transformation matrix.

[0058] The neural network MASt3R is a feed-forward method for 3D scene reconstruction. Based on the point map prediction and global alignment of the neural network MASt3R, general 3D scene reconstruction can be achieved without scene-specific pre-training.

[0059] See also Figure 2 As shown, it is a flow chart of an embodiment of the present invention; first, a dynamic global alignment (DGA) module is designed to improve the confidence of difficult-to-match pixels by dynamically adjusting the function, and the transformation matrix of each view mapped from the pixel coordinate system to the world coordinate system can be learned through iterative optimization. Second, SAM2 is introduced to construct multi-view target segmentation masks. Third, in the interactive segmentation step, the target mask from multiple perspectives is retrieved based on the user prompt information, and the two-dimensional mask is transformed to the world coordinate system based on the transformation matrix to achieve real-time segmentation.

[0060] Specifically, in S101, multi-view images of a specific scene are preprocessed and then combined in pairs and sent to a neural network; in the preprocessing of the images, the maximum side length of the images is scaled to 512 according to the original ratio; for each group of views, the model infers the point cloud of the two images in the same camera coordinate system.

[0061] Specifically, the global alignment training step includes the following steps:

[0062] Step a1: for multiple views of a specific scene, the point cloud and confidence map corresponding to each pair of image combinations output by the neural network in step 1 are given;

[0063] Step a2: Perform panoramic segmentation on the first view based on SAM2, extract the segmentation mask of all targets as prompt information, and input SAM2 again to obtain the segmentation mask corresponding to each target under all views. ;

[0064] Step a3, for each view , multiply the corresponding confidence map with the corresponding soft segmentation mask to obtain the weighted confidence, and then pass it through the sigmoid function , mapping the weighted confidence to Interval, the specific operations are:

[0065] ;

[0066] in, Represents the confidence score after applying the adjustment of the soft segmentation mask.

[0067] Step a4, initialize a transformation matrix for each view. The matrix consists of camera intrinsic parameters, extrinsic parameters and depth information, which can realize the projection transformation from pixel coordinates to world coordinates. Given a 2D point coordinate from a single view , and the depth map , camera intrinsic parameter matrix and the external parameter matrix , the coordinates of this point in the world coordinate system It can be expressed as:

[0068] ;

[0069] For each view , the corresponding 3D point in the world coordinate system (in ) are derived using the above transformation matrix.

[0070] Step a5: For each set of images, use MAST3R to generate a pair of matching images. , which means Each sample point in Is there a corresponding matching point? Based on the weighted confidence score, the dynamic adjustment function is defined as follows:

[0071] ;

[0072] in, It can improve the attention to the hard-to-match pixels. Adjustment factor is defined as:

[0073] ;

[0074] Step a6, apply the inverse function of sigmoid Map the adjusted confidence scores back to the original range:

[0075] ;

[0076] Step a7: traverse the point clouds corresponding to all image combinations and calculate their coordinates with the corresponding world coordinate system. of Loss, the final dynamic global alignment calculation method is:

[0077] ;

[0078] After the above iterative training, each perspective can learn an optimized transformation matrix, which can align the corresponding view from the pixel coordinate system to the unified world coordinate system.

[0079] Specifically, in the interactive segmentation step, the user inputs prompt information, and the target segmentation mask under multiple views can be obtained based on SAM2. After the transformation matrix learned by dynamic global alignment, the target can be mapped to a unified three-dimensional space to achieve efficient interactive three-dimensional segmentation. The specific calculation process is:

[0080] ;

[0081] in, Represents a collection of all views. Indicates that the user prompt is in the view The segmentation mask under It represents the point cloud set of the target corresponding to the prompt in the world coordinate system, that is, the three-dimensional segmentation result.

[0082] The effects of the embodiments of the present invention are further illustrated by the following simulation experiments.

[0083] 1) Simulation conditions: The present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python.

[0084] 2) Simulation content: Take the NVOS and SPIn-NeRF datasets, reconstruct different scenes according to the above steps and test them using the reference view. Tables 1 and 2 are the accuracy and running time test results of the present invention and other methods on the two datasets, respectively. It can be found that compared with other methods, the present invention has the best efficiency while ensuring better accuracy. Among them, WildSeg3D (the method of the present invention) is the result of the present invention, and the accuracy evaluation indicators are the mean intersection over union ratio mIoU and the mean accuracy mAc. The mIoU / mAcc of the present invention on the NVOS dataset can reach 94.1% / 99.0%, and the mIoU / mAcc on the SPIn-NeRF dataset can reach 94.0% / 98.6%. Compared with other methods, the present invention greatly improves the efficiency of interactive segmentation while ensuring that the accuracy is at the forefront.

[0085] Table 1 - Comparison of mIoU and mAcc metrics with state-of-the-art methods on the NVOS dataset:

[0086]

[0087] Table 2 - Comparison of mIoU and mAcc metrics with state-of-the-art methods on the SPIn-NeRF dataset:

[0088]

[0089] Extensive experiments conducted on NVOS and SPIn-NeRF datasets show that, compared with other competitive methods, the method of the embodiment of the present invention achieves optimal efficiency while ensuring performance.

[0090] See also Figure 3 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0091] The image preprocessing module 301 preprocesses multi-view images of a specific scene;

[0092] The point cloud generation module 302 inputs the preprocessed images into the neural network MASt3R as a group of two view images, and obtains point cloud data of each group of input images in the same camera coordinate system;

[0093] The global alignment training module 303 traverses the point cloud corresponding to each group of input images, and learns the transformation matrix of each view mapped to the world coordinate system based on the dynamic global alignment strategy;

[0094] The interactive segmentation module 304 obtains the two-dimensional segmentation masks of all viewing angles based on the user prompt and SAM2, and aligns the two-dimensional segmentation masks to the world coordinate system through the transformation matrix.

[0095] It can be seen that the present invention can realize real-time interactive three-dimensional segmentation in any scenario. Compared with the prior art, it achieves significant efficiency improvement while ensuring performance.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D segmentation method based on sparse view 3D reconstruction and dynamic alignment, characterized in that: The following steps are involved: Image preprocessing step, preprocessing multi-view images of specific scenes; In the point cloud generation step, the preprocessed images are input into the neural network MASt3R with two view images as a group, and the point cloud data of each input image group in the same camera coordinate system is obtained; In the global alignment training step, the point cloud corresponding to each group of input images is traversed, and the transformation matrix that maps each view to the world coordinate system is learned based on the dynamic global alignment strategy; In the interactive segmentation step, based on user prompts and SAM2, two-dimensional segmentation masks of all views are obtained, and the two-dimensional segmentation masks are aligned to the world coordinate system through the transformation matrix.

2. The 3D segmentation method based on sparse view 3D reconstruction and dynamic alignment according to claim 1, characterized in that: The neural network MASt3R is represented as: ： ； in, represents the neural network MASt3R, represents a set of input images that are fed into the neural network MASt3R each time, Respectively indicate the corresponding Point cloud data, confidence maps and dense feature maps; Respectively indicate the corresponding Point cloud data, confidence maps and dense feature maps; , indicating that and Combination of pictures, and All located in In the camera coordinate system of the viewing angle; W represents the image width of the input image group, and H represents the image height of the input image group.

3. The 3D segmentation method based on sparse view 3D reconstruction and dynamic alignment according to claim 1, characterized in that: The global alignment training step comprises the following steps: Receive the preprocessed multi-view image, perform panoramic segmentation on the first view based on SAM2, and extract the segmentation masks of all objects as prompt information; Input the segmentation masks of all targets into SAM2 to obtain the segmentation masks corresponding to each target under all viewing angles. ; The confidence map corresponding to each view And the corresponding soft segmentation mask Multiply them together to obtain the weighted confidence, and then map the weighted confidence to interval, get the confidence score , expressed as: ； Among them, v represents the view, , N represents the total number of views; i represents the pixel index, Represents the sigmoid function; Initialize a transformation matrix for each view to achieve the projection transformation from the two-dimensional pixel coordinates in the corresponding view to the world coordinates; For each set of input images, the dense feature map generated by the neural network MASt3R is used Compute a pair of matching graphs ; The confidence score is dynamically adjusted using the matching graph to obtain the dynamically adjusted confidence score, which is expressed as: ； ； in, represents the dynamically adjusted confidence score, is the adjustment factor, and Respectively represent the adjustment factors corresponding to the matched pixels and the unmatched pixels; Represents a matching graph The i-th pixel in ; Map the dynamically adjusted confidence score back to the original value range, expressed as: ； in, Represents the inverse function of sigmoid; Calculate the point cloud corresponding to all input image groups and the corresponding world coordinate system of Loss, the final dynamic global alignment calculation method is: ； in, Indicates gradient descent; Represents the point in the world coordinate system obtained by calculating the transformation matrix of the point cloud data; represents the scale factor, represents the external parameter matrix, Represents the coordinates of the world coordinate system calculated by the optimized transformation matrix of the point cloud data; represents the set consisting of photo combination e; By traversing the point clouds corresponding to all input image groups, an optimized transformation matrix can be learned for each perspective.

4. The 3D segmentation method based on sparse view 3D reconstruction and dynamic alignment according to claim 3, characterized in that: The transformation matrix includes the camera intrinsic matrix, extrinsic matrix and depth information. For a two-dimensional point coordinate from a single view , whose corresponding coordinates in the world coordinate system are It is expressed as: ； in, represents the depth map, represents the camera intrinsic parameter matrix, Represents the camera extrinsic matrix.

5. The 3D segmentation method based on sparse view 3D reconstruction and dynamic alignment according to claim 4, characterized in that: The interactive segmentation step is expressed as: ； in, Represents a collection of all views. Indicates that the user prompt is in the view The segmentation mask under It represents the point cloud set of the target corresponding to the prompt in the world coordinate system, that is, the three-dimensional segmentation result.

6. A 3D segmentation system based on sparse view 3D reconstruction and dynamic alignment, characterized in that: The method for implementing the three-dimensional segmentation method based on sparse view three-dimensional reconstruction and dynamic alignment according to any one of claims 1 to 5 comprises: Image preprocessing module, which preprocesses multi-view images of specific scenes; The point cloud generation module inputs the preprocessed images into the neural network MASt3R with two view images as a group, and obtains the point cloud data of each input image group in the same camera coordinate system; The global alignment training module traverses the point cloud corresponding to each group of input images and learns the transformation matrix that maps each view to the world coordinate system based on the dynamic global alignment strategy; The interactive segmentation module obtains the 2D segmentation masks of all views based on user prompts and SAM2, and aligns the 2D segmentation masks to the world coordinate system through the transformation matrix.

Images