SAM-based multi-view 3D segmentation method and system
By selecting the enclosure frame, calculating camera parameters and performing sparse point cloud reconstruction in multi-view 3D scenes, the problems of instability and consistency of object segmentation in the existing technology are solved, and automated and stable 3D object segmentation is realized, suitable for products Showcases and museum exhibitions.
Patent Information
- Application Number
- CN202411134853.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-19
AI Technical Summary
The prior art is difficult to achieve automated, stable and consistent object segmentation in multi-view 3D scenes, especially in scenes such as commodity displays and museum exhibitions. The traditional 2D segmentation scheme does not have the effect of segmentation of 3D objects and requires manual input. Prospect location information affects automation and availability.
By selecting the enclosure box in the camera, taking object photos, calculating camera parameters to reconstruct sparse point clouds, extracting the position information of the enclosure box, eliminating the scene point cloud, reprojecting it to a specific camera perspective, selecting the object box, and inputting it into the segmentation algorithm to obtain consistent segmentation results.
It realizes automated, stable and clear 3D segmentation of objects, ensuring that object segmentation at different perspectives keeps clear boundaries, has high 3D consistency, and improves the segmentation effect and automation level.
Smart Images

Figure CN118982545A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a multi-view 3D segmentation method and system based on SAM. Background Art
[0002] In scenes such as product display, museum 3D exhibition, and 3D object reconstruction, it is usually necessary to remove the background to achieve a better display effect. Traditional scene segmentation and reconstruction include the following solutions: 1. The 2D-based segmentation solution SAM (segement anything model, an open source 2D segmentation algorithm) has a refined segmentation effect and is quickly sought after by everyone. This solution can automatically segment the input image into several parts, or input the foreground point or foreground frame to segment the foreground part. However, as mentioned above, this solution is only for 2D scenes and requires manual input of foreground position information. It needs to be optimized in terms of usability and automation.
[0003] 2. Segment-anything-rgbd: This is a derivative of the SAM solution. When only image information is input, SAM segments objects too finely, usually dividing a whole into multiple parts. This solution wants to segment the foreground part as a whole based on the depth information. This solution first obtains the depth information of the 2D image based on the zoe-depth model, and then segments the rgb image and the depth image based on SAM, combining the segmentation effects of the two or a relatively stable foreground segmentation effect. However, this solution is mainly trained for the human body, while the main body of our scene is an object. After some tests, it was found that the effect is not Lupin in the multi-view 3D scene.
[0004] 3. SegmentAnything in 3D with NeRFs: This solution is dedicated to solving the problem of automated segmentation. For a series of pictures from different perspectives, you only need to click on some points on the object in a certain perspective. After 3D reconstruction through NeRF (Chinese name: neural rendering, a type of 3D reconstruction method using neural networks), these points can be migrated to other perspectives and foreground points can be removed based on confidence. This solution roughly meets our needs, but the effect is not stable. It only uses some uncertain points as SAM input for segmentation, and floaters are likely to appear at the boundaries. This is because SAM works better with box input than point input. Therefore, how to provide a multi-view 3D segmentation method and system based on SAM is a problem that needs to be solved urgently. Summary of the invention
[0005] The embodiments of the present invention provide a multi-view 3D segmentation method and system based on SAM to solve the above-mentioned technical problems existing in the prior art.
[0006] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0007] According to a first aspect of an embodiment of the present invention, a multi-view 3D segmentation method based on SAM is provided.
[0008] In one embodiment, a multi-view 3D segmentation method based on SAM includes: Select a bounding box in the camera and take a photo of the object at a specific point; Based on the object photos, calculate the camera parameters and reconstruct the sparse point cloud of the object photos; Extract the location information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; Reproject the object point cloud to a specific camera perspective, and select the object frame by coordinate conversion; Input the object box into the segmentation algorithm to obtain consistent segmentation results.
[0009] In one embodiment, selecting a bounding box in a camera and taking a photo of an object at a specific point includes: Get the current camera position and adjust a bounding box in the camera according to the location of the object; Set multiple specific points around the object, and take a photo of the object when the camera position coincides with the specific points.
[0010] In one embodiment, the step of calculating camera parameters based on the object photo and reconstructing a sparse point cloud of the object photo includes: Use 3D reconstruction software to extract image features of the object image and then perform feature matching; The sparse point cloud reconstruction and optimization of the object photos are performed using 3D reconstruction software to obtain the scene point cloud.
[0011] In one embodiment, extracting the position information of the bounding box, removing the scene point cloud outside the bounding box, and obtaining the object point cloud includes: Based on the camera parameters, calculate the pixel positions of the cuboid formed by the bounding box; The least square method is used to calculate the position information of the cuboid in the 3D reconstruction software; The scene point cloud outside the cuboid is eliminated, and the point cloud inside the cuboid is retained as the object point cloud.
[0012] In one embodiment, the calculation formula for calculating the pixel position of the cuboid formed by the bounding box includes:
[0013]
[0014]
[0015] In the formula, Indicates the pixel positions of the eight vertices of the cuboid; KA indicates the camera intrinsic parameter; KC indicates the camera extrinsic parameter, Represents the 3D position of the cuboid in the coordinate system of the augmented reality application.
[0016] In one embodiment, the calculation formula for calculating the position information of the cuboid in the three-dimensional reconstruction software is:
[0017] Where KC represents the camera internal parameter; Represents the camera extrinsic parameters corresponding to the i-th object photo.
[0018] In one embodiment, reprojecting the object point cloud to a specific camera perspective and selecting the object frame by coordinate conversion includes: Reproject the object point cloud to a specific camera perspective and calculate the projection coordinates of the object point cloud on each object photo; Based on the positional relationship of the projection coordinates, the maximum and minimum values of the projection coordinates are calculated, and the object in the object image is reframed and defined as an object frame.
[0019] In one embodiment, the calculation formula for calculating the projection coordinates of the object point cloud on each object photo is:
[0020]
[0021]
[0022] Where KC represents the camera internal parameter; represents the camera extrinsic parameter corresponding to the i-th object photo; points represents the object point cloud; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
[0023] In one embodiment, the calculation formula for calculating the maximum and minimum values of the projection coordinates is:
[0024]
[0025] In the formula, Indicates the minimum value of the projection coordinate on the x-axis; Indicates the maximum value of the projection coordinate on the x-axis; Indicates the minimum value of the projection coordinate on the y-axis; Indicates the maximum value of the projection coordinate on the y-axis; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
[0026] In one embodiment, the expression of the object frame is: .
[0027] According to a second aspect of the embodiments of the present invention, a multi-view 3D segmentation system based on SAM is provided.
[0028] In one embodiment, the SAM-based multi-view 3D segmentation system comprises: The camera shooting module is used to select a bounding box in the camera and take a picture of the object at a specific point; Point cloud reconstruction module, used to calculate camera parameters based on object photos and perform sparse point cloud reconstruction on object photos; The point cloud extraction module is used to extract the position information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; The coordinate conversion module is used to reproject the object point cloud to a specific camera perspective and select the object frame through coordinate conversion; The image segmentation module is used to input the object box into the segmentation algorithm to obtain consistent segmentation results.
[0029] According to a third aspect of an embodiment of the present invention, a computer device is provided.
[0030] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0031] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.
[0032] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.
[0033] The technical solution provided by the embodiment of the present invention may include the following beneficial effects: by obtaining the bounding box framed by the user in space and mapping it to a multi-perspective 2D picture, while ensuring its compactness and accuracy; using such a 2D box as a prompt word for SAM, a relatively consistent segmentation effect can be obtained in the multi-perspective segmentation; objects can be automatically, stably and clearly segmented in 3D, which has practical value in scenarios such as product display / museum exhibition, etc.
[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0036] Figure 1 is a flow chart of a multi-view 3D segmentation method based on SAM according to an exemplary embodiment; Figure 2 is a system block diagram of a multi-view 3D segmentation system based on SAM according to an exemplary embodiment; Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION
[0037] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.
[0038] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like used herein to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this article and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description herein, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal connection of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0039] As used herein, the term "plurality" means two or more than two, unless otherwise specified.
[0040] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.
[0041] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.
[0042] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0043] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.
[0044] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0045] Figure 1An embodiment of the SAM-based multi-view 3D segmentation method of the present invention is shown.
[0046] In this optional embodiment, the SAM-based multi-view 3D segmentation method includes: Step S101, select a bounding box in the camera, and take a photo of the object at a specific point; Step S103: Calculate camera parameters based on the object photo, and perform sparse point cloud reconstruction on the object photo; Step S105: extract the position information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; Step S107, reprojecting the object point cloud to a specific camera perspective, and selecting the object frame by coordinate conversion; Step S109: input the object frame into the segmentation algorithm to obtain a consistent segmentation result.
[0047] In this optional embodiment, when selecting a bounding box in the camera and taking a photo of an object at a specific point, the current camera position can be obtained, and a bounding box can be adjusted in the camera according to the position of the object; multiple specific points are set around the object, and when the camera position coincides with the specific points, a photo of the object is taken.
[0048] In this optional embodiment, when calculating the camera parameters based on the object photo and reconstructing the sparse point cloud of the object photo, the image features of the object photo can be extracted using the three-dimensional reconstruction software, and then feature matching can be performed; the sparse point cloud reconstruction and optimization of the object photo can be performed using the three-dimensional reconstruction software to obtain the scene point cloud.
[0049] In this optional embodiment, when extracting the position information of the bounding box and eliminating the scene point cloud outside the bounding box to obtain the object point cloud, the pixel position of the cuboid formed by the bounding box can be calculated based on the camera parameters; the position information of the cuboid in the three-dimensional reconstruction software is calculated using the least squares method; the scene point cloud outside the cuboid is eliminated, and the point cloud inside the cuboid is retained as the object point cloud.
[0050] In this optional embodiment, the calculation formula for calculating the pixel position of the cuboid formed by the bounding box includes:
[0051]
[0052]
[0053] In the formula, Indicates the pixel positions of the eight vertices of the cuboid; KA indicates the camera intrinsic parameter; KC indicates the camera extrinsic parameter, Represents the 3D position of the cuboid in the coordinate system of the augmented reality application.
[0054] In this optional embodiment, the calculation formula for calculating the position information of the cuboid in the three-dimensional reconstruction software is:
[0055] Where KC represents the camera internal parameter; Represents the camera extrinsic parameters corresponding to the i-th object photo.
[0056] In this optional embodiment, when reprojecting the object point cloud to a specific camera perspective and selecting the object frame through coordinate conversion, the object point cloud can be reprojected to the specific camera perspective, and the projection coordinates of the object point cloud on each object photo are calculated; based on the positional relationship of the projection coordinates, the maximum and minimum values of the projection coordinates are calculated, and the object in the object picture is framed again to be defined as an object frame.
[0057] In this optional embodiment, the calculation formula for calculating the projection coordinates of the object point cloud on each object photo is:
[0058]
[0059]
[0060] Where KC represents the camera internal parameter; represents the camera extrinsic parameter corresponding to the i-th object photo; points represents the object point cloud; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
[0061] In this optional embodiment, the calculation formula for calculating the maximum and minimum values of the projection coordinates is:
[0062]
[0063] In the formula, Indicates the minimum value of the projection coordinate on the x-axis; Indicates the maximum value of the projection coordinate on the x-axis; Indicates the minimum value of the projection coordinate on the y-axis; Indicates the maximum value of the projection coordinate on the y-axis; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
[0064] In this optional embodiment, the expression of the object frame is: .
[0065] Figure 2 An embodiment of the SAM-based multi-view 3D segmentation system of the present invention is shown.
[0066] In this optional embodiment, the SAM-based multi-view 3D segmentation system includes: The camera shooting module 201 is used to select a bounding box in the camera and take a picture of the object at a specific point; The point cloud reconstruction module 203 is used to calculate the camera parameters based on the object photo and perform sparse point cloud reconstruction on the object photo; Point cloud extraction module 205, used to extract the position information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; A coordinate conversion module 207 is used to reproject the object point cloud to a specific camera view and select the object frame by coordinate conversion; The image segmentation module 209 is used to input the object frame into the segmentation algorithm to obtain a consistent segmentation result.
[0067] In specific applications, in scenarios such as product display, museum 3D exhibitions, and 3D object reconstruction, it is usually necessary to remove the background to achieve a better display effect.
[0068] The present invention proposes a 3D segmentation algorithm based on SAM for the scene where the same object is photographed at multiple different viewing angles, so that the object segmentation at different viewing angles maintains clear boundaries and has a certain 3D consistency.
[0069] The advantages of the present invention include the following aspects: 1. Automation: Many segmentation algorithms require some manual intervention, such as selecting a foreground point / foreground frame for each picture and then performing segmentation based on the input. These methods are not conducive to the automatic generation of large-scale data. However, the present invention only needs to input a series of pictures taken at different viewing angles to perform automatic segmentation, greatly improving the segmentation effect.
[0070] 2. More stable: Some automatic segmentation algorithms perform automatic segmentation in the following ways: 1. Input the image into the target detection model to get the name of the foreground object. 2. Get the target detection box from the object name. 3. Input the target detection box into the SAM model to get the segmentation solution. Although this solution is highly automated, our tests found that it is very unstable in our scenario, mainly because the target detection box is not accurate and compact enough, which greatly reduces the segmentation effect of SAM. However, our solution has been tested in multiple scenarios and has been stable.
[0071] 3. 3D consistency: SAM is a 2D segmentation solution that processes each image separately. This will inevitably lead to inaccurate segmentation in certain viewing angles, such as segmenting part of the background into the foreground, while the background is segmented correctly in other viewing angles. Our solution uses the 3D information of the object to correct this incorrect segmentation and ensure the consistency of the segmentation of the series of images at all viewing angles.
[0072] To achieve the above functional advantages, the overall process of the present invention can be summarized as follows.
[0073] 1. The present invention requires a specially developed app to be used. The app's functions are: using augmented reality software (AR software, such as ARKit) to obtain the current camera position, the user adjusts a bounding box according to the location of the object so that the bounding box can just surround the object. After determining the bounding box, select specific points around the object for shooting, and when the camera position coincides with the selected position, take a photo of the object.
[0074] 2. After obtaining the photos taken at a specific location, use 3D reconstruction software to recalculate the camera parameters and reconstruct the sparse point cloud of the object. For example, Colmap is an open source 3D reconstruction software that can obtain the internal and external parameters of the camera corresponding to each view and the reconstructed point cloud of the scene through the input of multi-view images.
[0075] The point cloud reconstruction results are as follows. The calculation steps of the 3D reconstruction software include: a) Use the feature_extractor command of the 3D reconstruction software to extract image features; b) Use the mapper (feature matcher) command of the 3D reconstruction software to perform feature matching; c) Use the bundle_adjuster (reconstruction optimizer) command of the 3D reconstruction software to optimize the reconstruction; Finally, the 3D reconstruction software can simultaneously output the optimized camera parameters and scene point cloud reconstruction information.
[0076] 3. The above point cloud is the scene point cloud reconstructed by the 3D reconstruction software, and the position of the object is framed by the user in the acquisition APP. According to the bounding box, the point cloud outside the box can be eliminated to obtain the object point cloud.
[0077] The specific operations are: a) Calculate the specific pixel position of the bounding box cuboid based on the camera internal and external parameters returned by the augmented reality software. The calculation process is as follows:
[0078]
[0079]
[0080] Among them, KA and KC are the camera internal and external parameters obtained by augmented reality software, is the 3D position of the box (cuboid) in the coordinate system of the augmented reality software. The box consists of 8 vertices. is the projection of the calculated point cloud on the image, so It also contains the pixel positions of 8 points, each point contains x-axis and y-axis coordinates, and is an 8*2 matrix.
[0081] For the same object, the present invention takes n pictures with different perspectives. Therefore, there are n groups of KC, which share a camera intrinsic parameter KA. The 3D space where the cuboid is located is consistent, so they share a group The camera extrinsic parameter corresponding to the i-th image is recorded as , correspondingly, in the i-th image, the projection pixel position of the cuboid is recorded as .
[0082] b) The position of the cuboid in the 3D reconstruction software is obtained by the least squares method:
[0083] Among them, KC is the camera intrinsic parameter calculated by the 3D reconstruction software. Multiple shooting angles share one camera intrinsic parameter. Each shooting angle has different camera extrinsic parameters. The camera extrinsic parameter corresponding to the i-th image is recorded as , KC and MC are both 4*4 matrices, so their product The i-th row and j-th column of ,like Represents the value of the 1st row and 2nd column in matrix A.
[0084] For the jth vertex of the cuboid, there are n sets of projections in n different shooting angles, satisfying the following equations:
[0085] Among them, KA and KC are the camera internal and external parameters of the 3D reconstruction software, , is the pixel position of the projection of the j-th vertex of the cuboid on the i-th shooting angle, is the 3D position corresponding to the fixed point in the 3D reconstruction software coordinate system.
[0086] c) Eliminate the point cloud outside the box, determine whether any point is inside the cuboid, and keep the point cloud of the object inside the cuboid as points.
[0087] In addition, the object point cloud can also be obtained using other reconstruction algorithms. This example solution is obtained by 3D reconstruction software. For example, after using Nerf (Chinese name neural rendering, a type of 3D reconstruction method using neural networks) to perform 3D reconstruction within the framed range, the Marching cubes algorithm (an algorithm for generating surface meshes from 3D data) is used to obtain the point cloud. Or the 3DGS algorithm (Chinese name 3D Gaussian splashing, a 3D reconstruction method) is used to obtain the reconstructed point cloud, and then the frame is selected according to the range. Or the reconstructed point cloud is directly calculated in real time in the augmented reality software and selected.
[0088] 4. Reproject the object point cloud (or patch, the current example gives a point cloud) to a specific camera perspective, and frame the object in the image based on the maximum and minimum values of the projected pixel positions. The frame can be used as a prompt for SAM segmentation. The specific steps are as follows: a) Calculate the projection of the point cloud on the i-th image:
[0089]
[0090]
[0091] In the formula, KC and The meaning of the representation is explained in step 3.b. In step 3.c, the point cloud points retained in the box are calculated. As calculated above, they represent the x-axis coordinates and y-axis coordinates of the projection of points on the i-th image.
[0092] b) Calculate the maximum and minimum values of the projection coordinates:
[0093]
[0094] in , calculated by 4.a, represents the x-axis and y-axis coordinates of points projected on the i-th image.
[0095] The object box can be represented as .
[0096] 5. Input the object box into SAM to obtain consistent segmentation results.
[0097] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.
[0098] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0099] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.
[0100] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.
[0101] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0102] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A multi-view 3D segmentation method based on SAM, characterized in that: include: Select a bounding box in the camera and take a photo of the object at a specific point; Based on the object photos, calculate the camera parameters and reconstruct the sparse point cloud of the object photos; Extract the location information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; Reproject the object point cloud to a specific camera perspective, and select the object frame by coordinate conversion; Input the object box into the segmentation algorithm to obtain consistent segmentation results.
2. The multi-view 3D segmentation method based on SAM according to claim 1, characterized in that: The step of selecting a bounding box in the camera and taking a photo of the object at a specific point includes: Get the current camera position and adjust a bounding box in the camera according to the location of the object; Set multiple specific points around the object, and take a photo of the object when the camera position coincides with the specific points.
3. The multi-view 3D segmentation method based on SAM according to claim 1, characterized in that: The method of calculating camera parameters based on the object photo and reconstructing a sparse point cloud of the object photo includes: Use 3D reconstruction software to extract image features of the object image and then perform feature matching; The sparse point cloud reconstruction and optimization of the object photos are performed using 3D reconstruction software to obtain the scene point cloud.
4. The multi-view 3D segmentation method based on SAM according to claim 1, characterized in that: The step of extracting the position information of the bounding box and removing the scene point cloud outside the bounding box to obtain the object point cloud comprises: Based on the camera parameters, calculate the pixel positions of the cuboid formed by the bounding box; The least square method is used to calculate the position information of the cuboid in the 3D reconstruction software; The scene point cloud outside the cuboid is eliminated, and the point cloud inside the cuboid is retained as the object point cloud.
5. The multi-view 3D segmentation method based on SAM according to claim 4, characterized in that: The calculation formula for calculating the pixel position of the cuboid formed by the bounding box includes: In the formula, Indicates the pixel positions of the eight vertices of the cuboid; KA indicates the camera intrinsic parameter; KC indicates the camera extrinsic parameter, Represents the 3D position of the cuboid in the coordinate system of the augmented reality application.
6. The SAM-based multi-view 3D segmentation method according to claim 4, characterized in that: The calculation formula for calculating the position information of the cuboid in the 3D reconstruction software is: Where KC represents the camera internal parameter; Represents the camera extrinsic parameters corresponding to the i-th object photo.
7. The multi-view 3D segmentation method based on SAM according to claim 1, characterized in that: The object point cloud is reprojected to a specific camera perspective, and the object frame is selected by coordinate conversion. Reproject the object point cloud to a specific camera perspective and calculate the projection coordinates of the object point cloud on each object photo; Based on the positional relationship of the projection coordinates, the maximum and minimum values of the projection coordinates are calculated, and the object in the object image is reframed and defined as an object frame.
8. The multi-view 3D segmentation method based on SAM according to claim 7, characterized in that: The calculation formula for calculating the projection coordinates of the object point cloud on each object photo is: Where KC represents the camera internal parameter; represents the camera extrinsic parameter corresponding to the i-th object photo; points represents the object point cloud; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
9. The SAM-based multi-view 3D segmentation method according to claim 8, characterized in that: The calculation formula for calculating the maximum and minimum values of the projection coordinates is: In the formula, Indicates the minimum value of the projection coordinate on the x-axis; Indicates the maximum value of the projection coordinate on the x-axis; Indicates the minimum value of the projection coordinate on the y-axis; Indicates the maximum value of the projection coordinate on the y-axis; Represents the x-axis coordinate of the object point cloud projected on the i-th image; Represents the y-axis coordinate of the object point cloud projected on the i-th image.
10. A SAM-based multi-view 3D segmentation system, used to implement the SAM-based multi-view 3D segmentation method according to any one of claims 1 to 9, characterized in that: The system includes: The camera shooting module is used to select a bounding box in the camera and take a picture of the object at a specific point; Point cloud reconstruction module, used to calculate camera parameters based on object photos and perform sparse point cloud reconstruction on object photos; The point cloud extraction module is used to extract the position information of the bounding box, remove the scene point cloud outside the bounding box, and obtain the object point cloud; The coordinate conversion module is used to reproject the object point cloud to a specific camera perspective and select the object frame through coordinate conversion; The image segmentation module is used to input the object box into the segmentation algorithm to obtain consistent segmentation results.
Citation Information
Patent Citations
Method for segmenting target point cloud in combination with 2D image
CN117422730A
Digital twin modeling method and system based on SAM large model and NeRF
CN117671138A
Method for 3D scanning of a real object
US20220028100A1
