Target object segmentation method based on visual large model
Through the target object segmentation method based on the visual big model, the combination of three-dimensional point clouds and two-dimensional images is used to solve the generality and accuracy problems in industrial vision applications, and achieve more efficient target object recognition and segmentation effects.
Patent Information
- Application Number
- CN202410956779.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
The existing deep learning methods are not universal in industrial vision applications, making it difficult to accurately identify different data, and the training and application time is high, especially the differences in the target object segmentation labeling and detection methods, making it difficult to achieve the accuracy of the crawling requirements. sex.
Using the target object segmentation method based on the visual big model, by obtaining three-dimensional point clouds and two-dimensional images, chunking processing, principal component analysis and bounding box projection, combined with the bounding box and point prompt input of the visual big model for target object segmentation, improve the Image segmentation accuracy.
It improves the recognition accuracy and versatility of visual big models in industrial vision applications, reduces the consumption of training and application time, enhances the accuracy of target object segmentation, and is suitable for a variety of industrial scenarios.
Smart Images

Figure CN118918124A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of visual technology, and in particular to a method, device, electronic device, storage medium and program product for target object segmentation based on a large visual model. Background Art
[0002] Deep learning in the field of vision is developing rapidly. Visual recognition, including object detection, semantic segmentation, and instance segmentation, has certain applications in various fields. However, the main application is still in supervised learning, which relies on a large amount of annotation and training to obtain a good model effect. At the same time, in applications, different data needs to be collected for annotation training. It is difficult to use the same trained model to segment and recognize different data, and the versatility is not strong.
[0003] The emergence of large models has greatly improved the versatility of deep learning. Large models generally refer to models with more parameters obtained by training deeper networks based on deep learning algorithms. They are mainly used in natural language processing. With the development of technology, large models in a broad sense include large visual models and multimodal large models. The most well-known large visual model is the Segment Anything Model (SAM) released by Meta Platform Inc., hereinafter referred to as the SAM model. With sufficient data and parameter training, it covers a wide range of use cases, allowing the model to recognize and segment different objects in different scenarios with high accuracy.
[0004] Unlike general deep learning models, the SAM model does not need to be trained by labeling the current data. It can directly input the image to get the current scene, segment everything, and distinguish all targets. It can also input pixel positions or pixel boxes as prompts to allow the model to segment the corresponding targets more accurately based on the prompts. That is, the SAM model can use prompt information to segment different data. The development of large language models can also combine language models with images. If this can be applied to 3D vision, it can greatly improve the actual segmentation effect. However, the versatility of the model still has certain limitations in industrial vision, and more research and efforts are needed to achieve practical application effects.
[0005] In short, the existing mature and applicable deep learning methods rely on large data sets and are difficult to be universal. They need to be trained on different data. In addition, the detection methods and segmentation methods are quite different in labeling, training, and application. Segmentation labeling is more difficult than detection, and training and application are time-consuming. General large models (such as SAM models) cannot achieve very accurate recognition in some industrial vision fields and cannot meet the requirements of grasping. Summary of the invention
[0006] The present disclosure provides a target object segmentation method based on a visual large model, comprising:
[0007] Obtain the three-dimensional point cloud of the target scene in the camera coordinate system and the two-dimensional image in the image coordinate system;
[0008] Performing block processing on the three-dimensional point cloud to obtain at least one point cloud block;
[0009] Performing principal component analysis on the point cloud block to obtain the main direction of the point cloud block, and then obtaining a directed bounding box of the point cloud block;
[0010] Based on the transformation relationship between the camera coordinate system and the image coordinate system, the directed bounding box is projected onto the plane where the two-dimensional image is located, and a two-dimensional bounding box of the target object is obtained as a bounding box prompt;
[0011] Determining whether the number of target objects in the directed bounding box is one;
[0012] If the number of target objects in the directed bounding box is one, the following process is performed:
[0013] The bounding box prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the bounding box prompt to obtain a segmentation result.
[0014] According to the target object segmentation method of at least one embodiment of the present disclosure, if the number of target objects in the directed bounding box is more than one, the following process is performed:
[0015] Extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set;
[0016] Based on the transformation relationship between the camera coordinate system and the image coordinate system, the points in the extraction point set are projected onto the plane where the two-dimensional image is located, and a projection point set is obtained as a point prompt;
[0017] The bounding box prompt and the point prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the bounding box prompt and the point prompt to obtain a segmentation result.
[0018] According to at least one embodiment of the present disclosure, a target object segmentation method, wherein a plurality of point cloud distribution representative points or a plurality of random sampling points are extracted from the point cloud block to obtain an extracted point set, comprises:
[0019] voxelize the point cloud block to obtain a voxelized point cloud block;
[0020] A plurality of point cloud distribution representative points or a plurality of random sampling points are extracted from the voxelized point cloud block to obtain the extracted point set.
[0021] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0022] Extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set;
[0023] Based on the transformation relationship between the camera coordinate system and the image coordinate system, the points in the extraction point set are projected onto the plane where the two-dimensional image is located, and a projection point set is obtained as a point prompt;
[0024] The point prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the point prompt to obtain a segmentation result.
[0025] According to at least one embodiment of the present disclosure, the target object segmentation method further includes:
[0026] Projecting the points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a set of projection points;
[0027] generating an initial two-dimensional mask image based on the projection point set;
[0028] Performing dilation and / or corrosion processing on the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint;
[0029] If the number of target objects in the directed bounding box is one, the following process is performed:
[0030] The mask prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt and the bounding box prompt to obtain a segmentation result.
[0031] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0032] Projecting the points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a set of projection points;
[0033] generating an initial two-dimensional mask image based on the projection point set;
[0034] Performing dilation and / or corrosion processing on the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint;
[0035] The mask prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the prompt to obtain a segmentation result.
[0036] According to at least one embodiment of the present disclosure, the target object segmentation method further includes:
[0037] Detect the target object in the two-dimensional image using a deep learning-based detection algorithm to obtain at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame as a detection frame prompt;
[0038] If the number of target objects in the directed bounding box is one, the following process is performed:
[0039] The detection box prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the detection box prompt and the bounding box prompt to obtain a segmentation result.
[0040] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0041] Detect the target object in the two-dimensional image using a deep learning-based detection algorithm to obtain at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame as a detection frame prompt;
[0042] The detection frame prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the detection frame prompt to obtain a segmentation result.
[0043] According to at least one embodiment of the present disclosure, the target object segmentation method further includes:
[0044] Using a pre-acquired image template including feature information of the target object, performing feature matching based on the feature information of the target object on the two-dimensional image to obtain matching feature points of the two-dimensional image;
[0045] Generating a two-dimensional mask image for the two-dimensional image based on the matching feature points of the two-dimensional image as a mask prompt;
[0046] Extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame prompt;
[0047] If the number of target objects in the directed bounding box is one, the following process is performed:
[0048] The mask prompt, the detection box prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt, the detection box prompt and the bounding box prompt to obtain a segmentation result.
[0049] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0050] Using a pre-acquired image template including feature information of the target object, performing feature matching based on the feature information of the target object on the two-dimensional image to obtain matching feature points of the two-dimensional image;
[0051] Generating a two-dimensional mask image for the two-dimensional image based on the matching feature points of the two-dimensional image as a mask prompt;
[0052] Extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame prompt;
[0053] The mask prompt and the detection box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt and the detection box prompt to obtain a segmentation result.
[0054] According to the target object segmentation method of at least one embodiment of the present disclosure, the target object feature information includes position information of the target object and pixel gradient information of the target object.
[0055] According to at least one embodiment of the present disclosure, the target object segmentation method further includes:
[0056] Performing feature enhancement processing on the two-dimensional image to obtain a feature-enhanced two-dimensional image;
[0057] Performing foreground extraction on the feature enhanced image to obtain a foreground image;
[0058] Performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt;
[0059] If the number of target objects in the directed bounding box is one, the following process is performed:
[0060] The target object mask prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt and the bounding box prompt to obtain a segmentation result.
[0061] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0062] Performing feature enhancement processing on the two-dimensional image to obtain a feature-enhanced two-dimensional image;
[0063] Performing foreground extraction on the feature enhanced image to obtain a foreground image;
[0064] Performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt;
[0065] The target object mask prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt to obtain a segmentation result.
[0066] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0067] Inputting a language prompt to the multimodal large model based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt;
[0068] The target object detection frame prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection frame prompt to obtain a segmentation result.
[0069] According to at least one embodiment of the present disclosure, the target object segmentation method further includes:
[0070] Inputting a language prompt to the multimodal large model based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt;
[0071] If the number of target objects in the directed bounding box is one, the following process is performed:
[0072] The target object detection box prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection box prompt and the bounding box prompt to obtain a segmentation result.
[0073] According to the target object segmentation method of at least one embodiment of the present disclosure, the language prompt is obtained by the following method:
[0074] Marking the target object in one or more of the two-dimensional images to obtain a marked two-dimensional image;
[0075] The labeled two-dimensional image is input into the multimodal large model, and the multimodal large model outputs language describing the target object based on the labeled two-dimensional image as the language prompt.
[0076] According to the target object segmentation method of at least one embodiment of the present disclosure, alternatively, the target object segmentation method includes:
[0077] Marking a carrier body carrying the target object in the two-dimensional image to obtain a carrier body marking frame;
[0078] Performing regional segmentation on the two-dimensional image based on the carrier mark frame to obtain a two-dimensional image after regional segmentation, wherein the two-dimensional image after regional segmentation includes one or more mark frame regions;
[0079] The two-dimensional image after the region segmentation is input into the visual large model, so that the visual large model performs target object segmentation based on the two-dimensional image after the region segmentation to obtain a segmentation result.
[0080] According to at least one embodiment of the target object segmentation method of the present disclosure, the two-dimensional image after the region segmentation is input into the visual large model, and the visual large model performs target object segmentation based on the two-dimensional image after the region segmentation to obtain a segmentation result, including:
[0081] At least one segmentation level is configured for the visual large model, so that the visual large model performs target object segmentation based on the segmentation level and the two-dimensional image after the region segmentation, and obtains a segmentation result corresponding to the segmentation level.
[0082] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the target object segmentation method of any embodiment of the present disclosure.
[0083] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the target object segmentation method of any embodiment of the present disclosure is implemented.
[0084] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the target object segmentation method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.
[0086] Figure 1 It is a flowchart of a target object segmentation method according to an embodiment of the present disclosure.
[0087] Figure 2 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0088] Figure 3 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0089] Figure 4 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0090] Figure 5 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0091] Figure 6 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0092] Figure 7 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0093] Figure 8 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0094] Fig. 9 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0095] Fig.10 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0096] Fig.11 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0097] Fig.12 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0098] Fig.13 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0099] Fig.14 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0100] Fig.15 It is a schematic block diagram of the structure of a target object segmentation device using a hardware implementation of a processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0101] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.
[0102] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0103] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged in addition without departing from the technical concept of the present disclosure.
[0104] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.
[0105] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.
[0106] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values and / or the values provided that will be recognized by those of ordinary skill in the art.
[0107] Related technical description:
[0108] Prompt engineering is an important part of the use of large models (such as SAM models). Prompt is a pre-input during model inference. This input can well control the behavior of the large model (the SAM model is taken as an example below). According to the changes in the input prompt, the output results of the large model will also change accordingly. And because this part (that is, output based on the "prompt") is the inference part of the large model, it does not require training and is more efficient.
[0109] The SAM model can segment everything and separate all pixels in the input image, but its distinguishing scale cannot be determined and classification cannot be achieved. There are three main types of prompt inputs for the SAM model. One type is mask prompts, which can obtain more accurate segmentation effects on the basic mask. For example, first, the SAM model can obtain a preliminary regional mask of the target object based on the threshold method or edge detection technology. This regional mask, i.e., the basic mask, only roughly outlines the position and size of the target object, but it is not completely accurate. If the SAM model needs to identify a certain category of target objects (such as workpieces), it can first use the detection algorithm of the target object of that category to locate the position of the target object and generate a mask. Then, this mask is input into the SAM model as a prompt. The model will segment the image according to the mask prompt and generate an accurate mask of a certain category of target objects. One type is the prompt box, which is one or more 2D detection boxes, allowing the SAM model to focus on segmentation within this box; another type is the prompt point. For a target object, there can be one or more points as prompts. As the number of prompt points increases, the segmentation effect of the model will also be improved; the above three types of prompts can be combined as input, or only one can be used as input.
[0110] One purpose of the present disclosure is to obtain more accurate results that are more suitable for the scene by constructing prompts of different modes, so as to realize the application of large visual models (such as SAM models) in industrial scenes.
[0111] The present disclosure may use a 3D camera (or a depth camera) to collect a 3D point cloud and a 2D image of a scene or object. The 3D camera may obtain a 2D image and a 3D point cloud of a scene or object by integrating a traditional 2D camera and a depth sensor (such as an infrared transmitter, a laser scanner, etc.). The present disclosure does not specifically limit the structure of the 3D camera.
[0112] Figure 1 It is a flowchart of a target object segmentation method according to an embodiment of the present disclosure.
[0113] refer to Figure 1 In some embodiments of the present disclosure, a target object segmentation method M100 based on a visual large model includes:
[0114] S102, obtaining a three-dimensional point cloud of the target scene in a camera coordinate system and a two-dimensional image in an image coordinate system;
[0115] S104, performing block processing on the three-dimensional point cloud to obtain at least one point cloud block;
[0116] S106, performing principal component analysis on the point cloud block to obtain the main direction of the point cloud block, and then obtaining a directed bounding box of the point cloud block;
[0117] S108, projecting the directed bounding box onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system, obtaining a two-dimensional bounding box of the target object and using it as a bounding box prompt;
[0118] S110, determining whether the number of target objects in the directed bounding box is one;
[0119] If the number of target objects in the directed bounding box is one, the following process is performed:
[0120] The bounding box prompt is input into the visual large model, so that the visual large model segments the target object in the two-dimensional image based on the bounding box prompt to obtain the segmentation result.
[0121] The visual large model described in the present disclosure can be various large models that can perform image segmentation tasks, such as the existing SAM model, or the EntitySeg model, or the SEEM (Semantic-Enhanced InteractiveSegmentation Model) model, or other visual large models that can perform image segmentation tasks, all of which fall within the protection scope of the present disclosure.
[0122] The target scene may be various industrial scenes such as performing a grasping operation on a target object (ie, a target object), for example, performing a destacking operation on a stack of boxes (ie, a target object) in the scene.
[0123] In step S104, point cloud segmentation can be performed by clustering methods (including DBSCAN clustering, Euclidean clustering, region growing, etc.), and RANSAC point cloud fitting method can also be used, mainly for target objects of special shapes, such as planar objects, cylindrical objects, etc. The present disclosure can use plane fitting and other methods to fit large planar point clouds according to the scene.
[0124] In some embodiments of the present disclosure, the three-dimensional point cloud can be downsampled to reduce the running time, and then a filtering method is used to remove some redundant points to reduce interference, and then the point cloud is segmented through clustering or fitting methods. The selection or adjustment of the point cloud segmentation process based on the specific scenario by those skilled in the art falls within the protection scope of the present disclosure.
[0125] In the present disclosure, the directed bounding box of the point cloud block is a three-dimensional bounding box.
[0126] In the present disclosure, the transformation relationship between the camera coordinate system and the image coordinate system can be obtained through a pre-calibration process of the 3D camera, which belongs to the prior art and will not be described in detail in the present disclosure.
[0127] The target object segmentation method of this embodiment uses the obtained two-dimensional bounding box of the target object as a bounding box prompt and inputs it into the visual large model to enhance the image segmentation capability of the visual large model, thereby obtaining a more accurate target object segmentation result.
[0128] However, in some embodiments of the present disclosure, due to the limitations of the point cloud block segmentation method, in some scenarios, the directed bounding box of the point cloud block may not be accurate enough. For example, a large point cloud obtained by plane fitting cannot obtain a two-dimensional bounding box for each target object, that is, there may be multiple target objects in a two-dimensional bounding box obtained based on a directed bounding box. In order to improve the accuracy of image segmentation, for the case where the number of target objects in the directed bounding box is more than one, the present disclosure further improves the above-mentioned target object segmentation method.
[0129] Figure 2 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0130] refer to Figure 2 In some embodiments of the present disclosure, in the target object segmentation method M100 of the present disclosure, if the number of target objects in the directed bounding box is more than one, the following process is performed:
[0131] S1111, extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set;
[0132] S1112, projecting the points in the extraction point set onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system, obtaining a projection point set as a point prompt;
[0133] S112, inputting the bounding box prompt and the point prompt into the visual large model, so that the visual large model segments the target object of the two-dimensional image based on the bounding box prompt and the point prompt to obtain a segmentation result.
[0134] This implementation further enhances the image segmentation capability of the visual large model through the obtained point prompts.
[0135] In some embodiments of the present disclosure, the points in the extracted point set are representative points of the point cloud distribution, that is, regular points in the point cloud block. These regular points can be feature points in the point cloud extracted based on a feature extraction algorithm (which can be a feature extraction algorithm in the PCL (Point Cloud Library) point cloud processing library), such as points with large curvature, points with changing normal directions, etc. These regular points can also be obtained by dividing the point cloud block into different point cloud clusters through a cluster analysis method, and then selecting representative points from each point cloud cluster. These regular points can also be obtained by selecting more points in high-density areas of the point cloud block and fewer points in low-density areas through a density analysis method, thereby obtaining representative points. This method can balance the representativeness of the points in the point cloud block.
[0136] In some other embodiments of the present disclosure, the points in the extraction point set are random sampling points. A preset number of points can be randomly extracted directly from the point cloud block to form the extraction point set, and the randomness and uniformity of the sampling can be ensured as much as possible. This can be achieved using existing random extraction algorithms.
[0137] For the target object segmentation method in the above embodiment, extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set preferably includes:
[0138] voxelize the point cloud block to obtain a voxelized point cloud block;
[0139] Extract multiple point cloud distribution representative points or multiple random sampling points from the voxelized point cloud block to obtain an extracted point set.
[0140] Point cloud voxelization generally includes the following steps:
[0141] Determine voxel resolution: First, determine the appropriate voxel resolution based on the density of the point cloud and the accuracy required for processing. Voxel resolution determines the size of the voxel, that is, the size of the space occupied by each voxel.
[0142] Create Voxel Grid: Creates a voxel grid in 3D space at the selected resolution, covering the entire point cloud block.
[0143] Point cloud aggregation within a voxel: traverse each point in the point cloud and assign each point to the corresponding voxel. If there are multiple points in a voxel, you can choose to keep all points, take the average, or select the center point as the representative point of the voxel.
[0144] Figure 3 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0145] refer to Figure 3 In some embodiments of the present disclosure, the target object segmentation method M200 includes:
[0146] S202, obtaining a three-dimensional point cloud of the target scene in a camera coordinate system and a two-dimensional image in an image coordinate system;
[0147] S204, performing block processing on the three-dimensional point cloud to obtain at least one point cloud block;
[0148] S206, extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set;
[0149] S208, projecting the points in the extraction point set onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system, obtaining a projection point set as a point prompt;
[0150] S210, inputting the point prompt into the visual large model, so that the visual large model segments the target object of the two-dimensional image based on the point prompt to obtain a segmentation result.
[0151] In this embodiment, the obtained point prompts can be used alone to enhance the image segmentation capability of the visual large model.
[0152] Figure 4 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0153] refer to Figure 4 In some embodiments of the present disclosure, the target object segmentation method M100 of the present disclosure further includes:
[0154] S111, projecting points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a projection point set;
[0155] generating an initial two-dimensional mask image based on the set of projection points;
[0156] Performing dilation and / or erosion processing on the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint;
[0157] If the number of target objects in the directed bounding box is one, the following process is performed:
[0158] S112, inputting the mask prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt and the bounding box prompt to obtain a segmentation result.
[0159] refer to Figure 4 ,In this implementation, when the number of target objects in the directed ,bounding box is one, a mask hint acquisition process is performed, and the ,acquired mask hint and bounding box hint are used to enhance the image ,segmentation capability of the large visual model.
[0160] In some embodiments of the present disclosure, an initial two-dimensional mask image may be obtained based on the following method:
[0161] Determine foreground pixels: For each point in the projected point set, it has a corresponding pixel position on the two-dimensional image plane. Mark these pixels as foreground, which can be assigned a value of 1 or 255 (if it is a grayscale image).
[0162] Determine background pixels: All pixels except foreground pixels are considered background. Mark these pixels as 0.
[0163] In this embodiment, a mask prompt is obtained by obtaining a set of projection points, and based on the bounding box prompt, the mask prompt is further used to enhance the image segmentation capability of the large visual model.
[0164] Figure 5 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0165] refer to Figure 5 , the target object segmentation method M300 of this embodiment includes:
[0166] S302, obtaining a three-dimensional point cloud of the target scene in a camera coordinate system and a two-dimensional image in an image coordinate system;
[0167] S304, performing block processing on the three-dimensional point cloud to obtain at least one point cloud block;
[0168] S306, projecting the points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a set of projection points;
[0169] S308, generating an initial two-dimensional mask image based on the projection point set;
[0170] S310, dilating and / or corroding the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint;
[0171] S312, inputting the mask prompt into the visual large model, so that the visual large model segments the target object in the two-dimensional image based on the prompt to obtain a segmentation result.
[0172] In this embodiment, the image segmentation capability of the large visual model can be enhanced by using mask hints obtained based on the projection point set alone.
[0173] Figure 6 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0174] refer to Figure 6 In some embodiments of the present disclosure, the target object segmentation method M100 of the present disclosure further includes:
[0175] S111, detecting a target object in a two-dimensional image using a deep learning-based detection algorithm, obtaining at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame, and using the two-dimensional detection frame prompt as a detection frame prompt, wherein the detection frame prompt includes the two-dimensional detection frame and its central pixel point;
[0176] If the number of target objects in the directed bounding box is one, the following process is performed:
[0177] S112, inputting the detection box prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the detection box prompt and the bounding box prompt to obtain a segmentation result.
[0178] Among them, the deep learning detection algorithm can be, for example, the YOLO (You Only Look Once) algorithm. Those skilled in the art can adjust or select the deep learning detection algorithm, all of which fall within the protection scope of the present disclosure.
[0179] The deep learning detection method is relatively simple to label, requires less data, has relatively low accuracy requirements, and has a short training and actual detection time. However, the detection often fails to achieve the actual grasping accuracy, or may cause subsequent point cloud matching to produce mismatches. Therefore, in some embodiments of the present disclosure, the deep learning detection algorithm can be used to generate a detection box as a detection box prompt to the visual large model to enhance its image segmentation ability to achieve better image segmentation effects. Among them, the deep learning detection algorithm can be pre-labeled and trained through the collected two-dimensional images, which will not be repeated in this disclosure.
[0180] The target object segmentation method of this embodiment further uses the detection box prompt obtained by the deep learning-based detection algorithm on the basis of the bounding box prompt to enhance the image segmentation ability of the large visual model.
[0181] Figure 7 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0182] refer to Figure 7 In some embodiments of the present disclosure, the target object segmentation method M400 of this embodiment includes:
[0183] S402, obtaining a two-dimensional image of the target scene in an image coordinate system;
[0184] S404: Detect the target object in the two-dimensional image using a deep learning-based detection algorithm (e.g., You Only Look Once (YOLO) algorithm), obtain at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame, and use them as a detection frame prompt, where the detection frame prompt includes the two-dimensional detection frame and its central pixel point;
[0185] S406: Input the detection frame prompt to the visual large model, so that the visual large model segments the target object of the two-dimensional image based on the detection frame prompt to obtain a segmentation result.
[0186] In this embodiment, the obtained detection box prompts can be used alone to enhance the image segmentation capability of the visual large model.
[0187] Figure 8 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0188] refer to Figure 8 In some embodiments of the present disclosure, the target object segmentation method M100 of the present disclosure further includes:
[0189] S111, using a pre-acquired image template including feature information of a target object to perform feature matching based on the feature information of the target object on the two-dimensional image, to obtain matching feature points of the two-dimensional image;
[0190] Generate a two-dimensional mask image for the two-dimensional image based on matching feature points of the two-dimensional image and use it as a mask hint;
[0191] Extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame hint;
[0192] If the number of target objects in the directed bounding box is one, the following process is performed:
[0193] S112, inputting the mask prompt, the detection box prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt, the detection box prompt and the bounding box prompt to obtain a segmentation result.
[0194] The feature matching of this embodiment (which is a 2D matching method) can be achieved by rotating, scaling, and other processes a basic image (i.e., an image template) and then matching it in the actual image using the target object feature information (e.g., gradient information, etc.). The problem with this method is that the precision and accuracy are not high enough, so the key features of the target object can be used to find a basic position of the target object in the two-dimensional image by adjusting parameters (i.e., step S111), and then the visual large model can be segmented to improve the segmentation accuracy and reduce the segmentation time of the visual large model.
[0195] Exemplarily, features for 2D matching can be extracted from a scene image (i.e., a two-dimensional image), and an image template can be made by cutting out or drawing. The image template is then rotated and / or scaled, and more preferably, an image pyramid level is set, the template image is trained, and corresponding features are generated and saved. Then, the two-dimensional image that needs to be segmented as the target object is input, and the features generated by the image template are matched in the two-dimensional image. The number and score of the feature points finally obtained are subjected to IOU and / or non-maximum suppression to finally obtain a relatively reasonable result as the matching feature point of the two-dimensional image. Then, based on the above-mentioned rotation angle and / or scaling ratio, the two-dimensional image is converted to obtain an actual matching result, and a mask of a two-dimensional image is obtained as a mask prompt.
[0196] In this embodiment, based on the bounding box prompt, the mask prompt obtained based on feature matching and the detection box prompt obtained based on the two-dimensional mask image are further used to enhance the image segmentation capability of the large visual model.
[0197] Fig. 9 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0198] refer to Fig. 9 The target object segmentation method M500 of this embodiment further includes:
[0199] S502, obtaining a two-dimensional image of the target scene in an image coordinate system;
[0200] S504, using a pre-acquired image template including feature information of the target object to perform feature matching based on the feature information of the target object on the two-dimensional image, to obtain matching feature points of the two-dimensional image;
[0201] S506, generating a two-dimensional mask image for the two-dimensional image based on the matching feature points of the two-dimensional image and using it as a mask prompt;
[0202] S508, extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame prompt;
[0203] S510, inputting the mask prompt and the detection box prompt into the visual large model, so that the visual large model segments the target object of the two-dimensional image based on the mask prompt and the detection box prompt to obtain a segmentation result.
[0204] In this embodiment, the image segmentation capability of the large visual model can be enhanced by using mask hints obtained based on feature matching and detection frame hints obtained based on a two-dimensional mask image.
[0205] The target object feature information described in this embodiment preferably includes the position information of the target object (ie, the position information in the image) and the pixel gradient information of the target object.
[0206] Fig.10 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0207] refer to Fig.10 In some embodiments of the present disclosure, the target object segmentation method M100 of the present disclosure includes:
[0208] S111, performing feature enhancement processing on the two-dimensional image to obtain a feature enhanced two-dimensional image;
[0209] Perform foreground extraction on the feature enhanced image to obtain a foreground image;
[0210] Performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt;
[0211] If the number of target objects in the directed bounding box is one, the following process is performed:
[0212] S112, inputting the target object mask prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt and the bounding box prompt to obtain a segmentation result.
[0213] In this embodiment, the two-dimensional image is first feature enhanced, and the foreground image is extracted based on the feature enhanced two-dimensional image. The foreground image is then segmented based on an image segmentation algorithm (various existing image segmentation algorithms can be used) to extract the mask of each target object.
[0214] For some scenes, such as scenes where target objects (such as boxes) are evenly placed, various image segmentation algorithms such as watershed algorithm, distance transform algorithm, edge extraction algorithm, etc. can be used to extract features in the image and then achieve image segmentation, but the main problem with these methods is inaccuracy. The results of these methods can be used as prompts to input into the visual large model to achieve accurate detection.
[0215] Exemplarily, for a scene image (two-dimensional image), the image is preprocessed using methods such as binarization, dilation and erosion in image processing to obtain a feature-enhanced two-dimensional image.
[0216] For feature-enhanced two-dimensional images, distance transform and other methods can be used to calculate the center and outline of each part through the distance relationship between each pixel and the background, and then extract the foreground image. The foreground image is then segmented based on the image segmentation algorithm.
[0217] In this embodiment, on the basis of the bounding box prompt, the target object mask prompt obtained based on the feature enhanced two-dimensional image is further used to enhance the image segmentation capability of the visual large model.
[0218] Fig.11 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0219] refer to Fig.11 The target object segmentation method M600 of this embodiment includes:
[0220] S602, obtaining a two-dimensional image of the target scene in an image coordinate system;
[0221] S604, performing feature enhancement processing on the two-dimensional image to obtain a feature enhanced two-dimensional image;
[0222] S606, performing foreground extraction on the feature enhanced image to obtain a foreground image;
[0223] S608, performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt;
[0224] S610: Input the target object mask prompt to the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt to obtain a segmentation result.
[0225] Fig.12 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0226] refer to Fig.12 The target object segmentation method M700 of this embodiment includes:
[0227] S702, obtaining a two-dimensional image of the target scene in an image coordinate system;
[0228] S704: inputting a language prompt to a multimodal large model (e.g., Grounding DINO) based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt;
[0229] S706: Input the target object detection box prompt to the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection box prompt to obtain a segmentation result.
[0230] The big language model is an artificial intelligence model with a huge number of parameters for language understanding and generation. It is trained on a large amount of text data and can perform a variety of tasks, including language understanding, translation, summarization, etc.
[0231] The multimodal big model described in the present disclosure combines the language understanding ability of the language big model with the analysis of other modal data to achieve a fusion effect.
[0232] The Grounding DINO large model preferably used in the present disclosure is a multimodal large model that performs target detection on an image by inputting a language prompt and detects the target corresponding to the language.
[0233] This embodiment uses a multimodal large model to directly generate prompts, which can detect target objects of known target object types. For example, if there is a pill box (target object) in the scene, you can input "pill Box" as a language prompt, and the multimodal large model can identify this target object and obtain a detection box and a center point. However, this detection result is not accurate enough and is often difficult to meet the needs of use. Therefore, the present disclosure uses this result as a prompt to input it into the visual large model to accurately segment the target object.
[0234] Generally speaking, large models such as Grounding DINO have strong language comprehension capabilities and do not require training. You can input a sentence as a prompt, and the model will split the sentence into individual words, and then detect the target corresponding to each word. It can detect multiple targets at the same time and then input them into the large visual model, thereby enabling the classification of target objects of different categories.
[0235] In this implementation, the image segmentation capability of the visual large model may be enhanced based solely on the target object detection box prompt obtained from the multimodal large model.
[0236] Fig.13 It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0237] refer to Fig.13In some embodiments of the present disclosure, the target object segmentation method M100 of the present disclosure includes:
[0238] S111, inputting a language prompt to a multimodal large model (e.g., Grounding DINO) based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt;
[0239] If the number of target objects in the directed bounding box is one, the following process is performed:
[0240] S112, inputting the target object detection box prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection box prompt and the bounding box prompt to obtain a segmentation result.
[0241] In this embodiment, on the basis of the bounding box prompt, the target object detection box prompt obtained based on the multimodal large model is further used to enhance the image segmentation capability of the visual large model.
[0242] In some embodiments of the present disclosure, the language prompt described above is obtained by the following method:
[0243] Marking target objects in one or more two-dimensional images to obtain marked two-dimensional images;
[0244] The labeled two-dimensional image is input into the multimodal large model, and the multimodal large model outputs language describing the target object based on the labeled two-dimensional image as a language prompt.
[0245] This embodiment can obtain more accurate language prompts. If the segmented words and sentences are manually filled in as language prompts, it is necessary to test which word and sentence has a better effect. Therefore, the present disclosure proposes the above-mentioned more general language prompt acquisition method that does not require input of prompt words and does not require training.
[0246] In this embodiment, for a two-dimensional image, the target object to be identified is marked, and the image is sent to the Grounding DINO model for reasoning (i.e., back propagation), and the language describing the marked target object output by the Grounding DINO model is used as the target phrase. Then, this phrase is used as a language prompt, i.e., a prompt word, to reason all the two-dimensional images of the current scene, and the target object detection frame is obtained as the detection frame prompt, and the center point of the detection frame is used as a point prompt, which is input into the visual model.
[0247] Fig.14It is a flowchart of a target object segmentation method according to another embodiment of the present disclosure.
[0248] refer to Fig.14 The target object segmentation method M800 of this embodiment includes:
[0249] S802, obtaining a two-dimensional image of the target scene in an image coordinate system;
[0250] S804, marking a carrier (such as a material frame or a pallet) carrying the target object in the two-dimensional image to obtain a carrier marking frame;
[0251] S806, performing region segmentation on the two-dimensional image based on the carrier mark frame to obtain a two-dimensional image after region segmentation, wherein the two-dimensional image after region segmentation includes one or more mark frame regions;
[0252] S808, inputting the two-dimensional image after the region segmentation into the visual large model, so that the visual large model performs target object segmentation based on the two-dimensional image after the region segmentation to obtain a segmentation result.
[0253] Preferably, the above-described inputting of the two-dimensional image after region segmentation into the visual large model, so that the visual large model performs target object segmentation based on the two-dimensional image after region segmentation to obtain the segmentation result, includes:
[0254] At least one segmentation level is configured for the visual large model, so that the visual large model performs target object segmentation based on the segmentation level and the two-dimensional image after regional segmentation to obtain a segmentation result corresponding to the segmentation level.
[0255] For simple scenes such as industrial pallets or industrial material frames, the scenes are basically fixed, and the positions of the pallets or material frames are also fixed. Therefore, this embodiment can mark the material frames or pallets in the two-dimensional image, mark the material frames or pallets where the target object is located with a marking frame (i.e., the carrier marking frame), and then crop out the area within the marking frame. The two-dimensional image after area cropping (i.e., area segmentation) is used as the input of the SAM large model, and the target object is segmented directly without input prompts. Preferably, the segmentation level of the SAM large model can be adjusted to determine how refined the segmentation needs to be, so that the segmentation result of the entire scene image can be obtained, and the entire visual large model can be applied to the current scene.
[0256] From the above description of the target object segmentation method of each embodiment, it can be seen that the present disclosure constructs various types of prompts to enhance the image segmentation ability of the visual large model, and in some specific cases or specific scenes, the image segmentation ability of the visual large model is further enhanced, thereby improving the accuracy of target object segmentation in two-dimensional images.
[0257] For the target object segmentation method of each embodiment of the present disclosure, the segmentation of the visual large model such as the SAM model used therein is semantic segmentation, and segmentation is not performed according to the category of the target object. Therefore, the category of the target object can be input into the SAM model as the actual category based on the category of the target object in the target scene. For embodiments involving point cloud processing, target object detection, and target object matching, the category of the target object can be set for different point cloud processing results, target object detection results, and target object matching results, and after being input into the visual large model, it becomes the category of the segmented target object.
[0258] The segmentation results obtained by the target object segmentation method disclosed in the present invention are more accurate than those of existing methods, and can effectively guide the robot system to perform operations such as grasping the target object in the target scene.
[0259] Each step of the target object segmentation method in each of the above-mentioned embodiments of the present disclosure can be implemented based on a computer program.
[0260] Fig.15 It is a schematic block diagram of the structure of a target object segmentation device using a hardware implementation of a processor according to an embodiment of the present disclosure.
[0261] refer to Fig.15 , the target object segmentation device 1000 based on the visual big model of this embodiment includes:
[0262] The 3D point cloud and 2D image acquisition module 1002 acquires the 3D point cloud in the camera coordinate system and the 2D image in the image coordinate system of the target scene captured by the camera;
[0263] The point cloud block acquisition module 1004 performs block processing on the three-dimensional point cloud to obtain at least one point cloud block;
[0264] The point cloud block directed bounding box acquisition module 1006 performs principal component analysis on the point cloud block to obtain the main direction of the point cloud block, and then obtains the directed bounding box of the point cloud block;
[0265] A prompt generation module 1008 projects the directed bounding box onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system, obtains a two-dimensional bounding box of the target object and uses it as a bounding box prompt; and
[0266] A target object quantity determination module 1010, the target object quantity determination module 1010 determines whether the number of target objects in the directed bounding box is one;
[0267] The visual large model 1012, if the number of target objects in the directed bounding box is one, performs the following process:
[0268] The bounding box prompt is input into the visual large model 1012, so that the visual large model segments the target object of the two-dimensional image based on the bounding box prompt to obtain a segmentation result.
[0269] The target object segmentation device of the present disclosure may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by the corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for being implemented by a processor, or implemented by some combination.
[0270] The hardware structure of the target object segmentation device of the present disclosure can be implemented using a bus architecture. The bus architecture may include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 may also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.
[0271] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.
[0272] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).
[0273] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.
[0274] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.
[0275] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0276] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.
[0277] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.
[0278] Therefore, the present disclosure also provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the target object segmentation method of any embodiment of the present disclosure.
[0279] The present disclosure further provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the target object segmentation method of any embodiment of the present disclosure.
[0280] The present disclosure also provides a computer program product, including a computer program, which implements the target object segmentation method of any embodiment of the present disclosure when the computer program is executed by a processor.
[0281] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.
Claims
1. A target object segmentation method based on a visual large model, characterized in that: include: Obtain the three-dimensional point cloud of the target scene in the camera coordinate system and the two-dimensional image in the image coordinate system; Performing block processing on the three-dimensional point cloud to obtain at least one point cloud block; Performing principal component analysis on the point cloud block to obtain the main direction of the point cloud block, and then obtaining a directed bounding box of the point cloud block; Based on the transformation relationship between the camera coordinate system and the image coordinate system, the directed bounding box is projected onto the plane where the two-dimensional image is located, and a two-dimensional bounding box of the target object is obtained as a bounding box prompt; as well as Determining whether the number of target objects in the directed bounding box is one; If the number of target objects in the directed bounding box is one, the following process is performed: The bounding box prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the bounding box prompt to obtain a segmentation result.
2. The target object segmentation method according to claim 1, characterized in that: If the number of target objects in the directed bounding box is more than one, the following process is performed: Extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set; Based on the transformation relationship between the camera coordinate system and the image coordinate system, the points in the extraction point set are projected onto the plane where the two-dimensional image is located, and a projection point set is obtained as a point prompt; as well as The bounding box prompt and the point prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the bounding box prompt and the point prompt to obtain a segmentation result.
3. The target object segmentation method according to claim 2, characterized in that: in, Extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set includes: voxelize the point cloud block to obtain a voxelized point cloud block; and A plurality of point cloud distribution representative points or a plurality of random sampling points are extracted from the voxelized point cloud block to obtain the extracted point set.
4. The target object segmentation method according to claim 1, characterized in that: Alternatively, the target object segmentation method includes: Extracting a plurality of point cloud distribution representative points or a plurality of random sampling points from the point cloud block to obtain an extracted point set; Projecting the points in the extraction point set onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system, obtaining a projection point set as a point prompt; and The point prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the point prompt to obtain a segmentation result.
5. The target object segmentation method according to claim 1, characterized in that: The target object segmentation method also includes: Projecting the points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a set of projection points; generating an initial two-dimensional mask image based on the projection point set; Performing dilation and / or erosion processing on the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint; and If the number of target objects in the directed bounding box is one, the following process is performed: The mask prompt and the bounding box prompt are input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt and the bounding box prompt to obtain a segmentation result.
6. The target object segmentation method according to claim 1, characterized in that: Alternatively, the target object segmentation method includes: Projecting the points in the point cloud block onto the plane where the two-dimensional image is located based on the transformation relationship between the camera coordinate system and the image coordinate system to obtain a set of projection points; generating an initial two-dimensional mask image based on the projection point set; Performing dilation and / or erosion processing on the initial two-dimensional mask image to obtain a refined two-dimensional mask image, and using the refined two-dimensional mask image as a mask hint; and The mask prompt is input into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the prompt to obtain a segmentation result.
7. The target object segmentation method according to any one of claims 1 to 6, characterized in that: The target object segmentation method also includes: Detecting the target object in the two-dimensional image using a deep learning-based detection algorithm, obtaining at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame as a detection frame prompt; and If the number of target objects in the directed bounding box is one, the following process is performed: Inputting the detection box prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the detection box prompt and the bounding box prompt to obtain a segmentation result; Optionally, alternatively, the target object segmentation method includes: Detecting the target object in the two-dimensional image using a deep learning-based detection algorithm, obtaining at least one two-dimensional detection frame and a central pixel point of the two-dimensional detection frame as a detection frame prompt; and Inputting the detection frame prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the detection frame prompt to obtain a segmentation result; Optionally, the target object segmentation method further includes: Using a pre-acquired image template including feature information of the target object, performing feature matching based on the feature information of the target object on the two-dimensional image to obtain matching feature points of the two-dimensional image; Generating a two-dimensional mask image for the two-dimensional image based on the matching feature points of the two-dimensional image as a mask prompt; Extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame prompt; and If the number of target objects in the directed bounding box is one, the following process is performed: Inputting the mask prompt, the detection box prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt, the detection box prompt and the bounding box prompt to obtain a segmentation result; Optionally, alternatively, the target object segmentation method includes: Using a pre-acquired image template including feature information of the target object, performing feature matching based on the feature information of the target object on the two-dimensional image to obtain matching feature points of the two-dimensional image; Generating a two-dimensional mask image for the two-dimensional image based on the matching feature points of the two-dimensional image as a mask prompt; Extracting a two-dimensional detection frame and a center point of the two-dimensional detection frame of the target object from the two-dimensional mask image and using them as a detection frame prompt; Inputting the mask prompt and the detection frame prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the mask prompt and the detection frame prompt to obtain a segmentation result; Optionally, the target object feature information includes position information of the target object and pixel gradient information of the target object; Optionally, the target object segmentation method further includes: Performing feature enhancement processing on the two-dimensional image to obtain a feature-enhanced two-dimensional image; Performing foreground extraction on the feature enhanced image to obtain a foreground image; Performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt; and If the number of target objects in the directed bounding box is one, the following process is performed: Inputting the target object mask prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt and the bounding box prompt to obtain a segmentation result; Optionally, alternatively, the target object segmentation method includes: Performing feature enhancement processing on the two-dimensional image to obtain a feature-enhanced two-dimensional image; Performing foreground extraction on the feature enhanced image to obtain a foreground image; Performing segmentation processing on the foreground image based on an image segmentation algorithm, obtaining a target object mask, a target object mask frame, and a target object mask frame center point of each target object in the foreground image, and using the target object mask, the target object mask frame, and the target object mask frame center point as a target object mask prompt; and Inputting the target object mask prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object mask prompt to obtain a segmentation result; Optionally, alternatively, the target object segmentation method includes: Inputting a language prompt to the multimodal large model based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt; Inputting the target object detection frame prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection frame prompt to obtain a segmentation result; Optionally, the target object segmentation method further includes: Inputting a language prompt to the multimodal large model based on the category of the target object, so that the multimodal large model performs target object detection on the two-dimensional image based on the language prompt, obtains a target object detection frame and a detection frame center point, and uses the target object detection frame and the detection frame center point as a target object detection frame prompt; and If the number of target objects in the directed bounding box is one, the following process is performed: Inputting the target object detection frame prompt and the bounding box prompt into the visual large model, so that the visual large model performs target object segmentation on the two-dimensional image based on the target object detection frame prompt and the bounding box prompt to obtain a segmentation result; Optionally, the language prompt is obtained by the following method: Marking the target object in one or more of the two-dimensional images to obtain a marked two-dimensional image; and Inputting the marked two-dimensional image into the multimodal large model, the multimodal large model outputting language describing the target object based on the marked two-dimensional image as the language prompt; Optionally, alternatively, the target object segmentation method includes: Marking a carrier body carrying the target object in the two-dimensional image to obtain a carrier body marking frame; Performing region segmentation on the two-dimensional image based on the carrier mark frame to obtain a two-dimensional image after region segmentation, wherein the two-dimensional image after region segmentation includes one or more mark frame regions; and Inputting the two-dimensional image after the region segmentation into the visual large model, so that the visual large model performs target object segmentation based on the two-dimensional image after the region segmentation to obtain a segmentation result; Optionally, the two-dimensional image after the region segmentation is input into the visual large model, so that the visual large model performs target object segmentation based on the two-dimensional image after the region segmentation to obtain a segmentation result, including: At least one segmentation level is configured for the visual large model, so that the visual large model performs target object segmentation based on the segmentation level and the two-dimensional image after the region segmentation, and obtains a segmentation result corresponding to the segmentation level.
8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the target object segmentation method according to any one of claims 1 to 7.
9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the target object segmentation method according to any one of claims 1 to 7 when executed by a processor.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the target object segmentation method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Image automatic labeling method and device based on visual SLAM, and storage medium
CN113362363A
Target detection method and device, equipment and medium
CN118196398A
Cited By
Target detection method and device in screen door scene
CN119693602A
A target detection method and device in a shielded door scene
CN119693602B
Robot operation method and system based on space-time constraint enhancement
CN121374640A
Robot manipulation methods and systems based on spatiotemporal constraints
CN121374640B
