Three-dimensional point cloud segmentation and robot control method, device and equipment

Abstract

The embodiment of the invention provides a three-dimensional point cloud segmentation and robot control method, device and equipment. The method comprises the following steps: acquiring views of a target object at a plurality of different visual angles; inputting each view of the target object into the first two-dimensional multi-modal large model to obtain all parts included in the target object; segmenting each view by using a two-dimensional segmentation model to obtain an area where each component is located in each view; for each three-dimensional point in the three-dimensional point cloud of the target object, determining a part to which an area where a projection point of the three-dimensional point in each view is located belongs, and obtaining a projection result used for reflecting the projection condition of the three-dimensional point; and according to each projection result, each three-dimensional point is divided to a respective maximum hit part, a first segmentation result of the three-dimensional point cloud is obtained, and the maximum hit part is a part which appears most in the projection results of the three-dimensional points. The accuracy of 3D point cloud segmentation can be improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional vision technology, and in particular to three-dimensional point cloud segmentation and robot control methods, devices and equipment. Background Technology

[0002] Human visual perception possesses the generalization ability to break down objects into multiple parts and to analyze unseen objects, which is crucial for understanding the structure, semantics, mobility, and functionality of objects. Three-dimensional (3D) part segmentation technology, simulating this human ability, plays a vital role in numerous applications such as robot manipulation, augmented reality / virtual reality, and shape analysis and synthesis. 3D part segmentation (i.e., 3D point cloud segmentation) refers to dividing 3D points according to the parts they belong to. However, current technologies for 3D point cloud segmentation based on 3D segmentation models are still in the early stages of exploration, and related research is insufficient, resulting in relatively low segmentation accuracy. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, and device for 3D point cloud segmentation and robot control, so as to achieve accurate segmentation of 3D point clouds. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of the present invention provide a three-dimensional point cloud segmentation method, the method comprising:

[0005] Obtain views of the target object from multiple different perspectives;

[0006] Input each of the views of the target object into the first two-dimensional multimodal large model to obtain all the components contained in the target object;

[0007] The views are segmented using a two-dimensional segmentation model to obtain the regions where each component is located in each view;

[0008] For each three-dimensional point in the three-dimensional point cloud of the target object, determine the component to which the region where the projection point of the three-dimensional point is located in each of the views belongs, and obtain the projection result that reflects the three-dimensional point;

[0009] Based on each projection result, each three-dimensional point is divided into its most frequently hit component to obtain the first segmentation result of the three-dimensional point cloud; wherein, the most frequently hit component is the component that appears most often in the projection result of the three-dimensional point.

[0010] In one possible implementation, the step of segmenting each of the views using a two-dimensional segmentation model to obtain the region where each component is located in each of the views includes:

[0011] Add tooltips to each of the aforementioned views;

[0012] The component to which each of the aforementioned prompts belongs is determined using a second two-dimensional multimodal large language model;

[0013] Each of the aforementioned prompt points, the component to which each prompt point belongs, and each of the aforementioned views are input into a two-dimensional segmentation model to obtain the region where each of the aforementioned components is located in each of the aforementioned views.

[0014] In one possible implementation, adding the cue points to each of the views includes:

[0015] For each of the aforementioned views, cue points are added to the views in a manner that minimizes the cue point loss of the views; wherein, the cue point loss includes minimum distance loss and / or uniformity loss; the minimum distance loss is positively correlated with the distance of each pixel in the view to the nearest cue point; the uniformity loss is negatively correlated with the spacing between the cue points;

[0016] And / or,

[0017] Each of the views is input into a two-dimensional segmentation model to obtain the regions included in each view. For each view, prompt points are added to the view according to the proportion of each region in the view, wherein the number of prompt points added to each region is positively correlated with the proportion of that region in the view.

[0018] In one possible implementation, the step of inputting each of the views into a two-dimensional segmentation model to obtain the region included in each of the views includes:

[0019] Each of the views is input into the two-dimensional segmentation model to obtain the region included by each of the views in the two-dimensional segmentation model under unrestricted conditions.

[0020] In one possible implementation, the method further includes:

[0021] Outliers are removed from the first segmentation result to obtain the second segmentation result, wherein outliers are three-dimensional points whose distance from other three-dimensional points segmented into the same component is greater than a preset distance threshold.

[0022] In one possible implementation, the method further includes:

[0023] For each unsuccessful 3D point in the second segmentation result, determine the nearest successful 3D point to the unsuccessful 3D point, and segment the unsuccessful 3D point to the component to which the determined successful 3D point belongs. Here, an unsuccessful 3D point is a 3D point that has not been segmented to any component, and a successful 3D point is a 3D point that has been segmented to any component.

[0024] Secondly, embodiments of the present invention provide a robot control method, the method comprising:

[0025] The robot's image sensor is used to acquire views of the target object from multiple different perspectives, and the robot's laser sensor is used to acquire the three-dimensional point cloud of the target object.

[0026] Based on the view and the three-dimensional point cloud, perform any of the three-dimensional point cloud segmentation methods described in the first aspect to obtain a first segmentation result of the three-dimensional point cloud;

[0027] Based on the first segmentation result, the robot is controlled to interact with the target object.

[0028] Thirdly, embodiments of the present invention provide a three-dimensional point cloud segmentation device, the device comprising:

[0029] The acquisition module is used to acquire views of a target object from multiple different perspectives.

[0030] The input module is used to input each of the views of the target object into the first two-dimensional multimodal large model to obtain all the components contained in the target object;

[0031] The segmentation module is used to segment each of the views using a two-dimensional segmentation model to obtain the region where each component is located in each of the views;

[0032] The statistics module is used to count the components to which the projection points of the three-dimensional points in the three-dimensional point cloud of the target object belong in each of the three-dimensional points, and to obtain the projection results that reflect the three-dimensional points.

[0033] The summary module is used to divide each of the three-dimensional points into their respective most hit components according to each of the projection results, so as to obtain the first segmentation result of the three-dimensional point cloud; wherein, the most hit component is the component that appears most frequently in the projection results of the three-dimensional points.

[0034] In one possible implementation, the segmentation module includes:

[0035] The first segmentation submodule is used to add prompt points to each of the views respectively;

[0036] The second segmentation submodule is used to determine the component to which each of the prompt points belongs using the second two-dimensional multimodal large language model;

[0037] The third segmentation submodule is used to input each of the prompt points, the component to which each prompt point belongs, and each of the views into the two-dimensional segmentation model to obtain the area where each component is located in each of the views.

[0038] In one possible implementation, the first segmentation submodule includes:

[0039] The first segmentation unit is configured to add cue points to each of the views in a manner that minimizes the cue point loss of the view; wherein the cue point loss includes minimum distance loss and / or uniformity loss; the minimum distance loss is positively correlated with the distance of each pixel in the view to the nearest cue point; and the uniformity loss is negatively correlated with the spacing between the cue points.

[0040] And / or,

[0041] The second segmentation unit is used to input each of the views into the two-dimensional segmentation model to obtain the regions included in each of the views; for each view, according to the proportion of each region in the view, a prompt point is added to the view, wherein the number of prompt points added to each region is positively correlated with the proportion of that region in the view.

[0042] In one possible implementation, the second segmentation unit includes:

[0043] The first segmentation subunit is used to input each of the views into the two-dimensional segmentation model to obtain the regions included by each of the views in the two-dimensional segmentation model under unrestricted conditions.

[0044] In one possible implementation, the device further includes:

[0045] The elimination module is used to remove outliers from the first segmentation result to obtain a second segmentation result, wherein the outlier is a three-dimensional point whose distance from other three-dimensional points segmented into the same component is greater than a preset distance threshold.

[0046] In one possible implementation, the device further includes:

[0047] The segmentation module is used to determine the nearest resultable 3D point to each resultless 3D point in the second segmentation result, and segment the resultless 3D point to the component to which the determined resultable 3D point belongs. Here, the resultless 3D point is a 3D point that has not been segmented to any component, and the resultable 3D point is a 3D point that has been segmented to any component.

[0048] Fourthly, embodiments of the present invention provide a robot control device, the device comprising:

[0049] The acquisition module is used to acquire views of the target object from multiple different perspectives using the robot's image sensor, and to acquire the three-dimensional point cloud of the target object using the robot's laser sensor.

[0050] An execution module is configured to execute the three-dimensional point cloud segmentation method described in the first aspect based on the view and the three-dimensional point cloud to obtain a first segmentation result of the three-dimensional point cloud;

[0051] An interaction module is used to control the robot to interact with the target object based on the first segmentation result.

[0052] Fifthly, embodiments of the present invention provide an electronic device, comprising:

[0053] Memory, used to store computer programs;

[0054] The processor, when executing the program stored in the memory, implements the three-dimensional point cloud segmentation method of the first aspect or the robot control method of the second aspect.

[0055] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional point cloud segmentation method of the first aspect or the robot control method of the second aspect.

[0056] In a seventh aspect, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the three-dimensional point cloud segmentation method of the first aspect or the robot control method of the second aspect.

[0057] Beneficial effects of the embodiments of the present invention:

[0058] This invention provides a 3D point cloud segmentation and robot control method, apparatus, and device. By acquiring views of a target object from multiple different perspectives, it can comprehensively obtain the object's appearance features. Different perspectives provide complementary information, avoiding problems such as occlusion and information loss that may occur with a single perspective. Utilizing the feature extraction and semantic understanding capabilities of a multimodal large model, each view is input into a first 2D multimodal large model. The multimodal large model can fuse information from different modalities (such as visual features of images, possible text descriptions, etc.) to more accurately identify various components within the object, reducing misjudgments and omissions. A 2D segmentation model is used to segment each view, obtaining the region where each component is located in each view, thus obtaining the positional information of each component on the 2D plane, which serves as an important basis for subsequently associating 3D points with components. By statistically analyzing the components to which the projection points of 3D points in each view belong, information from multiple perspectives is comprehensively considered, avoiding the bias that may arise from relying solely on a single perspective for judgment. By acquiring the component to which each 3D point belongs in different view projection points, the component to which each 3D point most likely belongs can be quickly and accurately determined, i.e., the component with the most hits, and finally the first segmentation result is obtained, thereby improving the accuracy of 3D point cloud segmentation.

[0059] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0061] Figure 1 A schematic diagram of the first flowchart of the 3D point cloud segmentation method provided in this application;

[0062] Figure 2 A schematic diagram of the second flowchart of the 3D point cloud segmentation method provided in this application;

[0063] Figure 3a A schematic diagram of the third process for the 3D point cloud segmentation method provided in this application;

[0064] Figure 3b A schematic diagram of the fourth process for the 3D point cloud segmentation method provided in this application;

[0065] Figure 3c A schematic diagram of the fifth process for the 3D point cloud segmentation method provided in this application;

[0066] Figure 4 A schematic diagram illustrating the phased approach of the 3D point cloud segmentation method provided in this application;

[0067] Figure 5 A flowchart illustrating the robot control method provided in this application;

[0068] Figure 6 A schematic diagram of the 3D point cloud segmentation device provided in this application;

[0069] Figure 7 A schematic diagram of the structure of the robot control device provided in this application;

[0070] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0072] In existing technologies, 3D part segmentation based on 3D segmentation models covers two major tasks: semantic segmentation and instance segmentation. Semantic segmentation uses 3D backbone networks (such as PointNet) to assign semantic labels to each geometric element of a 3D point cloud or voxel; instance segmentation, on the other hand, needs to distinguish different instances under the same semantic category, which is often achieved through grouping strategies or region proposals.

[0073] Meanwhile, to alleviate the reliance on fully supervised data, weakly supervised learning schemes have emerged, such as supervision based on two-dimensional (2D) or 3D bounding boxes; some studies focus on scene-level 3D part segmentation, which can segment multiple objects in complex environments simultaneously; and other studies are dedicated to the hierarchical and fine-grained decomposition of 3D shapes, as well as enhancing the semantic characterization of 3D shapes by learning continuous implicit semantic fields.

[0074] For 3D segmentation training with small samples or low annotations, many methods employ strategies such as pre-training tasks and auxiliary losses to improve the generalization ability of 3D segmentation models. Furthermore, while image-text pre-trained models are widely used in tasks such as 3D shape generation, their application in 3D part segmentation is still in its early stages of exploration. Although 3D segmentation models have made some progress in 3D part segmentation, the following shortcomings remain:

[0075] Training 3D segmentation models relies on large amounts of finely annotated 3D point cloud data. However, 3D model construction and component-level annotation are costly, and datasets are limited in size, making it difficult to cover various objects and complex shapes in real-world applications. Furthermore, existing 3D segmentation models perform poorly with objects outside the training categories or novel shapes, lacking structural understanding and semantic transfer capabilities for new objects. Related research attempts to enhance the learning effect of 3D segmentation models by mapping 2D views to 3D shapes; however, 2D detection or segmentation results are difficult to directly reconstruct into 3D space, easily leading to inaccurate segmentation, missed detections, or misclassifications. While image-text pre-trained models perform well on vision-language tasks, their direct application to 3D component segmentation suffers from significant modal differences and insufficient feature alignment, thus failing to fully realize their potential.

[0076] Based on this, embodiments of the present invention provide a 3D point cloud segmentation method, see [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of a first flowchart of a 3D point cloud segmentation method provided in an embodiment of the present invention. The method includes:

[0077] S101, Obtain views of the target object from multiple different perspectives.

[0078] S102, input each view of the target object into the first 2D multimodal large model to obtain all the components contained in the target object.

[0079] S103, use a 2D segmentation model to segment each view to obtain the area where each component is located in each view.

[0080] S104, for each 3D point in the 3D point cloud of the target object, determine the component to which the area where the projection point of the 3D point is located in each view belongs, and obtain the projection result that reflects the 3D point.

[0081] S105, based on each projection result, divide each 3D point into its respective most hit component to obtain the first segmentation result of the 3D point cloud.

[0082] Among them, the component with the most hits is the component that appears most frequently in the projection results of 3D points.

[0083] By applying the above embodiments, and acquiring views of the target object from multiple different perspectives, the object's appearance features can be comprehensively obtained. Different perspectives provide complementary information, avoiding problems such as occlusion and information loss that may occur with a single perspective. Utilizing the feature extraction and semantic understanding capabilities of a multimodal large model, each view is input into a first 2D multimodal large model. This model can fuse information from different modalities (such as visual features of the image, possible text descriptions, etc.) to more accurately identify each component of the object, reducing false positives and false negatives. A 2D segmentation model is used to segment each view, obtaining the region where each component is located in each view, thus obtaining the positional information of each component on the two-dimensional plane, serving as an important basis for subsequently associating 3D points with components. By statistically analyzing the components to which 3D points belong at their projection points in each view, information from multiple perspectives is comprehensively considered, avoiding the bias that may arise from relying solely on a single perspective. By acquiring the components to which each 3D point belongs at its projection points in different views, the component most likely to belong to each 3D point can be quickly and accurately determined, i.e., the component with the most hits, ultimately obtaining the first segmentation result, thereby improving the accuracy of 3D point cloud segmentation.

[0084] Understandably, in this article Figures 1 to 3c The steps S102 and S103 shown can be executed in parallel, or one step can be executed first and then the other step can be executed, or the two steps can be executed alternately. There is no specific limitation on the execution order of the steps.

[0085] Steps S101-S105 will be explained below:

[0086] In step S101, appearance information of the target object is captured from different angles to gain a more comprehensive and accurate understanding of its structural features. Since a single viewpoint often only presents partial information about an object, views from different perspectives complement each other, providing a rich data foundation for subsequent image processing.

[0087] Different perspectives can be obtained in various ways. For example, multiple cameras can be placed around the target object at different positions to take pictures, or a single camera can be used to rotate around the target object to take pictures. Alternatively, a 3D scanning device can be used to obtain image information of the object from different angles during the scanning process, ultimately resulting in a set of views of the target object from multiple different perspectives. The obtained views can be color views or non-color views, without any specific limitations.

[0088] In step S102, the 2D multimodal large model is pre-trained with a large amount of data and has the ability to accurately identify object parts. Guided by prompts, it can analyze views from different perspectives and identify all the parts contained in the target object. The prompts can be pre-set in the 2D multimodal large model or entered by the user when inputting views from different perspectives.

[0089] Specifically, the views of the target object from different perspectives are input into the pre-trained first 2D multimodal large model, and prompts such as "Please give the name of the object and what parts the object is usually divided into" are input. The first 2D multimodal large model will extract and analyze features for each view, and based on its learned knowledge, it will determine and output all the components contained in the target object. For example, for a car, it may identify the body, wheels, windows, doors and other components.

[0090] In step S103, the 2D segmentation model is used to segment each view to clarify the specific position and range of each component in each view, that is, to determine the boundary of the component on the two-dimensional view.

[0091] For example, 2D segmentation models can be SAM (segment anything model), U-Net, FCN (Fully Convolutional Networks), etc. U-Net is a fully convolutional neural network used for image segmentation.

[0092] Specifically, for each view where all components have been identified, a 2D segmentation model is used for processing. The 2D segmentation model classifies the pixels in the view, grouping pixels belonging to the same component into the same region, thus obtaining the specific region where each component is located in each view. For example, in a view of a car, the 2D segmentation model can accurately divide the wheel area, body area, etc.

[0093] In step S104, the association between points in the 3D point cloud and components in the 2D view is established. By statistically analyzing the components to which the projection points of 3D points belong in different views, the component to which each 3D point in the 3D point cloud belongs is determined.

[0094] Specifically, when determining the component to which the projection points of 3D points in various views belong, in one possible embodiment, each 3D point in the 3D point cloud can be projected onto views from different perspectives to obtain corresponding projection points, thereby determining the component to which each 3D point belongs. In another possible embodiment, the component regions in the 2D view can be back-projected using camera parameters to the coordinate system of the 3D points, thereby determining the component to which each 3D point belongs. In other embodiments, each 3D point in the 3D point cloud can be projected onto a preset unified coordinate system. This coordinate system can be reasonably selected according to the actual application scenario and requirements, for example, establishing a coordinate system with the center of the target object as the origin. Simultaneously, the component regions are also projected onto this preset coordinate system, thereby determining the component to which each 3D point belongs.

[0095] Taking the projection of each 3D point in a 3D point cloud onto views from different perspectives as an example, after obtaining the corresponding projection points, we examine how these projection points fall within the various component regions obtained from the 2D segmentation model. We then statistically analyze the component information to which the projection point belongs, thus obtaining the projection result of the 3D point. For example, the projection points of a 3D point on multiple views may fall into the wheel region, the body region, etc. The projection result reflects the projection situation of the 3D point, that is, it records the component information to which the 3D point belongs.

[0096] In step S105, based on the projection results of each 3D point obtained in step S104, each 3D point in the 3D point cloud is assigned to the component to which it is most likely to belong, thereby completing the segmentation of the 3D point cloud and obtaining the first segmentation result.

[0097] Specifically, for each 3D point in the 3D point cloud, its projection result is examined, and the component that appears most frequently in the projection result is identified as the most frequently encountered component for that 3D point. Then, the 3D point is assigned to the point cloud corresponding to this most frequently encountered component. After performing this operation on each 3D point in the 3D point cloud, the first segmentation result of the 3D point cloud is obtained. For example, if the projection result of a 3D point shows that its projection point appears most frequently in the wheel component region, then that 3D point is segmented into the 3D point cloud corresponding to the wheel component.

[0098] When using a 2D segmentation model to segment each view and obtain the region where each component is located in each view, in one possible embodiment, the 2D segmentation model can segment each view without restrictions to obtain the region where each component is located in each view.

[0099] In another possible embodiment, to improve the accuracy of segmenting the regions where each component is located in each view, thereby improving the accuracy of 3D point cloud segmentation, cue points can be added in each view in advance, and the component to which each cue point belongs can be determined to improve the segmentation accuracy. Specifically, see [link to documentation]. Figure 2 , Figure 2 This is a second flowchart illustrating the 3D point cloud segmentation method provided in an embodiment of the present invention. The method includes:

[0100] S101, Obtain views of the target object from multiple different perspectives.

[0101] S102, input each view of the target object into the first 2D multimodal large model to obtain all the components contained in the target object.

[0102] S1031, Add tooltips to each view.

[0103] S1032, the component to which each prompt point belongs is determined by using the second 2D multimodal large language model.

[0104] S1033, input each prompt point, the component to which each prompt point belongs, and each view into the 2D segmentation model to obtain the area where each component is located in each view.

[0105] S104: For each 3D point in the 3D point cloud of the target object, count the components to which the projection points of the 3D points in each view belong, and obtain the projection results that reflect the 3D points.

[0106] S105, based on each projection result, divide each 3D point into its respective most hit component to obtain the first segmentation result of the 3D point cloud.

[0107] Among them, the component with the most hits is the component that appears most frequently in the projection results of 3D points.

[0108] Steps S1031, S1032, and S1033 are detailed steps of step S103. Steps S101, S102, S104, and S105 have been explained in the preceding text and will not be repeated here.

[0109] In step S1031, the purpose of adding the cue points is to provide reference location information for the subsequent component identification and segmentation process.

[0110] Cue points can be added to each view in various ways. For example, in one possible embodiment, for components with obvious geometric features, cue points can be added at their center, vertex, or key inflection points of their edges. If the view is a 3D object photographed from a specific perspective, cue points can also be added at key locations that may correspond to components, based on the object's projection characteristics in the view. The number of cue points can be flexibly adjusted according to the complexity of the view and the location of the components. For more complex views corresponding to target objects with many components, the number of cue points can be appropriately increased to provide richer positional information.

[0111] For example, in a 2D engineering view containing multiple mechanical parts, a hint point can be added at the center of a circular part; and a hint point can be added at each of its four vertices for a rectangular part.

[0112] In step S1032, the semantic understanding and reasoning capabilities of the second 2D multimodal large language model are used to determine the component corresponding to each prompt point based on the position of the prompt point in the view and the multimodal information contained in the view (such as the visual features of the image, possible text annotations, etc.).

[0113] For example, the second 2D multimodal large language model receives input information including cue points and views. The model first extracts features from the view, analyzing visual features such as color, texture, and shape in the image, while simultaneously combining this with any existing textual information (such as view titles and annotations) for a comprehensive understanding. Then, based on the location information of the cue point, the model performs semantic reasoning within the view to determine which component that location is most likely to belong to. For instance, the model might match the shape features surrounding the cue point with known component templates, or determine the component to which the cue point belongs based on the description of the component in the text annotations.

[0114] In step S1033, each prompt point, the component to which each prompt point belongs, and each view are input into the 2D segmentation model. The 2D segmentation model uses the position information of the prompt point and the component information to which it belongs, combined with the multimodal information contained in the multiple input view perspectives, to segment the view.

[0115] Applying the above embodiments, hint points are added to each view respectively, and by utilizing the information processing and understanding capabilities of the second 2D multimodal large language model, the components to which each hint point belongs are determined respectively. This model can fuse multimodal information, such as the visual features of the image and possible text descriptions, etc., so as to more comprehensively and accurately understand the component information corresponding to each hint point, reducing the error of component recognition. The hint points containing accurate component attribution information and each view are input into the 2D segmentation model, enabling the 2D segmentation model to refer to the position information of each annotation point (hint point) and the accurate information of the component to which it belongs, so that the model can more accurately determine the regions where each component is located in each view, thereby enabling the accurate association of the 3D point cloud and the components, and further improving the accuracy of 3D point cloud segmentation.

[0116] When adding hint points to each view, in one possible embodiment, hint points can be added to the view in a manner that minimizes the hint point loss of the view. Specifically, the aforementioned step S1031 includes:

[0117] S10311, for each view, add hint points to the view in a manner that minimizes the hint point loss of the view.

[0118] Among them, the hint point loss includes the minimum distance loss and / or the uniformity loss. The minimum distance loss is positively correlated with the distance from each pixel point in the view to the nearest hint point, and the uniformity loss is negatively correlated with the spacing between each hint point.

[0119] The minimum distance loss restricts to make the distance from all pixel points to the nearest annotation point the smallest. The uniformity loss restricts to make the distribution of annotation points as uniform as possible.

[0120] Exemplarily, assuming there are N pixel points in the view and M positions are selected for annotation among the N pixel points (M < N), the minimum distance loss can be expressed by the following formula (1):

[0121]

[0122] The uniformity loss can be expressed by the following formula (2):

[0123]

[0124] The hint point loss can be expressed by the following formula (3):

[0125]

[0126] Among them, N is the total number of pixels in the view, M is the number of annotation points, V i is the coordinate of the i-th pixel, W jLet d(V) be the coordinates of the j-th labeled point. i W j ) represents pixel V i To the marked point W j The distance between them, W j Let W be the coordinates of the j-th labeled point. k Let d(W) be the coordinates of the k-th labeled point. j W k ) is the label point W k To the marked point W j The distance between them, ∈ is a preset parameter, α is the weight coefficient of minimum distance loss, and β is the weight coefficient of uniformity loss.

[0127] By applying the above embodiments, adding cue points in a manner that minimizes the loss of cue point information for each view can optimize the layout of cue points. The cue point loss includes minimum distance loss and / or uniformity loss. Minimum distance loss is positively correlated with the distance between each pixel in the view and the nearest cue point. This means that when adding cue points, each pixel is placed as close as possible to at least one cue point, ensuring that each cue point effectively covers the pixel information of its surrounding area, reducing information loss due to insufficient cue points. Uniformity loss is negatively correlated with the spacing between cue points, promoting a uniform distribution of cue points in the view. A uniformly distributed distribution of cue points avoids situations where local areas are too densely or sparsely populated with cue points. Adding cue points in a manner that minimizes the loss of cue point information provides a more accurate data foundation for subsequent cue point-based component recognition and segmentation, improving the accuracy of subsequent 2D segmentation models in segmenting the regions where each component is located in each view. This lays the foundation for accurate association between 3D point clouds and components, as well as for final 3D point cloud segmentation, thus improving the accuracy of 3D point cloud segmentation.

[0128] In another possible embodiment, a cue point annotation algorithm can be designed to add cue points in each view based on the proportion of pixels contained in each component to the total number of pixels in the image, so that the annotation points on each component are as evenly distributed as possible. Specifically, the aforementioned step S1031 further includes:

[0129] S10312, input each view into the 2D segmentation model to obtain the region included in each view.

[0130] In one possible embodiment, each view can be input into the 2D segmentation model for segmentation, that is, no restrictions are placed on the 2D segmentation model, so that the 2D segmentation model can segment as many regions as possible from each view, thereby improving the segmentation accuracy.

[0131] In other possible embodiments, the views along with some prior knowledge can be input into the 2D segmentation model, so that the 2D segmentation model performs segmentation under the constraints of this prior knowledge, thereby improving the accuracy of segmentation. Prior knowledge includes, but is not limited to, the number of parts included in the view, the approximate position of each part, etc.

[0132] S10313, For each view, add tooltips to the view according to the proportion of each area in the view.

[0133] The number of tooltips added to each area is positively correlated with the area's proportion in the view.

[0134] Specifically, each view encompasses its own area, which is the area where the component in that view resides. For each area within a view, its proportion within the entire view needs to be calculated. This proportion can be determined by the ratio of the number of pixels within that area to the total number of pixels in the entire view. For example, if a view has 10,000 pixels and a certain area contains 2,000 pixels, then that area's proportion is 20%.

[0135] After determining the proportion of each area, the number of tooltips is determined based on the proportion of each area in the view. The number of tooltips added to each area is positively correlated with its proportion in the view; that is, the larger the proportion of an area, the more tooltips are added. For example, an area with a proportion of 20% will have 6 tooltips added, while an area with a proportion of 10% will only have 3 tooltips added.

[0136] By applying the above embodiments and adding cue points according to region proportions, the uneven distribution of cue points can be avoided. In traditional methods, cue points may be randomly or evenly distributed, leading to insufficient attention to important regions (regions containing larger components) and excessive attention to secondary regions, thus affecting segmentation accuracy. By adding cue points according to region proportions, the model's ability to identify and segment key regions is improved, avoiding excessive resource waste on secondary regions. This not only improves the accuracy of region segmentation but also saves computational resources.

[0137] When each view is input into a 2D segmentation model, and the 2D segmentation model segments the view to obtain the region included in each view, in order to enable the 2D segmentation model to segment as many components as possible, each view is input into the 2D segmentation model separately, and the region included in each view is obtained by the 2D segmentation model outputting the region included in each view under unrestricted conditions.

[0138] The preceding text has provided a detailed example of how to perform segmentation to obtain the first segmentation result. It can be understood that the first segmentation result has already segmented each 3D point in the 3D point cloud into its respective component, and therefore the first segmentation result can be used as the final segmentation result.

[0139] In other embodiments, the first segmentation result may not be used as the final segmentation result. Instead, further processing may be applied to the first segmentation result. This processing includes, but is not limited to, outlier removal and segmentation of resultless points. The following will provide illustrative examples of three processing methods: processing including only outlier removal, processing including only segmentation of resultless points, and processing including both outlier removal and segmentation of resultless points.

[0140] First, see Figure 3a , Figure 3a The diagram shown illustrates the process when only outlier removal is included, comprising:

[0141] S101, Obtain views of the target object from multiple different perspectives.

[0142] S102, input each view of the target object into the first 2D multimodal large model to obtain all the components contained in the target object.

[0143] S103, use a 2D segmentation model to segment each view to obtain the area where each component is located in each view.

[0144] S104: For each 3D point in the 3D point cloud of the target object, count the components to which the projection points of the 3D points in each view belong, and obtain the projection results that reflect the 3D points.

[0145] S105, based on each projection result, divide each 3D point into its respective most hit component to obtain the first segmentation result of the 3D point cloud.

[0146] S106, remove outliers from the first segmentation result to obtain the second segmentation result.

[0147] Outliers are defined as 3D points whose distance from other 3D points segmented into the same component is greater than a preset interval threshold. For example, suppose that in the first segmentation result, a total of 100 3D points are segmented into component A, denoted as 3D points 1 to 100. If the distance between 3D points 1 to 99 is not greater than the preset interval threshold, while the distance between 3D point 100 and 3D points 1 to 99 is greater than the preset interval threshold, then 3D point 100 is an outlier.

[0148] In this document, a 3D point whose distance from other 3D points segmented into the same component is greater than a preset interval threshold can refer to either a 3D point whose distance from all other 3D points segmented into the same component is greater than the preset interval threshold, or a 3D point whose distance from a portion of the 3D points segmented into the same component is greater than the preset interval threshold. This portion should account for at least 50% of the total number of 3D points segmented into the same component.

[0149] It's understandable that components in reality are usually continuous. Therefore, if a 3D point is an outlier, it means that the distance between that 3D point and other 3D points of the same component is too far, meaning that the component is spatially discontinuous, which is obviously unreasonable. Therefore, it can be deduced that these outliers should not belong to the component and should be removed.

[0150] Please see again. Figure 3b , Figure 3b The diagram shown illustrates the process for splitting only the data without result points, including:

[0151] S101, Obtain views of the target object from multiple different perspectives.

[0152] S102, input each view of the target object into the first 2D multimodal large model to obtain all the components contained in the target object.

[0153] S103, use a 2D segmentation model to segment each view to obtain the area where each component is located in each view.

[0154] S104: For each 3D point in the 3D point cloud of the target object, count the components to which the projection points of the 3D points in each view belong, and obtain the projection results that reflect the 3D points.

[0155] S105, based on each projection result, divide each 3D point into its respective most hit component to obtain the first segmentation result of the 3D point cloud.

[0156] S1071, for each unresulted 3D point in the first segmentation result, determine the nearest 3D point with results to the unresulted 3D point, and segment the unresulted 3D point to the component to which the determined 3D point with results belongs.

[0157] Here, "no result 3D point" refers to a 3D point that has not been segmented into any component, while "result 3D point" refers to a 3D point that has been segmented into any component. For details on how 3D points are segmented into components, please refer to the aforementioned explanations regarding S101 to S105, which will not be repeated here.

[0158] This section only explains how 3D points without results are generated:

[0159] It is understandable that both the view of the target object and the 3D point cloud of the target object have a certain degree of inaccuracy. Therefore, some 3D points in the 3D point cloud have projections outside the target object in all views, that is, not within the area of ​​any component, making it impossible to segment these 3D points to any component according to the aforementioned S101 to S105.

[0160] Furthermore, for some 3D points located in the intersection area of ​​two components, their projection in some views (hereinafter referred to as the first view) is located in one component (hereinafter referred to as component B), while in other views (hereinafter referred to as the second view) their projection is located in another component (hereinafter referred to as component C). If the number of the first view and the second view are equal, then the number of components B and components C in the projection result of the 3D point is equal. That is, there is no component that appears most frequently in the projection result, which makes it impossible to divide these 3D points into arbitrary components according to the aforementioned S101 to S105.

[0161] Of course, there are other situations that prevent these 3D points from being divided into arbitrary components according to the aforementioned S101 to S105. The above two situations are just examples and do not limit how the unrestrained 3D points in this application are specifically generated.

[0162] By applying the above embodiments, the unsuccessful 3D points in the segmentation results can be further segmented, thereby improving the accuracy of the segmentation results.

[0163] Please see again. Figure 3c , Figure 3c The diagram shows a flowchart illustrating the process of simultaneously including out-of-population removal and no-result point segmentation, including:

[0164] S101, Obtain views of the target object from multiple different perspectives.

[0165] S102, input each view of the target object into the first 2D multimodal large model to obtain all the components contained in the target object.

[0166] S103, use a 2D segmentation model to segment each view to obtain the area where each component is located in each view.

[0167] S104: For each 3D point in the 3D point cloud of the target object, count the components to which the projection points of the 3D points in each view belong, and obtain the projection results that reflect the 3D points.

[0168] S105, based on each projection result, divide each 3D point into its respective most hit component to obtain the first segmentation result of the 3D point cloud.

[0169] S106, remove outliers from the first segmentation result to obtain the second segmentation result.

[0170] S1072, for each unresulted 3D point in the second segmentation result, determine the nearest 3D point with results to the unresulted 3D point, and segment the unresulted 3D point to the component to which the determined 3D point with results belongs.

[0171] S101 to S106 have been explained above. S1072 is similar to S1701, except that the object is replaced by each unresulted 3D point in the first segmentation result with each unresulted 3D point in the second segmentation result.

[0172] Please watch it again below. Figure 1 , Figure 1 The diagram illustrates the 3D point cloud segmentation method provided in this application from the perspective of the steps included in the process. To more clearly explain the 3D point cloud segmentation method provided in this application, an illustrative explanation will be given below from the perspective of data changes. (See also...) Figure 4 :

[0173] Figure 4 The example shown includes four stages:

[0174] In the first stage, the six views of the target object (front view, top view, left view, right view, bottom view, and rear view) are input into the first 2D multimodal large model, which then outputs the names of the objects and components included in these views. This corresponds to the aforementioned S101 and S102.

[0175] In the second stage, the six views of the target object are input into the 2D segmentation model for the first time. The 2D segmentation model outputs the regions included in each view under unrestricted conditions, and then determines the positions of the added cue points in each region in a way that minimizes the cue point loss. The number of cue points added in each region is proportional to the proportion of the region in its respective view. For cue point loss, please refer to the aforementioned formula (3), which will not be repeated here. Since the process of determining the positions of the added cue points in each region can be regarded as adding cue points in each view, the relevant algorithm for implementing this process can be called the cue point annotation algorithm. Stage two is equivalent to the aforementioned S10312 and S10313.

[0176] In the third stage, according to the determined locations of the added cue points, cue points are added to each region of each view. These are then input into the second 2D multimodal large model to obtain the component to which each cue point belongs, output by the second 2D multimodal large language model. Then, each view is input into the 2D segmentation model a second time, and segmentation is performed using each cue point and its corresponding component as constraints, resulting in the region where each component is located, output by the 2D segmentation model. Stage three is equivalent to the aforementioned S1032 and S1033.

[0177] In the fourth stage, the 3D point cloud is segmented by projecting 3D points onto the 3D point cloud and identifying the components to which the projected area belongs, thus obtaining the segmentation result of the 3D point cloud. Stage four is equivalent to the aforementioned S104 and S105.

[0178] Corresponding to the aforementioned 3D point cloud segmentation method, this application also provides a robot control method, see [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating a robot control method provided in an embodiment of the present invention. The method includes:

[0179] S501 uses the robot's image sensor to acquire views of the target object from multiple different perspectives, and uses the robot's laser sensor to acquire the 3D point cloud of the target object.

[0180] S502, based on the view and the 3D point cloud, execute any of the aforementioned 3D point cloud segmentation methods to obtain the first segmentation result of the 3D point cloud.

[0181] S503, based on the first segmentation result, control the robot to interact with the target object.

[0182] For details on step S502, please refer to the aforementioned explanation of the 3D point cloud segmentation method; it will not be repeated here. Steps S501 and S503 will be explained below:

[0183] In step S501, the robot can be any robot capable of image acquisition and point cloud acquisition, and able to interact with target objects, such as industrial robotic arms, automated sorting robots, and warehouse handling robots. The robot's image sensor can be an image acquisition device such as an RGB camera, a fisheye camera, or an industrial camera, and the laser sensor can be an acquisition device capable of acquiring depth information such as a LiDAR, a structured light scanner, or a binocular camera.

[0184] In step S503, based on the first segmentation result, the robot is controlled to interact with the target object. Specifically, for an industrial robotic arm robot, it can accurately grasp specific parts of the target object based on the segmentation result to complete a high-precision assembly task. An automated sorting robot can quickly and accurately classify target objects and place them in designated areas based on the segmentation information. A warehouse handling robot can rationally plan the handling path based on the segmentation result, safely and efficiently handle target objects, avoid collisions with other objects, and achieve intelligent warehouse operations.

[0185] By applying the above embodiments, and by performing a 3D point cloud segmentation method based on the collected views of the target object from different perspectives and the 3D point cloud, the robot can automatically complete the segmentation operation of the target object without the need for complex manual segmentation work. This reduces the error and time cost of manual segmentation operations. By controlling the robot to interact with the target object based on the first segmentation result, the robot can perform operation tasks more accurately and reduce repetitive operations caused by inaccurate segmentation operations. Therefore, the accuracy and efficiency of the robot's interaction with the target object are improved.

[0186] Corresponding to the aforementioned 3D point cloud segmentation method, this application also provides a 3D point cloud segmentation apparatus, such as... Figure 6 As shown, it includes:

[0187] The acquisition module 601 is used to acquire views of the target object from multiple different perspectives;

[0188] Input module 602 is used to input each of the views of the target object into the first 2D multimodal large model to obtain all the components contained in the target object;

[0189] The segmentation module 603 is used to segment each of the views using a 2D segmentation model to obtain the area where each component is located in each of the views;

[0190] The statistics module 604 is used to determine the component to which the projection point of the three-dimensional point in each view belongs for each three-dimensional point in the three-dimensional point cloud of the target object, and to obtain the projection result reflecting the three-dimensional point.

[0191] The summary module 605 is used to divide each of the three-dimensional points into their respective most hit components according to each of the projection results, so as to obtain the first segmentation result of the three-dimensional point cloud, wherein the most hit component is the component that appears most frequently in the projection results of the three-dimensional points.

[0192] By applying the above embodiments, and acquiring views of the target object from multiple different perspectives, the object's appearance features can be comprehensively obtained. Different perspectives provide complementary information, avoiding problems such as occlusion and information loss that may occur with a single perspective. Utilizing the feature extraction and semantic understanding capabilities of a multimodal large model, each view is input into a first 2D multimodal large model. This model can fuse information from different modalities (such as visual features of the image, possible text descriptions, etc.) to more accurately identify each component of the object, reducing false positives and false negatives. A 2D segmentation model is used to segment each view, obtaining the region where each component is located in each view, thus obtaining the positional information of each component on the two-dimensional plane, serving as an important basis for subsequently associating 3D points with components. By statistically analyzing the components to which 3D points belong at their projection points in each view, information from multiple perspectives is comprehensively considered, avoiding the bias that may arise from relying solely on a single perspective. By acquiring the components to which each 3D point belongs at its projection points in different views, the component most likely to belong to each 3D point can be quickly and accurately determined, i.e., the component with the most hits, ultimately obtaining the first segmentation result, thereby improving the accuracy of 3D point cloud segmentation.

[0193] In one possible embodiment, the segmentation module includes:

[0194] The first segmentation submodule is used to add prompt points to each of the views respectively;

[0195] The second segmentation submodule is used to determine the component to which each of the prompt points belongs using the second 2D multimodal large language model;

[0196] The third segmentation submodule is used to input each of the prompt points, the component to which each prompt point belongs, and each of the views into the 2D segmentation model to obtain the area where each component is located in each of the views.

[0197] In one possible embodiment, the first segmentation submodule includes:

[0198] The first segmentation unit is used to add cue points to each of the views in a manner that minimizes the loss of cue points in the views.

[0199] The cue point loss includes minimum distance loss and / or uniformity loss; the minimum distance loss is positively correlated with the distance of each pixel in the view to the nearest cue point; the uniformity loss is negatively correlated with the spacing between cue points.

[0200] In one possible embodiment, the first segmentation submodule includes:

[0201] The second segmentation unit is used to input each of the views into the 2D segmentation model to obtain the regions included by each of the views.

[0202] The third segmentation unit is used to add prompts to each of the views according to the proportion of each region in the view, wherein the number of prompts added to each region is positively correlated with the proportion of that region in the view;

[0203] The second segmentation unit includes:

[0204] The first segmentation subunit is used to input each of the views into the 2D segmentation model to obtain the regions included by each of the views in the 2D segmentation model under unrestricted conditions.

[0205] In one possible embodiment, the device further includes: a rejection module, used to reject outliers in the first segmentation result to obtain a second segmentation result, wherein the outlier is a 3D point whose distance from other 3D points segmented to the same component is greater than a preset distance threshold.

[0206] In one possible embodiment, the apparatus further includes: a segmentation module, configured to, for each unresulted 3D point in the second segmentation result, determine the nearest achievable 3D point to the unresulted 3D point, and segment the unresulted 3D point to the component to which the determined achievable 3D point belongs, wherein an unresulted 3D point is a 3D point that has not been segmented to any component, and an achievable 3D point is a 3D point that has been segmented to any component.

[0207] Corresponding to the aforementioned robot control method, this application also provides a robot control device, such as... Figure 7 As shown, it includes:

[0208] The acquisition module 701 is used to acquire views of the target object from multiple different perspectives using the robot's image sensor, and to acquire the three-dimensional point cloud of the target object using the robot's laser sensor.

[0209] Execution module 702 is used to execute any of the aforementioned three-dimensional point cloud segmentation methods based on the view and the three-dimensional point cloud to obtain a first segmentation result of the three-dimensional point cloud;

[0210] The interaction module 703 is used to control the robot to interact with the target object based on the first segmentation result.

[0211] By applying the above embodiments, and by performing a 3D point cloud segmentation method based on the collected views of the target object from different perspectives and the 3D point cloud, the robot can automatically complete the segmentation operation of the target object without the need for complex manual segmentation work. This reduces the error and time cost of manual segmentation operations. By controlling the robot to interact with the target object based on the first segmentation result, the robot can perform operation tasks more accurately and reduce repetitive operations caused by inaccurate segmentation operations. Therefore, the accuracy and efficiency of the robot's interaction with the target object are improved.

[0212] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0213] Memory 803 is used to store computer programs;

[0214] When processor 801 executes a program stored in memory 803, it performs the following steps:

[0215] Obtain views of the target object from multiple different perspectives;

[0216] Input each of the views of the target object into the first two-dimensional multimodal large model to obtain all the components contained in the target object;

[0217] The views are segmented using a two-dimensional segmentation model to obtain the regions where each component is located in each view;

[0218] For each three-dimensional point in the three-dimensional point cloud of the target object, determine the component to which the region where the projection point of the three-dimensional point is located in each of the views belongs, and obtain the projection result that reflects the three-dimensional point;

[0219] Based on each projection result, each three-dimensional point is divided into its most frequently hit component to obtain the first segmentation result of the three-dimensional point cloud, wherein the most frequently hit component is the component that appears most often in the projection result of the three-dimensional point.

[0220] In one possible embodiment, the step of segmenting each of the views using a two-dimensional segmentation model to obtain the region where each component is located in each of the views includes:

[0221] Add tooltips to each of the aforementioned views;

[0222] The component to which each of the aforementioned prompts belongs is determined using a second two-dimensional multimodal large language model;

[0223] Each of the aforementioned prompt points, the component to which each prompt point belongs, and each of the aforementioned views are input into a two-dimensional segmentation model to obtain the region where each of the aforementioned components is located in each of the aforementioned views.

[0224] In one possible embodiment, adding the cue points to each of the views includes:

[0225] For each of the aforementioned views, cue points are added to the views in a manner that minimizes the cue point loss of the views; wherein, the cue point loss includes minimum distance loss and / or uniformity loss; the minimum distance loss is positively correlated with the distance of each pixel in the view to the nearest cue point; the uniformity loss is negatively correlated with the spacing between the cue points;

[0226] And / or,

[0227] For each of the views, cue points are added to the views in a manner that minimizes the cue point loss of the views; wherein, the cue point loss includes minimum distance loss and / or uniformity loss; the minimum distance loss is positively correlated with the distance of each pixel in the view to the nearest cue point; the uniformity loss is negatively correlated with the spacing between the cue points.

[0228] In one possible embodiment, inputting each of the views into a two-dimensional segmentation model to obtain the region included in each of the views includes:

[0229] Each of the views is input into the two-dimensional segmentation model to obtain the region included by each of the views in the two-dimensional segmentation model under unrestricted conditions.

[0230] In one possible embodiment, the method further includes:

[0231] Outliers are removed from the first segmentation result to obtain the second segmentation result, wherein outliers are three-dimensional points whose distance from other three-dimensional points segmented into the same component is greater than a preset distance threshold.

[0232] In one possible embodiment, the method further includes:

[0233] For each unsuccessful 3D point in the second segmentation result, determine the nearest successful 3D point to the unsuccessful 3D point, and segment the unsuccessful 3D point to the component to which the determined successful 3D point belongs. Here, an unsuccessful 3D point is a 3D point that has not been segmented to any component, and a successful 3D point is a 3D point that has been segmented to any component.

[0234] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0235] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0236] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0237] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0238] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described three-dimensional point cloud segmentation methods or robot control methods.

[0239] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the three-dimensional point cloud segmentation methods or robot control methods described above.

[0240] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0241] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0242] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and electronic device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0243] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method of segmenting a three-dimensional point cloud, the method comprising: The method comprises: obtaining views of a target object under multiple different viewing angles; inputting each of the views of the target object into a first two-dimensional multi-modal large model to obtain all components contained in the target object; segmenting each of the views using a two-dimensional segmentation model to obtain regions in which each of the components is located in each of the views; for each three-dimensional point in a three-dimensional point cloud of the target object, determining a component to which a region in which a projection point of the three-dimensional point in each of the views is located belongs, to obtain a projection result reflecting the projection of the three-dimensional point; according to each of the projection results, dividing each of the three-dimensional points into a respective most-hit component to obtain a first segmentation result of the three-dimensional point cloud; wherein the most-hit component is a component that appears most frequently in the projection result of the three-dimensional point.

2. The method of claim 1, wherein, The segmentation of each of the views using a two-dimensional segmentation model to obtain the regions in which each of the components is located in each of the views comprises: adding a prompt point in each of the views; determining a component to which each of the prompt points belongs using a second two-dimensional multi-modal large language model; inputting each of the prompt points, the component to which each of the prompt points belongs, and each of the views into a two-dimensional segmentation model to obtain the regions in which each of the components is located in each of the views.

3. The method of claim 2, wherein, The adding of a prompt point in each of the views comprises: for each of the views, adding a prompt point in the view in a manner that minimizes the prompt point loss of the view; wherein the prompt point loss comprises a minimum distance loss and / or a uniformity loss; the minimum distance loss is positively correlated with the distance between each pixel point in the view and the nearest prompt point; and the uniformity loss is negatively correlated with the distance between each prompt point; and / or, inputting each of the views into a two-dimensional segmentation model to obtain regions included in each of the views; for each of the views, adding a prompt point in the view according to the proportion of each region in the view, wherein the number of prompt points added in each region is positively correlated with the proportion of the region in the view.

4. The method of claim 3, wherein, The inputting of each of the views into a two-dimensional segmentation model to obtain the regions included in each of the views comprises: inputting each of the views into a two-dimensional segmentation model to obtain the regions included in each of the views output by the two-dimensional segmentation model without restrictions.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: removing outliers in the first segmentation result to obtain a second segmentation result, wherein the outliers are three-dimensional points having a distance greater than a preset distance threshold from other three-dimensional points segmented into the same component.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: for each no-result three-dimensional point in the second segmentation result, determining a result three-dimensional point closest to the no-result three-dimensional point, and segmenting the no-result three-dimensional point into a component to which the determined result three-dimensional point belongs, wherein a no-result three-dimensional point is a three-dimensional point that is not segmented into any component, and a result three-dimensional point is a three-dimensional point that is segmented into any component.

7. A robot control method characterized by, The method comprises: obtaining views of a target object under multiple different viewing angles using an image sensor of a robot, and obtaining a three-dimensional point cloud of the target object using a laser sensor of the robot; According to the views and the three-dimensional point cloud, a three-dimensional point cloud segmentation method according to any one of claims 1 to 6 is performed to obtain a first segmentation result of the three-dimensional point cloud; According to the first segmentation result, the robot is controlled to interact with the target object.

8. A three-dimensional point cloud segmentation apparatus, characterized by, The device comprises: An acquisition module is configured to acquire views of a target object from different perspectives; An input module is configured to input each view of the target object into a first two-dimensional multi-modal large model to obtain all components contained in the target object; A segmentation module is configured to segment each view using a two-dimensional segmentation model to obtain regions in which each component is located in each view; A statistics module is configured to determine, for each three-dimensional point in a three-dimensional point cloud of the target object, a component to which a region in which a projection point of the three-dimensional point in each view belongs, to obtain a projection result reflecting the three-dimensional point; A summary module is configured to divide each three-dimensional point to a most hit component according to each projection result, to obtain a first segmentation result of the three-dimensional point cloud; wherein the most hit component is a component that appears most frequently in the projection result of the three-dimensional point.

9. A robot control device characterized by comprising: The device comprises: An acquisition module is configured to acquire views of a target object from different perspectives using an image sensor of a robot, and to acquire a three-dimensional point cloud of the target object using a laser sensor of the robot; An execution module is configured to perform a three-dimensional point cloud segmentation method according to any one of claims 1 to 6 according to the views and the three-dimensional point cloud, to obtain a first segmentation result of the three-dimensional point cloud; An interaction module is configured to control the robot to interact with the target object according to the first segmentation result.

10. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored on the memory to implement the method steps of any one of claims 1 to 6, or the method steps of claim 7.

Images