A grasp pose estimation method and related devices

Abstract

The embodiment of the application discloses a kind of grasping pose estimation method and related equipment, it is characterized in that, the method includes: based on the M sample points in the object point cloud of target object, determine the M sample point clouds of the object point cloud, each sample point cloud in the M sample point clouds is the point cloud of the object point cloud in corresponding target space, the corresponding target space is the spherical space with the M sample points in corresponding sample point as ball center and with preset length as radius;Wherein, the number of points included in each sample point cloud is greater than preset value;Based on the M sample point clouds and grasping end model parameter, M grasping poses are calculated, and the grasping end model parameter includes the width and depth of grasping end model;Select one grasping pose from the M grasping poses as the target grasping pose for grasping the target object.Using the embodiment of the application can improve the robot grasping efficiency, accuracy in complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision grasping, and in particular to a grasping pose estimation method and related equipment. BACKGROUND

[0002] The grasping technology of robots has a wide application prospect in industries such as industry, agriculture, logistics transportation, and service, and has a huge market prospect. However, for various complex scenes such as objects of unlimited categories, stacking, and tilting, there is still a lack of relatively efficient and accurate grasping pose estimation methods at present, and more complex automated equipment is usually required in actual scenes, which often means larger floor space and higher introduction cost. At present, some enterprises still choose to involve manual labor. Therefore, improving the efficiency and accuracy of robot grasping in complex scenes to replace manual labor is particularly important for enterprises to reduce costs and increase efficiency. SUMMARY

[0003] The technical problem to be solved by the embodiments of the present application is to provide a grasping pose estimation method and related equipment to improve the efficiency and accuracy of robot grasping in complex scenes.

[0004] In a first aspect, the embodiments of the present application provide a grasping pose estimation method, characterized in that the method comprises: determining M sample point clouds of an object point cloud of a target object based on M sample points in the object point cloud, each sample point cloud in the M sample point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius; wherein the number of points included in each sample point cloud is greater than a preset value; M is an integer greater than 1; calculating M grasping poses based on the M sample point clouds and grasping end model parameters, the grasping end model parameters including the width and depth of a grasping end model; and selecting one grasping pose from the M grasping poses as a target grasping pose for grasping the target object.

[0005] It is a common method to detect the object grasping pose quickly and stably by using visual information, and then the robot (such as a two-fingered gripper) operates the object according to the grasping pose. The traditional robot grasping method is to establish an object grasping pose dataset, and then match based on the template. The size, precision of the model library and computer resources directly affect the grasping pose estimation. In unstructured environments such as home environments, there are various types of objects, different shapes and irregular postures. The traditional method lacks good generalization performance. The skilled person in the art obtains the feature information of the target object based on the RGB image, and then generates the grasping pose based on the feature information. However, it is found in actual use that since the grasping pose is generated based on only the planar features (i.e. two-dimensional data features) of the target object, the object stacking, object tilting and other features are ignored, thereby reducing the accuracy of the robot grasping and failing to adapt to complex scenes such as object overlap and irregular objects.

[0006] In the embodiment of the present application, the grasping pose is estimated based on the three-dimensional object point cloud data of the target object and the grasping end model parameters, and the three-dimensional features of the target object are considered, thereby improving the grasping efficiency and accuracy of the robot in complex scenes. Specifically, since the target object may be stacked and irregular, the target grasping pose cannot be directly calculated, so M sampling points are first taken on the object point cloud of the target object, and the number of point clouds in the spherical space with each sampling point as the center and a predetermined length as the radius needs to be greater than a predetermined value, so as to determine the points close to the center of the target object as the sampling points, thereby avoiding the problem of low efficiency of candidate poses caused by random sampling. Further, M target object sampling point clouds (i.e. point clouds in the spherical space with each sampling point as the center and a predetermined length as the radius) are obtained according to the M sampling points, and then the optimal grasping pose for grasping each sampling point cloud can be generated based on the three-dimensional features of each sampling point cloud and the grasping end model parameters, thereby obtaining M candidate grasping poses. Next, the overall target object can be combined to select one from the multiple candidate poses as the grasping pose for grasping the target object, thereby improving the grasping efficiency and accuracy of the robot in complex scenes.

[0007] In one possible implementation, the method further comprises: obtaining the object point cloud from the color map and the depth map of the target object captured by the camera, or obtaining the object point cloud by scanning the target object by a laser radar.

[0008] In the embodiments of the present application, the RGB image and the depth image of the target object can be obtained through the camera, and then the object point cloud (i.e., the three-dimensional data of the target object) of the target object can be obtained by analyzing and processing the RGB image and the depth image. Alternatively, the object point cloud can be directly obtained by scanning the target object through the laser radar. Further, the estimation of the grasping pose can be performed based on the three-dimensional data of the target object, thereby improving the efficiency and accuracy of the robot grasping in a complex scene.

[0009] In a possible implementation, the method further includes: preprocessing the object point cloud to obtain N sampling points, N being an integer greater than M; obtaining N sampling point clouds of the object point cloud based on the N sampling points, each sampling point cloud in the N sampling point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sampling point in the N sampling points as the center and a preset length as the radius; determining whether the number of points included in each sampling point cloud in the N sampling point clouds is greater than a preset value; and determining the sampling points corresponding to the M sampling point clouds greater than the preset value as the M sampling points.

[0010] In the embodiments of the present application, N sampling points can be randomly selected from the object point cloud of the target object, and then N spherical spaces can be obtained with each sampling point as the center and a preset length as the radius. Further, the number of points included in each spherical space is determined, and if the number of points is greater than a preset value, the sampling point corresponding to the spherical space can be determined as a sampling point, so as to eliminate the sampling points close to the edge of the object from the multiple sampling points, thereby improving the efficiency of the grasping pose.

[0011] In a possible implementation, the M grasping poses are calculated based on the M sampling point clouds and the grasping end model parameters, including: obtaining a surface normal of each sampling point cloud based on the normal vector of each point in each sampling point cloud in the M sampling point clouds; and calculating the M grasping poses based on the surface normal of each sampling point cloud in the M sampling point clouds and the grasping end model parameters.

[0012] In the embodiments of the present application, for each sampling point cloud, the multiple points included in each sampling point cloud constitute a local three-dimensional graph of the target object. Since the target object can be of any shape, the local three-dimensional graph of the target object is generally composed of a curved surface. Further, the surface normal of the sampling point cloud can be calculated based on the normal vector of each point in the sampling point cloud. Since the surface normal is perpendicular to the tangent plane of the curved surface, the surface normal can be taken as the coordinate axis vector of the grasping direction of the grasping end coordinate system (it can be understood that the direction of the surface normal determines the direction of the falling of the grasping). Then, the grasping pose can be directly calculated based on the surface normal and the parameters of the grasping end model.

[0013] In a possible implementation, the M grasp poses are calculated based on the surface normal of each of the M sampled point clouds and the grasp end model parameter, including: calculating a grasp center point of each of the M sampled point clouds based on the surface normal of each of the M sampled point clouds and the depth of the grasp end model; the grasp center point of each of the M sampled point clouds being on the surface normal of the corresponding sampled point cloud; obtaining M initial grasp poses based on the grasp center point of each of the M sampled point clouds and the width of the grasp end model; and performing translation and / or rotation on the M initial grasp poses respectively to obtain the M grasp poses.

[0014] In the embodiment of the present application, for each sampled point cloud, the direction of the grasp end falling can be determined based on the surface normal of the sampled point cloud. Further, the sampled point is moved in the direction of the surface normal by a distance of a grasp end depth based on the depth of the grasp end model, and is taken as a grasp center point (i.e., the grasp end of the gripper), so as to avoid collision between the grasp end and the sampled point cloud. Next, the initial grasp pose can be determined based on the grasp center point and the width of the grasp end. Finally, the initial grasp pose is translated and / or rotated to obtain the optimal grasp pose for grasping the corresponding sampled point cloud. Through the method provided in the embodiment of the present application, the grasp pose is estimated based on the three-dimensional object point cloud data of the target object and the grasp end model parameter, and the three-dimensional features of the target object are considered, thereby improving the grasping efficiency and accuracy of the robot in a complex scene.

[0015] In a possible implementation, the method further includes: performing collision detection on the M grasp poses based on the object point cloud of the target object; and determining L grasp poses that meet a preset collision detection result from the M grasp poses as L candidate grasp poses, L being an integer greater than 0.

[0016] In the embodiment of the present application, each of the M grasp poses is only the optimal grasp pose for grasping the corresponding sampled point cloud (i.e., the local part of the target object), but there may be a grasp pose in the M grasp poses that has a collision relationship with the local part of the target object and / or the desktop when grasping the entire target object. Therefore, collision detection can be performed on the M grasp poses based on the object point cloud of the target object, so as to eliminate the grasp pose that has a collision relationship with the target object or the desktop, thereby improving the efficiency of the candidate grasp pose.

[0017] In a possible implementation, the selecting one of the M grasping poses as a target grasping pose for grasping the target object comprises: obtaining L local point clouds based on the L candidate grasping poses, each of the L local point clouds being a point cloud of a portion of the object point cloud intersected by a corresponding candidate grasping pose; obtaining pose evaluation information based on the L local point clouds and the object point cloud; the pose evaluation information comprising an evaluation score corresponding to each of the L local point clouds, each evaluation score corresponding to a local point cloud being obtained based on feature information of the local point cloud and feature information of the object point cloud; and determining the candidate grasping pose corresponding to the local point cloud with the highest evaluation score as the target grasping pose.

[0018] In the embodiment of the application, a corresponding local point cloud is obtained based on each candidate grasping pose, and then each local point cloud is scored in combination with feature information of the local point cloud and feature information of the object point cloud. In this process, the candidate grasping pose can be scored in combination with factors such as the relationship of the grasping position relative to the grasped object and the relationship of the grasping position relative to all objects, and then the candidate grasping pose with the highest evaluation score is determined as the optimal grasping pose for grasping the target object. Therefore, the method provided in the application can adapt to more diverse object grasping (such as selecting a grasping pose closest to the center of a long strip-shaped object as a more optimal sample) and stacked object grasping (such as selecting a grasping pose on the outermost layer of a stacked object as a more optimal sample), and adapt to more complex scenarios.

[0019] In a second aspect, the application provides a grasping pose estimation device, which is characterized in that the device comprises: a first processing unit configured to determine M sample point clouds of an object point cloud of a target object based on M sample points in the object point cloud, each of the M sample point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius; wherein the number of points included in each of the sample point clouds is greater than a preset value; and M is an integer greater than 1; a second processing unit configured to calculate M grasping poses based on the M sample point clouds and grasping end model parameters, the grasping end model parameters comprising a width and a depth of a grasping end model; and a third processing unit configured to select one of the M grasping poses as a target grasping pose for grasping the target object.

[0020] In a possible implementation, the device further comprises a fourth processing unit configured to obtain the object point cloud from a color map and a depth map of the target object captured by a camera, or obtain the object point cloud by scanning the target object using a laser radar.

[0021] In a possible implementation, the device further includes a fifth processing unit, configured to: pre-process the object point cloud to obtain N to-be-sampled points, N being an integer greater than M; and obtain N to-be-sampled point clouds of the object point cloud based on the N to-be-sampled points, each to-be-sampled point cloud in the N to-be-sampled point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding to-be-sampled point in the N to-be-sampled points as a spherical center and a preset length as a radius; determine whether a number of points included in each to-be-sampled point cloud in the N to-be-sampled point clouds is greater than a preset value; and determine to-be-sampled points corresponding to M to-be-sampled point clouds greater than the preset value as the M sampling points.

[0022] In a possible implementation, the second processing unit is specifically configured to: obtain a surface normal of each of the M sampling point clouds based on a normal vector of each point in each of the M sampling point clouds; and calculate the M grasping poses based on the surface normal of each of the M sampling point clouds and the grasping end model parameters.

[0023] In a possible implementation, the second processing unit is specifically configured to: calculate a grasping center point of each of the M sampling point clouds based on the surface normal of each of the M sampling point clouds and a depth of the grasping end model, the grasping center point of each of the M sampling point clouds being on the surface normal of the sampling point cloud corresponding to the sampling point corresponding to the grasping center point; obtain M initial grasping poses based on the grasping center point of each of the M sampling point clouds and a width of the grasping end model; and respectively perform translation and / or rotation on the M initial grasping poses to obtain the M grasping poses.

[0024] In a possible implementation, the device further includes a sixth processing unit, configured to: perform collision detection on the M grasping poses based on the object point cloud of the target object; and determine L grasping poses satisfying a preset collision detection result in the M grasping poses as L candidate grasping poses, L being an integer greater than 0.

[0025] In a possible implementation, the third processing unit is specifically configured to: obtain L local point clouds based on the L candidate grasping poses, each local point cloud in the L local point clouds being a point cloud of an intersection part of a corresponding candidate grasping pose and the object point cloud; obtain pose evaluation information based on the L local point clouds and the object point cloud, the pose evaluation information including an evaluation score corresponding to each local point cloud in the L local point clouds, and the evaluation score corresponding to each local point cloud being obtained based on feature information of the L local point clouds and feature information of the object point cloud; and determine a candidate grasping pose corresponding to a local point cloud with a highest evaluation score as the target grasping pose.

[0026] In a third aspect, a computer storage medium is provided. The computer storage medium stores a computer program which, when executed by a processor, implements the method of any one of the first aspect.

[0027] In a fourth aspect, an electronic device is provided. The electronic device includes a processor configured to support the electronic device to implement the corresponding functions in the method of the first aspect. The electronic device can further include a memory coupled with the processor, which stores necessary program instructions and data for the electronic device. The electronic device can further include a communication interface for the electronic device to communicate with other devices or communication networks.

[0028] In a fifth aspect, a chip system is provided. The chip system includes a processor configured to support the electronic device to implement the functions involved in the first aspect, such as generating or processing the information involved in the method of the first aspect. In a possible design, the chip system further includes a memory configured to store necessary program instructions and data for the electronic device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0029] In a sixth aspect, a computer program is provided. The computer program includes instructions which, when executed by a computer, cause the computer to perform the method of any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1A A system architecture diagram of a grasp pose estimation method provided by an embodiment of the present application.

[0031] Figure 1A An automatic grasping diagram provided by an embodiment of the present application.

[0032] Figure 2 A flow diagram of a grasp pose estimation method in an embodiment of the present application.

[0033] Figure 3 A diagram of an object point cloud and a sampling point cloud provided by an embodiment of the present application.

[0034] Figure 4 A diagram of obtaining an object point cloud based on a scene point cloud provided by an embodiment of the present application.

[0035] Figure 5 A diagram of grasp pose generation provided by an embodiment of the present application.

[0036] Figure 6An initial grasping pose translation schematic diagram provided for the embodiment of the present application.

[0037] Figure 7 An initial grasping pose rotation schematic diagram provided for the embodiment of the present application.

[0038] Figure 8 Another grasping pose generation schematic diagram provided for the embodiment of the present application.

[0039] Figure 9 A translation center and rotation center schematic diagram provided for the embodiment of the present application.

[0040] Figure 10 A candidate grasping pose generation schematic diagram provided for the embodiment of the present application.

[0041] Figure 11 An optimal grasping pose schematic diagram provided for the embodiment of the present application.

[0042] Figure 12 A grasping pose evaluation model network structure schematic diagram provided for the embodiment of the present application.

[0043] Figure 13 A grasping pose estimation device schematic diagram provided for the embodiment of the present application.

[0044] Figure 14 A point cloud preprocessing schematic diagram provided for the embodiment of the present application.

[0045] Figure 15 A candidate pose selection schematic diagram provided for the embodiment of the present application.

[0046] Figure 16 A grasping pose evaluation schematic diagram provided for the embodiment of the present application.

[0047] Figure 17 A structure schematic diagram of a grasping pose estimation device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0049] The terms "first", "second", "third", and "fourth" and the like in the description and in the claims of the present application and the accompanying drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. The terms "comprises", "comprising", "includes", "including" and the like are to be construed open-ended, allowing for instances where there are equivalents to processes, methods, systems, products, or devices that do not literally include the recited steps or elements, but that otherwise achieve the same result. Processes, methods, systems, products, or devices that do not literally contain the specific steps or elements but that achieve the same result are considered to be within the scope of the present application.

[0050] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that any of the embodiments described herein can be combined with any of the other embodiments unless specifically noted otherwise.

[0051] The following explains some terms used in the present application to facilitate understanding by those skilled in the art.

[0052] (1) Point cloud, a collection of point data of the appearance surface of a product obtained by a measuring instrument, also known as a point cloud. The number of points obtained by a three-dimensional coordinate measuring machine is usually small, and the distance between points is also large, which can be referred to as a sparse point cloud. The point cloud obtained by a three-dimensional laser scanner or a photographic scanner has a large number of points and is relatively dense, which can be referred to as a dense point cloud.

[0053] (2) Deep learning (DL), a kind of machine learning technology based on deep neural network algorithm, which mainly uses multiple nonlinear transformations to process and analyze data. It is mainly applied to the perception and decision-making scenarios in the field of artificial intelligence, such as image and voice recognition, natural language translation, computer game playing, etc.

[0054] (3) Grasp pose detection, estimation of the grasp configuration of the end of a grasp executor.

[0055] Based on the above-mentioned technical problems, in order to facilitate understanding of the embodiments of the application, the system architecture based on the embodiments of the application will be described first. Please refer to Figure 1A , Figure 1AA system architecture schematic diagram of a grasping pose estimation method provided in an embodiment of the present application is used to solve the problems of low efficiency and low accuracy of an automatic grasping device (such as a manipulator) in a complex scene. The system architecture can include an automatic grasping device 101, a target object 102, and a computer 103. The automatic grasping device 101 can be used to automatically grasp the target object 102; the target object 102 can be any object and is not limited in category, stacking, tilting, etc.; and the computer 103 can be used to generate a grasping pose for the automatic grasping device 101.

[0056] The detailed description is as follows:

[0057] The automatic grasping device 101 can include a grab, a manipulator, a fixed frame, a rotating shaft, a power wheel, a power arm, a moving frame, and the like. Common automatic grasping devices 101 include intelligent robots, manipulators, and the like. The components of the automatic grasping device 101 for grasping objects include but are not limited to two-fingered grippers, three-fingered grippers, vacuum cups, dexterous hands, and the like. In the embodiment of the present application, the automatic grasping device 101 can establish a connection with the computer 103, and then can receive the grasping pose generated by the computer 103 for the target object 102, so that the automatic grasping device 101 can grasp the target object 102 based on the grasping pose. For example, as shown in Figure 1B , the computer 103 can generate a grasping pose for the target object 102, and then send the grasping pose to the automatic grasping device 101, so that the automatic grasping device 101 can grasp the target object 102 based on the grasping pose. Figure 1B A schematic diagram of automatic grasping provided in an embodiment of the present application, in which 3D data of an object can be collected in real time from a grasping position, and then the object can be grasped and placed at a target position by an automatic grasping system and device (such as the automatic grasping device 101).

[0058] The computer 103 is an electronic computing machine for high-speed calculation, which can perform numerical calculation and logical calculation, and also has a storage memory function. The computer 103 is a modern intelligent electronic device capable of automatically and high-speed processing of massive data according to a program. The computer 103 can include supercomputers, industrial control computers, network computers, personal computers, embedded computers, and more advanced computers such as biological computers, photonic computers, and quantum computers. In some embodiments, the computer 103 can be a personal digital assistant (PDA) computer, a tablet computer, and a laptop computer. In the embodiment of the present application, the computer 103 can establish a connection with the automatic grasping device 101, and can generate a grasping pose for the automatic grasping device 101 by using the grasping pose estimation method provided in the present application, and then send the grasping pose to the automatic grasping device 101, so that the automatic grasping device 101 can grasp the target object 102 based on the grasping pose. The grasping pose estimation method provided in the present application will be described in detail in subsequent embodiments, and will not be described here.

[0059] It can be understood that, Figure 1A The system architecture in the above embodiment is only some exemplary implementations provided by the embodiment of the present application, and the system architecture in the embodiment of the present application includes but is not limited to the above implementations.

[0060] The specific method architecture based on which the embodiment of the present application is described below. Referring to Figure 2 , Figure 2 is a flowchart of a grasping pose estimation method in the embodiment of the present application. The grasping pose estimation method in the embodiment of the present application will be described below in combination with the accompanying Figure 2 and the system architecture in the above Figure 1A It should be noted that, in order to more specifically describe the grasping pose estimation method in the embodiment of the present application, the present application describes the corresponding execution subject as an electronic device in each flow step, but this does not mean that the corresponding method flow in the embodiment of the present application can only be performed by the described execution subject.

[0061] Step S201: The electronic device determines M sample point clouds of an object point cloud of a target object based on M sample points in the object point cloud.

[0062] Specifically, each sample point cloud in the M sample point clouds is a point cloud of the object point cloud in a corresponding target space, and the corresponding target space is a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius; wherein the number of points included in each sample point cloud is greater than a preset value; and M is an integer greater than 1. It can be understood that the target object is an object to be grasped, which can be any object and can be placed in any place (such as a plane, an inclined plane, etc.) in any pose. The object point cloud is a mass point set of the target surface characteristics obtained by data collection on the target object. Each point can include X, Y, Z geometric coordinates, intensity values, classification values, etc., and these points combined together form a point cloud. The object point cloud can more truly restore the three-dimensional effect of the target object and realize visualization. The sample point cloud is composed of part of the point set of the object point cloud, and the number of points included in each sample point cloud is greater than a preset value to improve the efficiency of grasping pose.

[0063] For example, as shown in Figure 3 , Figure 3 is a schematic diagram of an object point cloud and a sample point cloud provided by the embodiment of the present application. In the diagram, M sample points can be determined on the object point cloud of the target object, and then M spherical spaces can be obtained with each sample point as a spherical center and a preset length as a radius. Further, the point set in the spherical space is determined as a sample point cloud, thereby obtaining M sample point clouds of the target object.

[0064] In a possible implementation, a color graph (Red Green Blue, RGB) and a depth graph of the target object captured by a camera are used to obtain a scene point cloud of the target object, and the scene point cloud is preprocessed to obtain the object point cloud. Specifically, the scene point cloud is pruned, and non-object point clouds are removed by calculating a table equation to obtain the object point cloud. Optionally, the object point cloud is transformed from a point cloud coordinate system to a table coordinate system according to the table equation.

[0065] For example, as shown in FIG. 1, Figure 4 Figure 4 A schematic diagram for obtaining an object point cloud based on a scene point cloud is provided in an embodiment of the present application. First, a point cloud is generated, specifically, a scene point cloud is generated according to an RGB graph and a depth graph of a target object. Second, non-object point clouds are removed, specifically, non-object parts in the scene point cloud are removed to obtain an object point cloud. Methods that can be used but are not limited to include calculating a table equation to retain part of the point cloud on the table within a certain distance, filtering the point cloud to remove outliers, dividing the area where the object is likely to appear, and the like. Third, coordinate system transformation is performed, specifically, the point cloud and the camera coordinate are transformed to a table coordinate system (for example, the z-axis is upward, and the table is an x-y coordinate system). Methods that can be used but are not limited to include calculating a transformation matrix from the current point cloud coordinate system to the table coordinate system according to the table equation.

[0066] In a possible implementation, the method further includes: obtaining the object point cloud from a color graph and a depth graph of the target object captured by a camera, or obtaining the object point cloud by scanning the target object by using a laser radar. Specifically, an RGB graph and a depth graph of the target object can be obtained by using a camera, and then the RGB graph and the depth graph are analyzed and processed to obtain the object point cloud of the target object. Alternatively, the object point cloud can be directly obtained by scanning the target object by using a laser radar. Further, the estimation of the grasping pose can be based on the three-dimensional data of the target object, thereby improving the efficiency and accuracy of robot grasping in a complex scene.

[0067] ​In a possible implementation, the method further includes: preprocessing the object point cloud to obtain N to-be-sampled points, N being an integer greater than M; obtaining N to-be-sampled point clouds of the object point cloud based on the N to-be-sampled points, each to-be-sampled point cloud in the N to-be-sampled point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding to-be-sampled point in the N to-be-sampled points as a spherical center and a preset length as a radius; determining whether a number of points included in each to-be-sampled point cloud in the N to-be-sampled point clouds is greater than a preset value; and determining to-be-sampled points corresponding to M to-be-sampled point clouds greater than the preset value as the M sampling points. Specifically, N to-be-sampled points can be randomly selected from the object point cloud of the target object, and then N spherical spaces can be obtained with each to-be-sampled point as a spherical center and a preset length as a radius. Further, the number of points included in each spherical space is determined, and if the number of points is greater than a preset value, the to-be-sampled point corresponding to the spherical space can be determined as a sampling point, so as to eliminate to-be-sampled points close to the edge of the object from the plurality of to-be-sampled points, thereby improving the efficiency of the grasping pose. For example, a point is randomly sampled on the to-be-sampled point cloud, and other points within a sampling point radius are obtained according to a set search radius, and if there are not enough points within the radius range, the point is eliminated.

[0068] Step S202: The electronic device calculates M grasping poses based on the M to-be-sampled point clouds and grasping end model parameters.

[0069] Specifically, the grasping end model parameters include, but are not limited to, width, depth, and size information such as inner and outer diameters of the grasping end model. It can be understood that the grasping end model includes, but is not limited to, a two-fingered gripper, a three-fingered gripper, a suction cup, and the like.

[0070] In a possible implementation, the calculating the M grasping poses based on the M sampled point clouds and the grasping end model parameters comprises: obtaining a surface normal of each of the M sampled point clouds based on a normal vector of each point in each of the M sampled point clouds; and calculating the M grasping poses based on the surface normal of each of the M sampled point clouds and the grasping end model parameters. Specifically, for each sampled point cloud, a plurality of points included in the sampled point cloud form a local three-dimensional graph of the target object, and since the target object can be of any shape, the local three-dimensional graph of the target object is generally formed by a curved surface. Further, the surface normal of the sampled point cloud can be calculated based on the normal vector of each point in the sampled point cloud, and since the surface normal is perpendicular to the tangent plane of the curved surface, the surface normal can be taken as a coordinate axis vector of the grasping direction of the grasping end coordinate system (it can be understood that the direction of the surface normal determines the direction of the grasping and clamping). Then, the grasping end point of the grasping and clamping can be directly calculated based on the surface normal and the parameters of the grasping end model, and further, the grasping pose can be obtained based on the grasping end point.

[0071] For example, as shown in FIG. 2, Figure 5 Figure 5 FIG. 2 is a schematic diagram of grasping pose generation provided by an embodiment of the present application, in which M sampled point clouds are obtained based on the object point cloud of the target object, and further, the M grasping poses can be calculated based on the surface normal of each of the M sampled point clouds and the grasping end model parameters.

[0072] Optionally, the normal vector estimation is performed on each point in the sampled point cloud, and a unit direction vector of the point cloud pointing to the camera is calculated, and if the angle between the unit direction vector of the point cloud pointing to the camera and the normal vector is greater than 90 degrees, the direction of the normal vector is reversed to obtain a processed normal vector. It should be noted that the unit direction vector of the point cloud pointing to the camera to some extent indicates the grasping direction, and the grasping end (such as a grasping and clamping) is grasped from the direction of the point cloud to the camera, and the normal vector is needed to obtain the direction of the grasping pose. If the angle between the normal vector and the unit direction vector of the point cloud pointing to the camera is greater than 90 degrees, it means that the normal vector is in the opposite direction, so the normal vector is reversed. For example, the unit direction vector of a point pointing to the camera is upward relative to the table, but the normal vector of the point is downward, and the normal vector needs to be reversed to ensure the correct grasping direction. If the normal vector is downward, it will cause the grasping and clamping to grasp from the bottom to the top.

[0073] ​In a possible implementation, the M grasp poses are calculated based on the surface normal of each of the M sampled point clouds and the grasp end model parameter, including: calculating a grasp center point of each of the M sampled point clouds based on the surface normal of each of the M sampled point clouds and the depth of the grasp end model; the grasp center point of each of the M sampled point clouds is on the surface normal of the corresponding sampled point cloud; obtaining M initial grasp poses based on the grasp center point of each of the M sampled point clouds and the width of the grasp end model; and performing translation and / or rotation on the M initial grasp poses respectively to obtain the M grasp poses. Specifically, for a sampled point cloud, the surface normal of the sampled point cloud can be used to determine the direction in which the grasp end (such as a gripper) falls. Further, based on the depth of the grasp end model, the sampled point is moved in the direction of the surface normal by a distance of, for example, the depth of a gripper, and is used as the grasp center point (i.e., the grasp end of the gripper) to avoid collision between the gripper and the sampled point cloud. Next, based on the grasp center point and the width of the gripper, the initial grasp pose can be determined. Finally, the initial grasp pose is translated and / or rotated to obtain the optimal grasp pose for grasping the corresponding sampled point cloud. For example, as shown in FIG. 4, Figure 6 , Figure 6 FIG. 4 is a schematic diagram of translation of an initial grasp pose according to an embodiment of the present application, in which the grasp end model simulates grasping of the corresponding local point cloud scene based on the grasp center point. If the local point cloud is not at the center of the gripper, the initial grasp pose can be translated to make the local point cloud as close to the center of the gripper as possible. For another example, as shown in FIG. 5, Figure 7 , Figure 7 FIG. 5 is a schematic diagram of rotation of an initial grasp pose according to an embodiment of the present application, in which the gripper model simulates grasping of the corresponding local point cloud scene based on the grasp center point. In order to make the gripper grasp the object more stably, the initial grasp pose can be rotated to the center position of the rotation angle range corresponding to the local point cloud without collision.

[0074] In a possible implementation, the surface normal is calculated based on the random sampling point of the preprocessed point cloud, and the center position of the translation and rotation angle is directly calculated based on the grasp end parameter (i.e., the gripper model parameter) and the boundary of the collision detection to generate the grasp pose. Specifically, as shown in FIG. 6, Figure 8 , Figure 8Another schematic diagram of generating a grasping pose provided by an embodiment of the present application is shown in the figure. For a sample point cloud, according to the normal vector projection matrix and the eigenvector of each point included, the surface normal, the minor principal curvature (which can represent the longer direction of the object), and the major principal curvature (which can represent the width of the object) perpendicular to the two are estimated. Further, the surface normal can be taken as the coordinate axis vector of the grasping direction of the grasping end coordinate system, and the sample point coordinates can be moved along the surface normal by a gripper space distance as the grasping end, i.e., the grasping center point. Next, as shown in Figure 9 Figure 9 A schematic diagram of a translation center and a rotation center provided by an embodiment of the present application is shown in the figure. The translation distance of the grasping center relative to the intersecting part point cloud (which can also be referred to as a local point cloud) is calculated, and the gripper center is moved along the major principal curvature direction. The center position of the rotation angle range corresponding to the non-collision relationship between the gripper and the point cloud within the set active angle range is calculated along the surface normal, the best grasping pose corresponding to the current sample point is determined according to the grasping physical model and the force closure principle.

[0075] Step S203: The electronic device selects one grasping pose from the M grasping poses as a target grasping pose for grasping the target object.

[0076] In a possible implementation, the method further includes: performing collision detection on the M grasping poses based on the object point cloud of the target object; and determining L grasping poses in the M grasping poses that satisfy a preset collision detection result as L candidate grasping poses, L being an integer greater than 0. Specifically, since each grasping pose in the M grasping poses is only the optimal grasping pose for grasping the corresponding sample point cloud (i.e., the local of the target object), but there may be a grasping pose in the M grasping poses that has a collision relationship with the local of the target object and / or the desktop when grasping the entire target object. Therefore, collision detection can be performed on the M grasping poses based on the object point cloud of the target object, so as to eliminate the grasping poses that have a collision relationship with the target object or the desktop, thereby improving the efficiency of the candidate grasping poses.

[0077] For example, as shown in Figure 10 Figure 10 ​​A schematic diagram for generating a candidate grasping pose is provided for the embodiments of the present application. In the diagram, the object point cloud in the desktop coordinate system can be determined first, and then the object point cloud is processed and the normal vector is estimated. Next, a plurality of sampling points are selected from the object point cloud, and the grasping pose coordinate vector is calculated based on the plurality of sampling points. Finally, collision detection filters out the grasping poses that do not meet the requirements. Specifically, the grasping point cloud corresponding to each sampled grasping pose is subjected to collision detection with the object point cloud and the desktop, respectively. If collision with the object point cloud is found, it is skipped (i.e. the grasping pose is rejected); if collision with the desktop is found, the grasping point cloud is moved upwards by a certain distance. The number of points of the intersection part of the grasping point cloud and the object is calculated. If the number meets the set requirement, the current grasping pose is determined as the candidate grasping pose.

[0078] In a possible implementation, the selecting one grasping pose from the M grasping poses as a target grasping pose for grasping the target object includes: obtaining L local point clouds based on the L candidate grasping poses, each of the L local point clouds being a point cloud of an intersection part of a corresponding candidate grasping pose and the object point cloud; obtaining pose evaluation information based on the L local point clouds and the object point cloud; the pose evaluation information including an evaluation score corresponding to each of the L local point clouds, and each evaluation score corresponding to each of the local point clouds being obtained based on feature information of the L local point clouds and feature information of the object point cloud; and determining a candidate grasping pose corresponding to a local point cloud with the highest evaluation score as the target grasping pose. Specifically, a corresponding local point cloud is obtained based on each candidate grasping pose, and then each local point cloud is scored in combination with the feature information of the local point cloud and the feature information of the object point cloud. In this process, the candidate grasping pose can be scored in combination with factors such as the relationship of the grasping position relative to the grasped object and the relationship of the grasping position relative to all objects, and then the candidate grasping pose with the highest evaluation score is determined as the optimal grasping pose for grasping the target object. Therefore, the method provided by the present application can adapt to more diverse object grasping (such as selecting the grasping pose closest to the center of the long strip-shaped object as a more optimal sample) and stacked object grasping (such as selecting the grasping pose of the outermost layer of the stacked object as a more optimal sample), and adapt to more complex scenes.

[0079] It should be noted that the calculation of the local point cloud can be based on the input grasping pose and the grasping end parameter to calculate a local three-dimensional point cloud centered on the center of the grasping end. Methods that can be used but are not limited to include calculating a local point cloud that coincides in space, directly calculating a local point cloud with the size information in the grasping end as the radius, and the like.

[0080] In a possible implementation, the selecting one of the M grasping poses as a target grasping pose for grasping the target object comprises: generating M local point clouds based on the M grasping poses, each of the M local point clouds being a point cloud of a portion of the object point cloud intersected by a corresponding candidate grasping pose; obtaining pose evaluation information based on the M local point clouds and the object point cloud; the pose evaluation information comprising an evaluation score corresponding to each of the M local point clouds, each of the evaluation scores corresponding to the local point clouds being obtained based on feature information of the M local point clouds and feature information of the object point cloud; and determining the candidate grasping pose corresponding to the local point cloud with the highest evaluation score as the target grasping pose.

[0081] For example, as shown in Figure 11 , Figure 11 is a schematic diagram of an optimal grasping pose provided by an embodiment of the present application. In the diagram, a local point cloud can be obtained based on a plurality of candidate grasping poses and a grasping model, and the local point cloud is scored by a grasping pose evaluation model, and then the candidate grasping pose corresponding to the local point cloud with the highest score is confirmed as the optimal grasping pose. In the embodiment of the present application, a multi-branch grasping pose evaluation model with input of a local point cloud and an object point cloud is used. Since the relationship of a grasping position relative to a grasped object and the relationship of the grasping position relative to all objects are combined, a more optimal label is set for samples with a grasping position closer to the center of an object and a grasping position closer to the outer layer of an object, which can adapt to more diverse object grasping (such as selecting a grasping pose closest to the center of a long strip-shaped object as a more optimal sample) and stacked object grasping (such as selecting a grasping pose at the outermost layer of a stacked object as a more optimal sample), and adapt to more complex scenarios.

[0082] For example, as shown in Figure 12 , Figure 12A network structure diagram of a grasping pose evaluation model provided in an embodiment of the present application is first trained. The input is a local three-dimensional point cloud calculated according to a grasping pose (which can be any pose of a grasped object) and an object three-dimensional point cloud (which can be a point cloud of any object), and the output is a multi-level classification label of point clouds such as optimal (highest evaluation score), suboptimal, grasping failure (lowest evaluation score), and the like. The object three-dimensional point cloud includes a single object point cloud and a point cloud of a plurality of objects stacked or combined, the former being used to learn the grasping position and the relative position relationship of a single object, and the latter being used to learn the grasping position and the relative position relationship of a plurality of objects. For example, for data of a single object, samples closer to the center of the object are assigned more optimal labels, and for data of a plurality of objects, samples with a more outer grasping position are assigned more optimal labels. When the grasping pose evaluation model is trained, the local point cloud and the object point cloud corresponding to the candidate grasping pose generated based on the present application are input into the trained grasping pose evaluation model. The model is divided into two branches, the first branch extracts local point cloud features, and the second branch extracts object point cloud features. After the features are fused, the local point cloud is classified in multiple levels to obtain grasping classification and score, and the optimal grasping pose with the highest score is selected as the final grasping pose. It should be noted that the calculation of the local point cloud can be based on the input grasping pose and the grasping end parameters to calculate the local three-dimensional point cloud centered on the center of the grasping end. Methods such as but not limited to calculating the local point cloud that overlaps in space, directly calculating the local point cloud with the size information in the grasping end as the radius, and the like can be used.

[0083] In an embodiment of the present application, the grasping pose is estimated based on the three-dimensional object point cloud data of the target object and the grasping end model (such as a grasping model) parameters, taking into account the three-dimensional features of the target object, thereby improving the grasping efficiency and accuracy of the robot in a complex scene. Specifically, since the form of the target object is not limited, the target object may be stacked or irregular, and the target grasping pose cannot be directly calculated, therefore M sampling points are first taken on the object point cloud of the target object, and the number of point clouds in the spherical space with each sampling point as the center and a predetermined length as the radius needs to be greater than a predetermined value, so as to determine the points close to the center of the target object as sampling points, thereby avoiding the problem of low efficiency of candidate poses caused by random sampling. Further, M target object sampling point clouds (i.e., point clouds in the spherical space with a sampling point as the center and a predetermined length as the radius) are obtained according to the M sampling points, and then the optimal grasping pose for grasping each sampling point cloud can be generated based on the three-dimensional features of each sampling point cloud and the grasping model parameters, thereby obtaining M candidate grasping poses. Next, a target grasping pose can be selected from the plurality of candidate poses in combination with the overall target object, thereby improving the grasping efficiency and accuracy of the robot in a complex scene.

[0084] See Figure 13 ,Figure 13 A schematic diagram of a grasping pose estimation device provided for an embodiment of the present application, wherein the device comprises a point cloud preprocessing module, a candidate grasping pose generation module based on translation and rotation angle calculation, a local three-dimensional point cloud calculation module, a training module, and a multi-branch grasping pose evaluation network module. Among them,

[0085] The point cloud preprocessing module is used to reduce the scene point cloud, remove the non-object point cloud by calculating the desktop equation, and obtain the object point cloud. Further, the object point cloud coordinate system can be transformed to the desktop coordinate system according to the desktop equation.

[0086] The candidate grasping pose generation module based on translation and rotation angle calculation is used to calculate the surface normal of the region sampling point cloud using random sampling points on the preprocessed point cloud, and directly calculate the center position of the translation and rotation angle according to the grasping end parameter (i.e. the related parameters of the grasping model) and the boundary of the collision detection, to generate the candidate grasping pose.

[0087] The local three-dimensional point cloud calculation module is used to input the grasping pose and the grasping end parameter, and calculate the local three-dimensional point cloud centered on the grasping end center. Methods that can be used but are not limited to include calculating the coincident local point cloud in space, directly calculating the local point cloud with the size information in the grasping end as the radius, etc.

[0088] The training module is used to calculate the local three-dimensional point cloud based on sample data, i.e. a data set composed of object point cloud, grasping pose and positive and negative label data, and grasping end parameter; and train the multi-branch grasping pose evaluation network by taking the local three-dimensional point cloud and the object point cloud as network input.

[0089] The multi-branch grasping pose evaluation network module is used to input the local three-dimensional point cloud and the corresponding object point cloud into the trained grasping pose evaluation network. The network is divided into two branches. The first branch extracts the local point cloud features, and the second branch extracts the object point cloud features. After the features are fused, the local point cloud is classified at multiple levels to obtain the evaluation level and score of the grasping pose corresponding to the local three-dimensional point cloud, and the optimal grasping pose is obtained.

[0090] It should be noted that manual debugging can also be included in the present application. The manual debugging includes manual debugging of the point cloud preprocessing module, i.e. background point cloud removal visualization, manual checking of the effect and real-time adjustment of the background point cloud removal parameters through the interactive interface; manual debugging of the candidate grasping pose generation module, i.e. candidate grasping pose simulation and visualization, manual checking of the effect and real-time adjustment of the grasping pose parameters through the interactive interface; manual debugging of the grasping pose evaluation network module, i.e. visualizing the network evaluation effect, and adding the local point cloud with evaluation error to the training data set for updating the model. The detailed description is as follows,

[0091] The manual debugging of the point cloud preprocessing module is only needed once each time the new scene is switched to, and subsequent automatic point cloud preprocessing is performed by using the debugged parameters. Figure 14 As shown in the figure, Figure 14 A point cloud preprocessing schematic diagram is provided for the embodiment of the application, in which the point cloud preprocessing module removes the background point cloud according to the automatically identified table surface equation and the preset range parameters; the interactive interface displays the effect of removing the background point cloud; the user checks whether the background removal is clean or not, and if not, the next step is executed, and if yes, the process is ended; the user clicks the unremoved background part to mark it as background, and clicks the removed error background part to mark it as foreground; the point cloud preprocessing module adjusts the table surface equation and the range parameters according to the user marking, and removes the background point cloud again.

[0092] The manual debugging of the candidate grasping pose generation module is only needed once each time the new scene and the new grasping parameter are switched to, and subsequent automatic candidate grasping pose generation is performed by using the debugged parameters. Figure 15 As shown in the figure, Figure 15 A candidate pose selection schematic diagram is provided for the embodiment of the application, in which the candidate grasping pose generation module calculates the candidate pose according to the preset parameters; the interactive interface displays the candidate grasping pose effect, which can be a simplified or original robot arm end effector model grid superimposed with the original three-dimensional point cloud according to the candidate grasping pose; the user checks whether the candidate pose effect meets the requirements, and if not, the next step is executed, and if yes, the process is ended; the user adjusts the sampling parameters, such as the grasping parameter, the camera coordinate, the point cloud scale unit, the set table surface distance, the set search radius, the set angle and distance activity range, etc., on the interactive interface; and the candidate grasping pose generation module recalculates the candidate pose according to the adjusted parameters.

[0093] The manual debugging of the grasping pose evaluation network module is performed for a new scene and grasping parameter, the visual effect of the grasping pose evaluation network is manually checked, the evaluation error sample is screened out, and is added to the original data set for updating the model. Figure 16 As shown in the figure, Figure 16A schematic diagram of the grasp pose evaluation provided by the embodiment of the present application is shown in the figure, wherein the grasp pose evaluation network module infers a batch of data to obtain the ratings and scores of each grasp pose; the interactive interface displays the grasp pose effect and score, and can display the simplified or original mechanical arm end effector model grid according to the grasp pose and the original three-dimensional point cloud superposition; the user checks whether the candidate pose score meets the requirements, if not, the next step is performed, if all requirements are met, the process is ended; the user clicks the grasp pose visualization grid with classification errors or a larger score gap in the interactive interface; the three-dimensional local point cloud corresponding to the poor grasp pose selected by the user is saved and input into the training model to update the grasp pose evaluation network.

[0094] In the embodiment of the present application, the point cloud preprocessing module and the candidate grasp pose generation module with artificial debugging are used, the background removal, coordinate transformation, grasp pose sampling configuration and other operations of the input scene point cloud can be more efficiently completed through artificial preprocessing and sampling parameter debugging and real-time visualization, and only artificial debugging is required once when switching to a new scene, so that the new scene can be adapted faster; the artificial debugging of the grasp pose evaluation network training result can more efficiently complete the effect evaluation of the evaluation network, and new data can also be obtained for improving the network effect.

[0095] The above describes the method of the embodiment of the present application in detail, and the related equipment of the embodiment of the present application is provided below.

[0096] Please refer to Figure 17 , Figure 17 The structure diagram of the grasp pose estimation device provided by the embodiment of the present application is provided, the grasp pose estimation device 30 can include a first processing unit 301, a second processing unit 302, a third processing unit 303, a fourth processing unit 304, a fifth processing unit 305 and a sixth processing unit 306. The detailed description of each module is as follows.

[0097] The first processing unit 301 is configured to determine M sample point clouds of an object point cloud of a target object based on M sample points in the object point cloud, each sample point cloud in the M sample point clouds is a point cloud of the object point cloud in a corresponding target space, and the corresponding target space is a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius; wherein the number of points included in each sample point cloud is greater than a preset value; and M is an integer greater than 1.

[0098] The second processing unit 302 is configured to calculate M grasp poses based on the M sample point clouds and grasp end model parameters, wherein the grasp end model parameters include the width and depth of a grasp end model.

[0099] The third processing unit 303 is configured to select one of the M grasping poses as a target grasping pose for grasping the target object.

[0100] In a possible implementation, the device further includes a fourth processing unit 304 configured to obtain the object point cloud from a color map and a depth map of the target object captured by a camera, or obtain the object point cloud by scanning the target object using a laser radar.

[0101] In a possible implementation, the device further includes a fifth processing unit 305 configured to: pre-process the object point cloud to obtain N sampling points, N being an integer greater than M; obtain N sampling point clouds of the object point cloud based on the N sampling points, each of the N sampling point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sampling point in the N sampling points as a spherical center and a preset length as a radius; determine whether a number of points included in each of the N sampling point clouds is greater than a preset value; and determine the M sampling points as sampling points corresponding to M sampling point clouds greater than the preset value.

[0102] In a possible implementation, the second processing unit 302 is specifically configured to: obtain a surface normal of each of the M sampling point clouds based on a normal vector of each point in each of the M sampling point clouds; and calculate the M grasping poses based on the surface normal of each of the M sampling point clouds and the grasping end model parameters.

[0103] In a possible implementation, the second processing unit 302 is specifically configured to: calculate a grasping center point of each of the M sampling point clouds based on the surface normal of each of the M sampling point clouds and a depth of the grasping end model; the grasping center point of each of the M sampling point clouds being on the surface normal of the sampling point cloud corresponding to the sampling point corresponding to the grasping center point; obtain M initial grasping poses based on the grasping center point of each of the M sampling point clouds and a width of the grasping end model; and respectively perform translation and / or rotation on the M initial grasping poses to obtain the M grasping poses.

[0104] In a possible implementation, the device further includes a sixth processing unit 306 configured to: perform collision detection on the M grasping poses based on the object point cloud of the target object; and determine L grasping poses satisfying a preset collision detection result from the M grasping poses as L candidate grasping poses, L being an integer greater than 0.

[0105] In one possible implementation, the third processing unit 303 is specifically configured to: obtain L local point clouds based on the L candidate grasping poses, wherein each local point cloud is a point cloud representing the intersection of the corresponding candidate grasping pose and the object point cloud; obtain pose evaluation information based on the L local point clouds and the object point cloud; the pose evaluation information includes an evaluation score corresponding to each of the L local point clouds, wherein the evaluation score for each local point cloud is obtained based on the feature information of the L local point clouds and the feature information of the object point cloud; and determine the candidate grasping pose corresponding to the local point cloud with the highest evaluation score as the target grasping pose.

[0106] It should be noted that the functions of each functional unit in the grasping pose estimation device 30 described in this embodiment of the invention are as described above. Figure 2 The relevant descriptions of steps S201-S203 performed by the electronic device in the method embodiment are not repeated here.

[0107] This application provides a computer storage medium, characterized in that the computer storage medium stores a computer program, which, when executed by a processor, implements any of the above-mentioned grasping pose estimation methods.

[0108] This application provides an electronic device including a processor configured to support the implementation of corresponding functions in any of the above-described grasping pose estimation methods. The electronic device may also include a memory coupled to the processor, which stores necessary program instructions and data. The electronic device may also include a communication interface for communicating with other devices or communication networks.

[0109] This application provides a chip system including a processor for supporting an electronic device in implementing the functions described above, such as generating or processing information involved in the aforementioned grasping pose estimation method. In one possible design, the chip system further includes a memory for storing necessary program instructions and data for the electronic device. This chip system can be composed of chips or may include chips and other discrete devices.

[0110] This application provides a computer program, characterized in that the computer program includes instructions that, when executed by a computer, cause the computer to perform the aforementioned grasping pose estimation method.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0112] It should be noted that, for the foregoing method embodiments, the sequences of the described actions can be modified, and certain actions can be performed in other sequences or even concurrently. Also, certain actions can be optional based on the circumstances.

[0113] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the apparatus embodiments described above are merely schematic; the division of the units is only a logical function division; there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be implemented by using some interfaces, and the indirect couplings or communication connections can be implemented in electronic, mechanical, or other forms.

[0114] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0115] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0116] The integrated unit described above, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc., and specifically can be a processor in the computer device) to execute all or part of the steps of the above-mentioned methods of various embodiments of the present application. Among them, the aforementioned storage medium can include: a U disk, a mobile hard disk, a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, abbreviated as: ROM) or a random access memory (Random Access Memory, abbreviated as: RAM) and various program code storage media.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A grasp pose estimation method, characterized by, The method comprises: determining M sample point clouds of the object point cloud based on M sample points in the object point cloud of the target object, each of the M sample point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius; wherein the number of points included in each of the sample point clouds is greater than a preset value; M is an integer greater than 1; calculating M grasping poses based on the M sample point clouds and grasping end model parameters, the grasping end model parameters including the width and depth of the grasping end model; selecting one of the M grasping poses as a target grasping pose for grasping the target object.

2. The method of claim 1, wherein, The method further comprises: obtaining the object point cloud from a color map and a depth map of the target object captured by a camera, or obtaining the object point cloud by scanning the target object with a laser radar.

3. The method of claim 1 or 2, wherein, The method further comprises: preprocessing the object point cloud to obtain N to-be-sampled points, N being an integer greater than M; obtaining N sample point clouds of the object point cloud based on the N to-be-sampled points, each of the N sample point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding to-be-sampled point in the N to-be-sampled points as a spherical center and a preset length as a radius; determining whether the number of points included in each of the N sample point clouds is greater than a preset value; determining the M sample points as the M sample points corresponding to the to-be-sampled points greater than the preset value.

4. The method of claim 3, wherein, The method further comprises: obtaining a surface normal of each of the sample point clouds based on a normal vector of each point in each of the M sample point clouds; calculating the M grasping poses based on the surface normal of each of the sample point clouds and the grasping end model parameters.

5. The method of claim 4, wherein, The method further comprises: calculating a grasping center point of each of the sample point clouds based on the surface normal of each of the sample point clouds and the depth of the grasping end model; the grasping center point of each of the sample point clouds being on the surface normal of the corresponding sample point cloud corresponding to the corresponding sample point; obtaining M initial grasping poses based on the grasping center point of each of the sample point clouds and the width of the grasping end model; performing translation and / or rotation on the M initial grasping poses respectively to obtain the M grasping poses.

6. The method of claim 5, wherein, The method further comprises: performing collision detection on the M grasping poses based on the object point cloud of the target object; determining L candidate grasping poses from the M grasping poses that meet a preset collision detection result, L being an integer greater than 0.

7. The method of claim 6, wherein, The method further comprises: selecting one of the M grasping poses as a target grasping pose for grasping the target object. Obtain L local point clouds based on the L candidate grasping poses, each of the L local point clouds being a point cloud of a portion of the object point cloud intersected by a corresponding candidate grasping pose; Obtain pose evaluation information based on the L local point clouds and the object point cloud, the pose evaluation information including an evaluation score corresponding to each of the L local point clouds, each evaluation score being obtained based on feature information of the L local point clouds and feature information of the object point cloud; Determine the candidate grasping pose corresponding to the local point cloud with the highest evaluation score as the target grasping pose.

8. A grasp pose estimation device, characterized by, The device comprises: A first processing unit configured to determine M sample point clouds of an object point cloud of a target object based on M sample points in the object point cloud, each of the M sample point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding sample point in the M sample points as a spherical center and a preset length as a radius, and the number of points included in each of the sample point clouds being greater than a preset value, M being an integer greater than 1; A second processing unit configured to calculate M grasping poses based on the M sample point clouds and grasping end model parameters, the grasping end model parameters including a width and a depth of a grasping end model; A third processing unit configured to select one of the M grasping poses as a target grasping pose for grasping the target object.

9. The apparatus of claim 8, wherein, The device further comprises a fourth processing unit configured to: Obtain the object point cloud from a color map and a depth map of the target object captured by a camera, or obtain the object point cloud by scanning the target object using a laser radar.

10. The apparatus of claim 8 or 9, wherein, The device further comprises a fifth processing unit configured to: Preprocess the object point cloud to obtain N to-be-sampled points, N being an integer greater than M; Obtain N to-be-sampled point clouds of the object point cloud based on the N to-be-sampled points, each of the N to-be-sampled point clouds being a point cloud of the object point cloud in a corresponding target space, the corresponding target space being a spherical space with a corresponding to-be-sampled point in the N to-be-sampled points as a spherical center and a preset length as a radius; Determine whether the number of points included in each of the N to-be-sampled point clouds is greater than a preset value; Determine the M to-be-sampled points corresponding to the M to-be-sampled point clouds whose number of points is greater than the preset value as the M sample points.

11. The apparatus of claim 10, wherein, The second processing unit is specifically configured to: Obtain a surface normal of each of the M sample point clouds based on a normal vector of each point in each of the M sample point clouds; Calculate the M grasping poses based on the surface normal of each of the M sample point clouds and the grasping end model parameters.

12. The apparatus of claim 11, wherein, The second processing unit is specifically configured to: Calculate a grasping center point of each of the M sample point clouds based on the surface normal of each of the M sample point clouds and the depth of the grasping end model, the grasping center point of each of the M sample point clouds being on the surface normal of the sample point cloud corresponding to the corresponding sample point. M initial grasping poses are obtained based on a grasping center point of each of the sampling point clouds and a width of the grasping end model; The M initial grasping poses are respectively translated and / or rotated to obtain the M grasping poses.

13. The apparatus of claim 12, wherein, The device further comprises a sixth processing unit, configured to: perform collision detection on the M grasping poses based on the object point cloud of the target object; L grasping poses satisfying a preset collision detection result among the M grasping poses are determined as L candidate grasping poses, L being an integer greater than 0.

14. The apparatus of claim 13, wherein, The third processing unit is specifically configured to: obtain L local point clouds based on the L candidate grasping poses, each of the L local point clouds being a point cloud of an intersection part of a corresponding candidate grasping pose and the object point cloud; obtain pose evaluation information based on the L local point clouds and the object point cloud, the pose evaluation information including an evaluation score corresponding to each of the L local point clouds, the evaluation score corresponding to each of the local point clouds being obtained based on feature information of the L local point clouds and feature information of the object point cloud; determine a candidate grasping pose corresponding to a local point cloud with the highest evaluation score as the target grasping pose.

15. A computer storage medium, comprising, The computer storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-7.

Images