A parcel grasping method, device and apparatus

Abstract

The application provides a parcel grabbing method, device and equipment, the method comprises the following steps: obtaining a target RGB image and a target three-dimensional point cloud corresponding to a parcel table; determining K parcel image regions based on the target RGB image, K is a positive integer; determining K initial parcel point clouds corresponding to the K parcel image regions from the target three-dimensional point cloud, segmenting the K initial parcel point clouds according to the target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds; determining a three-dimensional grabbing point corresponding to each target parcel point cloud, sorting all three-dimensional grabbing points based on the grabbing parameters corresponding to each three-dimensional grabbing point, and selecting a three-dimensional grabbing point from all three-dimensional grabbing points as a target grabbing point based on the sorting result; wherein the target grabbing point is used to enable a robot to perform a grabbing operation on a parcel on the parcel table based on the target grabbing point. Through the technical scheme of the application, a suitable grabbing point (i.e. the best grabbing point) can be found, and the grabbing success rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of logistics, and particularly relates to a parcel grabbing method, device and equipment. BACKGROUND

[0002] With the sharp rise in labor costs of logistics sorting centers and the continuous improvement of logistics transportation efficiency, the demand for automation of parcel sorting is gradually increasing. Therefore, a vision-guided robot can be used to perform automatic grabbing of parcels, so as to realize the automation demand of parcel sorting. The vision-guided robot performing automatic grabbing of parcels refers to using a camera to collect an RGB image corresponding to a parcel supply table, analyzing a grabbing point based on the RGB image, and then controlling the robot to perform a grabbing operation on the parcel based on the grabbing point.

[0003] However, since there are many parcels on the parcel supply table, a large number of parcels are densely distributed. When the grabbing point is analyzed based on the RGB image, the interference between the parcels will be great, and a suitable grabbing point may not be found based on the RGB image, so that the robot cannot perform a grabbing operation on the parcel based on the suitable grabbing point. SUMMARY

[0004] Therefore, the present application provides a parcel grabbing method, device and equipment, which can control the robot to perform a grabbing operation on the parcel based on a suitable grabbing point, and improve the success rate of grabbing.

[0005] The present application provides a parcel grabbing method, which comprises:

[0006] obtaining a target RGB image corresponding to a parcel supply table, and obtaining a target three-dimensional point cloud corresponding to the parcel supply table;

[0007] determining K parcel image regions based on the target RGB image, K being a positive integer;

[0008] determining K initial parcel point clouds corresponding to the K parcel image regions from the target three-dimensional point cloud, segmenting the K initial parcel point clouds according to target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds, M being greater than or equal to K;

[0009] determining a three-dimensional grabbing point corresponding to each target parcel point cloud, sorting all three-dimensional grabbing points based on grabbing parameters corresponding to each three-dimensional grabbing point, and selecting a three-dimensional grabbing point as a target grabbing point from all three-dimensional grabbing points based on a sorting result; wherein the target grabbing point is used to control the robot to perform a grabbing operation on the parcel on the parcel supply table based on the target grabbing point.

[0010] The present application provides a parcel grabbing device, which comprises:

[0011] An acquisition module is configured to acquire a target RGB image corresponding to the parcel table and acquire a target three-dimensional point cloud corresponding to the parcel table.

[0012] A determination module is configured to determine K parcel image regions based on the target RGB image, and determine K initial parcel point clouds corresponding to the K parcel image regions from the target three-dimensional point cloud.

[0013] A segmentation module is configured to segment the K initial parcel point clouds according to target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds, where M is greater than or equal to K.

[0014] A processing module is configured to determine a three-dimensional grasping point corresponding to each target parcel point cloud, sort all three-dimensional grasping points based on grasping parameters corresponding to each three-dimensional grasping point, and select a three-dimensional grasping point as a target grasping point from all three-dimensional grasping points based on a sorting result. The target grasping point is used to enable the robot to perform a grasping operation on the parcel on the parcel table based on the target grasping point.

[0015] The present application provides an electronic device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; wherein the processor is configured to execute the machine executable instructions to implement the parcel grasping method described above.

[0016] As can be seen from the above technical solutions, in the embodiment of the present application, after determining K parcel image regions based on the target RGB image corresponding to the parcel table, K initial parcel point clouds corresponding to the K parcel image regions can be determined from the target three-dimensional point cloud corresponding to the parcel table, and the K initial parcel point clouds can be segmented to obtain M target parcel point clouds. That is, the target grasping point is determined based on the M target parcel point clouds, rather than based on the K parcel image regions, so that a suitable grasping point (i.e. the best grasping point) can be found, and the robot can be controlled to perform a grasping operation on the parcel based on the suitable grasping point. In the dense parcel scenario, a high-robustness and high-success-rate grasping effect can be obtained, and the grasping success rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0018] Figure 1 is a flowchart of the parcel grasping method in an embodiment of the present application;

[0019] Figure 2 is a flowchart of a parcel grabbing method in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an application scenario of a parcel supply table in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a bounding box of a parcel supply table in an embodiment of the present application;

[0022] Figure 5 is a flowchart of a parcel grabbing method in an embodiment of the present application;

[0023] Figure 6 is a structural schematic diagram of a parcel grabbing device in an embodiment of the present application;

[0024] Figure 7 is a hardware structure diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The terms used in the embodiments of the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the present application and claims are intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0026] It should be understood that although the terms first, second, third, etc. can be used in the embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the present application. In addition, depending on the context, the word "if" used can be interpreted as "when" or "upon" or "in response to determining".

[0027] An embodiment of the present application proposes a parcel grabbing method, as shown in Figure 1 The method can include:

[0028] Step 101, acquiring a target RGB image corresponding to a parcel supply table, and acquiring a target three-dimensional point cloud corresponding to the parcel supply table.

[0029] For example, obtaining the target RGB image corresponding to the parcel table can include but is not limited to: acquiring an initial RGB image corresponding to the parcel table through an RGB camera, the initial RGB image including the parcel table and a plurality of parcels on the parcel table. The RGB image of the parcel table region can be selected from the initial RGB image, for example, a parcel table bounding box can be pre-configured, the RGB image of the parcel table region can be selected from the initial RGB image through the parcel table bounding box, an image recognition algorithm can also be used to select the RGB image of the parcel table region from the initial RGB image, a machine learning model can also be used to select the RGB image of the parcel table region from the initial RGB image, and no limitation is made to this, as long as the RGB image of the parcel table region can be obtained. The target RGB image can be determined based on the RGB image of the parcel table region.

[0030] For example, obtaining the target three-dimensional point cloud corresponding to the parcel table can include but is not limited to: acquiring an initial three-dimensional point cloud corresponding to the parcel table through a depth camera, the initial three-dimensional point cloud including the parcel table and a plurality of parcels on the parcel table. The three-dimensional point cloud of the parcel table region can be determined based on the initial three-dimensional point cloud. The target three-dimensional point cloud can be determined based on the three-dimensional point cloud of the parcel table region after the three-dimensional point cloud of the parcel table region is obtained.

[0031] The three-dimensional point cloud of the parcel table region can be determined based on the initial three-dimensional point cloud, which can include: a parcel table bounding box can be pre-configured, the three-dimensional point cloud of the parcel table region can be selected from the initial three-dimensional point cloud through the parcel table bounding box, an image recognition algorithm can also be used to select the three-dimensional point cloud of the parcel table region from the initial three-dimensional point cloud, a machine learning model can also be used to select the three-dimensional point cloud of the parcel table region from the initial three-dimensional point cloud, and no limitation is made to this, as long as the three-dimensional point cloud of the parcel table region can be obtained.

[0032] Alternatively, the initial three-dimensional point cloud (i.e. the three-dimensional point cloud in the depth camera coordinate system) can be converted into a three-dimensional point cloud in the parcel table coordinate system first, a parcel table bounding box can be pre-configured, the three-dimensional point cloud of the parcel table region can be selected from the three-dimensional point cloud in the parcel table coordinate system through the parcel table bounding box, an image recognition algorithm can also be used to select the three-dimensional point cloud of the parcel table region from the three-dimensional point cloud in the parcel table coordinate system, a machine learning model can also be used to select the three-dimensional point cloud of the parcel table region from the three-dimensional point cloud in the parcel table coordinate system, and no limitation is made to this, as long as the three-dimensional point cloud of the parcel table region can be obtained.

[0033] Alternatively, the initial three-dimensional point cloud can be converted into a three-dimensional point cloud in a third coordinate system (such as a coordinate system other than the depth camera coordinate system and the supply table coordinate system, such as a world coordinate system), and the supply table bounding box can be pre-configured. The three-dimensional point cloud in the supply table region can be selected from the three-dimensional point cloud in the third coordinate system by the supply table bounding box, or an image recognition algorithm can be used to select the three-dimensional point cloud in the supply table region from the three-dimensional point cloud in the third coordinate system, or a machine learning model can be used to select the three-dimensional point cloud in the supply table region from the three-dimensional point cloud in the third coordinate system. No limitation is made to this, and the three-dimensional point cloud in the supply table region can be obtained.

[0034] For example, determining the target three-dimensional point cloud based on the three-dimensional point cloud in the supply table region can include, but is not limited to: determining the three-dimensional point cloud in the supply table region as the target three-dimensional point cloud; or the three-dimensional point cloud in the supply table region includes multiple initial three-dimensional points, and a target three-dimensional point can be selected from the multiple initial three-dimensional points, and the height of the target three-dimensional point is greater than a preset height threshold; and generating a target three-dimensional point cloud based on all target three-dimensional points.

[0035] For example, determining the target three-dimensional point cloud based on the three-dimensional point cloud in the supply table region can include, but is not limited to: determining the three-dimensional point cloud in the supply table region as the target three-dimensional point cloud; or the three-dimensional point cloud in the supply table region includes multiple initial three-dimensional points, and a target three-dimensional point can be selected from the multiple initial three-dimensional points, and the height of the target three-dimensional point is greater than a preset height threshold; and generating a target three-dimensional point cloud based on all target three-dimensional points.

[0036] Step 102, determining K package image regions based on the target RGB image, K being a positive integer.

[0037] For example, the target RGB image can be input to a trained instance segmentation model, and the instance segmentation model can perform instance segmentation on the target RGB image to obtain K package image regions. Alternatively, an image recognition algorithm can be used to recognize the target RGB image to obtain K package image regions. Of course, other methods can also be used as long as the package image regions can be found in the target RGB image.

[0038] Step 103, determining K initial package point clouds corresponding to the K package image regions from the target three-dimensional point cloud, and segmenting the K initial package point clouds according to target features corresponding to the K initial package point clouds to obtain M target package point clouds, M can be greater than or equal to K.

[0039] For example, the K initial wrapping point clouds are segmented according to the target features corresponding to the K initial wrapping point clouds to obtain M target wrapping point clouds, which can include but is not limited to: for each initial wrapping point cloud, all three-dimensional points in the initial wrapping point cloud are mapped to a voxel three-dimensional grid to obtain a plurality of hyper voxels corresponding to the initial wrapping point cloud. All hyper voxels can be clustered based on the target features corresponding to each hyper voxel to obtain N clustering groups, N being a positive integer, and a clustering group can include at least one hyper voxel; wherein for each hyper voxel, if the target feature corresponding to the hyper voxel is the same as the target feature corresponding to the adjacent hyper voxel of the hyper voxel, the hyper voxel and the adjacent hyper voxel of the hyper voxel belong to the same clustering group, and if the target feature corresponding to the hyper voxel is different from the target feature corresponding to the adjacent hyper voxel of the hyper voxel, the hyper voxel and the adjacent hyper voxel of the hyper voxel belong to different clustering groups; wherein the target feature is a concave feature or a convex feature. N target wrapping point clouds corresponding to the initial wrapping point cloud are generated based on the N clustering groups.

[0040] For example, the target features corresponding to the initial wrapping point cloud can include but are not limited to neighborhood features (i.e., concave-convex features), color features, texture features, normal features, etc., without limitation. In subsequent embodiments, the neighborhood features corresponding to the initial wrapping point cloud are taken as an example for illustration.

[0041] Of course, the above is only an example, and as long as the K initial wrapping point clouds can be segmented into K initial wrapping point clouds, without limitation. For example, for each initial wrapping point cloud, if at least two wrapping point clouds corresponding to the initial wrapping point cloud are identified (the hyper voxel clustering algorithm is only an example of an identification algorithm, and other algorithms such as a density-based clustering algorithm, a partition-based clustering algorithm (k-means), and a hierarchical clustering algorithm can also be used to identify whether the initial wrapping point cloud corresponds to at least two wrapping point clouds, without limitation), if at least two wrapping point clouds are identified, the initial wrapping point cloud is segmented into at least two target wrapping point clouds, and if only one wrapping point cloud is identified, the initial wrapping point cloud is maintained.

[0042] Step 104, determining a three-dimensional grabbing point corresponding to each target wrapping point cloud, sorting all three-dimensional grabbing points based on the grabbing parameters corresponding to each three-dimensional grabbing point, and selecting a three-dimensional grabbing point from all three-dimensional grabbing points as a target grabbing point based on the sorting result. For example, the target grabbing point is used to enable the robot to perform a grabbing operation on the package on the feeding table based on the target grabbing point.

[0043] For example, determining the three-dimensional grasping point corresponding to each target wrapping point cloud can include but is not limited to: for each target wrapping point cloud, mapping the three-dimensional points in the target wrapping point cloud to two-dimensional points, and selecting a two-dimensional point (such as the clustering center point of the maximum inscribed circle) from the mapped two-dimensional points as a target two-dimensional point, and determining a candidate three-dimensional point corresponding to the target two-dimensional point from all three-dimensional points of the target wrapping point cloud. Based on this, the candidate three-dimensional point can be determined as the three-dimensional grasping point corresponding to the target wrapping point cloud; or the candidate three-dimensional point can be subjected to angular perturbation and / or translation perturbation to obtain a plurality of perturbed three-dimensional points, and the candidate three-dimensional point and the plurality of perturbed three-dimensional points are determined as the three-dimensional grasping point corresponding to the target wrapping point cloud.

[0044] Of course, the above is only an example of determining a three-dimensional grasping point, and is not limited thereto. For example, a three-dimensional point that meets a first preset feature (the first preset feature can be configured according to experience) can be directly selected from the three-dimensional points in the target wrapping point cloud, and the selected three-dimensional point is taken as a candidate three-dimensional point. Alternatively, the three-dimensional points in the target wrapping point cloud are mapped to two-dimensional points, and a two-dimensional point that meets a second preset feature (the second preset feature can be configured according to experience) is selected from the mapped two-dimensional points as a target two-dimensional point, and a candidate three-dimensional point corresponding to the target two-dimensional point is determined from all three-dimensional points of the target wrapping point cloud.

[0045] For example, sorting all three-dimensional grasping points based on the grasping parameters corresponding to each three-dimensional grasping point can include but is not limited to: for each three-dimensional grasping point, the three-dimensional grasping point can correspond to a plurality of grasping parameters, and a target score corresponding to the three-dimensional grasping point is determined based on the grasping score corresponding to each grasping parameter corresponding to the three-dimensional grasping point; and based on the target score corresponding to each three-dimensional grasping point, all three-dimensional grasping points are sorted in order of target score from high to low or in order of target score from low to high.

[0046] For example, when all three-dimensional grasping points are sorted in order of target score from high to low, the first three-dimensional grasping point is selected as the target grasping point. Alternatively, when all three-dimensional grasping points are sorted in order of target score from low to high, the last three-dimensional grasping point is selected as the target grasping point.

[0047] Of course, other ways of sorting all three-dimensional grasping points can also be used, such as sorting all three-dimensional grasping points based on the distance between each three-dimensional grasping point and the robot, and the like, which is not limited thereto.

[0048] Exemplarily, the plurality of grasping parameters corresponding to the three-dimensional grasping point include, but are not limited to, at least one of the following: a height of a parcel corresponding to the three-dimensional grasping point (i.e., a height parameter), an area of the parcel corresponding to the three-dimensional grasping point (i.e., an area parameter), an angle between a normal of a parcel table plane and the three-dimensional grasping point (i.e., an angle parameter), a type of the parcel corresponding to the three-dimensional grasping point (i.e., a parcel type parameter), a target feature corresponding to the three-dimensional grasping point (i.e., a target feature parameter), and a distance between the three-dimensional grasping point and a robot (i.e., a distance parameter). Of course, the above are only a few examples of grasping parameters, and the grasping parameters are not limited in the embodiment.

[0049] Exemplarily, for the grasping score corresponding to each grasping parameter of the three-dimensional grasping point, the greater the height of the parcel corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the height parameter; the greater the area of the parcel corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the area parameter; the smaller the angle between the normal of the parcel table plane and the three-dimensional grasping point, the greater the grasping score corresponding to the angle parameter; the smaller the distance between the three-dimensional grasping point and the robot, the greater the grasping score corresponding to the distance parameter; the type of the parcel corresponding to the three-dimensional grasping point can be a hard parcel type or a soft parcel type, and the grasping score corresponding to the hard parcel type is greater than the grasping score corresponding to the soft parcel type; the target feature corresponding to the three-dimensional grasping point can be a convex feature or a concave feature, and the grasping score corresponding to the convex feature is greater than the grasping score corresponding to the concave feature.

[0050] As can be seen from the above technical solutions, in the embodiment of the present application, after determining K parcel image regions based on the target RGB image corresponding to the parcel table, K initial parcel point clouds corresponding to the K parcel image regions can be determined from the target three-dimensional point cloud corresponding to the parcel table, and the K initial parcel point clouds are segmented to obtain M target parcel point clouds. The target grasping point is determined based on the M target parcel point clouds, rather than based on the K parcel image regions, so that a suitable grasping point (i.e., the best grasping point) can be found, and the robot is controlled to perform a grasping operation on the parcel based on the suitable grasping point. In a dense parcel scene, a high-robustness and high-success-rate grasping effect can be obtained, and the grasping success rate is improved.

[0051] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0052] In order to realize the automation requirement of parcel sorting, a vision-guided robot can be used to perform automatic grasping of a parcel. For example, a camera is used to capture an RGB image corresponding to a parcel table, a grasping point is analyzed based on the RGB image, and then a robot is controlled to perform a grasping operation on a parcel based on the grasping point.

[0053] However, due to the large number of packages on the package supply table, a large number of packages are densely distributed, and when the grabbing point is analyzed based on the RGB image, the interference between the packages will be great, and the suitable grabbing point cannot be found based on the RGB image, so that the robot cannot perform the grabbing operation on the package based on the suitable grabbing point.

[0054] In view of the above finding, the embodiment of the present application proposes a package segmentation and grabbing point determination method based on RGBD visual guidance, which is a method of guiding the robot to perform automatic grabbing of the package based on the RGB image and the three-dimensional point cloud. In the dense package scene, a high-robustness and high-success-rate grabbing effect can be obtained, and the grabbing success rate is improved. A suitable grabbing point (i.e. the best grabbing point) can be found, so that the robot can perform the grabbing operation on the package based on the suitable grabbing point, and the grabbing success rate is improved.

[0055] In the embodiment of the present application, referring to Figure 2 As shown in the figure, the package grabbing method can include but is not limited to the processes of calibration of the working area of the package supply table, deep learning coarse segmentation of the adaptive window, secondary segmentation of the three-dimensional point cloud, package classification, grabbing point calculation of the package point cloud, disturbance of the grabbing point, and sorting of the grabbing point. The processes will be described in detail below.

[0056] First, the calibration of the working area of the package supply table.

[0057] Referring to Figure 3 As shown in the figure, the package supply table is used to place the disordered and stacked packages, and the RGBD camera is arranged on the upper side of the package supply table. The RGBD camera includes an RGB camera and a depth camera (which is composed of a Left camera and a Right camera). The RGB camera is used to collect the RGB image of the package supply table, and the depth camera is used to collect the three-dimensional point cloud of the package supply table. The coordinate system of the RGB camera is denoted as (Xc, Yc, Zc), the coordinate system of the depth camera is denoted as (Xd, Yd, Zd), and the coordinate system of the package supply table is denoted as (Xw, Yw, Zw).

[0058] Exemplarily, the conversion parameters Rw and Tw between the coordinate system of the depth camera and the coordinate system of the package supply table can be calibrated in advance. Rw represents the rotation matrix between the coordinate system of the depth camera and the coordinate system of the package supply table, and Tw represents the translation matrix between the coordinate system of the depth camera and the coordinate system of the package supply table. The conversion parameters Rc and Tc between the coordinate system of the depth camera and the coordinate system of the RGB camera can be calibrated in advance. Rc represents the rotation matrix between the coordinate system of the depth camera and the RGB camera, and Tc represents the translation matrix between the coordinate system of the depth camera and the RGB camera. The calibration process of the above parameters is not limited in the embodiment.

[0059] Referring to Figure 4As shown, in the calibration process of the working area of the parcel delivery table, the working area of the parcel delivery table can be calibrated in advance, for example, the parcel delivery table bounding box (i.e., a two-dimensional bounding box) is calibrated for an RGB image, and the parcel delivery table bounding box (i.e., a three-dimensional bounding box), also referred to as a 3D box, is calibrated for a three-dimensional point cloud.

[0060] In one possible implementation, the parcels on the parcel delivery table can be emptied, and an RGB image of the parcel delivery table can be captured by an RGB camera. After obtaining the RGB image, a bounding box of the working area can be drawn in the RGB image, for example, a user can manually draw the bounding box of the working area, i.e., the parcel delivery table bounding box, in the RGB image, which can be understood as the two-dimensional effective area of the parcel delivery table.

[0061] After obtaining the parcel delivery table bounding box, a parcel delivery table bounding box for a three-dimensional point cloud can be generated based on the parcel delivery table bounding box, which can be understood as the three-dimensional effective area of the parcel delivery table. For example, height information can be added based on the parcel delivery table bounding box, so that the parcel delivery table bounding box can be obtained.

[0062] In another possible implementation, the parcels on the parcel delivery table can be emptied, and an RGB image of the parcel delivery table can be captured by an RGB camera, and a three-dimensional point cloud of the parcel delivery table can be captured by a depth camera. The three-dimensional point cloud can be converted into a three-dimensional point cloud in the coordinate system of the parcel delivery table, for example, based on the conversion parameters between the coordinate system of the depth camera and the coordinate system of the parcel delivery table.

[0063] After obtaining the RGB image, a bounding box of the working area can be drawn in the RGB image, for example, a user can manually draw the bounding box of the working area, i.e., the parcel delivery table bounding box, in the RGB image, which can be understood as the two-dimensional effective area of the parcel delivery table.

[0064] After obtaining the three-dimensional point cloud in the coordinate system of the parcel delivery table, a bounding box of the working area can be drawn in the three-dimensional point cloud, for example, a user can manually draw the bounding box of the working area, i.e., the parcel delivery table bounding box, in the three-dimensional point cloud, which can be understood as the three-dimensional effective area of the parcel delivery table.

[0065] Of course, the above is only an example of calibrating the parcel delivery table bounding box and the parcel delivery table bounding box, which is not limited, for example, the RGB image can be input to a machine learning model, and the machine learning model can identify the area where the parcel delivery table is located from the RGB image and output the bounding box of the area where the parcel delivery table is located, i.e., the parcel delivery table bounding box. The three-dimensional point cloud can also be input to a machine learning model, and the machine learning model can identify the area where the parcel delivery table is located from the three-dimensional point cloud and output the bounding box of the area where the parcel delivery table is located, i.e., the parcel delivery table bounding box.

[0066] Second, the adaptive window deep learning rough segmentation.

[0067] In the adaptive window deep learning rough segmentation process, the RGB image corresponding to the packaging table can be acquired by the RGB camera, the three-dimensional point cloud corresponding to the packaging table can be acquired by the depth camera, the three-dimensional point cloud can be converted to the packaging table coordinate system, the three-dimensional point cloud can be background filtered by the packaging table 3D box, the RGB image can be background filtered by the packaging table bounding box, and the 2D image mask of the package can be obtained by deep learning on the RGB image without background.

[0068] For example, the adaptive window deep learning rough segmentation process can include the following steps:

[0069] Step S11, acquiring the initial RGB image corresponding to the packaging table by the RGB camera (i.e., the initial RGB image includes the packaging table and the multiple packages on the packaging table), and acquiring the initial three-dimensional point cloud corresponding to the packaging table by the depth camera (i.e., the initial three-dimensional point cloud includes the packaging table and the multiple packages on the packaging table).

[0070] For example, in the normal working process of the packaging table, multiple packages can be placed in disorder on the packaging table, and an RGBD camera (an RGBD camera is a camera that can acquire three-dimensional information and color information of an object, and data acquisition is performed by three-dimensional point cloud and RGB image respectively) is arranged on the upper side of the packaging table. The RGBD camera includes an RGB camera and a depth camera. The RGB image corresponding to the packaging table can be acquired by the RGB camera. For convenience of distinction, the RGB image is referred to as the initial RGB image. The three-dimensional point cloud corresponding to the packaging table can be acquired by the depth camera. For convenience of distinction, the three-dimensional point cloud is referred to as the initial three-dimensional point cloud.

[0071] Step S12, converting the initial three-dimensional point cloud to the three-dimensional point cloud in the packaging table coordinate system.

[0072] For example, based on the conversion parameters between the coordinate system of the depth camera and the packaging table coordinate system, the initial three-dimensional point cloud is converted to the three-dimensional point cloud in the packaging table coordinate system.

[0073] Step S13, selecting the three-dimensional point cloud of the serving tray area from the three-dimensional point cloud under the serving tray coordinate system through the serving tray bounding box (i.e. the three-dimensional effective area of the serving tray), and selecting the RGB image of the serving tray area from the initial RGB image through the serving tray bounding box (i.e. the two-dimensional effective area of the serving tray).

[0074] For example, the three-dimensional point cloud within the serving tray bounding box range can be selected from all three-dimensional point clouds under the serving tray coordinate system, and the three-dimensional point cloud within the serving tray bounding box range is the three-dimensional point cloud of the serving tray area, and the three-dimensional point cloud of the serving tray area is part of the three-dimensional point cloud.

[0075] The sub-image within the serving tray bounding box range can be selected from the initial RGB image, and the sub-image within the serving tray bounding box range is the RGB image of the serving tray area.

[0076] Step S14, aligning the three-dimensional point cloud of the serving tray area and the RGB image of the serving tray area, i.e. mapping the three-dimensional point information and the pixel value information of the same physical position to the same pixel point. For example, for any physical position on the serving tray, the three-dimensional point cloud includes the three-dimensional point information corresponding to the physical position, and the RGB image includes the pixel value information corresponding to the physical position. By aligning the three-dimensional point cloud and the RGB image, the three-dimensional point information and the pixel value information of the physical position can be mapped to the same pixel point.

[0077] For example, the three-dimensional point cloud of the serving tray area and the RGB image of the serving tray area can be aligned based on the conversion parameters (i.e. rotation matrix and translation matrix, which can also be referred to as the extrinsic parameters between the depth camera and the RGB camera) between the coordinate system of the depth camera and the coordinate system of the RGB camera, the intrinsic parameters of the depth camera, and the intrinsic parameters of the RGB camera. The alignment method is not limited in the embodiment.

[0078] Step S15, the three-dimensional point cloud of the serving tray area (i.e. the aligned three-dimensional point cloud) includes a plurality of initial three-dimensional points, a target three-dimensional point can be selected from the plurality of initial three-dimensional points, the height of the target three-dimensional point is greater than a preset height threshold, and a target three-dimensional point cloud is generated based on all target three-dimensional points, i.e. all target three-dimensional points remaining after filtering are combined to form a target three-dimensional point cloud.

[0079] For example, for each initial three-dimensional point, the initial three-dimensional point has corresponding height information, and it is determined whether the height corresponding to the initial three-dimensional point is greater than a preset height threshold (which can be configured according to experience). If yes, the initial three-dimensional point is taken as a target three-dimensional point, and if no, the initial three-dimensional point is not taken as a target three-dimensional point, and the initial three-dimensional point is filtered out. After the above operation is performed on all initial three-dimensional points, the target three-dimensional points remaining after filtering can be obtained, and these target three-dimensional points can form a target three-dimensional point cloud.

[0080] Step S16, determine target pixel points corresponding to the target three-dimensional points from the RGB image of the parcel table area (i.e. the aligned RGB image), and generate a target RGB image based on the target pixel points.

[0081] For example, since the three-dimensional point cloud and the RGB image have been aligned, a one-to-one correspondence between the three-dimensional points in the three-dimensional point cloud and the pixel points in the RGB image can be obtained, and therefore, after obtaining the target three-dimensional point cloud, for each target three-dimensional point in the target three-dimensional point cloud, a target pixel point corresponding to the target three-dimensional point can be determined from the RGB image, i.e. a plurality of target pixel points can be determined from the RGB image.

[0082] For each target pixel point in the RGB image, the pixel value of the target pixel point can be kept unchanged, and the remaining pixel points other than all the target pixel points can be regarded as interference background in the RGB image, i.e. the pixel value of the remaining pixel points can be set to 0, so as to remove the interference background in the RGB image, and the RGB image after removing the interference background is referred to as a target RGB image.

[0083] Step S17, determine K parcel image regions based on the target RGB image, K being a positive integer.

[0084] For example, the target RGB image can be input into a trained network model, and the network model can output K parcel image regions corresponding to the target RGB image, i.e. the parcel rough segmentation is realized based on the target RGB image to obtain the K parcel image regions. For example, the network model can be a deep learning model or a neural network model, and the type of the network model is not limited as long as the parcel rough segmentation can be realized.

[0085] In one possible implementation, the network model can be an instance segmentation model, which is a kind of deep learning model used to extract the position and mask of a target object in a two-dimensional image. For example, the target RGB image can be input into a trained instance segmentation model, and the instance segmentation model can perform instance segmentation on the target RGB image to obtain K parcel image regions in the target RGB image.

[0086] Since the target RGB image includes a plurality of parcels, when the instance segmentation model performs instance segmentation on the target RGB image, the image region corresponding to each parcel can be obtained, and the image region corresponding to each parcel can be output, and these image regions are referred to as parcel image regions, and the instance segmentation process is not limited.

[0087] In summary, the target RGB image can be used to implement coarse parcel segmentation to obtain K parcel image regions in the target RGB image. The coarse parcel segmentation process is not limited in this embodiment.

[0088] Third, secondary segmentation of the three-dimensional point cloud.

[0089] When multiple parcels are densely stacked, it is difficult to distinguish the boundaries between single-color or same-color parcels from the target RGB image. When coarse parcel segmentation is implemented based on the target RGB image, multiple parcels may be segmented into one parcel (i.e., an incorrect segmentation result), and it is also difficult to segment the top surface and the side surface of a single solid-color parcel. Therefore, in this embodiment, secondary segmentation of the three-dimensional point cloud can also be performed. In the secondary segmentation process of the three-dimensional point cloud, based on the K parcel image regions in the target RGB image, K initial parcel point clouds corresponding to the K parcel image regions can be determined from the target three-dimensional point cloud, i.e., the initial parcel point cloud corresponding to each parcel is obtained. Then, secondary segmentation is performed according to the neighborhood features (i.e., concave-convex features) of the initial parcel point cloud. The secondary segmentation can improve the segmentation accuracy and also segment the parcels according to the planes, which is more consistent with the grasping of the parcels by the robot end suction cup. The improvement of the segmentation accuracy means that if multiple parcels are segmented into one parcel based on the target RGB image, the multiple parcels are segmented into multiple parcels through secondary segmentation to avoid incorrect segmentation results. The segmentation of the parcels according to the planes means that the dividing line between the top surface and the side surface can be clearly seen from the three-dimensional point cloud. Therefore, the top surface and the side surface of a single parcel are segmented through secondary segmentation, i.e., each surface is a parcel. For example, assuming that there are three surfaces in the parcel in the target RGB image, the three surfaces in the target RGB image are segmented into one parcel, and after secondary segmentation, the three surfaces are segmented into three parcels, which can be understood as different parcels.

[0090] For example, the secondary segmentation process of the three-dimensional point cloud can include the following steps:

[0091] In step S21, K initial parcel point clouds corresponding to the K parcel image regions are determined from the target three-dimensional point cloud, i.e., the K parcel image regions and the K initial parcel point clouds correspond to each other.

[0092] Exemplarily, in the adaptive window deep learning coarse segmentation process, K wrapped image regions in the target RGB image and a target three-dimensional point cloud can be obtained. Since the three-dimensional point cloud and the RGB image have been aligned, a one-to-one correspondence between the three-dimensional points in the three-dimensional point cloud and the pixel points in the RGB image can be obtained. Therefore, for each pixel point in the wrapped image region, the three-dimensional point corresponding to the pixel point can be determined from the target three-dimensional point cloud, and the three-dimensional points corresponding to all the pixel points in the wrapped image region can constitute an initial wrapped point cloud corresponding to the wrapped image region. Obviously, for the K wrapped image regions in the target RGB image, K initial wrapped point clouds corresponding to the K wrapped image regions can be obtained.

[0093] In step S22, the K initial wrapped point clouds are segmented according to the neighborhood features corresponding to the K initial wrapped point clouds, to obtain M target wrapped point clouds. M can be greater than or equal to K.

[0094] For example, for each initial wrapped point cloud, the initial wrapped point cloud can be segmented based on the neighborhood features corresponding to the initial wrapped point cloud, to obtain a target wrapped point cloud corresponding to the initial wrapped point cloud, and all the target wrapped point clouds corresponding to the K initial wrapped point clouds correspond to the M target wrapped point clouds.

[0095] In a possible implementation, for each initial wrapped point cloud, the initial wrapped point cloud can be segmented based on the neighborhood features (i.e., concave-convex features) corresponding to the initial wrapped point cloud, by using a method of hyper-voxel segmentation and clustering, to obtain a target wrapped point cloud corresponding to the initial wrapped point cloud.

[0096] For example, all the three-dimensional points in the initial wrapped point cloud are mapped to a voxel three-dimensional grid, to obtain a plurality of hyper-voxels corresponding to the initial wrapped point cloud. Wherein, a point cloud is a collection of three-dimensional points, a voxel is a pixel in a three-dimensional space, and a hyper-voxel is a collection, and the elements of the collection are "volumes", which are essentially small blocks.

[0097] After obtaining the plurality of super voxels, the plurality of super voxels can be clustered based on the neighborhood features corresponding to each super voxel to obtain N cluster groups, N being a positive integer, and each cluster group can include at least one super voxel. In the clustering of the plurality of super voxels based on the neighborhood features corresponding to each super voxel, for each super voxel, if the neighborhood feature corresponding to the super voxel is the same as the neighborhood feature corresponding to the neighboring super voxel of the super voxel, the super voxel and the neighboring super voxel of the super voxel belong to the same cluster group; if the neighborhood feature corresponding to the super voxel is different from the neighborhood feature corresponding to the neighboring super voxel of the super voxel, the super voxel and the neighboring super voxel of the super voxel belong to different cluster groups. The neighborhood feature can be a concave feature or a convex feature. For example, after obtaining super voxel a1, super voxel a2, super voxel a3 and super voxel a4, it is assumed that super voxel a1 is adjacent to super voxel a2, super voxel a2 is adjacent to super voxel a3, super voxel a3 is adjacent to super voxel a4, the neighborhood feature corresponding to super voxel a1 is a concave feature, the neighborhood feature corresponding to super voxel a2 is a concave feature, the neighborhood feature corresponding to super voxel a3 is a convex feature, and the neighborhood feature corresponding to super voxel a4 is a convex feature. Then, the neighborhood feature corresponding to super voxel a1 is the same as the neighborhood feature corresponding to super voxel a2, and super voxel a1 and super voxel a2 belong to the same cluster group. The neighborhood feature corresponding to super voxel a2 is different from the neighborhood feature corresponding to super voxel a3, and super voxel a2 and super voxel a3 belong to different cluster groups. The neighborhood feature corresponding to super voxel a3 is the same as the neighborhood feature corresponding to super voxel a4, and super voxel a3 and super voxel a4 belong to the same cluster group. As a result, two cluster groups can be obtained, the first cluster group can include super voxel a1 and super voxel a2, and the second cluster group can include super voxel a3 and super voxel a4.

[0098] After obtaining the N cluster groups, N target wrapped point clouds corresponding to the initial wrapped point cloud can be generated based on the N cluster groups. For example, the first cluster group includes super voxel a1 and super voxel a2, and the second cluster group includes super voxel a3 and super voxel a4. The three-dimensional points corresponding to super voxel a1, the three-dimensional points corresponding to super voxel a2, the three-dimensional points corresponding to super voxel a3 and the three-dimensional points corresponding to super voxel a4 are determined from the initial wrapped point cloud. Based on this, the three-dimensional points corresponding to super voxel a1 and the three-dimensional points corresponding to super voxel a2 constitute the target wrapped point cloud corresponding to the first cluster group, and the three-dimensional points corresponding to super voxel a3 and the three-dimensional points corresponding to super voxel a4 constitute the target wrapped point cloud corresponding to the second cluster group, thereby obtaining two target wrapped point clouds.

[0099] As can be seen from the above, for each initial wrapped point cloud, the target wrapped point cloud corresponding to the initial wrapped point cloud can be obtained by performing secondary segmentation on the initial wrapped point cloud.

[0100] Fourth, wrapped classification.

[0101] In the secondary segmentation process of the three-dimensional point cloud, M target parcel point clouds can be obtained, and in the parcel classification process, M image regions corresponding to the M target parcel point clouds can be determined from the target RGB image, that is, the M target parcel point clouds and the M image regions are one-to-one corresponding. After obtaining the M image regions, for each image region, the image region is input to the trained classification network model (the classification network model can be a deep learning classification network model or a neural network model, and the classification network model is not limited), and the classification network model classifies based on the image region to obtain the type of the parcel, which can be a hard parcel type or a soft parcel type. The type of the parcel can be used as an attribute of the target parcel point cloud corresponding to the image region, that is, the type of the parcel corresponding to the target parcel point cloud is obtained.

[0102] In summary, in the parcel classification process, the type of the parcel corresponding to each target parcel point cloud can be obtained, which can be a hard parcel type or a soft parcel type, and the type of the parcel is not limited.

[0103] For example, the image region corresponding to each target parcel point cloud can be determined, and each image region can be input to the classification network model separately, and the classification network model can classify based on each image region, rather than inputting the target RGB image to the classification network model as a whole. In this way, the mutual influence between different parcels can be avoided, the influence of the environment can be avoided, and all image regions can be scaled to the same size, avoiding the influence of the camera category and the erection scheme, so as to improve the accuracy of classification.

[0104] Fifth, the calculation of the grabbing point of the parcel point cloud.

[0105] In the secondary segmentation process of the three-dimensional point cloud, M target parcel point clouds can be obtained, and in the parcel point cloud grabbing point calculation process, for each target parcel point cloud, point cloud plane fitting can be performed. In order to obtain a higher parcel grabbing success rate, the clustering center point of the maximum inscribed circle can be used as the grabbing point, so that the grabbing point has the maximum and balanced distance to each boundary point of the plane contour, which can ensure that the robot suction cup has the maximum contact surface with the parcel plane when grabbing, and ensure the grabbing success rate.

[0106] Exemplarily, for the process of calculating the grasping point of the wrapped point cloud, the process can include: for each target wrapped point cloud, mapping all three-dimensional points in the target wrapped point cloud to two-dimensional points (i.e., performing point cloud plane fitting), and selecting a two-dimensional point (such as the cluster center point of the maximum inscribed circle) from all two-dimensional points as the target two-dimensional point. After obtaining the target two-dimensional point, the three-dimensional point corresponding to the target two-dimensional point can be determined from all three-dimensional points of the target wrapped point cloud, and the three-dimensional point is referred to as a candidate three-dimensional point. After obtaining the candidate three-dimensional point, the candidate three-dimensional point can be determined as the three-dimensional grasping point corresponding to the target wrapped point cloud.

[0107] In the process of calculating the grasping point of the wrapped point cloud, for each target wrapped point cloud, the candidate three-dimensional point corresponding to the target wrapped point cloud can be determined as the three-dimensional grasping point corresponding to the target wrapped point cloud.

[0108] In the process of calculating the grasping point of the wrapped point cloud, for each target wrapped point cloud, the candidate three-dimensional point corresponding to the target wrapped point cloud can be determined as the three-dimensional grasping point corresponding to the target wrapped point cloud.

[0109] Sixth, perturbation of the grasping point.

[0110] In the secondary segmentation process of the three-dimensional point cloud, since the top surface and the side surface of a single wrapped are segmented by the secondary segmentation, i.e., each surface is a wrapped, each wrapped can be segmented into multiple grasping surfaces, i.e., the M target wrapped point clouds include target wrapped point clouds of multiple grasping surfaces of each wrapped, and each grasping surface can have a candidate three-dimensional point (i.e., a three-dimensional grasping point). However, when the robot grasps the candidate three-dimensional point, there can be a collision risk or a grasping risk. In order to further improve the grasping success rate, in the perturbation process of the grasping point, the candidate three-dimensional point can be perturbed, multiple perturbed three-dimensional points can be generated near the candidate three-dimensional point, such as angle perturbation and / or translation perturbation, and the robot's suction cup can be simulated to try to grasp according to different poses, so as to obtain multiple perturbed three-dimensional points corresponding to the candidate three-dimensional point.

[0111] Exemplarily, for the process of calculating the grasping point of the wrapped point cloud, the process can include: for each target wrapped point cloud, mapping all three-dimensional points in the target wrapped point cloud to two-dimensional points (i.e., performing point cloud plane fitting), and selecting a two-dimensional point (such as the cluster center point of the maximum inscribed circle) from all two-dimensional points as the target two-dimensional point. After obtaining the target two-dimensional point, the three-dimensional point corresponding to the target two-dimensional point can be determined from all three-dimensional points of the target wrapped point cloud, and the three-dimensional point is referred to as a candidate three-dimensional point. After obtaining the candidate three-dimensional point, the candidate three-dimensional point can be determined as the three-dimensional grasping point corresponding to the target wrapped point cloud.

[0112] For example, after obtaining the candidate three-dimensional point corresponding to the target wrapped point cloud, the candidate three-dimensional point can be subjected to angle perturbation, that is, the candidate three-dimensional point is offset by a certain angle to obtain a perturbed three-dimensional point. Since the candidate three-dimensional point has coordinates and a direction, when the candidate three-dimensional point is offset by a certain angle, it is equivalent to offsetting the direction of the candidate three-dimensional point by a certain angle, thereby obtaining a new three-dimensional point, that is, the perturbed three-dimensional point.

[0113] After obtaining the candidate three-dimensional point corresponding to the target wrapped point cloud, the candidate three-dimensional point can be subjected to translation perturbation, that is, the candidate three-dimensional point is offset by a certain distance to obtain a perturbed three-dimensional point. Since the candidate three-dimensional point has coordinates and a direction, when the candidate three-dimensional point is offset by a certain distance, it is equivalent to offsetting the coordinates of the candidate three-dimensional point by a certain distance, thereby obtaining a new three-dimensional point, that is, the perturbed three-dimensional point.

[0114] After obtaining the candidate three-dimensional point corresponding to the target wrapped point cloud, the candidate three-dimensional point can be subjected to angle perturbation and translation perturbation, that is, the candidate three-dimensional point is offset by a certain angle and distance to obtain a perturbed three-dimensional point.

[0115] As described above, by perturbing the candidate three-dimensional point multiple times (such as angle perturbation and / or translation perturbation), multiple perturbed three-dimensional points corresponding to the candidate three-dimensional point can be obtained. In this way, the candidate three-dimensional point and the multiple perturbed three-dimensional points can be determined as three-dimensional grasping points corresponding to the target wrapped point cloud.

[0116] Since the candidate three-dimensional point and the multiple perturbed three-dimensional points have different positions and / or angles, the suction cup of the robot grasps the candidate three-dimensional point according to pose A, the suction cup of the robot grasps the first perturbed three-dimensional point according to pose B, the suction cup of the robot grasps the second perturbed three-dimensional point according to pose C, and so on. Obviously, the candidate three-dimensional point and the multiple perturbed three-dimensional points correspond to different poses of the robot, that is, the suction cup of the robot grasps the candidate three-dimensional point and the multiple perturbed three-dimensional points according to different poses, thereby introducing the structural information of the suction cup.

[0117] Seventh, grasping point sorting.

[0118] In the perturbation process of the grasping point, for each target wrapped point cloud, multiple three-dimensional grasping points (candidate three-dimensional point and multiple perturbed three-dimensional points) corresponding to the target wrapped point cloud can be obtained. In this way, M target wrapped point clouds correspond to multiple three-dimensional grasping points. In the grasping point sorting process, all three-dimensional grasping points can be traversed, and all three-dimensional grasping points can be sorted based on the target score corresponding to each three-dimensional grasping point. Based on the sorting result, the three-dimensional grasping point with the highest overlap rate of the suction cup and the plane, which does not collide with other packages, and which does not grasp other packages, is selected. By determining the target score corresponding to each three-dimensional grasping point, the best grasping point can be calculated according to the suction cup structure, the grasping success rate, and the grasping probability.

[0119] For example, the ranking process for the grasping points can include the following steps:

[0120] In step S31, for each three-dimensional grasping point, which can correspond to multiple grasping parameters, the grasping score corresponding to each grasping parameter of the three-dimensional grasping point is determined.

[0121] For example, the multiple grasping parameters corresponding to the three-dimensional grasping point include but are not limited to at least one of the following:

[0122] The height of the package corresponding to the three-dimensional grasping point (i.e., the grasping parameter is the height parameter). For example, since the three-dimensional grasping point is the three-dimensional grasping point corresponding to the target package point cloud, and the target package point cloud corresponds to the package on the package supply table, and the height of each package on the package supply table can be obtained, the height of the package corresponding to the target package point cloud can be taken as the height of the package corresponding to the three-dimensional grasping point.

[0123] The area of the package corresponding to the three-dimensional grasping point (i.e., the grasping parameter is the area parameter). For example, since the three-dimensional grasping point is the three-dimensional grasping point corresponding to the target package point cloud, and the target package point cloud corresponds to the package on the package supply table, and the area of each package on the package supply table can be obtained, the area of the package corresponding to the target package point cloud can be taken as the area of the package corresponding to the three-dimensional grasping point.

[0124] The angle between the three-dimensional grasping point and the normal of the plane of the package supply table (i.e., the grasping parameter is the angle parameter). For example, since the three-dimensional grasping point has coordinates and a direction, the angle pointed by the direction of the three-dimensional grasping point is the angle between the three-dimensional grasping point and the normal of the plane of the package supply table.

[0125] The distance between the three-dimensional grasping point and the robot (i.e., the grasping parameter is the distance parameter). For example, since the three-dimensional grasping point has coordinates and a direction, i.e., the three-dimensional grasping point corresponds to a certain position point in the physical space, the distance between the three-dimensional grasping point and the robot can be obtained.

[0126] The type of the package corresponding to the three-dimensional grasping point (i.e., the grasping parameter is the package type parameter). For example, in the package classification process, the type of the package corresponding to each target package point cloud can be obtained, and the type of the package can be a hard package type or a soft package type. Since the three-dimensional grasping point is the three-dimensional grasping point corresponding to the target package point cloud, the type of the package corresponding to the target package point cloud can be taken as the type of the package corresponding to the three-dimensional grasping point, and the type of the package corresponding to the three-dimensional grasping point is a hard package type or a soft package type.

[0127] The neighborhood feature corresponding to the three-dimensional grasping point (i.e., the grasping parameter is the neighborhood feature parameter). For example, in the secondary segmentation process of the three-dimensional point cloud, each cluster group can correspond to a target package point cloud, and the cluster group can include multiple hyper-voxels corresponding to the same neighborhood feature. The neighborhood feature of the hyper-voxels in the cluster group can be taken as the neighborhood feature corresponding to the target package point cloud. The neighborhood feature can be a convex feature or a concave feature. Since the three-dimensional grasping point is the three-dimensional grasping point corresponding to the target package point cloud, the neighborhood feature corresponding to the three-dimensional grasping point can be taken as the neighborhood feature corresponding to the three-dimensional grasping point. The neighborhood feature corresponding to the three-dimensional grasping point can be a convex feature or a concave feature.

[0128] Of course, the above is only a few examples of the grasping parameter, and the grasping parameter is not limited in the embodiment.

[0129] For example, for each grasping parameter corresponding to the three-dimensional grasping point, a grasping score corresponding to the grasping parameter can be determined. For example, a grasping score corresponding to the height parameter can be determined. The greater the height of the package corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the height parameter. The smaller the height of the package corresponding to the three-dimensional grasping point, the smaller the grasping score corresponding to the height parameter. In summary, the grasping score is proportional to the height of the package corresponding to the three-dimensional grasping point, and the grasping score is not limited.

[0130] For example, a grasping score corresponding to the area parameter can be determined. The greater the area of the package corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the area parameter. The smaller the area of the package corresponding to the three-dimensional grasping point, the smaller the grasping score corresponding to the area parameter. In summary, the grasping score corresponding to the area parameter is proportional to the area of the package corresponding to the three-dimensional grasping point, and the grasping score is not limited.

[0131] For example, a grasping score corresponding to the angle parameter can be determined. The smaller the angle between the three-dimensional grasping point and the normal of the package table plane, the greater the grasping score corresponding to the angle parameter. The greater the angle between the three-dimensional grasping point and the normal of the package table plane, the smaller the grasping score corresponding to the angle parameter. In summary, the grasping score corresponding to the angle parameter is inversely proportional to the angle between the three-dimensional grasping point and the normal of the package table plane, and the determination method of the grasping score corresponding to the angle parameter is not limited.

[0132] For example, a grasping score corresponding to the distance parameter can be determined. The smaller the distance between the three-dimensional grasping point and the robot, the greater the grasping score corresponding to the distance parameter. The greater the distance between the three-dimensional grasping point and the robot, the smaller the grasping score corresponding to the distance parameter. In summary, the grasping score corresponding to the distance parameter is inversely proportional to the distance, and the determination method of the grasping score is not limited.

[0133] For example, the grabbing score corresponding to the package type parameter can be determined, the grabbing score corresponding to the package type parameter is determined as a first score (configured according to experience) when the type of the package corresponding to the three-dimensional grabbing point is a hard package type, the grabbing score corresponding to the package type parameter is determined as a second score (configured according to experience) when the type of the package corresponding to the three-dimensional grabbing point is a soft package type, and the first score is greater than the second score. In summary, the grabbing score corresponding to the hard package type is greater than the grabbing score corresponding to the soft package type.

[0134] For example, the grabbing score corresponding to the neighborhood feature parameter can be determined, the grabbing score corresponding to the neighborhood feature parameter is determined as a third score (configured according to experience) when the neighborhood feature corresponding to the three-dimensional grabbing point is a convex feature, the grabbing score corresponding to the neighborhood feature parameter is determined as a fourth score (configured according to experience) when the neighborhood feature corresponding to the three-dimensional grabbing point is a concave feature, and the third score is greater than the fourth score. In summary, the grabbing score corresponding to the convex feature is greater than the grabbing score corresponding to the concave feature.

[0135] In summary, for each three-dimensional grabbing point, the grabbing score corresponding to each grabbing parameter corresponding to the three-dimensional grabbing point can be determined, for example, the grabbing score corresponding to the height parameter, the grabbing score corresponding to the area parameter, the grabbing score corresponding to the angle parameter, the grabbing score corresponding to the distance parameter, the grabbing score corresponding to the package type parameter, and the grabbing score corresponding to the neighborhood feature parameter can be determined.

[0136] Step S32, for each three-dimensional grabbing point, based on the grabbing score corresponding to each grabbing parameter corresponding to the three-dimensional grabbing point, the target score corresponding to the three-dimensional grabbing point is determined, for example, the grabbing score corresponding to each grabbing parameter is weighted to obtain the target score corresponding to the three-dimensional grabbing point.

[0137] In a possible implementation, taking the three-dimensional grabbing point corresponding height parameter, area parameter, angle parameter, distance parameter, package type parameter and neighborhood feature parameter as examples, the target score can be determined by using the following formula: S=S1*W1+S2* W2+S3*W3+S4*W4+S5*W5+S6*W6. In the above formula, S represents the target score corresponding to the three-dimensional grabbing point, S1 represents the grabbing score corresponding to the height parameter, S2 represents the grabbing score corresponding to the area parameter, S3 represents the grabbing score corresponding to the angle parameter, S4 represents the grabbing score corresponding to the distance parameter, S5 represents the grabbing score corresponding to the package type parameter, and S6 represents the grabbing score corresponding to the neighborhood feature parameter. W1 represents the weight coefficient corresponding to the height parameter, W2 represents the weight coefficient corresponding to the area parameter, W3 represents the weight coefficient corresponding to the angle parameter, W4 represents the weight coefficient corresponding to the distance parameter, W5 represents the weight coefficient corresponding to the package type parameter, and W6 represents the weight coefficient corresponding to the neighborhood feature parameter. W1, W2, W3, W4, W5 and W6 can be configured according to experience.

[0138] Step S33, based on the target score corresponding to each three-dimensional grabbing point, the three-dimensional grabbing points are sorted in order of high to low target score or in order of low to high target score.

[0139] Step S34, when the three-dimensional grabbing points are sorted in order of high to low target score, the first three-dimensional grabbing point is selected as the target grabbing point. Or, when the three-dimensional grabbing points are sorted in order of low to high target score, the last three-dimensional grabbing point is selected as the target grabbing point.

[0140] Step S35, the robot is controlled to perform a grabbing operation on the package on the package supply table based on the target grabbing point, that is, the robot is guided to perform a grabbing operation on the package on the package supply table by the mechanical hand, that is, the mechanical hand performs a grabbing operation on the package on the package supply table by the suction cup. The package grabbing process in this embodiment is not limited.

[0141] After the robot performs a grabbing operation on the package on the package supply table based on the target grabbing point, the processes of adaptive window deep learning rough segmentation, three-dimensional point cloud secondary segmentation, package classification, grabbing point calculation of package point cloud, grabbing point disturbance, and grabbing point sorting need to be performed again to obtain another target grabbing point. The package on the package supply table is re-performed based on the target grabbing point. The grabbing operation is performed in this way.

[0142] In a possible implementation, referring to FIG. 1, the package grabbing method can include the following steps. Figure 5

[0143] Step 501, according to the application requirement, an RGBD camera is erected on the package supply table. ​

[0144] Step 502, empty the parcel on the parcel table, and obtain an RGBD image.

[0145] Step 503, manually draw the parcel table area.

[0146] Step 504, perform system calibration.

[0147] Step 505, obtain the workbench coordinate system and the three-dimensional bounding box through system calibration.

[0148] Step 506, obtain an RGBD image during normal operation.

[0149] Step 507, perform background filtering based on the system calibration result (i.e., the three-dimensional bounding box).

[0150] Step 508, perform coarse segmentation of the parcel on the filtered RGB image.

[0151] Step 509, obtain the three-dimensional point cloud of each coarse segmentation parcel, and perform secondary segmentation based on concave-convex properties.

[0152] Step 510, extract the RGBD information of each parcel according to the secondary segmentation result, use a classification network based on the RGBD information to classify the parcel material, and obtain the type of the parcel.

[0153] Step 511, extract the maximum inscribed circle clustering center of the planar point cloud of the parcel as a candidate grasping point.

[0154] Step 512, perform angle perturbation and / or translation perturbation on the candidate grasping point, introduce the structural information of the suction cup, generate multiple perturbed three-dimensional points corresponding to the candidate grasping point, and determine the candidate three-dimensional point and all perturbed three-dimensional points as three-dimensional grasping points, i.e., multiple three-dimensional grasping points can be obtained.

[0155] Step 513, perform score calculation and sorting on each three-dimensional grasping point, and obtain the best grasping point and grasping pose by comprehensively considering the grasping success probability, collision, and grasping probability.

[0156] It can be seen from the above technical solutions that, in the embodiment of the application, after determining K parcel image regions based on the target RGB image corresponding to the parcel supply table, K initial parcel point clouds corresponding to the K parcel image regions can be determined from the target three-dimensional point cloud corresponding to the parcel supply table, and the K initial parcel point clouds are segmented to obtain M target parcel point clouds, that is, the target grabbing point is determined based on the M target parcel point clouds, instead of being determined based on the K parcel image regions, so that a suitable grabbing point (i.e., the best grabbing point) can be found, thereby controlling the robot to perform a grabbing operation on the parcel based on the suitable grabbing point. In a dense parcel scene, a high-robustness and high-success-rate grabbing effect can be obtained, and the grabbing success rate is improved. A parcel segmentation and grabbing point determination method based on RGBD visual guidance is proposed, the three-dimensional bounding box obtained through system calibration effectively filters out interference background information, improves the generalization ability of instance segmentation to new scenes, the accuracy and robustness of the segmentation result. Through the secondary segmentation of the instance segmentation result by the three-dimensional point cloud, the probability of failure of the same color parcel segmentation can be reduced; through the classification of the parcel alone RGBD information, the classification accuracy can be improved without being affected by the scene and the erection scheme; through the point cloud plane segmentation of the parcel and the selection of the largest inscribed circle clustering center in the point cloud plane as the grabbing point, the precision of the grabbing point is improved; through the perturbation of the grabbing point and the introduction of the sucker structure information to generate a candidate pose, the best grabbing point and the grabbing pose are searched through iteration under the guidance of indicators such as the grabbing success probability, collision, and grabbing probability, and the grabbing effect is effectively improved.

[0157] Based on the same application concept as the above method, the embodiment of the application proposes a parcel grabbing device, as shown in Figure 6 The device can include:

[0158] The acquisition module 61 is configured to acquire a target RGB image corresponding to a parcel supply table and acquire a target three-dimensional point cloud corresponding to the parcel supply table.

[0159] The determination module 62 is configured to determine K parcel image regions based on the target RGB image and determine K initial parcel point clouds corresponding to the K parcel image regions from the target three-dimensional point cloud.

[0160] The segmentation module 63 is configured to segment the K initial parcel point clouds according to target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds, where M is greater than or equal to K.

[0161] The processing module 64 is configured to determine a three-dimensional grasping point corresponding to each target parcel point cloud, sort all the three-dimensional grasping points based on grasping parameters corresponding to each three-dimensional grasping point, and select a three-dimensional grasping point from all the three-dimensional grasping points as a target grasping point based on a sorting result; wherein the target grasping point is used to enable the robot to perform a grasping operation on a parcel on the parcel supply table based on the target grasping point.

[0162] For example, the acquisition module 61 is configured to acquire the target RGB image corresponding to the parcel supply table by: collecting an initial RGB image corresponding to the parcel supply table through an RGB camera; selecting an RGB image of a parcel supply table region from the initial RGB image; and determining the target RGB image based on the RGB image of the parcel supply table region. The acquisition module 61 is configured to acquire the target three-dimensional point cloud corresponding to the parcel supply table by: collecting an initial three-dimensional point cloud corresponding to the parcel supply table through a depth camera; determining a three-dimensional point cloud of the parcel supply table region based on the initial three-dimensional point cloud; and determining the target three-dimensional point cloud based on the three-dimensional point cloud of the parcel supply table region.

[0163] For example, the acquisition module 61 is configured to determine the target three-dimensional point cloud based on the three-dimensional point cloud of the parcel supply table region by: determining the three-dimensional point cloud of the parcel supply table region as the target three-dimensional point cloud; or selecting a target three-dimensional point from a plurality of initial three-dimensional points of the three-dimensional point cloud of the parcel supply table region, wherein the height of the target three-dimensional point is greater than a preset height threshold; and generating the target three-dimensional point cloud based on all the target three-dimensional points. The acquisition module 61 is configured to determine the target RGB image based on the RGB image of the parcel supply table region by: determining the RGB image of the parcel supply table region as the target RGB image; or determining a target pixel point corresponding to the target three-dimensional point from the RGB image of the parcel supply table region, and generating the target RGB image based on the target pixel point.

[0164] For example, the segmentation module 63 is configured to segment the K initial parcel point clouds based on target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds by: for each initial parcel point cloud, mapping all three-dimensional points in the initial parcel point cloud to a voxel three-dimensional grid to obtain a plurality of hyper voxels corresponding to the initial parcel point cloud; clustering all the hyper voxels based on target features corresponding to each hyper voxel to obtain N clustering groups, wherein each clustering group includes at least one hyper voxel; for each hyper voxel, if a target feature corresponding to the hyper voxel is the same as a target feature corresponding to a neighboring hyper voxel of the hyper voxel, the hyper voxel and the neighboring hyper voxel belong to the same clustering group; if the target feature corresponding to the hyper voxel is different from the target feature corresponding to the neighboring hyper voxel of the hyper voxel, the hyper voxel and the neighboring hyper voxel belong to different clustering groups; wherein the target feature is a concave feature or a convex feature; and generating N target parcel point clouds corresponding to the initial parcel point cloud based on the N clustering groups.

[0165] For example, when determining the three-dimensional grasping point corresponding to each target parcel point cloud, the processing module 64 is specifically configured to: for each target parcel point cloud, map the three-dimensional points in the target parcel point cloud to two-dimensional points, select a two-dimensional point from the mapped two-dimensional points as a target two-dimensional point, and determine a candidate three-dimensional point corresponding to the target two-dimensional point from all three-dimensional points in the target parcel point cloud; determine the candidate three-dimensional point as the three-dimensional grasping point corresponding to the target parcel point cloud; or perform angle perturbation and / or translation perturbation on the candidate three-dimensional point to obtain a plurality of perturbed three-dimensional points, and determine the candidate three-dimensional point and the plurality of perturbed three-dimensional points as the three-dimensional grasping points corresponding to the target parcel point cloud.

[0166] For example, when sorting all the three-dimensional grasping points based on the grasping parameters corresponding to each three-dimensional grasping point, the processing module 64 is specifically configured to: for each three-dimensional grasping point, the three-dimensional grasping point corresponds to a plurality of grasping parameters, determine a target score corresponding to the three-dimensional grasping point based on the grasping score corresponding to each grasping parameter corresponding to the three-dimensional grasping point; sort all the three-dimensional grasping points in order of target score from high to low or from low to high based on the target score corresponding to each three-dimensional grasping point; wherein the plurality of grasping parameters corresponding to the three-dimensional grasping point include at least one of the following: the height of the parcel corresponding to the three-dimensional grasping point, the area of the parcel corresponding to the three-dimensional grasping point, the angle between the three-dimensional grasping point and the normal of the parcel table plane, the type of the parcel corresponding to the three-dimensional grasping point, the target feature corresponding to the three-dimensional grasping point, and the distance between the three-dimensional grasping point and the robot.

[0167] For example, the greater the height of the parcel corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the height parameter; the greater the area of the parcel corresponding to the three-dimensional grasping point, the greater the grasping score corresponding to the area parameter; the smaller the angle between the three-dimensional grasping point and the normal of the parcel table plane, the greater the grasping score corresponding to the angle parameter; the smaller the distance between the three-dimensional grasping point and the robot, the greater the grasping score corresponding to the distance parameter; the type of the parcel corresponding to the three-dimensional grasping point is a hard package type or a soft package type, and the grasping score corresponding to the hard package type is greater than the grasping score corresponding to the soft package type; the target feature corresponding to the three-dimensional grasping point is a convex feature or a concave feature, and the grasping score corresponding to the convex feature is greater than the grasping score corresponding to the concave feature.

[0168] Based on the same application concept as the above method, an electronic device is provided in the embodiments of the present application, as shown in Figure 7As shown, the electronic device can include a processor 71 and a machine readable storage medium 72 having stored machine executable instructions capable of being executed by the processor 71; the processor 71 is configured to execute the machine executable instructions to implement the parcel grabbing method disclosed in the above examples of the present application.

[0169] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium having stored computer instructions, which, when executed by a processor, can implement the parcel grabbing method disclosed in the above examples of the present application.

[0170] The machine readable storage medium can be any electronic, magnetic, optical, or other physical storage device capable of storing information such as computer readable instructions, data, and the like. For example, the machine readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard drive), a solid state drive, any type of storage disk (such as a floppy disk, a DVD, etc.), or similar storage medium, or a combination thereof.

[0171] The systems, apparatuses, modules or units illustrated in the above embodiments can be implemented by computer entities or products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0172] For the convenience of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0173] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0174] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0175] Also, these computer program instructions can be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0176] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0177] The embodiments of the present application described above are merely used to illustrate the present application and should not be used to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall within the scope of claims of the present application.

Claims

1. A package grasping method characterized by, The method comprises: obtaining a target RGB image corresponding to the parcel table, and obtaining a target three-dimensional point cloud corresponding to the parcel table; determining K parcel image regions based on the target RGB image, K being a positive integer; determining K initial parcel point clouds corresponding to the K parcel image regions from the target three-dimensional point cloud, segmenting the K initial parcel point clouds according to target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds, M being greater than or equal to K; determining a three-dimensional grabbing point corresponding to each target parcel point cloud, sorting all three-dimensional grabbing points based on grabbing parameters corresponding to each three-dimensional grabbing point, and selecting a three-dimensional grabbing point as a target grabbing point from all three-dimensional grabbing points based on a sorting result; wherein the target grabbing point is used to enable a robot to perform a grabbing operation on a parcel on the parcel table based on the target grabbing point; wherein the determination of the three-dimensional grabbing point corresponding to each target parcel point cloud comprises: for each target parcel point cloud, determining a candidate three-dimensional point from three-dimensional points in the target parcel point cloud; determining the candidate three-dimensional point as the three-dimensional grabbing point corresponding to the target parcel point cloud; or, performing angle perturbation and / or translation perturbation on the candidate three-dimensional point to obtain a plurality of perturbed three-dimensional points, and determining the candidate three-dimensional point and the plurality of perturbed three-dimensional points as the three-dimensional grabbing points corresponding to the target parcel point cloud.

2. The method of claim 1, wherein the obtaining of the target RGB image corresponding to the parcel table comprises: acquiring an initial RGB image corresponding to the parcel table through an RGB camera; selecting an RGB image of a parcel table region from the initial RGB image; and determining a target RGB image based on the RGB image of the parcel table region; the obtaining of the target three-dimensional point cloud corresponding to the parcel table comprises: acquiring an initial three-dimensional point cloud corresponding to the parcel table through a depth camera; determining a three-dimensional point cloud of a parcel table region based on the initial three-dimensional point cloud; and determining a target three-dimensional point cloud based on the three-dimensional point cloud of the parcel table region.

3. The method of claim 2, wherein the determination of the target three-dimensional point cloud based on the three-dimensional point cloud of the parcel table region comprises: determining the three-dimensional point cloud of the parcel table region as the target three-dimensional point cloud; or the three-dimensional point cloud of the parcel table region comprises a plurality of initial three-dimensional points, a target three-dimensional point is selected from the plurality of initial three-dimensional points, the height of the target three-dimensional point is greater than a preset height threshold; and a target three-dimensional point cloud is generated based on all target three-dimensional points; the determination of the target RGB image based on the RGB image of the parcel table region comprises: determining the RGB image of the parcel table region as the target RGB image; or determining a target pixel point corresponding to a target three-dimensional point from the RGB image of the parcel table region, and generating a target RGB image based on the target pixel point.

4. The method of claim 1, wherein the segmentation of the K initial parcel point clouds according to target features corresponding to the K initial parcel point clouds to obtain M target parcel point clouds comprises: For each initial wrapping point cloud, all three-dimensional points in the initial wrapping point cloud are mapped to a voxel three-dimensional grid to obtain a plurality of hyper-voxels corresponding to the initial wrapping point cloud; All hyper-voxels are clustered based on target features corresponding to each hyper-voxel to obtain N clustering groups, N being a positive integer, and the clustering groups including at least one hyper-voxel; wherein for each hyper-voxel, if the target feature corresponding to the hyper-voxel is the same as the target feature corresponding to the adjacent hyper-voxel of the hyper-voxel, the hyper-voxel and the adjacent hyper-voxel of the hyper-voxel belong to the same clustering group, and if the target feature corresponding to the hyper-voxel is different from the target feature corresponding to the adjacent hyper-voxel of the hyper-voxel, the hyper-voxel and the adjacent hyper-voxel of the hyper-voxel belong to different clustering groups; wherein the target feature is a concave feature or a convex feature; N target wrapping point clouds corresponding to the initial wrapping point cloud are generated based on the N clustering groups.

5. The method of claim 1, wherein, The sorting of all three-dimensional grabbing points based on the grabbing parameters corresponding to each three-dimensional grabbing point includes: For each three-dimensional grabbing point, the three-dimensional grabbing point corresponds to a plurality of grabbing parameters, and a target score corresponding to the three-dimensional grabbing point is determined based on a grabbing score corresponding to each grabbing parameter corresponding to the three-dimensional grabbing point; All three-dimensional grabbing points are sorted in an order from high to low of target scores or in an order from low to high of target scores based on the target score corresponding to each three-dimensional grabbing point; Wherein, the plurality of grabbing parameters corresponding to the three-dimensional grabbing point include at least one of the following: a height of a package corresponding to the three-dimensional grabbing point, an area of the package corresponding to the three-dimensional grabbing point, an angle between the three-dimensional grabbing point and a normal of a table plane, a type of the package corresponding to the three-dimensional grabbing point, a target feature corresponding to the three-dimensional grabbing point, and a distance between the three-dimensional grabbing point and a robot.

6. The method of claim 5, wherein, the greater the height of the package corresponding to the three-dimensional grabbing point, the greater the grabbing score corresponding to the height parameter; the greater the area of the package corresponding to the three-dimensional grabbing point, the greater the grabbing score corresponding to the area parameter; the smaller the angle between the three-dimensional grabbing point and the normal of the table plane, the greater the grabbing score corresponding to the angle parameter; the smaller the distance between the three-dimensional grabbing point and the robot, the greater the grabbing score corresponding to the distance parameter; the type of the package corresponding to the three-dimensional grabbing point is a hard package type or a soft package type, and the grabbing score corresponding to the hard package type is greater than the grabbing score corresponding to the soft package type; the target feature corresponding to the three-dimensional grabbing point is a convex feature or a concave feature, and the grabbing score corresponding to the convex feature is greater than the grabbing score corresponding to the concave feature.

7. A package gripping device, characterized by The device includes: an acquisition module configured to acquire a target RGB image corresponding to a package table and acquire a target three-dimensional point cloud corresponding to the package table; a determination module configured to determine K package image regions based on the target RGB image, and determine K initial wrapping point clouds corresponding to the K package image regions from the target three-dimensional point cloud; a segmentation module configured to segment the K initial wrapping point clouds according to target features corresponding to the K initial wrapping point clouds to obtain M target wrapping point clouds, M being greater than or equal to K. The processing module is configured to determine a three-dimensional grasping point corresponding to each target parcel point cloud, sort all the three-dimensional grasping points based on grasping parameters corresponding to each three-dimensional grasping point, and select a three-dimensional grasping point from all the three-dimensional grasping points as a target grasping point based on a sorting result; wherein the target grasping point is used to enable the robot to perform a grasping operation on the parcel on the parcel supply table based on the target grasping point; The processing module is configured to determine a three-dimensional grasping point corresponding to each target parcel point cloud, sort all the three-dimensional grasping points based on grasping parameters corresponding to each three-dimensional grasping point, and select a three-dimensional grasping point from all the three-dimensional grasping points as a target grasping point based on a sorting result; wherein the target grasping point is used to enable the robot to perform a grasping operation on the parcel on the parcel supply table based on the target grasping point; 8. The apparatus of claim 7, wherein wherein The acquisition module is configured to acquire an initial RGB image corresponding to the parcel supply table through an RGB camera, select an RGB image of a parcel supply table region from the initial RGB image, and determine a target RGB image based on the RGB image of the parcel supply table region; and acquire an initial three-dimensional point cloud corresponding to the parcel supply table through a depth camera, determine a three-dimensional point cloud of the parcel supply table region based on the initial three-dimensional point cloud, and determine a target three-dimensional point cloud based on the three-dimensional point cloud of the parcel supply table region. The acquisition module is configured to acquire an initial RGB image corresponding to the parcel supply table through an RGB camera, select an RGB image of a parcel supply table region from the initial RGB image, and determine a target RGB image based on the RGB image of the parcel supply table region; and acquire an initial three-dimensional point cloud corresponding to the parcel supply table through a depth camera, determine a three-dimensional point cloud of the parcel supply table region based on the initial three-dimensional point cloud, and determine a target three-dimensional point cloud based on the three-dimensional point cloud of the parcel supply table region. The acquisition module is configured to acquire an initial RGB image corresponding to the parcel supply table through an RGB camera, select an RGB image of a parcel supply table region from the initial RGB image, and determine a target RGB image based on the RGB image of the parcel supply table region; and acquire an initial three-dimensional point cloud corresponding to the parcel supply table through a depth camera, determine a three-dimensional point cloud of the parcel supply table region based on the initial three-dimensional point cloud, and determine a target three-dimensional point cloud based on the three-dimensional point cloud of the parcel supply table region. The segmentation module is configured to segment the K initial wrapping point clouds according to target features corresponding to the K initial wrapping point clouds to obtain M target wrapping point clouds, and specifically configured to: for each initial wrapping point cloud, map all three-dimensional points in the initial wrapping point cloud to a voxel three-dimensional grid to obtain a plurality of hyper voxels corresponding to the initial wrapping point cloud; cluster all hyper voxels based on target features corresponding to each hyper voxel to obtain N cluster groups, the cluster group including at least one hyper voxel; for each hyper voxel, if a target feature corresponding to the hyper voxel is the same as a target feature corresponding to a neighboring hyper voxel of the hyper voxel, the hyper voxel and the neighboring hyper voxel belong to the same cluster group, and if the target feature corresponding to the hyper voxel is different from the target feature corresponding to the neighboring hyper voxel of the hyper voxel, the hyper voxel and the neighboring hyper voxel belong to different cluster groups; wherein the target feature is a concave feature or a convex feature; and generate N target wrapping point clouds corresponding to the initial wrapping point cloud based on the N cluster groups. The processing module is configured to sort all three-dimensional grabbing points based on grabbing parameters corresponding to each three-dimensional grabbing point, and specifically configured to: for each three-dimensional grabbing point, the three-dimensional grabbing point corresponds to a plurality of grabbing parameters, determine a target score corresponding to the three-dimensional grabbing point based on a grabbing score corresponding to each grabbing parameter corresponding to the three-dimensional grabbing point; sort all three-dimensional grabbing points in an order from high to low of target scores or in an order from low to high of target scores based on the target score corresponding to each three-dimensional grabbing point; wherein the plurality of grabbing parameters corresponding to the three-dimensional grabbing point include at least one of the following: a height of a package corresponding to the three-dimensional grabbing point, an area of the package corresponding to the three-dimensional grabbing point, an angle between the three-dimensional grabbing point and a normal of a feeding table plane, a type of the package corresponding to the three-dimensional grabbing point, a target feature corresponding to the three-dimensional grabbing point, and a distance between the three-dimensional grabbing point and a robot. The height parameter corresponds to a larger grabbing score when the height of the package corresponding to the three-dimensional grabbing point is larger; the area parameter corresponds to a larger grabbing score when the area of the package corresponding to the three-dimensional grabbing point is larger; the angle parameter corresponds to a larger grabbing score when the angle between the three-dimensional grabbing point and the normal of the feeding table plane is smaller; the distance parameter corresponds to a larger grabbing score when the distance between the three-dimensional grabbing point and the robot is smaller; the type of the package corresponding to the three-dimensional grabbing point is a hard package type or a soft package type, and the grabbing score corresponding to the hard package type is larger than the grabbing score corresponding to the soft package type; the target feature corresponding to the three-dimensional grabbing point is a convex feature or a concave feature, and the grabbing score corresponding to the convex feature is larger than the grabbing score corresponding to the concave feature.

9. An electronic device, comprising: The processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; wherein the processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-6. The processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; wherein the processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-6.

Images