Target object sorting method and system, electronic equipment and computer readable storage medium

Abstract

The invention discloses a target object sorting method and system, electronic equipment and a computer readable storage medium, the method is applied to a sorting robot, the sorting robot comprises multiple terminal execution tools, and the target object sorting method comprises the steps that a scene image of a target scene is obtained, determining point cloud data of a target object in the target scene based on the scene image; wherein the scene image is matched with depth information; based on the scene image and the point cloud data, selecting a target execution tool matched with the target object from all end execution tools; and based on the point cloud data and the target execution tool, determining an execution point on the target object, and utilizing the target execution tool to abut against the execution point to sort the target object. In this way, the accuracy and stability of target object sorting can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a target object sorting method and system, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the wide application of sorting robots in industrial production and logistics sorting fields, how to effectively sort target objects in a target scene has become an important issue. In the prior art, the sorting robot has high precision when sorting target objects arranged in order and with uniform size, but the precision is greatly reduced when sorting target objects arranged out of order and with different sizes, resulting in insufficient stability of target object sorting. Therefore, how to improve the accuracy and stability of target object sorting has become a problem to be solved. SUMMARY

[0003] The technical problem solved by the present application is to provide a target object sorting method and system, an electronic device and a computer readable storage medium, which can improve the accuracy and stability of target object sorting.

[0004] To solve the above technical problem, the first aspect of the present application provides a target object sorting method, which is applied to a sorting robot, the sorting robot comprising a plurality of end execution tools, the target object sorting method comprising: acquiring a scene image of a target scene, and determining point cloud data of a target object in the target scene based on the scene image; wherein the scene image matches depth information; selecting a target execution tool matched with the target object from all the end execution tools based on the scene image and the point cloud data; determining an execution point on the target object based on the point cloud data and the target execution tool, and sorting the target object by using the target execution tool to resist the execution point.

[0005] To solve the above technical problem, the second aspect of the present application provides a target object sorting system, which is applied to a sorting robot, the sorting robot comprising a plurality of end execution tools, the target object sorting system comprising: an acquisition module, a selection module and an execution module, the acquisition module being configured to acquire a scene image of a target scene, and determine point cloud data of a target object in the target scene based on the scene image; wherein the scene image matches depth information; the selection module being configured to select a target execution tool matched with the target object from all the end execution tools based on the scene image and the point cloud data; and the execution module being configured to determine an execution point on the target object based on the point cloud data and the target execution tool, and sort the target object by using the target execution tool to resist the execution point.

[0006] To solve the above technical problems, the third aspect of the present application provides an electronic device, comprising a memory and a processor coupled with each other, wherein the memory stores program data, and the processor invokes the program data to execute the method of the first aspect.

[0007] To solve the above technical problems, the fourth aspect of the present application provides a computer readable storage medium having program data stored thereon, wherein the program data is executed by a processor to implement the method of the first aspect.

[0008] The beneficial effects of the present application are as follows: Different from the prior art, the present application acquires a scene image of a target scene, wherein the scene image matches depth information, determines a target object in the target scene based on the scene image, and determines point cloud data of the target object in a three-dimensional space. Based on the scene image and the point cloud data, a target execution tool that matches the target object is screened from all end execution tools of a sorting robot, so that the target execution tool is more suitable for the target object, reduces the probability that the target object is difficult to be accurately sorted due to disordered stacking of the target object, and improves the accuracy of sorting the target object using the target execution tool. Based on the point cloud data and the target execution tool, an execution point selected by the target execution tool when sorting the target object is determined, so that when the sizes of different target objects are inconsistent, an appropriate execution point is selected on the target object, and the target execution tool is used to bear on the execution point, so that the target execution tool exerts an external force on the target object, thereby sorting the target object and improving the accuracy and stability of sorting the target object. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0010] Figure 1 is a flowchart of an embodiment of the target object sorting method of the present application;

[0011] Figure 2 is a flowchart of another embodiment of the target object sorting method of the present application;

[0012] Figure 3 is a structural schematic diagram of an embodiment of the target object sorting system of the present application;

[0013] Figure 4 is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 5is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments, and the adaptive combination can be made between different embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0016] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is only a description of the associated relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of existence of A alone, existence of A and B together, and existence of B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" herein means two or more than two.

[0017] The target sorting method provided by the present application is applied to a sorting robot, which includes a plurality of end execution tools for sorting targets, and the corresponding execution subject is a processing unit capable of data processing, which is integrated in the robot or exists independently of the robot and interacts with the robot.

[0018] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the target sorting method of the present application, which comprises:

[0019] S101: acquiring a scene image of a target scene, and determining point cloud data of a target object in the target scene based on the scene image; wherein the scene image matches depth information.

[0020] Specifically, the scene image of the target scene is acquired, wherein the scene image matches depth information, the target object in the target scene is determined based on the scene image, and the point cloud data of the target object in the three-dimensional space is determined.

[0021] It should be noted that the target objects are stacked in the target scene, and the scene image is collected by a camera, wherein the camera can be a depth camera or a conventional camera.

[0022] In an embodiment, a scene image captured by a depth camera in a target scene is acquired, depth information matched by the scene image is determined, an image region corresponding to a target object in the scene image is extracted, at least part of pixels in the image region is converted to a three-dimensional space by using camera intrinsic parameters of the camera and the depth information, and point cloud data of the target object is obtained.

[0023] In an embodiment, a scene image captured by a camera in a target scene is acquired, depth of pixels in the scene image is estimated by using a trained depth estimation model, depth information matched by the scene image is obtained, a scene point cloud matched by the target scene is determined based on the scene image and the depth information, and point cloud data of a target object in the target scene is determined based on the scene image and the scene point cloud.

[0024] In some implementation scenarios, a scene image captured by a depth camera in a target scene and a depth image captured by a sensor on the depth camera are acquired, the scene image and the depth image are aligned to determine depth information matched by the scene image, the scene image is subjected to semantic segmentation to obtain an image region corresponding to a target object, part of pixels or all pixels in the image region are converted to a three-dimensional space by using camera intrinsic parameters of the camera and the depth information, and point cloud data of the target object is obtained.

[0025] In some implementation scenarios, an image captured by a camera in a target scene is acquired, a depth value of each pixel in the image is predicted by using a depth estimation model, and a depth image with the same resolution as the image is generated, wherein the depth estimation model is obtained by supervised training of an image labeled with a depth value. Based on the scene image and the depth information, a scene point cloud matched by the target scene is determined, a target object recognized in the scene image is acquired, the target object is aligned with the scene point cloud, and point cloud data of the target object in the target scene is determined.

[0026] S102: selecting a target execution tool matched by a target object from all end execution tools based on the scene image and the point cloud data.

[0027] Specifically, a target execution tool matched by a target object is selected from all end execution tools of a sorting robot based on the scene image and the point cloud data, so that the target execution tool is more suitable for the target object.

[0028] It can be understood that, by screening the end execution tools, the probability that the target object is difficult to be sorted accurately due to disordered stacking of the target object is reduced, thereby improving the accuracy of sorting the target object by using the target execution tool.

[0029] In an embodiment, visual features corresponding to the scene image and point cloud features corresponding to the point cloud data are acquired, an end execution tool matched by a target object is screened based on the visual features and the point cloud features, and a target execution tool matched by the target object is determined.

[0030] In an embodiment, the target object position corresponding to the target object is determined based on the scene image, the target object shape corresponding to the target object is determined based on the point cloud data, the end execution tool is screened according to the target object position and the target object shape, and the target execution tool matched with the target object is determined.

[0031] It should be noted that the plurality of end execution tools at least include a gripper and a suction cup. When the sorting robot includes two mechanical arms, one of the two mechanical arms can be provided with the gripper and the other mechanical arm can be provided with the suction cup, or both of the two mechanical arms can be provided with the gripper and the suction cup.

[0032] In some implementation scenarios, one of the two mechanical arms of the sorting robot is provided with the gripper and the other mechanical arm is provided with the suction cup. The visual feature corresponding to the scene image and the point cloud feature corresponding to the point cloud data are obtained, wherein the visual feature is related to the size of the target object, and the point cloud feature is related to the surface curvature of the target object. The target execution tool matched with the target object is selected from the gripper and the suction cup as the target execution tool based on the visual feature and the point cloud feature, and the mechanical arm corresponding to the target execution tool is determined.

[0033] In some implementation scenarios, both of the two mechanical arms of the sorting robot are provided with the gripper and the suction cup. The position of the target object relative to the sorting robot is determined based on the scene image, the target object shape corresponding to the target object is determined based on the point cloud data, the mechanical arm matched with the target object is determined according to the position of the target object, and the target execution tool matched with the target object is selected from the gripper and the suction cup according to the target object shape.

[0034] S103: determining an execution point on the target object based on the point cloud data and the target execution tool, and using the target execution tool to sort the target object at the execution point.

[0035] Specifically, the execution point selected by the target execution tool on the target object when sorting the target object is determined based on the point cloud data and the target execution tool, so as to select an appropriate execution point on the target object when the sizes of different target objects are inconsistent. The target execution tool is used to resist the execution point, so that the target execution tool exerts an external force on the target object, thereby achieving sorting of the target object.

[0036] It can be understood that the execution point corresponds to a force point when the target execution tool exerts an external force on the target object. When the target execution tool is a gripper, the execution point corresponds to a gripping point of the gripper. When the target execution tool is a suction cup, the execution point corresponds to a suction point of the suction cup.

[0037] In an embodiment, a target object contour of the target object is generated based on the point cloud data, a plurality of target object surfaces corresponding to the target object contour are determined, an execution point matching the target execution tool is selected from the target object surfaces, and the target execution tool is applied to the execution point so that the target execution tool exerts an external force on the target object, thereby achieving sorting of the target object.

[0038] In an embodiment, at least one candidate surface is selected from the target object based on the point cloud data, a surface curvature corresponding to each candidate surface is determined, an execution surface matching the target execution tool is selected according to the surface curvature, and an execution point is selected on the surface of the target object based on the target execution tool and the execution surface, wherein the execution point is arranged on at least the execution surface. The target execution tool is applied to the execution point so that the target execution tool exerts an external force on the target object, thereby achieving sorting of the target object.

[0039] In some implementation scenarios, when the target execution tool is a gripper, a largest surface is selected from the target object surfaces, an execution point is selected from the largest surface, a direction of a shortest axis of the target object is taken as a direction of gripping of the gripper, and a corresponding execution point is selected on an opposite surface.

[0040] In some implementation scenarios, when the target execution tool is a suction cup, an execution surface with a curvature less than a curvature threshold and an area greater than an area threshold is selected from the target object surfaces, and a plurality of execution points are selected from the execution surface as suction points of the suction cup.

[0041] It can be understood that the target object can be moved after the target execution tool exerts an external force on the target object, thereby achieving sorting of the target object. The purposes of sorting of the target object include, but are not limited to, transfer of the target object and delivery of the target object.

[0042] In a specific implementation scenario, the target object corresponds to a delivery package, and the target execution tool is applied to the execution point and moves the sorting robot, thereby achieving delivery of the delivery package.

[0043] In a specific implementation scenario, the target object corresponds to a cargo box, and the target execution tool is applied to the execution point and moves the sorting robot, thereby achieving transfer and storage of the cargo box.

[0044] The above scheme obtains a scene image of the target scene, wherein the scene image is matched with depth information, determines the target object in the target scene based on the scene image, and determines the point cloud data of the target object in three-dimensional space. Based on the scene image and point cloud data, a target execution tool that matches the target object is screened from all the end execution tools of the sorting robot, so that the target execution tool and the target object are more compatible, reducing the probability of difficulty in accurately sorting the target objects due to disorderly stacking of the target objects, thereby improving the accuracy of sorting the target objects using the target execution tool. Based on the point cloud data and the target execution tool, the execution point selected on the target object by the target execution tool when sorting the target object is determined, so that when the sizes of different target objects are inconsistent, an appropriate execution point can be selected on the target object, and the target execution tool is used to support the execution point so that the target execution tool can apply external force to the target object, thereby achieving sorting of the target object and improving the accuracy and stability of target object sorting.

[0045] See also Figure 2 , Figure 2 : is a flow chart of another embodiment of the target object sorting method of the present application, the method comprising:

[0046] S201: Acquire a scene image of a target scene, and determine point cloud data of a target object in the target scene based on the scene image; wherein the scene image is matched with depth information.

[0047] Specifically, a scene image of a target scene is acquired, wherein the scene image is matched with depth information, a target object in the target scene is determined based on the scene image, and point cloud data of the target object is determined in a three-dimensional space.

[0048] In one embodiment, a scene image of a target scene is obtained, and point cloud data of a target object in the target scene is determined based on the scene image, including: obtaining a scene image of the target scene, performing semantic segmentation on the scene image, and obtaining an image area corresponding to the target object; obtaining camera intrinsic parameters corresponding to a camera that captures the scene image, and determining point cloud data of the target object in the target scene based on the camera intrinsic parameters, the image area, and depth information matched thereto.

[0049] Specifically, the scene image captured by the camera in the target scene is obtained, and the scene image is semantically segmented to accurately obtain the image area corresponding to the target object, improve the recognition accuracy of the target object, obtain the camera intrinsic parameters corresponding to the camera, and use the depth information and camera intrinsic parameters to map the two-dimensional image area obtained after segmentation to three-dimensional space, generate point cloud data of the target object in the target scene, so that the point cloud data can be accurately matched with the target object in the actual scene.

[0050] In some implementation scenarios, the camera for collecting the scene image is a depth camera, and the scene image and depth information are collected by using a depth camera with RGB-D function, where RGB corresponds to the color of the target scene, D corresponds to the distance between the target object in the target scene and the camera, the depth camera collects an RGB image and a depth image, the RGB image is the scene image, the depth image includes depth information, and the RGB image and the depth image are time-stamped matched and pixel-aligned. The RGB image is subjected to semantic segmentation to segment the image area matched with each independent target object. The segmented 2D object area is mapped to a three-dimensional space by using the aligned depth information and the camera internal parameter to generate 3D point cloud data corresponding to each object.

[0051] Optionally, after the scene image of the target scene is collected and the point cloud data of the target object in the target scene is determined based on the scene image, the method further includes: determining the occlusion degree of the target object based on the image area and the point cloud data, obtaining a priority score matched with the occlusion degree, and generating a sorting order of all target objects in the target scene based on the priority score.

[0052] Specifically, the occlusion degree of the target object is analyzed based on the image area and the point cloud data, the occlusion degree of the target object is determined, a priority score matched with the occlusion degree is obtained, and a sorting order of all target objects in the target scene is generated according to the priority score, thereby improving the success rate and efficiency of intelligent grabbing of the sorting robot in a complex environment.

[0053] In some implementation scenarios, the image area and the point cloud data are input into an object occlusion grading model, the occlusion degree of the target object is determined by using the object occlusion grading model, and a priority score matched with the occlusion degree is output. The object occlusion grading model is obtained by supervised training of target objects labeled with the occlusion degree, the object occlusion grading model analyzes the visual and geometric features such as the height, area and curvature of the object to determine the priority score, and improves the rationality of the sorting order.

[0054] S202: Selecting a target execution tool matched with the target object from all end execution tools based on the scene image and the point cloud data.

[0055] Specifically, a target execution tool matched with the target object is selected from all end execution tools of the sorting robot based on the scene image and the point cloud data, and the adaptability of the target execution tool to the target object is improved.

[0056] In some implementation scenarios, the sorting robot is provided with mechanical arms on both sides, and each of the mechanical arms is provided with multiple end execution tools. Based on the scene image and the point cloud data, a target execution tool matched with the target object is selected from all the end execution tools, including: obtaining the spatial position of the target object relative to the sorting robot, determining the mechanical arm matched with the target object based on the spatial position; obtaining the visual features of the scene image and the point cloud features of the point cloud data, and determining the target execution tool used on the mechanical arm matched with the target object based on the visual features and the point cloud features.

[0057] Specifically, the sorting robot is provided with mechanical arms on both sides, and each of the two mechanical arms is provided with multiple end execution tools. The spatial position of the target object relative to the sorting robot is obtained, and a more matched mechanical arm is selected for the target object based on the spatial position, so as to ensure that the mechanical arm can operate on the target object and reduce the probability of interference of the mechanical arm.

[0058] Further, the visual features of the scene image and the point cloud features of the point cloud data are obtained, so as to intelligently judge the end execution tool adapted to the target object in combination with the visual features and the point cloud features, dynamically select the adapted target execution tool to perform the sorting task, and intelligently arrange a more adapted mechanical arm to perform the sorting task according to the position of the object, thereby effectively improving the versatility of the end execution tool of the robot and the working efficiency of the robot.

[0059] It can be understood that in other implementation scenarios, when the sorting robot is provided with more than two mechanical arms, the mechanical arm matched with the target object and the target execution tool on the mechanical arm can still be determined based on the above-mentioned manner of selecting the mechanical arm and the end execution tool.

[0060] S203: Determine the three-dimensional contour information corresponding to the target object based on the point cloud data.

[0061] Specifically, based on the point cloud data, the three-dimensional contour of the target object is analyzed to obtain three-dimensional contour information with less data amount.

[0062] It should be noted that the three-dimensional contour information includes but is not limited to a convex hull contour and a three-dimensional bounding box (3D Bounding Box), wherein the convex hull contour can be obtained by processing the point cloud data by using a convex hull algorithm, and the 3D Bounding Box can be obtained by performing geometric feature analysis and bounding box parameter calculation on the point cloud data.

[0063] S204: Determine the execution point on the target object based on the point cloud data and the target execution tool.

[0064] Specifically, based on the point cloud data and the target execution tool, the execution point selected by the target execution tool on the target object when sorting the target object is determined, so as to select an adapted execution point on the target object when the sizes of different target objects are inconsistent.

[0065] It should be noted that the plurality of end execution tools include grippers and suction cups to adapt to different sizes and different shapes of target objects.

[0066] In some implementation scenarios, when the target execution tool is a gripper, the execution point on the target object is determined based on the point cloud data and the target execution tool, including: determining the center point and the axial length in each axial direction of the target object based on the three-dimensional contour information, and determining the execution point and the grabbing direction of the gripper based on the center point and the axial length.

[0067] Specifically, the center point position of the target object and the axial length of the three axes are determined based on the simplified three-dimensional contour information compared with the point cloud data, which improves the efficiency of data analysis. The center point is taken as the execution point when the gripper grabs the target object, and the direction matched with the shortest axial length of the object is taken as the grabbing direction of the gripper to ensure the stability and safety of the gripper when grabbing the target object, and to reduce the probability of sliding or tilting.

[0068] In an embodiment, when the target execution tool is a gripper, the execution point on the target object is determined based on the point cloud data and the target execution tool, including: inputting the point cloud data into a pose estimation model to obtain a grabbing pose of the gripper when grabbing the target object output by the pose estimation model, and determining the execution point of the gripper based on the grabbing pose.

[0069] Specifically, the point cloud data is input into a pre-trained pose estimation model to obtain a grabbing pose of the gripper when grabbing the target object output by the pose estimation model. The pose estimation model corresponds to a neural network model, and the pose estimation model is obtained by supervised training using objects labeled with the grabbing pose, so as to determine the pose required by the gripper when grabbing the target object, thereby facilitating the control of the gripper. The execution point of the gripper when grabbing the target object is determined based on the grabbing pose.

[0070] In some implementation scenarios, when the target execution tool is a suction cup, the execution point on the target object is determined based on the point cloud data and the target execution tool, including: selecting a projection plane from all planes of the target object based on the point cloud data; projecting the point cloud data to the projection plane to obtain a point cloud plane, obtaining a plurality of sampling points from the point cloud plane to obtain a sampling point set; selecting an execution point of the suction cup when sucking the target object from the sampling point set, and determining a sucking pose of the suction cup when sucking the target object based on the execution point and the three-dimensional contour information; wherein the sucking pose includes the association relationship between the axial direction of the suction cup when sucking the target object and the three-dimensional contour information.

[0071] Specifically, based on the point cloud data, all planes corresponding to the target object are determined, and a plane meeting the projection requirement is selected as the projection plane from all the planes corresponding to the target object, where the projection plane can be a plane with the largest area or other planes.

[0072] Further, the point cloud data is projected to the projection plane, so that the point cloud data is gathered in the projection plane, and a corresponding point cloud plane is obtained to facilitate sampling of the point cloud. A plurality of sampling points are selected from the point cloud plane to obtain a sampling point set. A sampling point matched with the suction cup is selected from the sampling point set as an execution point when the target object is sucked by the suction cup. Based on the execution point and three-dimensional contour information of the target object, an axial direction when the target object is sucked by the suction cup is determined, and a suction pose when the target object is sucked by the suction cup is obtained.

[0073] In a specific implementation scenario, after the execution point on the target object is determined, the axial direction when the target object is sucked by the suction cup is determined, where the Z-axis is always perpendicular to the object surface, ensuring that the suction force or clamping force applied during suction is optimal; the Y-axis is parallel to the longest axis of the object, ensuring that any large rotation during subsequent robot carrying and operation is perpendicular to the object long axis direction as much as possible, avoiding failure or sliding due to excessive torque; and the X-axis is determined according to the previous two axes and the right-hand rule.

[0074] In some implementation scenarios, projecting the point cloud data to the projection plane obtains a point cloud plane, and obtaining a plurality of sampling points from the point cloud plane obtains a sampling point set, including: projecting the point cloud data to the projection plane to obtain the point cloud plane, and obtaining a plane center of the point cloud plane; taking the plane center as a sampling point, and determining a polar coordinate system with the plane center as a pole; determining a plurality of sampling directions in the polar coordinate system according to a preset angle interval, and obtaining corresponding sampling points on the sampling directions of the point cloud plane to obtain the sampling point set.

[0075] Specifically, the point cloud data is projected to the projection plane, so that the point cloud data is gathered in the projection plane, and a corresponding point cloud plane is obtained. A plane center of the point cloud plane is obtained, a polar coordinate system is established with the plane center as a pole, and sampling is performed in the polar coordinate system according to a preset angle interval.

[0076] It should be noted that in the polar coordinate system, a sampling direction is determined every radian r0, and a sampling is performed in the sampling direction. Among them, since the point cloud data has discreteness, assuming that the farthest distance of the point set after the point cloud data is projected in the sampling direction from the polar point is Lmax, the ratio0*Lmax is taken as a marker, and the point closest to the marker in the sampling direction is selected as a sampling point. Among them, ratio0 is a proportionality coefficient, which corresponds to a value between 0 and 1. In this way, a total of N samplings (N is the maximum integer not exceeding π / r0) are performed, and a total of N+1 sampling points including the polar point are obtained, wherein the sampling points are candidate suction pose points. Therefore, on the basis of plane segmentation, a plurality of candidate suction pose points are quickly and efficiently returned, which provides more abundant alternative solutions for subsequent kinematics calculation and obstacle avoidance planning of the robot, and enhances the flexibility and robustness of the grasping task in a complex scene.

[0077] In some implementation scenarios, before determining the suction pose of the suction cup when sucking the target object based on the execution point and the three-dimensional contour information, when selecting the execution point of the suction cup when sucking the target object from the sampling point set, the method further includes: obtaining all adjacent points of the sampling point within a preset range in the point cloud data, determining the curvature between the sampling point and each adjacent point; filtering the sampling points in the sampling point set based on the curvature to obtain an updated sampling point set.

[0078] Specifically, the following steps are performed for each sampling point in the sampling point set: determining the point corresponding to the sampling point in the point cloud data, obtaining all adjacent points within a preset range from the point in the point cloud data, determining the curvature between the sampling point and each adjacent point, wherein assuming that there are K adjacent points, determining whether the curvature between the sampling point and the adjacent points is greater than a threshold value theta0, assuming that there are k points whose curvature is greater than the threshold value theta0 among the K points, and when k / K is greater than a threshold value theta1, discarding the sampling point, thereby eliminating the sampling points with large curvature in the peripheral region and avoiding poor suction effect of the suction cup due to large curvature.

[0079] S205: using the target execution tool to resist the execution point, and determining an obstacle avoidance path when sorting the target object based on the three-dimensional contour information corresponding to the target object.

[0080] Specifically, using the target execution tool to resist the execution point, and determining the pose of the mechanical arm and the target object corresponding to the target execution tool after resisting the execution point based on the three-dimensional contour information corresponding to the target object, thereby planning an obstacle avoidance path for sorting the target object, avoiding the probability of sorting failure due to collision of the target object, and improving the stability of sorting the target object.

[0081] It can be understood that the three-dimensional contour information enables the robot to quickly and efficiently perform subsequent path planning and obstacle avoidance calculation, and the highly abstracted geometric representation is suitable for low-power and efficient real-time processing of the embedded control unit of the robot.

[0082] In the embodiment, based on the scene image and the point cloud data, a matching mechanical arm and a target execution tool on the mechanical arm are selected for the target object, based on the point cloud data, the three-dimensional contour of the target object is analyzed to obtain three-dimensional contour information with less data amount. According to different cases that the gripper is used as the target execution tool and the suction cup is used as the end execution tool, the execution point is selected according to the strategy matched with the target execution tool, the target execution tool is used to hold the execution point, based on the three-dimensional contour information corresponding to the target object, the pose of the mechanical arm and the target object corresponding to the target execution tool holding the execution point is determined, so as to plan an obstacle avoidance path for sorting the target object, avoid the probability of sorting failure caused by the collision of the target object, and improve the stability of the sorting of the target object.

[0083] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of an embodiment of a target object sorting system of the application, which is applied to a sorting robot including various end execution tools. The target object sorting system 30 includes an acquisition module 301, a selection module 302 and an execution module 303. The acquisition module 301 is configured to acquire a scene image of a target scene, and determine point cloud data of a target object in the target scene based on the scene image. The scene image matches depth information. The selection module 302 is configured to select a target execution tool matched with the target object from all the end execution tools based on the scene image and the point cloud data. The execution module 303 is configured to determine an execution point on the target object based on the point cloud data and the target execution tool, and use the target execution tool to hold the execution point to sort the target object.

[0084] The above scheme, the acquisition module 301 acquires a scene image of a target scene, wherein the scene image matches depth information, determines a target object in the target scene based on the scene image, and determines point cloud data of the target object in a three-dimensional space. The selection module 302 selects a target execution tool matched with the target object from all the end execution tools of the sorting robot based on the scene image and the point cloud data, so that the target execution tool is more suitable for the target object, reduces the probability that the target object is difficult to be accurately sorted due to disordered stacking of the target object, and improves the accuracy of sorting the target object by using the target execution tool. The execution module 303 determines an execution point on the target object when the target execution tool sorts the target object based on the point cloud data and the target execution tool, so as to select a suitable execution point on the target object when the sizes of different target objects are inconsistent, use the target execution tool to hold the execution point, so that the target execution tool exerts an external force on the target object, thereby sorting the target object, and improving the accuracy and stability of sorting the target object.

[0085] In an embodiment, the selection module 302 is further configured to determine the three-dimensional contour information corresponding to the target object based on the point cloud data. The execution module 303 is further configured to determine the obstacle avoidance path when sorting the target object based on the three-dimensional contour information of the target object by using the execution point of the target execution tool.

[0086] In an embodiment, the plurality of end execution tools includes a gripper and a suction cup. When the target execution tool is the gripper, the execution module 303 is further configured to determine the center point and the axial length in each axial direction corresponding to the target object based on the three-dimensional contour information, determine the execution point and the grasping direction of the gripper based on the center point and the axial length, or input the point cloud data into a pose estimation model to obtain the grasping pose of the gripper matched by the pose estimation model when the gripper grasps the target object, and determine the execution point of the gripper based on the grasping pose.

[0087] In an embodiment, the plurality of end execution tools includes a gripper and a suction cup. When the target execution tool is the suction cup, the execution module 303 is further configured to select a projection plane from all planes of the target object based on the point cloud data, project the point cloud data to the projection plane to obtain a point cloud plane, obtain a plurality of sampling points from the point cloud plane to obtain a sampling point set, select the execution point of the suction cup when the suction cup sucks the target object, and determine the sucking pose of the suction cup when the suction cup sucks the target object based on the execution point and the three-dimensional contour information; wherein the sucking pose includes the association relationship between the axial direction of the suction cup when the suction cup sucks the target object and the three-dimensional contour information.

[0088] In an embodiment, the execution module 303 is further configured to project the point cloud data to the projection plane to obtain the point cloud plane, obtain the plane center of the point cloud plane, take the plane center as a sampling point, and determine a polar coordinate system with the plane center as a pole, determine a plurality of sampling directions in the polar coordinate system according to a preset angle interval, obtain corresponding sampling points in the sampling directions of the point cloud plane to obtain a sampling point set.

[0089] In an embodiment, the execution module 303 is further configured to obtain all adjacent points of the sampling point within a preset range in the point cloud data, determine the curvature between the sampling point and each adjacent point, and screen the sampling points in the sampling point set based on the curvature to obtain an updated sampling point set.

[0090] In an embodiment, the two sides of the sorting robot are respectively provided with a mechanical arm, and the mechanical arm is provided with a plurality of end execution tools. The selection module 302 is further configured to obtain the spatial position of the target object relative to the sorting robot, determine the matched mechanical arm of the target object based on the spatial position, obtain the visual features of the scene image and the point cloud features of the point cloud data, and determine the target execution tool used on the matched mechanical arm of the target object based on the visual features and the point cloud features.

[0091] In an embodiment, the acquisition module 301 is further configured to acquire a scene image of the target scene, perform semantic segmentation on the scene image to obtain an image region corresponding to the target object, acquire camera intrinsic parameters of a camera used to capture the scene image, and determine point cloud data of the target object in the target scene based on the camera intrinsic parameters, the image region, and matching depth information of the image region.

[0092] In an embodiment, the selection module 302 is further configured to determine an occlusion degree of the target object based on the image region and the point cloud data, acquire a priority score matching the occlusion degree, and generate a sorting order of all target objects in the target scene based on the priority score.

[0093] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 40 includes a memory 401 and a processor 402 coupled with each other. The memory 401 stores program data (not shown in the figure). The processor 402 invokes the program data to implement the method in any of the above embodiments. For related content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0094] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. The computer readable storage medium 50 stores program data 500. The program data 500 is executed by a processor to implement the method in any of the above embodiments. For related content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0095] It should be noted that the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0097] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0098] The above is only the embodiment of the present application, and does not limit the protection scope of the present application. Any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, is also included in the protection scope of the present application.

Claims

1. A method for sorting target objects, characterized in that: Applied to a sorting robot, the sorting robot includes a plurality of end-effectors, and the target object sorting method includes: Acquire a scene image of a target scene, and determine point cloud data of a target object in the target scene based on the scene image; wherein the scene image is matched with depth information; Selecting a target execution tool that matches the target object from all the end-effectors based on the scene image and the point cloud data; Based on the point cloud data and the target execution tool, an execution point on the target object is determined, and the target execution tool is used to push against the execution point to sort the target object.

2. The target object sorting method according to claim 1, characterized in that: Before determining the execution point on the target object based on the point cloud data and the target execution tool, and using the target execution tool to push against the execution point to sort the target object, the method further includes: Determining three-dimensional contour information corresponding to the target object based on the point cloud data; The step of utilizing the target execution tool to push against the execution point to sort the target object includes: The target execution tool is used to press against the execution point, and an obstacle avoidance path is determined when sorting the target object based on the three-dimensional contour information corresponding to the target object.

3. The target object sorting method according to claim 2, characterized in that: The end effector includes a gripper and a suction cup. When the target effector is the gripper, determining the effector point on the target object based on the point cloud data and the target effector includes: Determine the center point and the axial length of the target object in each axial direction based on the three-dimensional contour information, and determine the execution point and grasping direction of the gripper based on the center point and the axial length; or The point cloud data is input into a posture estimation model to obtain a gripping posture matched by the gripper when gripping the target object, output by the posture estimation model, and an execution point of the gripper is determined based on the gripping posture.

4. The target object sorting method according to claim 2, characterized in that: The end effector includes a gripper and a suction cup. When the target effector is the suction cup, determining the effector point on the target object based on the point cloud data and the target effector includes: Selecting a projection plane from all planes of the target object based on the point cloud data; Projecting the point cloud data onto the projection plane to obtain a point cloud plane, and acquiring a plurality of sampling points from the point cloud plane to obtain a sampling point set; An execution point when the suction cup sucks the target object is selected from the sampling point set, and a suction posture when the suction cup sucks the target object is determined based on the execution point and the three-dimensional contour information; wherein the suction posture includes an association relationship between an axial direction when the suction cup sucks the target object and the three-dimensional contour information.

5. The target object sorting method according to claim 4, characterized in that: The step of projecting the point cloud data onto the projection plane to obtain a point cloud plane, and acquiring a plurality of sampling points from the point cloud plane to obtain a sampling point set includes: Projecting the point cloud data onto the projection plane to obtain a point cloud plane, and obtaining the plane center of the point cloud plane; Taking the center of the plane as a sampling point, and determining a polar coordinate system with the center of the plane as a pole; A plurality of sampling directions are determined in the polar coordinate system according to preset angular intervals, and corresponding sampling points are acquired in the sampling directions of the point cloud plane to obtain the sampling point set.

6. The target object sorting method according to claim 4, characterized in that: Before selecting an execution point when the suction cup sucks the target object from the sampling point set and determining a suction posture when the suction cup sucks the target object based on the execution point and the three-dimensional contour information, the method further includes: Obtaining all adjacent points within a preset range corresponding to the sampling point in the point cloud data, and determining the curvature between the sampling point and each adjacent point; The sampling points in the sampling point set are screened based on the curvature to obtain the updated sampling point set.

7. The target object sorting method according to claim 1, characterized in that: Mechanical arms are respectively provided on both sides of the sorting robot, and a plurality of end-effector tools are provided on each of the mechanical arms; The selecting, based on the scene image and the point cloud data, a target execution tool matching the target object from all the end-effectors includes: Obtaining a spatial position of the target object relative to the sorting robot, and determining a robotic arm that matches the target object based on the spatial position; The visual features of the scene image and the point cloud features of the point cloud data are obtained, and based on the visual features and the point cloud features, the target execution tool used on the robotic arm that matches the target object is determined.

8. The target object sorting method according to claim 1, characterized in that: The acquiring a scene image of a target scene and determining point cloud data of a target object in the target scene based on the scene image includes: Acquire a scene image of the target scene, perform semantic segmentation on the scene image, and obtain an image region corresponding to the target object; Obtain camera intrinsic parameters corresponding to a camera that captures the scene image, and determine point cloud data of a target object in the target scene based on the camera intrinsic parameters, the image area and its matching depth information.

9. The target object sorting method according to claim 8, characterized in that: After acquiring a scene image of a target scene and determining point cloud data of a target object in the target scene based on the scene image, the method further includes: Determining an occlusion degree of the target object based on the image area and the point cloud data, and obtaining a priority score for matching the occlusion degrees; Based on the priority scores, a sorting order of all the targets in the target scene is generated.

10. A target object sorting system, characterized in that: Applied to a sorting robot, the sorting robot includes a variety of end-effector tools, and the target object sorting system includes: An acquisition module, configured to acquire a scene image of a target scene, and determine point cloud data of a target object in the target scene based on the scene image; wherein the scene image is matched with depth information; A selection module, configured to select a target execution tool that matches the target object from all the end execution tools based on the scene image and the point cloud data; An execution module is used to determine an execution point on the target object based on the point cloud data and the target execution tool, and use the target execution tool to push against the execution point to sort the target object.

11. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images