Method and apparatus for acquiring grabbing pose of target object, and electronic device

Abstract

A method for acquiring the grabbing pose of a target object. The method comprises: setting a detection area for a three-dimensional point cloud in a point cloud coordinate system acquired from a scenario, so as to obtain a three-dimensional detection area comprising a point cloud of a target object; converting the three-dimensional detection area into a two-dimensional detection area; performing target detection and analysis on the basis of the two-dimensional detection area and a color image acquired from the scenario, so as to obtain at least one target object and feature information of the target object that comprises spatial geometric features in an image coordinate system; converting a three-dimensional point cloud of the three-dimensional detection area in a point cloud coordinate system into a robot coordinate system, so as to obtain a three-dimensional point cloud in the robot coordinate system; acquiring feature information in the robot coordinate system on the basis of the feature information of the target object in the image coordinate system; and obtaining a grabbing point of the target object on the basis of the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, so as to obtain the grabbing pose of the target object. Further provided are an apparatus (1000) for acquiring a grabbing pose of a target object, and an electronic device.

Description

Method and device for acquiring target object grasping posture, and electronic equipment Technical Field

[0001] The present invention relates to a method and device for acquiring the grasping posture of a target object, an electronic device, a readable storage medium and a robot grasping system. Background Art

[0002] Deep learning is developing rapidly in the field of vision. Related 2D deep learning methods include target detection, semantic segmentation, instance segmentation and other methods.

[0003] 2D deep learning models can learn advanced features from data, offering greater adaptability. A single 2D deep learning model can detect and identify a variety of targets. However, the results of detection and recognition using a 2D deep learning model are difficult to use directly for grasping the target object, potentially leading to incorrect grasps. Furthermore, the accuracy of grasp points generated by 2D deep learning models is low. Therefore, it is crucial to perform a series of processing on the 2D detection and recognition results using point clouds.

[0004] In existing technologies, post-processing of detection and recognition results primarily involves filtering based on point cloud height, 2D score, and the number of target point clouds. This inefficiently utilizes point cloud information, preventing high-accuracy detection and recognition, and ultimately preventing completely accurate capture. Furthermore, existing post-processing methods primarily address specific scenarios and are incapable of adapting to a wide range of scenarios. Existing post-processing methods also struggle to address misaligned capture caused by deformation of stacked objects, often requiring manual intervention. Furthermore, they are unable to capture tilted objects perpendicular to their surfaces, and are unable to support diverse sorting requirements.

[0005] Summary of the Invention

[0006] The present invention provides a method and device for acquiring the grasping posture of a target object, an electronic device, a readable storage medium, and a robot grasping system.

[0007] According to one aspect of the present invention, a method for acquiring a grasping posture of a target object is provided, comprising: setting a detection area for a three-dimensional point cloud in a point cloud coordinate system obtained from a scene to obtain a three-dimensional detection area including a point cloud of a target object; converting the three-dimensional detection area into a two-dimensional detection area based on camera intrinsic parameters; performing target detection and analysis based on the two-dimensional detection area and a color image obtained from the scene to obtain at least one target object and feature information of the target object in an image coordinate system including spatial geometric features; converting the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area from a point cloud coordinate system to a robot coordinate system to obtain a three-dimensional point cloud in a robot coordinate system; obtaining feature information in a robot coordinate system based on the feature information in the image coordinate system of the target object; and obtaining a grasping point of the target object based on the feature information in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0008] According to at least one embodiment of the present invention, the method for acquiring the target object grasping posture further includes: filtering the three-dimensional point cloud in the point cloud coordinate system based on the three-dimensional detection area to filter out the point cloud outside the three-dimensional detection area; wherein, when filtering the three-dimensional point cloud in the point cloud coordinate system, the contour features of the three-dimensional point cloud are maintained to maintain the spatial correspondence between the three-dimensional point cloud in the point cloud coordinate system and the color image.

[0009] According to at least one embodiment of the present invention, a method for acquiring the target object grasping posture obtains feature information in a robot coordinate system based on the feature information in the image coordinate system of the target object, including: converting the feature information in the image coordinate system of the target object to a point cloud coordinate system based on the spatial correspondence between the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area and the color image, to obtain the feature information in the point cloud coordinate system of the target object; and converting the feature information in the point cloud coordinate system of the target object to feature information in the robot coordinate system.

[0010] According to the target object grasping posture acquisition method of at least one embodiment of the present invention, the spatial geometric features include a minimum circumscribed rectangular frame and / or a spatial angle.

[0011] According to the target object grasping posture acquisition method of at least one embodiment of the present invention, the feature information of the target object in the image coordinate system also includes target object score information, target object mask information and target object category information.

[0012] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, includes: judging whether the grasping point of the target object exceeds the three-dimensional point cloud in the robot coordinate system based on the spatial geometric features in the feature information of the target object in the robot coordinate system; if exceeded, discarding the grasping point that exceeds the three-dimensional point cloud in the robot coordinate system and updating the grasping point of the target object; and obtaining the grasping posture of the target object based on the updated grasping point.

[0013] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, includes: performing category judgment on the target object category information in the feature information of each target object in the robot coordinate system to determine whether the target object can be grasped, so as to obtain a target object that can be grasped; and obtaining the grasping point of the target object based on the feature information of the target object that can be grasped in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0014] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, includes: converting the geometric dimensions of the minimum circumscribed rectangular box in the feature information of each target object in the image coordinate system to the point cloud coordinate system to obtain the geometric dimensions of the minimum circumscribed rectangular box in the point cloud coordinate system; comparing the geometric dimensions of the minimum circumscribed rectangular box in the point cloud coordinate system of the target object with the actual geometric dimensions of the target object to determine whether the difference between the two is greater than or greater than or equal to a difference threshold; if greater than or greater than or equal to the difference threshold, discarding the target object to update the target object obtained by the target detection; and obtaining the grasping point of each target object based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0015] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, includes: converting the mask area in the feature information of each target object in the image coordinate system to the point cloud coordinate system to obtain the mask area in the point cloud coordinate system; comparing the mask area in the point cloud coordinate system of the target object with the actual area of ​​the target object to determine whether the difference between the two is greater than or greater than equal to a difference threshold; if greater than or greater than equal to the difference threshold, filtering out the target object to update the target object obtained by the target detection; and obtaining the grasping point of the target object based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0016] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: sorting all target objects based on the height information in the feature information of each target object in the robot coordinate system to obtain the top target object; comparing the height of the top target object with the heights of other target objects to obtain a first difference between the heights of each other target object and the height of the top target object; comparing the first difference with the thickness of the target object in the height direction to determine whether each first difference is greater than or greater than or equal to 1 / 2 of the thickness, and if so, discarding the corresponding target object to obtain the upper target object; and obtaining a grasping point of the upper target object based on the feature information of each upper target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the upper target object.

[0017] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of an upper-level target object based on feature information of each upper-level target object in a robot coordinate system and a three-dimensional point cloud in the robot coordinate system to obtain a grasping posture of the upper-level target object, includes: judging whether there is a point cloud above each upper-level target object to judge whether there is a target object above the uppermost target object; if so, discarding the upper-level target object to update the upper-level target object; and obtaining a grasping point of the upper-level target object based on the feature information of the updated upper-level target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain a grasping posture.

[0018] According to a target object grasping posture acquisition method of at least one embodiment of the present invention, whether there is a point cloud above each upper-layer target object to determine whether there is a target object above the upper-layer target object includes: determining whether the number of point clouds above each upper-layer target object is greater than a quantity threshold; and if greater than, determining that there is a target object above the upper-layer target object and discarding the upper-layer target object.

[0019] According to at least one embodiment of the present invention, a target object grasping posture acquisition method is used to determine whether there is a point cloud above each upper-layer target object to determine whether there is a target object above the upper-layer target object, including: circumferentially expanding the minimum circumscribed rectangular frame of each upper-layer target object based on a preset size to obtain an expanded area; determining whether the ratio of the number of point clouds above the expanded area of ​​the upper-layer target object to the number of upper-layer target objects is greater than a ratio threshold; and if greater than, determining that there is a target object above the upper-layer target object and discarding the upper-layer target object.

[0020] According to at least one embodiment of the present invention, the method for acquiring the grasping posture of a target object obtains the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: obtaining the intersection-and-union ratio between the minimum bounding rectangles of each two target objects; judging whether the intersection-and-union ratio of the minimum bounding rectangles of the two target objects is greater than or greater than or equal to the intersection-and-union ratio threshold; if so, judging that the two target objects have an overlapping relationship; for the two target objects that have an overlapping relationship, calculating the mask of the first target object and the mask of the second target object A first area ratio of the intersection area of ​​the minimum circumscribed rectangular frame of the object and the mask of the first target object is calculated, and a second area ratio of the intersection area of ​​the mask of the second target object and the minimum circumscribed rectangular frame of the first target object and the mask of the second target object is calculated; the size relationship between the first area ratio and the second area ratio is determined, if the first area ratio is greater than the second area ratio, the first target object is determined to be an overlapped target object, if the second area ratio is greater than the first area ratio, the second target object is determined to be an overlapped target object; and the overlapped target object is discarded to update the target object obtained by the target detection.

[0021] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: acquiring the minimum circumscribed rectangular box of each target object and the normal direction of the minimum circumscribed rectangular box based on the grasping point of each target object; acquiring its rotation matrix relative to the robot coordinate system based on the normal direction of the minimum circumscribed rectangular box of each target object; and acquiring the grasping posture of each target object based on the product of the rotation matrix of the target object and the rotation matrix of the grasping point of the target object.

[0022] According to at least one embodiment of the target object grasping posture acquisition method of the present invention, the minimum bounding rectangle and the normal direction of the minimum bounding rectangle of each target object are obtained based on the grasping point of each target object, including: based on the relevant feature information of the grasping point of each target object, obtaining the point cloud corresponding to each target object in the robot coordinate system; calculating the main direction of the point cloud of each target object based on the principal component analysis method; and obtaining the minimum bounding rectangle and the normal direction of the minimum bounding rectangle of the point cloud of each target object based on the main direction of the point cloud of each target object.

[0023] According to at least one embodiment of the present invention, a method for acquiring a target object grasping posture, obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, includes: obtaining a point cloud normal determination sub-area containing the grasping point based on the grasping point of each target object; obtaining the normal direction of all point cloud points or a preset number of point cloud points including the grasping point in the point cloud normal determination sub-area; obtaining the average normal direction of the point cloud normal determination sub-area of ​​each target object based on the normal direction of all point cloud points or a preset number of point cloud points including the grasping point in the point cloud normal determination sub-area; obtaining its point cloud normal determination sub-area rotation matrix relative to the robot coordinate system based on the average normal direction; and obtaining the grasping posture of each target object based on the point cloud normal determination sub-area rotation matrix of the target object and the grasping point rotation matrix of the target object's grasping point relative to the robot coordinate system.

[0024] According to at least one embodiment of the present invention, a method for acquiring the target object grasping posture is used to acquire the grasping posture of each target object based on the point cloud normal determination sub-area rotation matrix of the target object and the grasping point rotation matrix of the target object relative to the robot coordinate system, including: multiplying the grasping point rotation matrix of each target object with the point cloud normal determination sub-area rotation matrix to obtain the grasping point update rotation matrix of each target object to obtain the grasping posture of each target object.

[0025] The method for acquiring a target object grasping posture according to at least one embodiment of the present invention further includes: sorting all target objects based on the spatial geometric features of each target object whose grasping posture has been acquired, and obtaining sorting information for performing grasping.

[0026] According to the method for acquiring the grasping posture of a target object in at least one embodiment of the present invention, the sorting information includes sorting information along a first direction and sorting information along a second direction perpendicular to the first direction.

[0027] According to another aspect of the present invention, a device for acquiring a grasping posture of a target object is provided, comprising: a detection area setting module, the detection area setting module performing a detection area setting on a three-dimensional point cloud in a point cloud coordinate system obtained from a scene to obtain a three-dimensional detection area including a point cloud of a target object; a detection area conversion module, the detection area conversion module converting the three-dimensional detection area into a two-dimensional detection area based on a camera intrinsic parameter; a target object detection and analysis module, the target object detection and analysis module performing target detection and analysis based on the two-dimensional detection area and a color image obtained from the scene to obtain at least one target object and feature information of the target object in an image coordinate system including spatial geometric features; a first conversion module, the first conversion module converting the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area from a point cloud coordinate system to a robot coordinate system to obtain a three-dimensional point cloud in a robot coordinate system; a second conversion module, the second conversion module obtaining feature information in a robot coordinate system based on the feature information of the target object in the image coordinate system; and a grasping posture generation module, the grasping posture generation module obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain a grasping posture of the target object.

[0028] According to at least one embodiment of the present invention, the target object grasping posture acquisition device further includes: a target object sorting module, which sorts all target objects based on the spatial geometric features of each target object whose grasping posture has been acquired to obtain sorting information for performing grasping.

[0029] According to another aspect of the present invention, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the target object grasping posture acquisition method of any embodiment of the present invention.

[0030] According to another aspect of the present invention, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the target object grasping posture acquisition method of any embodiment of the present invention.

[0031] According to yet another aspect of the present invention, a robot grasping system is provided, comprising: a readable storage medium according to any one embodiment of the present invention; and a robot configured to execute a grasping operation based on an execution instruction stored in the readable storage medium.

[0032] According to the robot grasping system of at least one embodiment of the present invention, the robot includes a robot arm having a robot hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings illustrate exemplary embodiments of the present invention and together with the description serve to explain the principles of the present invention. These drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification.

[0034] FIG1 is a flow chart of a method for acquiring a target object grasping posture according to an embodiment of the present invention.

[0035] FIG2 is a flow chart of a method for acquiring a target object grasping posture according to another embodiment of the present invention.

[0036] FIG3 is a schematic diagram of a process for acquiring feature information in a robot coordinate system based on feature information in an image coordinate system of a target object according to an embodiment of the present invention.

[0037] FIG4 is a schematic diagram of a grab point filtering process based on a three-dimensional ROI according to an embodiment of the present invention.

[0038] FIG5 is a schematic diagram of a category filtering process according to an embodiment of the present invention.

[0039] FIG6 is a schematic diagram of a filtering process based on target object size according to an embodiment of the present invention.

[0040] FIG7 is a schematic diagram of a filtering process based on mask area according to an embodiment of the present invention.

[0041] FIG8 is a schematic diagram of a top-level target object acquisition process according to an embodiment of the present invention.

[0042] FIG. 9 is a schematic diagram of a top-level target object screening step according to an embodiment of the present invention.

[0043] FIG. 10 is a flow chart illustrating a step of filtering overlapping target objects according to an embodiment of the present invention.

[0044] FIG11 is a schematic diagram of a process for obtaining the position and posture of a target object according to an embodiment of the present invention.

[0045] FIG12 is a schematic diagram of a process for obtaining the position and posture of a target object according to another embodiment of the present invention.

[0046] FIG13 is a flow chart of a method for acquiring a target object grasping posture according to another embodiment of the present invention.

[0047] FIG14 is a schematic block diagram of the structure of a target object grasping posture acquisition device using a hardware implementation of a processing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following describes in detail the target object grasping posture acquisition method, device, electronic device, readable storage medium and robot grasping system of the present invention in conjunction with Figures 1 to 14.

[0049] FIG1 is a flow chart of a method for acquiring a target object grasping posture according to an embodiment of the present invention.

[0050] 1 , in some embodiments of the present invention, the target object grasping posture acquisition method S100 of the present invention includes the following steps.

[0051] S102 : Setting a detection region (ROI) for the three-dimensional point cloud in the point cloud coordinate system obtained from the scene to obtain a three-dimensional detection region (3D ROI) including the point cloud of the target object.

[0052] S104: Convert the three-dimensional detection region (3D ROI) into a two-dimensional detection region (2D ROI) based on the camera intrinsic parameters.

[0053] S106. Perform target detection and analysis based on a two-dimensional detection region (2D ROI) and a color image (e.g., an RGB image) obtained from the scene (preferably using a 2D target detection algorithm based on deep learning) to obtain at least one target object and feature information of the target object in the image coordinate system, including spatial geometric features.

[0054] S108 , converting the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection region (3D ROI) from the point cloud coordinate system to the robot coordinate system to obtain the three-dimensional point cloud in the robot coordinate system.

[0055] S110 . Acquire feature information in the robot coordinate system based on the feature information in the image coordinate system of the target object.

[0056] S112 : Obtaining a grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain a grasping posture of the target object.

[0057] The target object may be a material bar, a material box, a material bag, etc., and the target object may be placed in a carrier (such as a plastic basket, a pallet, etc.).

[0058] The three-dimensional point cloud described in the present invention can be obtained based on the following method.

[0059] Laser ranging method: Laser ranging method is to emit laser to the surface of an object through a laser transmitter. After the laser hits the surface of the object, the photoelectric converter is used to convert the reflected laser signal into an electrical signal, and the three-dimensional coordinates of the object surface are calculated through an algorithm to obtain point cloud data.

[0060] Structured light method: The structured light method is a method of three-dimensional scanning based on projecting grating patterns. A projector is usually used to project a specific grating pattern, and then the reflected image is captured by a camera to obtain the three-dimensional coordinates of the object surface and obtain point cloud data.

[0061] RGB-D camera method: RGB-D cameras can simultaneously acquire color images and depth images. By converting depth images into point cloud data, the three-dimensional shape information of the object surface can be obtained.

[0062] In some embodiments of the present invention, the three-dimensional detection area set in the present invention only includes the target object point cloud and the vehicle point cloud.

[0063] By converting a three-dimensional detection region (3D ROI) into a two-dimensional detection region (2D ROI), the present invention can avoid interference of redundant objects in 2D target detection based on deep learning and improve the reliability of 2D recognition.

[0064] The deep learning-based 2D target detection described in the present invention can be performed based on an existing 2D target detection algorithm. The deep learning-based 2D target detection algorithm can be R-CNN (Selective Search+CNN+SVM), SPP-net (ROI Pooling), Fast R-CNN (Selective Search+CNN+ROI), Faster R-CNN (RPN+CNN+ROI), R-FCN and other algorithms or other algorithms. Under the guidance of the technical solution of the present invention, those skilled in the art can also use other types of 2D target detection algorithms, which all fall within the protection scope of the present invention.

[0065] In the present invention, the two-dimensional detection area obtained by converting the three-dimensional detection area based on the camera intrinsic parameters (intrinsic parameter matrix, distortion matrix) is the two-dimensional detection area in the image coordinate system.

[0066] In step S106 described above, the detection of the target object may include identifying and obtaining an approximate position and a rough rectangular frame of the target object, and the analysis of the target object may include processing such as instance segmentation to obtain the accurate edge, accurate position, and detection rectangular frame (minimum bounding rectangular frame) of the target object. Those skilled in the art, inspired by the technical solutions of the present invention, may adjust the specific steps of target detection and analysis performed in step S106 of the present invention, and all such adjustments shall fall within the scope of protection of the present invention.

[0067] The target object and its feature information in the image coordinate system, including spatial geometric features, can be obtained through image detection / segmentation using deep learning algorithms, such as the minimum enclosing rectangular frame of the target object, spatial angle, target object score information, target object mask information, target object category information, etc.

[0068] Due to the different training levels of the deep learning model, the target object segmentation results may not be accurate enough, and the categories, masks, scores, etc. may be wrong. Therefore, the 2D recognition results obtained in step S106 cannot be directly used by the robot to grasp the target object, and subsequent steps are required.

[0069] The target object grasping posture acquisition method of the present invention obtains feature information in the robot coordinate system based on the feature information in the image coordinate system of the target object through step S110. The purpose is to combine the feature information in the image coordinate system of the target object obtained by target detection and analysis based on the two-dimensional detection area and the color image obtained from the scene, that is, the target object score information, target object category and other information provided by the 2D data with the target center point, target depth information, target spatial position, target area, target length and width information, and target rotation information provided by the three-dimensional point cloud (that is, the information provided by the 3D data), and then convert them into the robot coordinate system so that the information obtained based on the 2D data and the information obtained based on the 3D data can be used to obtain the grasping point and grasping posture of the target object in the same coordinate system (robot coordinate system), so that the robot grasping system can directly grasp.

[0070] In some embodiments of the present invention, referring to FIG2 , the target object grasping posture acquisition method S100 of the present invention further includes: S107 , filtering the three-dimensional point cloud in the point cloud coordinate system based on the three-dimensional detection region (3D ROI) to filter out the point cloud outside the three-dimensional detection region.

[0071] When filtering the three-dimensional point cloud in the point cloud coordinate system, the contour features of the three-dimensional point cloud are maintained to maintain the spatial correspondence between the three-dimensional point cloud and the color image in the point cloud coordinate system.

[0072] The present invention filters the point cloud in step S107 to remove interference targets outside the 3D ROI, thereby filtering out redundant point clouds and reducing the impact of flying points in the camera point cloud on subsequent data calculations.

[0073] In some embodiments of the present invention, referring to FIG3 , step S110 of the present invention, acquiring feature information in the robot coordinate system based on feature information in the image coordinate system of the target object, includes the following steps.

[0074] S1102 : Based on the spatial correspondence between the three-dimensional point cloud and the color image in the point cloud coordinate system of the three-dimensional detection area, the feature information of the target object in the image coordinate system is converted to the point cloud coordinate system to obtain the feature information of the target object in the point cloud coordinate system.

[0075] S1104: Convert the feature information of the target object in the point cloud coordinate system to feature information in the robot coordinate system.

[0076] In the present invention, preferably, the feature information of each target object in the image coordinate system is not directly converted to the robot coordinate system, but is first converted to the point cloud coordinate system and then converted to the robot coordinate system, thereby adding relevant feature information in the point cloud coordinate system (i.e., 3D feature information) to obtain the feature information of the target object in the point cloud coordinate system, and then obtain the feature information of the target object in the robot coordinate system.

[0077] The present invention converts the 2D deep learning detection results (the target object and the feature information of the target object in the image coordinate system) obtained above, that is, the detection results in the image coordinate system, into points in the point cloud coordinate system through step S1102, so as to associate them with the 3D feature information in the point cloud coordinate system (including the target center point, target depth information, target spatial position, target area, target length and width information, target rotation information, etc.), thereby adding the 3D feature information to the 2D feature information of the target object.

[0078] Further, through step S1104, the feature information of the target object in the point cloud coordinate system with added 3D feature information is converted to the robot coordinate system, and the corresponding grasping points in the robot coordinate system are obtained according to the hand-eye calibration matrix. It is possible to generate a grasping pose type that can be directly used by the robot grasping system for grasping, including the actual grasping point pose, score, category, mask, minimum bounding box (minimum circumscribed rectangular box), average depth, etc. of each target object, so as to be transferred as a fixed data type in subsequent processing. The processing process of steps S1102 and S1104 of the present invention improves the utilization rate of point cloud information, is conducive to improving grasping accuracy, and transferring it as a fixed data type can achieve multi-scene adaptation.

[0079] The present invention performs coordinate system conversion on the point cloud after ROI filtering, and uses the hand-eye calibration matrix (which can be obtained by hand-eye calibration of the camera and robot) to convert the three-dimensional point cloud from the point cloud coordinate system to the robot coordinate system for subsequent processing.

[0080] Taking into account that the grasping results converted from the 2D detection results may exceed the 3D ROI range on a certain coordinate axis, and the grasping of all target objects to be grasped must be within the 3D ROI, therefore, in some preferred embodiments of the present invention, grasping point filtering based on the 3D ROI is performed. Referring to Figure 4, in step S112, a grasping point filtering process based on the 3D ROI is included: based on the spatial geometric features in the feature information of the target object in the robot coordinate system, it is judged whether the grasping point of the target object exceeds the 3D point cloud in the robot coordinate system; if it exceeds, the grasping point that exceeds the 3D point cloud in the robot coordinate system is discarded and the grasping point of the target object is updated; based on the updated grasping point, the grasping posture of the target object is obtained.

[0081] In some preferred embodiments of the present invention, referring to FIG5 , in step S112, a category filtering process is also included: performing category judgment on the target object category information in the feature information of each target object in the robot coordinate system to determine whether the target object can be grasped, so as to obtain a target object that can be grasped; obtaining the grasping point of the target object based on the feature information of the target object that can be grasped in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0082] In some preferred embodiments of the present invention, referring to Figure 6, in step S112, a filtering step based on the size of the target object is also included: the geometric dimensions of the minimum circumscribed rectangular box in the feature information of each target object in the robot coordinate system are converted to the point cloud coordinate system to obtain the geometric dimensions of the minimum circumscribed rectangular box in the point cloud coordinate system; the geometric dimensions of the minimum circumscribed rectangular box in the point cloud coordinate system of the target object are compared with the actual geometric dimensions of the target object to determine whether the difference between the two is greater than or greater than or equal to a difference threshold; if it is greater than or greater than or equal to the difference threshold, the target object is discarded to update the target object obtained by target detection; based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of each target object is obtained to obtain the grasping posture of the target object.

[0083] In some preferred embodiments of the present invention, referring to Figure 7, in step S112, a filtering step based on the mask area is also included: the mask area in the feature information of each target object in the robot coordinate system is converted to the point cloud coordinate system to obtain the mask area in the point cloud coordinate system; the mask area in the point cloud coordinate system of the target object is compared with the actual area of ​​the target object to determine whether the difference between the two is greater than or greater than or equal to a difference threshold; if it is greater than or greater than or equal to the difference threshold, the target object is filtered out to update the target object obtained by target detection; based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the target object is obtained to obtain the grasping posture of the target object.

[0084] In some embodiments of the present invention, preferably, with reference to FIG8 , in step S112, an upper target object acquisition process is further included: all target objects are sorted based on the height information in the feature information of each target object in the robot coordinate system to obtain the uppermost target object; the height of the uppermost target object is compared with the heights of other target objects to obtain a first difference between the heights of each other target object and the height of the uppermost target object (i.e., a height difference, which can be obtained based on the difference between the coordinates of the target object in the height direction); the above-mentioned first difference is compared with the thickness of the target object in the height direction (which can be pre-set, such as the actual thickness of the target object) to determine whether each first difference is greater than or greater than 1 / 2 of the above-mentioned thickness. If so, the corresponding target object is discarded to obtain the upper target object; based on the feature information of each upper target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the upper target object is obtained to obtain the grasping posture of the upper target object.

[0085] By performing the above-mentioned target object filtering, all the obtained target objects are sorted by height in the robot coordinate system to obtain the height of the top target object, and then according to the actual thickness of the target object, the target objects below the height of the top target object and less than half the thickness of the target object are filtered out, that is, the target objects whose height difference with the height of the top target object is greater than or equal to 1 / 2 of the target object thickness are filtered out to obtain the upper target object, which can filter out most of the lower results.

[0086] For the results obtained by the above-mentioned upper-layer filtering process, preferably, step S112 of the present invention, obtaining the grasping points of each upper-layer target object based on the feature information of each upper-layer target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the upper-layer target object, also includes an upper-layer target object screening step, referring to Figure 9: judging whether there is a point cloud above each upper-layer target object to judge whether there is a target object above the upper-layer target object; if so, discarding the upper-layer target object to update the upper-layer target object; obtaining the grasping points of the upper-layer target object based on the feature information of the updated upper-layer target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture.

[0087] In a preferred embodiment of the present invention, the upper-layer target object screening process described above is implemented based on the following steps: determining whether the number of point clouds above each upper-layer target object is greater than a quantity threshold; if greater than, determining that there is a target object above the upper-layer target object and discarding the upper-layer target object.

[0088] In another preferred embodiment of the present invention, the upper-layer target object screening process described above is implemented based on the following steps: the minimum circumscribed rectangular frame of each upper-layer target object is circumferentially expanded based on a preset size to obtain an expanded area; whether the ratio of the number of point clouds above the expanded area of ​​the upper-layer target object to the number of upper-layer target objects is greater than a ratio threshold; if greater, it is determined that there is a target object above the upper-layer target object and the upper-layer target object is discarded.

[0089] For the target object grasping posture acquisition method S100 in each of the above embodiments, preferably, S112, obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, also includes an overlapping target object filtering step, referring to Figure 10: obtaining the intersection-and-union ratio (IOU) between the minimum bounding rectangles of each two target objects overUnion); determine whether the intersection-over-union ratio of the minimum bounding rectangles of the two target objects is greater than or greater than or equal to the intersection-over-union ratio threshold; if so, determine that the two target objects have an overlapping relationship; for the two target objects in an overlapping relationship, calculate a first area ratio of the intersection area of ​​the mask of the first target object and the minimum bounding rectangle of the second target object to the mask of the first target object, and calculate a second area ratio of the intersection area of ​​the mask of the second target object and the minimum bounding rectangle of the first target object to the mask of the second target object; determine the size relationship between the first area ratio and the second area ratio, if the first area ratio is greater than the second area ratio, determine the first target object as the overlapped target object, if the second area ratio is greater than the first area ratio, determine the second target object as the overlapped target object; and discard the overlapped target object to update the target object obtained by target detection.

[0090] After obtaining the upper-layer target object, it may not be possible to completely filter the lower-layer results for scenes with a large degree of inclination. In order to realize the grasping operation of the upper-layer target object including the inclined target object perpendicular to the target object surface, the present invention preferably obtains the position and posture of the target object based on the following steps, referring to Figure 11: based on the grasping point of each target object, the minimum circumscribed rectangular box of each target object and the normal direction of the minimum circumscribed rectangular box are obtained; based on the normal direction of the minimum circumscribed rectangular box of each target object, its rotation matrix relative to the reference plane (such as the horizontal plane) (that is, the rotation matrix relative to the robot coordinate system) is obtained; based on the product of the rotation matrix of the target object and the rotation matrix of the grasping point of the target object, the grasping posture of each target object is obtained.

[0091] Among them, based on the grasping point of each target object (each target object has a grasping point, and the grasping point of each target object itself has a basic rotation matrix, that is, the grasping point rotation matrix; the grasping point position is preferably the surface geometric center position of the target object. In the present invention, the surface geometric center position is preferably obtained based on the two-dimensional image, that is, the color image and the three-dimensional point cloud of the target object), the minimum bounding rectangle and the normal direction of the minimum bounding rectangle of each target object are obtained, preferably based on the following method: based on the relevant feature information of the grasping point of each target object, the point cloud corresponding to each target object in the robot coordinate system is obtained, and the principal component analysis method is preferably used to calculate the main direction of the point cloud of each target object, and then the minimum bounding rectangle and the normal direction of the minimum bounding rectangle of this point cloud (that is, the point cloud of the target object) are obtained; based on the normal direction of the minimum bounding rectangle of each target object, the rotation matrix of each target object relative to the robot coordinate system is obtained; based on the product (dot product) of the rotation matrix of the target object and the grasping point rotation matrix of the target object, the grasping posture of each target object is obtained, thereby realizing posture correction.

[0092] Figure 12 shows the process of obtaining the posture of the target object in another preferred embodiment of the present invention: based on the grasping point of each target object, a point cloud normal determination sub-area containing the grasping point is obtained; the normal direction of all point cloud points or a preset number of point cloud points including the grasping point in the point cloud normal determination sub-area is obtained; based on the normal direction of all point cloud points or a preset number of point cloud points including the grasping point in the point cloud normal determination sub-area, the average normal direction of the point cloud normal determination sub-area of ​​each target object is obtained; based on the average normal direction, its point cloud normal determination sub-area rotation matrix relative to the reference plane (such as the horizontal plane), that is, relative to the robot coordinate system, is obtained; and the grasping posture of each target object is obtained based on the point cloud normal determination sub-area rotation matrix of the target object and the grasping point rotation matrix of the target object's grasping point relative to the robot coordinate system.

[0093] Among them, the point cloud normal judgment sub-area containing the grasping point of each target object has a preset shape and preset size. The preset shape is preferably a circular shape, and the preset size is pre-set based on the size / type of the target object (material bag, material box, etc.). Those skilled in the art can adjust the preset shape and preset size under the guidance of the technical solution of the present invention, which falls within the scope of protection of the present invention.

[0094] In some embodiments of the present invention, the process of obtaining the position and posture of the target object shown in FIG12 includes the following steps:

[0095] For each target object, based on the relevant feature information of the grasping point, the point cloud and center point position of the target object in the robot coordinate system are obtained, and the point cloud of a circular area with a preset radius is taken at the center point position, and the normal calculation is performed on the point cloud points (all or a preset number) including the grasping point in the circular area to obtain the normal direction of each point cloud point; the average normal direction of the circular area is obtained based on the normal direction of each point cloud point; the grasping posture of each target object is obtained based on the average normal rotation matrix of the average normal direction relative to the robot coordinate system (that is, the point cloud normal determination sub-area rotation matrix described above), and based on the average normal rotation matrix of the target object and the grasping point rotation matrix of the target object's grasping point relative to the robot coordinate system.

[0096] Preferably, the grasping posture of each target object is obtained based on the average normal rotation matrix of the target object and the grasping point rotation matrix of the target object, including: multiplying the grasping point rotation matrix of each target object (the grasping point related feature information includes the basic grasping point posture, that is, the grasping point rotation matrix) with the average normal rotation matrix (preferably point multiplication) to obtain the grasping point update rotation matrix of each target object, thereby obtaining the inclined grasping posture of each target object.

[0097] In the preferred embodiment described above, after first obtaining the target object's grasping point pose, i.e., the basic grasping pose, the grasping point rotation matrix of each grasping point relative to the robot coordinate system is obtained because the grasping point pose itself includes position information and rotation angle information. The average normal direction of the target object's point cloud normal determination sub-area, including the grasping point, is then calculated. This allows the grasping pose to be calibrated and updated, improving the grasping pose accuracy and ensuring grasping accuracy. Furthermore, this solution first obtains the grasping point rotation matrix of each grasping point relative to the robot coordinate system, and then calculates the average normal direction of the target object's point cloud normal determination sub-area, including the grasping point (i.e., the average normal direction is calculated at the end of the process). This aims to reduce the number of normal calculations, especially to further reduce the amount of calculation after interference has been reduced in the previous steps.

[0098] In the present invention, the purpose of calculating the average normal is to make the normal calculation more accurate and to avoid incorrect grasping posture of the target object obtained due to the tilt direction.

[0099] FIG13 shows a process of a method S100 for acquiring a grasping posture of a target object according to another embodiment of the present invention. This embodiment also includes: S114, sorting all target objects based on the spatial geometric features of each target object whose grasping posture has been acquired, and obtaining sorting information for performing grasping.

[0100] The sorting information preferably includes sorting information along a first direction and sorting information along a second direction perpendicular to the first direction.

[0101] In some embodiments of the present invention, sorting can be performed based on features such as area, score, height, etc., or target objects on the same layer can be sorted in order of up, down, left, and right. The target objects can first be divided into horizontal and vertical directions according to spatial geometric features such as side length and rotation angle, and the horizontal and vertical target objects can be sorted up, down, left, and right respectively, and finally merged.

[0102] The present invention also provides a target object grasping posture acquisition device 1000, which includes the following modules.

[0103] The detection area setting module 1002 performs detection area setting on the three-dimensional point cloud in the point cloud coordinate system obtained from the scene to obtain a three-dimensional detection area including the target object point cloud.

[0104] The detection area conversion module 1004 converts the three-dimensional detection area into a two-dimensional detection area based on the camera intrinsic parameters.

[0105] The target object detection and analysis module 1006 performs target detection and analysis based on the two-dimensional detection area and the color image obtained from the scene to obtain at least one target object and feature information of the target object in the image coordinate system including spatial geometric features.

[0106] The first conversion module 1008 converts the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area from the point cloud coordinate system to the robot coordinate system to obtain the three-dimensional point cloud in the robot coordinate system.

[0107] The second conversion module 1010 obtains feature information in the robot coordinate system based on the feature information in the image coordinate system of the target object.

[0108] The grasping posture generation module 1012 obtains the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

[0109] In some embodiments of the present invention, the target object grasping posture acquisition device 1000 further includes: a target object sorting module 1014, which sorts all target objects based on the spatial geometric features of each target object whose grasping posture has been acquired, and obtains sorting information for performing grasping.

[0110] Each of the above modules can be implemented through a computer program module.

[0111] 14 , which shows a schematic block diagram of the structure of a target object grasping posture acquisition device 1000 implemented in hardware using a processing system according to an embodiment of the present invention.

[0112] The target object grasping posture acquisition device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart can be executed by the corresponding module, and the device may include one or more of these modules. The module can be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by the processor, or implemented by some combination.

[0113] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0114] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0115] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which the functions may be performed in a manner not shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art of the art to which the embodiments of the present invention pertain. The processor performs the various methods and processes described above. For example, the method embodiments of the present invention can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0116] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0117] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0118] It should be understood that various parts of the present invention can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0120] Furthermore, the functional units in various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0121] The present invention also provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the target object grasping posture acquisition method of any embodiment of the present invention.

[0122] The present invention also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the target object grasping posture acquisition method of any embodiment of the present invention.

[0123] The present invention also provides a robot grasping system, comprising: the readable storage medium of the present invention; and a robot, wherein the robot performs a grasping operation based on an execution instruction stored in the readable storage medium.

[0124] The robot described in the present invention may be a robotic arm with a robotic hand or other types of robots.

Claims

1. A method for acquiring a target object grasping posture, characterized in that: include: Setting a detection area for the three-dimensional point cloud in the point cloud coordinate system obtained from the scene to obtain a three-dimensional detection area including the point cloud of the target object; Converting the three-dimensional detection area into a two-dimensional detection area based on camera intrinsic parameters; Performing target detection and analysis based on the two-dimensional detection area and the color image obtained from the scene to obtain at least one target object and feature information of the target object in an image coordinate system including spatial geometric features; The three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area is converted from the point cloud coordinate system to the robot coordinate system to obtain the three-dimensional point cloud in the robot coordinate system; Acquiring feature information in a robot coordinate system based on the feature information in the image coordinate system of the target object; as well as Based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the target object is obtained to obtain the grasping posture of the target object.

2. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Also includes: Filtering the three-dimensional point cloud in the point cloud coordinate system based on the three-dimensional detection area to filter out the point cloud outside the three-dimensional detection area; Wherein, when filtering the three-dimensional point cloud in the point cloud coordinate system, the contour features of the three-dimensional point cloud are maintained to maintain the spatial correspondence between the three-dimensional point cloud in the point cloud coordinate system and the color image.

3. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Acquiring feature information in a robot coordinate system based on the feature information in the image coordinate system of the target object includes: Based on the spatial correspondence between the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area and the color image, the feature information of the target object in the image coordinate system is converted to the point cloud coordinate system to obtain the feature information of the target object in the point cloud coordinate system; and The feature information of the target object in the point cloud coordinate system is converted to feature information in the robot coordinate system.

4. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: The geometric size of the minimum bounding rectangle in the feature information of each target object in the image coordinate system is converted to the point cloud coordinate system to obtain the geometric size of the minimum bounding rectangle in the point cloud coordinate system; Compare the geometric size of the minimum circumscribed rectangular frame of the target object in the point cloud coordinate system with the actual geometric size of the target object to determine whether the difference between the two is greater than or equal to a difference threshold; If it is greater than or equal to the difference threshold, the target object is discarded to update the target object obtained by the target detection; and Based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, a grasping point of each target object is obtained to obtain a grasping posture of the target object.

5. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: The mask area in the feature information of each target object in the image coordinate system is converted to the point cloud coordinate system to obtain the mask area in the point cloud coordinate system; Compare the mask area of ​​the target object in the point cloud coordinate system with the actual area of ​​the target object to determine whether the difference between the two is greater than or equal to a difference threshold; If it is greater than or equal to the difference threshold, filtering out the target object to update the target object obtained by the target detection; and Based on the updated feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the target object is obtained to obtain the grasping posture of the target object.

6. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: Sort all target objects based on the height information in the feature information of each target object in the robot coordinate system to obtain the top target object; Compare the height of the top target object with the heights of the other target objects to obtain a first difference between the heights of the other target objects and the height of the top target object; Comparing the first difference with the thickness of the target object in the height direction, determining whether each first difference is greater than or equal to 1 / 2 of the thickness, and if so, discarding the corresponding target object to obtain an upper target object; and Based on the feature information of each upper-layer target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the upper-layer target object is obtained to obtain the grasping posture of the upper-layer target object.

7. The method for acquiring the target object grasping posture according to claim 6, characterized in that: Obtaining the grasping point of the upper target object based on the feature information of each upper target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the upper target object, including: Determine whether there is a point cloud above each upper target object to determine whether there is a target object above the uppermost target object; If it exists, discard the upper target object to update the upper target object; and Based on the updated feature information of the upper target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system, the grasping point of the upper target object is obtained to obtain the grasping posture.

8. The method for acquiring the target object grasping posture according to claim 7, characterized in that: Determining whether there is a point cloud above each upper target object to determine whether there is a target object above the upper target object includes: Determine whether the number of point clouds above each upper target object is greater than a number threshold; and If it is greater, it is determined that there is a target object above the upper target object and the upper target object is discarded.

9. The method for acquiring the target object grasping posture according to claim 7, characterized in that: Determining whether there is a point cloud above each upper target object to determine whether there is a target object above the upper target object includes: The minimum circumscribed rectangular frame of each upper target object is circumferentially expanded based on a preset size to obtain an expanded area; Determine whether a ratio of the number of point clouds above the extended area of ​​the upper target object to the number of the upper target object is greater than a ratio threshold; and If it is greater, it is determined that there is a target object above the upper target object and the upper target object is discarded.

10. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: Obtain the intersection-and-union ratio between the minimum bounding rectangles of every two target objects; Determine whether the intersection-and-union ratio of the minimum circumscribed rectangular boxes of the two target objects is greater than or equal to the intersection-and-union ratio threshold; If yes, it is determined that the two target objects have an overlapping relationship; For the two target objects that are in an overlapping relationship, a first area ratio of an intersection area between a mask of a first target object and a minimum circumscribed rectangular frame of a second target object to the mask of the first target object is calculated, and a second area ratio of an intersection area between a mask of the second target object and the minimum circumscribed rectangular frame of the first target object to the mask of the second target object is calculated; Determine the magnitude relationship between the first area ratio and the second area ratio, and if the first area ratio is greater than the second area ratio, determine the first target object as the overlapped target object; if the second area ratio is greater than the first area ratio, determine the second target object as the overlapped target object; and The overlapped target object is discarded to update the target object obtained by the target detection.

11. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: Based on the grasping point of each target object, the minimum bounding rectangle of each target object and the normal direction of the minimum bounding rectangle are obtained; Obtaining a rotation matrix of each target object relative to the robot coordinate system based on the normal direction of the minimum bounding rectangle of each target object; and The grasping posture of each target object is obtained based on the product of the rotation matrix of the target object and the grasping point rotation matrix of the target object.

12. The method for acquiring the target object grasping posture according to claim 11, characterized in that: Based on the grasping points of each target object, the minimum bounding rectangle of each target object and the normal direction of the minimum bounding rectangle are obtained, including: Based on the relevant feature information of the grasping point of each target object, a point cloud corresponding to each target object in the robot coordinate system is obtained; Calculate the main direction of the point cloud of each target object based on the principal component analysis method; and The minimum bounding rectangle of the point cloud of each target object and the normal direction of the minimum bounding rectangle are obtained based on the main direction of the point cloud of each target object.

13. The method for acquiring the target object grasping posture according to claim 1, characterized in that: Obtaining the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object, including: Based on the grasping point of each target object, a point cloud normal determination sub-area including the grasping point is obtained; Obtaining the normal directions of all point cloud points or a preset number of point cloud points including the grasping point in the point cloud normal determination sub-area; Obtaining an average normal direction of the point cloud normal determination sub-region of each target object based on the normal directions of all point cloud points or a preset number of point cloud points in the point cloud normal determination sub-region including the grasping point; Obtaining a point cloud normal determination sub-area rotation matrix relative to the robot coordinate system based on the average normal direction; and The grasping posture of each target object is obtained based on the sub-area rotation matrix determined based on the point cloud normal of the target object and the grasping point rotation matrix of the target object relative to the robot coordinate system.

14. A device for acquiring a target object's grasping posture, characterized in that: include: A detection area setting module, which sets a detection area for a three-dimensional point cloud in a point cloud coordinate system obtained from the scene to obtain a three-dimensional detection area including a point cloud of a target object; A detection area conversion module, wherein the detection area conversion module converts the three-dimensional detection area into a two-dimensional detection area based on camera intrinsic parameters; a target object detection and analysis module, wherein the target object detection and analysis module performs target detection and analysis based on the two-dimensional detection area and the color image obtained from the scene to obtain at least one target object and feature information of the target object in an image coordinate system including spatial geometric features; A first conversion module, which converts the three-dimensional point cloud in the point cloud coordinate system of the three-dimensional detection area from the point cloud coordinate system to the robot coordinate system to obtain the three-dimensional point cloud in the robot coordinate system; A second conversion module, which acquires feature information in a robot coordinate system based on the feature information in the image coordinate system of the target object; as well as A grasping posture generation module is used to obtain the grasping point of the target object based on the feature information of the target object in the robot coordinate system and the three-dimensional point cloud in the robot coordinate system to obtain the grasping posture of the target object.

15. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the target object grasping posture acquisition method according to claim 1.

Images