Object grasping method and device based on grasping point pose, equipment and medium

Abstract

The present application provides an object grasping method based on a grasping point pose, which comprises: obtaining S first grasping point poses of a first target object, wherein the first target object comprises an object determined from N objects to be grasped; determining a target grasping point pose from the S first grasping point poses; calculating a pose distance between the target grasping point pose and at least one second grasping point pose of M second grasping point poses, wherein the M second grasping point poses comprise poses of at least one grasping failure object in the N objects to be grasped, S, M or N are integers greater than or equal to 1; and stopping grasping the first target object in the case that any one of the calculated pose distances is less than a first distance threshold. The present application also provides an object grasping device based on a grasping point pose, an electronic device and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent logistics, and more particularly to an object grasping method and device based on grasping point pose, equipment and medium. BACKGROUND

[0002] With the development of automation technology, in some scenarios, automation machinery gradually replaces manual work. For example, an object is grasped by a mechanical arm and moved to a target position, instead of manual carrying.

[0003] Generally, when the mechanical arm grasps the object, there may be a situation of grasping failure. The result calculated by the control algorithm indicates that the object currently grasped unsuccessfully has the highest priority, and the grasping path is successfully planned. Therefore, the mechanical arm may repeatedly attempt to grasp after grasping failure. If the object currently grasped unsuccessfully does not have the possibility of being grasped successfully, then the repeated grasping of the mechanical arm will cause the consumption of time and resources, and reduce the overall grasping efficiency. SUMMARY

[0004] In view of the above problems, the present application provides an object grasping method and device based on grasping point pose, equipment and medium, which can improve the overall grasping efficiency.

[0005] An aspect of an embodiment of the present application provides an object grasping method, comprising: obtaining S first grasping point poses of a first target object, wherein the first target object comprises an object determined from N objects to be grasped; determining a target grasping point pose from the S first grasping point poses; calculating a pose distance between the target grasping point pose and at least one second grasping point pose of M second grasping point poses, wherein the M second grasping point poses comprise a pose of at least one grasping failure object of the N objects to be grasped, S, M or N are integers greater than or equal to 1; and stopping grasping the first target object in the case that any of the calculated pose distances is less than a first distance threshold.

[0006] In an embodiment of the present application, when any of the calculated pose distances is less than a first distance threshold, the method further comprises: calculating an attribute difference between a first region on the first target object and at least one of S-1 second regions, wherein the first region is a region where the target grasp point pose is located, the S-1 second regions one by one correspond to regions where the remaining first grasp point poses in the S first grasp point poses are located, and the attribute difference is used to represent a difference degree between attribute information of the first region and attribute information of a second region; and when the attribute difference between the first region and any of the S-1 second regions satisfies a preset condition, taking the first grasp point pose corresponding to the second region as a new target grasp point pose.

[0007] In an embodiment of the present application, the preset condition comprises a first preset condition, and the calculation of the attribute difference between the first region on the first target object and at least one of the S-1 second regions comprises: setting i = 1, and performing the following operations in a loop: obtaining a shape contour of the first region and a shape contour of an i-th second region based on an image of the first target object, wherein i is an integer greater than or equal to 1 and less than or equal to S-1; calculating a shape contour difference between the shape contour of the first region and the shape contour of the i-th second region; and setting i = i + 1 when the shape contour difference does not satisfy the first preset condition.

[0008] In an embodiment of the present application, the method further comprises: stopping the loop operation when the shape contour difference satisfies the first preset condition; and taking the first grasp point pose corresponding to the i-th second region as a new target grasp point pose.

[0009] In an embodiment of the present application, the preset condition comprises a second preset condition, and the calculation of the attribute difference between the first region on the first target object and at least one of the S-1 second regions comprises: setting i = 1, and performing the following operations in a loop: obtaining a gravity distribution of the first region and a gravity distribution of an i-th second region, wherein i is an integer greater than or equal to 1 and less than or equal to S-1; calculating a gravity distribution difference between the gravity distribution of the first region and the gravity distribution of the i-th second region; and setting i = i + 1 when the gravity distribution difference does not satisfy the second preset condition.

[0010] In an embodiment of the present application, the method further comprises: stopping the loop operation when the gravity distribution difference satisfies the second preset condition; and taking the first grasp point pose corresponding to the i-th second region as a new target grasp point pose.

[0011] According to an embodiment of the present application, the stopping the first target object from being grabbed in a case where any of the calculated pose distances is less than a first distance threshold comprises: deleting the target grasp point pose from a priority queue in a case where any of the calculated pose distances is less than a second distance threshold, wherein the second distance threshold is less than the first distance threshold, and the priority queue is used to determine a grasping order of at least one pending target object in the N objects to be grasped, and the first target object is a highest-priority object in the at least one pending target object.

[0012] According to an embodiment of the present application, the stopping the first target object from being grabbed in a case where any of the calculated pose distances is less than a first distance threshold further comprises: lowering a priority order of the target grasp point pose in the priority queue in a case where any of the calculated pose distances is greater than or equal to a second distance threshold and less than the first distance threshold.

[0013] According to an embodiment of the present application, the method further comprises: based on any of the calculated pose distances being greater than or equal to the first distance threshold, grasping the first target object based on the target grasp point pose.

[0014] According to an embodiment of the present application, the M second grasp point poses are recorded in a failure record queue, and the method further comprises: in a case where the first target object is a failed-grasp object, the target grasp point pose is written into the failure record queue as a second object pose.

[0015] According to an embodiment of the present application, the method further comprises: based on a number N of the N objects to be grasped and a number S of grasp point poses of each object to be grasped, setting a maximum number of pose records of the failure record queue to be less than or equal to N*S-1, wherein M is less than or equal to N*S-1; and before the target grasp point pose is written into the failure record queue as the second object pose, further comprising: if M is equal to N*S-1, deleting a second grasp point pose recorded earliest in the M second grasp point poses from the failure record queue.

[0016] Another aspect of the embodiment of the present application provides an object grasping device, comprising: a pose obtaining module configured to obtain S first grasping point poses of a first target object, wherein the first target object comprises an object determined from N objects to be grasped; a pose determining module configured to determine a target grasping point pose from the S first grasping point poses; a distance calculating module configured to calculate a pose distance between the target grasping point pose and at least one second grasping point pose of M second grasping point poses, wherein the M second grasping point poses comprise poses of at least one grasping failure object of the N objects to be grasped, S, M or N is an integer greater than or equal to 1; and an object grasping module configured to stop grasping the first target object if any of the calculated pose distances is less than a first distance threshold.

[0017] Another aspect of the embodiment of the present application provides an electronic device, comprising: one or more processors; and a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

[0018] Another aspect of the embodiment of the present application also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method described above.

[0019] Compared with the repeated attempt to grasp in the related art after the mechanical arm fails to grasp the object, the embodiment of the present application can first determine the target grasping point pose from the S first grasping point poses of the first target object, and then take the M second grasping point poses as a reference, and determine that the first target object is likely to fail to be grasped if the pose distance between the target grasping point pose and at least one second grasping point pose of the M second grasping point poses is less than the first distance threshold, so as to stop grasping, thereby avoiding the consumption of time and resources caused by repeated attempts, and improving the overall grasping efficiency.

[0020] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the contents of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other purposes, features and advantages of the present application will be more clearly understood through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0022] Figure 1 a flowchart schematically showing an object grasping method according to the embodiment of the present application is shown;

[0023] Figure 2 A flowchart of determining a new target grasping point pose is shown schematically according to an embodiment of the present application.

[0024] Figure 3 A flowchart of calculating attribute difference is shown schematically according to an embodiment of the present application.

[0025] Figure 4 A flowchart of calculating attribute difference is shown schematically according to another embodiment of the present application.

[0026] Figure 5 A flowchart of writing a target grasping point pose into a failure record queue is shown schematically according to an embodiment of the present application.

[0027] Figure 6 A schematic diagram of a priority queue and a failure record queue is shown schematically according to an embodiment of the present application.

[0028] Figure 7 A flowchart of an object grasping method is shown schematically according to another embodiment of the present application.

[0029] Figure 8 A structural block diagram of an object grasping device is shown schematically according to an embodiment of the present application.

[0030] Figure 9 A structural schematic diagram of a computing device is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0032] Figure 1 A flowchart of an object grasping method is shown schematically according to an embodiment of the present application.

[0033] As Figure 1 shown, the object grasping method of this embodiment includes operation S110 to operation S160.

[0034] In operation S110, S first grasping point poses of a first target object are acquired, wherein the first target object includes an object determined from N objects to be grasped.

[0035] For example, the point cloud data of the N objects to be grasped is obtained by means of photographing or laser radar scanning. Then, a target object is determined by processing the point cloud data by means of a target detection algorithm (such as a classification algorithm or a template matching algorithm). The object pose of the target object or the grasp point pose can be obtained from the point cloud data.

[0036] The object pose and the grasp point pose are inherent attributes of the object. The object pose can be defined at any position, such as at the center of the object or not, above the object or not. After being defined, the object pose is uniquely determined for the object. In actual application, where the object pose is defined can be set by software (such as Mech-Vision software). For example, after obtaining a crankshaft model, Mech-Vision can edit the object pose on the loaded crankshaft.

[0037] In operation S120, a target grasp point pose is determined from the S first grasp point poses.

[0038] The grasp point pose can be determined according to the camera or radar position, the grasping manner of the robot arm, and other factors. The number of grasp point poses of an object to be grasped can be one or more. Taking a crankshaft as an example, there is generally one recess at the front, middle and tail of the crankshaft, and then three grasp point poses are defined at the three grasping positions. Since the object pose can be set according to actual conditions, the object pose and the grasp point pose can be the same or different. The grasp point pose can be obtained simultaneously with the object pose, or after or before the object pose. It should be known that the front and tail of the crankshaft can be flexibly defined according to the actual coordinate system, and the middle is located between the front and the tail.

[0039] In operation S120, the target grasp point pose can be determined by considering factors such as the shape of the gripper, the grasping manner, the geometric center of the object, the orientation of the grasping position, inputting a pre-trained grasp classification model, classifying the S grasp point poses, and determining the target grasp point pose. The point cloud data model can also be used for grasp simulation. For example, two contact points are selected at the position of each grasp point pose, the grasp angle is determined, collision detection is performed, and the grasping action is simulated, and finally the grasp point pose with the highest success rate is determined as the target grasp point pose.

[0040] In operation S130, the pose distance between the target grasp point pose and at least one of the M second grasp point poses is calculated, wherein the M second grasp point poses include the pose of at least one failed grasping object among the N objects to be grasped, and S, M or N is an integer greater than or equal to 1.

[0041] For example, the pose of a grasp point can be represented by a rotation-translation matrix [x, y, z, a, b, g], where x, y, z represent the displacement of three degrees of freedom, and a, b, g represent the spatial rotation of three degrees of freedom. The pose distance can be the Euclidean distance between the matrix representing the pose of the target grasp point and the matrix representing any second grasp point pose.

[0042] The failed grasp object refers to an object that has been grasped historically but failed to be grasped and is still among the N objects to be grasped. The first target object can also be a failed grasp object. The failed grasp can refer to that the gripper has contacted a to-be-grasped object, applied an external force, and performed a normal grasping operation, but the object has not been displaced or rotated, or has only been displaced or rotated slightly and has not left the current position. The failed grasp can also refer to that the object has not been grasped from the to-be-grasped position to the target position.

[0043] In operation S140, it is determined whether the pose distance obtained in operation S130 is less than a first distance threshold. If yes, operation S150 is performed. If no, operation S160 is performed.

[0044] In operation S150, the first target object is stopped from being grasped when any of the calculated pose distances is less than the first distance threshold.

[0045] The first distance threshold D1 can be used to represent the first movement amplitude of the object. Different first distance thresholds D1 can be defined for different object types. For example, for small objects, the movement amplitude can be small and the object can be grasped. For large objects, the movement amplitude can be large before the object can be grasped.

[0046] In some embodiments, different types of objects can be preset to correspond to different first distance thresholds D1. For example, when the first target object is obtained, the type of the first target object can be determined, and then the corresponding first distance threshold D1 can be obtained based on the type, and finally each pose distance can be compared with the first distance threshold D1. The object type can be classified according to volume, length, height, etc., or can be a specific object name, category, model. It should be noted that the to-be-grasped object referred to in the present application can be a regular object, i.e., an object whose shape can be expressed by a formula, such as a rectangle, a circle, etc. It can also be an irregular object.

[0047] In the related art, for example, when a mechanical arm sucks an object through a suction cup, if the surface condition of the object suction area is poor, causing the suction cup to fail to create a vacuum, the suction may fail. For another example, when the mechanical arm clamps an object through a clamp, for example, clamps a heavy metal part such as an automobile crankshaft, since the crankshafts are stacked with each other, some crankshafts may be clamped tightly, although they can be successfully clamped, but cannot be grabbed. From the visual calculation result, the grabbing priority of the above-mentioned failed objects is relatively high (for example, the objects at high positions are grabbed first and then the objects at low positions are grabbed), so these objects are always grabbed first each time. From the perspective of movement, the grabbing planning is successful, but the actual grabbing fails. When the above-mentioned situation occurs, the robot always repeatedly grabs the same object, resulting in a decrease in overall grabbing efficiency and a waste of time and grabbing resources.

[0048] Compared with the repeated attempt to grab in the related art after the mechanical arm fails to grab an object, the embodiment of the present application can first determine a target grabbing point pose from S first grabbing point poses of a first target object, then take M second grabbing point poses as references, and in the case that the pose distance between the target grabbing point pose and at least one of the M second grabbing point poses is less than a first distance threshold, it can be determined that the first target object may fail to be grabbed, so as to stop grabbing to avoid the waste of time and resources caused by repeated attempts, and improve the overall grabbing efficiency.

[0049] In operation S160, in the case that any of the calculated pose distances is greater than or equal to the first distance threshold, the first target object is grabbed based on the target grabbing point pose.

[0050] When the pose distance is greater than or equal to the first distance threshold D1, it can be determined that the grabbing point pose of the first target object has not been tried to grab, or although it has failed to grab, the moving amplitude meets the condition and can be tried to grab again. Therefore, by calculating the pose distance between the target grabbing point pose and each second grabbing point pose and comparing it with the first distance threshold, on the one hand, objects that have failed to grab and whose grabbing point poses have not changed can be filtered out, and on the other hand, grabbing operations can be performed on objects that have not been grabbed or can be tried to grab repeatedly, without omission. Thus, the single grabbing success rate and the overall grabbing efficiency are improved.

[0051] Figure 2 A flowchart for determining a new target grabbing point pose according to an embodiment of the present application is schematically shown.

[0052] As shown in Figure 2 In the case that any of the calculated pose distances is less than the first distance threshold, the object grabbing method of this embodiment includes operations S210-S220.

[0053] In operation S210, attribute differences between a first region on the first target object and at least one of S-1 second regions are calculated, where the first region is a region where the target grasp point pose is located, the S-1 second regions one by one correspond to regions where the remaining first grasp point poses in the S first grasp point poses are located, and the attribute differences are used to represent the difference degrees between attribute information of the first region and attribute information of the second region.

[0054] Taking a crankshaft as an example, in a case where three grasp point poses are defined at grooves in a front part, a middle part and a tail part of the crankshaft respectively. If the grasp point pose A at the groove in the front part is the target grasp point pose. The grasp point pose B at the groove in the middle part and the grasp point pose C at the groove in the tail part are the remaining first grasp point poses. Then the groove in the front part can be considered as the first region, and the grooves in the middle part and the tail part can be considered as two second regions. The attribute information of the first region or the attribute information of the second region can include shape profile attributes, gravity distribution attributes, color attributes, contact point friction attributes, etc. The difference degree can be the difference degree between the same attributes.

[0055] In operation S220, in a case where the attribute difference between the first region and any one of the S-1 second regions satisfies a preset condition, the first grasp point pose corresponding to the second region is taken as a new target grasp point pose.

[0056] Since the robot arm is based on the grasp point pose, contacts the region where the pose is located and performs a grasping action, it can be determined as a basis for determining a new target grasp point pose whether there is a difference between the two regions. For example, if the attributes of the two regions are consistent, the grasping success rate under the same grasping condition can be consistent, and there is no need to determine a new target grasp point pose based on the region.

[0057] The preset condition can be that the attribute difference between the first region and any one of the second regions satisfies a preset threshold, so that it can be considered that the second region has different grasping conditions relative to the first region. The above-mentioned preset threshold can be a probability difference value output by a neural network model, for example, the value of the grasping success rate of the first region is 50%, and the value of the grasping success rate of a certain second region is 80%, and the preset threshold can be 20% (only as an example). The grasp point pose corresponding to the second region can be taken as a new target grasp point pose.

[0058] According to the embodiment of the present application, in a case where the grasp point pose is recorded, more grasping options are provided for grasping the first target object. Since the first target object has the highest priority, that is, the grasping of the first target object can be verified by simulation, in a case where the grasping is stopped based on the target grasp point pose, a new target grasp point pose can be replaced, and the first target object is still attempted to be grasped, thereby improving the grasping success rate of the first target object.

[0059] Figure 3 A flowchart of calculating the property difference in operation S210 according to an embodiment of the present application is shown.

[0060] As shown in Figure 3 calculating the property difference between the first region on the first target object and at least one of the S-1 second regions in operation S210 includes operations S310-S350. The preset condition includes a first preset condition. In a case where the shape profile difference does not satisfy the first preset condition, operations S320, S330 and S350 can be executed in a loop.

[0061] In operation S310, based on the image of the first target object, a shape profile of the first region and a shape profile of the i-th second region are obtained, where i is an integer greater than or equal to 1 and less than or equal to S-1.

[0062] For example, an image of a crankshaft is obtained, and target detection is performed on the groove region where each grasping point pose is located. First, a rectangular frame can be drawn for each groove, and then further fitting is performed in the rectangular frame, and finally the shape profile of each groove is determined by fitting.

[0063] In operation S320, the shape profile difference between the shape profile of the first region and the shape profile of the i-th second region is calculated.

[0064] Taking the front groove (target grasping point pose position) and the middle groove as an example, an optional implementation is to classify the shape profiles of the front groove and the middle groove, for example, whether they are both z-edged profiles, where z is an integer greater than or equal to 3. If not, it indicates that there is a shape profile difference. If yes, further calculation is performed on the areas corresponding to the two shape profiles. It is determined whether the areas are consistent. If not, it indicates that there is a shape profile difference. If yes, it ends.

[0065] In operation S330, it is determined whether the shape profile difference obtained in operation S320 satisfies the first preset condition. If yes, operation S340 is performed. If not, operation S350 is performed.

[0066] The first preset condition can be determined according to the difference type of the shape profile. For example, if the z-edged profiles of the front groove and the middle groove are inconsistent, the first preset condition can be the number of edges. For example, if the areas of the front groove and the middle groove are inconsistent, the first preset condition can be an area threshold.

[0067] In operation S340, in a case where the shape profile difference satisfies the first preset condition, the loop operation is stopped. The first grasping point pose corresponding to the i-th second region is taken as a new target grasping point pose.

[0068] In operation S350, it is judged whether i is smaller than S-1. If yes, operation S320 is executed. If no, the current loop is ended.

[0069] According to the embodiment of the present application, the shape profile is taken as the attribute of the region, and the comparison can be conveniently performed. If the first grasping point pose corresponding to the ith second region is taken as the new target grasping point pose, the shape profile of the ith second region can be further subjected to collision detection with the gripper, grasping simulation, and other operations, so as to further improve the grasping success rate.

[0070] Figure 4 A flowchart of calculating the attribute difference in operation S210 according to another embodiment of the present application is schematically shown.

[0071] As shown in Figure 4 The calculation of the attribute difference between the first region on the first target object and at least one of the S-1 second regions in operation S210 includes operations S410-S450. The preset condition can further include a second preset condition. In the case where the gravity distribution difference does not satisfy the first preset condition, operations S420, S430, and S450 can be cyclically executed.

[0072] In operation S410, the gravity distribution of the first region and the gravity distribution of the ith second region are obtained, where i is an integer greater than or equal to 1 and smaller than or equal to S-1.

[0073] The gravity distribution refers to the gravity received by different regions of the object, for example. In an optional implementation, if a large or heavy object is grasped, the object can be placed alone, and a plurality of gravity sensors can be placed under the object in advance to determine the gravity distribution of the corresponding region according to the readings of the gravity sensors. In another optional implementation, whether the object is placed alone or stacked, the density distribution of each region of the object can be measured by ultrasonic method or light scanning method, and the corresponding gravity distribution can be calculated in combination with the area size of the shape profile.

[0074] In operation S420, the gravity distribution difference between the gravity distribution of the first region and the gravity distribution of the ith second region is calculated.

[0075] In operation S430, it is judged whether the gravity distribution difference obtained in operation S420 satisfies the second preset condition. If yes, operation S440 is executed. If no, operation S450 is executed.

[0076] In operation S440, in the case where the gravity distribution difference satisfies the second preset condition, the cyclic operation is stopped. The first grasping point pose corresponding to the ith second region is taken as the new target grasping point pose.

[0077] The gravity distribution difference can be a difference between the gravity of the two regions. The second preset condition can be a preset difference threshold. In the case that the density of the same object is uniform, the small gravity can mean a small volume and a small contact area. When the object is grabbed by using an electromagnetic clamp, the small gravity of the region that is attracted can cause insufficient suction. At this time, it can be determined whether the gravity of the i th second region exceeds the difference threshold of the gravity of the first region. If yes, it is determined that the second preset condition is met. It should be understood that the second preset condition can be set according to the type of the clamp, the type of the object, the attribute of the region, and the like, and the present application does not make specific limitations.

[0078] In operation S450, it is determined whether i is less than S-1. If yes, operation S420 is performed. If no, the current loop is ended.

[0079] According to the embodiments of the present application, the gravity distribution is taken as the attribute of the region, and the comparison is combined with factors such as the shape contour and the type of the clamp, so that the new target point pose can be determined more accurately, and the success rate of grabbing is further improved.

[0080] Figure 5 A flowchart for writing the target grabbing point pose into the failure record queue according to the embodiments of the present application is schematically shown.

[0081] After operation S160 is performed, as shown in Figure 5 operation S510 to operation S540 can also be performed.

[0082] In operation S510, it is determined whether the first target object is successfully grabbed. If yes, the current operation is ended. If no, operation S520 is performed.

[0083] In operation S520, it is determined whether the number M of the current record in the failure record queue is equal to N*S-1. If yes, operation S530 is performed. If no, operation S540 is performed.

[0084] According to the embodiments of the present application, before operation S520 is performed, the maximum pose record number of the failure record queue can be set to be less than or equal to N*S-1 based on the number N of the N target objects to be grabbed and the number S of the grabbing point poses of each target object to be grabbed, wherein M is less than or equal to N*S-1. N*S is the product of N and S, and N*S-1 is the difference between N*S and 1.

[0085] Again taking the crankshaft as an example, N crankshafts to be grabbed are arranged in a logistics box, and after grabbing, the next logistics box is replaced for crankshaft grabbing. If the maximum number of pose records in the failure record queue is not limited, in the case of continuous operation of the robot arm, the object poses of the failed crankshafts in other logistics boxes may still be recorded in the queue. In fact, when the crankshafts in the current logistics box are grabbed, the object poses of the crankshafts in other logistics boxes are meaningless, that is, the comparison result between the calculated pose distance and the first distance threshold is meaningless. Therefore, the maximum number of pose records can be set to be less than or equal to N*S-1.

[0086] In operation S530, if M is equal to N*S-1, the earliest recorded second grasp point pose in the M second grasp point poses is deleted from the failure record queue.

[0087] Taking N as 3 as an example, there are 3 objects to be grabbed, and the number of grasp point poses of each object to be grabbed is 3. If the maximum number of pose records is 9, and the three objects to be grabbed all fail to be grabbed, the pose distance between the grasp point pose of each object to be grabbed and the grasp point pose in the failure record queue may be less than the first distance threshold, and then the robot arm may stop grabbing any object to be grabbed, resulting in that the grabbing work cannot be carried out.

[0088] If the maximum number of pose records is 8, and the three objects to be grabbed all fail to be grabbed, one grasp point pose can be squeezed out of the queue. Specifically, if three objects A, B and C are grabbed, and each planning can successfully grab, but each grabbing fails, if the maximum number of records is set to 8, the actual effect is: after failing to grab A, the target grasp point pose of A is recorded, after updating the other two grasp point poses to the target grasp point pose, both fail to be grabbed, and three grasp point poses of A are recorded. Similarly, after failing to grab B, there are six grasp point poses of A and B recorded. After failing to grab C, if it still fails to be grabbed based on the three grasp point poses. The initial target grasp point pose of A is out of the queue, and at this time, there are six grasp point poses of B and C, and two other grasp point poses of A. At this time, A can try to grab. Thus, the robot arm can continuously apply external force to the object to improve the success rate of grabbing by changing the position or attitude of the object.

[0089] In operation S540, in the case of failing to grab the first target object, the first target object is a failed object, and the target grasp point pose is written into the failure record queue as a second object pose.

[0090] According to an embodiment of the present application, when each object fails to be grabbed, the current target grabbing point pose can be written into a failure record queue as a reference for subsequent grabbing of objects. For example, if the first target object is a failed-to-be-grabbed object, the pose distance is calculated before the next grabbing planning, and compared with the first distance threshold to determine whether it can be grabbed. This avoids the situation that the first target object does not move, resulting in the second grabbing still failing.

[0091] Figure 6 A schematic diagram of a priority queue and a failure record queue according to an embodiment of the present application is shown schematically.

[0092] As shown in Figure 6 , the priority queue can include target grabbing point poses and P-1 pending target object grabbing point poses.

[0093] According to an embodiment of the present application, first, P pending target objects are determined from N objects to be grabbed, where P is an integer greater than or equal to 1.

[0094] Next, the grabbing priorities of the P pending target objects are determined.

[0095] Then, the object with the highest priority among the P pending target objects is taken as the first target object.

[0096] Next, S first grabbing point poses of the first target object and each of the remaining pending target objects are obtained.

[0097] Next, the corresponding target grabbing point pose is determined from the S first grabbing point poses of each object.

[0098] Finally, the corresponding target grabbing point pose is written into the priority queue based on the grabbing priorities of the P pending target objects.

[0099] For example, when N crankshafts in a logistics box are grabbed, the crankshafts are stacked. The N crankshafts are photographed to obtain point cloud data of a number of uppermost crankshafts. First, target detection is performed on the number of uppermost crankshafts, and they are taken as P pending target objects, where P is less than or equal to the number of uppermost crankshafts. Then, further calculation is performed to determine one of the objects as the first target object. For example, the object with the highest grabbing success rate can be taken as the first target object with the highest priority, and the objects can be arranged in priority according to the grabbing success rate (only as an example).

[0100] According to an embodiment of the present application, based on the above-mentioned sorting of grabbing priorities, the grabbing point pose of each of the P pending target objects is written into the priority queue, where the priority queue is used to determine the grabbing order of the P pending target objects.

[0101] As shown in Figure 6 , the failure record queue includes second grasping point poses 1~M. The robot arm performs grasping according to the order in the priority queue in Figure 6 . Since the target grasping point pose has the highest priority, the first target object is grasped first. Before performing grasping planning, the first object pose can be compared with the M second object poses in the failure record queue one by one. The following will be described in combination with Figure 7 .

[0102] Figure 7 A flowchart of an object grasping method according to another embodiment of the application is schematically shown.

[0103] As shown in Figure 7 , the object grasping method of this embodiment can include operation S110, operation S120, operation S160, and can further include operation S710~operation S760.

[0104] In operation S110, S first grasping point poses of a first target object are obtained, wherein the first target object includes an object determined from N objects to be grasped.

[0105] In operation S120, a target grasping point pose is determined from the S first grasping point poses.

[0106] In operation S710, a pose distance between the target grasping point pose and the i-th second grasping point pose is calculated.

[0107] In operation S720, it is determined whether the pose distance obtained in operation S710 is less than a second distance threshold. If yes, operation S730 is performed. If no, operation S740 is performed.

[0108] The second distance threshold D2 can be used to represent the second movement amplitude of the object. For different object types, different second distance thresholds D2 can be defined. In some embodiments, different types of objects can correspond to different second distance thresholds D2. For example, when the second target object is obtained, the type of the second target object can be determined, and then the corresponding second distance threshold D2 is obtained based on the type, and finally each pose distance is compared with the second distance threshold D2.

[0109] In operation S730, in the case where any one of the calculated pose distances is less than the second distance threshold, the target grasping point pose is deleted from the priority queue, wherein the second distance threshold is less than the first distance threshold, the priority queue is used to determine the grasping order of at least one pending target object in the N objects to be grasped, and the first target object is the object with the highest priority among the at least one pending target object.

[0110] Referring toFigure 6 The target grasping point pose is deleted from the priority queue, i.e. the priority of the pending target object 1 grasping point pose becomes the highest, and there are P-1 object poses recorded in the priority queue. Since P target objects can be determined before each grasping planning, and the grasping is performed in turn. The second distance threshold D2 is used to determine whether the first target object has the feasibility of trying to grasp. If the movement amplitude of the first target object is less than the second movement amplitude, it can be considered that the success rate of grasping is very low, and the grasping priority should not be allocated, and the grasping is abandoned this time.

[0111] In operation S740, it is judged whether the pose distance obtained in operation S710 is less than the first distance threshold. If yes, operation S750 is performed. If no, operation S760 is performed.

[0112] In operation S750, the priority order of the target grasping point pose in the priority queue is lowered in the case that any of the obtained pose distances is greater than or equal to the second distance threshold and less than the first distance threshold.

[0113] In the case that the pose distance is greater than or equal to the second distance threshold and less than the first distance threshold, it is considered that the target grasping point pose may have failed to grasp before, but has a certain movement amplitude, at this time the first target object can be affected by other objects, such as being pressed on the first target object. After grasping other objects, the first target object is tried to grasp again, and the success rate of grasping is relatively high. Referring to Figure 6 The first object pose can be inserted between any two of the pending target object poses 1-P-1, for example, the first object pose can be inserted between the pending target object 1 grasping point pose and the pending target object 1 grasping point pose, i.e. the priority order is lowered by one.

[0114] The number of priorities by which the target grasping point pose is lowered can be determined according to the actual running situation. Taking the crankshaft as an example, it can be observed that after grasping a certain number of crankshafts, the crankshafts in the logistics box can be loosened, and then the degradation order is set to this value. Because the crankshafts that can be placed in front are generally easy to grasp, and the reason for lowering the grasping priority is that it is too tight to actually grasp. But when the crankshaft is loosened (i.e. the object pose is changed), the crankshaft can still be grasped first.

[0115] In some embodiments, referring to Figure 6If the to-be-determined target object poses 1-P-1 have been deleted from the priority queue due to successful or unsuccessful grasping, etc. Even if the priority order of the target grasping point pose is lowered to be after the to-be-determined target object pose S-1, in the case where only the target grasping point pose is in the priority queue, the pose distance of the i th second object pose is still greater than or equal to the second distance threshold and less than the first distance threshold, it can be set that if the priority of the target grasping point pose cannot be lowered, the target grasping point pose is deleted.

[0116] In operation S760, it is judged whether i is less than M. If yes, operation S710 is performed. If no, operation S160 is performed.

[0117] Reference Figure 6 The target grasping point pose is compared with each of the second grasping point poses 1-M, a pose distance is obtained, and it is determined whether the target grasping point pose can be grasped. After it is determined that the target grasping point pose can be grasped, the grasping pose, path, angle, gripper, etc. are planned. For example, the target grasping point pose is obtained first, and after it is determined that the target grasping point pose can be grasped, the grasping path, angle, gripper, etc. are determined, which can save computing resources.

[0118] In operation S160, the first target object is grasped.

[0119] According to the embodiment of the present application, stopping grasping the first target object can be divided into two implementation measures of deleting in the priority queue or lowering the priority order in the priority queue. Some scenes where the crankshaft is very tightly stuck and cannot be loosened no matter how hard it is tried can be flexibly handled, and the situation where the crankshaft is not grasped again after trying to grasp it. Or in other scenes, as the crankshaft is grasped, the crankshaft that is not grasped may be loosened due to the influence of grasping, so for the crankshaft that is tried but not grasped, only the priority is lowered, and when the grasping reaches a certain degree, the situation where the grasping is tried again. Thus, it can be adapted to the complex and variable working conditions in the grasping process, and the overall grasping efficiency is improved.

[0120] Based on the above object grasping method, the present application further provides an object grasping device. The following will be combined Figure 8 with the description of the device.

[0121] Figure 8 The structure block diagram of the object grasping device 800 according to the embodiment of the present application is schematically shown.

[0122] As Figure 8 shown, the object grasping device 800 of the embodiment includes a pose acquisition module 810, a pose determination module 820, a distance calculation module 830, and an object grasping module 830.

[0123] The pose obtaining module 810 may, for example, perform operation S110, to obtain S first grasping point poses of a first target object, wherein the first target object comprises an object determined from the N objects to be grasped.

[0124] The pose determining module 820 may, for example, perform operation S120, to determine a target grasping point pose from the S first grasping point poses.

[0125] The distance calculating module 830 may, for example, perform operation S130, to calculate a pose distance between the target grasping point pose and at least one second grasping point pose of the M second grasping point poses, wherein the M second grasping point poses comprise a pose of at least one failed grasping object of the N objects to be grasped, and S, M or N is an integer greater than or equal to 1.

[0126] The object grasping module 840 may, for example, perform operation S150, to stop grasping the first target object if any of the calculated pose distances is less than a first distance threshold. The object grasping module 840 may also perform operation S160, to grasp the first target object based on the target grasping point pose if any of the calculated pose distances is greater than or equal to the first distance threshold.

[0127] The object grasping module 840 may, for example, also be configured to perform operation S730, to delete the target grasping point pose from a priority queue if any of the calculated pose distances is less than a second distance threshold, wherein the second distance threshold is less than the first distance threshold, and the priority queue is used to determine a grasping order of at least one pending target object of the N objects to be grasped, and the first target object is a highest-priority object of the at least one pending target object.

[0128] The object grasping module 840 may, for example, also be configured to perform operation S740, to lower a priority order of the target grasping point pose in the priority queue if any of the calculated pose distances is greater than or equal to the second distance threshold and less than the first distance threshold.

[0129] The object grasping device 800 can further include a grasping point pose updating module. The grasping point pose updating module can perform operation S210 to operation S220. In a case where any one of the calculated poses is less than a first distance threshold, the grasping point pose updating module calculates an attribute difference between a first region on the first target object and at least one of S-1 second regions, where the first region is a region where the target grasping point pose is located, and the S-1 second regions one-to-one correspond to regions where the remaining first grasping point poses in the S first grasping point poses are located. The attribute difference is used to represent a difference between attribute information of the first region and attribute information of the second region. In a case where the attribute difference between the first region and any one of the S-1 second regions satisfies a preset condition, the first grasping point pose corresponding to the second region is taken as a new target grasping point pose.

[0130] The grasping point pose updating module can further perform operation S310 to operation S350. The grasping point pose updating module obtains a shape contour of the first region and a shape contour of the i-th second region based on the image of the first target object, where i is an integer greater than or equal to 1 and less than or equal to S-1. The grasping point pose updating module calculates a shape contour difference between the shape contour of the first region and the shape contour of the i-th second region. In a case where the shape contour difference does not satisfy a first preset condition, i is set to i+1. In a case where the shape contour difference satisfies the first preset condition, the loop operation is stopped. The first grasping point pose corresponding to the i-th second region is taken as a new target grasping point pose.

[0131] The grasping point pose updating module can further perform operation S410 to operation S450. The grasping point pose updating module obtains a gravity distribution of the first region and a gravity distribution of the i-th second region, where i is an integer greater than or equal to 1 and less than or equal to S-1. The grasping point pose updating module calculates a gravity distribution difference between the gravity distribution of the first region and the gravity distribution of the i-th second region. In a case where the gravity distribution difference does not satisfy a second preset condition, i is set to i+1. In a case where the gravity distribution difference satisfies the second preset condition, the loop operation is stopped. The first grasping point pose corresponding to the i-th second region is taken as a new target grasping point pose.

[0132] The object grabbing device 800 can further include a pose recording module, and operations S510-S540 can be performed to, in a case where the first target object is failed to be grabbed, the first target object being a failed-to-be-grabbed object, write the target grabbing point pose as a second object pose into a failure record queue. Before the target grabbing point pose is written as the second object pose into the failure record queue, further including: if M is equal to N*S-1, deleting the earliest recorded second grabbing point pose in the M second grabbing point poses from the failure record queue. The maximum pose record quantity of the failure record queue can be set to be less than or equal to N*S-1 based on the quantity N of the N objects to be grabbed and the quantity S of the grabbing point poses of each object to be grabbed, where M is less than or equal to N*S-1.

[0133] Figure 9 A structural diagram of a computing device according to an embodiment of the present application is shown, and the embodiment of the present application does not limit the specific implementation of the computing device.

[0134] As shown in Figure 9 the computing device can include a processor 902, a communications interface 904, a memory 906, and a communications bus 908.

[0135] Among them:

[0136] The processor 902, the communications interface 904, and the memory 906 complete mutual communication through the communications bus 908.

[0137] The communications interface 904 is configured to communicate with network elements such as clients or other servers.

[0138] The processor 902 is configured to execute the program 910, and specifically can execute the related steps in the above object grabbing method embodiments.

[0139] Specifically, the program 910 can include program code including computer operation instructions.

[0140] The processor 902 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the computing device can be the same type of processor, such as one or more CPUs. It can also be different types of processors, such as one or more CPUs and one or more ASICs.

[0141] Memory 906, for storing programs 910. Memory 906 can include both RAM and nonvolatile memory, such as at least one disk storage.

[0142] The program 910 can specifically be used to make the processor 902 perform the object grasping method in any of the above method embodiments. The specific implementation of each step in the program 910 can refer to the corresponding description in the corresponding steps and units in the above object grasping embodiments, which will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the foregoing method embodiments, which will not be described here

[0143] The application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments. It can also exist independently and not be assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0144] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus.

[0145] Various general purpose systems can also be used with the teachings herein. General purpose systems typically include a floppy disk drive, a CD-ROM drive, a tape drive, a disk array, a disk drive, a memory, a hard drive, or a combination thereof. General purpose systems can also include a removable memory port. A user can enter commands and information into the general purpose system through input devices such as a keyboard, a mouse, a pen, a microphone, a trackball, a touch screen, a touch pad, a voice recognition system, a gesture recognition system, a bar code reader, or a combination thereof. These and other input devices are connected to the processing unit(s) through the user interface circuitry implemented in the input / output interface. The user interface circuitry presents information about the system on output devices such as a display screen, a printer, a speaker, a light emitting diode, a liquid crystal display, a cathode ray tube, a plasma display panel, a thin film transistor display, an organic light emitting diode display, a thin-film transistor display, a monitor, a television, a projector, a flat panel display, a solid state memory, a computer, a virtual system, or a combination thereof. One or more output devices are connected to the processing unit(s) through the user interface circuitry implemented in the input / output interface.

[0146] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0147] Similarly, it is to be understood that the above description is only one implementation of the application and that the application is not limited to the embodiments described above. Numerous alternatives will be readily apparent to those skilled in the art. For example, one or more features of the various embodiments of the application can be combined with each other, or the application can be used in conjunction with other technologies. Furthermore, many of the features of the application are optional and can be implemented or not implemented in various embodiments of the application. Furthermore, the description sometimes uses terms like "include" or "has," which themselves do not imply that "comprising," "containing," or "having" is the only way for the subject to be implemented. Other alternatives will be apparent to an artisan of ordinary skill in the art from the descriptions and figures provided.

[0148] Accordingly, the claims appended to this detailed description are hereby expressly incorporated into this detailed description, wherein each claim by itself is a separate embodiment of the application.

[0149] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into more sub-modules or sub-units or sub-components. Any combination of all the features described in the present specification (including the accompanying claims, abstract and drawings), and any method or apparatus of the present application so described is within the scope of the present application. Unless specifically stated otherwise, each feature of the present specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose, so that each feature is not essential to the present application.

[0150] Further, those skilled in the art will appreciate that a combination of features of different embodiments can be meant to be within the scope of the application and form a different embodiment. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0151] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Those skilled in the art will appreciate that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present application. The present application can also be implemented as a program (for example, a computer program and a computer program product) for performing part or all of the methods described herein on a device or an apparatus. Such a program implementing the present application can be stored on a computer readable medium, or can have one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0152] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain task. The use of the term'means' in a claim is intended to refer to a combination of devices, apparatuses or means for carrying out a task. The word 'first','second', 'third', etc. do not imply any order. The use of these terms is to be construed as an indication of particular embodiments. Steps in the above-described embodiments, unless otherwise specified, are not to be construed as necessarily limiting the order in which the steps are performed.

Claims

1. An object grasping method based on grasping point pose, comprising: Obtain S first grasping point poses of the first target object, wherein the first target object includes objects determined from N objects to be grasped, and the first grasping point poses are defined at the grasping position for the first target object and represented by rotation and translation matrices. The target grasping point pose is determined from the S first grasping point poses; Calculate the pose distance between the target grasping point pose and at least one of the M second grasping point poses, wherein the M second grasping point poses include the pose of at least one failed grasping object among the N objects to be grasped, the second grasping point pose is defined at the grasping position for the failed grasping object, and is represented by a rotation and translation matrix, where S, M and N are all integers greater than or equal to 1; If any of the calculated pose distances is less than a first distance threshold, the grasping of the first target object is stopped.

2. The method according to claim 1, wherein, If any of the calculated pose distances is less than a first distance threshold, the method further includes: Calculate the attribute difference between a first region on the first target object and at least one of the S-1 second regions, wherein the first region is the region where the target grasping point pose is located, and the S-1 second regions correspond one-to-one with the regions where the other first grasping point poses are located in the S first grasping point poses, and the attribute difference is used to characterize the degree of difference between the attribute information of the first region and the attribute information of the second region. If the attribute difference between the first region and any one of the S-1 second regions meets the preset conditions, the first grasping point pose corresponding to the second region is taken as the new target grasping point pose.

3. The method according to claim 2, wherein, The preset conditions include a first preset condition. The calculation of the attribute difference between the first region on the first target object and at least one of the S-1 second regions includes: setting i = 1, and repeatedly performing the following operations. Based on the image of the first target object, the shape contour of the first region and the shape contour of the i-th second region are obtained, where i is an integer greater than or equal to 1 and i is less than or equal to S-1. The shape contour difference between the shape contour of the first region and the shape contour of the i-th second region is calculated; If the shape contour difference does not meet the first preset condition, let i = i + 1.

4. The method according to claim 3, wherein, The method further includes: If the shape contour difference meets the first preset condition, the loop operation stops; and The pose of the first grasping point corresponding to the i-th second region is taken as the new target grasping point pose.

5. The method according to claim 2, wherein, The preset conditions include a second preset condition. The calculation of the attribute difference between the first region on the first target object and at least one of the S-1 second regions includes: setting i = 1, and repeatedly performing the following operations. Obtain the gravity distribution of the first region and the gravity distribution of the i-th second region, where i is an integer greater than or equal to 1 and i is less than or equal to S-1; The difference in gravity distribution between the first region and the i-th second region is calculated. If the difference in gravity distribution does not meet the second preset condition, let i = i + 1.

6. The method according to claim 5, wherein, The method further includes: The cyclic operation stops when the gravity distribution difference meets the second preset condition; and The pose of the first grasping point corresponding to the i-th second region is taken as the new target grasping point pose.

7. The method according to claim 1, wherein, The step of stopping the grasping of the first target object when any of the calculated pose distances is less than the first distance threshold includes: If any of the calculated pose distances is less than the second distance threshold, the target grasping point pose is removed from the priority queue, wherein the second distance threshold is less than the first distance threshold, and the priority queue is used to determine the grasping order of at least one undetermined target object among the N objects to be grasped, and the first target object is the object with the highest priority among the at least one undetermined target object.

8. The method according to claim 7, wherein, The step of stopping the grasping of the first target object when any of the calculated pose distances is less than the first distance threshold further includes: If any of the calculated pose distances is greater than or equal to the second distance threshold and less than the first distance threshold, the priority order of the target grab point pose in the priority queue is reduced.

9. The method according to claim 1, wherein, The method further includes: If any of the calculated pose distances is greater than or equal to the first distance threshold, the first target object is grasped based on the target grasping point pose.

10. The method according to claim 9, wherein, The failure record queue records the poses of the M second grasping points, and the method further includes: If the first target object fails to be grasped, the first target object is a failed grasp object, and the pose of the target grasp point is written as the pose of the second object into the failure record queue.

11. The method according to claim 10, wherein, The method further includes: Based on the number N of the N objects to be grabbed and the number S of the grab point poses of each object, the maximum number of pose records in the failure record queue is set to be less than or equal to N*S-1, where M is less than or equal to N*S-1. Before writing the target grab point pose as the second object pose into the failure record queue, the method further includes: If M equals N*S-1, then the earliest recorded second grasping point pose among the M second grasping point poses is deleted from the failure record queue.

12. An object grasping device based on grasping point pose, comprising: The pose acquisition module is used to acquire the poses of S first grasping points of the first target object, wherein the first target object includes an object determined from N objects to be grasped, and the poses of the first grasping points are defined at the grasping position for the first target object and are represented by a rotation and translation matrix. The pose determination module is used to determine the pose of the target grasping point from the S first grasping point poses; The distance calculation module is used to calculate the pose distance between the target grasping point pose and at least one of the M second grasping point poses, wherein the M second grasping point poses include the pose of at least one failed grasping object among the N objects to be grasped, the second grasping point pose is defined at the grasping position for the failed grasping object, and is represented by a rotation and translation matrix, where S, M and N are all integers greater than or equal to 1; The object grasping module is used to stop grasping the first target object if any of the calculated pose distances is less than a first distance threshold.

13. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 11.

Images