A robot arm sorting and grasping control method, system and device

By obtaining point cloud data at the end of the robot arm, using ICP point cloud registration algorithm and path planning algorithm, combined with background interference analysis, correcting the offset and planning the crawling path, the problem of positioning error of the robot arm in complex environments is solved. Improved grab accuracy and adaptability.

CN118809624BActive Publication Date: 2024-11-15TIANJIN SHIJUE INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411303120.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-19
Publication Date
2024-11-15
Estimated Expiration
2044-09-19

AI Technical Summary

Technical Problem

Misjudgment and positioning errors caused by background interference and scale changes in robotic arms in complex environments affect the accuracy of grabbing strategies and path planning.

Method used

By obtaining point cloud data at the end of the robot arm, using ICP point cloud registration algorithm and path planning algorithm, combined with the analysis of background interference degree, correct the offset and plan the grab path to enhance the robot arm's adaptability to complex environments.

Benefits of technology

It improves the accuracy of object positioning, reduces the error in scale estimation, and enhances the grasping accuracy and adaptability of the robotic arm in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118809624B_ABST
    Figure CN118809624B_ABST
Patent Text Reader

Abstract

The present application relates to the field of robotic arm grasping control technology, and specifically to a robotic arm sorting grasping control method, system and device, the method comprising: obtaining point cloud data of a target object collected by the end of the robotic arm during the sorting grasping process; determining the degree of background interference to the robotic arm grasping target; determining the difference in the change of background interference to the robotic arm grasping target at adjacent moments; obtaining the offset of the robotic arm grasping target in the point cloud data of the target object at adjacent moments; correcting the offset based on the difference in change and planning the robotic arm grasping path; obtaining the position and posture of the robotic arm when the end effector of the robotic arm reaches the end point; and controlling the robotic arm sorting grasping in combination with the robotic arm grasping path, the position and posture. The present application enhances the adaptability of the robotic arm to complex environments and the accuracy of task execution by dynamically adjusting the offset and path planning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of robot arm grasping control, and in particular to a robot arm sorting and grasping control method, system and device. Background Art

[0002] Robotic arm sorting and grasping control technology is a key technology in the field of industrial automation. With the rise of Industry 4.0, intelligent manufacturing has become a trend in global industrial development. As an important tool for realizing automated production, robotic arms can quickly and accurately complete tasks such as sorting, handling, and assembly, reduce the need for manual operation, and reduce production costs. Therefore, the development of sorting and grasping control technology is particularly important.

[0003] When the robot arm is controlled to move, the change of the camera angle of view will cause the background environment in the image to change due to the movement of the robot arm. This background change will introduce noise and irrelevant information, thereby interfering with the accurate recognition and positioning of the target object. When the distance between the camera and the target object changes, even if the actual size of the target object does not change, its proportion in the image will change accordingly. This scale change may lead to misjudgment of the actual size of the object, thereby affecting the calculation of the offset of the object relative to the robot arm, resulting in errors in the robot arm's continuous tracking of the position and size of the object, which in turn affects the robot arm's grasping strategy and path planning. Due to the insufficient recognition and compensation of background interference and scale changes, the existing technology cannot effectively solve the problem of accurate recognition and positioning of target objects in complex environments, causing the position and posture offset of the robot arm end effector during grasping and the accuracy of grasping path planning. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a robot arm sorting and grasping control method, system and device, and the technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a robot arm sorting and grasping control method, the method comprising the following steps:

[0006] Obtaining point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object includes the area where each independent object is located;

[0007] Based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robot grasps the target and the boundary points in the two-dimensional point cloud data of the area where all other independent objects are located projected onto the XY plane, the degree of background interference to the robot grasping target is determined;

[0008] Based on the distribution similarity between the boundary points of the point cloud data of the area where the robot grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot grasping target, the change difference of the background interference to the robot grasping target at adjacent moments is determined;

[0009] The ICP point cloud registration algorithm is used to obtain the offset of the target grasped by the robot arm in the point cloud data of the target object at adjacent moments; the offset is corrected based on the change difference, the actual position of the target grasped by the robot arm in the point cloud data at the next moment is re-determined, and the path planning algorithm is used to plan the grasping path of the robot arm;

[0010] The translation deviation, translation direction and rotation matrix output by the modified ICP point cloud registration algorithm are used to output the position and posture of the robot arm when the end effector of the robot arm reaches the end point using inverse dynamics; at the same time, the robot arm grasping path, the position and posture are combined to control the robot arm sorting and grasping.

[0011] Preferably, the method for determining the degree of background interference of the robot arm grasping the target is as follows:

[0012] The proportion of elements corresponding to the minimum distances smaller than the average level among all boundary points in the two-dimensional point cloud data of the area where the robot grasps the target is projected onto the XY plane is used as the degree of overall background interference to the robot grasping target;

[0013] Based on the concentrated distribution of elements corresponding to the minimum distances smaller than the average level in the minimum distances of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target and projects it onto the XY plane, determine the overall degree to which the robot arm grasps the target and is interfered by different backgrounds;

[0014] The degree of interference of the overall background to the target grasped by the robot arm is combined with the overall degree of interference from different backgrounds to obtain the degree of interference of the background to the target grasped by the robot arm.

[0015] Preferably, the method for determining the overall degree to which the robot arm grasps the target being disturbed by different backgrounds is as follows:

[0016] The minimum distances of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target are projected onto the XY plane form a boundary feature sequence;

[0017] Mark the elements in the boundary feature sequence that are smaller than the average level as 0, otherwise mark them as 1; obtain the run length of all elements that are smaller than the average level based on the marked values;

[0018] For each run, calculate the ratio of the run length of each run to the length of the boundary feature sequence; calculate the mean of the minimum distances of all elements in each run; calculate the ratio of the ratio value to the mean;

[0019] The sum of the ratio results of all strokes is taken as the overall degree to which the robot arm is subjected to interference from different backgrounds when grasping the target.

[0020] Preferably, the run is a sequence consisting of numerical values ​​of the same mark that appear continuously in the boundary feature sequence.

[0021] Preferably, the method for determining the difference in the change of the background interference of the robot arm grasping the target at adjacent moments is:

[0022] The structural similarity and distribution similarity are integrated to determine the feature similarity of the robot grasping the target at adjacent moments;

[0023] Obtain the difference in the degree of background interference when the robot grasps the target at adjacent moments;

[0024] Based on the feature similarity and the difference, the difference in changes in the background interference of the robot arm grasping the target at adjacent moments is determined.

[0025] Preferably, the method for determining the feature similarity of the robot arm grasping the target at adjacent moments is further obtained by multiplying the structural similarity and the distribution similarity.

[0026] Preferably, the process of obtaining the offset is: using the ICP point cloud registration algorithm for the point cloud data of the target object at adjacent moments, outputting the translation vector in the space variation matrix as the offset of the robot arm to grasp the target.

[0027] Preferably, the specific method of correction is expressed as: , where is the offset of the corrected robot arm grasping the target, is the offset of the robot grasping the target, B is the difference in the background interference of the robot grasping the target at adjacent moments, is the preset adjustment factor.

[0028] In a second aspect, an embodiment of the present application provides a robot arm sorting and grasping control system, the system comprising:

[0029] The robot arm grasping data acquisition module is used to obtain the point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object contains the area where each independent object is located;

[0030] The robot grasping target analysis module is used to determine the degree of background interference of the robot grasping target based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robot grasping target is projected onto the XY plane and the boundary points in the two-dimensional point cloud data of the area where all other independent objects are projected onto the XY plane;

[0031] Based on the distribution similarity between the boundary points of the point cloud data of the area where the robot grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot grasping target, the change difference of the background interference to the robot grasping target at adjacent moments is determined;

[0032] The robot grasping path planning module is used to obtain the offset of the robot grasping target in the point cloud data of the target object at adjacent moments by using the ICP point cloud registration algorithm; correct the offset based on the change difference, re-determine the actual position of the robot grasping target in the point cloud data at the next moment, and plan the robot grasping path by using the path planning algorithm;

[0033] The translation deviation, translation direction and rotation matrix output by the modified ICP point cloud registration algorithm are used inverse dynamics to output the position and posture of the robot arm when the end effector of the robot arm reaches the end point; at the same time, the robot arm sorting and grasping are controlled by combining the robot arm grasping path, the position and posture.

[0034] In the third aspect, an embodiment of the present application also provides a robot arm sorting and grasping control device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, it implements the steps of any one of the above-mentioned robot arm sorting and grasping control methods.

[0035] It can be seen from the above embodiments that the robot arm sorting and grasping control method, system and device provided in the embodiments of the present application have at least the following beneficial effects:

[0036] This application proposes a robot arm sorting and grasping control method, system and device, which aims to solve the problem of the influence of background point cloud on the scale estimation of target object, and combines alpha Shapes algorithm is used to analyze the distribution of all boundary points in the two-dimensional point cloud data of the area where the robot grasps the target and project it to the XY plane, and determine the degree of background interference on the target grasped by the robot. It is used to reflect the degree to which the boundary points of the target object are affected by the background point cloud, and solves the influence of background interference on the accuracy of scale estimation. In view of the instability of object scale estimation under continuous adjacent moments, the structural similarity measurement algorithm and the three-dimensional scale-invariant feature transformation matching algorithm are used to analyze the similarity between the distribution of boundary points of the point cloud data in the area where the robot grasps the target and the contours composed of boundary points in the two-dimensional point cloud data. Combined with the degree of background interference on the target grasped by the robot, the influence of background interference on the stability of object scale estimation in continuous adjacent moments is eliminated. The original offset of the object is obtained through the ICP point cloud registration algorithm, and the offset is corrected in combination with the change difference. The robot grasping path is planned to achieve more accurate object position positioning, reduce the positioning error caused by scale change, improve the accuracy of object scale estimation and the accuracy of robot positioning, and enhance the adaptability of the robot to complex environments and the accuracy of task execution by dynamically adjusting the offset and path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flowchart of a method for controlling a robot arm sorting and grasping according to an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a robot arm sorting and grasping control device provided in one embodiment of the present application;

[0040] Figure 3 A flow chart of a method for determining the degree to which a robot arm grasping a target is interfered by background interference provided in one embodiment of the present application;

[0041] Figure 4 A flow chart of a method for determining the overall degree to which a robot arm grasps a target subject to interference from different backgrounds, provided in one embodiment of the present application;

[0042] Figure 5A flow chart of a method for determining the difference in changes in background interference of a target grasped by a robot arm at adjacent moments provided in one embodiment of the present application;

[0043] Figure 6 A schematic diagram of a module of a robotic arm sorting and grasping control system provided for one embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of a robot arm sorting and grasping control method, system and device proposed in the present application, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0045] Unless otherwise specified and limited, terms such as "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such articles or devices. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items. All technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application.

[0046] The following is a detailed description of a specific scheme of a robot arm sorting and grasping control method, system and device provided by the present application in conjunction with the accompanying drawings.

[0047] See also Figure 1 , which shows a flowchart of a robot arm sorting and grasping control method provided by an embodiment of the present application, the method comprising the following steps:

[0048] The first step is to obtain the point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object contains the area where each independent object is located.

[0049] In the embodiment of the present application, a six-degree-of-freedom robotic arm is selected as the control target, an RGB-D camera is rigidly connected to the six-degree-of-freedom robotic arm, the RGB-D camera is fixed to the end effector of the robotic arm, so that the depth camera can move with the robotic arm, that is, the EYE-IN-HAND connection method is adopted, and the time interval of the image captured by the RGB-D camera is set to 0.1s, and images are captured by the end of the robotic arm during the sorting and grasping process.

[0050] In the embodiment of the present application, a schematic diagram of a robot arm sorting and grasping control device is shown in the attached Figure 2 shown.

[0051] In the attached Figure 2 In the figure, T1 is the platform on which the robot is installed, T2 is the robot body, T3 is the end of the robot, and T4 is the RGB-D camera.

[0052] Through the hand-eye calibration of the RGB-D camera vision sensor, the target object can be accurately positioned in the three-dimensional space, and the RGB image and corresponding depth image containing the target object can be obtained. The collected RGB image and depth image are converted to generate point cloud data of the target object.

[0053] At the same time, the three-dimensional kd tree algorithm is used to perform spatial filtering on the point cloud data to remove the desktop point cloud in the background and retain only the point cloud data related to the target object. The point cloud data converted from the RGB-D image retains the color information. On this basis, the filtered point cloud data is used as input, and the color-based region growing algorithm is used to automatically select seed points to segment the point cloud data of the target object, and the point cloud data of the target object is segmented into individual independent objects. Each growth region output by the color-based region growing algorithm can be used to distinguish the individual objects in the point cloud data for subsequent analysis.

[0054] Among them, the color-based region growing algorithm and the three-dimensional kd tree algorithm are both well-known technologies and will not be described in detail.

[0055] The second step is to determine the degree of background interference to the target being grasped by the robotic arm based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robotic arm grasps the target and the boundary points in the two-dimensional point cloud data of the areas where all other independent objects are located and projected onto the XY plane.

[0056] In the process of the robotic arm grasping an independent object, when the background and the target object are very close in space or overlap, the filtering algorithm cannot completely remove all background point clouds, which include the environmental point clouds and the point clouds of other adjacent objects. The background point clouds in the independent object will have a significant impact on the displacement of the robotic arm, interfering with the robotic arm's accurate estimation of the object scale.

[0057] If the background point cloud is mixed with the target object point cloud, the boundary of the object will be unclear, resulting in errors in the scale information estimated by the robot arm. The control system of the robot arm will incorrectly calculate the scale of the object, resulting in inaccurate grasping points, which will cause the robot arm to move to the wrong position and fail to successfully grasp the target object.

[0058] To avoid collisions, the robot arm usually observes from above to reduce the risk of collision, so scale occlusions at different object heights in the point cloud data will not cause scale interference. For example, if the robot arm observes from top to bottom, then scale interference usually occurs in the XY plane and will not affect the top view of the object.

[0059] For the Mth depth image collected, the independent object that the robot needs to grasp is recorded as the robot grasping target, and the point cloud data of the robot grasping target is used as input. The point cloud data of the robot grasping target is mapped to the XY plane through projection to obtain the two-dimensional point cloud data of the robot grasping target, and the alpha shapes algorithm is used to output the contour image composed of boundary points and boundary points in the two-dimensional point cloud data of the robot grasping target, and the same method is used to output the boundary points of the two-dimensional point cloud data of all independent objects. Among them, the three-dimensional point cloud data is mapped to the two-dimensional space, and the boundary points output by the alpha shapes algorithm are in a closed connected state. The alpha shapes algorithm is a well-known technology and will not be repeated.

[0060] In order to further enhance the grasping accuracy of the target object by the robot arm, for each boundary point in the two-dimensional point cloud data of the area where the robot arm grasps the target is projected to the XY plane, the minimum distance between each boundary point and the boundary points in the two-dimensional point cloud data of the area where the remaining independent objects are projected to the XY plane is calculated.

[0061] In various implementations of the embodiments of the present application, Euclidean distance, DTW distance, etc. may be used to calculate the distance. Euclidean distance and DTW distance are well-known technologies and will not be described in detail.

[0062] A boundary point is randomly selected as the starting point and the end point, and the walking starts from any position of the starting point. The minimum distance between each boundary point and the boundary points of the other independent objects is formed into a boundary feature sequence for the robot arm to grasp the target in the walking order, so that the robot arm can more accurately identify the boundary of the target object.

[0063] In the embodiment of the present application, the flow chart of the method for determining the degree of background interference of the robot arm grasping the target is as shown in the attached figure. Figure 3 As shown, specifically:

[0064] Q1, based on the minimum distance of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target projected to the XY plane, the proportion of elements corresponding to the minimum distance that is smaller than the average level is used as the degree of overall background interference to the robot arm grasping target.

[0065] The mean of all elements in the boundary feature sequence of the target grasped by the robot arm is calculated to evaluate the relative distance between the boundary point of the target grasped by the robot arm and the rest of the objects, and the proportion of elements in the boundary feature sequence whose minimum distance is less than the mean is calculated.

[0066] By calculating the proportion of elements in the boundary feature sequence whose minimum distance is less than the mean, the degree a of background interference when the robot grasps the target is obtained. a reflects the degree to which the boundary points of the object are affected by the background point cloud. The larger the value of a, the closer the spatial distance between the boundary points of the target object grasped by the robot and the background point cloud is, which increases the difficulty of positioning when the robot grasps the target, making the robot more susceptible to background interference when grasping.

[0067] Q2, based on the concentrated distribution of elements corresponding to the minimum distances of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target and is projected onto the XY plane, which are smaller than the average level, determines the overall degree to which the robot arm grasps the target being interfered by different backgrounds.

[0068] In the embodiment of the present application, a flow chart of a method for determining the overall degree to which a robot arm grasps a target and is disturbed by different backgrounds is shown in the attached figure. Figure 4 As shown, specifically:

[0069] The elements in the boundary feature sequence of the target to be grasped by the robot arm that are greater than or equal to the mean are marked as 1, and the elements that are less than the mean are marked as 0. The runs of all elements less than the mean are obtained, where the run is a sequence of values ​​of the same mark that appear continuously in the boundary feature sequence. The proportion of the length of each run to the length of the boundary feature sequence is calculated, and the mean of the minimum distances between all elements in each run and the boundary points of other independent objects is calculated.

[0070] set up is the ratio of the length of the i-th run to the length of the boundary feature sequence. Each run can be considered as the influence of a background point cloud on the robot arm grasping the target. The larger the value of , the greater the influence range of the i-th background point cloud on the robot arm's grasping of the target point cloud; is the mean of the minimum distances between all elements in the ith run and the boundary points of the remaining independent objects, The smaller the value of, the stronger the influence of the i-th background point cloud on the robot arm's grasping of the target point cloud.

[0071] By calculating the relationship Calculate the overall degree b of the robot arm's grasping target affected by different background interferences. Y is the number of runs in the boundary feature sequence. The calculation of b takes into account the differences in the impact of background point clouds in different regions on the target object. It can be understood as the impact of background objects in different directions or positions on the robot arm's grasping accuracy during the grasping process. The larger the b value, the more complex or significant the impact of the background point cloud on the target object, which means that the robot arm is subject to more background interference in a specific direction.

[0072] Q3, the degree to which the robot arm grasps the target affected by the overall background interference is integrated with the overall degree to which the robot arm grasps the target affected by different background interferences to obtain the degree to which the robot arm grasps the target affected by the background interferences.

[0073] The obtained degree of background interference of the target grasped by the robotic arm is used to evaluate the influence of background interference on the accuracy of the target object when the robotic arm grasps the target object.

[0074] It can be understood that fusion can be divided into forward fusion and reverse fusion. This embodiment adopts the forward fusion method. Forward fusion is a fusion method such as addition and multiplication between data. The specific forward fusion method is determined by the implementer to be a suitable fusion method, and this application does not impose any special restrictions.

[0075] In the embodiment of the present application, the relationship is Calculate the degree of background interference A when the robot grasps the target. A represents the degree of influence of background interference on the accuracy of the target object when the robot grasps the target object.

[0076] In other embodiments of the present application, the relationship is Calculate the degree A of background interference when the robot grasps the target.

[0077] It should be understood that the larger the A value is, the greater the impact of the surface background on the grasping path and accuracy of the robot arm.

[0078] The calculation of A provides the robot arm with a quantitative assessment of the impact of background interference on the grasping accuracy of the target object, which helps to optimize the grasping strategy and improve the accuracy and reliability of the robot arm's grasping operation. By identifying and quantifying background interference, the robot arm can better adapt to complex environments and achieve precise grasping.

[0079] The third step is to determine the difference in the change of background interference to the robot arm's grasping target at adjacent moments based on the distribution similarity between the boundary points of the point cloud data of the area where the robot arm grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot arm's grasping target.

[0080] In the EYE-IN-HAND robotic arm, the position of the camera will change during the movement of the robotic arm, and the position of the target object to be grasped in the captured image will also change.

[0081] In two RGB images taken continuously, the Horn-Schunck optical flow algorithm is used to track the target grasped by the robot arm, ensuring that the robot arm updates the position information of the target object in real time, thereby adjusting its motion trajectory to achieve accurate grasping. The Horn-Schunck optical flow algorithm is a well-known technology and will not be described in detail.

[0082] At the same time, the above method is used to calculate the contour image of the target grasped by the robot arm in the M+1th depth image composed of the boundary points in its two-dimensional point cloud data, as well as the degree of background interference to the target grasped by the robot arm in the M+1th depth image , to evaluate the impact of background factors on target object recognition and positioning, which helps the robotic arm control system adjust its strategy and reduce errors.

[0083] In the embodiment of the present application, the flow chart of the method for determining the difference in the background interference of the robot arm grasping the target at adjacent moments is as shown in the attached figure. Figure 5 As shown, specifically:

[0084] W1, combines structural similarity with distribution similarity to determine the feature similarity of the robot grasping the target at adjacent moments.

[0085] For the contour images of the target grasped by the robot arm in the Mth and M+1th depth images, which are composed of the boundary points in the two-dimensional point cloud data, the two contour images are used as input, and the structural similarity between the two contour images is output using the structural similarity measurement (SSIM) algorithm. Among them, the structural similarity measurement algorithm is a well-known technology and will not be described in detail. In other implementations of the embodiments of the present application, other methods such as mean square error can also be used to calculate the structural similarity.

[0086] Subsequently, the point cloud data of the area where the robot grasps the target in the Mth and M+1th depth images is used as input, and the three-dimensional scale-invariant feature transform matching (SIFT3D) algorithm is used to output the distribution similarity between the point cloud data of the area where the robot grasps the target. Among them, the three-dimensional scale-invariant feature transform matching (SIFT3D) algorithm is a well-known technology and will not be described in detail. In other implementations of the embodiments of the present application, other methods such as the iterative closest point algorithm can also be used to calculate the distribution similarity.

[0087] For the Mth and M+1th depth images, the feature similarity c between the robot grasping targets in the two depth images at different shooting positions is obtained by fusing the structural similarity between the two contour images with the distribution similarity between the point cloud data of the area where the robot grasping target is located.

[0088] In the embodiment of the present application, the product of the structural similarity and the distribution similarity is used as the feature similarity. In other embodiments of the embodiment of the present application, the sum of the structural similarity and the distribution similarity is used as the feature similarity.

[0089] It should be understood that c quantifies the similarity between the targets grasped by the robot in the two depth images, structural similarity measures the structural similarity of the two contour images, reflecting the visual similarity of the edge content of the point cloud data of the target grasped by the robot, and distribution similarity measures the spatial distribution similarity of the point cloud data in the area where the robot grasps the target, including shape, size and relative position, etc., expressing the consistency of the robot's recognition of the target object in continuous actions. The larger the value of c, the more similar the features of the objects in the two images, which means that the background interference has less impact on the scale of the object, which helps the robot to grasp accurately.

[0090] W2, obtains the difference in the degree of background interference when the robot grasps the target at adjacent moments.

[0091] By calculating the difference d between the degree of background interference of the target grasped by the robot arm in the Mth and M+1th depth images, the degree of background interference of the target grasped by the robot arm in the two depth images at adjacent moments is obtained.

[0092] In the implementation of the embodiment of the present application, the absolute value of the difference between the degree of background interference of the robot arm grasping the target in the Mth and M+1th depth images is used as the difference. In other implementations of the embodiment of the present application, the difference between the degree of background interference of the robot arm grasping the target in the M+1th and Mth depth images can also be calculated as the difference.

[0093] It should be understood that d reflects the change in the impact of background interference on object scale estimation, that is, whether the background interference becomes more significant or decreases in consecutive images, and reflects the change in the background interference of the robot arm at different action stages. The larger the value of d, the more significant the change in background interference, the larger the error in object scale estimation, and the greater the impact of the background on the robot arm's grasping target, which helps the robot arm maintain accurate scale estimation and grasping.

[0094] W3, based on the feature similarity and the difference, determine the change difference of the background interference of the robot arm grasping the target at adjacent moments.

[0095] In the embodiment of the present application, the relationship is calculated as follows: , calculate the change difference B of the background interference of the robot arm grasping the target at adjacent moments. In another embodiment of the present application, by calculating the relationship: , calculate the change difference B of the background interference of the robot arm grasping the target at adjacent moments. In other implementations of the embodiment of the present application, by calculating the relationship: , calculate the change difference B of the background interference of the robot arm grasping the target at adjacent moments.

[0096] The specific implementation method can be reasonably selected by the implementer based on the feature similarity and the degree of influence of the difference on B, and the details will not be repeated here.

[0097] It should be understood that B evaluates the inaccuracy of the object scale estimation in the continuous depth images. The larger the value of B, the more unstable the object scale estimation in the continuous images, the greater the influence of background interference, and the greater the offset error of the robot grasping the target. By analyzing the B value, the main factors affecting the grasping accuracy can be identified, which can then serve as the basis for optimizing the error accumulation of the robot motion control strategy.

[0098] The same method is then used to calculate the difference in changes in background interference between each independent object in two adjacent depth images at adjacent moments.

[0099] The fourth step is to use the ICP point cloud registration algorithm to obtain the offset of the robot arm grasping target in the point cloud data of the target object at adjacent moments; correct the offset based on the change difference, re-determine the actual position of the robot arm grasping target in the point cloud data at the next moment, and use the path planning algorithm to plan the robot arm grasping path.

[0100] When controlling the robot, firstly, the point cloud data of the target object at the Mth and M+1th moments are used as input, and the ICP point cloud registration algorithm is used to align the point cloud data at the M+1th moment with the point cloud data at the Mth moment. The registration process will find the best spatial transformation, including rotation and translation, so that the point clouds at two adjacent moments overlap as much as possible in space. The calculated spatial transformation matrix contains the translation vector K of the target object from the Mth moment to the M+1th moment. The translation vector K in the spatial change matrix is ​​extracted and used as the offset of the robot grasping the target, as the deflection parameter that the robot needs to control the movement of the robot when the robot grasps the target. Among them, the ICP point cloud registration algorithm is a well-known technology and will not be repeated.

[0101] However, due to the movement of the robot arm in the acquisition interval between two consecutive moments, the acquired background point cloud data will affect the point cloud data of the target object and further increase the scale change of the target object at the two moments.

[0102] Therefore, the offset of the target grasped by the robot arm is corrected. The specific correction formula is: , where is the offset of the corrected robot arm grasping the target, is the offset of the robot grasping the target, B is the difference in the background interference of the robot grasping the target at adjacent moments, It is a preset adjustment coefficient, and in the embodiment of the present application, its value is 0.05.

[0103] It should be understood that the correction formula takes into account the scale changes caused by the movement of the robot and the background interference, by adjusting the coefficient Determines the strength of the correction. In the embodiment of the present application Being set to a smaller value means that the correction is a gradual process and will not have a significant impact on the original offset. The corrected offset is intended to provide a more accurate target object position and improve positioning accuracy by reducing the impact of scale changes.

[0104] It should be noted that The value of can be modified according to the actual conditions such as the lighting changes in the working environment, the reflective characteristics of the object, and the resolution and noise level of the point cloud data. The value is not unique.

[0105] After correcting the offset of the target grasped by the robot, the corrected offset is substituted into the spatial transformation matrix of the ICP point cloud registration algorithm to update the translation vector K in the transformation matrix. The updated translation vector should reflect the actual offset change, so as to obtain the actual position of the point cloud data of the target grasped by the robot at the M+1th moment. At the same time, the translation deviation, translation direction and rotation matrix output by the ICP point cloud registration algorithm can be obtained to characterize the correction results of the target position and posture of the end effector of the robot.

[0106] Furthermore, the starting point and end point of the robot arm's grasping path at the M+1th moment are defined. The starting point is the position when the robot arm starts to perform the task, that is, the position of the robot arm at the previous moment in the adjacent acquisition moments. The end point is the position that the robot arm needs to reach after completing the task, that is, the actual position of the robot arm grasping the target in the point cloud data at the M+1th moment.

[0107] The point cloud data, starting point and end point at the M+1th moment are used as input, and the RRT (rapidly explored random tree) path planning algorithm is used to generate a path that avoids obstacles, which is recorded as the robot arm grasping path to ensure the smoothness and efficiency of the robot arm's movement.

[0108] The target position and posture of the end effector of the manipulator are corrected according to the correction offset, so that the RRT algorithm can take into account the scale changes caused by the movement of the manipulator and the background interference, thereby providing the manipulator with an accurate grasping path, and adjusting the algorithm parameters according to the specific conditions of the working environment to achieve the best planning effect. Among them, the rapid exploration random tree path planning algorithm is a well-known technology and will not be repeated. In other implementations of the embodiments of the present application, the PRM probabilistic roadmap algorithm can also be used to obtain the manipulator grasping path.

[0109] At this point, the above method can be used to complete the planning of the robot arm's grasping path for the robot arm to grasp the target between two consecutive adjacent moments.

[0110] The fifth step is to use inverse dynamics to output the translation deviation, translation direction and rotation matrix output by the corrected ICP point cloud registration algorithm, and then output the position and posture of the robot arm when the end effector of the robot arm reaches the end point; at the same time, the robot arm's grasping path, the position and posture are combined to control the robot arm's sorting and grasping.

[0111] Substituting the corrected offset into the ICP point cloud registration algorithm, the translation deviation, translation direction and rotation matrix outputted are the inputs of inverse kinematics. Inverse kinematics converts the position and posture in 3D space into the solution of six joint variables, and applies the inverse kinematics algorithm to solve the angle of each joint, and outputs the position and posture of the robot arm when the end effector of the robot arm reaches the end point. Among them, inverse dynamics is a well-known technology and will not be described in detail.

[0112] Through the robot arm's grasping path of the robot arm to the robot arm grasping target and the position and posture of the robot arm when the robot arm end effector reaches the end point, precise motion control of the robot arm from the starting point to the end point is achieved, including precise arrival of position and posture, and the grasping control of the robot arm to the robot arm grasping target between consecutive adjacent acquisition moments is completed.

[0113] Please see attached Figure 6 , Figure 6 Schematic diagram of a module of a robot arm sorting and grasping control system provided in an embodiment of the present application. In this embodiment, each module included in the terminal is used to execute each step in an embodiment corresponding to a robot arm sorting and grasping control method.

[0114] See also Figure 6 The system includes: a robotic arm grasping data acquisition module, a robotic arm grasping target analysis module, and a robotic arm grasping path planning module.

[0115] The robot arm grasping data acquisition module is used to obtain the point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object contains the area where each independent object is located;

[0116] The robot grasping target analysis module is used to determine the degree of background interference of the robot grasping target based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robot grasping target is projected onto the XY plane and the boundary points in the two-dimensional point cloud data of the area where all other independent objects are projected onto the XY plane;

[0117] Based on the distribution similarity between the boundary points of the point cloud data of the area where the robot grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot grasping target, the change difference of the background interference to the robot grasping target at adjacent moments is determined;

[0118] The robot grasping path planning module is used to obtain the offset of the robot grasping target in the point cloud data of the target object at adjacent moments by using the ICP point cloud registration algorithm; correct the offset based on the change difference, re-determine the actual position of the robot grasping target in the point cloud data at the next moment, and plan the robot grasping path by using the path planning algorithm;

[0119] The translation deviation, translation direction and rotation matrix output by the modified ICP point cloud registration algorithm are used inverse dynamics to output the position and posture of the robot arm when the end effector of the robot arm reaches the end point; at the same time, the robot arm sorting and grasping are controlled by combining the robot arm grasping path, the position and posture.

[0120] Based on the same inventive concept as the above method, an embodiment of the present application also provides a robot arm sorting and grasping control device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above-mentioned robot arm sorting and grasping control methods are implemented.

[0121] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0122] It should be noted that, unless otherwise specified and limited, terms such as "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, an element defined by the sentence "including one..." does not exclude the existence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items.

[0123] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not invented by the present application.

[0124] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A robot arm sorting and grasping control method, characterized in that: The method comprises the following steps: Obtaining point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object includes the area where each independent object is located; Based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robot grasps the target and the boundary points in the two-dimensional point cloud data of the area where all other independent objects are located projected onto the XY plane, the degree of background interference to the robot grasping target is determined; Based on the distribution similarity between the boundary points of the point cloud data of the area where the robot grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot grasping target, the change difference of the background interference to the robot grasping target at adjacent moments is determined; The ICP point cloud registration algorithm is used to obtain the offset of the target grasped by the robot arm in the point cloud data of the target object at adjacent moments; the offset is corrected based on the change difference, the actual position of the target grasped by the robot arm in the point cloud data at the next moment is re-determined, and the path planning algorithm is used to plan the grasping path of the robot arm; The translation deviation, translation direction and rotation matrix output by the modified ICP point cloud registration algorithm are used to output the position and posture of the robot arm when the end effector of the robot arm reaches the end point using inverse dynamics; at the same time, the robot arm grasping path, the position and posture are combined to control the robot arm sorting and grasping.

2. A robot arm sorting and grasping control method as claimed in claim 1, characterized in that: The method for determining the degree to which the robot grasps the target and is disturbed by the background is as follows: The proportion of elements corresponding to the minimum distances smaller than the average level among all boundary points in the two-dimensional point cloud data of the area where the robot grasps the target is projected onto the XY plane is used as the degree of overall background interference to the robot grasping target; Based on the concentrated distribution of elements corresponding to the minimum distances smaller than the average level in the minimum distances of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target and projects it onto the XY plane, determine the overall degree to which the robot arm grasps the target and is interfered by different backgrounds; The degree of interference of the overall background to the target grasped by the robot arm is combined with the overall degree of interference from different backgrounds to obtain the degree of interference of the background to the target grasped by the robot arm.

3. A robot arm sorting and grasping control method as claimed in claim 2, characterized in that: The method for determining the overall degree to which the robot grasps the target being disturbed by different backgrounds is as follows: The minimum distances of all boundary points in the two-dimensional point cloud data of the area where the robot arm grasps the target are projected onto the XY plane form a boundary feature sequence; Mark the elements in the boundary feature sequence that are smaller than the average level as 0, otherwise mark them as 1; obtain the run length of all elements that are smaller than the average level based on the marked values; For each run, calculate the ratio of the run length of each run to the length of the boundary feature sequence; calculate the mean of the minimum distances of all elements in each run; calculate the ratio of the ratio value to the mean; The sum of the ratio results of all strokes is taken as the overall degree to which the robot arm is subjected to interference from different backgrounds when grasping the target.

4. A robot arm sorting and grasping control method as claimed in claim 3, characterized in that: The run is a sequence consisting of the values ​​of the same mark that appear continuously in the boundary feature sequence.

5. A robot arm sorting and grasping control method as claimed in claim 1, characterized in that: The method for determining the difference in the change of the background interference of the robot arm grasping the target at adjacent moments is as follows: The structural similarity and distribution similarity are integrated to determine the feature similarity of the robot grasping the target at adjacent moments; Obtain the difference in the degree of background interference when the robot grasps the target at adjacent moments; Based on the feature similarity and the difference, the difference in changes in the background interference of the robot arm grasping the target at adjacent moments is determined.

6. A robot arm sorting and grasping control method as claimed in claim 5, characterized in that: The method for determining the feature similarity of the robot arm grasping the target at adjacent moments is further obtained by multiplying the structural similarity and the distribution similarity.

7. A robot arm sorting and grasping control method as claimed in claim 1, characterized in that: The process of obtaining the offset is as follows: using the ICP point cloud registration algorithm on the point cloud data of the target object at adjacent moments, outputting the translation vector in the space variation matrix as the offset of the robot arm grasping the target.

8. A robot arm sorting and grasping control method as claimed in claim 1, characterized in that: The expression of the specific method of the correction is: , where is the offset of the corrected robot arm grasping the target, is the offset of the robot grasping the target, B is the difference in the background interference of the robot grasping the target at adjacent moments, is the preset adjustment factor.

9. A robot arm sorting and grasping control system, implementing the method as claimed in claim 1, characterized in that: The system comprises: The robot arm grasping data acquisition module is used to obtain the point cloud data of the target object collected by the end of the robot arm during the sorting and grasping process; the point cloud data of the target object contains the area where each independent object is located; The robot grasping target analysis module is used to determine the degree of background interference of the robot grasping target based on the distribution of the minimum distances between all boundary points in the two-dimensional point cloud data of the area where the robot grasping target is projected onto the XY plane and the boundary points in the two-dimensional point cloud data of the area where all other independent objects are projected onto the XY plane; Based on the distribution similarity between the boundary points of the point cloud data of the area where the robot grasps the target at adjacent moments, the structural similarity between the contours composed of the boundary points in its two-dimensional point cloud data, and the difference in the degree of background interference to the robot grasping target, the change difference of the background interference to the robot grasping target at adjacent moments is determined; The robot grasping path planning module is used to obtain the offset of the robot grasping target in the point cloud data of the target object at adjacent moments by using the ICP point cloud registration algorithm; correct the offset based on the change difference, re-determine the actual position of the robot grasping target in the point cloud data at the next moment, and plan the robot grasping path by using the path planning algorithm; The translation deviation, translation direction and rotation matrix output by the modified ICP point cloud registration algorithm are used to output the position and posture of the robot arm when the end effector of the robot arm reaches the end point using inverse dynamics; at the same time, the robot arm grasping path, the position and posture are combined to control the robot arm sorting and grasping.

10. A robot arm sorting and grasping control device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of a robot arm sorting and grasping control method as described in any one of claims 1-8 are implemented.

Citation Information

Cited By

  • Device capable of quickly and directionally sorting and accurately positioning bulk parts

    CN120618903A