Mechanical arm positioning and grabbing method and device, computer equipment and storage medium

Abstract

The invention relates to a mechanical arm positioning and grabbing method and device, computer equipment and a storage medium. According to the method, the registration technology is utilized to obtain the grabbing position of the actual object which is successfully registered, then the space conversion relation between the teaching object and the actual object is calculated, based on the space conversion relation, the information of the registered object is effectively utilized, parts which are not well imaged are compensated through an optimization algorithm, and the imaging accuracy is improved. And the grabbing position of the residual object which is not successfully registered is speculated, so that the problem of registration failure caused by poor imaging of the high-reflection part is solved. According to the method, the grabbing success rate and the reliability of the system are remarkably improved, it is guaranteed that the parts can be accurately grabbed in the high-reflection environment, and the stability and efficiency of the whole system are improved.

Description

Technical Field

[0001] The present application relates to the field of robotic arm technology, and in particular to a robotic arm positioning and grasping method, device, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of industrial automation, robots have been widely used, especially in the handling and assembly of parts. For part grasping, traditional methods mainly rely on preset paths and visual guidance systems. For conventional parts, visual devices such as 3D cameras or laser scanners can obtain their three-dimensional information more accurately, thereby guiding the robot to grasp them accurately. However, when encountering parts with highly reflective surfaces (such as painted parts), the problem becomes complicated. Highly reflective surfaces can cause interference such as light spots and reflections in the images obtained by the 3D camera, affecting the quality and integrity of the point cloud data, and thus affecting the accuracy of the registration algorithm.

[0003] In the prior art, some processing methods for highly reflective parts have been developed. For example:

[0004] (1) Filtering technology: By applying various filtering algorithms, such as median filtering and mean filtering, we try to reduce reflection interference.

[0005] (2) Light source adjustment: Reduce the impact of reflected light by adjusting the angle and intensity of the light source.

[0006] (3) Multi-view imaging: Using multi-view imaging technology, shooting from multiple angles can synthesize more accurate three-dimensional images.

[0007] (4) Feature extraction algorithm: Use more complex feature extraction algorithms to try to extract useful information from the disturbed point cloud data.

[0008] Although existing technologies can alleviate the impact of high reflection on image quality to a certain extent, there are still many limitations in practical applications:

[0009] (1) Limitations of filtering technology:

[0010] When dealing with high reflective interference, filtering technology often leads to loss of image details, affecting the accuracy of subsequent registration. The parameter setting in the filtering process is relatively complex and difficult to automatically adjust in a dynamic environment.

[0011] (2) Limitations of light source adjustment:

[0012] Light source adjustment requires precise control of lighting conditions, but in industrial production environments, lighting conditions are often difficult to maintain stable. The reflective properties of highly reflective parts vary greatly at different angles, and the light source adjustment method is difficult to adapt to the changing environment.

[0013] (3) Limitations of feature extraction algorithms:

[0014] Complex feature extraction algorithms require high computing resources and are difficult to run efficiently in real-time systems. The robustness of feature extraction algorithms is poor under high reflective interference, resulting in a high registration failure rate.

[0015] (4) Reliability of the overall system:

[0016] In a complex industrial environment, there are many interference factors, such as vibration, temperature change, etc., which will affect the effectiveness of existing technologies.

[0017] That is, the existing methods are difficult to guarantee sustained high precision and high reliability in practical applications, and the success rate and efficiency of grasping cannot meet the needs of industrial production. Summary of the invention

[0018] Based on this, it is necessary to provide a robot arm positioning and grasping method, device, computer equipment, computer readable storage medium and computer program product that can grasp with accuracy and reliability in order to solve the above technical problems.

[0019] In a first aspect, the present application provides a method for positioning and grasping a robot arm. The method comprises:

[0020] Acquire point cloud data of actual objects in a current grasping scene; there are multiple actual objects to be grasped in the current scene;

[0021] Registering the point cloud data of the actual object with the object template point cloud of the current grasping scene to obtain a successfully registered actual object and a grasping position of the successfully registered actual object;

[0022] Obtaining a spatial conversion relationship between the teaching object and the actual object according to the successfully registered grasping position of the actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene;

[0023] According to the spatial conversion relationship and the teaching grasping position of the teaching object, the target grasping position of all objects in the current grasping scene is obtained; the all objects include the actual objects that have been successfully registered and the remaining objects that have not been successfully registered;

[0024] Based on the target grasping positions of all the objects, the robotic arm is controlled to grasp the objects.

[0025] In one embodiment, the spatial transformation relationship includes a rotation matrix and a translation vector;

[0026] The method of obtaining a spatial conversion relationship between the teaching object and the actual object based on the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene includes:

[0027] Acquire a set of actual grasping points corresponding to the grasping position of the actual object that has been successfully registered;

[0028] Acquire a teaching object corresponding to the actual object in the current grasping scene, and a teaching grasping point set corresponding to the teaching grasping position of the teaching object;

[0029] Calculating the centroid of the actual grasping point set and the teaching grasping point set;

[0030] According to the centroid, the actual grasping point set and the teaching grasping point set are respectively converted into the centroid coordinate system to obtain a first coordinate set corresponding to the actual grasping point set and a second coordinate set corresponding to the teaching grasping point set;

[0031] Calculate the covariance matrix of the first coordinate set and the second coordinate set according to the first coordinate set and the second coordinate set;

[0032] Performing singular value decomposition on the covariance matrix, and calculating the rotation matrix between the teaching object and the actual object according to the decomposition result;

[0033] The translation vector is calculated according to the first coordinate set, the second coordinate set and the rotation matrix.

[0034] In one embodiment, the step of obtaining point cloud data of an actual object in a current grasping scene includes:

[0035] Collect the depth image and texture image of the current captured scene;

[0036] Detecting actual objects in the texture image to obtain the number of actual objects on the texture image and the positions of the actual objects on the texture image;

[0037] Segmenting the depth image and the texture image according to the positions of the actual objects to obtain a depth map and a texture map of each of the actual objects;

[0038] Point cloud data of each actual object is obtained by conversion according to the depth map and the texture map of each actual object.

[0039] In one embodiment, the teaching grasping position of the teaching object includes the horizontal coordinate and the vertical coordinate of the teaching grasping position; the target grasping position includes the horizontal coordinate, the vertical coordinate and the depth value of the target grasping position;

[0040] The calculation formula for the target grasping position of all objects in the current grasping scene is:

[0041]

[0042] Among them, x i and i are the horizontal and vertical coordinates of the teaching grasping position, R is the rotation matrix, Z avg is the average depth value of the actual object that has been successfully registered, and t is the translation vector.

[0043] In one embodiment, the point cloud data of the actual object is registered with the object template point cloud used in the current grasping scene during the teaching process to obtain the successfully registered actual object and the grasping position of the successfully registered actual object, including:

[0044] Extracting feature points of the point cloud data of the actual object and the object template point cloud of the current capture scene;

[0045] Perform feature point matching through feature descriptors to obtain matching feature point pairs between the point cloud data of the actual object and the point cloud of the object template;

[0046] Obtaining an initial registration result of the point cloud data of the actual object and the point cloud of the object template according to the matching feature point pairs;

[0047] The initial registration result is optimized by using a truncated least squares estimation method to obtain a rough registration result;

[0048] Perform ICP registration according to the rough registration result to obtain the corresponding relationship between the point cloud data of the actual object and the point cloud of the object template, as well as the actual object that has been successfully registered;

[0049] The grasping position of the successfully registered actual object is calculated according to the corresponding relationship between the point cloud data of the actual object and the point cloud of the object template.

[0050] In one embodiment, the method further comprises:

[0051] Determine whether the number of successfully registered actual objects is greater than a preset value. If the number of successfully registered actual objects is greater than the preset value, stop the registration, and execute the step of obtaining the spatial conversion relationship between the teaching object and the actual object based on the grasping position of the successfully registered actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene.

[0052] In one embodiment, the method further comprises:

[0053] The first camera and the robot arm are pre-calibrated in the eye-on-hand manner, and the relationship between the second camera and the flange end of the robot arm is pre-calibrated; the first camera is used to collect the depth image and texture image of the current grasping scene; the second camera is fixed to the flange end of the robot arm to perform part grasping;

[0054] An object template point cloud is prepared in advance, and a teaching object is grasped based on the object template point cloud, and a grasping position of the teaching object is recorded.

[0055] In a second aspect, the present application also provides a robot arm positioning and grasping device. The device comprises:

[0056] An acquisition module, used to acquire point cloud data of an actual object in a current grasping scene; there are multiple actual objects to be grasped in the current scene;

[0057] A registration module, used to register the point cloud data of the actual object with the object template point cloud of the current grasping scene, to obtain a successfully registered actual object and a grasping position of the successfully registered actual object;

[0058] A conversion module, used to obtain a spatial conversion relationship between the teaching object and the actual object according to the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene;

[0059] The positioning module is used to obtain the target grasping position of the remaining objects that have not been successfully registered in the current grasping scene according to the spatial conversion relationship and the taught grasping position of the taught object.

[0060] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of positioning and grasping the robot arm in the above-mentioned embodiments are implemented.

[0061] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of positioning and grasping the robot arm in the above-mentioned embodiments are implemented.

[0062] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of positioning and grasping the robot arm in the above-mentioned embodiments are implemented.

[0063] The above-mentioned robot arm positioning grasping method, device, computer equipment, storage medium and computer program product use registration technology to obtain the grasping position of the actual object that has been successfully registered, and then calculate the spatial conversion relationship between the teaching object and the actual object. Based on the spatial conversion relationship, the information of the registered object is effectively used to compensate for the parts with poor imaging through the optimization algorithm, and the grasping position of the remaining objects that have not been successfully registered is inferred, which solves the problem of poor imaging of highly reflective parts leading to registration failure. This method significantly improves the grasping success rate and system reliability, ensures that parts can be accurately grasped even in highly reflective environments, and improves the stability and efficiency of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a process of a robot arm positioning and grasping method in one embodiment;

[0065] Figure 2 A schematic flow chart of the steps of obtaining the spatial conversion relationship between the teaching object and the actual object in one embodiment;

[0066] Figure 3 A schematic diagram of a flow chart of a robot arm positioning and grasping method in another embodiment;

[0067] Figure 4 This is a schematic diagram of the registration results;

[0068] Figure 5 It is a structural block diagram of a mechanical arm positioning and grasping device in one embodiment;

[0069] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] A robot arm positioning and grasping method is applied to a robot arm grasping system. The robot arm grasping system comprises three 3D cameras, two 6-axis robot arms, a suction cup gripper and an industrial computer. Two first cameras are respectively fixed on the ends of two robot arms for imaging parts, and the relationship between the robot arms and the suction cup is like eyes on hands. The second camera is placed on the ground and is also fixed on the robot arm flange for grasping parts. The industrial computer is connected to the 3D camera and the robot arm, and is used to obtain the image data collected by the 3D camera, analyze the image data, determine the grasping position, and control the robot arm to perform the grasping action based on the grasping position.

[0072] A robot arm positioning and grasping method is applied to the above-mentioned industrial computer, such as Figure 1 As shown, the following steps are included:

[0073] Step 102, obtaining point cloud data of actual objects in the current grasping scene; there are multiple actual objects to be grasped in the current scene.

[0074] In one embodiment, the robot positioning grasping method can grasp objects in different grasping scenes. In each grasping scene, the objects are placed regularly. For example, a grasping scene is an industrial application scene, and parts are grasped. Specifically, the material frame in the industry usually contains dozens or even hundreds of regularly placed parts, and the point cloud data of the parts in the grasping scene is obtained.

[0075] Step 104 , registering the point cloud data of the actual object with the object template point cloud of the current grasping scene, to obtain a successfully registered actual object and a grasping position of the successfully registered actual object.

[0076] In some cases, in order to improve the accuracy and reliability of the robot's positioning and grasping, a teaching process may be required. Teaching refers to guiding the robot to complete a series of actions through manual operation or preset programs so that the robot can learn and remember information such as the location, shape, and grasping method of the object.

[0077] In one embodiment, the industrial computer can prepare object template point clouds for different grasping scenes in advance, and perform grasping teaching based on the object template point clouds, and store the object module point clouds in the grasping scene, as well as the corresponding teaching grasping position and other information. When the robot arm locates and grasps, it identifies the current grasping scene and obtains the object template point cloud of the current grasping scene for registration.

[0078] Specifically, the registration may include two steps: coarse registration and fine registration. By registering the point cloud data of the actual object with the object template point cloud used in the current grasping scene during the teaching process, a successfully registered actual object and a grasping position of the successfully registered actual object can be obtained.

[0079] Step 106 , obtaining a spatial conversion relationship between the teaching object and the actual object according to the successfully registered grasping position of the actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene.

[0080] Step 108, obtaining the target grasping positions of all objects in the current grasping scene according to the spatial conversion relationship and the taught grasping positions of the taught objects; the all objects include actual objects that have been successfully registered and remaining objects that have not been successfully registered.

[0081] In industrial production, the automated grasping of highly reflective parts (such as painted parts) is a challenging task. Due to the highly reflective properties of the part surface, 3D cameras are often disturbed when acquiring their three-dimensional images, resulting in poor imaging, which in turn affects the registration success rate and grasping efficiency.

[0082] Therefore, during the registration process, there may be some objects that fail to be registered due to poor imaging, and the precise grasping positions of these objects cannot be obtained due to the registration failure. For this part, the present application uses an optimization algorithm to calculate the grasping positions of the objects that fail to be registered.

[0083] Specifically, the spatial conversion relationship between the teaching object and the actual object is obtained based on the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene. Based on the spatial conversion relationship, the information of the registered object is effectively used, and the parts with poor imaging are compensated through the optimization algorithm, and the grasping positions of the remaining objects that have not been successfully registered are inferred, thereby improving the stability and accuracy of the overall grasping system.

[0084] Step 110: Based on the target grasping positions of all the objects, control the robot arm to grasp the objects.

[0085] Specifically, according to the target grasping position of each object and the pre-calibrated relationship between the camera and the end of the flange of the robotic arm, the robotic arm is controlled to perform the grasping action, and the grasping result is feedback-adjusted.

[0086] The above-mentioned robot arm positioning grasping method uses registration technology to obtain the grasping position of the actual object that has been successfully registered, and then calculates the spatial conversion relationship between the teaching object and the actual object. Based on the spatial conversion relationship, the information of the registered object is effectively used to compensate for the parts with poor imaging through the optimization algorithm, and the grasping position of the remaining objects that have not been successfully registered is inferred, which solves the problem of poor imaging of highly reflective parts leading to registration failure. This method significantly improves the grasping success rate and system reliability, ensures that parts can be accurately grasped even in highly reflective environments, and improves the stability and efficiency of the overall system.

[0087] In another embodiment, the obtaining of point cloud data of actual objects in the current grasping scene includes: collecting depth images and texture images in the current grasping scene; detecting actual objects in the texture image to obtain the number of actual objects on the texture image and the positions of the actual objects on the texture image; segmenting the depth image and texture image according to the positions of the actual objects to obtain depth maps and texture maps of each actual object; and converting the depth maps and texture maps of each actual object to obtain point cloud data of each actual object.

[0088] Specifically, after adjusting the camera parameters and obtaining the depth map and texture map, the actual objects in the texture image are detected using deep learning or contour detection methods to detect the number and position of objects in the image. The detected detection frame is cut on the depth map, leaving only the depth information within the detection frame, and finally the depth map of each part is obtained; the point cloud data of each actual object is obtained by converting the depth map and texture map.

[0089] In this embodiment, when there is a tailing, the coordinates of the parts that do not actually exist are eliminated through the target detection results to avoid misoperation and grasping failure. This method ensures the accuracy and reliability of the grasping operation, reduces the probability of misoperation, and further improves the grasping efficiency and overall production quality of the system.

[0090] In another embodiment, the spatial transformation relationship includes a rotation matrix and a translation vector.

[0091] Specifically, Figure 2 As shown, according to the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene, the spatial conversion relationship between the teaching object and the actual object is obtained, including:

[0092] Step 202: Acquire a set of actual grasping points corresponding to the grasping position of the actual object that has been successfully registered.

[0093] Step 204 : acquiring a teaching object corresponding to the actual object in the current grasping scene, and a teaching grasping point set corresponding to the teaching grasping position of the teaching object.

[0094] Specifically, the calculated grasping position and corresponding ID, as well as the taught part ID and grasping position are bound together, and the X coordinates and Y coordinates of the two sets of points are regarded as a series of corresponding plane points. Suppose the grasping point set of the taught part is {(x i ,y i )}, the currently calculated grab point set is {(x' i ,y' i )}. The two sets of grasping points are considered as planar point pairs.

[0095] Step 206: Calculate the centroid of the actual grasping point set and the teaching grasping point set.

[0096] Specifically, calculate the centroids C and C' of the two sets of grasping points:

[0097]

[0098] Step 208 , converting the actual grasping point set and the teaching grasping point set into a centroid coordinate system according to the centroid, to obtain a first coordinate set corresponding to the actual grasping point set and a second coordinate set corresponding to the teaching grasping point set.

[0099] Specifically, decentralize and transform all points to the centroid coordinate system:

[0100]

[0101] Step 210: Calculate the covariance matrix of the first coordinate set and the second coordinate set according to the first coordinate set and the second coordinate set.

[0102] Specifically, the covariance calculation formula is:

[0103]

[0104] Step 212, performing singular value decomposition on the covariance matrix, and calculating the rotation matrix between the teaching object and the actual object according to the decomposition result.

[0105] Perform SVD decomposition on the covariance matrix:

[0106] H=UΣV T

[0107] Compute the rotation matrix:

[0108] R=VU T

[0109] Step 214: Calculate the translation vector according to the first coordinate set, the second coordinate set and the rotation matrix.

[0110] The calculation formula of the translation vector is:

[0111] t=C'-RC

[0112] In this embodiment, a gripping method based on registration technology is introduced, and the gripping position of the parts that failed to be registered is calculated through an optimization algorithm. Specifically, by binding the feature points of the teaching part with the part whose gripping position has been calculated, and calculating the rotation and translation transformation of the two sets of feature points, the gripping position of all parts can be derived, and accurate gripping can be achieved even in poor imaging conditions.

[0113] In another embodiment, the teaching grasping position of the teaching object includes the abscissa and ordinate of the teaching grasping position; and the target grasping position includes the abscissa, ordinate and depth value of the target grasping position.

[0114] The calculation formula for the target grasping position of all objects in the current grasping scene is:

[0115]

[0116] Among them, x i and i are the horizontal and vertical coordinates of the teaching grasping position, R is the rotation matrix, Z avg is the average depth value of the actual object that has been successfully registered, and t is the translation vector.

[0117] In this embodiment, the average depth Z of the registration result part is avg As the Z value of the derived part, the grasping coordinates (X, Y, Z) of all parts are finally obtained. This method derives the grasping position of all parts by binding the teaching part with the feature points of the registered part and calculating the rotation and translation matrix. At the same time, the Z value is derived by the average depth of the registration result part to reduce repeated calculations. This method improves the computing efficiency and resource utilization of the system, quickly calculates the grasping position and derives the Z value, making the system respond faster when processing large-scale data, and improving production efficiency.

[0118] In another embodiment, the point cloud data of the actual object is registered with the object template point cloud of the current grasping scene to obtain the actual object that has been successfully registered, and the grasping position of the actual object that has been successfully registered, including: extracting feature points of the point cloud data of the actual object and the object template point cloud of the current grasping scene; matching feature points through feature descriptors to obtain matching feature point pairs of the point cloud data of the actual object and the object template point cloud; obtaining an initial registration result of the point cloud data of the actual object and the object template point cloud based on the matching feature point pairs; optimizing the initial registration result using truncated least squares estimation to obtain a coarse registration result; performing ICP registration based on the coarse registration result to obtain the corresponding relationship between the point cloud data of the actual object and the object template point cloud, and the actual object that has been successfully registered; and calculating the grasping position of the actual object that has been successfully registered based on the corresponding relationship between the point cloud data of the actual object and the object template point cloud.

[0119] Specifically, coarse registration can solve a global optimal solution as follows: extract feature points from the two input point clouds, match the feature points through feature descriptors, and obtain the initial correspondence. Use the matched feature point pairs to generate an initial registration estimate. For the initial estimated registration result, the truncated least squares estimation method is then used to further optimize the registration result. Specifically, it minimizes the impact of outliers by solving an optimization problem with a truncated loss function. The truncated loss function will truncate the error in the case of outliers, so that the impact of outliers is limited to a smaller range. The truncated least squares estimation problem is transformed into a mixed integer programming problem, and the final registration result is obtained by solving this globally optimal integer programming problem.

[0120] Perform ICP fine matching on the data after rough registration to obtain the fine matching result, and calculate the grasping position of the corresponding part through the result.

[0121] The present invention adopts the Teaser++ algorithm, which enhances the robustness of the registration process through truncated least squares estimation and global optimization, can effectively filter noise and outliers, and ensure the accuracy of the registration results. This method improves the robustness and adaptability of the system in complex environments, and the registration results are still accurate even in the case of a large amount of noise and outliers.

[0122] In another embodiment, Figure 3 As shown, a robot arm positioning and grasping method comprises the following steps:

[0123] Step 1: Pre-calibrate the first camera and the robot arm in the eye-on-hand manner, and pre-calibrate the relationship between the second camera and the flange end of the robot arm; the first camera is used to collect the depth image and texture image of the current grasping scene; the second camera is fixed at the flange end of the robot arm to perform part grasping.

[0124] Step 2: Pre-prepare an object template point cloud, grasp the teaching object based on the object template point cloud, and record the grasping position of the teaching object.

[0125] Specifically, a frame of point cloud is collected, and a template is made for each object in the point cloud, and the corresponding grasping points of the template are taught. All grasping points of the parts under the teaching template are used as the true value for subsequent optimization derivation.

[0126] Step 3: Obtain point cloud data of actual objects in the current grasping scene, where there are multiple actual objects to be grasped in the current scene.

[0127] Specifically, a depth image and a texture image of the current capture scene are collected, and the actual objects in the texture image are detected to obtain the number of the actual objects on the texture image and the positions of the actual objects on the texture image; according to the positions of the actual objects, the depth image and the texture image are segmented to obtain the depth map and the texture map of each actual object, and according to the depth map and the texture map of each actual object, the point cloud data of each actual object is converted.

[0128] Step 4: Align the point cloud data of the actual object with the object template point cloud of the current grasping scene to obtain the successfully aligned actual object and the grasping position of the successfully aligned actual object.

[0129] Specifically, the process includes coarse registration and fine registration. Coarse registration can solve a global optimal solution, as follows: extract feature points from the two input point clouds, match the feature points through feature descriptors, and obtain the initial correspondence. Use the matched feature point pairs to generate an initial registration estimate. For the initial estimated registration result, the truncated least squares estimation method is then used to further optimize the registration result. Specifically, it minimizes the impact of outliers by solving an optimization problem with a truncated loss function. The truncated loss function will truncate the error in the case of outliers, so that the impact of outliers is limited to a smaller range. The truncated least squares estimation problem is converted into a mixed integer programming problem, and the final registration result is obtained by solving this global optimal integer programming problem. The registration result is as follows Figure 4 shown.

[0130] Perform ICP fine matching on the data after rough registration to obtain the fine matching result, and calculate the grasping position of the corresponding part through the result.

[0131] Step 5: Determine whether the number of actual objects successfully registered is greater than a preset value. If so, stop the registration and execute step 6.

[0132] In traditional industry, the material frame usually contains dozens or even hundreds of regularly placed parts. It is too inefficient to align and calculate the grasping position one by one. In this embodiment, only a preset number of objects need to be aligned, and the position of the remaining objects can be deduced by using the position law of the objects in the particle frame, which greatly reduces the number of alignments and thus reduces the time for calculating the grasping position. In one embodiment, the preset number can be 3, and the grasping position can be derived by using an optimized method. The position of all parts can be calculated through at least 3 points, which can make full use of the position law of the parts in the material frame, greatly reducing the number of alignments and thus reducing the time for calculating the grasping position.

[0133] Step 6: According to the successfully registered grasping position of the actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene, the spatial conversion relationship between the teaching object and the actual object is obtained.

[0134] By binding the calculated grasping position and corresponding ID, and the taught part ID and grasping position, the X coordinates and Y coordinates of the two sets of points are regarded as a series of corresponding plane points. Suppose the grasping point set of the taught part is {(x i ,y i )}, the currently calculated grab point set is {(x' i ,y' i )}. Consider the two sets of grasping points as plane point pairs and calculate the rotation and translation matrices by the following method:

[0135] Center of mass calculation: Calculate the center of mass C and C' of the two sets of grasping points:

[0136]

[0137] Decentralization: Convert all points to the centroid coordinate system:

[0138]

[0139] Compute the covariance matrix:

[0140]

[0141] Singular value decomposition: SVD decomposition of the covariance matrix:

[0142] H=UΣV T

[0143] Compute the rotation matrix:

[0144] R=VU T

[0145] Calculate the translation vector:

[0146] t=C'-RC

[0147] Step 7: According to the spatial transformation relationship and the teaching grasping position of the teaching object, the target grasping position of all objects in the current grasping scene is obtained.

[0148] Using the calculated rotation matrix R and translation vector t, the grasping positions of all parts are derived and calculated.

[0149] Final grab coordinates:

[0150]

[0151] For the tailings, the coordinates of the parts that do not actually exist are removed based on the target detection results. The average depth Z of the parts is obtained by the registration result. avg The Z value of the deduced part is used to finally obtain the grabbing coordinates (X, Y, Z) of all parts.

[0152] Step 8: Based on the target grasping positions of all the objects, control the robotic arm to grasp the objects.

[0153] The present invention can quickly derive the grasping positions of all parts by binding the teaching parts with the feature points of the registered parts and using the optimization algorithm to calculate the rotation and translation matrices. In addition, the Z value is derived by the average depth of the registration result parts, which reduces the waste of resources caused by repeated calculations.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present application also provides a mechanical arm positioning and grasping device for implementing the mechanical arm positioning and grasping method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more mechanical arm positioning and grasping device embodiments provided below can refer to the limitations of the mechanical arm positioning and grasping method above, and will not be repeated here.

[0156] In one embodiment, Figure 5 As shown, a mechanical arm positioning and grasping device is provided, comprising:

[0157] The acquisition module 502 is used to acquire point cloud data of actual objects in the current grasping scene; there are multiple actual objects to be grasped in the current scene.

[0158] The registration module 504 is used to register the point cloud data of the actual object with the object template point cloud of the current grasping scene to obtain the successfully registered actual object and the grasping position of the successfully registered actual object.

[0159] The conversion module 506 is used to obtain a spatial conversion relationship between the teaching object and the actual object according to the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene.

[0160] The positioning module 508 is used to obtain the target grasping position of the remaining objects that have not been successfully registered in the current grasping scene according to the spatial conversion relationship and the taught grasping position of the taught object.

[0161] The grasping module 510 controls the robot arm to grasp the objects based on the target grasping positions of all the objects.

[0162] In another embodiment, the spatial transformation relationship includes a rotation matrix and a translation vector;

[0163] A conversion module is used to obtain the actual grasping point set corresponding to the grasping position of the actual object that has been successfully registered; obtain the teaching object corresponding to the actual object in the current grasping scene, and the teaching grasping point set corresponding to the teaching grasping position of the teaching object; calculate the center of mass of the actual grasping point set and the teaching grasping point set; according to the center of mass, the actual grasping point set and the teaching grasping point set are respectively converted to the center of mass coordinate system to obtain a first coordinate set corresponding to the actual grasping point set and a second coordinate set corresponding to the teaching grasping point set; according to the first coordinate set and the second coordinate set, calculate the covariance matrix of the two; perform singular value decomposition on the covariance matrix, and calculate the rotation matrix between the teaching object and the actual object according to the decomposition result; according to the first coordinate set, the second coordinate set and the rotation matrix, calculate the translation vector. In another embodiment, an acquisition module is used to collect a depth image and a texture image in a current capture scene; detect the actual objects in the texture image to obtain the number of the actual objects on the texture image and the positions of the actual objects on the texture image; segment the depth image and the texture image according to the positions of the actual objects to obtain a depth map and a texture map of each of the actual objects; and convert the depth map and the texture map of each of the actual objects to obtain point cloud data of each of the actual objects.

[0164] In another embodiment, the teaching grasping position of the teaching object includes the horizontal coordinate and the vertical coordinate of the teaching grasping position; the target grasping position includes the horizontal coordinate, the vertical coordinate and the depth value of the target grasping position;

[0165] The calculation formula for the target grasping position of all objects in the current grasping scene is:

[0166]

[0167] Among them, x i and i are the horizontal and vertical coordinates of the teaching grasping position, R is the rotation matrix, Z avg is the average depth value of the actual object that has been successfully registered, and t is the translation vector.

[0168] In another embodiment, a registration module is used to extract feature points of the point cloud data of the actual object and the object template point cloud of the current grasping scene; perform feature point matching through feature descriptors to obtain matching feature point pairs between the point cloud data of the actual object and the object template point cloud; obtain an initial registration result between the point cloud data of the actual object and the object template point cloud based on the matching feature point pairs; optimize the initial registration result using truncated least squares estimation to obtain a coarse registration result; perform ICP registration based on the coarse registration result to obtain the corresponding relationship between the point cloud data of the actual object and the object template point cloud, as well as the actual object that has been successfully registered; calculate the grasping position of the successfully registered actual object based on the corresponding relationship between the point cloud data of the actual object and the object template point cloud.

[0169] In another embodiment, a judgment module is further included to judge whether the number of actual objects successfully registered is greater than a preset value. When the judgment result of the judgment module is yes, the registration is stopped.

[0170] In another embodiment, a pre-processing module is also included, which is used to pre-calibrate the first camera and the robotic arm in the eye-on-hand manner, and pre-calibrate the relationship between the second camera and the flange end of the robotic arm; the first camera is used to collect the depth image and texture image of the current grasping scene; the second camera is fixed to the flange end of the robotic arm, and is used to perform part grasping; the object template point cloud is pre-made, and the teaching object is grasped based on the object template point cloud, and the grasping position of the teaching object is recorded. Each module in the above-mentioned robotic arm positioning and grasping device can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0171] In one embodiment, a computer device is provided. The computer device may be an industrial computer, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for positioning and grasping a robot arm is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0172] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of positioning and grasping the robot arm in the above embodiments when executing the computer program.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of positioning and grasping the robot arm in the above-mentioned embodiments are implemented.

[0175] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of positioning and grasping the robot arm in the above-mentioned embodiments.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot arm positioning and grasping method, characterized in that: The method comprises: Acquire point cloud data of actual objects in a current grasping scene; there are multiple actual objects to be grasped in the current scene; Registering the point cloud data of the actual object with the object template point cloud of the current grasping scene to obtain a successfully registered actual object and a grasping position of the successfully registered actual object; Obtaining a spatial conversion relationship between the teaching object and the actual object according to the successfully registered grasping position of the actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene; According to the spatial conversion relationship and the teaching grasping position of the teaching object, the target grasping position of all objects in the current grasping scene is obtained; the all objects include the actual objects that have been successfully registered and the remaining objects that have not been successfully registered; Based on the target grasping positions of all the objects, the robotic arm is controlled to grasp the objects.

2. The method according to claim 1, characterized in that The spatial transformation relationship includes a rotation matrix and a translation vector; The method of obtaining a spatial conversion relationship between the teaching object and the actual object based on the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene includes: Acquire a set of actual grasping points corresponding to the grasping position of the actual object that has been successfully registered; Acquire a teaching object corresponding to the actual object in the current grasping scene, and a teaching grasping point set corresponding to the teaching grasping position of the teaching object; Calculating the centroid of the actual grasping point set and the teaching grasping point set; According to the centroid, the actual grasping point set and the teaching grasping point set are respectively converted into the centroid coordinate system to obtain a first coordinate set corresponding to the actual grasping point set and a second coordinate set corresponding to the teaching grasping point set; Calculate the covariance matrix of the first coordinate set and the second coordinate set according to the first coordinate set and the second coordinate set; Performing singular value decomposition on the covariance matrix, and calculating the rotation matrix between the teaching object and the actual object according to the decomposition result; The translation vector is calculated according to the first coordinate set, the second coordinate set and the rotation matrix.

3. The method according to claim 1, characterized in that The step of obtaining point cloud data of an actual object in the current grasping scene includes: Collect the depth image and texture image of the current captured scene; Detecting actual objects in the texture image to obtain the number of actual objects on the texture image and the positions of the actual objects on the texture image; Segmenting the depth image and the texture image according to the positions of the actual objects to obtain a depth map and a texture map of each of the actual objects; Point cloud data of each actual object is obtained by conversion according to the depth map and the texture map of each actual object.

4. The method according to claim 2 or 3, characterized in that: The teaching grasping position of the teaching object includes the horizontal coordinate and the vertical coordinate of the teaching grasping position; the target grasping position includes the horizontal coordinate, the vertical coordinate and the depth value of the target grasping position; The calculation formula for the target grasping position of all objects in the current grasping scene is: Among them, x i and i are the horizontal and vertical coordinates of the teaching grasping position, R is the rotation matrix, Z avg is the average depth value of the actual object that has been successfully registered, and t is the translation vector.

5. The method according to claim 3, characterized in that: The step of registering the point cloud data of the actual object with the object template point cloud applied in the teaching process of the current grasping scene to obtain the successfully registered actual object and the grasping position of the successfully registered actual object includes: Extracting feature points of the point cloud data of the actual object and the object template point cloud of the current capture scene; Perform feature point matching through feature descriptors to obtain matching feature point pairs between the point cloud data of the actual object and the point cloud of the object template; Obtaining an initial registration result of the point cloud data of the actual object and the point cloud of the object template according to the matching feature point pairs; The initial registration result is optimized by using a truncated least squares estimation method to obtain a rough registration result; Perform ICP registration according to the rough registration result to obtain the corresponding relationship between the point cloud data of the actual object and the point cloud of the object template, as well as the actual object that has been successfully registered; The grasping position of the successfully registered actual object is calculated according to the corresponding relationship between the point cloud data of the actual object and the point cloud of the object template.

6. The method according to claim 1, characterized in that The method further comprises: Determine whether the number of successfully registered actual objects is greater than a preset value. If the number of successfully registered actual objects is greater than the preset value, stop the registration, and execute the step of obtaining the spatial conversion relationship between the teaching object and the actual object based on the grasping position of the successfully registered actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene.

7. The method according to claim 1, characterized in that The method further comprises: The first camera and the robot arm are pre-calibrated in the eye-on-hand manner, and the relationship between the second camera and the flange end of the robot arm is pre-calibrated; the first camera is used to collect the depth image and texture image of the current grasping scene; the second camera is fixed to the flange end of the robot arm to perform part grasping; An object template point cloud is prepared in advance, and a teaching object is grasped based on the object template point cloud, and a grasping position of the teaching object is recorded.

8. A mechanical arm positioning and grasping device, characterized in that: The device comprises: An acquisition module, used to acquire point cloud data of an actual object in a current grasping scene; there are multiple actual objects to be grasped in the current scene; A registration module, used to register the point cloud data of the actual object with the object template point cloud of the current grasping scene, to obtain a successfully registered actual object and a grasping position of the successfully registered actual object; A conversion module, used to obtain a spatial conversion relationship between the teaching object and the actual object according to the grasping position of the actual object that has been successfully registered and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene; The positioning module is used to obtain the target grasping position of the remaining objects that have not been successfully registered in the current grasping scene according to the spatial conversion relationship and the taught grasping position of the taught object.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images