Mechanical arm positioning and grabbing method and device, computer device and storage medium

Abstract

The application relates to a mechanical arm positioning and grabbing method and device, computer equipment and a storage medium. The method uses registration technology to obtain a successful registration of the actual object grabbing position, calculates the space conversion relationship between the teaching object and the actual object, effectively uses the information of the registered object, compensates for the poor imaging of the part through an optimization algorithm, and infers the grabbing position of the remaining object which fails to be registered, thereby solving the problem of registration failure caused by poor imaging of the high-reflectivity part. The method significantly improves the success rate of grabbing and the reliability of the system, ensures accurate grabbing of the part in a high-reflectivity environment, and improves the stability and efficiency of the overall system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arms, in particular to a mechanical arm positioning and grabbing method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] In the field of industrial automation, robots have been widely used, especially in the handling and assembly of parts. For part grabbing, traditional methods mainly rely on preset paths and visual guidance systems. For regular parts, 3D cameras or laser scanners and other visual devices can accurately obtain their three-dimensional information, thereby guiding the robot to perform accurate grabbing. However, when encountering parts with high light reflection surfaces (such as painted parts), the problem becomes complex. High light reflection surfaces can cause light spots and reflections in the images obtained by 3D cameras, affecting the quality and integrity of the point cloud data, and thus affecting the accuracy of the registration algorithm.

[0003] In the prior art, some processing methods for high light reflection parts have been developed. For example:

[0004] (1) Filtering technology: Various filtering algorithms such as median filtering and mean filtering are applied to try to reduce reflection interference.

[0005] (2) Light source adjustment: Adjust the angle and intensity of the light source to reduce the impact of reflected light.

[0006] (3) Multi-view imaging: Use multi-view imaging technology to take multiple angle shots and synthesize more accurate three-dimensional images.

[0007] (4) Feature extraction algorithm: Use more complex feature extraction algorithms to try to extract useful information from the disturbed point cloud data.

[0008] Although the existing technology can alleviate the impact of high light reflection on image quality to some extent, there are still many limitations in practical application:

[0009] (1) Limitations of filtering technology:

[0010] Filtering technology often causes loss of image details when dealing with high light reflection interference, affecting the accuracy of subsequent registration. The parameter setting in the filtering process is complex and difficult to automatically adjust in a dynamic environment.

[0011] (2) Limitations of light source adjustment:

[0012] Light source adjustment requires precise control of lighting conditions, but in industrial production environments, lighting conditions are often difficult to maintain stable. The light reflection characteristics of high light reflection parts vary greatly at different angles, and the light source adjustment method is difficult to adapt to changing environments.

[0013] (3) Limitations of feature extraction algorithms:

[0014] Complex feature extraction algorithms require high computational resources and are difficult to run efficiently in real-time systems. The robustness of feature extraction algorithms is poor under high light interference, leading to a high failure rate of registration.

[0015] (4) Reliability of the overall system:

[0016] In complex industrial environments, there are various interference factors such as vibration and temperature changes, which can affect the effectiveness of existing technologies.

[0017] That is, existing methods are difficult to ensure sustained high precision and high reliability in practical applications, and the success rate and efficiency of grabbing cannot meet the needs of industrial production. SUMMARY

[0018] Therefore, it is necessary to provide a mechanical arm positioning and grabbing method, device, computer equipment, computer readable storage medium and computer program product capable of improving the precision and reliability of grabbing.

[0019] In a first aspect, the present application provides a mechanical arm positioning and grabbing method. The method comprises:

[0020] Obtaining point cloud data of actual objects in the current grabbing scene; there are multiple actual objects to be grabbed in the current scene;

[0021] Registering the point cloud data of the actual objects with the object template point cloud of the current grabbing scene to obtain successfully registered actual objects and the grabbing positions of the successfully registered actual objects;

[0022] Obtaining the spatial conversion relationship between the teaching objects and the actual objects according to the grabbing positions of the successfully registered actual objects and the teaching grabbing positions of the teaching objects corresponding to the actual objects in the current grabbing scene;

[0023] Obtaining the target grabbing positions of all objects in the current grabbing scene according to the spatial conversion relationship and the teaching grabbing positions of the teaching objects; the all objects include the successfully registered actual objects and the remaining actual objects that are not successfully registered;

[0024] Controlling the mechanical arm to grab the objects based on the target grabbing positions of the all objects.

[0025] In one embodiment, the spatial conversion relationship includes a rotation matrix and a translation vector.

[0026] The spatial conversion relationship between the teaching object and the actual object is obtained according to the successful registration of the actual object and the teaching grasp position of the teaching object corresponding to the actual object in the current grasp scene, and includes:

[0027] An actual grasp point set corresponding to the successful registration of the actual object is obtained.

[0028] A teaching object corresponding to the actual object in the current grasp scene and a teaching grasp point set corresponding to the teaching grasp position of the teaching object are obtained.

[0029] Centroids of the actual grasp point set and the teaching grasp point set are calculated.

[0030] The actual grasp point set and the teaching grasp point set are respectively converted to a centroid coordinate system according to the centroids, to obtain a first coordinate set corresponding to the actual grasp point set and a second coordinate set corresponding to the teaching grasp point set.

[0031] A covariance matrix of the first coordinate set and the second coordinate set is calculated.

[0032] The covariance matrix is singular value decomposed, and a rotation matrix between the teaching object and the actual object is calculated according to a decomposition result.

[0033] The translation vector is calculated according to the first coordinate set, the second coordinate set and the rotation matrix.

[0034] In one of the embodiments, the point cloud data of the actual object in the current grasp scene is obtained, including:

[0035] A depth image and a texture image in the current grasp scene are collected.

[0036] The actual object in the texture image is detected to obtain the number of the actual objects on the texture image and the positions of the actual objects on the texture image.

[0037] The depth image and the texture image are segmented according to the positions of the actual objects to obtain the depth map and the texture map of each actual object.

[0038] The point cloud data of each actual object is converted from the depth map and the texture map of each actual object.

[0039] In one of the embodiments, the teaching grasp position of the teaching object includes the horizontal coordinate and the vertical coordinate of the teaching grasp position, and the target grasp position includes the horizontal coordinate, the vertical coordinate and the depth value of the target grasp position.

[0040] The calculation formula of the target grasping position of all objects in the current grasping scene is:

[0041]

[0042] wherein x i and y i are the horizontal coordinate and the vertical coordinate of the demonstration grasping position respectively, R is a rotation matrix, Z avg is the average depth value of the actual object for which the registration is successful, and t is a translation vector.

[0043] In an embodiment, the registration of the point cloud data of the actual object with the object template point cloud applied in the demonstration process of the current grasping scene, to obtain an actual object for which the registration is successful and a grasping position of the actual object for which the registration is successful, comprises:

[0044] extracting feature points of the point cloud data of the actual object and the object template point cloud of the current grasping scene;

[0045] performing feature point matching through a feature descriptor to obtain a matching feature point pair of the point cloud data of the actual object and the object template point cloud;

[0046] obtaining an initial registration result of the point cloud data of the actual object and the object template point cloud according to the matching feature point pair;

[0047] optimizing the initial registration result by using a truncated least squares estimation method to obtain a coarse registration result;

[0048] performing ICP registration according to the coarse registration result to obtain a corresponding relationship of the point cloud data of the actual object and the object template point cloud, and an actual object for which the registration is successful;

[0049] calculating the grasping position of the actual object for which the registration is successful according to the corresponding relationship of the point cloud data of the actual object and the object template point cloud.

[0050] In an embodiment, the method further comprises:

[0051] judging whether the number of actual objects for which the registration is successful is greater than a preset value, and if the number of actual objects for which the registration is successful is greater than the preset value, stopping the registration and performing the step of obtaining the spatial conversion relationship between the demonstration object and the actual object according to the grasping position of the actual object for which the registration is successful and the demonstration grasping position of the demonstration object corresponding to the actual object in the current grasping scene.

[0052] In an embodiment, the method further comprises:

[0053] The first camera and the robot arm are calibrated in eye-in-hand manner in advance, and the second camera and the flange end of the robot arm are calibrated in advance; the first camera is used to collect a depth image and a texture image of a current grasping scene; the second camera is fixed on the flange end of the robot arm and is used to perform grasping of a part;

[0054] An object template point cloud is prepared in advance, and a demonstration object is grasped based on the object template point cloud, and a grasping position of the demonstration object is recorded.

[0055] In a second aspect, the present application further provides a robot arm positioning and grasping device. The device comprises:

[0056] An acquisition module is configured to acquire point cloud data of actual objects in a current grasping scene; the current scene comprises a plurality of actual objects to be grasped;

[0057] A registration module is configured to register the point cloud data of the actual objects with an object template point cloud of the current grasping scene, to obtain actual objects with successful registration and grasping positions of the actual objects with successful registration;

[0058] A conversion module is configured to obtain a spatial conversion relationship between a demonstration object and an actual object according to the grasping positions of the actual objects with successful registration and demonstration grasping positions of demonstration objects corresponding to the actual objects in the current grasping scene;

[0059] A positioning module is configured to obtain target grasping positions of remaining objects with unsuccessful registration in the current grasping scene according to the spatial conversion relationship and the demonstration grasping positions of the demonstration objects.

[0060] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the robot arm positioning and grasping of each of the above embodiments when executing the computer program.

[0061] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the robot arm positioning and grasping of each of the above embodiments when executed by a processor.

[0062] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program implements the steps of the robot arm positioning and grasping of each of the above embodiments when executed by a processor.

[0063] The mechanical arm positioning and grabbing method, device, computer equipment, storage medium and computer program product use registration technology to obtain a successful registration of the actual object, calculate the spatial conversion relationship between the teaching object and the actual object, effectively use the information of the registered object, compensate for the poor imaging of the parts through an optimization algorithm, and predict the grabbing position of the remaining objects that are not successfully registered, thereby solving the problem of registration failure caused by poor imaging of high-reflectivity parts. The method significantly improves the success rate of grabbing and the reliability of the system, ensures accurate grabbing of parts in a high-reflectivity environment, and improves the stability and efficiency of the overall system. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A flowchart of a mechanical arm positioning and grabbing method in an embodiment;

[0065] Figure 2 A flowchart of a step of obtaining a spatial conversion relationship between a teaching object and an actual object in an embodiment;

[0066] Figure 3 A flowchart of a mechanical arm positioning and grabbing method in another embodiment;

[0067] Figure 4 A registration result schematic diagram;

[0068] Figure 5 A structural block diagram of a mechanical arm positioning and grabbing device in an embodiment;

[0069] Figure 6 An internal structure diagram of a computer equipment in an embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0071] A mechanical arm positioning and grabbing method is applied to a mechanical arm grabbing system. The mechanical arm grabbing system includes three 3D cameras, two 6-axis mechanical arms, a suction cup gripper and an industrial computer. Two first cameras are respectively fixed on the ends of the two mechanical arms, used for part imaging, and have an eye-in-hand relationship with the mechanical arm. A second camera is placed on the ground and a suction cup is also fixed on the flange of the mechanical arm, used for performing part grabbing. The industrial computer is connected with the 3D camera and the mechanical arm, used for obtaining image data collected by the 3D camera, analyzing the image data, determining the grabbing position, and controlling the mechanical arm to perform a grabbing action based on the grabbing position.

[0072] A mechanical arm positioning and grabbing method is applied to the industrial computer as described above, which comprises the following steps: Figure 1 As shown in the figure, the steps include:

[0073] Step 102, obtaining the point cloud data of the actual object in the current grabbing scene; there are multiple actual objects to be grabbed in the current scene.

[0074] In one embodiment, the mechanical arm positioning and grabbing method can grab objects in different grabbing scenes. In each grabbing scene, the objects are regularly placed. For example, one grabbing scene is an industrial application scene for grabbing parts. Specifically, the industrial material frame usually contains dozens or even hundreds of regularly placed parts, and the point cloud data of the parts in the grabbing scene is obtained.

[0075] Step 104, registering the point cloud data of the actual object with the object template point cloud of the current grabbing scene to obtain the actual object with successful registration and the grabbing position of the actual object with successful registration.

[0076] In some cases, in order to improve the accuracy and reliability of the mechanical arm positioning and grabbing, a teaching process may be needed. Teaching refers to guiding the mechanical arm to complete a series of actions through manual operation or preset program, so that the mechanical arm can learn and remember the position, shape and grabbing method of the object and other information.

[0077] In one embodiment, the industrial computer can pre-produce object template point clouds for different grabbing scenes, and perform grabbing teaching based on the object template point clouds, store the object module point cloud and the corresponding teaching grabbing position information in the grabbing scene. When the mechanical arm is positioned and grabbed, the current grabbing scene is identified, and the object template point cloud of the current grabbing scene is obtained for registration.

[0078] Specifically, the registration can include two steps of coarse registration and fine registration. By registering the point cloud data of the actual object with the object template point cloud applied in the teaching process of the current grabbing scene, the actual object with successful registration and the grabbing position of the actual object with successful registration can be obtained.

[0079] Step 106, obtaining the spatial conversion relationship between the teaching object and the actual object according to the grabbing position of the actual object with successful registration and the teaching grabbing position of the corresponding teaching object of the actual object in the current grabbing scene.

[0080] Step 108, obtaining the target grabbing position of all objects in the current grabbing scene according to the spatial conversion relationship and the teaching grabbing position of the teaching object; the all objects include the actual object with successful registration and the remaining actual object with unsuccessful registration.

[0081] In industrial production, the automated picking of high-reflective parts (such as painted parts) is a challenging task. Due to the high-reflective characteristics of the part surface, the 3D camera is often disturbed when acquiring its three-dimensional image, resulting in poor imaging effect, which in turn affects the registration success rate and picking efficiency.

[0082] Therefore, there may be a part of the imaging of the object that fails to register, and the accurate picking position of the object cannot be obtained due to the registration failure. For this part, the picking position of the registration-failed object is calculated by an optimization algorithm in the present application.

[0083] Specifically, the spatial conversion relationship between the teaching object and the actual object is obtained according to the picking position of the actual object that successfully registers and the teaching picking position of the teaching object corresponding to the actual object in the current picking scene. Based on the spatial conversion relationship, the information of the registered object is effectively utilized, and the picking position of the remaining object that fails to register is inferred by an optimization algorithm to compensate for the poor imaging of the part, thereby improving the stability and accuracy of the overall picking system.

[0084] Step 110, based on the target picking position of the entire object, controlling the robot arm to pick the object.

[0085] Specifically, according to the target picking position of each object and the relationship between the camera and the flange end of the robot arm that is calibrated in advance, the robot arm is controlled to perform picking action, and the picking result is adjusted.

[0086] The above-mentioned robot positioning and picking method utilizes registration technology to obtain the picking position of the actual object that successfully registers, and then calculates the spatial conversion relationship between the teaching object and the actual object. Based on the spatial conversion relationship, the information of the registered object is effectively utilized, and the picking position of the remaining object that fails to register is inferred by an optimization algorithm to compensate for the poor imaging of the part, thereby solving the problem of registration failure caused by poor imaging of high-reflective parts. This method significantly improves the picking success rate and the reliability of the system, ensures accurate picking of parts in a high-reflective environment, and improves the stability and efficiency of the overall system.

[0087] In another embodiment, the acquisition of the point cloud data of the actual object in the current picking scene comprises: collecting a depth image and a texture image in the current picking scene; detecting the actual object in the texture image to obtain the number of actual objects on the texture image and the position of the actual objects on the texture image; segmenting the depth image and the texture image according to the position of the actual objects to obtain the depth map and the texture map of each actual object; and converting the depth map and the texture map of each actual object to obtain the point cloud data of each actual object.

[0088] Specifically, after adjusting the camera parameters and obtaining the depth map and the texture map, a deep learning or contour detection method is used to detect the actual objects in the texture image, detect the number of objects in the image and the position of the objects in the image. The detected bounding boxes are cut on the depth map, leaving only the depth information within the bounding boxes, and finally the depth map of each part is obtained; the point cloud data of each actual object is obtained by converting the depth map and the texture map.

[0089] In this embodiment, when encountering tailing material, the coordinates of the parts that do not actually exist are removed through target detection results to avoid misoperation and failure of grabbing. This method ensures the accuracy and reliability of the grabbing operation, reduces the probability of misoperation, and further improves the grabbing efficiency and overall production quality of the system.

[0090] In another embodiment, the spatial conversion relationship includes a rotation matrix and a translation vector.

[0091] Specifically, as shown in Figure 2 According to the grabbing position of the actual object of the successful registration and the teaching grabbing position of the teaching object corresponding to the actual object in the current grabbing scene, the spatial conversion relationship between the teaching object and the actual object is obtained, including:

[0092] Step 202, obtaining an actual grabbing point set corresponding to the grabbing position of the actual object of the successful registration.

[0093] Step 204, obtaining a teaching object corresponding to the actual object in the current grabbing scene, and a teaching grabbing point set corresponding to the teaching grabbing position of the teaching object.

[0094] Specifically, by binding the calculated grabbing position and the corresponding ID, and the teaching part ID and the grabbing position, the X coordinates and Y coordinates of the two sets of points are regarded as a series of corresponding plane points. Let the grabbing point set of the teaching part be {(x i ,y i )}, and the currently calculated grabbing point set be {(x' i ,y' i )}. The two sets of grabbing points are regarded as plane point pairs.

[0095] Step 206, calculating the centroids of the actual grabbing point set and the teaching grabbing point set.

[0096] Specifically, the centroids C and C' of the two sets of grabbing points are calculated:

[0097]

[0098] Step 208, according to the center of mass, the actual set of grasp points, and the set of demonstration grasp points are respectively converted to the center of mass coordinate system, to obtain the first coordinate set corresponding to the actual set of grasp points, and the second coordinate set corresponding to the set of demonstration grasp points.

[0099] Specifically, decentralization, all points are converted to the center of mass coordinate system:

[0100]

[0101] Step 210, according to the first coordinate set and the second coordinate set, the covariance matrix of the two is calculated.

[0102] Specifically, the covariance calculation formula is:

[0103]

[0104] Step 212, singular value decomposition is performed on the covariance matrix, and a rotation matrix between the demonstration object and the actual object is calculated according to the decomposition result.

[0105] SVD decomposition is performed on the covariance matrix:

[0106] H = UΣV T

[0107] The rotation matrix is calculated:

[0108] R = VU T

[0109] Step 214, according to the first coordinate set, the second coordinate set and the rotation matrix, the translation vector is calculated.

[0110] Wherein, the calculation formula of the translation vector is:

[0111] t = C'-RC

[0112] In this embodiment, a grasping method based on registration technology is introduced, and an optimization algorithm is used to calculate the grasping position of the registration failed part. Specifically, by binding the feature points of the demonstration part and the parts whose grasping positions have been calculated, and calculating the rotation and translation transformation of the two sets of feature points, the grasping positions of all parts can be derived, and accurate grasping can be achieved even in poor imaging conditions.

[0113] In another embodiment, the demonstration grasp position of the demonstration object includes the horizontal coordinate and the vertical coordinate of the demonstration grasp position; and the target grasp position includes the horizontal coordinate, the vertical coordinate and the depth value of the target grasp position.

[0114] The calculation formula of the target grasp position of all objects in the current grasping scene is:

[0115]

[0116] wherein x i and y i are the horizontal and vertical coordinates of the teaching grasping position, R is a rotation matrix, Z avg is the average depth value of the successfully registered actual object, and t is a translation vector.

[0117] In this embodiment, the average depth Z avg of the registered part is used to derive the Z value of the part, and finally the grasping coordinates (X, Y, Z) of all parts are obtained. This method binds the feature points of the teaching part and the registered part, and calculates the rotation and translation matrix to derive the grasping position of all parts. At the same time, the average depth of the registered part is used to derive the Z value, reducing repeated calculation. This method improves the calculation efficiency and resource utilization of the system, quickly calculates the grasping position and derives the Z value, so that the system has shorter response time when processing large-scale data, and improves the production efficiency.

[0118] In another embodiment, the registration of the point cloud data of the actual object with the object template point cloud of the current grasping scene to obtain a successfully registered actual object and the grasping position of the successfully registered actual object comprises: extracting feature points of the point cloud data of the actual object and the object template point cloud of the current grasping scene; performing feature point matching through a feature descriptor to obtain a matching feature point pair of the point cloud data of the actual object and the object template point cloud; obtaining an initial registration result of the point cloud data of the actual object and the object template point cloud according to the matching feature point pair; optimizing the initial registration result by using a truncated least squares estimation method to obtain a coarse registration result; performing ICP registration according to the coarse registration result to obtain a correspondence between the point cloud data of the actual object and the object template point cloud, and a successfully registered actual object; and calculating the grasping position of the successfully registered actual object according to the correspondence between the point cloud data of the actual object and the object template point cloud.

[0119] Specifically, the coarse registration can solve a global optimal solution, specifically as follows: feature points are extracted from the input two point clouds, feature point matching is performed through feature descriptors, and an initial correspondence relationship is obtained. An initial registration estimate is generated using the matched feature point pairs. For the initial registration estimate, a truncated least squares estimation method is then used to further optimize the registration result. Specifically, it reduces the influence of outliers to the greatest extent by solving an optimization problem with a truncated loss function. The truncated loss function truncates the error in the case of outliers, so that the influence of outliers is limited within a small range. The truncated least squares estimation problem is converted into a mixed integer programming problem, and the final registration result is obtained by solving this globally optimal integer programming problem.

[0120] ICP fine matching is performed on the data after coarse registration to obtain a fine matching result, and the grabbing position of the corresponding part is calculated through the result.

[0121] The present application adopts the Teaser++ algorithm, enhances the robustness of the registration process through truncated least squares estimation and global optimization, can effectively filter noise and outliers, and ensures the accuracy of the registration result. This method improves the robustness and adaptability of the system in complex environments, and even in the presence of a large amount of noise and outliers, the registration result is still accurate.

[0122] In another embodiment, as shown in Figure 3 A mechanical arm positioning and grabbing method, comprising the following steps:

[0123] Step 1: Pre-calibrate the first camera and the mechanical arm on the hand, and pre-calibrate the relationship between the second camera and the flange end of the mechanical arm; the first camera is used to collect the depth image and the texture image of the current grabbing scene; the second camera is fixed at the flange end of the mechanical arm and is used to perform part grabbing.

[0124] Step 2: Pre-make an object template point cloud, and grab a teaching object based on the object template point cloud, and record the grabbing position of the teaching object.

[0125] Specifically, a frame of point cloud is collected, and a template is made for each object in the point cloud, and the grabbing point corresponding to the template is taught. All part grabbing points under the teaching template are taught, and the teaching value is used as the true value for later optimization derivation.

[0126] Step 3: Obtain the point cloud data of the actual objects in the current grabbing scene, and there are multiple actual objects to be grabbed in the current scene.

[0127] Specifically, a depth image and a texture image in a current grasping scene are collected, actual objects in the texture image are detected to obtain a number of the actual objects on the texture image and positions of the actual objects on the texture image, the depth image and the texture image are segmented according to the positions of the actual objects to obtain a depth map and a texture map of each of the actual objects, and point cloud data of each of the actual objects is converted according to the depth map and the texture map of each of the actual objects.

[0128] Step 4: The point cloud data of the actual objects are registered with object template point clouds of the current grasping scene to obtain actual objects with successful registration and grasping positions of the actual objects with successful registration.

[0129] Specifically, the process includes coarse registration and fine registration. The coarse registration can solve a global optimal solution, and specifically as follows: feature points are extracted from the two input point clouds, feature point matching is performed through feature descriptors to obtain an initial correspondence. An initial registration estimate is generated using the matched feature points. For the initial registration result of the estimate, a truncated least squares estimation method is then used to further optimize the registration result. Specifically, it solves an optimization problem with a truncated loss function to minimize the influence of outliers. The truncated loss function truncates the error in the case of outliers, so that the influence of outliers is limited within a small range. The truncated least squares estimation problem is converted into a mixed integer programming problem, and the final registration result is obtained by solving this globally optimal integer programming problem. The registration result is shown in FIG. 8. Figure 4

[0130] ICP fine matching is performed on the data after coarse registration to obtain a fine matching result, and a grasping position of a corresponding part is calculated through the result.

[0131] Step 5: It is judged whether the number of actual objects with successful registration is greater than a preset value. If yes, the registration is stopped, and step 6 is performed.

[0132] In the traditional industry, a frame usually contains dozens or even hundreds of regularly placed parts. It is too low in efficiency to perform registration and calculate the grasping position one by one. In the embodiment, only a preset number of objects need to be registered, and the positions of the remaining objects can be derived by using the position regularity of the objects in the frame, so that the number of registrations and the time for calculating the grasping position are greatly reduced. In an embodiment, the preset number can be 3, and the grasping position can be calculated through the positions of the 3 points to obtain the positions of all the parts, so that the position regularity of the parts in the frame can be fully utilized, and the number of registrations and the time for calculating the grasping position are greatly reduced.

[0133] ​Step 6: According to the registered successful grasping position of the actual object and the teaching grasping position of the corresponding teaching object in the current grasping scene, the spatial conversion relationship between the teaching object and the actual object is obtained.

[0134] By the grasping position and the corresponding ID that have been calculated, and the binding of the teaching part ID and the grasping position, the X coordinates and the Y coordinates of the two groups of points are regarded as a series of corresponding plane points. Let the grasping point set of the teaching part be {(x i ,y i )}, and the currently calculated grasping point set be {(x' i ,y' i )}. The two groups of grasping points are regarded as plane point pairs, and the rotation and translation matrices are calculated by the following method:

[0135] Centroid calculation: the centroids C and C' of the two groups of grasping points are calculated:

[0136]

[0137] Decentering: all points are converted to the centroid coordinate system:

[0138]

[0139] Covariance matrix calculation:

[0140]

[0141] Singular value decomposition: SVD decomposition is performed on the covariance matrix:

[0142] H = UΣV T

[0143] Rotation matrix calculation:

[0144] R = VU T

[0145] Translation vector calculation:

[0146] t = C' - RC

[0147] Step 7: According to the spatial conversion relationship and the teaching grasping position of the teaching object, the target grasping position of all objects in the current grasping scene is obtained.

[0148] Using the calculated rotation matrix R and translation vector t, the grasping positions of all parts are derived and calculated.

[0149] Final grasping coordinates:

[0150]

[0151] For the tailing case, the coordinates of the parts that do not actually exist are removed according to the target detection result. The average depth Z of the parts through the registration result avg As the Z value of the derived parts, the final coordinates (X, Y, Z) of the parts are obtained.

[0152] Step 8: Based on the target grabbing position of the object, the mechanical arm is controlled to grab the object.

[0153] The present application can quickly derive the grabbing position of all parts by binding the feature points of the parts to the registered parts and using an optimization algorithm to calculate the rotation and translation matrix. In addition, the Z value is derived by the average depth of the registered parts, reducing the waste of resources caused by repeated calculation.

[0154] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0155] Based on the same inventive concept, the present application also provides a mechanical arm positioning and grabbing device for implementing the above-mentioned mechanical arm positioning and grabbing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more mechanical arm positioning and grabbing device embodiments provided below can refer to the limitations of the mechanical arm positioning and grabbing method in the above text, which will not be repeated here.

[0156] In one embodiment, as Figure 5 shown, a mechanical arm positioning and grabbing device is provided, comprising:

[0157] The acquisition module 502 is configured to acquire point cloud data of actual objects in a current grabbing scene; and the current scene includes a plurality of actual objects to be grabbed.

[0158] The registration module 504 is configured to register the point cloud data of the actual objects with the object template point cloud of the current grabbing scene to obtain actual objects with successful registration and the grabbing position of the actual objects with successful registration.

[0159] The conversion module 506 is configured to obtain a spatial conversion relationship between the teaching object and the actual object according to the grasping position of the actual object in the registration success and the grasping position of the teaching object corresponding to the actual object in the current grasping scene.

[0160] The positioning module 508 is configured to obtain a target grasping position of a remaining object in the current grasping scene according to the spatial conversion relationship and the grasping position of the teaching object.

[0161] The grasping module 510 is configured to control the robot arm to grasp the objects based on the target grasping positions of all the objects.

[0162] In another embodiment, the spatial conversion relationship includes a rotation matrix and a translation vector.

[0163] The conversion module is configured to obtain an actual grasping point set corresponding to the grasping position of the actual object in the registration success; obtain a teaching object corresponding to the actual object in the current grasping scene and a teaching grasping point set corresponding to the grasping position of the teaching object; calculate the centroids of the actual grasping point set and the teaching grasping point set; convert the actual grasping point set and the teaching grasping point set to a centroid coordinate system respectively according to the centroids to obtain a first coordinate set corresponding to the actual grasping point set and a second coordinate set corresponding to the teaching grasping point set; calculate a covariance matrix of the first coordinate set and the second coordinate set according to the first coordinate set and the second coordinate set; perform singular value decomposition on the covariance matrix, and calculate a rotation matrix between the teaching object and the actual object according to the decomposition result; and calculate the translation vector according to the first coordinate set, the second coordinate set and the rotation matrix. In another embodiment, the acquisition module is configured to acquire a depth image and a texture image in the current grasping scene; detect the actual objects in the texture image to obtain the number of the actual objects on the texture image and the positions of the actual objects on the texture image; segment the depth image and the texture image according to the positions of the actual objects to obtain a depth image and a texture image of each actual object; and convert the depth image and the texture image of each actual object to obtain point cloud data of each actual object.

[0164] In another embodiment, the grasping position of the teaching object includes a horizontal coordinate and a vertical coordinate of the grasping position of the teaching object; and the target grasping position includes a horizontal coordinate, a vertical coordinate and a depth value of the target grasping position.

[0165] The formula for calculating the target grasping positions of all the objects in the current grasping scene is:

[0166]

[0167] wherein, x i and y i are the horizontal and vertical coordinates of the demonstration grasping position, R is a rotation matrix, Z avg is the average depth value of the actual object for which the registration is successful, and t is a translation vector.

[0168] In another embodiment, the registration module is configured to extract feature points of the point cloud data of the actual object and the object template point cloud of the current grasping scene, perform feature point matching through a feature descriptor to obtain a matching feature point pair of the point cloud data of the actual object and the object template point cloud, obtain an initial registration result of the point cloud data of the actual object and the object template point cloud according to the matching feature point pair, optimize the initial registration result by using a truncated least squares estimation method to obtain a coarse registration result, perform ICP registration according to the coarse registration result to obtain a correspondence relationship between the point cloud data of the actual object and the object template point cloud and an actual object for which the registration is successful, and calculate a grasping position of the actual object for which the registration is successful according to the correspondence relationship between the point cloud data of the actual object and the object template point cloud.

[0169] In another embodiment, the judging module is further configured to judge whether the number of actual objects for which the registration is successful is greater than a preset value. When the judging result of the judging module is yes, the registration is stopped.

[0170] In another embodiment, the pre-processing module is further configured to pre-calibrate the first camera and the robot arm in eye-in-hand and pre-calibrate the relationship between the second camera and the flange end of the robot arm, the first camera is configured to collect a depth image and a texture image of the current grasping scene, the second camera is fixed at the flange end of the robot arm and is configured to perform grasping of a part, the object template point cloud is pre-made, and the demonstration object is grasped based on the object template point cloud, and the grasping position of the demonstration object is recorded. Each module in the robot arm positioning and grasping device can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0171] In one embodiment, a computer device is provided, which can be an industrial computer, and an internal structure diagram of the computer device can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. Wireless mode can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a robot positioning and grabbing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0172] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0173] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the robot positioning and grabbing in each of the above embodiments.

[0174] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the robot positioning and grabbing in each of the above embodiments.

[0175] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to implement the steps of the robot positioning and grabbing in each of the above embodiments.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0177] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0178] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A robotic arm positioning and grasping method, characterized in that, The method includes: Acquire point cloud data of the actual objects in the current grasping scene; there are multiple actual objects to be grasped in the current grasping scene; The point cloud data of the actual object is registered with the point cloud of the object template in the current capture scene to obtain the successfully registered actual object and the capture position of the successfully registered actual object. Based on the successfully registered grasping position of the actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scenario, the spatial transformation relationship between the teaching object and the actual object is obtained, including the rotation matrix and translation vector; the steps for obtaining the spatial transformation relationship between the teaching object and the actual object include: Obtain the set of actual gripping points corresponding to the gripping position of the successfully registered actual object; Obtain the teaching object corresponding to the actual object in the current grasping scenario, and the set of teaching grasping points corresponding to the teaching grasping position of the teaching object; Calculate the centroids of the actual grasp point set and the taught grasp point set; Based on the centroid, the actual grasping point set and the teaching grasping point set are respectively transformed to the centroid coordinate system to obtain the first coordinate set corresponding to the actual grasping point set and the second coordinate set corresponding to the teaching grasping point set. Calculate the covariance matrix of the first coordinate set and the second coordinate set; Singular value decomposition is performed on the covariance matrix, and the rotation matrix between the teaching object and the actual object is calculated based on the decomposition results. The translation vector is calculated based on the first coordinate set, the second coordinate set, and the rotation matrix; Based on the spatial transformation relationship and the teaching grasping position of the teaching object, the target grasping position of all objects in the current grasping scene is obtained; the all objects include the successfully registered actual objects and the remaining unregistered actual objects. Based on the target grasping positions of all the objects, the robotic arm is controlled to grasp the objects.

2. The method according to claim 1, characterized in that, The acquisition of point cloud data of the actual objects in the current grasping scene includes: Collect depth and texture images of the current capture scene; The actual objects in the texture image are detected to obtain the number of actual objects in the texture image and the position of the actual objects in the texture image; Based on the position of the actual object, the depth image and texture image are segmented to obtain the depth map and texture map of each actual object; Point cloud data of each actual object is obtained by converting the depth map and texture map of each actual object.

3. The method according to claim 2, characterized in that, The teaching object's grasping position includes the x-coordinate and y-coordinate of the teaching grasping position; the target grasping position includes the x-coordinate, y-coordinate, and depth value of the target grasping position; The formula for calculating the target grasping position of all objects in the current grasping scenario is: in, and These are the x and y coordinates of the teaching grasping position, respectively. Let be a rotation matrix. The average depth value of the successfully registered actual object. It is a translation vector.

4. The method according to claim 3, characterized in that, The step of registering the point cloud data of the actual object with the point cloud of the object template used in the teaching process of the current grasping scene to obtain the successfully registered actual object and the grasping position of the successfully registered actual object includes: Extract the point cloud data of the actual object and the feature points of the point cloud of the object template in the current capture scene. Feature point matching is performed using feature descriptors to obtain matching feature point pairs between the point cloud data of the actual object and the point cloud of the object template. Based on the matching feature point pairs, the initial registration result of the point cloud data of the actual object and the point cloud of the object template is obtained; The initial registration result is optimized using the truncated least squares estimation method to obtain the coarse registration result. Based on the coarse registration result, ICP registration is performed to obtain the correspondence between the point cloud data of the actual object and the point cloud of the object template, as well as the actual object that has been successfully registered. Based on the correspondence between the point cloud data of the actual object and the point cloud of the object template, the grasping position of the successfully registered actual object is calculated.

5. The method according to claim 1, characterized in that, The method further includes: Determine whether the number of successfully registered actual objects is greater than a preset value. If the number of successfully registered actual objects is greater than the preset value, stop the registration and execute the step of obtaining the spatial transformation relationship between the teaching object and the actual object based on the grasping position of the successfully registered actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scene.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The first camera and the robotic arm are pre-calibrated to be in the eye-to-hand position, and the relationship between the second camera and the flange end of the robotic arm is pre-calibrated. The first camera is used to acquire depth and texture images of the current grasping scene. The second camera is fixed to the flange end of the robotic arm for grasping parts. A point cloud template of an object is pre-made, and the teaching object is grasped based on the point cloud template, and the grasping position of the teaching object is recorded.

7. A robotic arm positioning and gripping device, characterized in that, The apparatus for implementing the method according to any one of claims 1 to 6 comprises: The acquisition module is used to acquire point cloud data of actual objects in the current grasping scene; there are multiple actual objects to be grasped in the current grasping scene. The registration module is used to register the point cloud data of the actual object with the point cloud of the object template in the current capture scene to obtain the successfully registered actual object and the capture position of the successfully registered actual object. The conversion module is used to obtain the spatial conversion relationship between the teaching object and the actual object based on the grasping position of the successfully registered actual object and the teaching grasping position of the teaching object corresponding to the actual object in the current grasping scenario. The positioning module is used to obtain the target grasping position of the remaining unregistered objects in the current grasping scenario based on the spatial transformation relationship and the teaching grasping position of the teaching object.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images