3D visual guidance mechanical arm grabbing method for irregular objects

Abstract

The invention belongs to the technical field of mechanical arm automatic grabbing, and particularly relates to a 3D visual guidance mechanical arm grabbing method for irregular objects. Comprising the following steps that (1) a 3D visual guidance mechanical arm grabbing system is built, specifically, a 3D camera based on line structured light is adopted to guide a mechanical arm, the mechanical arm is a visual flexible six-axis mechanical arm, and after the mechanical arm completes accurate grabbing of all objects and carries out placement according to requirements, it is regarded that the grabbing task is successfully completed; and the camera in the 3D camera is calibrated, and the calibration of the camera is realized by calling a CalibrateCamera function in the OpenCV library. Through mutual cooperation of the 3D camera and the mechanical arm, intelligent grabbing of irregular objects such as industrial parts is achieved, more accurate grabbing operation is carried out according to the actual application situation, and the robot does not depend on training of three-dimensional point cloud samples. And the 3D camera can accurately obtain external information, and the grabbing requirement of the mechanical arm is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic grasping by a robot arm, and in particular relates to a 3D vision-guided grasping method for an irregular object by a robot arm. Background Art

[0002] With the increase in labor costs and the diversification of product demand, the widespread application of robots in various fields has become more and more significant. In particular, multi-robot production lines with industrial robots as the core have become the only way for the current manufacturing industry to transform and upgrade. In this process, the research and application of 3D vision technology has become one of the research focuses at home and abroad. In robotic automated assembly, accurate object recognition and precise three-dimensional spatial positioning are key prerequisites for the robot to perform operations such as grasping. Due to the uncertainty of the object's position and posture, as well as possible occlusion and overlap, the grasping operation of the robot arm has become more and more complicated. Therefore, it is imperative to conduct in-depth research and improvement on the grasping system to ensure that the robot arm can perform various operations accurately and effectively.

[0003] At present, the traditional robotic arm automatic grasping system mainly relies on two-dimensional image training and position coordinate point matching, resulting in low object posture accuracy and automation and intelligence levels. Compared with images that only obtain two-dimensional information of objects, three-dimensional point clouds can directly obtain the three-dimensional geometric shape of the object surface. At the same time, point cloud acquisition is a mapping process from three-dimensional space to three-dimensional space, which mainly involves rotation and translation transformations, avoiding the influence of projection transformations in two-dimensional images, thereby greatly reducing the interference of imaging viewpoint changes on imaging results, and can provide richer information for model reconstruction and target recognition tasks. In view of this, a grasping position vector map based on the position coordinates of the robotic arm and the object has emerged. When the robotic arm receives the vector map, it can analyze a set of grasping path data between the robotic arm and the object, and guide the robotic arm to the target position of the object. However, due to the lack of three-dimensional information of the object (such as depth information and geometric shape), misgrasping and low precision may occur during grasping. Traditional sorting methods lack external perception capabilities, while 3D vision can fully obtain environmental information. Combining robots to form a 3D vision-based robotic arm grasping has become a mainstream solution for irregular objects.

[0004] The patent number CN202410599863.9 discloses a multi-target intelligent sorting method for flexible parts, including: partitioning the picking material frame, scanning the entire material frame through an X-ray scanning module, and analyzing and processing the image generated by the scan; extracting the edge, texture and density information of the inspected object in each area, identifying and locating the inspected object; locking the calibrated and identified flexible part position, determining the picking point of each flexible part in the partition, and driving the robotic arm to pick up in the partitions in turn; after the partition picking is completed, the parts in the picking material frame are shuffled, and re-scanned through the secondary X-ray scanning module until the object recognition module cannot recognize the flexible parts, and the picking material frame is replaced for subsequent sorting. The advantage of this invention is that the robotic arm has a fast grasping speed, which can effectively prevent the omission of flexible parts, thereby improving the accuracy of sorting. However, the system still uses the traditional two-dimensional shooting method to locate the coordinates of the object, which makes it impossible to obtain the three-dimensional information of the object, so it is not suitable for the robotic arm to grasp the complex and irregular objects.

[0005] The patent number CN202410789869.2 discloses a multi-target sorting method for multi-robot collaboration, which includes the following steps: building a pipeline multi-robot sorting system, using the Yolov8 algorithm to perform target detection on the RGB images taken by the visual robot arm, and obtaining the object boundary box detected by Yolov8, combining it with its corresponding depth image, simulating the sorting scene, building a LightGBM regression model, training and verifying the model performance, and optimizing the task allocation and scheduling sequence of each robot arm based on the expected sorting time of the robot arm, and minimizing the total sorting time; using the particle swarm algorithm to optimize the robot arm scheduling process; after each robot arm obtains the grasping task sequence, multi-arm path planning and obstacle avoidance are performed based on the APF-RRT algorithm. The advantage of this invention is that it effectively reduces the time wasted by the robot arm in collaborative tasks due to waiting for each other, outputs the optimal grasping sequence, thereby improving the sorting efficiency, and selecting the best path to avoid obstacles. However, the algorithm mainly relies on the training and processing of two-dimensional images, which requires the collection of a large number of two-dimensional pictures of scenes and object targets. This process is relatively complex and time-consuming, and does not involve the 3D imaging part.

[0006] Patent No. CN201810942396.X discloses a method and device for obtaining gripping points of a robotic arm. The patent scans an object through the movement of the robotic arm itself, obtains a stereoscopic image of the object and constructs its three-dimensional model, demonstrating efficient and high-quality data acquisition capabilities. In addition, a simulated image of the actual scene is generated using the three-dimensional model of the object, and a deep neural network is trained based on these simulated images to predict the pose information of the object. The method performs foreground and background segmentation based on the depth map on the scanned stereo image to obtain a stereoscopic image of the object after removing the background. A preset number of training samples are generated by rendering the simulated image, and the deep neural network is trained using these samples. The current stereoscopic image is input into the network to calculate the pose information of the object in the image. Although this method makes full use of the three-dimensional point cloud data and achieves good results, it takes a long time to train the three-dimensional point cloud samples, and the process of generating the training set is relatively complicated. Summary of the invention

[0007] The purpose of the present invention is to provide a 3D vision-guided robotic arm grasping method for irregular objects to solve the problems existing in the prior art.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A 3D vision-guided robotic arm grasping method for irregular objects comprises the following steps:

[0010] (1) Build a 3D vision-guided robotic arm grasping system: Use a 3D camera based on line structured light to guide the robotic arm. The robotic arm is a visual flexible six-axis robotic arm. When the robotic arm completes the accurate grasping of all objects and places them as required, it is considered to have successfully completed this grasping task; calibrate the camera in the 3D camera. The camera calibration is implemented by calling the CalibrateCamera function in the OpenCV library, and the point cloud of the target object is collected in advance through the 3D camera, and a matching template is created using 3D mapping software;

[0011] (2) Line structured light 3D reconstruction: The laser in the 3D camera projects a line laser onto the target object pile, performs line structured light 3D reconstruction on the object pile, and finally completes the acquisition of the object pile point cloud through the encoded and decoded image;

[0012] (3) Hand-eye calibration: The camera and the robotic arm are calibrated to obtain the posture transformation between the camera coordinate system and the robotic arm base coordinate system. This transformation is achieved through a rotation and translation matrix [R C_r ,T C_r ] indicates that the visual information is converted to the robot base coordinate system through this matrix;

[0013] (4) Point cloud preprocessing: The obtained 3D scene object pile point cloud is filtered and denoised to achieve a smoothing effect, the object pile point cloud is segmented using the Euclidean clustering point cloud segmentation algorithm, and multiple candidate target point cloud sets are selected through screening;

[0014] (5) Object pose estimation: The pose estimation algorithm based on curvature point pair features is used to estimate the pose of the selected candidate target point cloud set to obtain the position and pose of the objects in each scene;

[0015] (6) Path planning and grasping operation: After the robot arm obtains the grasping posture sequence, it performs path planning and obstacle avoidance based on the RRT algorithm, and sends the path information to the robot arm to complete the grasping and placement of the target object.

[0016] Furthermore, the method of line structured light 3D reconstruction in step (2) is:

[0017] 2.1. System calibration: According to the wavelength of the laser in the 3D camera, select a suitable narrow-band filter, keep the position between the 3D camera and the laser fixed, and project the laser line onto the calibration plate with different postures to form two intersecting straight lines. Fit the points extracted by the laser line to the two straight lines to solve the laser plane equation;

[0018] 2.2 Image processing: The laser line energy distribution is normal distribution, bilateral filtering is used to remove Gaussian noise, and distortion correction is performed;

[0019] 2.3. Extract the center line of the laser stripe: extract the point where the first-order derivative is 0 and the second-order directional derivative takes the minimum value as the center point of the line;

[0020] 2.4. Point cloud stitching: Convert the camera coordinate system into a coordinate system with the object's motion direction as the X-axis. After stitching is completed, convert it back to the camera coordinate system. The stitching displacement of the two laser lines is the object's motion speed multiplied by the camera's shooting interval. The shooting speed is determined by the required scanning accuracy.

[0021] Furthermore, the method of hand-eye calibration in step (3) is:

[0022] The tip probe is fixed at the output end of the robot arm, and a calibration plate is placed in the working area. When the line structured light stripe pattern is projected onto the calibration plate, the 3D model of the calibration plate can be restored by the structured light algorithm. At this time, the coordinate point P of the corner point in the camera coordinate system can be obtained. i (x Ci ,y Ci ,z Ci ), by controlling the movement of the robotic arm to align the tip probe with the corner point of the calibration plate, the corresponding three-dimensional coordinate point Q in the robotic arm coordinate system is obtained i (x ri ,y ri ,zri ), use the least squares method to calculate the hand-eye calibration matrix [R C_r ,T C_r ], the mathematical model is:

[0023]

[0024] in is the mathematical model of the hand-eye calibration matrix, R C_r is the rotation component in the hand-eye calibration matrix, T C_r is the translation classification in the hand-eye calibration matrix, and n is the total number of corner points.

[0025] Furthermore, the method for point cloud segmentation in step (4) is:

[0026] The point cloud segmentation method uses Euclidean clustering for processing. For Euclidean clustering, the judgment criterion is Euclidean distance. For a point P in space, the KD-Tree nearest neighbor search algorithm is used to find the k points closest to point P. Among these points, those whose distance is less than the set threshold are clustered into the set Q. If the number of elements in Q does not increase, the entire clustering process ends. Otherwise, points other than point P are selected in the set Q and the process is repeated until the number of elements in Q does not increase.

[0027] Furthermore, the pose estimation algorithm based on curvature features in step (5) is based on the four feature vectors f of the feature PPF at the original point 1 、f 2 、f 3 、f 4 On the basis of this, the curvature information of the point cloud is introduced 5 and f 6 , is a six-dimensional feature vector composed of the distance information of two points, the normal vector and the average curvature, thereby enhancing the feature description of the point cloud, and defining the curvature point pair feature as F Cur-PPF (m 1 ,m 2 ), and its calculation formula is:

[0028] F Cur-PPF (m 1 ,m 2 )=(f 1 ,f 2 ,f 3 ,f 4 ,f 5 ,f 6 )=(||d|| 2 ,∠(n 1 ,d),∠(n 2 ,d),∠(n 1 ,n 2 ),q 1 ,q2 );

[0029] Among them, m 1 and m 2 are any two points in the object point cloud; n 1 and n 2 m 1 and m 2 The normal vector, q 1 and q 2 m 1 and m 2 The mean curvature of 2 -m 1 ,||d|| 2 is m 1 and m 2 The Euclidean distance, ∠(n 1 ,d) is n 1 The angle between ∠(n 2 ,d) is n 2 The angle between ∠(n 1 ,n 2 ) is n 1 and n 2 The angle between.

[0030] Furthermore, the method for estimating the object pose in step (5) is:

[0031] The six-dimensional point pair features obtained by the pose estimation algorithm based on curvature features are mapped to the template point cloud with similar features. The matrix transformation [R, T] from model to scene can be obtained by a reference point m in the model point cloud. i and a rotation angle α, that is, (m i ,α) is the model relative to the scene reference point s i The local coordinates of are expressed as:

[0032] Among them, s i is any point in the scene, m i is any point in the model, T m→g It is point m i Transformation matrix to the origin, T s→g It's point s i The transformation matrix to the origin, R x (α) is the rotation matrix of the model to the scene by an angle α around the x-axis.

[0033] Furthermore, the voting link of the curvature point pair feature assigns different weights to the votes according to the size of the curvature difference. The larger the curvature difference between the two points, the greater the difference in the curvature of the surface where they are located, and the more F Cur-PPF (m 1 ,m2 ) The more obvious the feature is, the better it can represent the shape characteristics of the model. The point pairs with larger curvature difference contain more significant information, and are distributed farther, so the calculated pose is more reliable. The weight calculation formula is:

[0034]

[0035] Where v is the number of votes greater than 1.

[0036] Furthermore, the RRT path planning method in step (6) is:

[0037] 6.1. Determine the starting position, weld position and obstacle position of the robot arm. The obstacles are divided into environmental obstacles and the expected trajectory planned by the remaining robot arms. The starting position is used as the root node of the RRT random tree.

[0038] 6.2. Randomly sample a point in the workspace as the exploration target point, find the node closest to the sampling point in the random number, extend a distance from the nearest node to the sampling point, and create a new node;

[0039] 6.3. Check whether the new node collides with the obstacle. If not, add it to the random tree and connect it to the nearest node.

[0040] 6.4. Search for neighbors within a defined radius near the newly generated node as candidates for replacing the parent node. Calculate the path cost from the neighboring node to the starting point plus the path cost to each neighboring node, and select the node with the smallest cost as the new parent node;

[0041] 6.5. After reselecting the parent node, further optimize the cost of the connection between random tree nodes;

[0042] 6.6. For the robot arm, if it can successfully connect to the target position, it is considered that a valid path has been found; if a valid path cannot be found within a limited number of iterations or time, re-sampling and path extension are performed;

[0043] 6.7. Repeat steps 6.1-6.6 to generate the final path of the robot arm's grasping and placement process.

[0044] Furthermore, the 3D vision-guided robotic arm grasping system is provided with a 3D camera, a bracket, and a robotic arm. The 3D camera is provided with a camera and a laser. The camera and the laser are both mounted on the bracket, and the robotic arm is arranged on one side of the bracket.

[0045] Furthermore, a linear sliding module is installed on the bracket, and the camera and laser are installed at the sliding end of the linear sliding module. The camera and laser are installed on the bracket through the sliding module. The linear sliding module is located above the platform, and a starting position sensor and an ending position sensor are installed at both ends of the linear sliding module.

[0046] The present invention has the following beneficial effects: through the cooperation between the 3D camera and the robot arm, the intelligent grasping of irregular objects such as industrial parts is realized, and more accurate grasping operations are performed according to actual application conditions, and it does not rely on the training of three-dimensional point cloud samples. The 3D camera can accurately obtain external information to meet the grasping requirements of the robot arm. The robot arm can grasp the stacked parts in the material box one by one, improve the grasping efficiency, and further improve the level of industrial automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the present invention.

[0048] Figure 2 It is a schematic diagram of hand-eye calibration of the present invention.

[0049] Figure 3 It is a schematic diagram of the European clustering process of the present invention.

[0050] Figure 4 It is a schematic diagram of the model description of the curvature point pair feature of the present invention.

[0051] Figure 5 It is a structural diagram of embodiment 1 of the present invention.

[0052] Figure 6 It is a schematic diagram of the structure of embodiment 2 of the present invention.

[0053] Figure 7 This is a real machine screen picture of the 3D camera visual guidance of the present invention.

[0054] Among them: 1. Camera; 2. Laser; 3. Bracket; 4. Robotic arm; 5. Linear sliding module; 6. Starting position sensor; 7. End position sensor; 8. Conveyor. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] Embodiment 1:

[0057] like Figure 1-5 As shown, this embodiment provides a 3D vision-guided robotic arm grasping method for irregular objects, comprising the following steps:

[0058] (1) Build a 3D vision-guided robotic arm grasping system: Use a 3D camera based on line structured light to guide the robotic arm 4, which is a visual flexible six-axis robotic arm. When the robotic arm 4 completes the accurate grasping of all objects and places them as required, it is considered to have successfully completed the grasping task; calibrate camera 1 in the 3D camera. The camera 1 calibration is implemented by calling the CalibrateCamera function in the OpenCV library, and the point cloud of the target object is collected in advance through the 3D camera, and a matching template is created using 3D mapping software.

[0059] (2) Line structured light 3D reconstruction: Laser 2 in the 3D camera projects a line laser onto the target object pile, performs line structured light 3D reconstruction on the object pile, and finally completes the acquisition of the object pile point cloud by encoding and decoding the image. The specific method is:

[0060] 2.1. System calibration: According to the wavelength of laser 2 in the 3D camera, select a suitable narrow-band filter, keep the position between camera 1 and laser 2 of the 3D camera fixed, and project the laser line onto the calibration plate with different postures to form two intersecting straight lines. Fit the points extracted by the laser line to the two straight lines to solve the laser plane equation;

[0061] 2.2 Image processing: The laser line energy distribution is normal distribution, bilateral filtering is used to remove Gaussian noise, and distortion correction is performed;

[0062] 2.3. Extract the center line of the laser stripe: extract the point where the first-order derivative is 0 and the second-order directional derivative takes the minimum value as the center point of the line;

[0063] 2.4. Point cloud stitching: Convert the coordinate system of camera 1 into the coordinate system with the object's movement direction as the X-axis. After stitching is completed, convert it back to the coordinate system of camera 1. The stitching displacement of the two laser lines is the object's movement speed multiplied by the interval between camera 1 photos. The shooting speed is determined by the required scanning accuracy.

[0064] (3) Hand-eye calibration: The camera 1 and the robot arm 4 are calibrated to obtain the posture transformation between the camera 1 coordinate system and the robot arm 4 base coordinate system. This transformation is achieved through a rotation and translation matrix [R C_r ,T C_r ] indicates that the visual information is converted into the robot 4-base coordinate system through this matrix.

[0065] The method of hand-eye calibration is:

[0066] A tip probe is fixed at the output end of the robot 4, and a calibration plate is placed in the working area. When the line structured light stripe pattern is projected onto the calibration plate, the 3D model of the calibration plate can be restored by the structured light algorithm. At this time, the coordinate point P of the corner point in the coordinate system of camera 1 can be obtained.i (x Ci ,y Ci ,z Ci ), by controlling the movement of the robot arm 4 to align the tip probe with the corner point of the calibration plate, the corresponding three-dimensional coordinate point Q in the coordinate system of the robot arm 4 is obtained i (x ri ,y ri ,z ri ), use the least squares method to calculate the hand-eye calibration matrix [R C_r ,T C_r ], the mathematical model is:

[0067]

[0068] in is the mathematical model of the hand-eye calibration matrix, R C_r is the rotation component in the hand-eye calibration matrix, T C_r is the translation classification in the hand-eye calibration matrix, and n is the total number of corner points.

[0069] (4) Point cloud preprocessing: The obtained 3D scene object pile point cloud is filtered and denoised to achieve a smoothing effect. The object pile point cloud is segmented using the Euclidean clustering point cloud segmentation algorithm, and multiple candidate target point cloud sets are selected through screening.

[0070] The point cloud segmentation method is as follows: the point cloud segmentation method uses Euclidean clustering for processing. For Euclidean clustering, the judgment criterion is Euclidean distance. For a point P in space, the KD-Tree nearest neighbor search algorithm is used to find the k points closest to point P. Among these points, those whose distance is less than the set threshold are clustered into the set Q. If the number of elements in Q does not increase, the entire clustering process ends. Otherwise, points other than point P are selected in the set Q and the process is repeated until the number of elements in Q does not increase.

[0071] (5) Object pose estimation: The pose estimation algorithm based on curvature point pair features is used to estimate the pose of the selected candidate target point cloud set to obtain the position and pose of the objects in each scene.

[0072] The pose estimation algorithm based on curvature features is to calculate the four eigenvectors f of the feature PPF at the original point 1 、f 2 、f 3 、f 4 On the basis of this, the curvature information of the point cloud is introduced 5 and f 6 , is a six-dimensional feature vector composed of the distance information of two points, the normal vector and the average curvature, thereby enhancing the feature description of the point cloud, and defining the curvature point pair feature as F Cur-PPF (m 1 ,m2 ), and its calculation formula is:

[0073] F Cur-PPF (m 1 ,m 2 )=(f 1 ,f 2 ,f 3 ,f 4 ,f 5 ,f 6 )=(||d|| 2 ,∠(n 1 ,d),∠(n 2 ,d),∠(n 1 ,n 2 ),q 1 ,q 2 );

[0074] Among them, m 1 and m 2 are any two points in the object point cloud; n 1 and n 2 m 1 and m 2 The normal vector, q 1 and q 2 m 1 and m 2 The mean curvature of 2 -m 1 ,||d|| 2 is m 1 and m 2 The Euclidean distance, ∠(n 1 ,d) is n 1 The angle between ∠(n 2 ,d) is n 2 The angle between ∠(n 1 ,n 2 ) is n 1 and n 2 The angle between.

[0075] The method for object pose estimation is:

[0076] The six-dimensional point pair features obtained by the pose estimation algorithm based on curvature features are mapped to the template point cloud with similar features. The matrix transformation [R, T] from model to scene can be obtained by a reference point m in the model point cloud. i and a rotation angle α, that is, (m i ,α) is the model relative to the scene reference point s i The local coordinates of are expressed as:

[0077] Among them, si is any point in the scene, m i is any point in the model, T m→g It is point m i Transformation matrix to the origin, T s→g It's point s i The transformation matrix to the origin, R x (α) is the rotation matrix of the model to the scene by an angle α around the x-axis.

[0078] The voting link of the curvature point pair feature assigns different weights to the votes according to the size of the curvature difference. The larger the curvature difference between the two points, the greater the difference in the curvature of the surface where they are located, and the more F Cur-PPF (m 1 ,m 2 ) The more obvious the feature is, the better it can represent the shape characteristics of the model. The point pairs with larger curvature difference contain more significant information, and are distributed farther, so the calculated pose is more reliable. The weight calculation formula is:

[0079]

[0080] Where v is the number of votes greater than 1.

[0081] (6) Path planning and grasping operation: After the robot arm 4 obtains the grasping posture sequence, it performs path planning and obstacle avoidance based on the RRT algorithm, and sends the path information to the robot arm 4 to complete the grasping and placement of the target object.

[0082] The RRT path planning method is:

[0083] 6.1. Determine the starting position, weld position and obstacle position of the robot 4, where the obstacles are divided into environmental obstacles and the expected trajectory planned by the remaining robot 4, and use the starting position as the root node of the RRT random tree;

[0084] 6.2. Randomly sample a point in the workspace as the exploration target point, find the node closest to the sampling point in the random number, extend a distance from the nearest node to the sampling point, and create a new node;

[0085] 6.3. Check whether the new node collides with the obstacle. If not, add it to the random tree and connect it to the nearest node.

[0086] 6.4. Search for neighbors within a defined radius near the newly generated node as candidates for replacing the parent node. Calculate the path cost from the neighboring node to the starting point plus the path cost to each neighboring node, and select the node with the smallest cost as the new parent node;

[0087] 6.5. After reselecting the parent node, further optimize the cost of the connection between random tree nodes;

[0088] 6.6. For the robot 4, if it can successfully connect to the target position, it is considered that a valid path has been found; if a valid path cannot be found within a limited number of iterations or time, re-sampling and path extension are performed;

[0089] 6.7. Repeat steps 6.1-6.6 to generate the path of the final robot arm 4 grabbing and placing process.

[0090] The objects to be grasped in this embodiment are various (i.e., inconsistent shapes) and irregular (i.e., irregular shapes) objects. The objects are transported by the conveyor 8 of the assembly line, and the objects are placed in a single layer on the conveyor 8. The objects need to be grasped to a specified position. The bracket 3 is installed on one side of the conveyor 8, the camera 1 and the laser 2 are fixed on the bracket 3, and the mechanical arm 4 is set on one side of the conveyor 8. The conveyor 8 conveys the objects at a certain speed, and the working surfaces of the camera 1 and the laser 2 are facing the conveyor 8. The 3D camera is calibrated before use, and the point cloud of the target object is collected with the 3D camera to create a matching template.

[0091] When in use, when the object pile on the conveyor passes through the working area of ​​the 3D camera, the laser of the laser 2 will be projected on the object pile, and the camera 1 will capture the laser stripe image in real time, extract the center coordinates of the laser stripe, calibrate the laser plane coefficient, and calculate the three-dimensional coordinates of a laser stripe in the current frame. After calibrating the speed, the coordinate system is converted to solve the speed along the moving direction, and the laser stripe of the next frame is spliced ​​and processed. This process is repeated to complete the acquisition of high-precision three-dimensional data of irregular object piles in the field of view. After obtaining the three-dimensional data, point cloud filtering, point cloud segmentation and other steps are performed, and the segmented point cloud is matched with the created template to obtain the pose information of the segmented point cloud. After the object pose is estimated and the path is planned, the robot arm 4 is controlled to complete the grasping and placement actions.

[0092] Embodiment 2: This embodiment provides a 3D vision-guided robotic arm grasping method for irregular objects. The method steps are basically the same as those in Embodiment 1, except that:

[0093] like Figure 6-7As shown, the grasping objects faced by this embodiment are irregular (i.e., irregularly shaped) objects. The objects can be of a single type (i.e., consistent shape) or of multiple types (i.e., inconsistent shapes). The objects are piled up in a material box and their positions are disordered. The material box is transported to the position of the 3D camera by the conveyor 8 of the assembly line and stops. The objects in the material box need to be moved to the specified position. A linear sliding module 5 is installed on the bracket 3, and the camera 1 and the laser 2 are installed at the sliding end of the linear sliding module 5. The starting position sensor 6 and the ending position sensor 7 are installed at both ends of the linear sliding module 5. The conveyor 8 is located below the linear sliding module 5, and the robot arm 4 is set on one side of the conveyor 8. The conveyor 8 transports the material box at a certain speed, and the working surfaces of the camera 1 and the laser 2 are facing the conveyor 8. The 3D camera is calibrated before use, and the point cloud of the target object is collected by the 3D camera to create a matching template.

[0094] When in use, after the material box is in place, the conveyor 8 stops, and the sliding end of the linear sliding module 5 moves from the starting position sensor 6 to the ending position sensor 7, scans the object pile in the material box and returns to the starting position sensor 6. During this process, the laser 2 will project the laser on the object pile, and the camera 1 will capture the laser stripe image in real time, extract the center coordinates of the laser stripe, calibrate the laser plane coefficient, and calculate the three-dimensional coordinates of a laser stripe in the current frame. After calibrating the speed, the coordinate system is converted to solve the speed along the moving direction, and the laser stripe of the next frame is spliced ​​and processed. This is repeated to complete the acquisition of high-precision three-dimensional data of the irregular object pile in the field of view. After obtaining the three-dimensional data, point cloud filtering and point cloud segmentation are performed, and the segmented point cloud is matched with the created template to obtain the posture information of the segmented point cloud. After the object posture estimation and path planning, the robot arm 4 is controlled to complete the grasping and placement actions.

[0095] After completing the grabbing and placement of the material pile scanned this time, the 3D camera scans again and controls the robot arm 4 to place it, and repeats this process until the material box is empty. The conveyor 8 continues to transport the next material box to the 3D camera, and the process is repeated continuously.

[0096] like Figure 7As shown in the figure, the actual machine screen of object recognition in two scene examples of three-way pipe and wooden column is shown. Since it is impossible to demonstrate which are the segmented candidate target point cloud sets and which are the objects matched by pose estimation only by black and white, they are distinguished by color. In actual use, due to the occlusion between objects, a large number of invalid point pairs will be generated when calculating features. Therefore, point cloud segmentation is performed first to filter out the candidate point cloud target set, thereby saving time. After the 3D camera scans the object pile, it generates a scene point cloud map, and then performs point cloud segmentation based on the Euclidean clustering point cloud segmentation algorithm. The segmented point cloud is matched with the created template, and the candidate target point cloud set with a relatively complete outline can be filtered out. The pose estimation algorithm based on the curvature point pair feature is used to estimate the pose of the filtered candidate target point cloud set to obtain the position and posture of the objects in each scene. After the voting link, the candidate target point cloud set that best represents the shape characteristics of the object is matched, and this is used as the object to be grasped first.

[0097] The above-described embodiments are merely descriptions of preferred implementations of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in the art to the technical solution of the present invention should all fall within the scope of protection of the present invention.

[0098] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A 3D vision-guided robotic arm grasping method for irregular objects, characterized in that: The following steps are involved: (1) Build a 3D vision-guided robotic arm grasping system: Use a 3D camera based on line structured light to guide the robotic arm. The robotic arm is a visual flexible six-axis robotic arm. When the robotic arm completes the accurate grasping of all objects and places them as required, it is considered to have successfully completed this grasping task; calibrate the camera in the 3D camera. The camera calibration is implemented by calling the CalibrateCamera function in the OpenCV library, and the point cloud of the target object is collected in advance through the 3D camera, and a matching template is created using 3D mapping software; (2) Line structured light 3D reconstruction: The laser in the 3D camera projects a line laser onto the target object pile, performs line structured light 3D reconstruction on the object pile, and finally completes the acquisition of the object pile point cloud through the encoded and decoded image; (3) Hand-eye calibration: The camera and the robotic arm are calibrated to obtain the posture transformation between the camera coordinate system and the robotic arm base coordinate system. This transformation is achieved through a rotation and translation matrix [R C_r ,T C_r ] indicates that the visual information is converted to the robot base coordinate system through this matrix; (4) Point cloud preprocessing: The obtained 3D scene object pile point cloud is filtered and denoised to achieve a smoothing effect, the object pile point cloud is segmented using the Euclidean clustering point cloud segmentation algorithm, and multiple candidate target point cloud sets are selected through screening; (5) Object pose estimation: The pose estimation algorithm based on curvature point pair features is used to estimate the pose of the selected candidate target point cloud set to obtain the position and pose of the objects in each scene; (6) Path planning and grasping operation: After the robot arm obtains the grasping posture sequence, it performs path planning and obstacle avoidance based on the RRT algorithm, and sends the path information to the robot arm to complete the grasping and placement of the target object.

2. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The method for line structured light 3D reconstruction in step (2) is: 2.

1. System calibration: According to the wavelength of the laser in the 3D camera, select a suitable narrow-band filter, keep the position between the 3D camera and the laser fixed, and project the laser line onto the calibration plate with different postures to form two intersecting straight lines. Fit the points extracted by the laser line to the two straight lines to solve the laser plane equation; 2.2 Image processing: The laser line energy distribution is normal distribution, bilateral filtering is used to remove Gaussian noise, and distortion correction is performed; 2.

3. Extract the center line of the laser stripe: extract the point where the first-order derivative is 0 and the second-order directional derivative takes the minimum value as the center point of the line; 2.

4. Point cloud stitching: Convert the camera coordinate system into a coordinate system with the object's motion direction as the X-axis. After stitching is completed, convert it back to the camera coordinate system. The stitching displacement of the two laser lines is the object's motion speed multiplied by the camera's shooting interval. The shooting speed is determined by the required scanning accuracy.

3. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The method of hand-eye calibration in step (3) is: The tip probe is fixed at the output end of the robot arm, and a calibration plate is placed in the working area. When the line structured light stripe pattern is projected onto the calibration plate, the 3D model of the calibration plate can be restored by the structured light algorithm. At this time, the coordinate point P of the corner point in the camera coordinate system can be obtained. i (x Ci ,y Ci ,z Ci ), by controlling the movement of the robotic arm to align the tip probe with the corner point of the calibration plate, the corresponding three-dimensional coordinate point Q in the robotic arm coordinate system is obtained i (x ri ,y ri ,z ri ), use the least squares method to calculate the hand-eye calibration matrix [R C_r ,T C_r ], the mathematical model is: in is the mathematical model of the hand-eye calibration matrix, R C_r is the rotation component in the hand-eye calibration matrix, T C_r is the translation classification in the hand-eye calibration matrix, and n is the total number of corner points.

4. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The method for point cloud segmentation in step (4) is: The point cloud segmentation method uses Euclidean clustering for processing. For Euclidean clustering, the judgment criterion is Euclidean distance. For a point P in space, the KD-Tree nearest neighbor search algorithm is used to find the k points closest to point P. Among these points, those whose distance is less than the set threshold are clustered into the set Q. If the number of elements in Q does not increase, the entire clustering process ends. Otherwise, points other than point P are selected in the set Q and the process is repeated until the number of elements in Q does not increase.

5. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The pose estimation algorithm based on curvature features in step (5) introduces the curvature information f5 and f6 of the point cloud on the basis of the four feature vectors f1, f2, f3, and f4 of the original point pair feature PPF. The six-dimensional feature vectors are composed of the distance information of two points, the normal vector, and the average curvature, thereby enhancing the feature description of the point cloud. The curvature point pair feature is defined as F Cur-PPF (m1,m2), the calculation formula is: <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> Cur-PPF <h2 style=";text-align:left;direction:ltr"> (m1,m2)=(f1,f2,f3,f4,f5,f6)=(||d||2,∠(n1,d),∠(n2,d),∠(n1,n2),q1,q2); Among them, m1 and m2 are any two points in the object point cloud; n1 and n2 are the normal vectors of m1 and m2 respectively, q1 and q2 are the average curvatures of m1 and m2 respectively, d=m2-m1, ||d||2 is the Euclidean distance between m1 and m2, ∠(n1,d) is the angle between n1 and d, ∠(n2,d) is the angle between n2 and d, and ∠(n1,n2) is the angle between n1 and n2.

6. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 5, characterized in that: The method for estimating the object pose in step (5) is: The six-dimensional point pair features obtained by the pose estimation algorithm based on curvature features are mapped to the template point cloud with similar features. The matrix transformation [R, T] from model to scene can be obtained by a reference point m in the model point cloud. i and a rotation angle α, that is, (m i ,α) is the model relative to the scene reference point s i The local coordinates of are expressed as: Among them, s i is any point in the scene, m i is any point in the model, T m→g It is point m i Transformation matrix to the origin, T s→g It's point s i The transformation matrix to the origin, R x (α) is the rotation matrix of the model to the scene by an angle α around the x-axis.

7. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 6, characterized in that: The voting link of the curvature point pair feature assigns different weights to the votes according to the size of the curvature difference. The larger the curvature difference between the two points, the greater the difference in the curvature of the surface where they are located, and the more F Cur-PPF The more obvious the (m1, m2) feature is, the better it can represent the shape characteristics of the model. The point pairs with larger curvature differences contain more significant information and are distributed farther away, so the calculated pose is more reliable. The weight calculation formula is: Where v is the number of votes greater than 1.

8. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The RRT path planning method in step (6) is: 6.

1. Determine the starting position, weld position and obstacle position of the robot arm. The obstacles are divided into environmental obstacles and the expected trajectory planned by the remaining robot arms. The starting position is used as the root node of the RRT random tree. 6.

2. Randomly sample a point in the workspace as the exploration target point, find the node closest to the sampling point in the random number, extend a distance from the nearest node to the sampling point, and create a new node; 6.

3. Check whether the new node collides with the obstacle. If not, add it to the random tree and connect it to the nearest node. 6.

4. Search for neighbors within a defined radius around the newly generated node as candidates for replacing the parent node. Calculate the path cost from the neighboring node to the starting point plus the path cost to each neighboring node, and select the node with the smallest cost as the new parent node; 6.

5. After reselecting the parent node, further optimize the cost of the connection between random tree nodes; 6.

6. For the robot arm, if it can successfully connect to the target position, it is considered that a valid path has been found; if a valid path cannot be found within a limited number of iterations or time, re-sampling and path extension are performed; 6.

7. Repeat steps 6.1-6.6 to generate the final path of the robot arm's grasping and placement process.

9. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 1, characterized in that: The 3D vision-guided robotic arm grasping system is provided with a 3D camera, a bracket, and a robotic arm. The 3D camera is provided with a camera and a laser. The camera and the laser are both installed on the bracket, and the robotic arm is arranged on one side of the bracket.

10. The 3D vision-guided robotic arm grasping method for irregular objects according to claim 9, characterized in that: A linear sliding module is installed on the bracket, and a camera and a laser are installed at the sliding end of the linear sliding module. The camera and the laser are installed on the bracket through the sliding module. The linear sliding module is located above the platform, and a starting position sensor and an ending position sensor are installed at both ends of the linear sliding module.

Images