A method for sorting disordered stacked materials based on deep learning and point cloud

Abstract

The present invention relates to a method for sorting disordered stacked materials based on deep learning and point cloud, the method comprising: constructing a data set of scattered disordered stacked materials; training a YOLOv5s material recognition model; constructing a three-dimensional point cloud of disordered stacked materials based on the target reasoning result of YOLOv5 and the depth image collected by a depth camera; estimating the grasping posture of the end of the manipulator using an improved random consistency sampling algorithm; estimating the material size using the YOLOv5 reasoning bounding box; estimating the optimal shooting height of the material using the camera field of view; realizing the grasping control of the manipulator by the host computer using serial communication; and classifying and sorting the materials using the estimated material size. The present invention is designed and developed based on the concepts of visual perception and deep learning, and the 2D color image deep learning reasoning result is fused with the 3D depth image to construct a point cloud, which improves the accuracy of recognition of disordered stacked materials and grasping posture estimation, and at the same time, classifies and sorts the materials according to the estimated material size, thereby improving the stability of material sorting.

Description

Technical Field

[0001] The present invention relates to the fields of logistics warehousing and robot control, and specifically to a method for sorting disordered stacked materials based on deep learning and point cloud. Background Art

[0002] The picking and sorting of fruits and vegetables in the agricultural field, the sorting and assembly of parts in the manufacturing industry, the identification and sorting of recyclable materials in the renewable resources field, the classification and sorting of express delivery in the logistics industry, etc. Due to technical limitations, the above applications often rely on manual operations, which have the problems of high labor intensity and low work efficiency. If robots are used instead of manual operations, the above application scenarios can all be attributed to the problem of robots sorting disordered stacked materials. Unlike the grasping of planar targets, materials in three-dimensional scenes are stacked in disorder and occlude each other. How to accurately identify and classify materials in disordered scenes, and accurately detect their position and posture in the robot's workspace, is the primary key issue in robot grasping operations. Summary of the invention

[0003] In response to the above-mentioned problems, the present invention proposes a method for sorting disordered stacked materials based on deep learning and point cloud.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A method for sorting disordered stacked materials based on deep learning and point cloud, the method comprising:

[0006] S1. Obtain the material data set required for training, preprocess the data set and complete data labeling;

[0007] S2. Use the YOLOv5 target detection network to train an algorithm model for identifying disordered stacked materials;

[0008] S3, constructing a 3D point cloud of disordered stacked materials based on the target recognition results of YOLOv5 and the images collected by the depth camera;

[0009] S4. Select the actual spatial coordinates of the center point of the bounding box identified by YOLOv5 as the end point of the robot grasping, use the improved random consistency sampling method to perform multi-plane estimation on the three-dimensional point cloud of disordered stacked materials, and select the plane normal vector with the largest number of points in the estimated plane and containing the end grasping point as the end posture of the robot grasping; estimate the lateral size of the material based on the point cloud data, and classify the size according to the estimated size of the material; combine the estimated size of the material and the field of view of the depth camera color frame to estimate the best shooting point of a single material;

[0010] S5. Convert each point in the camera coordinate system to the robot base coordinate system according to the coordinate transformation relationship to obtain the joint coordinates of the robot corresponding to each point;

[0011] S6. Use serial communication, Arduino UNO development board, Arduino Mega 2560 development board and host computer to control the end effector of the robot arm.

[0012] The data set in S1 collects multiple sets of color images of disordered stacked materials, converts the RGB images to 640×480 resolution, and uses methods such as rotation and flipping for data enhancement. The LabelImg software is used to annotate the complete materials on the surface and not blocked in the RGB images of disordered stacked materials with rectangular labels. The label saving format is selected in YOLO format, and the sample size after labeling is about 1700;

[0013] In S2, the YOLOv5 target detection network is used for model training. The samples are divided into training set, validation set and test set in a ratio of 3:1:1. About 1000 samples are used for model training, about 340 samples are used for model validation, and about 340 samples are used for model testing. After the model training is completed, the detect script of YOLOv5 is rewritten into the DetectAPI class for externally calling the reasoning API of YOLOv5.

[0014] In the S3, a depth camera is used to collect the aligned color image and depth image of the disorderly stacked materials to be sorted, wherein the color image is used for YOLOv5 reasoning, and the depth image is converted into an 8-bit grayscale image by absolute value scaling for constructing a three-dimensional point cloud. The larger the grayscale value, the farther the target is from the camera. The scale factor of the scaling represents the effective shooting distance of the required depth camera. The trained YOLOv5 model is used to reason the color image, and the material with the highest confidence in the reasoning result is selected as the current grasping target of the robot arm. The grayscale image is intercepted according to the identified bounding box, and the intercepted area is the grayscale (depth) image corresponding to the area identified by YOLOv5. According to the camera imaging model, each two-dimensional pixel point and the corresponding depth value in the identified bounding box are converted into the camera coordinate system. The position between each three-dimensional point in the camera coordinate system represents the actual spatial position, and the construction of the three-dimensional point cloud is completed.

[0015] The calculation of the grasping point of the robot arm in S4 is to select the three-dimensional point corresponding to the central pixel point of the YOLOv5 recognition bounding box in the three-dimensional point cloud as the end point of the robot arm grasping;

[0016] The calculation of the grasping posture of the manipulator in S4 is based on the method of realizing the best grasping plane estimation based on the improved random consistency sampling algorithm:

[0017] The random consistency sampling algorithm randomly selects three planes containing non-collinear points in the three-dimensional point cloud as the estimated plane model, calculates the normal vector of the estimated plane and a point on the plane, calculates the distances of all other points to the estimated plane, and takes the points with distances less than a threshold as the inliers of the plane model. The above steps are repeated for multiple iterations, and the plane model with the largest number of inliers is selected as the best estimated plane model of the algorithm in the iteration.

[0018] Improve the above algorithm. After each random consistency sampling is performed to obtain the best estimated plane, all the inner points of the estimated plane are removed from the three-dimensional point cloud. Random consistency sampling is performed again. The above steps are repeated until the number of points in the three-dimensional point cloud is less than 3. The multi-plane estimation of the three-dimensional point cloud is completed. The plane normal vector whose inner point contains the end point of the robot arm not grasped is selected from all the fitting planes as the end posture orientation of the robot arm grasping the target.

[0019] In the S4, the size estimation of the material is to convert the length and width of the YOLOv5 recognition bounding box into the distance in the actual space, and select the larger distance value as the estimated size of the material. The estimated size ranges from the actual long side size of the material to the diagonal size of the material.

[0020] In the estimation of the best shooting point of a single material in S4, the actual distance of the length and width of the above-mentioned boundary box and the field of view of the depth camera color frame corresponding to the direction are used to estimate the distance from the camera to the material, and the larger distance value is selected as the height from the best shooting point of the camera to the material. The height value is combined with the grasping point at the end of the robot arm to convert it into the best shooting actual spatial point of the single material;

[0021] In the S5, the OpenCV library is used to complete the hand-eye calibration of the camera and the robot arm to obtain the transformation matrix from the camera to the end of the robot arm, and then the transformation matrix from the end of the robot arm to the base is obtained according to the posture of the robot arm. The coordinate values ​​of each point in the camera coordinate system are transformed to the robot arm base coordinate system through the above two transformation matrices, and then the inverse kinematics model of the robot arm is solved to obtain the joint coordinates of the robot arm corresponding to each point.

[0022] The control strategy of the end effector of the manipulator implemented by serial communication, Arduino UNO development board, Arduino Mega 2560 development board and host computer in S6 is as follows: the host computer reads the depth image and color image aligned by the depth camera, and obtains the posture and size of the grasped material through the S2, S3 and S4 algorithms, and sends the posture of the material to the Arduino Mega 2560 development board through the serial port to control the end of the manipulator to move to the center point of the material, and then sends a start grasping instruction to the Arduino UNO development board to turn on the end suction cup actuator and the vacuum negative pressure sensor, and the manipulator drives the suction cup to press down. If the suction cup successfully absorbs the material, the vacuum negative pressure sensor will give a high-level signal and return it to the host computer. After receiving the high-level signal, the host computer sends an instruction to the Arduino Mega 2560 to control the manipulator to move to the initial position;

[0023] In the S6, materials are graded and sorted according to their sizes: a 35mm diameter suction cup and two 60mm diameter suction cups are installed at the end of the robot arm in a triangular shape. For materials with a size greater than or equal to 300mm, the upper computer will control the robot arm to use three suction cups to suck at the same time, and then put the materials into the large cargo box; for materials with a size greater than or equal to 150mm and less than 300mm, the upper computer will control the robot arm to use a 60mm diameter suction cup to suck, and then put the materials into the medium cargo box; for materials with a size less than 150mm, the upper computer will control the robot arm to use a 35mm diameter suction cup to suck, and then put the materials into the small cargo box, thereby achieving stable grasping and sorting of materials.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention breaks the limitation of grabbing a single fixed target in a single scene. By combining the deep learning target recognition network, it can be generalized to the grabbing of any type of object, can adapt to the changing and complex practical application scenes, and has wide adaptability.

[0026] The present invention integrates tasks such as disorderly stacked material identification and material sorting, which are automatically completed by a host computer and a robotic arm, greatly improving the material sorting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the architecture of the disorderly stacked material sorting system in the present invention.

[0028] Figure 2 This is a robot grasping scene diagram in a specific embodiment of the present invention.

[0029] Figure 3 This is a structural diagram of the suction cup of the robot arm actuator in a specific embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the deep learning network inference results in a specific embodiment of the present invention.

[0031] Figure 5 This is a diagram of the estimated grasping posture of the robotic arm in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0033] See also Figure 1 As an embodiment of the present invention, this embodiment provides a method for sorting disordered stacked materials based on deep learning and point cloud for express sorting needs in the material industry, and the method includes:

[0034] S1. Obtain the material data set required for training, preprocess the data set and complete data labeling;

[0035] The data sample of this embodiment is a dataset of unordered stacked express images. 640×480 resolution RGB images of unordered express are collected at different express delivery points. Data enhancement processing is performed on the collected images. Random flipping, rotation and other methods are used for processing. The LabelImg software is used to annotate the complete express on the surface and unobstructed in the RGB images of unordered express with rectangular labels. The label saving format is selected in YOLO format. The sample size after labeling is about 1700;

[0036] S2. Use the YOLOv5 target detection network to train an algorithm model for identifying disordered stacked materials;

[0037] In S2, the YOLOv5 target detection network is used for model training. The samples are divided into training set, validation set and test set in a ratio of 3:1:1. About 1000 samples are used for model training, about 340 samples are used for model validation, and about 340 samples are used for model testing. After the model training is completed, the detect script of YOLOv5 is rewritten into the DetectAPI class for externally calling the reasoning API of YOLOv5.

[0038] S3, constructing a 3D point cloud of disordered stacked materials based on the target recognition results of YOLOv5 and the images collected by the depth camera;

[0039] The trained YOLOv5s model is deployed to the host computer, and the RealSense D435 depth camera is used to collect the aligned color image and depth image of the unordered express to be sorted; the color image is input into the YOLOv5s model for reasoning to obtain the 2D express recognition result, such as Figure 4The depth image is converted into an 8-bit grayscale image through absolute value scaling. The larger the grayscale value, the farther the target is from the camera. The scaling factor represents the effective shooting distance of the required depth camera. The depth image is intercepted using the bounding box with the highest confidence in the inference result, and the intercepted depth image is converted into a three-dimensional point cloud in the camera coordinate system according to the camera pinhole imaging model. The position between each three-point in the camera coordinate system represents the relative position in the actual space.

[0040] S4. Select the actual spatial coordinates of the center point of the bounding box identified by YOLOv5 as the end point of the robot grasping, and its coordinates in the robot base coordinate system as the end position of the robot grasping. Use the improved random consistency sampling method to perform multi-plane estimation on the three-dimensional point cloud of disordered stacked materials, and select the plane normal vector with the largest number of points in the estimated plane and containing the end grasping point as the end posture of the robot grasping; estimate the lateral size of the material based on the point cloud data, and classify the size according to the estimated size of the material; combine the estimated size of the material and the field of view of the depth camera color frame to estimate the best shooting point of a single material;

[0041] Random consistency sampling algorithm randomly selects the plane where three non-collinear points are located in the 3D point cloud as the estimated plane model, calculates the normal vector of the estimated plane and a point on the plane, calculates the distance from all other points to the estimated plane, and the points with distances less than the threshold are regarded as the inliers of the plane model. Repeat the above steps for multiple iterations, and select the plane model with the largest number of inliers in the iteration as the best estimated plane model of the algorithm;

[0042] Improve the above algorithm. After each random consistency sampling is performed to obtain the best estimated plane, all the inner points of the estimated plane are removed from the three-dimensional point cloud. Random consistency sampling is performed again. The above steps are repeated until the number of points in the three-dimensional point cloud is less than 3. The multi-plane estimation of the three-dimensional point cloud is completed. The plane normal vector whose inner point contains the end point of the robot arm is not grasped is selected from all the fitting planes as the end posture orientation of the robot arm grasping the target, such as Figure 5 ;

[0043] After the end-of-arm grasping pose estimation is completed, the size and optimal shooting point of the grasped express are estimated, and the length and width of the YOLO inference bounding box are converted to the camera coordinate system. At this time, the length and width represent the actual size of the identified express bounding box. The larger of the two is selected as the estimated value of the express size, and the subsequent express grasping control and sorting are performed based on this estimated value. The range of this size is from the actual long side size of the express to the diagonal size of the express; the actual distance of the length and width of the above-mentioned bounding box and the field of view of the depth camera color frame corresponding to this direction are used to estimate the distance from the camera to the express, and the larger distance is selected as the height from the camera's optimal shooting point to the express, and the height value is combined with the grasping point at the end of the robot arm to convert it into the optimal actual spatial point for shooting a single express;

[0044] S5. Convert each point in the camera coordinate system to the robot base coordinate system according to the coordinate transformation relationship to obtain the joint coordinates of the robot corresponding to each point;

[0045] Use the printed calibration plate and OpenCV library to complete the hand-eye calibration of the camera and the robot's eye on the hand to obtain the transformation matrix from the camera to the end of the robot. According to the posture of the robot, the transformation matrix from the end of the robot to the base is obtained. The coordinate values ​​of each point in the camera coordinate system are transformed to the robot base coordinate system through the above two transformation matrices, and then the inverse kinematics model of the robot is used to solve the joint coordinates of the robot corresponding to each point.

[0046] S6. Use serial communication, Arduino UNO development board, Arduino Mega 2560 development board and host computer to control the end effector of the robot arm, such as Figure 2 .

[0047] The host computer collects the color image and depth image aligned by the depth camera, and uses the trained YOLOv5s and pose estimation algorithm to calculate the pose and size of the express. The pose is sent to the Arduino Mega2560 development board through serial communication to control the end of the robotic arm to move to the center of the express. Then, a start grabbing command is sent to the Arduino UNO development board to turn on the end suction cup actuator and vacuum negative pressure sensor. The robotic arm drives the suction cup to press down. If the suction cup successfully absorbs the express, the vacuum negative pressure sensor will give a high-level signal and return it to the host computer. After receiving the signal, the host computer sends a command to the ArduinoMega 2560 development board to control the robotic arm to move to the initial position.

[0048] The end of the robot arm is triangular and has three suction cups with diameters of 35mm, 60mm, and 60mm respectively (such as Figure 3), where the 35mm suction cup is used as the main control suction cup, that is, the end effector point of the robot arm. For express with an estimated size less than 150mm, the host computer will control the robot arm to use a 35mm diameter suction cup to absorb and then place it in a small cargo box; for express with a size greater than or equal to 150mm and less than 300mm, the host computer will control the robot arm to use a 60mm diameter suction cup to absorb and then place it in a medium cargo box; for express with a size greater than or equal to 300mm, the host computer will control the robot arm to use three suction cups to absorb at the same time and then place it in a large cargo box. The express is absorbed by size classification to achieve stable sorting of the express.

[0049] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0050] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for sorting disordered stacked materials based on deep learning and point cloud, characterized in that: The method comprises the following steps: S1. Obtain the material data set required for training, preprocess the data set and complete data labeling; S2. Use the YOLOv5 target detection network to train an algorithm model for identifying disordered stacked materials; S3, constructing a 3D point cloud of scattered and disordered stacked materials based on the target recognition results of YOLOv5 and the images collected by the depth camera; S4. Select the actual spatial coordinates of the center point of the bounding box identified by YOLOv5 as the end point of the robot grasping, use the improved random consistency sampling algorithm to perform multi-plane estimation on the three-dimensional point cloud of disordered stacked materials, and select the plane normal vector with the largest number of points in the estimated plane and containing the end grasping point as the end posture of the robot grasping; estimate the lateral size of the material based on the point cloud data, and classify the material into large, medium and small according to the estimated size; combine the estimated size and the field of view of the depth camera color frame to estimate the best shooting point for a single material; S5. Convert each point in the camera coordinate system to the robot base coordinate system according to the coordinate transformation relationship to obtain the joint coordinates of the robot corresponding to each point; S6. Use serial communication, Arduino UNO development board, Arduino Mega 2560 development board and host computer to control the end effector of the robot arm; The random consistency sampling algorithm randomly selects three planes containing non-collinear points in the three-dimensional point cloud as the estimated plane model, calculates the normal vector of the estimated plane and a point on the plane, calculates the distances of all other points to the estimated plane, and takes the points with distances less than a threshold as the inliers of the plane model. The above steps are repeated for multiple iterations, and the plane model with the largest number of inliers is selected as the best estimated plane model of the algorithm in the iteration. The random consistency sampling algorithm is improved. After each random consistency sampling is performed to obtain the best estimated plane, all the inner points of the estimated plane are removed from the three-dimensional point cloud, and random consistency sampling is performed again. The above steps are repeated until the number of points in the three-dimensional point cloud is less than 3, and the multi-plane estimation of the three-dimensional point cloud is completed. The plane normal vector whose inner point contains the end point of the robot arm not grasped is selected from all fitting planes as the end posture orientation of the robot arm grasping the material.

2. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: The data set in S1 collects multiple groups of color images of disordered stacked materials, converts the RGB images to a resolution of 640×480, and uses rotation and flipping for data enhancement. The LabelImg software is used to annotate the complete materials on the surface and not obstructed in the RGB images of disordered stacked materials with rectangular labels, and the label saving format is selected in YOLO format.

3. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: In S2, the YOLOv5 target detection network is used for model training. The samples are divided into a training set, a validation set and a test machine in a ratio of 3:1:

1. The training set is used for model training, the validation set is used for model validation, and the test machine is used for model testing. After the model training is completed, the detect script of YOLOv5 is rewritten into the DetectAPI class for externally calling the reasoning API of YOLOv5.

4. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: In the S3, a depth camera is used to collect the aligned color image and depth image of the disorderly stacked materials to be sorted, wherein the color image is used for YOLOv5 reasoning, and the depth image is converted into an 8-bit grayscale image by absolute value scaling for constructing a three-dimensional point cloud. The larger the grayscale value, the farther the material is from the camera. The scale factor of the scaling represents the effective shooting distance of the required depth camera. The trained YOLOv5 model is used to reason the color image, and the material with the highest confidence in the reasoning result is selected as the current grasping target of the robot arm. The grayscale image is intercepted according to the identified bounding box, and the intercepted area is the grayscale / depth image corresponding to the area identified by YOLOv5. According to the camera imaging model, each two-dimensional pixel point and the corresponding depth value in the identified bounding box are converted into the camera coordinate system. The position between each three-dimensional point in the camera coordinate system represents the actual spatial position, and the construction of the three-dimensional point cloud is completed.

5. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: In S4, the size estimation of the material is to convert the length and width of the YOLOv5 recognition bounding box into a distance in the actual space, and select the larger distance value as the estimated size of the material. The estimated size ranges from the actual long side size to the diagonal size of the material; In the estimation of the best shooting point for a single material in S4, the actual distance of the length and width of the bounding box and the field of view of the depth camera color frame corresponding to the direction are used to estimate the distance from the camera to the material, and the larger distance value is selected as the height from the best shooting point of the camera to the material. The height value is combined with the grasping point at the end of the robot arm to convert it into the best actual spatial point for shooting a single material.

6. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: In the S5, the OpenCV library is used to complete the hand-eye calibration of the camera and the robotic arm to obtain the transformation matrix from the camera to the end of the robotic arm, and then the transformation matrix from the end of the robotic arm to the base is obtained according to the posture of the robotic arm. The coordinate values ​​of each point in the camera coordinate system are transformed to the robotic arm base coordinate system through the transformation matrix, and then the inverse kinematics model of the robotic arm is solved to obtain the joint coordinates of the robotic arm corresponding to each point.

7. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 1, characterized in that: In the S6, the control strategy of the end effector of the robotic arm is realized by serial communication, Arduino UNO development board, Arduino Mega 2560 development board and host computer: the host computer reads the depth image and color image aligned by the depth camera, and obtains the posture and size of the grabbed disorderly stacked materials through the S2, S3 and S4 algorithms, and sends the posture of the material to the Arduino Mega 2560 development board through the serial port to control the end of the robotic arm to move to the center point of the material, and then sends a start grasping instruction to the Arduino UNO development board to turn on the end suction cup actuator and the vacuum negative pressure sensor, and the robotic arm drives the suction cup to press down. If the suction cup successfully absorbs the material, the vacuum negative pressure sensor will give a high-level signal and return it to the host computer. After receiving the high-level signal, the host computer sends an instruction to the Arduino Mega 2560 to control the robotic arm to move to the initial position.

8. The method for sorting disordered stacked materials based on deep learning and point cloud according to claim 7, characterized in that: In the S6, the materials are graded and sorted according to their sizes: a suction cup with a diameter of 35 mm and two suction cups with a diameter of 60 mm are installed at the end of the robot arm in a triangular shape. For materials with a size greater than or equal to 300 mm, the upper computer will control the robot arm to use three suction cups to suck at the same time, and then put the materials into the large cargo box; for materials with a size greater than or equal to 150 mm and less than 300 mm, the upper computer will control the robot arm to use a suction cup with a diameter of 60 mm to suck, and then put the materials into the medium cargo box; for materials with a size less than 150 mm, the upper computer will control the robot arm to use a suction cup with a diameter of 35 mm to suck, and then put the materials into the small cargo box.

Images