A Fruit Grading and Sorting Method Based on Deep Learning and Unordered Grasping

Abstract

The present invention discloses a fruit grading and sorting method based on deep learning and unordered grasping, comprising the following steps: establishing a data set and a deep learning model of target fruits; training the deep learning model; determining the grade of target fruits through the deep learning model; collecting target fruit images through a binocular camera to obtain depth images; establishing a point cloud model of target fruits according to the depth images; determining the pose information and size of target fruits according to the depth images and the point cloud model; determining a manipulator grasping strategy by using a collision avoidance algorithm according to the pose information; determining the opening and closing size of the end gripper of the manipulator according to the size of the target fruits; the grasping strategy determines the optimal scheme for grasping the target fruits; according to the optimal scheme and the opening and closing size, driving the manipulator to grasp the target fruits and placing them at designated positions according to corresponding grades. The beneficial effects of the present invention are: realizing unordered grasping of target fruits and achieving high-precision grading and sorting.

Description

Technical Field

[0001] The present invention relates to the field of image target recognition, and particularly to a fruit grading and sorting method based on deep learning and unordered grasping. Background Art

[0002] Currently, for the grading and sorting of fruits, most of them are manually graded and sorted after picking. However, it is difficult for the human eye to distinguish mildew and spots on the surface of fruits, resulting in problems such as grading omissions; in addition, fruits need to be classified based on their own size, color, shape and other characteristics. As the number of characteristics increases, the workload of people also increases, making it easier for workers to get tired during the grading and sorting work, and thus resulting in grading errors. In recent years, traditional machine vision has been used for grading and sorting, but manual feature extraction is required, resulting in low efficiency and low recognition accuracy. Therefore, it is particularly important to provide a reliable, efficient and stable grading and sorting system. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a fruit grading and sorting method based on deep learning and unordered grasping, including the following steps:

[0004] S1. Establish a data set of target fruits; construct a deep learning model;

[0005] S2. Train the deep learning model according to the data set to obtain a trained deep learning model;

[0006] S3. Input the target fruit image into the trained deep learning model, and determine the grade of the target fruit through the deep learning model;

[0007] S4. Collect the target fruit image through a binocular camera and perform three-dimensional reconstruction to obtain a depth image; establish a point cloud model of the target fruit according to the depth image;

[0008] S5. Determine the pose information and size of the target fruit according to the depth image and the point cloud model;

[0009] S6. Determine the mechanical arm grasping strategy by using a collision avoidance algorithm through the pose information;

[0010] S7. Determine the opening and closing size of the end gripper of the mechanical arm according to the size of the target fruit;

[0011] S8. The grasping strategy determines the optimal solution for grasping the target fruit;

[0012] S9. According to the optimal solution and the opening and closing size, drive the mechanical arm to grasp the target fruit and place it at a specified position according to the corresponding grade.

[0013] Further, step S1 is specifically as follows:

[0014] S11: Classify the target fruits according to the grade classification standard;

[0015] S12: Obtain the data of fruits at different grades according to the grade of the target fruit;

[0016] S13: Collect the RGB images of each grade of the target fruit that the robotic arm needs to grasp;

[0017] S14: Label the fruit categories in the RGB images to generate a dataset of the target fruit.

[0018] Further, in step S2, the deep learning model uses the YOLOV3-SPP model.

[0019] Further, step S4 is specifically:

[0020] S41: Complete the calibration of the binocular camera using the Zhang Zhengyou calibration method;

[0021] S42: Collect the target fruit images using the binocular camera;

[0022] S43: Use the three-dimensional reconstruction method in the image processing database to obtain the depth image of the target fruit;

[0023] S44: Establish a point cloud model of the target fruit based on the depth image of the target fruit.

[0024] Further, step S5 is specifically:

[0025] S51: Determine the height information of the target fruit based on the depth image and the point cloud model;

[0026] S52: Preprocess the point cloud model by performing image cropping and noise removal operations on the point cloud model through the software CloudCompare to obtain a preprocessed point cloud model;

[0027] S54: Perform three-dimensional editing, cleaning, and stitching on the preprocessed point cloud model through the software MeshLab in sequence to obtain a final point cloud model.

[0028] S55: Calculate the size information of the target fruit based on the final point cloud model.

[0029] Further, step S6 is specifically:

[0030] S61. Determine the grasping strategy through the pose information;

[0031] S62. Determine the grasping order from high to low based on the obtained height information; for those with larger height information, grasp them first;

[0032] S63. Calculate the approaching target fruit grasping trajectory and the manipulator withdrawal trajectory according to the collision avoidance algorithm;

[0033] S64. Adopt a grasping scheme with multiple grasping poses corresponding to one target fruit pose;

[0034] S65. Determine the overall grasping strategy according to the grasping sequence, the target fruit grasping trajectory and the manipulator withdrawal trajectory, and the grasping scheme.

[0035] Further, step S9 is specifically as follows:

[0036] S91. Determine the motion trajectory of the manipulator according to the optimal scheme;

[0037] S92. Drive the manipulator to achieve precise positioning of the target fruit by using the five-point positioning method;

[0038] S93. Grasp the target fruit according to the motion trajectory and the opening and closing size, and place the target fruit at the specified position according to the identified grade.

[0039] The beneficial effects provided by the present invention are: realizing the unordered grasping of target fruits by combining a deep learning model, and realizing high-precision grading and sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0042] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of the method of the present invention; a fruit grading and sorting method based on deep learning and unordered grasping, including the following:

[0043] S1. Establish a data set of target fruits; construct a deep learning model;

[0044] It should be noted that step S1 is specifically as follows:

[0045] S11: Distinguish the target fruits according to the grade classification standard;

[0046] In the embodiments of the present invention, mainly taking citrus as an example, refer to Table 1:

[0047] Table 1 Citrus grade classification

[0048]

[0049] S12: Obtain fruit data of each different grade according to the target fruit grade;

[0050] It should be noted that the fruit data here refers to fruit images.

[0051] S13: Collect RGB images of each grade of the target fruit that the robotic arm needs to grasp; it should be noted that among the RGB images of each grade of the target fruit, there may be some other fruits other than the target fruit.

[0052] S14: Label the fruit categories in the RGB images to generate a data set of the target fruit.

[0053] S2. Train a deep learning model according to the data set to obtain a trained deep learning model;

[0054] It should be noted that in step S2, the deep learning model uses the YOLOV3-SPP model. Of course, this is not used as a limitation here, and other technicians in the field can also use other deep learning models.

[0055] S3. Input the target fruit image into the trained deep learning model, and determine the target fruit grade of the target through the deep learning model;

[0056] It should be noted that regarding the training process of the deep learning model, it is a conventional training process and will not be elaborated in detail in this application.

[0057] S4. Collect the target fruit image through a binocular camera and perform three-dimensional reconstruction to obtain a depth image; establish a point cloud model of the target fruit according to the depth image;

[0058] It should be noted that step S4 is specifically:

[0059] S41: Complete the calibration of the binocular camera using the Zhang Zhengyou calibration method;

[0060] It should be noted that the Zhang Zhengyou calibration method is a relatively conventional existing calibration method, and it will not be elaborated in detail and the principle derivation in this application; of course, this is not used as a limitation here either; other technicians in the field can also use other calibration methods;

[0061] S42: Collect the target fruit image using the binocular camera;

[0062] It should be noted that using the binocular camera to collect the target fruit image is mainly to obtain depth information;

[0063] S43: Use the three-dimensional reconstruction method in the image processing database to obtain the depth image of the target fruit;

[0064] It should be noted that the image processing database can use existing open-source databases, such as OPENCV image processing or industrial HALCON image processing software; for the method of three-dimensional reconstruction, the corresponding function package in the image processing database can be used for processing.

[0065] S44: Establish a point cloud model of the target fruit based on the depth image of the target fruit.

[0066] It should be noted that establishing a point cloud model of the target fruit based on the depth image can still be processed using an image processing library.

[0067] S5. Determine the pose information and size of the target fruit based on the depth image and the point cloud model;

[0068] It should be noted that step S5 is specifically as follows:

[0069] S51: Determine the height information of the target fruit based on the depth image and the point cloud model;

[0070] It should be noted that the fruit height information can be obtained by inputting the depth image and the point cloud model into 3D processing software;

[0071] S52: Preprocess the point cloud model by performing image cropping and noise removal operations on the point cloud model through the software CloudCompare to obtain a preprocessed point cloud model;

[0072] S54: Perform 3D editing, cleaning, and stitching on the preprocessed point cloud model through the software MeshLab in sequence to obtain a final point cloud model.

[0073] S55: Calculate the size information of the target fruit based on the final point cloud model.

[0074] It should be noted that calculating the size information of the target fruit based on the point cloud model can be directly exported or corresponding operations can be performed through the software MeshLab.

[0075] S6. Determine the manipulator grasping strategy using a collision avoidance algorithm based on the pose information;

[0076] It should be noted that step S6 is specifically as follows:

[0077] S61. Determine the grasping strategy based on the pose information;

[0078] S62. Determine the grasping order from high to low based on the obtained height information; for those with larger height information, grasp first;

[0079] S63. Calculate the grasping trajectory approaching the target fruit and the manipulator withdrawal trajectory according to the collision avoidance algorithm;

[0080] S64. Adopt a grasping scheme where one target fruit pose corresponds to multiple grasping poses;

[0081] It should be noted that for one target fruit pose, there can be multiple robotic arm grasping schemes. When considering the grasping scheme alone, it does not involve specific planning of the grasping sequence and the grasping trajectory;

[0082] S65. Determine the overall grasping strategy according to the grasping sequence, the target fruit grasping trajectory, the robotic arm withdrawal trajectory, and the grasping scheme.

[0083] The following is an example: First, determine the grasping sequence, with the target fruit having the highest height information being grasped first. Under the condition of determining the grasping sequence, generate multiple grasping trajectories; consider the grasping efficiency and grasping time of multiple grasping trajectories, and finally determine the grasping scheme;

[0084] S7. Determine the opening and closing size of the end gripper of the robotic arm according to the size of the target fruit;

[0085] It should be noted that the opening and closing size of the end gripper is slightly larger than the size of the target fruit when approaching the target fruit;

[0086] S8. The grasping strategy determines the optimal scheme for grasping the target fruit;

[0087] S9. According to the optimal scheme and the opening and closing size, drive the robotic arm to grasp the target fruit and place it at the specified position according to the corresponding grade.

[0088] Step S9 is specifically as follows:

[0089] S91. Determine the motion trajectory of the robotic arm according to the optimal scheme;

[0090] S92. Adopt the five - point positioning method to drive the robotic arm to achieve precise positioning of the target fruit;

[0091] S93. According to the motion trajectory and the opening and closing size, grasp the target fruit and place the target fruit at the specified position according to the recognized grade.

[0092] Finally, it should be noted that hand - eye calibration is also required between the robotic arm and the binocular camera to finally complete the corresponding grasping and cooperation. The hand - eye calibration process is a conventional technical means and will not be elaborated further here.

[0093] The beneficial effects of the present invention are: By combining a deep - learning model, unordered grasping of target fruits is achieved, and high - precision grading and sorting are realized.

[0094] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A fruit grading and sorting method based on deep learning and disordered grasping, characterized in that: It includes the following steps: S1. Establish a dataset of target fruits; construct a deep learning model; The specific steps of S1 are as follows: S11: Classify the target fruits according to the grade classification standard; The fruit grades include four levels, which are distinguished by transverse diameter, color and fruit surface. Among them: Grade I: Transverse diameter The coloring rate of the color is greater than or equal to 25%, and the fruit surface is clean with a large number of scars; Grade Ⅱ: Transverse diameter The coloring rate of the color and luster is greater than or equal to 55%, and the fruit surface is clean with a small number of scars; Grade Ⅲ: Transverse diameter The coloring rate of the color is greater than or equal to 75%, and the fruit surface is clean with slight scars; Grade Ⅳ: Transverse diameter Greater than or equal to 80, the coloring rate of the color is greater than or equal to 90%, and the fruit surface is clean with very few scars; The unit of the transverse diameter is mm; S12: Obtain fruit data of each different grade according to the target fruit grade; S13: Collect RGB images of each grade of the target fruits that the robotic arm needs to grasp; S14: Label the fruit categories in the RGB images to generate a dataset of target fruits; S2. Train the deep learning model according to the dataset to obtain a trained deep learning model; S3. Input the target fruit image into the trained deep learning model, and determine the grade of the target fruit through the deep learning model; S4. Collect the target fruit image through a binocular camera and perform three-dimensional reconstruction to obtain a depth image; establish a point cloud model of the target fruit according to the depth image; S5. Determine the pose information and size of the target fruit according to the depth image and the point cloud model; The specific steps of S5 are: S51: Determine the height information of the target fruit according to the depth image and the point cloud model; S52: Preprocess the point cloud model by performing image cropping and noise removal operations on the point cloud model through the software CloudCompare to obtain a preprocessed point cloud model; S54: Perform three-dimensional editing, cleaning and stitching on the preprocessed point cloud model through the software MeshLab in sequence to obtain a final point cloud model; S55: Calculate the size information of the target fruit according to the final point cloud model; S6. Determine the grasping strategy of the robotic arm by using the collision avoidance algorithm according to the pose information; The specific steps of S6 are: S61: Determine the grasping strategy through the pose information; S62: Determine the grasping order from high to low according to the obtained height information; for those with larger height information, grasp them first; S63: Calculate the grasping trajectory approaching the target fruit and the withdrawal trajectory of the robotic arm according to the collision avoidance algorithm; S64: Adopt a grasping scheme with multiple grasping poses corresponding to one target fruit pose; S65: Determine the overall grasping strategy according to the grasping order, the grasping trajectory and the withdrawal trajectory of the target fruit, and the grasping scheme; S7. Determine the opening and closing size of the end gripper of the robotic arm according to the size of the target fruit; S8. The grasping strategy determines the optimal scheme for grasping the target fruit; S9. Drive the robotic arm to grasp the target fruit and place it at the specified position according to the corresponding grade according to the optimal scheme and the opening and closing size.

2. The fruit grading and sorting method based on deep learning and disordered grasping according to claim 1, characterized in that: In step S2, the YOLOV3-SPP model is adopted for the deep learning model.

3. A fruit grading and sorting method based on deep learning and unordered grasping according to claim 1, characterized in that: The specific steps of S4 are: S41: Complete the calibration of the binocular camera by using the Zhang Zhengyou calibration method; S42: Collect the target fruit image by using the binocular camera; S43: Obtain the depth image of the target fruit by using the three-dimensional reconstruction method in the image processing database; S44: Establish a point cloud model of the target fruit according to the depth image of the target fruit.

4. The fruit grading and sorting method based on deep learning and disordered grasping according to claim 1, wherein: The specific steps of S9 are: S91: Determine the motion trajectory of the robotic arm according to the optimal scheme; S92. Use the five-point positioning method to drive the robotic arm to achieve precise positioning of the target fruit; S93. According to the movement trajectory and the opening and closing size, grasp the target fruit and place the target fruit at the designated position according to the identified grade.

Images