An apple grading method based on binocular vision

Abstract

The invention discloses an apple grading method based on binocular vision, comprising the following steps: S1: using a binocular camera to acquire an apple image; S2: dividing the apple image acquired by the binocular camera into a left-eye image and a right-eye image; S3: inputting the left-eye image into a target recognition convolutional neural network to recognize all apples in the left-eye image; S4: extracting the apple image through a convolutional neural network recognition frame; S5: calculating the three-dimensional position coordinates of the apple from the center point of the recognition frame through a binocular stereo matching algorithm; S6: preprocessing the extracted apple image to remove useless image parts; S7: grading the preprocessed image according to the diameter size and the fruit shape coefficient; S8: sending the position of the apple and the corresponding grade to a grasping device to end the grading.

Description

Technical Field

[0001] The invention relates to an apple grading method based on binocular vision and belongs to the field of image recognition equipment. Background Art

[0002] At present, fruit grading in my country is mainly done by machines. Traditional apple grading is mainly mechanical. One method is to use a tiltable weighing tray to weigh the apples, and the apples are tilted by the tray to roll into the corresponding level frame according to their weight. The other method is to use funnels of different sizes to let the apples pour from top to bottom, and the apples are sorted into different levels of frames from different funnels according to their sizes. In the above two mechanical grading methods, collisions are inevitable during the apple grading process, causing damage to the apple skin. The present invention develops an apple grading method based on binocular vision. Summary of the invention

[0003] In view of the above shortcomings, the present invention provides an apple grading method based on binocular vision.

[0004] The present invention is achieved through the following technical solutions:

[0005] An apple grading method based on binocular vision comprises the following steps:

[0006] S1: Use a binocular camera to obtain apple images;

[0007] S2: Segment the apple image obtained by the binocular camera into a left image and a right image;

[0008] S3: Input the left image into the object recognition convolutional neural network to identify all the apples in the left image;

[0009] S4: Extracting apple images through convolutional neural network recognition boxes;

[0010] S5: Calculate the three-dimensional position coordinates of the apple using the binocular stereo matching algorithm to obtain the center point of the recognition frame;

[0011] S6: preprocessing the extracted apple image to remove useless image parts;

[0012] S7: classifying the preprocessed images according to diameter size and fruit shape coefficient;

[0013] S8: Send the position of the apple and the corresponding grade to the grasping device to complete the grading.

[0014] Furthermore, the left-eye image is input into the target recognition convolutional neural network to identify all the apples in the left-eye image. The specific steps are: input the left-eye image into a pre-trained Mobile-SSD convolutional neural network to identify the area where all the apples contained in the left-eye image are located.

[0015] Furthermore, the specific steps of extracting the apple image by using the convolutional neural network recognition frame are: using the Mobile-SSD target to detect the key area of ​​the neural network, and obtaining the key area image by cropping;

[0016] The key area image is expanded outward by three pixels at an equal distance as the apple image required for grading.

[0017] The key area is the area where the apple is located in the left-eye image.

[0018] Furthermore, the three-dimensional position coordinates of the apple are obtained by calculating the center point of the recognition frame through a binocular stereo matching algorithm as described in S5, and the specific steps are: subtracting the pixel coordinates of the lower left corner and the upper right corner of the convolutional neural network recognition frame to obtain the center pixel coordinates of the recognition frame coordinates;

[0019] The pixel coordinates of the center point are obtained through the binocular stereo matching algorithm to obtain the actual three-dimensional coordinates of the center point as the position coordinates of the apple.

[0020] Furthermore, the extracted apple image is preprocessed in S6 to remove useless image parts, and the specific process is as follows:

[0021] S601: filtering the extracted apple image;

[0022] S602: Binarizing the filtered image;

[0023] S603: performing morphological processing on the binarized image;

[0024] S604: extracting all contours in the binary image after morphological processing;

[0025] S605: Delete the redundant contours outside the apple contour and keep the useful contours.

[0026] Furthermore, the specific steps of filtering the apple image are: convolving the original image through a two-dimensional convolution with a convolution kernel size of 3x3; performing mean filtering on the image after convolution processing; and extracting the R channel image of the image after mean filtering.

[0027] Among them, the R channel refers to the red channel of the image in the RGB color space.

[0028] Furthermore, the morphological processing of the binarized image is mainly performed by reducing the white outline of the binarized image by 3 pixels inward through corrosion processing.

[0029] Furthermore, the redundant contours outside the apple contour are deleted, and the specific steps are: detecting all contours in the image after morphological processing; and selecting the largest contour among the contours for retention.

[0030] The purpose of morphological processing is to separate the apple images that are touching each other.

[0031] Further, in step S7, the pre-processed images are classified according to the diameter and the fruit shape coefficient, and the specific process is as follows:

[0032] S701: Determine the diameter of the apple by using the minimum circumscribed circle;

[0033] S702: Obtain the vertical diameter of the apple through a binocular camera;

[0034] S703: calculating the shape coefficient of the apple according to the diameter and longitudinal diameter of the apple;

[0035] S704: apples are graded according to their diameter and shape coefficient.

[0036] Furthermore, the specific process of determining the apple diameter by using the minimum circumscribed circle is as follows: obtaining the pixel diameter of the minimum circumscribed circle and substituting it into the fitting curve to obtain the actual diameter as the apple diameter.

[0037] The fitting curve is the functional relationship between the pixel diameter and the actual diameter in the image, which is obtained through calibration;

[0038] The specific calibration process is to obtain the functional relationship between pixel diameter and actual diameter by fitting the actual diameters of 30 apples with the corresponding pixel diameters through the least squares method.

[0039] Furthermore, the method of obtaining the longitudinal diameter of the apple by using a binocular camera comprises the following specific steps: simultaneously inputting the left-eye and right-eye images into a binocular stereo matching algorithm to obtain a depth map; subtracting the minimum depth value of the corresponding position of the apple image from the platform depth value to obtain the longitudinal diameter of the apple; and using the ratio of the apple diameter to the apple longitudinal diameter as the apple shape coefficient.

[0040] Furthermore, the grading by apple diameter and fruit shape coefficient complies with the national standard GB / T 10651-2008 "Fresh Apples" and the national and industry standards for fruit diameter and fruit shape grading corresponding to the variety of apples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The present invention is a flowchart of an apple grading method based on binocular vision.

[0042] Figure 2 This is an example diagram of an image acquired by a binocular camera provided in an embodiment of the present invention.

[0043] Figure 3 This is an example diagram of the left-eye image recognition result in an embodiment of the present invention.

[0044] Figure 4 This is a flowchart of apple image preprocessing in an embodiment of the present invention.

[0045] Figure 5 exemplifying images before and after image preprocessing in an embodiment of the present invention.

[0046] Figure 6 This is a flowchart of apple grading in an embodiment of the present invention.

[0047] Figure 7 This is an example diagram of the calculation results of the minimum circumscribed circle of an apple in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described again. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0050] Example 1

[0051] like Figure 1 As shown, an apple grading method based on binocular vision comprises the following steps:

[0052] S1: Use a binocular camera to obtain an apple image.

[0053] S2: Segment the apple image obtained by the binocular camera into a left image and a right image.

[0054] It should be noted that the image acquired by the binocular camera is segmented because the image acquired by the binocular camera is a whole image of the left and right objects (such as Figure 2 The image shown is captured by a binocular camera. The image segmentation of a binocular camera is to divide the whole image into left and right eyes from the horizontal pixel 1 / 2.

[0055] S3: Input the left image into the object recognition convolutional neural network to identify all the apples in the left image.

[0056] It should be noted that the step of inputting the left-eye image into the target recognition convolutional neural network to identify all apples in the left-eye image is as follows: inputting the left-eye image into the pre-trained Mobile-SSD convolutional neural network to identify the areas where all apples in the left-eye image are located (such as Figure 3 The left image recognition result is shown).

[0057] S4: Extracting apple images through convolutional neural network recognition boxes.

[0058] It should be noted that the specific steps of extracting the apple image through the convolutional neural network recognition frame are: using the Mobile-SSD target to detect the key area of ​​the neural network, and obtaining the key area image by cropping; expanding the key area image outward by three pixels equidistantly as the apple image required for grading.

[0059] The key area is the area where the apple is located in the left-eye image.

[0060] S5: Calculate the three-dimensional position coordinates of the apple by using the binocular stereo matching algorithm to calculate the center point of the recognition frame.

[0061] It should be noted that the three-dimensional position coordinates of the apple are calculated by the binocular stereo matching algorithm using the center point of the recognition frame. The specific steps are: subtracting the pixel coordinates of the lower left corner and the upper right corner of the convolutional neural network recognition frame to obtain the center pixel coordinates of the recognition frame coordinates; and using the binocular stereo matching algorithm to obtain the actual three-dimensional coordinates of the center point as the apple position coordinates.

[0062] S6: Preprocess the extracted apple image to remove useless image parts.

[0063] Example 2

[0064] This embodiment describes step S6 in detail based on the above steps.

[0065] like Figure 4 As shown, the specific process is:

[0066] S601: Filter the extracted apple image.

[0067] It should be noted that, in the filtering process of the apple image, the original image is first convolved by a two-dimensional convolution with a convolution kernel size of 3x3, wherein the convolution kernel is as follows:

[0068] ,

[0069] The image after convolution processing is subjected to mean filtering; and the R channel image of the image after mean filtering is extracted.

[0070] Among them, the R channel refers to the red channel of the image in the RGB color space.

[0071] S602: Binarize the filtered image.

[0072] S603: Perform morphological processing on the binarized image.

[0073] It should be noted that the morphological processing of the binarized image is mainly to reduce the white outline of the binarized image by 3 pixels inward through corrosion processing.

[0074] The purpose of morphological processing is to separate the apple images that are touching each other.

[0075] S604: Extract all contours in the binary image after morphological processing.

[0076] S605: Delete the redundant contours outside the apple contour and keep the useful contours.

[0077] It should be noted that the specific process of deleting the redundant contours outside the apple contour is as follows: detecting all contours in the image after morphological processing; selecting the largest contour among the contours for retention (such as Figure 5 Shown are images before and after image preprocessing).

[0078] S7: Classify the preprocessed images according to diameter size and fruit shape coefficient.

[0079] Example 3

[0080] This embodiment describes step S7 in detail based on the above steps.

[0081] like Figure 6 As shown, the specific process is:

[0082] S701: Determine the diameter of the apple by using the minimum circumscribed circle.

[0083] It should be noted that the specific process of determining the apple diameter by the minimum circumscribed circle is as follows: fitting the minimum circumscribed circle of the retained contour to obtain the circumscribed circle of the contour, and using the diameter of the minimum circumscribed circle as the pixel diameter (e.g. Figure 7 The figure shows the calculation result of the minimum circumscribed circle of an apple); the pixel diameter of the minimum circumscribed circle is obtained and substituted into the fitting curve to obtain the actual diameter as the diameter of the apple.

[0084] The fitting curve is the functional relationship between the pixel diameter in the image and the actual diameter, which is obtained through calibration.

[0085] The specific calibration process is to obtain the functional relationship between pixel diameter and actual diameter by fitting the actual diameters of 30 apples with the corresponding pixel diameters through the least squares method.

[0086] S702: Obtain the longitudinal diameter of the apple through a binocular camera.

[0087] It should be noted that the specific steps of obtaining the longitudinal diameter of the apple by using a binocular camera are: inputting the left and right eye images into the binocular stereo matching algorithm at the same time to obtain a depth map; and obtaining the longitudinal diameter of the apple by subtracting the minimum depth value of the corresponding position of the apple image from the platform depth value.

[0088] S703: Calculate the shape coefficient of the apple according to the diameter and longitudinal diameter of the apple.

[0089] It should be noted that the apple shape coefficient is calculated by using the apple diameter and the longitudinal diameter, and the ratio of the apple diameter to the apple longitudinal diameter is used as the apple shape coefficient.

[0090] S704: apples are graded according to their diameter and shape coefficient.

[0091] It should be noted that the apples used in this embodiment are Red Fuji apples, and the classification based on apple diameter and fruit shape coefficient follows the national standard GB / T 10651-2008 "Fresh Apples" and the industry standard NY / T1075-2006 "Red Fuji Apples", and the specific classification standards are as follows:

[0092] The diameter is greater than or equal to 70 and the fruit shape coefficient is greater than or equal to 0.9, which is special grade; the diameter is greater than or equal to 60 and less than 70, and the fruit shape coefficient is greater than or equal to 0.75 and less than 0.9, which is first grade; the diameter is greater than or equal to 60 and less than 65, and the fruit shape coefficient is less than 0.5, which is second grade; the diameter is less than 60, and the fruit shape coefficient is less than 0.5, which is outclass.

[0093] S8: Send the position of the apple and the corresponding grade to the grasping device to complete the grading.

Claims

1. An apple grading method based on binocular vision, characterized in that: The following steps are involved: S1: Use a binocular camera to obtain apple images; S2: Segment the apple image obtained by the binocular camera into a left image and a right image; S3: Input the left image into the object recognition convolutional neural network to identify all apples in the left image; S4: Extract the apple image through the convolutional neural network recognition frame; S5: Calculate the three-dimensional position coordinates of the apple by using the binocular stereo matching algorithm to obtain the center point of the recognition frame; S6: Preprocess the extracted apple image to remove useless image parts; S7: classifying the preprocessed images according to diameter size and fruit shape coefficient; The specific process is: S701: Determine the diameter of the apple by using the minimum circumscribed circle; S702: Obtain the vertical diameter of the apple through a binocular camera; Its specific steps; Input the left and right eye images into the binocular stereo matching algorithm at the same time to obtain a depth map; The vertical diameter of the apple is obtained by subtracting the minimum depth value of the corresponding position of the apple image from the platform depth value; S703: calculating the shape coefficient of the apple according to the diameter and longitudinal diameter of the apple; S704: apples are graded according to their diameter and shape coefficient; S8: Send the position of the apple and the corresponding grade to the grasping device to complete the grading.

2. The apple grading method based on binocular vision according to claim 1, characterized in that: The left-eye image is input into the target recognition convolutional neural network to identify all apples in the left-eye image. The specific steps are: input the left-eye image into the pre-trained Mobile-SSD convolutional neural network to identify the area where all apples contained in the left-eye image are located.

3. The apple grading method based on binocular vision according to claim 1, characterized in that: The specific steps of extracting the apple image through the convolutional neural network recognition frame are: using the Mobile-SSD target to detect the key area of ​​the neural network, and obtaining the key area image by cropping; The key area image is expanded outward by three pixels at equal intervals as the apple image required for grading; wherein the key area is the area where the apple is located in the left eye image.

4. The apple grading method based on binocular vision according to claim 1, characterized in that: The three-dimensional position coordinates of the apple are obtained by calculating the center point of the recognition frame through a binocular stereo matching algorithm. The specific steps are: subtracting the pixel coordinates of the lower left corner and the upper right corner of the convolutional neural network recognition frame to obtain the center pixel coordinates of the recognition frame coordinates; and obtaining the actual three-dimensional coordinates of the center point as the apple position coordinates through a binocular stereo matching algorithm for the center point pixel coordinates.

5. The method for grading apples based on binocular vision according to claim 1, characterized in that: The extracted apple image is preprocessed to remove useless image parts. The specific process is as follows: S601: filtering the extracted apple image; S602: binarizing the filtered image; S603: performing morphological processing on the binarized image; S604: extract all contours in the binary image after morphological processing; S605: delete redundant contours outside the apple contour and retain useful contours.

6. The method for grading apples based on binocular vision according to claim 5, characterized in that: The specific steps of filtering the apple image are as follows: first, the original image is convolved with a two-dimensional convolution with a convolution kernel size of 3x3, then the convolution image is mean filtered, and finally the R channel image of the mean filtered image is extracted; Among them, the R channel refers to the red channel of the image in the RGB color space.

7. The method for grading apples based on binocular vision according to claim 5, characterized in that: The specific steps of performing morphological processing on the binarized image are as follows: the white outline of the binarized image is reduced by 3 pixels inward by corrosion processing.

8. The method for grading apples based on binocular vision according to claim 5, characterized in that: The specific steps of deleting the redundant outline outside the apple outline are as follows: Detect all contours in the morphologically processed image; Filter out the largest contour among the contours and retain them; The purpose of morphological processing is to separate the apple images that are touching each other.

9. The method for grading apples based on binocular vision according to claim 8, characterized in that: The specific process of determining the apple diameter by the minimum circumscribed circle is as follows: Fit the retained contour with a minimum circumscribed circle to obtain the circumscribed circle of the contour, and use the minimum circumscribed circle diameter as the pixel diameter; Get the pixel diameter of the minimum circumscribed circle and substitute it into the fitting curve to get the true diameter as the diameter of the apple; The fitting curve is the functional relationship between the pixel diameter and the actual diameter in the image, which is obtained through calibration; The specific calibration process is to obtain the functional relationship between pixel diameter and actual diameter by fitting the actual diameters of 30 apples with the corresponding pixel diameters through the least squares method.

Images