A method and device for cherry picking and classification based on machine vision

Abstract

The present invention discloses a cherry picking and classification method based on machine vision, including: collecting images of cherries hanging on branches and including fruit stalks, filtering and removing noise as original images; performing image processing on the original images, extracting the fruit stalk features of the cherries and determining the picking point position for picking; performing image collection on the picked cherries to obtain images for classification; performing image processing on the classification images, extracting the contour features of the cherries, and calculating the size of the cherries based on the contour features; classifying the cherries according to the size data of the cherries; and also discloses a picking and classification device including: a collection module; a classification transmission module; a connection module; and a control module. The present invention designs a picking and classification method based on machine vision specifically for cherries in view of the characteristics that cherries are small spherical fruits with long fruit stalks, which can improve the efficiency of cherry picking.

Description

Technical Field

[0001] The present invention relates to the technical field of cherry picking, and in particular to a method and device for cherry picking and classification based on machine vision. Background Art

[0002] The contradiction between the rapid development of fruit and vegetable production and the shortage of agricultural labor and excessive labor intensity is becoming increasingly apparent. The replacement of the complex manual labor of selective harvesting can only be achieved through in-depth research on automated picking technology. Therefore, the research and application of automated picking technology for fruit crops is of great significance for reducing the labor intensity of agricultural practitioners, liberating agricultural labor, and improving the intensive production level of fruits and vegetables. The primary task of automated picking is to use the visual system to identify and locate the target of ripe apples. Relevant algorithms based on the field of image processing research to realize the automated detection, identification and positioning of fruits have become a research hotspot in the current research and development and application of automated fruit picking.

[0003] The "spherical fruit picking robot based on machine vision and its fruit picking method" is disclosed in Chinese patent literature. Its publication number is CN113843810A and the publication date is 2021-12-28. It includes a picking device, a collecting device, a driving device, a console and a telescopic hose. The picking device includes a robotic arm base, a base rotating motor, a robotic arm device, an end effector, an internal visual sensor, a large fruit shearing device, a small and medium-sized fruit clamp and an external visual sensor. The end effector is connected to the collecting device through a telescopic hose. The picking robot controls the picking device, the driving device and the collecting device through the console to complete the fruit picking operation. This technology can realize the efficient picking of spherical fruits with sparse growth distribution such as apples, pears, oranges, lemons, grapefruits, durian, kiwis, grapes, etc., greatly improves the cost performance of the picking robot, reduces the cost of fruit picking, and greatly improves the quality of picked fruits. However, this technology can only be used to pick spherical fruits that are sparsely distributed, including large spherical fruits, medium-sized spherical fruits with stalks, medium-sized spherical fruits without stalks, and small spherical fruits with branches. Figure 7 As shown, cherries are small spherical fruits with long stalks, and they grow in clusters. Each cluster is relatively dense and not in strings. They are not within the range of fruit types applicable to the existing automated harvesting technology. Therefore, an automated harvesting technology for cherries is needed. Summary of the invention

[0004] The present invention aims to overcome the problem that the prior art lacks automated cherry picking technology and other fruit and vegetable picking technologies are inefficient when applied to cherry picking. A cherry picking and classification method and device based on machine vision are provided. In view of the characteristics of cherries as small spherical fruits with long stalks, a picking and classification method based on machine vision specifically for cherries is designed, which can improve the cherry picking efficiency.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A cherry picking and classification method based on machine vision, comprising:

[0007] S1, collecting the image of cherries hanging on the branches including the fruit stalks and filtering and removing noise as the original image;

[0008] S2, performing image processing on the original image, extracting the characteristics of the cherry stem and determining the picking point for picking;

[0009] S3, collecting images of the picked cherries to obtain images for classification;

[0010] S4. Perform image processing on the classification image to extract contour features of the cherries, and calculate the size of the cherries based on the contour features; S5. Classify the cherries according to the size data of the cherries.

[0011] The present invention first captures an image of cherries hanging on a branch, the image including the cherry fruit and the stalk; when picking cherries, the fruit and part of the stalk are picked, and picking the fruit alone is likely to damage the fruit, so in the design of automated picking, the stalk of the cherry is sheared to complete the picking of the cherries; the present invention processes the original image to extract an image of the stalk and its vicinity, extracts the stalk feature from the image, and determines the picking point according to the stalk feature to pick the cherries; the picked cherries are imaged to obtain an image for classification, extracts the shape outline of the cherries, and uses the number of pixels contained in the shape outline of the cherries as a basis for judging the size of the cherries; the present invention can make cherry picking more convenient and efficient; at the same time, the size of the cherries is judged according to the number of pixels in the cherry image, and classification can be performed automatically, saving the cost of manual classification.

[0012] Preferably, S2 includes the following steps:

[0013] S21, extracting the grayscale of the original image using the color factor to obtain an original grayscale image;

[0014] S22, converting the original image into a HIS model image and performing segmentation based on the hue H to obtain a mask image;

[0015] S23, combining the original grayscale image and the mask image to exclude the fruit area to obtain a grayscale image of the fruit stalk and its vicinity;

[0016] S24, using the Otsu algorithm to perform binarization processing on the grayscale image in S23, and obtain the connection point position between the fruit stalk and the fruit and the growth point position between the fruit stalk and the branch;

[0017] S25, performing weighted calculation on the connection point position and the growth point position to obtain the picking point position.

[0018] In the present invention, the sizes of the original grayscale image and the mask image are the same, and the mask image is divided into a cherry fruit area and other areas; because the color of cherries is quite different from the color of surrounding green leaves, a more ideal and accurate cherry fruit area segmentation can be obtained after segmentation based on hue H through the HIS model; the original grayscale image excludes the fruit area after mask image processing, thereby obtaining a grayscale image containing only the fruit stalk and its vicinity, thereby avoiding the influence of the fruit color problem on the binarization result during the subsequent binarization processing; the connection point position and the growth point position can be found in the binary map of the fruit stalk and its vicinity, and the line connecting the two positions is approximately equal to the length of the fruit stalk, so that the picking point position on the fruit stalk can be obtained by weighted calculation according to actual needs.

[0019] Preferably, the color factor is one of the three primary color components R, G, B or one of the composite components of the three primary color components; a number of historically collected cherry images are sampled, and the component values ​​of the cherry area color and the component values ​​of the background area color are calculated with the same color factor to form a scatter plot, and the color factor that minimizes the intersection of the component values ​​of the two area colors in the scatter plot is selected to perform grayscale extraction on the original image.

[0020] In the present invention, the color factors include R component, G component, B component, |GB| component, |GR| component and |GB|+|GR| component, etc., three primary color components and different collective components of the three primary color components; when the component values ​​of the cherry area color and the component values ​​of the background area color calculated by a certain color factor have the least intersection of the component values ​​in the scatter diagram, it means that the grayscale color difference between the cherry area and the background area in the grayscale image obtained by grayscale extraction using the color factor is the largest, and subsequent image processing is easier.

[0021] Preferably, the contour tracing algorithm is used to obtain the contours of the fruit stalk and the trunk for the binary image in S24; after the binary image and the mask image are overlapped, the contours of the fruit stalk and the trunk and the boundary of the fruit region on the mask image intersect at points A1 and B1, and the midpoint of the line connecting point A1 and point B1 is the connection point position;

[0022] In the process of moving from point A1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point A i So that α = S CAi / S Ai is greater than the set threshold, and in the process of moving from point B1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point B j So that α = S CBj / S Bj If it is greater than the set threshold, then point A i and point B j The midpoint of the connecting line is the location of the growth point;

[0023] S Ai and S Bj Circle a i and circle b j The number of pixels contained in circle a i and circle b j Point A i and point B j A circle with center S and radius S. CAi and S CBj are the number of pixels within the two circles and the outlines of the fruit stalk and branches.

[0024] In the present invention, after binarization processing, the fruit stalk and the branch outline are white or black, and the other background colors are black or white. The fruit stalk and the branch outline are white for illustration, one end of the fruit stalk is connected to the branch at the growth point, and the other end has a connection point with the fruit. Since the fruit stalk has a certain width, the connection point where the fruit stalk outline intersects with the boundary of the fruit area is a curve, and the two end points of the curve are selected as point A1 and point B1, and the focus of the line connecting the two points is used as the connection point; a circle is obtained by taking any point on the outline where the fruit stalk area is located as the center and the line segment |A1B1| as the radius, and the area of ​​the white part included in this circle is in a ratio α to the area of ​​the entire circle. When the center of the circle is on the outline of the fruit stalk area, the ratio α is less than a certain set threshold value, and when it is greater than the set threshold value, it means that the area of ​​the white part included in this circle not only includes the fruit stalk area but also includes the branch area or other fruit stalk areas, so the position of the growth point can be determined according to this principle.

[0025] Preferably, S4 includes the following steps:

[0026] S41, filtering and removing noise from the classification image and converting it into a grayscale image for classification;

[0027] S42, using an edge detection operator to process the grayscale image for classification, and extracting the outline of the cherry;

[0028] S43, calculating the number of pixels within the range surrounded by the cherry outline.

[0029] In the present invention, images of picked cherries are collected, and the pixels and sizes of each cherry image are guaranteed to be the same and the actual areas included in the images are the same. Therefore, when all variables are consistent, the number of pixels within the range surrounded by the cherry outline in the image can be used to replace the area of ​​the cherry as the measurement value of the cherry size. At the same time, an edge detection operator is used to process the grayscale image for classification, so that a more accurate shape outline of the cherry can be obtained.

[0030] A cherry picking and sorting device based on machine vision, comprising:

[0031] The acquisition module is used to acquire images of cherries hanging on branches including stalks and to pick cherries;

[0032] The classification transmission module is used to collect images of picked cherries and calculate the size of the cherries for classification;

[0033] A connection module is used to connect the collection module and the classification transmission module, and drive the movement of the collection module;

[0034] The control module is used to control the operation of the cherry picking and sorting device.

[0035] In the present invention, the acquisition module captures and shoots images of cherries on branches, determines the picking points on the corresponding cherry stalks after processing and analyzing the cherry images, and then picks the cherries; the connection module drives the movement of the acquisition module, and puts the cherries picked by the acquisition module into the classification and transmission module; the classification and transmission module arranges the cherries one by one and transmits them, and captures the image of each cherry to obtain the size of the cherry, and transmits the cherries to different locations for storage according to the size of the cherry.

[0036] Preferably, the acquisition module includes at least three connecting arms, each of which is provided with a ranging and positioning device; a picking device is provided at the end of the first connecting arm; an image acquisition device is provided at the end of the second connecting arm; and a holding device is provided at the end of the third connecting arm.

[0037] In the present invention, each connecting arm is retractable and bendable, so that the picking device or image acquisition device can pick cherries or take images from different positions and angles; installing devices with different functions at the end of each connecting arm can make image acquisition and cherry picking more free, and when there are leaves affecting, the angle and position can be adjusted to work again; a distance measuring and positioning device is set on each connecting arm, and the position of the cherry can be located by the distance from three different points to the cherry for subsequent picking; the holding device can make an elastic net bag or other device for temporarily holding the picked cherries to avoid damage to the falling cherries.

[0038] Preferably, the classification and transmission module comprises an input unit, which transmits the picked cherries to a plurality of branch classification units through a sorting unit; an image acquisition unit is arranged directly above each of the branch classification units.

[0039] After the cherries in the holding device of the present invention are transferred to the input unit, the input unit continues to transmit them. When passing through the sorting unit, the cherries can be diverted to different branch classification units, and one cherry is transmitted in the branch classification unit each time, and the front and rear adjacent cherries are separated by a certain distance; the image acquisition unit is arranged directly above the branch classification unit, and the field of view width of the image acquisition unit is equal to the width of the branch classification unit, so as to ensure that the size and pixels of each acquired image are the same.

[0040] The present invention has the following beneficial effects: in view of the fact that cherries are small spherical fruits with long stalks, a picking and classification method based on machine vision specifically for cherries is designed according to the long stalks, thereby reducing the large amount of manpower required for manual picking, automating the picking and identification, and improving the efficiency of cherry picking; after picking is completed, a cherry classification process based on machine vision is also provided, which can classify cherries according to their size, reducing the subsequent process required for manual classification, saving manpower and material resources, and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the cherry picking and classification method of the present invention;

[0042] Figure 2 It is a schematic diagram of the cherry picking and sorting device of the present invention;

[0043] Figure 3 is the original grayscale image of cherries in the embodiment of the present invention;

[0044] Figure 4 is a mask image of cherries in an embodiment of the present invention;

[0045] Figure 5 is a grayscale image of the fruit stalk and its vicinity in an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of the positions of the connection points and the growth points in an embodiment of the present invention;

[0047] Figure 7 It is a schematic diagram of cherries hanging on a branch in the background technology. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, a cherry picking and classification method based on machine vision includes:

[0050] S1. Collect the image of cherries hanging on the branches including the fruit stalks and filter and remove the noise as the original image.

[0051] S2, performing image processing on the original image, extracting the stalk features of the cherry and determining the picking point for picking; S2 includes the following steps:

[0052] S21, extracting the grayscale of the original image using the color factor to obtain an original grayscale image;

[0053] S22, converting the original image into a HIS model image and performing segmentation based on the hue H to obtain a mask image;

[0054] S23, combining the original grayscale image and the mask image to exclude the fruit area to obtain a grayscale image of the fruit stalk and its vicinity;

[0055] S24, using the Otsu algorithm to perform binarization processing on the grayscale image in S23, and obtain the connection point position between the fruit stalk and the fruit and the growth point position between the fruit stalk and the branch;

[0056] S25, performing weighted calculation on the connection point position and the growth point position to obtain the picking point position.

[0057] S3. Capture images of the picked cherries to obtain images for classification.

[0058] S4, performing image processing on the classification image, extracting contour features of cherries, and calculating the size of cherries based on the contour features; S4 includes the following steps:

[0059] S41, filtering and removing noise from the classification image and converting it into a grayscale image for classification;

[0060] S42, using an edge detection operator to process the grayscale image for classification, and extracting the outline of the cherry;

[0061] S43, calculating the number of pixels within the range surrounded by the cherry outline.

[0062] S5. Classify the cherries according to their size data.

[0063] The color factor is one of the three primary color components R, G, and B or one of the composite components of the three primary color components; a number of historically collected cherry images are sampled, and the component values ​​of the cherry area color and the component values ​​of the background area color are calculated with the same color factor to form a scatter plot, and the color factor that minimizes the intersection of the component values ​​of the two areas in the scatter plot is selected to perform grayscale extraction on the original image.

[0064] like Figure 6 As shown, the binary image in S24 uses a contour tracking algorithm to obtain the contours of the fruit stalk and the branch; after the binary image and the mask image are overlapped, the contours of the fruit stalk and the branch and the boundary of the fruit area on the mask image intersect at points A1 and B1, and the midpoint of the line connecting point A1 and point B1 is the connection point position;

[0065] In the process of moving from point A1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point A i So that α = S CAi / S Ai is greater than the set threshold, and in the process of moving from point B1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point B j So that α = S CBj / S Bj If it is greater than the set threshold, then point A i and point B j The midpoint of the connecting line is the location of the growth point;

[0066] S Ai and S Bj Circle a i and circle b j The number of pixels contained in circle a i and circle b j Point A i and point B j A circle with center S and radius S. CAi and S CBj are the number of pixels within the two circles and the outlines of the fruit stalk and branches.

[0067] The present invention first captures an image of cherries hanging on a tree branch, the image comprising cherry fruits and stalks; when picking cherries, the fruits and part of the stalks are picked, and picking the fruits alone is likely to damage the fruits, so in the design of automated picking, the stalks of the cherries are sheared to complete the picking of the cherries; the present invention processes the original image to extract an image of the stalk and its vicinity, extracts the stalk features from the image, and determines the picking point based on the stalk features to pick the cherries; the picked cherries are re-imaged to obtain a classification image, and after image processing of the classification image, the shape outline of the cherries can be extracted, and the number of pixels contained in the shape outline of the cherries is used as a basis for judging the size of the cherries; the present invention performs image acquisition and recognition based on the long stalk features of cherries to determine the picking point for picking, which is more convenient and efficient; at the same time, the size of the cherries is judged based on the number of pixels in the cherry image, and classification can be performed automatically, saving manual classification costs.

[0068] In the present invention, the sizes of the original grayscale image and the mask image are the same, and the mask image is divided into a cherry fruit area and other areas; because the color of cherries is quite different from the color of surrounding green leaves, a more ideal and accurate cherry fruit area segmentation can be obtained after segmentation based on hue H through the HIS model; the original grayscale image excludes the fruit area after mask image processing, thereby obtaining a grayscale image containing only the fruit stalk and its vicinity, thereby avoiding the influence of the fruit color problem on the binarization result during the subsequent binarization processing; the connection point position and the growth point position can be found in the binary map of the fruit stalk and its vicinity, and the line connecting the two positions is approximately equal to the length of the fruit stalk, so that the picking point position on the fruit stalk can be obtained by weighted calculation according to actual needs.

[0069] In the present invention, the color factors include R component, G component, B component, |GB| component, |GR| component and |GB|+|GR| component, etc., three primary color components and different collective components of the three primary color components; when the component values ​​of the cherry area color and the component values ​​of the background area color calculated by a certain color factor have the least intersection of the component values ​​in the scatter diagram, it means that the grayscale color difference between the cherry area and the background area in the grayscale image obtained by grayscale extraction using the color factor is the largest, and subsequent image processing is easier.

[0070] In the present invention, after binarization processing, the fruit stalk and the branch outline are white or black, and the other background colors are black or white. The fruit stalk and the branch outline are white for illustration, one end of the fruit stalk is connected to the branch at the growth point, and the other end has a connection point with the fruit. Since the fruit stalk has a certain width, the connection point where the fruit stalk outline intersects with the boundary of the fruit area is a curve, and the two end points of the curve are selected as point A1 and point B1, and the focus of the line connecting the two points is used as the connection point; a circle is obtained by taking any point on the outline where the fruit stalk area is located as the center and the line segment |A1B1| as the radius, and the area of ​​the white part included in this circle is in a ratio α to the area of ​​the entire circle. When the center of the circle is on the outline of the fruit stalk area, the ratio α is less than a certain set threshold value, and when it is greater than the set threshold value, it means that the area of ​​the white part included in this circle not only includes the fruit stalk area but also includes the branch area or other fruit stalk areas, so the position of the growth point can be determined according to this principle.

[0071] In the present invention, images of picked cherries are collected, and the pixels and sizes of each cherry image are guaranteed to be the same and the actual areas included in the images are the same. Therefore, when all variables are consistent, the number of pixels within the range surrounded by the cherry outline in the image can be used to replace the area of ​​the cherry as the measurement value of the cherry size. At the same time, an edge detection operator is used to process the grayscale image for classification, so that a more accurate shape outline of the cherry can be obtained.

[0072] like Figure 2As shown, a cherry picking and sorting device based on machine vision includes:

[0073] The acquisition module 2 is used to acquire images of cherries hanging on branches and including stalks and to pick cherries; the acquisition module includes at least three connecting arms, each of which is provided with a distance measuring and positioning device 24; a picking device 5 is provided at the end of the first connecting arm 21; an image acquisition device 6 is provided at the end of the second connecting arm 22; and a holding device 7 is provided at the end of the third connecting arm 23. The classification and transmission module 4 is used to acquire images of the picked cherries and calculate the size of the cherries for classification; the classification and transmission module includes an input unit 41, which transmits the picked cherries to a plurality of branch classification units 43 through a sorting unit 42; an image acquisition unit 44 is provided directly above each branch classification unit. The connection module 3 is used to connect the acquisition module with the classification and transmission module and drive the movement of the acquisition module; the control module 1 is used to control the operation of the cherry picking and classification device.

[0074] In the present invention, the acquisition module captures and shoots images of cherries on branches, determines the picking points on the corresponding cherry stalks after processing and analyzing the cherry images, and then picks the cherries; the connection module drives the movement of the acquisition module, and puts the cherries picked by the acquisition module into the classification and transmission module; the classification and transmission module arranges the cherries one by one and transmits them, and captures the image of each cherry to obtain the size of the cherry, and transmits the cherries to different locations for storage according to the size of the cherry.

[0075] In the present invention, each connecting arm is retractable and bendable, so that the picking device or image acquisition device can pick cherries or take images from different positions and angles; installing devices with different functions at the end of each connecting arm can make image acquisition and cherry picking more free, and when there are leaves affecting, the angle and position can be adjusted to work again; a distance measuring and positioning device is set on each connecting arm, and the position of the cherry can be located by the distance from three different points to the cherry for subsequent picking; the holding device can make an elastic net bag or other device for temporarily holding the picked cherries to avoid damage to the falling cherries.

[0076] After the cherries in the holding device of the present invention are transferred to the input unit, the input unit continues to transmit them. When passing through the sorting unit, the cherries can be diverted to different branch classification units, and one cherry is transmitted in the branch classification unit each time, and the front and rear adjacent cherries are separated by a certain distance; the image acquisition unit is arranged directly above the branch classification unit, and the field of view width of the image acquisition unit is equal to the width of the branch classification unit, so as to ensure that the size and pixels of each acquired image are the same.

[0077] In an embodiment of the present invention, the content of the present invention is explained in the order of the entire cherry picking and classification process. Before picking, the control module controls the acquisition module to reach a position near the cherries to be picked, and the cherries to be picked are initially positioned by the ranging and positioning devices on the three connecting arms of the acquisition module, and the relative positions of the cherries, the picking device, the image acquisition device and the holding device are determined, so that the holding device is located below the cherries and can directly catch the falling cherries. The image acquisition device can freely fine-tune the angle and position through the second connecting arm, and capture an image of cherries hanging on the branches, which includes the cherry fruit, the fruit stalk and part of the branch connected to the fruit stalk.

[0078] The collected image is filtered to remove some noise and then used as the original image for the next step of processing. First, the grayscale of the original image is extracted by the color factor to obtain the original grayscale image. The original image obtained by shooting is a color RGB model image. When the original image is converted into a grayscale image, there will be different grayscale images according to different conversion methods, and the grayscale difference between the cherry area and the background area will also be different due to different conversion methods. Commonly used grayscale conversion can extract grayscale according to one of the components of the three primary colors, or perform grayscale conversion by weighted average of the three primary colors. In this embodiment, some conversion methods are exemplified, including R component, G component, B component, |GB| component, |GR| component, |GB|+|GR| component, etc., such as component synthesis of μ1|α1G-β1B|+μ2|α2G-γ1R|+μ3|γ2B-β2R|, where α, β, γ and μ are all parameters. Select multiple cherry pictures collected historically, sample the component values ​​of the color factor of the cherry area color and the component values ​​of the same color factor of the background area, and draw them into a scatter plot; select the color factor that makes the component values ​​of the cherry area and the background area in the scatter plot have the least intersection as the color factor for grayscale extraction in this embodiment, so as to obtain the original grayscale image, the image is as follows: Figure 3 shown.

[0079] Converting the original image from the RGB model to the HIS model uses hue H, saturation S, and intensity I, which are more in line with human visual perception, to represent the color components. The model conversion formula is:

[0080]

[0081]

[0082] After converting the original image into a HIS image, the mask image can be obtained by segmenting the image by hue H, as shown in Figure 4As shown, the black part in the image is the fruit area, and the white part is the background area. The image processed by the mask image can delete the image of the fruit area and only retain the background image.

[0083] Therefore, the original grayscale image is processed by the mask image to obtain Figure 5 The grayscale image of the fruit stalk and its surrounding area is shown in the figure. In this image, the grayscale image of the fruit stalk and branches is retained, and the image of the fruit area is excluded. In the subsequent binarization process, the color of the fruit area is prevented from interfering with the processing result. In this embodiment, the Otsu algorithm is used to perform binarization on the grayscale image of the fruit stalk and branches to obtain the following image: Figure 6 The binary image shown in the figure shows that the white part is the fruit stalk area and the branch area, and the black part is the part outside the fruit stalk area and the branch area. The contour tracing algorithm is used to obtain the contour of the fruit stalk and the branch area. The contour is the intersection of the white and black parts of the binary image. After the binary image is overlapped with the mask image, the intersection of the fruit stalk contour and the boundary of the fruit area can be obtained, as shown in Figure 6 As shown in the curve between the midpoint A1 and the point B1, the fruit stalk in the figure has a certain width, so both point A1 and point B1 can represent the connecting part between the fruit stalk and the fruit. In this embodiment, the midpoint C1 of the line connecting the two points is selected as the connecting point between the fruit stalk and the fruit.

[0084] Point A1 moves pixel by pixel along the left side of the fruit stalk contour away from the fruit area. In this process, any point A on the fruit stalk contour i Draw a circle with the center as the circle and the length of the line segment |A1B1| as the radius, where i is a natural number greater than or equal to 1. When moving in the contour area of ​​the fruit handle, circle a i It includes the white part of the fruit stem area and the black part of the background area. From the shape characteristics of the cherry fruit stem, we can know that in circle a i The proportion of the white part in the whole circle is less than a set threshold. When the fruit stem is basically straight, α = S CAi / S Ai It fluctuates around half, that is, half is white in the fruit stem area and half is black in the background area. When the fruit stem is partially bent, due to the shape characteristics of the cherry fruit stem, the width of the fruit stem is much smaller than the length, so α = S CAi / S Ai It will be slightly larger than one-half. In this embodiment, a fluctuation range of 10% is selected, that is, the threshold is set to 0.55. When α is greater than 0.55, it can be determined that the circle a i The white area in not only includes the fruit stalk but also the branches or other fruit stalks. At this time, the point A with the smallest i value among all the points on the contour that satisfy α greater than 0.55 is determined. i It is the connection between the fruit stalk and the branch. Point B iThe determination method is the same as point A i The same, there is a connection between the fruit stalk and the branch, point A i and point B i The line connecting the fruit stalk and the branch can be regarded as the connecting plate. i and point B i The midpoint C of the line i Serves as the growth point of the fruit stalk on the branch.

[0085] After getting the connection point C1(x1,y1) and the growth point C i (x i ,y i ), the weighted average of the connection point and the growth point is used to obtain the picking point C c (ax1+bx i ,ay1+by i ), where a+b=1, and both a and b are positive parameters. In the actual picking process, you can choose to keep a longer fruit stalk or a shorter fruit stalk for picking according to your needs. The following explanation is given when a and b are both 0.5. At this time, the midpoint of the line connecting the connection point and the growth point is selected as the picking point for picking. The first connecting arm in the collection module moves to control the collection device to reach the position of the picking point to shear the fruit stalk, cut the cherries, and make the cherries fall into the holding device. When the cherry stalk has a certain curvature, the midpoint of the line connecting the connection point and the growth point is outside the stalk area. At this time, a parallel line of the line segment |A1B1| is drawn through the midpoint. The parallel line has two intersections with the stalk contour area. Any one of the intersections or the midpoint of the two intersections can be selected as a new picking point for picking cherries.

[0086] When the number of cherries in the holding device reaches the quantity threshold, or the quality of the cherries in the holding device exceeds the quality threshold, the connection module controls the acquisition module to recycle, drives the holding device to the top of the input unit of the classification and transmission module, and transfers the cherries in the holding device to the input unit. The entire classification and transmission module can be transmitted through a conveyor belt. When the cherries on the input unit are transmitted to the entrance of the sorting unit, the sorting unit diverts the piles of cherries to different branch classification units, so that the cherries transmitted on the classification branch unit are transmitted one after another, and there is a certain interval between two adjacent cherries, ensuring that the image acquisition unit will only capture the image of one cherry each time.

[0087] After collecting the classification image of cherries, the classification image is filtered and converted into a classification grayscale image. Then the grayscale image for classification is calculated by edge detection operators. The edge detection operators include Robert operator, Prewitt operator, Sobel operator and Canny operator, etc., and any one of the operators can be selected for edge detection to extract the contour features of cherries. The contour enclosing range of cherries can reflect the size of cherries; when the pixels of the classification image size collected by the image acquisition unit are the same and the actual shooting area is the same, the contour area of ​​cherries in the image can be used as the actual size value of cherries as the basis for classification. The contour area of ​​cherries can be represented by the number of pixels contained in the contour enclosing range of cherries. Therefore, the pixel number interval and the corresponding cherry grade classification can be set in advance according to actual needs; after collecting the classification image for the picked cherries, the number of pixels in the contour range of cherries is calculated, and the cherries are classified into the corresponding category according to the interval where the number of pixels is located, and the cherries are transmitted to the storage location corresponding to the category through the branch classification unit for storage.

[0088] The above embodiments are further elaborations and illustrations of the present invention for ease of understanding, and are not limitations of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cherry picking and classification method based on machine vision, characterized in that: include: S1, collecting the image of cherries hanging on the branches including the fruit stalks and filtering and removing noise as the original image; S2, performing image processing on the original image, extracting the characteristics of the cherry stem and determining the picking point for picking; The original grayscale image is obtained by using the color factor to extract the grayscale of the original image; The original image is converted into a HIS model image and segmented based on the hue H to obtain a mask image; The grayscale image of the fruit stalk and its surrounding area is obtained by combining the original grayscale image and the mask image to exclude the fruit area; The grayscale image is binarized using the Otsu algorithm to obtain the connection point between the fruit stalk and the fruit and the growth point between the fruit stalk and the branch; The position of the picking point is obtained by weighted calculation of the connection point position and the growth point position; S3, collecting images of the picked cherries to obtain images for classification; S4, performing image processing on the classification image, extracting contour features of the cherries, and calculating the size of the cherries based on the contour features; S5. Classify the cherries according to their size data.

2. The method for picking and sorting cherries based on machine vision according to claim 1, characterized in that: The color factor is one of the three primary color R, G, B components or one of the composite components of the three primary color components; a number of historically collected cherry images are sampled, and the component values ​​of the cherry area color and the component values ​​of the background area color are calculated with the same color factor to form a scatter plot, and the color factor that minimizes the intersection of the component values ​​of the two area colors in the scatter plot is selected to perform grayscale extraction on the original image.

3. The method for picking and sorting cherries based on machine vision according to claim 1, characterized in that: The contour tracing algorithm is used to obtain the contours of the fruit stalk and the trunk for the binary image in S24; after the binary image and the mask image are overlapped, the contours of the fruit stalk and the trunk and the boundary of the fruit region on the mask image intersect at points A1 and B1, and the midpoint of the line connecting point A1 and point B1 is the connection point position; In the process of moving from point A1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point A i So that α=S CAi / S Ai is greater than the set threshold, and in the process of moving from point B1 along the contour of the fruit stalk and the branch away from the fruit area, there is a point B j So that α=S CBj / S Bj If it is greater than the set threshold, then point A i and point B j The midpoint of the line is the location of the growth point; S Ai and S Bj They are circle a i and circle b j The number of pixels contained in circle a i and circle b j Point A i and point B j A circle with center S and radius S. CAi and S CBj are the number of pixels within the two circles and the outlines of the fruit stalk and branches.

4. The method for picking and sorting cherries based on machine vision according to claim 1, characterized in that: The S4 includes the following steps: S41, filtering and removing noise from the classification image and converting it into a grayscale image for classification; S42, using an edge detection operator to process the grayscale image for classification, and extracting the outline of the cherry; S43, calculating the number of pixels within the range surrounded by the cherry outline.

5. A cherry picking and sorting device based on machine vision, applicable to the method according to any one of claims 1 to 4, characterized in that: include: The acquisition module is used to acquire images of cherries hanging on branches including stalks and to pick cherries; The classification transmission module is used to collect images of picked cherries and calculate the size of the cherries for classification; A connection module is used to connect the collection module and the classification transmission module, and drive the movement of the collection module; The control module is used to control the operation of the cherry picking and sorting device.

6. The cherry picking and sorting device based on machine vision according to claim 5, characterized in that: The acquisition module includes at least three connecting arms, each of which is provided with a distance measuring and positioning device; a picking device is provided at the end of the first connecting arm; an image acquisition device is provided at the end of the second connecting arm; and a containing device is provided at the end of the third connecting arm.

7. The cherry picking and sorting device based on machine vision according to claim 5, characterized in that: The classification and transmission module includes an input unit, which transmits the picked cherries to a plurality of branch classification units through a sorting unit; an image acquisition unit is arranged directly above each of the branch classification units.

Images