A machine vision-based plug seedling grading and transplanting method

Abstract

A kind of plug seedling grading transplanting method based on machine vision, comprising the following steps: obtaining the original image of plug seedling grabbed by manipulator, the original image is processed in HSV format, the binary image of leaf region of interest and matrix region of interest in HSV format seedling image is obtained respectively, two binary images are overlapped and added to obtain mask image, the mask image is overlapped with the original image to obtain black background seedling image, the mask image is converted to obtain transparent image, the black background seedling image and transparent image are added to obtain white background seedling image, the white background seedling image is used to grade the plug seedling to be transplanted, and the manipulator transports the plug seedling to be transplanted to corresponding seedling tray according to the grading result.The present application can improve the accuracy of grading evaluation by specific image extraction, reduce the damage rate of seedling transplanting, reduce the calculation amount in grading, and improve the seedling recognition efficiency.

Description

Technical Field

[0001] This invention belongs to the field of plug seedling transplanting technology, specifically relating to a machine vision-based method for graded transplanting of plug seedlings. Background Technology

[0002] Transplanting seedlings from plug trays is a widely used transplanting method in facility agriculture. To meet the increasing demand for growing space and nutrients for seedlings, they need to be transplanted from high-density plug trays to low-density plug trays.

[0003] Mechanized transplanting of plug seedlings is of great significance to the development of my country's vegetable and flower industries. Compared with traditional manual transplanting techniques, mechanized transplanting of plug seedlings has significant advantages such as improved transplanting efficiency, reduced labor intensity, and lower costs. However, plug seedlings with many empty or damaged seedlings are not conducive to the sale of finished seedlings or subsequent mechanized transplanting. Therefore, strict control of the quality of plug seedlings is necessary for mechanized transplanting. In controlling seedling quality, images of the seedlings are usually captured by a vision camera to determine whether the seedlings are up to standard or to classify them, and the judgment results are transmitted to the robotic arm for corresponding operations. However, existing seedling images acquired during transplanting often contain cluttered backgrounds, resulting in a large computational load for subsequent seedling grading, low seedling recognition speed, and low accuracy in seedling information judgment. Summary of the Invention

[0004] The purpose of this invention is to provide a machine vision-based method for grading and transplanting seedlings in seed trays, in order to solve the problems of large computational load and low seedling recognition speed caused by cluttered backgrounds in seedling images during transplanting.

[0005] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0006] A machine vision-based method for grading and transplanting seedlings in plug trays includes the following steps:

[0007] (1) The robotic arm picks up the seedlings in the plug trays to be graded and transplanted;

[0008] (2) Obtain the original image of the seedlings to be transplanted in the plug tray, and convert the original image to obtain an HSV format seedling image;

[0009] (3) Process the HSV format seedling image obtained in (2) to obtain binary images of the leaf region of interest and the matrix region of interest in the seedling image respectively;

[0010] (4) The binary images of the leaf region of interest and the matrix region of interest obtained in (3) are superimposed and added to obtain a mask image;

[0011] (5) Take the mask image obtained in (4) and overlay it with the original image in (2) to obtain a seedling image with a black background, and take the mask image in (4) to perform pixel conversion to obtain a transparent image;

[0012] (6) Add the black background seedling image obtained in (5) to the transparent image to obtain a white background seedling image;

[0013] (7) Based on the white background seedling image obtained in (6), grade the seedlings to be transplanted in the plug trays;

[0014] (8) The robotic arm transports the seedlings to be transplanted from the plug trays to the corresponding seedling trays based on the grading results of (7).

[0015] Furthermore, the process of obtaining the original image of the seedling to be transplanted in (2) is as follows: the depth camera is positioned directly in front of the seedling in ...

[0016] Furthermore, in (3), the inRange function is used to determine whether each pixel in the HSV format seedling image is between [lowerb, upperb]. If the result is yes, 255 is filled in the corresponding pixel position in the output image. If the result is no, 0 is filled in the corresponding pixel position in the output image. The obtained image is a binary image. Among them, lowerb is the low value of the extracted color and upperb is the high value of the extracted color.

[0017] Furthermore, in (3), the low and high values ​​of the color are determined based on the color of the region of interest and by querying the HSV color space table. In the seedling image, the range of orange, yellow, green, cyan, and blue in the region of interest of the leaf is selected as the extraction threshold of the binary image of the leaf, and the range of black in the region of interest of the matrix is ​​selected as the extraction threshold of the binary image of the matrix.

[0018] Furthermore, the threshold for extracting the color of the leaf region of interest as yellow-green has a low value of [12, 43, 46] and a high value of [110, 255, 255]; the threshold for extracting the matrix region of interest as black has a low value of [0, 0, 0] and a high value of [180, 255, 46].

[0019] Furthermore, the extracted region of interest (ROI) image of the leaf is first subjected to median filtering and noise reduction using the medianBlur function, and then the outer contour is drawn to obtain a binary image of the ROI of the leaf.

[0020] Furthermore, the process of outlining the outer contour is as follows:

[0021] S1. Find the contour of the region of interest of the leaf: First, use the threshold function to perform binarization image processing, set the threshold to 235, and take 255 for pixels that exceed this threshold, otherwise take 0; then use the findContours function to find the contour of the region of interest of the leaf.

[0022] S2. Use the convexHull function to find the convex protrusions of the contour, and use the line function to draw lines to connect the outer contour.

[0023] S3. Use the connectedComponentsWithStats function to connect regions with the same number of pixels together.

[0024] Furthermore, the extracted matrix region of interest image is first subjected to morphological processing, and then the medianBlur function is used for median filtering and noise reduction to obtain a binary image of the matrix region of interest.

[0025] Furthermore, the morphological processing involves the following steps:

[0026] S1. Use the getStructuringElement function to create a rectangular kernel with a size of 15*10;

[0027] S2. The dilate function is used to fill small defects in the image, and the erode function is used to remove burrs from the image.

[0028] Furthermore, the grading process in (7) is as follows: the area of ​​the seedling leaves and the area of ​​the substrate in the plug tray are used as the basis for judgment, or the ratio of the white background to the area of ​​the region of interest is used as the basis for judgment, and the threshold level is set. The grading is carried out based on the threshold level reached by the seedling leaf area and the substrate area of ​​the seedling image with the white background.

[0029] The beneficial effects of this invention are:

[0030] The seedling grading and transplanting method of the present invention performs HSV format conversion, binary image conversion and mask image processing on the acquired seedling images in sequence. This can better remove the background in the seedling images. In the grading process, it can greatly reduce the amount of calculation for seedling grading, improve the seedling recognition efficiency, and at the same time help to obtain key seedling information and improve the accuracy of seedling information judgment. Attached image description:

[0031] Figure 1 This is a flowchart illustrating the image extraction process in a machine vision-based seedling grading and transplanting method for plug trays, as described in this embodiment.

[0032] Figure 2This is a schematic diagram illustrating the process of extracting images of seedlings from seed trays using the machine vision-based seedling grading and transplanting method in this embodiment.

[0033] Figure 3 This is a schematic diagram illustrating the process of extracting color images of seedlings from seed trays using the machine vision-based seedling grading and transplanting method described in this embodiment. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. In the description of the present invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0035] Combination Figures 1 to 3 As shown, this embodiment provides a machine vision-based method for grading and transplanting seedlings in plug trays, including the following steps:

[0036] (1) The robotic arm grabs the seedlings in the plug trays to be graded and transplanted. During implementation, each robotic arm gripper is controlled by a separate servo motor, which can achieve individual independent control.

[0037] (2) Position the depth camera directly in front of the seedlings in the plug tray held by the robotic arm. The depth camera should be 90mm away from the seedlings, and its height should be level with the height of the seedlings held by the robotic arm. To ensure the completeness of the seedling image extraction and the success rate of background removal, ensure that there are no other green or black objects in the field of view of the depth camera except for the seedlings in the plug tray. Then, the depth camera acquires the original image of the seedlings to be transplanted. The image acquisition method can be real-time acquisition until all the seedlings in the plug tray have been graded and transplanted. After brightening the acquired original image of the seedlings, the cvtColor function is used to convert it into an HSV image to facilitate the separation and conversion of the image and the background. Figure 2 In the image, the background color of the seedling HSV image is red. Converting the RGB image to an HSV image allows for a more intuitive description of the colors we need. H represents hue, S represents lightness / darkness, and V represents brightness / brightness. HSV format images make it easier to divide colors; simply setting different thresholds will suffice.

[0038] (3) Process the HSV format seedling image obtained in (2) to obtain binary images of the leaf region of interest and the substrate region of interest in the seedling image. It can be understood that the leaf region of interest is the colored region of interest, and the substrate region of interest is the black region of interest.

[0039] During processing, the `inRange` function is used to determine whether each pixel in the HSV format seedling image is within the range [lowerb, upperb]. If the result is yes, 255 is entered into the corresponding pixel position in the output image; if the result is no, 0 is entered into the corresponding pixel position in the output image, resulting in a binary image. Here, `lowerb` is the low value of the extracted color, and `upperb` is the high value of the extracted color. The low and high values ​​of the color are determined based on the color of the region of interest and by consulting the HSV color space table. For the leaf region of interest in the seedling image, the range of orange, yellow, green, cyan, and blue is selected as the extraction threshold for the leaf binary image. For the matrix region of interest, the range of black is selected as the extraction threshold for the matrix binary image. The binarization process traverses every pixel in the image and compares it with the set threshold. Pixels in the region of interest that exceed the set value are set to 255 (white), and those that are not are set to 0 (black). Through binarization, the region of interest (preserving the important parts) and its outline and shape can be quickly obtained.

[0040] In this embodiment, the threshold for extracting the color of the leaf region of interest as yellow-green has a low value of [12, 43, 46] and a high value of [110, 255, 255]; the threshold for extracting the matrix region of interest as black has a low value of [0, 0, 0] and a high value of [180, 255, 46]. Selecting yellow-green can extract as much seedling leaf information as possible, so as to avoid omissions or incomplete extraction of leaf information.

[0041] In this method, the extracted leaf region of interest image is first filtered by the medianBlur function to remove noise, and then the outer contour is drawn to form a binary image of the leaf region of interest. The median gray value of the pixel is used to replace the gray value of the pixel, so that the surrounding pixel values ​​are close to the real values, thereby eliminating isolated noise points. This method can preserve the edge details of the image while removing impulse noise and salt-and-pepper noise.

[0042] Specifically, the process of outlining the outer contour is as follows:

[0043] S1. Find the contour of the region of interest of the leaf: First, use the threshold function to perform binarization image processing, set the threshold to 235, and take 255 for pixels that exceed this threshold, otherwise take 0; then use the findContours function to find the contour of the region of interest of the leaf.

[0044] S2. Use the convexHull function to find the convex protrusions of the contour, and use the line function to draw lines to connect the outer contour. This can make the outer contour of the seedling clearer. Otherwise, the pixels at the very edge of the contour are the most difficult to distinguish, and there will be a blurry phenomenon.

[0045] S3. Use the connectedComponentsWithStats function to connect regions with the same pixel size. Connecting connected components removes smaller regions, which is equivalent to lossless noise reduction and removal of impurities.

[0046] In this process, the extracted matrix region of interest image is first subjected to morphological processing, and then the medianBlur function is used for median filtering and noise reduction to obtain the binary image of the matrix region of interest.

[0047] Specifically, the morphological processing involves the following steps:

[0048] First, the getStructuringElement function is used to create a rectangular kernel with a size of 15*10. Then, the dilate function is used to fill small defects in the image, and the erode function is used to remove burrs in the image. This can result in a more perfect target area, removal of interference, smoothing of boundaries, neighborhood connectivity, and filling of holes.

[0049] (4) The binary image of the leaf region of interest and the binary image of the matrix region of interest obtained in (3) are processed by bitwise_or function to perform basic image operations, and the two images are superimposed and added to obtain the mask image.

[0050] (5) The mask image obtained in (4) is superimposed on the original image in (2) using the bitwise_or function. The pixel values ​​in the region of interest remain unchanged, while the other pixel values ​​are all zero, to obtain a seedling image with a black background. The mask image in (4) is converted into a four-channel image using image_out.convert("RGBA"), and the pixel is converted using the putpixel function, that is, the black pixel (0,0,0,255) is converted into the white pixel (255,255,255,255), and the white pixel (255,255,255,255) is converted into the transparent pixel (0, 0, 0, 0), to obtain a transparent image.

[0051] (6) Take the black background seedling image obtained in (5) and the transparent image and add them together using the addWeighted function. The weights of the two images are both set to 1, and the image is of type uinit8. If the result of the direct calculation is greater than 255, it will be truncated to 255 (finally white) to obtain the white background seedling image.

[0052] (7) Based on the white background seedling image obtained in (6), the seedlings to be transplanted in the plug tray are graded, that is, the white background seedling image is used as the basis for subsequent seedling grading or removal. In practice, the area of ​​the seedling leaves and the area of ​​the substrate in the white background seedling image are used as the basis for judgment, or the ratio of the white background to the area of ​​the region of interest is used as the basis for judgment. A certain threshold level is set, and the seedling leaf area and substrate area in the white background seedling image are graded according to the threshold level reached. For example, deep learning can be used to grade the white background seedling image, and the grading result is finally obtained.

[0053] (8) The robot arm transmits the grading results from (7) to the robot arm via the Bluetooth module of the microcontroller. The robot arm then transports the seedlings to be transplanted from the seedling trays to the corresponding seedling trays, thus completing the grading and transplanting of the seedlings.

[0054] It should be noted that the above embodiments are only used to illustrate the present invention, but the present invention is not limited to the above embodiments. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A machine vision-based plug seedling grading and transplanting method, characterized in that, The method comprises the following steps: (1) a mechanical hand picks up the plug seedlings to be transplanted; (2) an original image of the plug seedlings to be transplanted is obtained, and an HSV format seedling image is obtained by processing the original image; (3) the HSV format seedling image obtained in (2) is processed to obtain a binary image of a leaf region of interest and a substrate region of interest in the seedling image, wherein in (3), an inRange function is used to determine whether each pixel in the HSV format seedling image is between [lowerb, upperb]; if the result is yes, 255 is filled in the corresponding pixel position of the output image, and if the result is no, 0 is filled in the corresponding pixel position of the output image, so that a binary image is obtained; wherein lowerb is a low value of the extracted color, and upperb is a high value of the extracted color, the low value and the high value of the color are determined according to the color of the region of interest and a query HSV color space table, the range of orange, yellow, green, cyan and blue in the leaf region of interest in the seedling image is selected as the extraction threshold of the leaf binary image, and the range of black in the substrate region of interest is selected as the extraction threshold of the substrate binary image; the extracted leaf region of interest image is first subjected to medianBlur function median filtering denoising, and then subjected to external contour line drawing to obtain a binary image of the leaf region of interest; the process of external contour line drawing is as follows: S1, find the contour of the leaf region of interest: first, perform binary image processing by using a threshold function, and set the threshold thresh to 235; if the pixel point exceeds the threshold, 255 is taken, otherwise 0 is taken; then find the contour of the leaf region of interest by using a findContours function; S2, find the convex protruding point of the contour by using a convexHull function, and connect the external contour by using a line function; S3, connect the regions with the same pixels together by using a connectedComponentsWithStats function; (4) the binary images of the leaf region of interest and the substrate region of interest obtained in (3) are subjected to overlapping addition processing to obtain a mask image; (5) the mask image obtained in (4) is superimposed with the original image in (2) to obtain a black background seedling image, and the mask image in (4) is subjected to pixel conversion to obtain a transparent image; (6) the black background seedling image obtained in (5) is added to the transparent image to obtain a white background seedling image; (7) the white background seedling image obtained in (6) is used to classify the plug seedlings to be transplanted; the classification process in (7) is as follows: the area of the leaf of the plug seedling and the area of the substrate are used as the judgment basis, or the area ratio of the white background to the region of interest is used as the judgment basis, a threshold level is set, and the white background seedling image is classified according to the threshold level reached by the leaf area and the substrate area of the seedling; (8) the mechanical hand transports the plug seedlings to be transplanted to the corresponding seedling tray according to the classification result in (7).

2. A machine vision based seedling grading and transplanting method as claimed in claim 1 wherein, The process of acquiring the original image of the transplanting plug seedling is as follows: the depth camera is directed at the plug seedling grabbed by the manipulator, and the depth camera acquires the original image of the transplanting plug seedling under the condition that the depth camera is in the field of view and no other green and black objects are present.

3. A machine vision based seedling grading and transplanting method as claimed in claim 1, wherein, The threshold value of the yellow-green color for extracting the color of the leaf blade region of interest is [12, 43, 46] for the low value and [110, 255, 255] for the high value; the threshold value of the black color for extracting the color of the substrate region of interest is [0, 0, 0] for the low value and [180, 255, 46] for the high value.

4. A machine vision based seedling grading and transplanting method as claimed in claim 3 wherein, The morphological processing is performed on the extracted substrate region of interest image, and then the medianBlur function is used for median filtering denoising to serve as the binary image of the substrate region of interest.

5. A machine vision based seedling grading and transplanting method as claimed in claim 4, wherein, The process of morphological processing is as follows: S1, a rectangular kernel with a size of 15*10 is established by using the getStructuringElement function; S2, the dilate function is used for inflation operation to fill in the small defects in the image, and the erode function is used for corrosion operation to remove burrs in the image.

Images