A multi-index passion fruit quality grading method based on machine vision

Abstract

The present invention discloses a multi-index passion fruit quality grading method based on machine vision, for shooting video data in a passion fruit production line, through a passion fruit shrinkage discrimination model (i.e., MobileNetV3 neural network) combined with traditional image processing, to achieve detection of three indicators of passion fruit size, maturity, and whether the peel is wrinkled, and passion fruit is divided into three grades in combination with passion fruit grading standard DB45 / T 2100‑2019, and the three grades include whether it is wrinkled, whether it is mature, and the range of fruit diameter. The present invention solves the problems of high cost, low efficiency, and high subjectivity of manual grading. The present invention is based on video analysis, and is more adaptable to operations on automated assembly lines, that is, lossless, fast, and accurate grading that can be achieved by passion fruit during conveyor belt transportation. The present invention comprehensively grades passion fruit size, maturity, and shrinkage according to three indicators, and the grading accuracy is higher.

Description

Technical Field

[0001] The invention relates to the technical field of non-destructive rapid grading of passion fruit quality, and in particular to a multi-index passion fruit quality grading method based on machine vision. Background Art

[0002] As the public's requirements for passion fruit quality become increasingly higher, selecting fruits of appropriate maturity based on transportation distance and time after harvesting will help ensure the optimal maturity of the fruit, thereby improving the competitiveness of the passion fruit market.

[0003] In the current automatic grading of passion fruit, most of them rely on mechanical methods such as perforated screens to achieve the grading of fruit size and ignore indicators such as color and shrinkage. However, in the actual storage process, color changes and shrinkage are closely related to the commodity performance of passion fruit. Although there are some passion fruit grading based on multiple indicators, most of them are based on manual judgment, which is labor-intensive, low in productivity, and the sorting accuracy is easily affected by subjective factors, making it difficult to achieve fast and accurate requirements. Therefore, it is a trend to develop an automatic, non-destructive, and multi-indicator comprehensive quality grading method that can be applied to passion fruit online production to improve production quality.

[0004] Machine vision technology can quickly extract the target's color, size and other appearance information, and has been widely used for non-destructive and rapid grading of fruits. In some production lines, machine vision is also used to grade other fruits such as mangoes and apples by multiple indicators. However, for passion fruit, traditional mechanical or single indicator grading is still the main method, and there is no multi-indicator comprehensive quality grading method that uses machine vision technology combined with image processing. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a multi-index passion fruit quality grading method based on machine vision. For the video data shot in the passion fruit production line, a passion fruit shrinkage discrimination model (i.e., MobileNetV3 neural network) is combined with traditional image processing to detect three indicators of passion fruit size, maturity, and whether the peel is wrinkled. Combined with the passion fruit grading standard DB45 / T 2100-2019, passion fruit is divided into three grades, which solves the problems of high cost, low efficiency, and high subjectivity of manual grading.

[0006] To achieve the above object, the technical solution provided by the present invention is: a multi-index passion fruit quality grading method based on machine vision, comprising the following steps:

[0007] 1) Calibrate the industrial camera to obtain the actual size of the unit pixel at the camera height, that is, the actual size of the unit pixel;

[0008] 2) Place the passion fruits on a conveyor belt, and use an industrial camera placed vertically above the conveyor belt to capture the video of the passion fruits being conveyed on the conveyor belt;

[0009] 3) Process each frame image of the video described in step 2) through the background subtraction model in the OpenCV software library to identify the passion fruits and generate their binary images;

[0010] 4) Use a 5×5 convolution kernel to perform morphological closing operation on the binary image described in step 3) to fill the holes in the passion fruit image, and then use a 25×25 convolution kernel to perform morphological opening operation on the image, cutting off the connection between the pedicel and the fruit of the passion fruit, preventing the interference of the pedicel on the size detection of the passion fruit;

[0011] 5) Set a specific number of pixels on the left and right of the image center as the counting interval. Based on the image processed in step 4), obtain the central x coordinate of the passion fruit. As the passion fruit moves, when the central coordinate of the passion fruit enters the above counting interval, start counting the number of image frames fps, and take a certain frame image as the analysis image for the three indicators of the size, maturity, and whether the pericarp is wrinkled of the passion fruit. At the same time, automatically record the serial number of the passion fruit after extracting the image for subsequent result statistics;

[0012] 6) Solve the minimum circumscribed rectangle and the inscribed rectangle for the contour of the passion fruit in the image described in step 5). Based on the minimum circumscribed rectangle, obtain the longest side H2, and crop the image based on the inscribed rectangle area;

[0013] 7) Input the image cropped in step 6) into the pre-established passion fruit wrinkling situation discrimination model to detect whether the passion fruit is wrinkled; According to the actual size size of a single pixel provided in step 1) and the longest side H2 of the minimum circumscribed rectangle obtained in step 6), obtain the actual size D of the passion fruit. The calculation formula is: D = H2×size; Convert the image cropped in step 6) to the HSV color space, set thresholds based on red and yellow, retain the areas that meet the threshold requirements, and set other parts as the background. In this way, binary images of the passion fruit areas under each color are obtained respectively. Add the binary images of red and yellow, and use the ratio of the total number of pixels in the retained area after segmentation to the total number of pixels in the overall passion fruit area as the maturity evaluation criterion. A ratio > the set value indicates maturity;

[0014] 8) Score and sum the three indicators of the passion fruit size, maturity, and whether it is wrinkled respectively, and divide those with the sum of scores > b into special grade, those with scores in the range of [a, b] into first grade, and those < a into second grade, where a < b.

[0015] Further, in step 1), by placing a white cube 3D printed part with side length c directly below the industrial camera and taking its photo, the minimum bounding rectangle of the reference calibration object is recognized through HSV color space conversion. Among them, in order to reduce errors, the average value (H1 + W1) / 2 of the sum of the length H1 and width W1 of the minimum bounding rectangle is used as the side length, so as to obtain the actual size corresponding to a single pixel under the height of this camera, that is, the actual size size of a single pixel, and the calculation is as shown in the following formula:

[0016]

[0017] Further, in step 2), the height between the industrial camera and the conveyor belt is set to 80 cm, and the size of the captured image is 640×480 pixels.

[0018] Further, in step 7), the discrimination model for the shrivelling condition of passion fruits has been established in advance, specifically the MobileNetV3 neural network. 1000 images of passion fruits with smooth epidermis and 500 images of passion fruits with shrivelled epidermis are collected against a black background as the dataset for training and validating the discrimination model for the shrivelling condition of passion fruits. Based on the background subtraction model of KNN in the OpenCV software library, the background of the images in the dataset is subtracted, and the fruit region of the passion fruit is extracted. The processed dataset is divided into a training set and a validation set at a ratio of 9:1, and is input into the discrimination model for the shrivelling condition of passion fruits for training and validation.

[0019] Further, in step 7), when obtaining the area of the reserved region, the mature region in the binary image is assigned a value of 1, and the total number of pixel counts can be obtained by calculating the sum of the pixels in the image.

[0020] Further, in step 7), the threshold settings for red and yellow in each channel of HSV are as follows:

[0021] The threshold setting for the H channel is: for red: 0 < h < 10 or 156 < h < 180; for yellow: 26 < h < 34;

[0022] The threshold setting for the S channel is: for red: 43 < s < 255; for yellow: 43 < h < 255;

[0023] The threshold setting for the V channel is: for red: 46 < v < 255; for yellow: 46 < v < 255;

[0024] Among them, h represents hue, s represents saturation, and v represents value.

[0025] Further, in step 8), according to the passion fruit grading standard DB45 / T 2100-2019, passion fruits are divided into three grades, and the three grades include the conditions of whether they are shrivelled, whether they are mature, and the fruit diameter range.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] 1. Based on video analysis, it is more adaptable to the operation on the automated assembly line, that is, the passion fruit can be graded non-destructively, quickly and accurately during the transportation on the conveyor belt.

[0028] 2. The MobileNetV3 neural network is used to classify whether the passion fruit peel is wrinkled. Its network model has few parameters, small computational complexity, and short inference time. It is suitable for edge computing devices such as mobile embedded devices with limited storage space and power consumption, and can be well deployed in assembly line equipment.

[0029] 3. Comprehensive grading is carried out based on the three indicators of passion fruit size, maturity and shrinkage, with higher grading accuracy.

[0030] 4. When judging maturity, different from the traditional method of traversing the image to count the number of binary image pixels, the area is calculated by summing the binary image pixels, which improves the detection speed. The speed of a single image reaches about 1ms / picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flow chart of the method of the present invention.

[0032] Figure 2 Flowchart for extracting images for interval ranges.

[0033] Figure 3 Schematic diagram of the binarized image obtained by the background subtraction model based on KNN.

[0034] Figure 4 This is the morphological closing operation to remove holes and connect the overall contour effect.

[0035] Figure 5 This is the effect of removing the fruit stalk after morphological opening operation.

[0036] Figure 6 Schematic diagram of images extracted from videos for multi-index detection of passion fruit.

[0037] Figure 7 Schematic diagram of the minimum enclosing rectangle and the orthogonal rectangle based on the passion fruit mark in the extracted image.

[0038] Figure 8 This is a schematic diagram of a passion fruit image cropped based on a contiguous rectangular area.

[0039] Fig. 9 Schematic diagram of the superimposed binary image after red and yellow threshold segmentation.

[0040] Fig.10This is the final detection effect diagram. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0042] like Figure 1 As shown, this embodiment provides a multi-index passion fruit quality grading method based on machine vision, which comprehensively grades the three indicators of size, maturity, and whether the Taiwanese purple passion fruit is wrinkled. The specific situation is as follows:

[0043] 1) Prepare Tainong passion fruit with smooth skin and wrinkled skin, place them on a black background, and take photos at the center of the viewing angle of an industrial camera. A total of 1000 passion fruit images with smooth skin and 500 passion fruit images with wrinkled skin are collected as data sets for training and verifying a pre-established passion fruit wrinkle discrimination model, wherein the passion fruit wrinkle discrimination model is specifically a MobileNetV3 neural network; further, the background subtraction model of KNN in the OpenCV software library is used to perform background subtraction processing on the images in the data set, and the passion fruit area is extracted; the processed data set is divided into a training set and a validation set in a ratio of 9:1, and input into the MobileNetV3 passion fruit wrinkle discrimination model for training and verification.

[0044] 2) Calibrate the camera by placing a white cube 3D print with a side length of c directly under the industrial camera and taking a photo of it. Then, perform the minimum bounding rectangle recognition on the reference calibration object through HSV color space conversion. In order to reduce the error, the average value (H1+W1) / 2 of the sum of the length (H1) and width (W1) of the minimum bounding rectangle is used as the side length, so as to obtain the actual size of the unit pixel at the camera shooting height, that is, the actual size of the unit pixel, as shown in the following formula:

[0045]

[0046] 3) In the actual detection process, the passion fruit is placed on a conveyor belt, and the binary image of the passion fruit in motion is identified through the background subtraction model. Figure 3 As shown in the figure, the model outputs the image with holes in the contour. The image is processed by morphological closing operation with a 5×5 convolution kernel to fill the holes and connect the overall contour. The processing result is as follows Figure 4 shown.

[0047] 4) Through the convolution kernel of 25×25, the image is morphologically opened to cut off the connection between the passion fruit stalk and the fruit, thus preventing the stalk from interfering with the passion fruit size detection. The processing results are shown in the figure below. Figure 5 shown.

[0048] 5) Set 140 pixels on each side of the center of the image as the counting interval. Based on each frame of the image in the video, obtain the x - coordinate of the center of the passion fruit. As Figure 2 shown, as the passion fruit moves, when the center coordinate of the passion fruit enters this counting interval, start counting the number of image frames (fps), and take the third - frame (fps = 3) image as the analysis image for the three indicators of the size, maturity, and whether the pericarp is shrunken of the passion fruit. At the same time, automatically record the serial number of the passion fruit after extracting the image for subsequent result statistics. As Figure 6 shown, this is the extracted image, and the passion fruit at the center of the image is the passion fruit to be analyzed.

[0049] 6) Solve the minimum - bounding rectangle and the circumscribed rectangle of the passion - fruit contour at the center of the extracted image. Based on the minimum - bounding rectangle, obtain the longest side H2. As Figure 7 shown; Crop the image based on the circumscribed - rectangle area. As Figure 8 shown.

[0050] 7) Detection of the passion - fruit shrinkage index: Input the obtained cropped image into the established passion - fruit shrinkage - situation discrimination model, and then it can be detected whether the passion fruit is shrunken.

[0051] 8) Detection of the passion - fruit size index: According to the actual size size of a unit pixel provided in step 2) and the larger side length H2 of the minimum - bounding rectangle provided in step 6), obtain the actual size D of the passion fruit. The calculation formula is: D = H2×size.

[0052] 9) Detection of the passion - fruit maturity index: Convert the cropped - image area obtained in step 6) to the HSV color space. Set thresholds based on red and yellow. Retain the areas that meet the threshold requirements, and set other parts as the background. In this way, obtain the binary images of the passion - fruit areas under each color respectively. Superimpose the binary images of red and yellow. As Fig. 9 shown, take the ratio of the total number of pixels in the retained area after segmentation to the total number of pixels in the overall passion - fruit area as the maturity - judgment criterion. A ratio > 60% indicates maturity.

[0053] Preferably, when calculating the area of the retained area, assign a value of 1 to the mature area in the binary image, and the total number of pixel counts can be obtained by calculating the sum of the pixels in the image.

[0054] Preferably, the threshold settings of red and yellow in each channel of HSV are as follows:

[0055] The threshold setting for the H channel is: for red: 0 < h < 10 or 156 < h < 180; for yellow: 26 < h < 34;

[0056] The threshold for the S channel is set as follows: Red: 43 < s < 255; Yellow: 43 < h < 255;

[0057] The threshold for the V channel is set as follows: Red: 46 < v < 255; Yellow: 46 < v < 255;

[0058] Among them, h represents hue, s represents saturation, and v represents value.

[0059] 10) Manually score and sum the three indicators of the size, maturity, and whether it is shrunken of the passion fruit. The scoring criteria are as follows:

[0060] Table 1 Manual scoring values for the three indicators

[0061]

[0062] 11) According to the passion fruit grading standard DB45 / T 2100 - 2019, the passion fruit is divided into three grades. The three grades include whether it is shrunken, whether it is mature, and the fruit diameter range. The specific grading is shown in Table 2. Calculate the total score according to step 10). Those with a score > 6 are classified as special grade, < 5 are classified as second grade, and the others are classified as first grade. The final detection effect is as Fig.10 shown. According to the grading of the grading standard, the various indicator situations included in each grade and the corresponding scores are as follows in the table:

[0063] Table 2 Grade division and score assignment for 12 situations

[0064]

[0065] In summary, this embodiment comprehensively grades the Tainong purple passion fruit considering the size, maturity, and whether the fruit peel is shrunken of the passion fruit. For special grade fruits, the grading accuracy is 94%, for first grade fruits, the grading accuracy is 97%, and for second grade fruits, the grading accuracy is 100%, showing a good discrimination effect. At the same time, it can accurately count the passion fruits. The method of the present invention directly obtains the video stream analysis through an industrial camera, with an average grading accuracy of 97% and an efficiency of 120 ms / fruit for grading one passion fruit. It shows that the method of the present invention can be effectively applied to the automated production line grading of passion fruits and is worthy of promotion.

[0066] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A multi-index passion fruit quality grading method based on machine vision, characterized in that: Including the following steps: 1) Calibrate the industrial camera to obtain the actual size corresponding to a unit pixel at the height of the camera, that is, the actual size of a unit pixel size; 2) Place the passion fruits on the conveyor belt, and use the industrial camera to be vertically placed above the conveyor belt to capture the video of the passion fruits being conveyed on the conveyor belt; 3) Process each frame of the video described in step 2) through the background subtraction model in the OpenCV software library to identify the passion fruits and generate their binary images; 4) Use a 5×5 convolution kernel to perform morphological closing operation on the binary image described in step 3) to fill the holes in the passion fruit image, and then use a 25×25 convolution kernel to perform morphological opening operation on the image, cutting off the connection between the stalk and the fruit of the passion fruit, preventing the stalk from interfering with the size detection of the passion fruit; 5) Set a specific number of pixels on the left and right of the image center as the counting interval. Based on the image processed in step 4), obtain the center x coordinate of the passion fruit. As the passion fruit moves, when the center coordinate of the passion fruit enters the above counting interval, start counting the number of image frames fps, and take a certain frame of the image as the analysis image for the three indicators of the size, maturity, and whether the peel is wrinkled of the passion fruit. At the same time, automatically record the serial number of the passion fruit after extracting the image for subsequent result statistics; 6) Solve the minimum circumscribed rectangle and the orthographic rectangle for the contour of the passion fruit in the image described in step 5). Based on the minimum circumscribed rectangle, obtain the longest side H2, and crop the image based on the orthographic rectangle area; 7) Input the image cropped in step 6) into the pre-established passion fruit shrinkage situation discrimination model to detect whether the passion fruit is wrinkled; According to the actual size of a unit pixel size provided in step 1) and the longest side H2 of the minimum circumscribed rectangle obtained in step 6), obtain the actual size D of the passion fruit. The calculation formula is: D = H2 × size; Convert the image cropped in step 6) to the HSV color space, set thresholds based on red and yellow, retain the areas that meet the threshold requirements, and set other parts as the background. In this way, binary images of the passion fruit areas under each color are obtained respectively. Add the binary images of red and yellow, and use the ratio of the total number of pixels in the retained area after segmentation to the total number of pixels in the overall passion fruit area as the maturity evaluation criterion. A ratio > the set value indicates maturity; 8) Score and sum the three indicators of the passion fruit size, maturity, and whether it is wrinkled respectively, and divide those with the sum of scores > b into special grade, those with scores between [a, b] into first grade, and those < a into second grade, where a < b.

2. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 1), place a white cube 3D printed part with a side length of c directly below the industrial camera and take its photo. Through HSV color space conversion, perform minimum circumscribed rectangle recognition on the reference calibration object; Among them, in order to reduce errors, use the average value (H1 + W1) / 2 of the sum of the length H1 and width W1 of the minimum circumscribed rectangle as the side length, so as to obtain the actual size corresponding to a unit pixel at the height of this camera, that is, the actual size of a unit pixel size. The calculation is as shown in the following formula:

3. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 2), the height between the industrial camera and the conveyor belt is set to 80 cm, and the size of the captured image is 640×480 pixels.

4. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 7), the discriminant model for the wrinkling condition of passion fruits has been established in advance, specifically the MobileNetV3 neural network. 1000 images of passion fruits with smooth skins and 500 images of passion fruits with wrinkled skins are collected against a black background as the dataset for training and validating the discriminant model for the wrinkling condition of passion fruits. The background of the images in the dataset is subtracted based on the KNN background subtraction model in the OpenCV software library, and the passion fruit fruit regions are extracted. The processed dataset is divided into a training set and a validation set at a ratio of 9:1 and input into the discriminant model for the wrinkling condition of passion fruits for training and validation.

5. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 7), when calculating the area of the retained region, the mature region in the binary image is assigned a value of 1, and the total number of pixels can be obtained by calculating the sum of the pixels in the image.

6. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 7), the threshold settings for red and yellow in each HSV channel are as follows: The threshold setting for the H channel is: for red, 0 < h < 10 or 156 < h < 180; for yellow, 26 < h < 34; The threshold setting for the S channel is: for red, 43 < s < 255; for yellow, 43 < h < 255; The threshold setting for the V channel is: for red, 46 < v < 255; for yellow, 46 < v < 255; Among them, h represents hue, s represents saturation, and v represents value.

7. A multi-index passion fruit quality grading method based on machine vision according to claim 1, characterized in that: In step 8), the passion fruits are divided into three grades according to the passion fruit grading standard DB45 / T 2100-2019. The three grades include the conditions of whether there is wrinkling, whether it is mature, and the fruit diameter range.

Images