A Method and System for Detecting Banana Bun Fullness Based on Shape and Color Information
By using a detection method based on shape and color information, and employing a Kinect v3 camera and an LDA classifier, the feature values of banana combs are automatically extracted. This solves the problems of time-consuming, labor-intensive, and subjective differences associated with manual detection, and achieves efficient, accurate, and automated detection of banana comb fullness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-15
- Publication Date
- 2026-03-10
AI Technical Summary
In existing technologies, the detection of banana fruit comb fullness relies on manual judgment, which is labor-intensive and resource-intensive, and subject to significant subjective differences, making it difficult to achieve automation and accuracy.
A detection method based on shape and color information is adopted. RGB-D images are acquired using a Kinect v3 camera. Feature values of banana combs are automatically extracted through adaptive threshold segmentation, edge detection, and LDA classifier to achieve automated detection.
It achieves automated and highly accurate banana comb fullness grading, avoids human subjective error, and improves detection efficiency and accuracy.
Smart Images
Figure CN115330721B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of banana comb fullness detection, and in particular to a method and system for detecting banana comb fullness based on shape and color information. Background Technology
[0002] The plumpness of bananas at harvest is crucial to their storage time and consumer acceptability. Generally, bananas are harvested when they are plump and firm, then ripened further at the sales location before being sold or consumed. However, bananas undergo a climacteric respiration period, meaning that many bananas suffer quality loss during storage and transportation, primarily due to the mixing of bananas at different ripeness levels. Therefore, accurately determining banana plumpness and processing bananas of the same plumpness level uniformly is essential for ensuring banana quality.
[0003] Currently, banana plumpness grading relies heavily on manual labor. Experienced fruit growers observe the banana's appearance to determine its plumpness; for example, less plump bananas have a bluish-green peel with clear edges, while plumper bananas have a yellowish-green peel with lost edges. However, manually judging banana plumpness is labor-intensive and resource-intensive, and subject to subjective differences.
[0004] With advancements in machine vision technology, the grading of more and more fruits has become automated. The Kinectv3 camera can simultaneously acquire color and depth images, extracting both color and shape information from the sample. While some RGB-D image technology is already used for fruit quality grading, there are currently no applications using this technology to detect the plumpness of bananas. Summary of the Invention
[0005] The primary objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for detecting the fullness of banana combs based on shape and color information. This method can effectively and accurately detect the fullness level of banana combs, solving problems such as the strong subjectivity, time-consuming and labor-intensive nature of manual detection.
[0006] The second objective of this invention is to provide a banana comb fullness detection system based on shape and color information.
[0007] The first objective of this invention is achieved through the following technical solution: a method for detecting the fullness of banana combs based on shape and color information, comprising the following steps:
[0008] 1) Prepare at least two different banana fruit combs with different plumpness levels, collect their color and depth images, and use the camera's internal parameters to align the images. The two aligned images constitute a four-channel RGB-D image.
[0009] 2) Preprocessing the RGB-D image: Adaptive threshold segmentation is performed on the color image in the RGB-D image to extract the region of interest, and nearest neighbor filtering is performed on the depth image in the RGB-D image to fill in the missing depth pixels;
[0010] 3) Perform edge detection on the preprocessed RGB-D image to separate and identify individual banana fingers;
[0011] 4) Extract three features from a single banana finger: average color value, finger length, and mid-section width;
[0012] 5) Take the average of the three features of each banana finger as the feature value of the whole banana comb. After obtaining the feature values of all banana combs, normalize the feature values of all banana combs to obtain the normalized banana comb data of different fullness levels.
[0013] 6) Randomly select a portion of the banana fruit comb data obtained after normalization as the training set, and the remainder as the test set;
[0014] 7) Import the data from the training set into the LDA classifier, which is used as the banana plumpness discrimination model, for training to obtain a trained banana plumpness discrimination model based on three features: color, finger length, and middle cross-sectional width.
[0015] 8) Input the data from the test set into the trained banana fullness discrimination model to obtain accurate banana comb fullness detection results, i.e., the corresponding fullness level of the banana comb.
[0016] Furthermore, in step 1), when acquiring images, the Kinect v3 camera is used to acquire color and depth images of the banana combs at the same time. The banana combs are from the same period, variety, and orchard, eliminating interference from time and place factors.
[0017] Furthermore, in step 2), the preprocessing of the RGB-D image is as follows:
[0018] a. Color Image: Extract the grayscale image of the R channel from the three channels of the RGB color image, perform adaptive threshold segmentation, then perform edge detection, convert the original image into a binary image, set all pixels in the foreground area to 0, and set all pixels in the background area to 255, then mask this binary image with the original image, set all pixels in the background area of the original image to 255, and leave the pixels in the foreground area unchanged.
[0019] b. Depth Image: Mask the binarized image from a and the depth image, set all pixels in the background region to 0, leave the pixels in the foreground region unchanged, and then iterate through each pixel in the foreground region to find the point with a pixel value of 0, which is the point where the depth information is missing. Correct the pixel value of this point to the average of the non-zero pixel values of its eight neighboring pixels, thus filling in all the pixels with missing depth information.
[0020] Furthermore, in step 3), the separation and identification of individual banana fingers are as follows:
[0021] The color image is converted to a grayscale image. The Canny edge detection algorithm is used to find the contour lines between the banana fingers on both the grayscale and depth images. The contour lines from the two images are then drawn on the same image. After morphological processing, the contour lines are closed. Each closed contour line is a banana finger. Incomplete banana fingers are removed by area filtering.
[0022] Furthermore, in step 4), the three features extracted from a single banana finger are as follows:
[0023] a. Average color value: The average pixel value of all pixels in the foreground region of a grayscale image;
[0024] b. Banana finger length: The central axis of the banana finger is extracted using a skeleton extraction algorithm. Let the horizontal coordinate of the banana finger image be the Y-axis and the vertical coordinate be the X-axis. Then the coordinates of each point on the central axis from top to bottom are (X1, Y1), (X2, Y2), (X3, Y3), and so on. The coordinates of the nth point are (X... n Y n The depth values corresponding to these points are denoted as H1, H2, H3, and so on, with the depth value corresponding to the nth point being H. n The formula for calculating the length L of the finger is:
[0025]
[0026] In the formula, r = 1, 2, ..., n-1;
[0027] c. Mid-section width: Fit the function expression of the midline, find the midpoint of the midline based on the length of the fruit finger, and obtain the function expression of the normal of the midline at its midpoint, denoted as f(x). Find the two intersection points of f(x) and the banana outline, denoted as (X... d1 Y d1 ) and (X d2 Y d2 If the width of the middle section is D, then the formula for calculating the width D is:
[0028]
[0029] Further, in step 7), the fullness level corresponding to each banana comb in the training set is imported into the LDA classifier as a predicted value, and the color, finger length, and middle cross-sectional width features corresponding to each banana comb are imported into the LDA classifier as feature vectors. The LDA classifier will generate a weight for each imported feature vector, thereby calculating the fullness level corresponding to the feature value, completing the training of the banana fullness discrimination model, and obtaining a trained banana fullness discrimination model based on three features: color, finger length, and middle cross-sectional width.
[0030] The second objective of this invention is achieved through the following technical solution: a banana comb fullness detection system based on shape and color information, used to implement the above-mentioned banana comb fullness detection method based on shape and color information, comprising:
[0031] The data acquisition module is used to acquire at least two different levels of banana fruit combs, collect their color and depth images, and perform image alignment using camera internal parameters. The two aligned images constitute a four-channel RGB-D image.
[0032] The data preprocessing module is used to preprocess RGB-D images, including: performing adaptive threshold segmentation on the color image in the RGB-D image to extract the region of interest, performing nearest neighbor filtering on the depth image in the RGB-D image, and filling in the missing depth pixels;
[0033] The single banana finger recognition module is used to perform edge detection on the pre-processed RGB-D image, and separate and identify the single banana finger.
[0034] The feature extraction module is used to extract three features from a single banana finger: the average color value, the finger length, and the width of the middle cross section.
[0035] The feature processing module is used to average the three features of each banana finger as the feature value of the whole banana comb. After obtaining the feature values of all banana combs, the feature values of all banana combs are normalized to obtain banana comb data of different fullness levels after normalization. A portion of the banana comb data is randomly selected as the training set and the rest as the test set.
[0036] The training module is used to train the LDA classifier, which serves as a banana plumpness discrimination model. The data in the training set is imported into the LDA classifier for training, resulting in a trained banana plumpness discrimination model based on three features: color, finger length, and mid-section width.
[0037] The testing module is used to input the data from the test set into the trained banana fullness discrimination model to obtain accurate banana comb fullness detection results, that is, the corresponding fullness level of the banana comb.
[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0039] 1. Using a single Kinect v3 device, both color and depth images can be acquired simultaneously, allowing for the simultaneous acquisition of color and shape information of the banana comb, while avoiding the cumbersome calibration of binocular vision cameras.
[0040] 2. It can automatically extract regions of interest, automatically identify individual banana fingers, automatically extract grading indicators for banana fingers, and automatically predict the fullness level of banana combs without manual labeling.
[0041] 3. Using the extracted indicators, the LDA classifier can objectively and fairly predict the fullness level of banana combs, eliminating the influence of subjective factors in human identification.
[0042] 4. The LDA classifier used in this invention has high classification accuracy and fast speed. Attached Figure Description
[0043] Figure 1 This is a flowchart of the method of the present invention.
[0044] Figure 2 These are the original color and depth images.
[0045] Figure 3 This is a schematic diagram of the preprocessed color and depth images.
[0046] Figure 4 A diagram illustrating the identification of a single banana finger.
[0047] Figure 5 This is a schematic diagram of the data extraction results.
[0048] Figure 6 This is a system architecture diagram of the present invention. Detailed Implementation
[0049] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0050] Example 1
[0051] This embodiment discloses a method for detecting the fullness of banana combs based on shape and color information. It utilizes a fusion of color and depth images to extract the shape and color information of each finger on the banana comb, thereby determining the fullness level of the banana comb. Figure 1As shown, the specific details are as follows:
[0052] 1) Prepare banana fruit combs of three different levels of plumpness: 60%, 70%, and 80%. Use a Kinect v3 camera to capture both color and depth images of the bananas in one go. The original images are shown below. Figure 2 As shown, the left side is the original color image, and the right side is the original depth image. After acquiring the two images, image alignment was performed using the camera's internal parameters to obtain a four-channel RGB-D image. The samples were banana bunches from the same period, variety, and orchard, thus eliminating interference from time and origin factors.
[0053] 2) Preprocess the RGB-D image as follows:
[0054] a. Color Image: Extract the grayscale image of the R channel from the RGB three channels of the color image, perform adaptive threshold segmentation, then perform edge detection, convert the original image into a binary image, set all pixels in the foreground area to 0, and set all pixels in the background area to 255. Then, mask this binary image with the original image, set all pixels in the background area of the original image to 255, and leave the pixels in the foreground area unchanged.
[0055] b. Depth Image: Mask the binarized image from a and the depth image, set all pixels in the background region to 0, leave the pixels in the foreground region unchanged, and then iterate through each pixel in the foreground region to find the point with a pixel value of 0, which is the point where the depth information is missing. Correct the pixel value of this point to the average of the non-zero pixel values of its eight neighboring pixels, thus filling in all the pixels with missing depth information.
[0056] The preprocessed image is as follows Figure 3 As shown, the left side is the color image after region of interest extraction, and the right side is the depth image after nearest neighbor filtering.
[0057] 3) Perform edge detection on the RGB-D image to separate and identify individual banana fingers, as follows:
[0058] The color image is converted to a grayscale image. The Canny edge detection algorithm is used to find the contour lines between the banana fingers on both the grayscale and depth images. These contour lines are then plotted on the same image, and morphological processing is performed to close the contour lines. Figure 4 As shown, each closed outline in the left image represents a banana finger. The incomplete banana fingers are removed by area filtering. The right image shows the outline of the complete banana fingers remaining after filtering.
[0059] 4) Extract three features from a single banana finger: average color value, finger length, and mid-section width, as detailed below:
[0060] a. Average color value: The average pixel value of all pixels in the foreground region of a grayscale image.
[0061] b. Finger length: The midline of the banana finger is extracted using a skeleton extraction algorithm, such as... Figure 5 As shown, let the horizontal coordinate of the banana finger image be the Y-axis and the vertical coordinate be the X-axis. Then the coordinates of each point on the central axis from top to bottom are (X1, Y1), (X2, Y2), (X3, Y3), and so on. The coordinate of the nth point is (X... n Y n The depth values corresponding to these points are denoted as H1, H2, H3, and so on, with the depth value corresponding to the nth point being H. n The formula for calculating the length L of the finger is:
[0062]
[0063] In the formula, r = 1, 2, ..., n-1;
[0064] c. Mid-section width: Fit the function expression of the midline, find the midpoint of the midline based on the finger length, and obtain the function expression of the normal to the midline at its midpoint, denoted as f(x). Figure 5 As shown, find the two intersection points of f(x) and the banana outline, denoted as (X, X). d1 Y d1 ) and (X d2 Y d2 If the width of the middle section is D, then the formula for calculating the width D is:
[0065]
[0066] 5) Take the average of the three features of each banana finger as the feature value of the whole banana comb. After obtaining the feature values of all banana combs, normalize the feature values of all banana combs to obtain the normalized banana comb data of different fullness levels.
[0067] 6) Randomly select 2 / 3 of the banana comb data obtained after normalization as the training set, and the rest as the test set.
[0068] 7) Import the data from the training set into the LDA classifier used as the banana plumpness discrimination model for training, and obtain a trained banana plumpness discrimination model based on three features: color, finger length, and mid-section width, as follows:
[0069] The fullness level corresponding to each banana comb in the training set is imported into the LDA classifier as the predicted value. The color, finger length, and mid-section width of each banana comb are also imported into the LDA classifier as feature vectors. The LDA classifier generates a weight for each imported feature vector, thereby calculating the fullness level corresponding to the feature value. This completes the training of the banana fullness discrimination model, resulting in a trained banana fullness discrimination model based on three features: color, finger length, and mid-section width.
[0070] 8) Input the data from the test set into the trained banana fullness discrimination model to obtain accurate banana comb fullness detection results, i.e., the corresponding fullness level of the banana comb. The effectiveness of this invention is shown in Table 1 below.
[0071] Table 1 shows the confusion matrix of the discrimination results between the training set and the test set.
[0072]
[0073] Experiments showed that the present invention achieved a discrimination accuracy of 83.9% and 81.48% for banana fruit combs of three different plumpness levels on the training and test sets, respectively, demonstrating good discrimination performance.
[0074] Example 2
[0075] This embodiment discloses a banana comb fullness detection system based on shape and color information, used to implement the banana comb fullness detection method based on shape and color information described in Embodiment 1, such as... Figure 6 As shown, the system includes the following functional modules:
[0076] The data acquisition module is used to acquire at least two different levels of banana fruit combs, collect their color and depth images, and perform image alignment using camera internal parameters. The two aligned images constitute a four-channel RGB-D image.
[0077] The data preprocessing module is used to preprocess RGB-D images, including: performing adaptive threshold segmentation on the color image in the RGB-D image to extract the region of interest, performing nearest neighbor filtering on the depth image in the RGB-D image, and filling in the missing depth pixels;
[0078] The single banana finger recognition module is used to perform edge detection on the pre-processed RGB-D image, and separate and identify the single banana finger.
[0079] The feature extraction module is used to extract three features from a single banana finger: the average color value, the finger length, and the width of the middle cross section.
[0080] The feature processing module is used to average the three features of each banana finger as the feature value of the whole banana comb. After obtaining the feature values of all banana combs, the feature values of all banana combs are normalized to obtain banana comb data of different fullness levels after normalization. A portion of the banana comb data is randomly selected as the training set and the rest as the test set.
[0081] The training module is used to train the LDA classifier, which serves as a banana plumpness discrimination model. The data in the training set is imported into the LDA classifier for training, resulting in a trained banana plumpness discrimination model based on three features: color, finger length, and mid-section width.
[0082] The testing module is used to input the data from the test set into the trained banana fullness discrimination model to obtain accurate banana comb fullness detection results, that is, the corresponding fullness level of the banana comb.
[0083] Example 3
[0084] This embodiment discloses a storage medium storing a program. When the program is executed by a processor, it implements the banana comb fullness detection method based on shape and color information as described in Embodiment 1.
[0085] The storage medium in this embodiment can be a disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), USB flash drive, portable hard drive, etc.
[0086] Example 4
[0087] This embodiment discloses a computing device, including a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the banana comb fullness detection method based on shape and color information described in Embodiment 1.
[0088] The computing device described in this embodiment may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer, programmable logic controller (PLC), or other terminal device with processor function.
[0089] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for detecting the fullness of banana bunches based on shape and color information, characterized by, The method comprises the following steps: 1) Prepare at least two different fullness grades of banana bunches, collect color images and depth images thereof, and perform image alignment by using internal camera parameters, so that the two images after alignment constitute a four-channel RGB-D image; 2) Preprocess the RGB-D image: perform adaptive threshold segmentation on the color image in the RGB-D image, extract the region of interest, and perform neighborhood filtering on the depth image in the RGB-D image to fill in the missing pixel points of the depth; 3) Perform edge detection on the preprocessed RGB-D image, separate and identify a single banana finger; 4) Extract the average color value, finger length and middle cross-sectional width of the single banana finger, specifically as follows: a. Average color value: the average value of the pixel values of all pixel points in the foreground region of the grayscale image; b. Banana finger length: The central axis of the banana finger is extracted using a skeleton extraction algorithm. Let the horizontal coordinate of the banana finger image be the Y-axis and the vertical coordinate be the X-axis. Then the coordinates of each point on the central axis from top to bottom are (X1, Y1), (X2, Y2), (X3, Y3), and so on. The coordinates of the nth point are (X... n Y n The depth values corresponding to these points are denoted as H1, H2, H3, and so on, with the depth value corresponding to the nth point being H. n The formula for calculating the length L of the finger is: In the formula, r = 1, 2, …, n-1; c. Mid-section width: Fit the function expression of the midline, find the midpoint of the midline based on the length of the fruit finger, and obtain the function expression of the normal of the midline at its midpoint, denoted as f(x). Find the two intersection points of f(x) and the banana outline, denoted as (X... d1 Y d1 ) and (X d2 Y d2 If the width of the middle section is D, then the formula for calculating the width of the middle section is: 5) Average the three characteristics of each banana finger to obtain the characteristic values of the whole banana bunch, normalize the characteristic values of all banana bunches to obtain normalized banana bunch data of different fullness grades; 6) Randomly select a part of the normalized banana bunch data as a training set, and the rest as a test set; 7) Import the data in the training set into the LDA classifier serving as a banana fullness discrimination model to perform training, and obtain a trained banana fullness discrimination model based on the three characteristics of color, finger length and middle cross-sectional width; 8) Input the data in the test set into the trained banana fullness discrimination model to obtain accurate banana bunch fullness detection results, i.e., the corresponding fullness grades of the banana bunch.
2. The method for detecting the fullness of banana bunch according to claim 1, characterized in that: In step 1), when collecting images, the color image and the depth image of the banana bunch are collected at one time by using a kinect v3 camera. The banana bunches are of the same period, the same variety and the same orchard, so that the interference of time and place factors is excluded.
3. The method for detecting the fullness of banana bunch according to claim 1, wherein, In step 2), the preprocessing of the RGB-D image is specifically as follows: a. Color image: extract the R channel grayscale image in the R channel of the color image, perform adaptive threshold segmentation, then perform edge detection, convert the original image into a binary image, the foreground region pixel points are all 0, and the background region pixel points are all 255, then perform masking of the binary image and the original image, set all the background region pixel points to 255, and the foreground region pixel points remain unchanged; b. Depth image: mask the binary image in a and the depth image, set all the pixel points in the background region to 0, and the pixel points in the foreground region remain unchanged, then traverse each pixel point in the foreground region, find the point with a pixel value of 0, which is the point with missing depth information, correct the pixel value of this point to the average value of the pixel values of the non-0 pixel points in its eight adjacent pixel points, and then fill in all the missing pixel points of the depth information.
4. The method for detecting the fullness of banana fingers based on shape and color information according to claim 1, characterized in that, In step 3), the separation and identification of a single banana finger are specifically as follows: The color image is converted into a gray image, the contour lines between the fingers of the banana are found on the gray image and the depth image respectively by using the canny edge detection algorithm, and the contour lines on the two images are drawn on the same image. After morphological processing, the contour lines are closed, and each closed contour line is a finger of the banana. The incomplete banana fingers are removed by using area screening.
5. The method for detecting the fullness of banana bunch according to claim 1, characterized in that: In step 7), the fullness grade corresponding to each banana comb in the training set is input into the LDA classifier as a prediction value, and the color, finger length and middle cross-sectional width features corresponding to each banana comb are input into the LDA classifier as a feature vector. The LDA classifier generates a weight corresponding to each feature vector input, thereby calculating the fullness grade corresponding to the feature value, completing the training of the banana fullness discrimination model, and obtaining a trained banana fullness discrimination model based on the three features of color, finger length and middle cross-sectional width.
6. A banana comb fullness detection system based on shape and color information, characterized by, The method for detecting the fullness of banana fingers based on shape and color information according to any one of claims 1-5, comprising: a data acquisition module for acquiring banana fingers of at least two different fullness grades, collecting color images and depth images thereof, and aligning the images by using internal camera parameters. The two aligned images constitute a four-channel RGB-D image; a data preprocessing module for preprocessing the RGB-D image, including: performing adaptive threshold segmentation on the color image in the RGB-D image, extracting the region of interest, and performing neighborhood filtering on the depth image in the RGB-D image to fill in the missing pixel points; a single banana finger recognition module for performing edge detection on the preprocessed RGB-D image, separating and recognizing single banana fingers; a feature extraction module for extracting the average color value, finger length and middle cross-sectional width of the single banana finger; a feature processing module for averaging the three features of each banana finger to obtain the feature values of the whole banana comb, normalizing the feature values of all banana combs after obtaining the feature values of all banana combs, obtaining normalized banana comb data of different fullness grades, and randomly selecting a part of the banana comb data as a training set and the rest as a test set; a training module for training the LDA classifier as a banana fullness discrimination model, inputting the data in the training set into the LDA classifier for training, and obtaining a trained banana fullness discrimination model based on the three features of color, finger length and middle cross-sectional width; a test module for inputting the data in the test set into the trained banana fullness discrimination model to obtain accurate banana comb fullness detection results, i.e., the fullness grade of the banana comb.