Nondestructive testing method for kiwi fruit hardness prediction based on structured hyperspectral system

Abstract

The invention provides a structured hyperspectral detection system and method capable of nondestructively predicting the shelf life hardness of kiwi fruits. The method comprises the following steps: generating sine stripe light with the spatial frequency of 60cycles / m through computer programming, projecting the sine stripe light to a measured object, shooting three phase pictures of-2 / 3pi, 0 and 2 / 3pi by using a hyperspectral camera, demodulating the phase pictures into a complete picture, and obtaining structured light spectral information. The method comprises the following steps: selecting kiwi fruit samples stored at room temperature for different time, collecting structured hyperspectral data, and obtaining a true value of sample hardness through destructive inspection; the structured light data is demodeled and preprocessed, sample structured light spectral information is extracted, and a hardness prediction model is constructed. The result shows that the Rc2 of the optimal prediction model of the structured hyperspectral system for the hardness of the kiwi fruit is 0.8697, and the Rp2 is 0.8204, which are obviously higher than those of a common hyperspectral technology. The method is used for predicting the fruit hardness, has high accuracy, and is especially used for detecting the hardness of kiwi fruits with unobvious mature signs in the storage process.

Description

Technical Field

[0001] The present invention relates to the field of non-destructive detection of fruit quality, and particularly to a detection method for non-destructively predicting the change in hardness of kiwifruit during shelf life. Background Art

[0002] Kiwifruit is popular in the consumer market due to its good sensory, nutritional and flavor characteristics. Usually, to reduce quality damage during harvesting and transportation, kiwifruit is harvested at the commercial maturity stage, when it is hard and has a high acidity and is inedible. As a climacteric fruit, hardness is the most direct indicator for consumers when choosing kiwifruit, and it is also an important indicator related to juiciness, sugar-to-acid ratio, and texture characteristics. However, kiwifruit does not show visible signs of ripening, such as color or odor changes, during storage. Therefore, it is difficult for consumers to judge the hardness and the required ripening time of kiwifruit when purchasing. Currently, relevant researchers have tried to use optical non-destructive detection techniques (such as hyperspectral / multispectral imaging, visible near-infrared spectroscopy, etc.) to predict the hardness of fruits. The most studied non-destructive fruit hardness assessment method, such as hyperspectral imaging technology, can capture the light information that partially penetrates the fruit tissue and scatters back to the camera. However, due to the limited penetration depth of light and the interference of the fruit skin texture information, the accuracy of hyperspectral technology in detecting fruit hardness is limited. In recent years, researchers have also developed structured illumination reflectance imaging (SIRI) technology based on spatial frequency domain reflectance for evaluating the quality of agricultural products. SIRI uses structured light or patterned light to illuminate an object. By changing the spatial frequency of illumination, the penetration depth of light into the tissue can be controlled, thus achieving advantages such as depth-resolved characterization, enhanced spatial resolution, and contrast. The present invention integrates the advantages of hyperspectral and structured illumination reflectance imaging technologies and aims to explore their potential for kiwifruit hardness prediction. Summary of the Invention

[0003] The present invention provides a detection method for fruit hardness based on a structured hyperspectral system. This method has a high accuracy in detecting fruit hardness, especially for the detection of kiwifruit hardness with large interference from the peel color and insignificant ripening signs during storage. The detection method for fruit hardness based on a structured hyperspectral system according to the present invention, the hyperspectral system includes a hyperspectral camera, a light source, an optical fiber conduit, a projector, and a computer. The computer is respectively connected to the hyperspectral camera and the projector. The light source is connected to the projector through the optical fiber conduit. The hyperspectral camera, the projector, and the sample to be measured are placed in a closed black box, and the background of the black box does not reflect light. By programming the computer to generate sinusoidal gray scale stripes, the sinusoidal stripes are projected onto the surface of the sample to be measured through the projector. The hyperspectral camera is used to collect pictures with different phases at a specific frequency, and then the phase pictures are demodulated into complete pictures. An appropriate-sized region of interest is selected and the structured spectral values of the region of interest are obtained for fruit hardness prediction.

[0004] The present invention adopts a new structured light illumination mode. The basic principle is to measure the spatial modulation transfer function (S-MTF) to quantitatively describe the absorption and scattering processes of light in a medium. Different modes of light (usually sinusoidal fringe light) are generated through computer programming. The sinusoidal gray-scale fringes are projected onto the object to be measured by a projector, and then three phase images at a specific frequency are collected by a hyperspectral camera, and then the phase images are combined and transformed into a complete image.

[0005] The structured hyperspectral system described in the present invention adds a hyperspectral camera to the structured light system, combining the advantages of structured light and hyperspectral technologies to achieve the purpose of detecting early damage to fruits. The system consists of a hyperspectral camera, a light source, a computer, and image acquisition software. The whole device is placed in a sealed black box, specifically including: a hyperspectral camera (band range 400 - 1000 nm), an adjustable light source which is a 150W halogen tungsten lamp, an optical fiber conduit, a projector (DLi CEL5500-Fiber, Digital Light Innovations, Austin, TX, USA), and the image acquisition software is the software supporting the hyperspectral camera.

[0006] According to the three phase modes of sinusoidal structured light, sample information is collected at three phases of -2 / 3π, 0, and 2 / 3π respectively. After demodulating the image, complete image information can be obtained. First, the spectra after demodulation at each pixel point are obtained, and then the spectral values of all pixel points in the region of interest are averaged to obtain the average spectral value of the region of interest in the image.

[0007] Sinusoidal image demodulation is to obtain the alternating component (AC) from the three collected phase images. Through the cancellation of the sinusoidal illumination mode at -2 / 3π, 0, and 2 / 3π, it can be obtained by the following formula:

[0008]

[0009] Where I1, I2, and I3 are three different phase images, and the phase difference between each other is 2π / 3.

[0010] The process of collecting the diffuse reflection image of the sample with the structured hyperspectral system is as follows: Sinusoidal gray-scale fringes with different phases at a specific frequency are generated through MATLAB (The MathWorks, Inc., Natick, MA, USA), and then imported into the software supporting the projector (DLP CEL conductor control software). The projector is controlled to generate structured light and project it onto the surface of the sample to be measured. The hyperspectral camera collects three diffuse reflection images and saves them in the computer in real time.

[0011] As an improvement, a projector CEL5500 developed by a professional company (Digital Light Innovations, Austin, TX, USA) is adopted, and there is a supporting software to control the output of structured light, which has more stable performance and is more convenient to use than a simple commercial projector;

[0012] As an improvement, the hyperspectral camera is selected to perform visible near-infrared imaging in the range of 400 - 1000 nm, and this spectral range is suitable for fruit quality detection.

[0013] As an improvement, Python is used for image demodulation and calculating the average spectral value of the region of interest. First, three phase images of each sample are read, and the phase difference between the three phases is 2π / 3. When demodulating the phase pictures, complete image information can be obtained through phase cancellation.

[0014] The present invention provides a structured hyperspectral detection system and method for predicting the firmness of climacteric fruits that will not show any signs of ripening during storage. The system includes: a hyperspectral camera, a halogen tungsten lamp, an optical fiber conduit, a projector, a computer, and image acquisition software. Sinusoidal fringe light is generated through computer programming and projected onto the object to be measured through a projection device. The hyperspectral camera is used to capture three phase pictures of -2 / 3π, 0, and 2 / 3π, and then the phase pictures are demodulated into complete pictures and spectral information is obtained. The firmness prediction method is to select samples stored at room temperature for different times, manually extract the structured light spectral information of the region of interest of the samples, and select characteristic bands. Subsequently, the structured light images of the samples at the characteristic bands are extracted, image demodulation and preprocessing are performed, and firmness prediction is carried out according to the image processing algorithm. Compared with the ordinary illumination mode, this system has many advantages: First, by changing the frequency of the sine wave, illumination modes with different penetration depths can be achieved; second, it has multi-band spectral and image information; in addition, the structured light image has better contrast and resolution, and more internal spectral information of the sample can be obtained. Brief Description of the Drawings

[0015] Figure 1 It is a schematic structural diagram of the structured hyperspectral system of the present invention;

[0016] Figure 2 It is (A) the firmness change and (B) the firmness distribution of 300 kiwifruits at different storage times;

[0017] Figure 3 It is a schematic diagram of the structured spectrum of the kiwifruit sample

[0018] Figure 4 It is to use SHAP analysis based on different spectral preprocessing S-HSI to quantify the influence of spectral features on random forest prediction; Detailed Description of the Invention

[0019] A detection method for predicting the hardness of fruits based on a structured hyperspectral system, and the specific implementation method is as follows:

[0020] 1. Materials and Methods

[0021] The "Zespri" kiwifruits were purchased from a local wholesale market in Nanjing, China. The purchased samples were stored at room temperature (26 °C) for about 12 hours to reduce the spectral data differences caused by temperature changes. Three hundred kiwifruits with consistent maturity, shape, and size, and without defects and diseases were selected and divided into 5 groups, with 60 in each group. The kiwifruits were stored at room temperature for different times of 0 d, 2 d, 4 d, 6 d, and 8 d to obtain samples with different hardnesses. At different storage times, one group was randomly selected for structured hyperspectral data collection. After the structured hyperspectral data collection was completed, a portable GY-4 fruit hardness tester was used to measure the fruit hardness. The peel was removed by 1 mm along the equatorial region, and a test probe with a diameter of 7.9 mm was inserted into the outer side of the peel to a depth of 10 mm to measure the hardness. The maximum force recorded in newtons (N) was the hardness value.

[0022] 2. Spectral Information Collection

[0023] The structured hyperspectral system group consists of a hyperspectral camera ((4250, HinaLea Imaging, Emeryville, CA, USA), with a band range of 400 - 1000 nm and a spectral resolution of 2 nm), a 150W adjustable halogen tungsten lamp, an optical fiber conduit, a projector (DLi CEL5500-Fiber, Digital Light Innovations, Austin, TX, USA), and image acquisition software, as Figure 1 shown. Specific-frequency sine gray-scale stripes were generated through computer programming using MATLAB (The Mathworks, Inc., Natick, MA, USA) software. After preliminary experiments, a spatial frequency of 60 cycles / m was selected to collect structured light spectral information. The sine gray-scale stripes were uploaded in the bitmap format of 8-bit gray level to the supporting control software of the projector (DLPCEL conductor control software), and projected onto the kiwifruit samples to be measured through the projector. The light source was set to 150W, and the incident angle relative to the vertical axis was about 30°. Each kiwifruit sample was placed on a display table facing the camera. Under the sine illumination mode, 3 reflected images were obtained for each band from the sample, with the three-phase offsets being -2 / 3π, 0, and 2 / 3π respectively, to collect structured light spectral information.

[0024] 3. Image Demodulation

[0025] First, the acquired images are corrected to correct the non-uniform pattern projection caused by the projector. Based on the total reflection signal and dark signal response generated by a Spectralon plate (Labsphere, Inc, North Sutton, NH) with a reflectivity of 98% (i.e., the camera is covered to completely block any incident light), according to the following formula

[0026]

[0027] where Rn is the corrected phase image, Rr is the original noise image, Rb is the acquired dark field image, and Rw is the white board calibration image of the calibration reference plate.

[0028] Sine image demodulation is to obtain the alternating component (AC) image from the three phase pictures with the fringes removed by demodulation. Through the cancellation of the sine illumination pattern at -2 / 3π, 0, and 2 / 3π, it can be obtained by the following formula:

[0029]

[0030] where I1, I2, and I3 are three different phase images with a phase difference of 2π / 3 between each other.

[0031] 4. Hardness prediction

[0032] First, the size range of the region of interest of the kiwifruit sample is delimited to extract the structured spectrum of this region. A prediction model is constructed based on the true hardness of the samples measured by the GY-4 fruit hardness tester for hardness prediction and analysis. After extracting the alternating current image, a 250 * 200 pixel equatorial center region is selected as the region of interest (ROI) to obtain the average spectral value of all pixel points within the ROI. Therefore, a total of 300 groups of structured light spectrum data were collected in the experimental group. Each group of 60 samples was divided into a training set and a prediction set at a ratio of 2:1. The original spectral data was preprocessed using three different methods: multiplicative scatter correction (MSC), standard normal variate (SNV), and first derivative (1st) using Python, and then SVM and RF modeling analysis were performed to predict the hardness of kiwifruit. Two common regression metrics were used to evaluate the prediction performance of the model, and these metrics include the coefficient of determination (R c 2 ) and root mean square error of calibration (RMSEC) for the calibration set, and the coefficient of determination (R p 2 ) and root mean square error of prediction (RMSEP) for the test set. Generally, when R c 2 and R p 2 are close to 1 and RMSEC and RMSEP are close to 0, it indicates that the fitting performance of the model is better.

[0033] 5. Experimental Results

[0034] 5.1 Analysis of the Change in Hardness during the Shelf Life

[0035] As Figure 2 shows the distribution of the measured hardness values of each kiwifruit sample during the shelf life. The hardness values of the overall kiwifruit samples are approximately normally distributed, indicating that the sample selection and reference data measurement are suitable for constructing a calibration model for practical applications. As the shelf life extends, the hardness of the kiwifruit shows a downward trend. The hardness changes little at the first three sampling points (0, 2, 4 days), with an average hardness value of around 20 N and no significant difference among the three groups. On the 6th day of the shelf life, a significant decrease in hardness was observed. On the 8th day, the hardness of the kiwifruit further decreased to 8 N, and most of the kiwifruit had completely softened. Some kiwifruit also emitted the smell of fermented ethanol and were no longer suitable for storage. Generally, kiwifruit produces ethylene during the ripening process, which reduces the hardness. At the edible softness, kiwifruit produces autocatalytic ethylene, resulting in a relatively short (3 - 4 days) optimal edible period.

[0036] 5.2 Kiwifruit Hardness Prediction Results Based on Structured Hyperspectral

[0037] Table 1 shows the results of calibrating and predicting the hardness of kiwifruit using a structured hyperspectral imaging (S - HSI) model with different spectral pre - processing methods. From the data results, whether it is the SVM model or the RF model, the 1st pre - processing method has the best calibration and prediction performance for the model. The Rc of the SVM model using the 1st pre - processing method 2 is 0.8357, and the Rp 2 is 0.7689. The Rc of the RF model using the 1st pre - processing method 2 is 0.8697, and the Rp 2 is 0.8204. From the modeling results, the RF model performs significantly better than the SVM in predicting hardness; Figure 3 shows the distribution of SHAP values of the top 20 characteristic bands that have the greatest impact on the prediction model. These characteristic bands are based on the original spectrum and the pre - processed S - HSI spectrum. From the characteristic bands with the greatest output impact, after spectral pre - processing, the sample points become more concentrated, and most are distributed in the positive direction of the X - axis, indicating that the characteristic values have a stable positive impact on hardness prediction.

[0038] Table 1 Results of calibrating and predicting the hardness of kiwifruit using the S - HSI model with different spectral pre - processing methods

[0039]

Claims

1. A non-destructive testing method for predicting the hardness of kiwifruit based on a structured hyperspectral system, characterized in that: A structured hyperspectral system is used to project sinusoidal fringe structured light with a spatial frequency of 60 cycles / m onto the kiwifruit sample to be measured, and three-phase images are collected. The diffuse reflection image of the measured sample is demodulated using a three-phase demodulation algorithm through Python, and the spectral reflectance of the average structured hyperspectrum in the central equatorial region of the kiwifruit is extracted. A prediction model is established to predict the hardness of the kiwifruit.

2. The non-destructive detection method for kiwifruit hardness prediction based on structured hyperspectrum according to claim 1, wherein: Kiwifruits with different simulated hardnesses are prepared according to the following steps: First, kiwifruit samples with the same maturity, shape, and size and without defects are placed at room temperature (26 °C) for about 12 hours to reduce the spectral data differences caused by temperature changes. Then, the samples are evenly divided into 5 groups, with 60 samples in each group. One group is taken out after storing at room temperature for 0 d, 2 d, 4 d, 6 d, and 8 d respectively. These five groups of samples are used to simulate the different hardnesses of kiwifruits during the shelf life. After collecting the structured light hyperspectral data of the samples, the true hardness value of each kiwifruit sample is measured using a GY-4 hardness meter. The peel with a thickness of 1 mm is removed along the equatorial region, and a test needle with a diameter of 7.9 mm is inserted into the pulp to a depth of 10 mm. The maximum applied force is recorded as the hardness value, with the unit of Newton (N).

3. The non-destructive detection method for predicting the hardness of kiwifruit based on structured hyperspectrum according to claim 1, wherein the composition characteristics of the structured hyperspectrum system include: Hyperspectral camera, light source, fiber optic catheter, DLi CEL5500-Fiber projector, and computer, computer image acquisition software, airtight black box; Sinusoidal gray scale fringes with 60 cycles / m and a phase difference of 2π / 3 are generated through computer programming using MATLAB (The Mathworks, Inc., Natick, MA, USA) software, and a total of three phase fringe images are projected onto the kiwifruit sample to be measured through the projector.

4. The non-destructive detection method for predicting the hardness of kiwifruit based on structured hyperspectral as claimed in claim 1, wherein: Python is used to demodulate the images of each band. When demodulating the phase images, complete image information can be obtained through phase cancellation. The alternating component (AC) image is obtained through the following formula: where I1, I2, and I3 are three different phase images at a frequency of 60 cycles / m. A 250*250 pixel central equatorial region is selected as the region of interest (ROI), the spectral value of each pixel within the ROI is calculated, and the average spectral value of the ROI is further obtained, which is recorded as the spectral reflectance of the structured hyperspectrum of the sample.

5. The non-destructive testing method for kiwifruit hardness prediction based on structured hyperspectrum according to claim 1, wherein the feature of establishing the hardness prediction model is that the original spectral data is preprocessed based on Python using three methods: multiplicative scatter correction (MSC), standard normal variate (SNV), and first derivative (1st). The samples are divided into a training set and a prediction set in a ratio of 2:1, and support vector machine (SVM) and random forest (RF) models are constructed to predict the hardness of the kiwifruit, and SHapley Additive exPlanations (SHAP) is used to explain the prediction results of the regression model.

Images