Kiwifruit quality detection method based on deep learning and hyperspectral imaging technology

Through the extended morphology-non-local capsule network algorithm and near-infrared hyperspectral imaging technology of deep learning, the pixel-level data characteristics and spatial-spectral characteristics of kiwi fruit are extracted, which solves the invasive and insufficient stability of traditional detection methods, and realizes the problem of Fast and accurate lossless identification and classification of kiwi fruit quality.

CN114965346BActive Publication Date: 2024-11-29HEBEI UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210636956.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-07
Publication Date
2024-11-29
Estimated Expiration
2042-06-07

AI Technical Summary

Technical Problem

The prior art is difficult to accurately and quickly identify the maturity and quality of kiwifruit. Traditional detection methods are invasive, destructive and labor-intensive. Hyperspectral imaging models have problems with insufficient stability and generalization capabilities in data processing.

Method used

Using the extended morphology-non-local capsule network (EMP-NLCapsNet) algorithm based on deep learning, using near-infrared hyperspectral imaging technology, a classification model is constructed to realize the different storage conditions through pixel-level data feature extraction and spatial-spectral feature fusion. Quick identification of kiwi fruit quality changes under.

Benefits of technology

Non-destructive detection is realized, the original spatial structure of hyperspectral images is retained, the stability and generalization ability of the model are improved, and the storage conditions of fruits can be accurately classified in complex background environments, reducing the subjectivity and destructiveness of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114965346B_ABST
    Figure CN114965346B_ABST
Patent Text Reader

Abstract

The present invention relates to a kiwifruit quality detection method based on deep learning and hyperspectral imaging technology, and belongs to the technical field of fruit quality detection. The method first prepares N kiwifruits with good shape, normal color and no decay and numbers the storage conditions, respectively puts them into a low-temperature test box and room temperature to simulate different storage environments, and performs near-infrared hyperspectral imaging after storing them in the two storage environments for different periods of time. Secondly, the spectral reflectance of all pixel points in the fruit area of ​​interest under different storage conditions is extracted using ENVI5.3 software, and the black-and-white correction is performed to establish a pixel-level classification model to achieve rapid identification of fruits under different storage conditions. The detection method of the present invention is efficient, non-destructive, and saves human resources, and is suitable for real-time monitoring and industrial management of kiwifruit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of nondestructive detection of food quality based on spatial-spectral information, and in particular to rapid identification of quality changes of kiwifruit under different storage conditions based on deep learning and hyperspectral imaging technology. Background Art

[0002] Kiwifruit is soft in texture, sweet and sour in taste, and rich in various vitamins, amino acids and trace elements needed by the human body. It is known as the "king of fruits" and is therefore deeply loved by consumers. Kiwifruit is a respiratory climacteric fruit. It will quickly soften and rot after ripening. The storage time of the fruit after picking is extremely limited. At present, low-temperature refrigeration is mainly used to store and transport the fruit. The low temperature environment can inhibit the ethylene synthesis of the fruit and reduce the ripening and softening speed after picking. Due to the uneven surface of kiwifruit and the lack of obvious color changes, it is difficult to judge the maturity and quality of kiwifruit from the surface, and untrained inspectors will inevitably make wrong judgments. According to statistics, the waste caused by discarded rotten food in China accounts for about 20% of the total food production value, especially fresh foods such as fruits, vegetables and seafood. Most of them cannot guarantee storage conditions during storage and transportation, and inspectors cannot accurately judge the maturity of their fruits and rot. Therefore, researchers have adopted a variety of methods to monitor and control the quality and storage environment of fruits. Traditional testing methods require invasive treatment of fruit samples to extract physical and chemical parameters such as soluble solids, moisture content and pH, which is subjective, destructive and labor-intensive. Therefore, it will be of great significance to the fruit industry to extract a technology that can quickly and accurately distinguish the storage temperature and time of fruits.

[0003] As a new method to replace traditional detection methods, hyperspectral imaging (HSI) technology combines the technical advantages of spectral imaging and computer vision based on the red, green and blue color model. It can obtain the spectral and spatial characteristics of samples of the detected objects in the visible and near infrared (VNIR) and near infrared (NIR) spectral regions. Therefore, it is widely used in the fields of maturity assessment of vegetables and fruits, quantitative and qualitative analysis of chemical components, and mechanical damage detection. However, hyperspectral images usually have hundreds of dimensional spectra, and the spectral data are highly continuous. Too much redundant data will greatly reduce the performance of the model, which brings great challenges to the subsequent data analysis and processing. In order to reduce the amount of data processing, the current food non-destructive detection research usually extracts the average spectral reflectance (i.e., object-level data) in the region of interest (ROI) of the image as a training sample. However, compared with the model based on pixel-level data, although the model based on object-level data reduces the model training parameters and shortens the training time, it destroys the spatial characteristics of solid food samples, so the stability and generalization ability of the model still need to be improved.

[0004] At present, there are literatures on the quality inspection of kiwifruit's early hidden damage, frostbite and post-harvest maturity, but the fruit quality is only studied in the VNIR range (400-1000nm) related to compounds such as chlorophyll and anthocyanin, and a traditional machine learning model is established based on the object-level data set to establish the relationship between the spectrum and kiwifruit quality parameters (such as hardness, soluble solids, pH, etc.), and finally predict and classify the fruit quality. This traditional machine learning algorithm usually sacrifices the spatial features in the sample data to reduce the model training parameters, so the performance of the model still needs to be improved. In addition, since food will be affected by scattering from the background environment and the surface of the imaging sample during hyperspectral imaging, the spectral data may experience baseline shift, curve deformation and other phenomena, which increases the difficulty of traditional machine learning algorithms to analyze data. Therefore, the present invention proposes an extended morphological-nonlocal capsule network (EMP-NLCapsNet) algorithm based on deep learning, selects the NIR spectral region (900-1700nm) to analyze the quality changes of kiwifruit stored at different temperatures for different times, and extracts the pixel-level data features of the fruit to classify the quality of kiwifruit under different storage conditions. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects in the existing food non-destructive research, replace the object-level data that will destroy the sample space characteristics with pixel-level data, and replace the spectral characteristics in the existing research with spatial-spectral characteristics to build a classification model. The present invention uses HSI technology in the NIR (900-1700nm) spectral region to study the quality changes of kiwifruit under different storage conditions, and uses the extended morphological-non-local capsule network (EMP-NLCapsNet) algorithm based on deep learning to classify fruits stored for different times (0, 2, 4, 6 days) at different temperatures (low temperature (4°C, 75% relative humidity) and room temperature (18±2°C)), and analyze the fruit quality changes corresponding to spatial-spectral characteristics under different storage conditions (temperature and time).

[0006] To achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0007] A kiwifruit quality detection method based on deep learning and hyperspectral imaging technology, which uses deep learning and near-infrared hyperspectral imaging technology to achieve rapid detection of kiwifruit quality. The steps of the method are:

[0008] S1: Prepare N kiwifruits with good shape, normal color and no decay, and store the kiwifruits under different storage conditions. The different storage conditions include different ambient temperatures and different storage days under different ambient temperatures, and there are multiple kiwifruits under each storage condition.

[0009] S2: Use the NIR-HSI acquisition system to collect images of the kiwifruit under different storage conditions in S1, and finally obtain hyperspectral image data containing 256 spectral bands of the kiwifruit under different storage conditions;

[0010] S3: Perform black and white correction on the hyperspectral image obtained in step S2 to obtain a corrected reflection image, and then manually select ROI data of kiwifruit under different storage conditions from the corrected reflection image using ENVI 5.3 software; since there are multiple kiwifruits under different storage conditions, randomly select E×E pixels from the ROI data obtained from all kiwifruits under the same storage conditions as the full spectrum data under that condition, and all the full spectrum data form a pixel-level data set; wherein the pixel-level data set is divided into a training set, a validation set, and a test set in a ratio of 35:35:30.

[0011] S4: Build the EMP-NLCapsNet classification algorithm;

[0012] The EMP-NLCapsNet algorithm includes the EMP algorithm, the PCA algorithm and the non-local capsule network NLCapsNet. The PCA algorithm is used to reduce the dimension of the full spectrum data to obtain the PC image. The PC image with a cumulative variance exceeding 99% is used as the basic image for creating the EMP. Secondly, the structuring element (SE) is used to obtain the shape, boundary and structural features of the PC image, and then the PC image is switched four times to form an EMP data block with spatial structural features. The EMP data block is stacked with the PC image to form a spatial-spectral feature fusion data block that is input into the non-local capsule network NLCapsNet. The non-local capsule network NLCapsNet includes a convolutional layer, a primary capsule layer Primarycaps, and a digital capsule Digitcaps; the convolutional layer is composed of two two-dimensional convolution operations (Conv2D) and a non-local module, which can extract the global deep spatial-spectral features in the input image and organically connect the high-spectral features with spatial structure. The non-local module is located between the two two-dimensional convolution operations, and the three are connected in series in sequence; the non-local module is composed of three convolution kernels: It consists of a convolution operation (Conv2D) with a kernel of 1×1×C, which calculates the correlation between all pixels in the feature map generated by the first 2D convolution operation and adjusts the weights of all positions in the feature map to simulate the relationship between distant pixels.

[0013] The training set is used to build the EMP-NLCapsNet model, the validation set is used to adjust the parameters that control the model complexity and monitor whether the model is overfitting, and the test set is used to test the generalization ability of the model and obtain a trained pixel-level classification model.

[0014] S5: After the kiwifruit stored in an unknown environment is processed by steps S2-S3, the complete ROI data of the fruit is obtained, and the ROI data is sent to the trained pixel-level classification model to achieve rapid classification of kiwifruit under different storage conditions (the output is the storage temperature and storage time of the fruit), so as to reflect the different qualities of the fruit with the storage conditions.

[0015] Furthermore, in step S3, the ROI data of kiwifruit under different storage conditions is obtained by selecting a pixel range of 30×30 to 60×60 pixels in the middle area of ​​the fruit from the corrected reflection image as ROI data; preferably, the size of the ROI data is 50×50 pixels; in step S4, the structural element SE is a circle with a radius of 2 pixels.

[0016] The near infrared wavelength of the NIR-HSI acquisition system is 900-1700 nm, and different storage conditions include storage for different time periods (0, 2, 4, 6 days) at low temperature (4° C., 75% relative humidity) and room temperature (18±2° C.).

[0017] The process of black and white correction in step S3 is: obtain a white correction image and a black correction image respectively through a white correction plate with a reflectivity of 99% of a standard diffuse reflection surface and a dark environment without light source and completely covering the lens; compare and correct each pixel point in the hyperspectral image with the obtained black and white correction images according to formula (1) to obtain a corrected reflection image:

[0018]

[0019] Among them, I R is the corrected reflection image, I raw is a hyperspectral image, I dark and I white are the light intensities of the black and white correction images respectively, and a and b are the spatial coordinates of the image.

[0020] The process of the non-local module is: the pixel point in the feature map is represented by x in ={x1,x2,…x z}, each pixel x i Each represents a C-dimensional vector, where z is the number of pixels in the feature map, i is an integer from 1 to z, and C is the number of channels in the feature map; the output of the non-local module is y out ={y1,y2,…,y z}, the similarity s between every two pixels is obtained by formula (2). Therefore, the non-local module can calculate the correlation between all pixels in the feature map generated by the first two-dimensional convolution operation and adjust the weights of all positions to simulate the relationship between distant pixels;

[0021]

[0022] Where x is the input data, i is the position index of the output data, and j is the index that enumerates all possible positions; θ(·), and g(·) both represent a two-dimensional convolutional layer, and the convolution kernels are Therefore, after two dot multiplication operations and one softmax regularization operation, the feature map obtained is Secondly, we use a 1×1×C convolution kernel to perform spatial convolution on the feature map, and obtain a feature map of H×W×C; finally, we combine the obtained feature map with the input image x in Do the addition operation to get the output y out .

[0023] The first layer in the NLCapsNet network is a standard two-dimensional convolution operation (Conv1) with a convolution kernel of 5×5. It obtains deep features by convolving the input data block and feeds the features into the non-local module. After the non-local module, a two-dimensional convolution operation (Conv2) with a convolution kernel of 3×3 is set to further extract deep features based on global information, reduce the number of primary capsules, and improve the running speed of dynamic routing. A total of 32 capsules are output in the PrimaryCaps layer, and each capsule outputs an 8-dimensional vector generated by a 3×3 convolution kernel. Secondly, the output capsule is fed into the Digitcaps layer for encoding data information and final classification through the dynamic routing algorithm. In the Digitcaps layer, l capsules representing categories are set, and each capsule has a 16-dimensional vector representing the entity class. The vector length represents the existence probability of the current input target entity. Finally, the vector output from the capsule network is fed into the decoder, and the feature vector learned by the capsule is reconstructed into the input image. The output node of the EMP-NLCapsNet algorithm is 4, the number of iterations of dynamic routing and network is set to 3 and 100 (epochs) respectively, the learning rate and attenuation of the network are 0.001 and 0.9 respectively, the adjacent data block pixel area is set to 11×11, and the Adam optimizer is used to update the parameters. The performance of EMP-NLCapsNet is evaluated by the overall accuracy OA and Kappa coefficient, where OA is the number of correctly classified pixels in the overall test set, and the Kappa coefficient is a metric that is weighted by classification accuracy. The mathematical expression is shown in equations (3) and (4).

[0024]

[0025]

[0026] Where T is the total number of pixels in the test set; l is the total number of categories; u represents each category; c uu is the pixel point that is correctly classified; c u+ With c +u is the sum of misclassified pixels, where u+ represents pixels in the uth row that are classified into other columns, and +u represents pixels in the uth column that are classified into other rows, both of which are incorrect classifications.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) The detection method of the present invention uses the EMP-NLCapsNet algorithm to establish a pixel-level classification model. During the detection, there is no need to perform invasive detection on the kiwifruit to extract the internal chemical composition content, and the sample will not be damaged during the image acquisition process of the NIR-HSI system. It is a non-destructive detection method.

[0029] (2) The detection method of the present invention selects the imaging spectral range of the NIR-HSI system. This spectral range is concentrated in the 900-1700nm band that can reflect the content of organic matter such as water, fat and amino acids in food, and can intuitively reflect the changes in the organic matter content of fruits under different storage conditions. In addition, when operating the NIR-HSI system, it only needs to place the sample to be tested in the center of the mobile platform, and after setting the imaging parameters, hyperspectral data collection can be performed, which has the characteristics of simple operation.

[0030] (3) The EMP-NLCapsNet algorithm proposed in the present invention is composed of an extended morphological profile (EMP) algorithm, a principal component analysis (PCA) algorithm and a non-local capsule network (NLCapsNet). EMP and PCA are used to extract the spatial structure and spectral features of kiwifruit hyperspectral images, respectively, and the extracted feature data blocks are stacked and input into the non-local capsule network (NLCapsNet) to achieve quality classification. Compared with the current food non-destructive testing technology, the EMP-NLCapsNet algorithm builds a network based on a pixel-level dataset, retaining the original spatial structure of the hyperspectral image. In addition, it solves the problems of the pooling operation in the traditional convolutional neural network losing the image spatial features and the overfitting phenomenon in the case of insufficient data, and adds a non-local module to simulate the long-distance dependency in the hyperspectral image, so that the model has strong robustness even in a dataset with complex background environment, small samples and overlapping target parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the NIR-HSI system constructed in the present invention.

[0032] Figure 2 It is the spectral characteristic of the fresh kiwi fruit (100 pixels randomly selected from storage D0 of the control group) in the NIR (900-1700nm) spectral range of the present invention.

[0033] FIG3( a ) is a graph showing the average spectral curve changes of kiwifruit at different storage times at room temperature in the present invention;

[0034] FIG3( b ) is a graph showing the average spectral curve changes of kiwifruit stored at low temperature (4° C.) for different periods of time in the present invention.

[0035] Figure 4 1 is a structural flow chart of the EMP-NLCapsNet algorithm in the present invention, in which the data in the figure takes the kiwifruit stored in D0 of the control group as an example.

[0036] Figure 5It is the EMP established based on the first four PC images of the intact fruits of the randomly selected control group after storage at room temperature for D0-D6 (from top to bottom, the control group was stored at room temperature for D0, D2, D4 and D6, respectively).

[0037] Figure 6 It is a schematic diagram of the structure of the non-local module in the present invention.

[0038] Figure 7 These are the accuracy and loss curves of the training set and validation set (Training and Validation) of the EMP-NLCapsNet algorithm in the present invention.

[0039] Figure 8 These are the false color and PC1-PC4 images of the interior of the fruit stored at room temperature D6 and at low temperature D2-D6 in the present invention.

[0040] Fig. 9 It is a time-varying classification diagram obtained by feeding the complete ROI of fruits stored at low temperature for different periods of time into the EMP-NLCapsNet model. DETAILED DESCRIPTION

[0041] The present invention is further explained below in conjunction with the embodiments and drawings, but this is not intended to limit the scope of protection of the present application.

[0042] The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology of the present invention comprises the following steps:

[0043] S1: Prepare N kiwifruits with good shape, normal color and no rot, and place the samples in different storage environments. Different storage conditions include different ambient temperatures and different storage days under different ambient temperatures, and the number of kiwifruits under each storage condition is multiple;

[0044] 80 Cuixiang kiwifruits from Zhouzhi County, Shaanxi Province, with uniform shape and size, no rot, no scars and normal color were prepared. All fruits were divided into 4 groups, each with 20 fruits, and the storage condition categories were numbered. 20 fresh fruits were randomly selected as the control group and stored at room temperature (18±2℃), and the rest of the fruits were placed in a low-temperature test box. The temperature and relative humidity of the low-temperature environment in the test box were set to 4℃ and 75%, respectively. A group of samples were removed from the test box every two days (i.e., storage days 0, 2, 4, and 6, D0, D2, D4, and D6), and hyperspectral imaging was performed together with the control group fruits stored at room temperature for the same time.

[0045] S2: Use the NIR-HSI (near infrared-hyperspectral imaging) acquisition system to collect hyperspectral images of kiwifruit samples, and finally obtain hyperspectral images containing 256 spectral bands;

[0046] Hyperspectral image acquisition is performed in a dark environment to avoid light interference in the data acquisition environment. Figure 1 As shown, the spectral range is selected as the near-infrared region. Figure 1 (a) is the composition of the existing hyperspectral imaging equipment, including a mobile platform, a lens, a light source, and a fixed frame, (b) is the information composition of the hyperspectral image, and (c) is the average spectral curve of the sample ROI. In this embodiment, the kiwi fruit sample is placed in the middle of the mobile platform, the distance between the lens and the sample is set to 100mm, the moving speed of the mobile platform is set to 8mm / s, the exposure time is set to 20ms, the spectral range is 900-1700nm, and the image is collected at a spectral resolution of 5nm, and finally a hyperspectral image of 256 bands is obtained. Since the hyperspectral image is presented as a three-dimensional data cube containing two-dimensional spatial information and one-dimensional spectral information, each dimension in its spectral dimension is a two-dimensional grayscale image of a single wavelength, so the hyperspectral image acquired by NIR-HSI has a total of 256 dimensions.

[0047] S3: In order to eliminate the influence of uneven illumination and random noise caused by environmental factors during the imaging process, the acquired hyperspectral image needs to be corrected in black and white, and finally the corrected reflection image is obtained. The hyperspectral image obtained in step S2 is corrected in black and white to obtain the corrected reflection image, and then the ROI data of kiwifruit under different storage conditions is manually selected from the corrected reflection image by ENVI 5.3 software, and the pixel range of the middle area of ​​the fruit is preferably 30×30~60×60 pixels; since there are multiple kiwifruits under different storage conditions, E×E pixels are randomly selected from the ROI data obtained from all kiwifruits under the same storage conditions as the full spectrum data under this condition and form a pixel-level data set. Among them, the pixel-level data set is divided into training set, validation set and test set according to the ratio of 35:35:30.

[0048] The black and white calibration image was obtained by using a white calibration plate with a reflectivity of 99% on a standard diffuse reflection surface (Labsphere, North Sutton, NH) and a dark environment without light source and completely covering the lens. Then, each pixel in the fruit image was compared and calibrated with the obtained black and white calibration image, as shown in formula (1):

[0049]

[0050] Among them, I R is the corrected reflection image, I rawis the original light intensity image of the sample under test, I dark and I white are the light intensities of the black and white corrected images respectively, and a and b are the spatial coordinates of the image.

[0051] Since the ROI extraction position of the fruit with skin will affect the performance of the classification model, in order to simultaneously obtain the spatial and spectral information of the peach core, seeds and pulp inside the kiwi fruit, the ENVI 5.3 software is used in this embodiment to extract a ROI containing 50×50 pixels in the central part of the whole fruit. Therefore, in this embodiment, there are 20 fruits in one group, and there are 1000×50 pixels in the ROI of each group of fruits. In order to reduce the amount of data and make the training data universal, 50×50 pixels are randomly selected from the 1000×50 pixels in each group of fruit ROI as full spectrum data, and a pixel-level data set is formed. Therefore, in the present invention, there are 200×50 pixel data for different storage temperatures (one storage temperature has four storage time data), 35% of the data is used as a training set to adjust the model parameters, 35% is used as a validation set to adjust the parameters of the control model complexity, and 30% is used as a test set to observe the generalization ability of the model.

[0052] In the control group storage D0, 100 pixels of the spectral curve are randomly selected, such as Figure 2 As shown, in Figure 2 The secondary absorption frequency doubling of OH in water and carbohydrate-related substances make the spectrum have main absorption peaks near 970nm, 1100nm and 1450nm. The CH stretching in anthocyanin compounds also makes the spectrum peak at 1300nm. The spectral characteristics of fresh kiwifruit can be observed.

[0053] Figure 3 shows the average reflectance spectra of fresh fruits at room temperature and low temperature (4°C) for different storage days. In the figure, D0, D2, D4, and D6 correspond to storage days 0, 2, 4, and 6, respectively. From the average spectrum in Figure 3 (a), it can be seen that the longer the storage time at room temperature, the lower the reflectance of the fruit. In addition, after the fruit is stored for D4, the absorption peak of the curve at 1200-1300nm becomes gentle, and the absorption peak in the range of 1300-1500nm disappears and intersects with the curves of D0 and D2. This phenomenon may be related to the OH stretching vibration, the second and third overtones of water, and the degradation of phenols, resulting in changes in the content of certain organic matter in the fruit. In addition, the changes in the spectral curve of the fruit stored at low temperature in Figure 3 (b) are similar to those under room temperature. Due to pigment damage and tissue atrophy, the longer the sample is stored at low temperature, the lower the reflectance of the fruit. However, there is no great difference in the average spectral curve of the samples stored at low temperature, which can prove from the perspective of spectral reflectance that the internal substances of the fruit in the range of 1200-1500nm have not disappeared.

[0054] S4: Build the EMP-NLCapsNet classification algorithm;

[0055] The EMP-NLCapsNet algorithm is composed of the EMP algorithm, the PCA algorithm and the non-local capsule network (NLCapsNet). First, PCA is used to reduce the dimension of the full spectrum data H×W×B in the spectral dimension (256) to generate the principal component (PC) image H×W×P containing the most input image information, and the cumulative explained variance of the PC image exceeds 99%. Secondly, morphological operations are performed on the generated PC images to extract the spatial structure in the image to generate EMP data blocks H×W×M, and the EMP data blocks are stacked with the PC images. The stacked data H×W×C is sent to NLCapsNet as the spatial-spectral features of the hyperspectral image for fruit quality classification under different storage environments. Among them, H, W and B are the height, width and band of the full spectrum data respectively, P is the dimension of the full spectrum data after dimensionality reduction, M is the dimension of the generated EMP data block, and C is the dimension of the fused spatial-spectral feature data block.

[0056] The morphological operations involved in the EMP algorithm require the use of a structuring element (SE) of known size and shape to obtain the shape, boundary and structural features of the image. The basic operations are erosion, dilation, opening and closing operations, and the mathematical definitions are shown in equations (2)-(5). Among them, erosion and dilation are the minimum and maximum values ​​in SE, respectively; the opening operation is a process of first erosion and then dilation, which can expand the erosion range of the image; the closing operation is a process of first dilation and then erosion, which can erode the dilated image to restore the dilated image structure. In order to reduce the correlation between spectra, Fauwel et al. [1] The full spectrum data is processed by dimensionality reduction, and the low-dimensional cube is used as the basic image for creating the morphological profile (MP). The opening and closing operations are used to amplify or reduce the light and dark structures in the image, so that the brightness of the created MP is uniform and the shape noise introduced by the outside world is reduced. The generated data block is the EMP data block, and the mathematical expression is shown in Equations (6) and (7).

[0057] In this embodiment, the PCA algorithm is used to reduce the dimension of the preprocessed data, and the PC images with a cumulative variance exceeding 99% (that is, the first 4 PC images) are used as the basic images for creating EMP. Secondly, the PC images are switched four times based on the circular SE with a radius of 2 to form 36 EMPs with spatial structural characteristics (9 for each principal component image, and 36 for the first four principal component images). In order to highlight the changes in fruit quality, the complete image of the fruit is used here instead of the data ROI in the training set for illustration. Figure 5The EMP is based on the first four PC images after the whole fruit of the randomly selected control group was stored at room temperature for D0-D6. It can be seen from the figure that with the extension of storage time, the shape of the fruit changes from round to shriveled, and the water content is significantly reduced, which is closely related to the fruit water content at 988, 1032, 1136 and 1164nm in the spectral curve, indicating that EMP can effectively extract the spatial structural features of hyperspectral images. Finally, the EMP data block is stacked with the PC image to form a fused spatial-spectral feature data block. In this embodiment, 40 spatial-spectral features are formed and fed into NLCapsNet for deep feature learning and classification.

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] Among them, I is the processed image, B is SE, B X is the center value of SE at pixel x, k is the input position index, n is the number of opening and closing operations, β is the closing operation, γ is the opening operation, and P is the dimension of the image after dimensionality reduction.

[0065] Capsule Network (CapsNet) is mainly composed of traditional convolutional layers, primary capsule layers (Primarycaps), and digital capsules (Digitcaps). There are a number of capsule basic units in the primary capsule layer (Primarycaps) and digital capsules (Digitcaps). Compared with traditional CNN, the input and output data of CapsNet are represented in the form of vectors representing specific entity types, and the dynamic routing algorithm replaces the pooling layer in the traditional architecture to make up for the defect of losing spatial information. In addition, the vector direction of the input and output data represents the target attribute of the entity, and the length represents the probability of the entity's existence. In CapsNet, the capsules in the same layer predict the instantiation parameters of the capsules in the lower layer through a certain transformation matrix. The dynamic routing algorithm keeps the prediction results of multiple capsules in this layer consistent and makes the capsules in the lower layer active. The dynamic routing algorithm is the core of CapsNet and works in the Digitcaps layer. Its dynamicity brings powerful explanatory power to the network algorithm and improves the network's anti-interference ability in noisy data sets. The essence of this algorithm is a clustering algorithm, which can iteratively adjust the weight coefficient between the child capsule and the parent capsule and allow them to be dynamically connected, so that the child capsule is more inclined to transmit information to the parent capsule with a larger coupling coefficient.

[0066] In addition, NLCapsNet adds a non-local module based on CapsNet, which can take the weighted sum of features at all positions in the input feature map as the response of a certain position, thus simulating the long-range dependency in the hyperspectral image, making the model robust even in datasets with complex background environments, small samples, and overlapping targets. The working principle of the non-local module in NLCapsNet is as follows: Figure 6 As shown, it is mainly used to emphasize the relevant information of each pixel in the fused spatial-spectral feature map generated by Conv1. The pixel point in the feature map is represented by x in ={x1,x2,…x z}, each pixel x i Both represent a C-dimensional vector, where z is the number of pixels in the feature map and C is the number of channels in the feature map. The output of the non-local module is y out ={y1,y2,…,y z}, the similarity s between every two pixels is expressed as formula (8). Therefore, the non-local module can calculate the correlation between all pixels in the feature map generated by Conv1 and adjust the weights of all positions to simulate the relationship between distant pixels.

[0067]

[0068] Where x is the input data, i is the position index of the output data, and j is the index that enumerates all possible positions; θ(·), and g(·) both represent Figure 6 A traditional two-dimensional convolutional layer (Conv2D) in the convolution layer, the convolution kernels are Therefore, after two point multiplication operations and one softmax regularization operation, the feature map obtained is Secondly, a spatial convolution operation is performed on the feature map using a 1×1×C convolution kernel to obtain a feature map of H×W×C. Finally, the obtained feature map is compared with the input image x in Do the addition operation to get the output y out .

[0069] The network architecture of EMP-NLCapsNet is shown in Table 1. The network part uses two two-dimensional convolution operations and a non-local module for feature learning, which can extract the global deep spatial-spectral features in the input image and organically connect the hyperspectral features with spatial structure. First, the stacked data H×W×C is converted into a three-dimensional data block of S×S×C as input data and fed into the NLCapsNet network for classification, where S represents the size of the convolution kernel. The first layer in the NLCapsNet network is a standard two-dimensional convolution operation (Conv1) with a convolution kernel of 5×5. The deep features can be obtained by convolving the input data block and feeding the features into the non-local module. In addition, a two-dimensional convolution operation (Conv2) with a convolution kernel of 3×3 is set after the non-local module to further extract deep features based on global information, reduce the number of primary capsules, and improve the running speed of dynamic routing. A total of 32 capsules are output in the PrimaryCaps layer, and each capsule outputs an 8-dimensional vector generated by a 3×3 convolution kernel. Secondly, the output capsule is fed into the Digitcaps layer for encoding data information and final classification through the dynamic routing algorithm. In the Digitcaps layer, l capsules representing categories are set, and each capsule has a 16-dimensional vector representing the entity class. The length of the vector represents the existence probability of the current input target entity. Finally, the vector output from the capsule network is fed into the decoder to reconstruct the feature vector learned by the capsule into the input image. In addition, the EMP-NLCapsNet network has 4 output nodes, the number of dynamic routing and network iterations is set to 3 and 100 times (epochs) respectively, the learning rate and attenuation of the network are 0.001 and 0.9 respectively, the adjacent data block pixel area is set to 11×11, and the Adam optimizer is used to update the parameters. Figure 7It is the accuracy and loss curve of the training set and validation set (Training and Validation) of the EMP-NLCapsNet algorithm in the present invention. In addition, the performance of EMP-NLCapsNet is evaluated by the overall accuracy (OA) and Kappa coefficient. Among them, OA is the number of correctly classified pixels in the overall test set, and the Kappa coefficient is a metric for weighted classification accuracy, and the mathematical expression is shown in formulas (9) and (10). Finally, the overall classification performance of EMP-NLCapsNet is shown in Table 2.

[0070]

[0071]

[0072] Where T is the total number of pixels in the test set; l is the total number of categories; u represents each category; c uu is the pixel point that is correctly classified; c u+ With c +u is the sum of misclassified pixels, where u+ represents pixels in the uth row that are classified into other columns, and +u represents pixels in the uth column that are classified into other rows, both of which are incorrect classifications.

[0073] The training set is used to build the EMP-NLCapsNet model, the validation set is used to adjust the parameters that control the model complexity and monitor whether the model is overfitting, and the test set is used to test the generalization ability of the model and obtain a trained pixel-level classification model.

[0074] Table 1 Convolution type and output mapping dimension of each layer in the EMP-NLCapsNet algorithm

[0075]

[0076] Table 2 Overall classification results of fruits under different storage conditions by EMP-NLCapsNet algorithm

[0077]

[0078] S5: After obtaining the pixel-level dataset, sensory evaluation of fruit samples under different storage conditions was carried out according to the Chinese national standards GB / T 40743-2021 and NY / T1794-2009 to verify the accuracy of the classification model; the verification results showed that the storage days and storage environment do affect the quality of the fruit, and the classification of storage conditions can reflect the changes in fruit quality.

[0079] According to the national standards GB / T 40743-2021 Kiwifruit Quality Grade and NY / T 1794-2009 Kiwifruit Grade Specification, the sensory characteristics of kiwifruit quality were evaluated, as shown in Table 3. Fresh kiwifruit has a regular shape, a clean and intact appearance, no rot, wrinkles or soft areas, no odor, and no insect pests. Symptoms of deterioration include patches on the surface, incomplete ring particles or woody forms, soft skin, rotten parts, lignified and watery flesh inside, fermented wine smell, sour and astringent. The quality of the fruit after storage can be roughly divided into three types: the fruit skin is moderately hard and soft, exudes a fresh fragrance, and is considered to be the best time to eat the fruit; the skin becomes soft and has a fermented wine smell. Although the quality is reduced, it can still be eaten; the skin rots and even has a sour and astringent wine smell, completely losing its edible value.

[0080] In S5, kiwifruit stored at room temperature for D6 and stored at low temperature for D2-D6 were cut axially, and the internal flesh was subjected to sensory analysis. The false color image and the first four PC images after PCA dimensionality reduction were established based on the information of the 29th, 42nd and 89th bands of the spectrum. Figure 8 As shown. The fruit stored at room temperature for D6 had lignified flesh inside, exuding the smell of fermented wine, and the skin became soft and began to rot. The fruit stored at low temperature for D2 had a soft skin and a fresh fragrance, which was the best time to eat the fruit; although the fruit stored for D4 and D6 lost some water, the skin was still hard, and no obvious difference could be seen. Figure 8 The cumulative explained variance of the first four PC images presented in the figure is 0.9972. The water content of the fruit stored in a low temperature environment has not decreased significantly, resulting in different degrees of overlap in PC3. Therefore, no obvious changes can be found in the external features and internal flesh of the fruit presented in the false color image. On the contrary, the fruit stored at room temperature has rotted, with basically no reflection, and the color has become dim in PC1. Therefore, the first four PC images can reflect the quality changes of the fruit stored at different temperatures for different times, which further verifies that HSI technology can evaluate the internal physical and chemical properties of kiwifruit through spectral information.

[0081] Table 3 Sensory characteristic standards for kiwifruit quality evaluation

[0082]

[0083] S6: After the kiwifruit stored in different environments is processed in steps S2-S3, the complete ROI data of the fruit is obtained, and the ROI data is sent to the trained pixel-level classification model to achieve rapid classification of kiwifruit under different storage conditions.

[0084] Since there is no obvious difference in the sensory characteristics of fruits stored at low temperatures for different periods of time, the EMP-NLCapsNet model is used to establish the relationship between deep hyperspectral features (full spectrum data) and fruits stored at low temperatures for different periods of time. In the present invention, the complete ROI data of fruits stored at low temperatures for different periods of time are randomly selected, and the EMP-NLCapsNet model established using 40 spatial-spectral features is still used to predict the storage time of each pixel in the ROI. Each pixel will present a classification result due to its different spectral reflectance value and spatial structure, and thus appear as a different color in the generated distribution map, such as Fig. 9 As shown. The color scale from 1 to 4 in the label represents the storage days from low to high (D0, D2, D4 and D6). In order to compare the ROI data with the fruits in daily life, the information of the 29th, 42nd and 89th bands of the spectrum was used to create a false color image of the kiwifruit ROI. Although people cannot see the internal quality of fruits stored at low temperatures for different periods of time with the naked eye, the differences between fruits stored at low temperatures for different periods of time can be clearly observed based on the classification diagram of the fruits.

[0085] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A kiwifruit quality detection method based on deep learning and hyperspectral imaging technology, the steps of the method are: S1: preparing N kiwifruits with good shapes, normal colors and no decay, and storing the kiwifruits under different storage conditions; the different storage conditions include different ambient temperatures and different storage days under different ambient temperatures, and the number of kiwifruits under each storage condition is multiple; S2: Use the NIR-HSI acquisition system to collect images of the kiwifruit under different storage conditions in S1, and finally obtain hyperspectral image data containing 256 spectral bands of the kiwifruit under different storage conditions; S3: Perform black and white correction on the hyperspectral image obtained in step S2 to obtain a corrected reflection image, and then manually select ROI data of kiwifruit under different storage conditions from the corrected reflection image using ENVI5.3 software. Since there are multiple kiwifruits under different storage conditions, E×E pixels are randomly selected from the ROI data obtained from all kiwifruits under the same storage condition as the full spectrum data under the storage condition. The full spectrum data under all storage conditions form a pixel-level data set; wherein, Divide the pixel-level dataset into training set, validation set and test set; S4: Build the EMP-NLCapsNet classification algorithm; The EMP-NLCapsNet algorithm includes the EMP algorithm, the PCA algorithm and the non-local capsule network NLCapsNet. The PCA algorithm is used to reduce the dimension of the full spectrum data to obtain the PC image, and the PC image with a cumulative variance exceeding 99% is used as the basic image for creating the EMP. A structural element SE is used to obtain the shape, boundary and structural features of the PC image, and then the PC image is switched four times to form an EMP data block with spatial structural features. The EMP data block is stacked with the PC image to form a spatial-spectral feature fusion data block, which is input into the non-local capsule network NLCapsNet. The non-local capsule network NLCapsNet includes a convolution layer, a primary capsule layer Primarycaps and a digital capsule Digitcaps; the convolution layer is composed of two two-dimensional convolution operations and a non-local module, which can extract the global deep spatial-spectral features in the input image and organically connect the high-spectral features with spatial structure. The non-local module is located between the two two-dimensional convolution operations, and the three are connected in series in sequence; the non-local module includes three convolution kernels: A 2D convolution layer with a convolution kernel of 1×1×C is used to calculate the correlation between all pixels in the feature map generated by the first 2D convolution operation, and adjust the weights of all positions in the feature map to simulate the relationship between distant pixels. The training set is used to build the EMP-NLCapsNet model, the validation set is used to adjust the parameters that control the model complexity and monitor whether the model is overfitting, and the test set is used to test the generalization ability of the model and obtain a trained pixel-level classification model. S5: After the kiwifruit stored in an unknown environment is processed in steps S2-S3, complete ROI data of the fruit is obtained, and the ROI data is sent to a trained pixel-level classification model to achieve rapid classification of kiwifruit under different storage conditions.

2. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 1 is characterized in that: The method for acquiring ROI data of kiwi fruit under different storage conditions in step S3 is as follows: a pixel range of 30×30 to 60×60 pixels in the middle area of ​​the fruit is selected from the corrected reflection image as ROI data; the structural element SE in step S4 is a circle with a radius of 2 pixels.

3. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 2 is characterized in that: In step 3, the size of the ROI data is 50×50 pixels.

4. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 1 is characterized in that: The near infrared wavelength of the NIR-HSI acquisition system is 900-1700 nm, and different storage conditions include storage for different time periods at low temperature and room temperature.

5. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 1 is characterized in that: The process of black and white correction in step S3 is: obtain a white correction image and a black correction image respectively through a white correction plate with a reflectivity of 99% of a standard diffuse reflection surface and a dark environment without light source and completely covering the lens; compare and correct each pixel point in the hyperspectral image with the obtained black and white correction images according to formula (1) to obtain a corrected reflection image: Among them, I R is the corrected reflection image, I raw is a hyperspectral image, I dark and I white are the light intensities of the black and white correction images respectively, and a and b are the spatial coordinates of the image.

6. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 1 is characterized in that: The process of the non-local module is: the pixel point in the feature map is represented by x in ={x1,x2,…x z }, each pixel xi represents a C-dimensional vector, where z is the number of pixels in the feature map, i is an integer from 1 to z, and C is the number of channels in the feature map; the output of the non-local module is y out ={y1,y2,…,y z }, the similarity s between every two pixels is obtained by formula (2). Therefore, the non-local module can calculate the correlation between all pixels in the feature map generated by the first two-dimensional convolution operation and adjust the weights of all positions to simulate the relationship between distant pixels; Where x is the input data, i is the position index of the output data, and j is the index that enumerates all possible positions; θ(·), and g(·) both represent a two-dimensional convolutional layer, and the convolution kernels are Therefore, after two dot multiplication operations and one softmax regularization operation, the feature map obtained is Secondly, we use a 1×1×C convolution kernel to perform spatial convolution on the feature map, and obtain a feature map of H×W×C; finally, we combine the obtained feature map with the input image x in Do the addition operation to get the output y out .

7. The kiwifruit quality detection method based on deep learning and hyperspectral imaging technology according to claim 1 is characterized in that: The first layer in the NLCapsNet network is a standard two-dimensional convolution operation with a convolution kernel of 5×5. The deep features are obtained by convolving the input data block and the features are sent to the non-local module. After the non-local module, a two-dimensional convolution operation with a convolution kernel of 3×3 is set to further extract deep features based on global information, reduce the number of primary capsules, and improve the running speed of dynamic routing. A total of 32 capsules are output in the PrimaryCaps layer, and each capsule outputs an 8-dimensional vector generated by a 3×3 convolution kernel. Secondly, the output capsule is fed into the Digitcaps layer for encoding data information and final classification through the dynamic routing algorithm. In the Digitcaps layer, l capsules representing categories are set, and each capsule has a 16-dimensional vector representing the entity class. The vector length indicates the existence probability of the current input target entity; finally, the vector output from the capsule network is fed into the decoder, and the feature vector learned by the capsule is reconstructed into the input image; the output node of the EMP-NLCapsNet algorithm is 4, the number of iterations of dynamic routing and network is set to 3 and 100 respectively, the learning rate and attenuation of the network are 0.001 and 0.9 respectively, the adjacent data block pixel area is set to 11×11, and the Adam optimizer is used to update the parameters; the performance of EMP-NLCapsNet is evaluated by the overall accuracy OA and Kappa coefficient, where OA is the number of correctly classified pixels in the overall test set, and the Kappa coefficient is a metric through weighted classification accuracy, and the mathematical expression is shown in equations (3) and (4); Where T is the total number of pixels in the test set; l is the total number of categories; u represents each category; c uu is the pixel point that is classified correctly; c u+ With c +u is the sum of misclassified pixels, where u+ represents pixels in the uth row that are classified into other columns, and +u represents pixels in the uth column that are classified into other rows, both of which are incorrect classifications.

Citation Information

Patent Citations

  • Hyperspectral image-based nondestructive detection method for soluble solid contents of kiwi fruits

    CN109187378A

  • A high-quality imaging method of a spectral imaging system based on a convolutional neural network

    CN109447891A