A method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks

Abstract

A method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks includes: using hyperspectral imaging technology to obtain peanut images within the wavelength range of 970-2500nm; obtaining peanut patches of hyperspectral images pixel by pixel according to a point-centered strategy; constructing a multi-scale point-centered convolutional neural network to extract deep spatial-spectral features of moldy peanuts and healthy peanut patches and classify moldy peanuts and healthy peanuts accordingly; using threshold segmentation to separate single peanuts; combining the classification results and segmentation results of the multi-scale point-centered convolutional neural network to determine whether the whole peanut is spoiled. The results show that the method of the present invention can effectively identify and extract moldy peanuts and can be used in peanut production, food testing and instrument development.

Description

Technical Field

[0001] The invention relates to the technical field of hyperspectral remote sensing, and in particular to a method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks. Background Art

[0002] Peanut is an important cash crop and oil crop. China is the country with the largest total production and consumption of peanuts and peanut oil in the world. In 2019, China's total peanut production was 17.52 million tons, accounting for 36.19% of the world's total production. Peanuts are prone to mold and produce aflatoxin due to inappropriate conditions during storage and transportation. Aflatoxin is a Class I carcinogen and cannot be inactivated by high-temperature cooking. At present, the quantitative detection methods of aflatoxin include thin film chromatography, liquid chromatography, and high-performance liquid chromatography. Although the above quantitative methods can obtain effective and accurate results, these methods are expensive, time-consuming and destructive, and require highly skilled operators and precise control of experimental conditions. They are not suitable for online monitoring or automation. Therefore, there is an urgent need for a technology that can be applied to the online detection and identification of moldy peanuts in a fast, real-time, non-destructive and accurate manner to meet the needs of actual industrial production. In this way, it not only helps to reduce the content of aflatoxin in peanuts and their products, but also has great significance for human health.

[0003] Hyperspectral images have the characteristic of "image and spectrum in one", which can integrate the spectral characteristics reflecting the target radiation with the image characteristics reflecting the target two-dimensional space. By comprehensively utilizing spectral and spatial characteristics, it can realize the accurate identification of moldy peanuts, which is conducive to the use of mechanical devices to remove moldy peanuts. Summary of the invention

[0004] The purpose of the present invention is to provide a method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks, and to realize rapid, real-time, non-destructive and accurate identification of moldy peanuts by constructing a convolutional neural network to extract deep spatial-spectral features.

[0005] The technical solution of the present invention is as follows:

[0006] The moldy peanut identification method combining near-infrared hyperspectral images and convolutional neural networks of the present invention comprises: step one: using hyperspectral imaging technology to obtain peanut images within a wavelength range of 970-2500nm; step two: according to a point-centered strategy, obtaining peanut spots of the hyperspectral image pixel by pixel; step three: constructing a multi-scale point-centered convolutional neural network to extract deep spatial-spectral features of moldy peanuts and healthy peanut spots and classify the moldy peanuts and healthy peanuts accordingly; step four: using threshold segmentation to achieve separation of single peanuts; step five: combining the classification result and segmentation result of the multi-scale point-centered convolutional neural network to determine whether the whole peanut is moldy.

[0007] Preferably, in the above-mentioned method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks, in step 2, the optimal peanut patch size obtained according to the point-centered strategy is 7×7×λ (λ is the number of bands for obtaining the hyperspectral image).

[0008] Preferably, in the above-mentioned method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks, in step three, the constructed multi-scale convolutional neural network includes a multi-scale convolutional layer, two convolutional layers, two fully connected layers and a SoftMax layer.

[0009] Preferably, in the above-mentioned moldy peanut identification method combining near-infrared hyperspectral images and convolutional neural networks, the multi-scale convolution layer includes three parallel parts: 1×1 convolution, 1×1 convolution and 3×3 convolution, 1×1 convolution and two consecutive 3×3 convolutions, and then the results of the three parallel convolution operations are connected in series.

[0010] Preferably, in the above-mentioned moldy peanut identification method combining near-infrared hyperspectral images and convolutional neural networks, the optimal number of convolution kernels of the first layer of convolution operation of the constructed multi-scale convolutional neural network is 5, and the number of convolution kernels of each subsequent layer is twice that of the previous layer.

[0011] Preferably, in the above-mentioned method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks, the multi-scale point-centered convolutional neural network takes the image patch as input, and the surrounding pixels are used as auxiliary information to determine the category of the center pixel of the image patch.

[0012] Preferably, in the above-mentioned method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks, it is characterized in that, in step four, the wavelength of 1279nm is used as the mask band, and its threshold value (α) is 0.2.

[0013] Preferably, in the above-mentioned method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks, in step five, whether a single peanut is moldy is determined by calculating the ratio of moldy pixels in a single peanut to all pixels, and the threshold value (θ) is 0.05.

[0014] According to the technical solution of the present invention, the beneficial effects produced are:

[0015] Hyperspectral images have the characteristic of "image and spectrum in one". The method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks of the present invention utilizes hyperspectral imaging technology to simultaneously obtain spectral features reflecting target radiation and image features reflecting target spatial information, and comprehensively utilizes spectral and spatial features. Through convolutional neural networks, deep spatial-spectral features are fused and extracted to improve the recognition accuracy of moldy peanuts. Through the established point-centered multi-scale convolutional neural network, rapid, real-time, lossless and accurate recognition of moldy peanuts can be achieved.

[0016] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1a This is a schematic diagram of obtaining a spot using a point-centric strategy. Figure 1b It is a schematic diagram of the distribution of the central pixels and surrounding pixels of the image spot.

[0018] Figure 2 Schematic diagram of the point-centered multi-scale convolutional neural network structure.

[0019] Figure 3 The following are the actual pictures of healthy peanuts and moldy peanuts.

[0020] Figure 4 The loss function of the point-centered multi-scale convolutional neural network changes with the number of iterations under different convolution kernels.

[0021] Figure 5 The classification accuracy of the point-centered multi-scale convolutional neural network for different input image spatial scales.

[0022] Figure 6 The features of healthy peanuts (bright spots) and moldy peanuts (dark spots) output by each layer of the network are visualized after t-SNE projection and dimensionality reduction.

[0023] Figure 7 This is the pixel-level classification result of peanut classification based on point-centered multi-scale convolutional neural network. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings.

[0025] The method of identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks of the present invention is a moldy peanut detection method based on the spectral-spatial features of hyperspectral images. By constructing a convolutional neural network to extract deep-level spatial-spectral features, rapid, real-time, non-destructive and accurate moldy peanut identification can be achieved.

[0026] The method of the present invention for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks comprises the following steps:

[0027] Step 1: Use a scanning device to perform near-infrared hyperspectral scanning on the moldy peanuts and healthy peanuts to be identified or detected to generate near-infrared hyperspectral image information. The scanning device can be a near-infrared imaging spectrometer, and the scanned image information at least includes image information and spectral information of each band.

[0028] Preprocess the hyperspectral image data:

[0029] Hyperspectral imaging technology is used to obtain the hyperspectral image Img of peanuts. Including: After measuring the reflectivity reference white board and black board, the hyperspectral image is corrected by the following formula. The correction formula is as follows:

[0030]

[0031] Where: R is the relative reflectance of the corrected hyperspectral image, R∈[0,1]; I S is the original hyperspectral DN value of peanut; I D is the DN value of the calibration image of the blackboard; I W DN value of the white plate calibration image.

[0032] At the same time, singular spectrum analysis is used to eliminate the noise in the spectrum, and its decomposition formula is:

[0033]

[0034] Among them, d is the decomposition window size of 90, U i 、V i and λ are the left singular vector, right singular vector and singular value respectively, and then the left singular vector, right singular vector and singular value of the first n components are selected to reconstruct the spectrum.

[0035] Step 2: Obtain the hyperspectral image spots of peanuts pixel by pixel according to the point-centered strategy. The spot extraction formula is:

[0036]

[0037] Patch i,jThe obtained patch has a size of L×L×λ, where L is the patch window size, λ is the number of bands, and L is generally an odd number; i and j are the row and column numbers of the central pixel of the obtained Patch.

[0038] Step 3: Construct a multi-scale point-centered convolutional neural network with patches as input, such as Figure 2 As shown in Figure 1. The network consists of a multi-scale convolution layer, two convolution layers, two fully connected layers and a SoftMax layer. The multi-scale convolution layer includes three parallel parts: 1×1 convolution, 1×1 convolution and 3×3 convolution, and 1×1 convolution and two consecutive 3×3 convolutions. No padding is done for 1×1 convolution, and a padding of size 1 is done for 3×3 convolution. The results of the three parallel convolution operations are then connected in series. The multi-scale convolution layer of the network is a depth-separable convolution structure, and the first layer of each branch contains multiple 1×1 convolutions. There is a certain redundancy in the spectral dimension of the original hyperspectral image, so the use of 1×1 convolution can compress the redundancy of the original image and reduce the number of model parameters. In order to extract deeper features and compress local spatial information, the multi-scale convolution layer includes two consecutive convolution layers, whose convolution kernel size is 3×3 and no padding is done during convolution. The result of the convolution is then flattened into a vector and connected to the fully connected layer. Then, based on the acquired feature vector, the relationship between the extracted features and the peanut category is established through the full connection of the fourth and fifth layers. The number of neurons in the first fully connected layer is 120, and the number of neurons in the second fully connected layer is 30. The fewer neurons in the fully connected layer are due to the fewer parameters of the network itself. In addition, the Dropout strategy is adopted in the fully connected layer to avoid overfitting. Finally, the output of F-5 is converted into a probability using the SoftMax function, indicating whether the central pixel of the patch is contaminated. The number of convolution kernels in the first layer of the network is N (N<<λ), and the number of convolution kernels in each subsequent layer is twice that of the previous layer.

[0039] Step 4: Segmentation of the range of single peanuts. When performing spectral scanning, since the spectral reflectance of the background is much lower than the reflectance of peanuts, threshold segmentation is used to separate single peanuts. The 1279nm band is used as the mask band. Using the threshold α as the boundary, the mask image is obtained to divide the image into peanut pixels and non-peanut pixels. Based on the mask image, the region growing algorithm is used to obtain the range of each peanut. Seed points are randomly generated on the mask image, and the region growing algorithm is performed. After obtaining the peanut range where the seed point is located, the peanut range is deleted from the mask image, and seed points are randomly generated again. The above algorithm is repeated until the range of all single peanuts is obtained.

[0040] Step 5: Obtain the particle-level classification results of moldy peanuts based on the classification results of the multi-scale point-centered convolutional neural network. On this basis, the segmentation results of a single peanut are used to calculate the proportion of moldy pixels in a single peanut (θ). When the proportion of moldy pixels in a certain peanut is greater than a certain threshold, the peanut is considered moldy; otherwise, it is considered healthy peanut.

[0041] Example:

[0042] The method of the present invention uses hyperspectral imaging technology to obtain hyperspectral images of moldy peanuts and healthy peanuts in the wavelength range of 970-2500nm. Specifically, it includes the following process:

[0043] A. Preprocessing of hyperspectral image data.

[0044] The hyperspectral data is preprocessed according to equations (1) and (2), wherein when singular spectrum analysis is used to reduce the noise of the spectrum, the reconstructed component n=20.

[0045] B. Obtain the hyperspectral image of peanuts pixel by pixel according to the point-centered strategy.

[0046] According to formula (3), the peanut patch is obtained, and the sliding step size is (1,1). The final patch size is L×L×λ, where the input window size L is 7 and the number of bands λ is 288.

[0047] C. Construct a point-centered multi-scale convolutional neural network.

[0048] The network consists of a multi-scale convolutional layer, two convolutional layers, two fully connected layers and a SoftMax layer. The multi-scale convolutional layer (MSC-1) includes three parallel parts: 1×1 convolution, 1×1 convolution and 3×3 convolution, and 1×1 convolution and two consecutive 3×3 convolutions. No padding is done for 1×1 convolution, and a padding of size 1 is done for 3×3 convolution. The results of the three parallel convolution operations are then concatenated. The multi-scale convolutional layer is followed by two consecutive convolutions (C-2 / C-3) with a convolution kernel size of 3×3 and no padding is done during convolution. Two fully connected layers (F-4 / F-5) are then used to classify the features extracted by convolution. The number of neurons in the first fully connected layer is 120, and the number of neurons in the second fully connected layer is 30. Finally, the output of F-5 is converted into a probability using the SoftMax function to determine whether the central pixel of the image spot is contaminated. The number of convolution kernels in the first layer of the network is 5, and the number of convolution kernels in each subsequent layer is twice that of the previous layer, that is, the output of MSC-1 is 7*7*N*7=7*7*35 (number of horizontal pixels*number of vertical pixels*number of convolution kernels), the output of C-2 is 5*5*2N*7=5*5*70, and the output of C-3 is 3*3*4N*7=3*3*140=1260. When training the network, the optimization method of cross entropy combined with stochastic gradient descent is adopted, and the batch size is set to 64. The learning rate is set to 0.00001, and the learning decay rate is 0.9. The smaller learning rate is used because the input of the network is small.

[0049] D.Single peanut range segmentation.

[0050] The 1279nm band is used as the mask band. With the threshold α=0.2 as the boundary, the mask image is obtained to divide the image into peanut pixels and non-peanut pixels. The obtained mask image is a 0 / 1 image, with peanuts as 1 and non-peanuts as 0. Seed points are randomly generated on the mask image and the region growing algorithm is performed. The region growing algorithm is a four-way region growing algorithm with a growth threshold of 0.1. After obtaining the peanut range of the seed point, the peanut range is deleted from the mask image, and the seed point is randomly generated again. The above algorithm is repeated until the range of all single peanuts is obtained.

[0051] E. Obtaining particle-level classification results.

[0052] The pixel-level classification results of moldy peanuts were obtained based on the classification results of the multi-scale point-centered convolutional neural network. On this basis, the segmentation results of a single peanut were used to calculate the proportion of moldy pixels (θ) in a single peanut. When θ>0.05 for a certain peanut, the peanut was considered moldy; otherwise, if the proportion of moldy pixels in the peanut was θ<0.05, the peanut was considered healthy.

[0053] Results analysis and feasibility of this method

[0054] An experiment on identifying moldy peanuts was conducted using three varieties of peanuts, Huayu, Silihong and Xiaobaisha, as research objects.

[0055] The hyperspectral images of three varieties of peanuts, Huayu, Silihong and Xiaobaisha, were processed according to the method flow. Figure 1(a) is a schematic diagram of peanut spot acquisition, which is obtained by sliding pixel by pixel in the hyperspectral image to obtain the spots corresponding to all pixels. Figure 1(b) is a schematic diagram of the distribution of central pixels and surrounding pixels of peanut spots.

[0056] Figure 2 Schematic diagram of the point-centered multi-scale convolutional neural network structure. Figure 3 The following are the actual pictures of healthy peanuts and moldy peanuts. Figure 4 The variation of cross entropy with the number of iterations of point-centered multi-scale convolutional neural network with different numbers of convolution kernels is given. When the number of convolution kernels in the first layer is 5, the classification accuracy reaches the best. Figure 5 The influence of different input patch spatial scales on classification accuracy is shown in Figure 2. When the input patch size is 7×7×λ, the classification accuracy reaches the optimal value. Figure 6 The pixel-level classification results are obtained by using the trained point-centered multi-scale to classify peanuts. Figure 7 The peanut particle level classification result is obtained by combining the pixel level classification result with the image segmentation result. At this time, α=0.2,θ=0.05.

[0057] Table 1 is the specific structural parameter table of the point-centered multi-scale convolutional neural network model.

[0058] Table 1 Point-centered multi-scale convolutional neural network structure

[0059]

[0060]

[0061] Table 2 is the statistical table of peanut pixel-level classification results.

[0062] Table 2 Pixel-level classification results of moldy peanuts

[0063]

[0064] Table 3 is a statistical table of peanut particle classification results.

[0065] Table 3 Pixel-level classification results of moldy peanuts

[0066]

[0067] The present invention combines near-infrared hyperspectral images and convolutional neural networks to identify moldy peanuts. The moldy peanut identification results can be imaged, and the exact location of the moldy peanuts can be detected and further separated. In order to obtain the deep spatial-spectral characteristics of moldy peanuts, the present invention proposes a point-centered multi-scale convolutional neural network to identify moldy peanuts. The recognition results show that the present invention has a high accuracy in identifying moldy peanuts, and the pixel-level classification accuracy is above 98%. Overall, when α=0.2 and θ=0.05, the particle-level classification accuracy is above 96%, indicating that it is feasible for detecting moldy peanuts.

[0068] The above description is the best embodiment according to the concept and working principle of the present invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks, characterized in that: The following steps are involved: Step 1: Use hyperspectral imaging technology to obtain peanut images in the wavelength range of 970-2500nm; Step 2: Obtain the peanut patch of the hyperspectral image pixel by pixel according to the point center strategy; Step 3: Construct a multi-scale point-centered convolutional neural network to extract the deep spatial-spectral features of moldy peanuts and healthy peanuts and classify them accordingly; Step 4: Use threshold segmentation combined with region growing algorithm to separate individual peanuts; Step 5: Combine the classification results and segmentation results of the multi-scale point-centered convolutional neural network to determine whether a single peanut is moldy; In step three, the constructed multi-scale point-centered convolutional neural network includes a multi-scale convolutional layer, two convolutional layers, two fully connected layers and a SoftMax layer; the multi-scale convolutional layer includes three parallel parts: 1×1 convolution, 1×1 convolution and 3×3 convolution, 1×1 convolution and two consecutive 3×3 convolutions, and then the results of the three parallel convolution operations are connected in series; The multi-scale point-centered convolutional neural network takes the peanut patch as input and the surrounding pixels as auxiliary information to determine the category of the center pixel of the patch. In step 2, the hyperspectral image spots of peanuts are obtained pixel by pixel according to the point-centered strategy. The spot extraction formula is: ， The size of the obtained patch is , where L is the spot window size, λ is the number of bands, and L is an odd number; and To obtain The row and column number of the center pixel.

2. The method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks according to claim 1, characterized in that: In step 2, the optimal peanut patch size obtained according to the point-centered strategy is , The number of bands to obtain the hyperspectral image.

3. According to the method for identifying moldy peanuts combining near-infrared hyperspectral images and convolutional neural networks described in claim 1, the optimal number of convolution kernels for the first layer of convolution operation of the constructed multi-scale point-centered convolutional neural network is 5, and the number of convolution kernels in each subsequent layer is twice that of the previous layer.

4. The method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks according to claim 1, characterized in that: In step 4, the wavelength of 1279 nm is used as the mask band.

5. The method for identifying moldy peanuts by combining near-infrared hyperspectral images and convolutional neural networks according to claim 1, characterized in that: In step five, whether a single peanut is moldy is determined by calculating the ratio of moldy pixels in a single peanut to all pixels.

Images