A classification and regression method for leaf hyperspectral images based on multi-scale cascade convolutional neural networks

Abstract

The present invention belongs to the field of plant science research, and discloses a leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network. First, in 3D-CNN, dilated convolution is embedded to construct a spectral-spatial feature extraction structure of different scales to achieve the fusion of multi-scale features; secondly, 1D-CNN is cascaded after 3D-CNN to further extract high-level abstract spectral features, and the proposed multi-scale 3D-1D-CNN network is explored for the optimal framework; finally, on two limited sample facility crop leaf data sets, the proposed multi-scale 3D-1D-CNN network model is compared with a benchmark model and a multi-scale 3D-CNN model to verify the effectiveness of the proposed method. The method of the present invention is helpful for the classification and regression of leaf hyperspectral images, and can also provide new ideas and technical assistance for other image classification and regression methods in the field of agricultural information technology.

Description

Technical Field

[0001] The invention belongs to the field of plant science research, and in particular relates to a leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network. Background Art

[0002] The biological processes of plant leaves, such as photosynthesis and transpiration, are closely related to the biochemical parameters of leaves, such as the chlorophyll content and water content of leaves. Taking basil leaves and pepper leaves as examples, basil is an economic crop of herbal spices that is cultivated in greenhouses in southern my country. Its leaves can be eaten fresh or dried for medicinal purposes. The chlorophyll content in basil leaves directly affects the growth, development, nutritional level and economic yield of basil. Pepper is a shallow-rooted plant with a high degree of cork and drought resistance. Severe water stress can easily cause significant damage to the physiological mechanism of pepper. Plant leaves are affected by the above biochemical parameters, and their corresponding visible-near infrared spectra will also show different response characteristics. Hyperspectral imaging, as a technology that integrates images and spectra, can simultaneously obtain continuous spectral information of each pixel on the image and continuous image information of each spectral band. Its spectral dimension consists of hundreds of continuous bands in the visible to infrared wavelength range. It has many advantages such as high resolution, rich band information, fast and lossless, and has been widely used in the field of plant phenotyping research. Since the three-dimensional data blocks of hyperspectral images have problems such as high spectral dimension, redundant adjacent band information and limited label samples, and the traditional hyperspectral image analysis process is cumbersome and the analysis results rely on expert experience, we need to establish an efficient leaf hyperspectral image classification and regression method, which lays the foundation for the subsequent prediction of plant biochemical parameter content and the establishment of stress diagnosis models.

[0003] At present, in the field of plant science research, deep learning models represented by convolutional neural networks (CNN) are increasingly being used for the analysis of hyperspectral images. CNN autonomously learns and extracts local and global features of data through multi-layer convolution and pooling operations. According to the convolution kernel structure of CNN, it can be divided into one-dimensional CNN (1D-CNN), two-dimensional CNN (2D-CNN) and three-dimensional CNN (3D-CNN). 1D-CNN is the most widely used hyperspectral image analysis model. It extracts and models deep spectral features based on one-dimensional spectral data. It usually requires manual extraction of the average spectrum or pixel point spectrum of the sample region of interest (ROI) in advance. It has been applied to the classification of cotton seeds, the identification of low temperature stress of corn plants, the prediction of water content of corn plants, the quantification of total anthocyanins in black wolfberry, and the detection of pesticide residues in leek leaves. 2D-CNN mainly extracts deep spatial features and can analyze and model the original or reduced-dimensional hyperspectral images. It has been applied to the classification of apple leaf status, the identification of disease-resistant rice seeds, the identification of diseased areas in the canopy of potato plants, and the evaluation of corn seed vitality. Compared with 1D-CNN and 2D-CNN feature learning in spectral and spatial dimensions respectively, 3D-CNN can extract deep spectral-spatial joint features end-to-end while maintaining the information continuity of adjacent spectral band features. It is particularly suitable for the analysis of three-dimensional data blocks of hyperspectral images with spatial-spectral continuity.

[0004] At present, the application of 3D-CNN in the field of plant science is still relatively rare, with only reports on soybean stem disease detection and cotton leaf aphid detection. This is because it is difficult and time-consuming to obtain labeled samples in plant research, and the 3D-CNN network has a large number of parameters and high computational complexity. For the limited labeled training samples in plant research, it is easy to overfit, and the generalization performance is reduced. Therefore, in the field of plant science research, it is urgent to explore an improved 3D-CNN model for leaf hyperspectral image classification and regression, which can reduce the model computational complexity when the training samples are limited, and improve the model generalization performance. Summary of the invention

[0005] The purpose of the present invention is to provide a leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network to solve the above-mentioned technical problems.

[0006] In order to solve the above technical problems, the specific technical solution of the leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network of the present invention is as follows:

[0007] A leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network comprises the following steps:

[0008] S1: Build a hyperspectral imaging system;

[0009] S2: Collect hyperspectral images of basil leaf and pepper leaf samples and preprocess the images;

[0010] S3: Introducing dilated convolution into 3D-CNN, expanding the receptive field of the convolution kernel without increasing network parameters and computational complexity, and constructing spectral-spatial feature extraction structures of different scales to achieve multi-scale feature fusion and explore the best

[0011] Multi-scale 3D-CNN network structure;

[0012] S4: A 1D-CNN network is cascaded after the optimal multi-scale 3D-CNN network described in S3, and 3D data is used as input to learn more abstract spectral features based on spectral-spatial joint feature extraction, and the optimal framework of the proposed multi-scale 3D-1D-CNN network is explored;

[0013] S5: Compare the optimal framework of the multi-scale 3D-1D-CNN network described in S4 with the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, and multi-scale 3D-CNN network models to verify the effectiveness of the method.

[0014] Furthermore, the S1: building a hyperspectral imaging system includes acquiring a hyperspectral image of a sample, setting a spectral band range and a spatial resolution, and correcting an original hyperspectral image.

[0015] Furthermore, the hyperspectral imaging system includes a SNAPSCAN VNIR spectral camera, a 35mm lens, a 150W annular halogen tungsten lamp light source, image acquisition software and an image acquisition platform; the spectral band range is 470-900nm, with a total of 140 bands; the SNAPSCAN VNIR spectral camera uses a built-in scanning imaging method to quickly acquire the hyperspectral image of the sample, without the need for relative displacement of the camera or the sample, the maximum spatial resolution of the SNAPSCAN VNIR spectral camera is 3650×2048, and the actual spatial resolution in data acquisition is adjusted according to the size of the leaf sample; the vertical distance between the 35mm lens and the sample is 34.5cm, and the exposure time is set to 30ms; the original hyperspectral image collected by the hyperspectral imaging system is subjected to the following black and white correction to calculate the reflectance of the sample, while reducing the influence of illumination and camera dark current on the acquisition of spectra of different samples:

[0016]

[0017] Where R 0 and R are the hyperspectral images before and after correction, respectively. W and D are the white board reference image and the dark current reference image, respectively.

[0018] Furthermore, for the basil leaves described in S2, three different light intensity treatments were applied to the sweet basil cultivated under artificial LED lighting, data collection was carried out when the basil grew for 40 days, the relative value of chlorophyll was measured, and 540 hyperspectral images were collected; for the pepper leaves described in S2, two water treatments were applied to healthy pepper plants with consistent growth after 50 days of cultivation, the samples of the control group were irrigated normally, and the samples of the experimental group were subjected to five days of continuous drought, and 600 hyperspectral images were collected;

[0019] Hyperspectral images and corresponding label data pairs are read using custom python functions.

[0020] Furthermore, the image preprocessing in S2 includes background segmentation, noise point removal, and size adjustment;

[0021] The background segmentation method is: using the 800nm ​​near-infrared band with the largest difference in spectral reflectance between the leaves and the background as the threshold band for background segmentation, and the threshold is set to 0.15;

[0022] The method for removing noise points is: using morphological transformation in opencv-python to achieve noise removal;

[0023] The size adjustment is: reducing the spatial dimension uniformly to 120×120.

[0024] Further, S2 includes dividing the preprocessed images into a training set, a validation set and a test set, and the method of dividing the training set, the validation set and the test set is a random sampling method; 380 samples are randomly selected from basil leaves as a training set, 80 as a validation set, and 80 as a test set; 400 samples are randomly selected from pepper leaves as a training set, 100 as a validation set, and 100 as a test set, and the data set is divided into multiple batches, and the number of batch samples is set to 8.

[0025] Furthermore, S3 describes constructing a spectral-spatial feature extraction structure of different scales, which is to embed standard 3D convolution and dilated 3D convolution in parallel in the same layer of the network, with each of the two convolution kernels occupying 50% of the number of channels. The feature maps output by different convolution kernels are then batch normalized and concatenated in the channel dimension, and ReLU nonlinear activation is performed.

[0026] Furthermore, the dilated 3D convolution described in S3 is implemented by inserting d-1 values ​​with a weight of 0 between adjacent weights of all rows and columns of the standard 3D convolution, where d is the dilation factor, which expands the receptive field of the convolution kernel without increasing network parameters or losing information; the convolution kernel of the standard 3D convolution is 3×3×3, d=1, and the receptive field is 3×3×3;

[0027] The convolution kernel of the expanded 3D convolution is 3×3×3, and the expanded 3D convolution kernel structures with d of 2, 3, and 4 are tested, corresponding to receptive fields of 5×5×5, 7×7×7, and 9×9×9, respectively. Convolution kernels with different receptive fields can extract spectral and spatial features of different scales in the feature map;

[0028] The optimal multi-scale 3D-CNN network structure is a model of splicing 3×3×3 dilated 3D convolution kernels with d=2. Furthermore, the S4 cascades a 1D-CNN network after the optimal multi-scale 3D-CNN network, and performs a 3D-CNN to 1D-CNN model performance test at different locations of the multi-scale 3DResNet-18 network to obtain the optimal multi-scale 3D-1D-CNN model;

[0029] The basic components of the best multi-scale 3D-CNN network cascade 1D-CNN network include convolution layer, batch normalization layer, ReLU nonlinear activation layer, mean pooling layer and fully connected layer. From the input layer to the output layer, the entire network is divided into 3D-CNN, 3D-CNN to 1D-CNN, and 1D-CNN stages;

[0030] In the 3D-CNN stage, the preprocessed hyperspectral image cube data block with a dimension of 120×120×140 is used as input, and a 3×3×3 convolution kernel with a step size of 2, 2, 1 is first used to extract local spectral-spatial features to generate a 64-channel three-dimensional spectral image, which is then input into a continuous optimal multi-scale 3D-CNN network to achieve spectral-spatial feature extraction and fusion of different scales;

[0031] In the 3D-CNN to 1D-CNN stage, after testing, the conversion effect of position ⑦ in the multi-scale 3D ResNet-18 network is the best. The three-dimensional feature map of the 3D-CNN output channel number × spatial height × spatial width × number of bands: 256 × 15 × 15 × 35 is converted into a one-dimensional feature of size 35 using a 15 × 15 × 1 3D convolution kernel;

[0032] In the 1D-CNN stage, continuous one-dimensional convolution kernels of size 3 are used to extract deep spectral features to obtain one-dimensional features of 512×17. The extracted high-level abstract features are then compressed and aggregated based on mean pooling to obtain features with stronger representation capabilities with a dimension of 512×1. Finally, the model prediction value is output through the fully connected layer. Compared with the best multi-scale 3D-CNN network without cascading 1D-CNN, the number of parameters of this model is reduced by about 35.83%.

[0033] Furthermore, the benchmark 1D-CNN, benchmark 2D-CNN, benchmark 3D-CNN, and multi-scale 3D-CNN described in S5 include a convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a mean pooling layer, and a fully connected layer;

[0034] The benchmark 1D-CNN is a one-dimensional convolutional neural network based on the residual network framework ResNet, which performs feature extraction in the spectral dimension;

[0035] The benchmark 2D-CNN is a two-dimensional convolutional neural network based on ResNet, which performs feature extraction in the spatial dimension; the benchmark 3D-CNN is a three-dimensional convolutional neural network based on ResNet, which performs joint feature extraction in the spectral + spatial dimensions;

[0036] The multi-scale 3D-CNN is the best multi-scale 3D-CNN network in S3, which can obtain spectral and spatial features of different scales;

[0037] All models are trained from scratch, and the initial network weights come from random initialization values;

[0038] The loss function of the regression model is mean square error, and the loss function of the classification model is cross entropy;

[0039] The model was optimized using the gradient descent method, with the momentum set to 0.9 and the initial learning rate varying between {1×10 -2 ,1×10 -3 ,1×10 -4 ,1×10 -5}, the learning rate drops by one order of magnitude every 30 training rounds, and the total number of training rounds is 80;

[0040] The performance evaluation index of the regression model is the determination coefficient R 2 and root mean square error RMSE;

[0041] The evaluation indicators of the classification model are F1-score and accuracy.

[0042] The leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network of the present invention has the following advantages:

[0043] (1) In view of the problem that 1D-CNN and 2D-CNN network models cannot extract spectral-spatial joint features from plant leaf hyperspectral images, the present invention proposes a multi-scale 3D-CNN network framework integrating dilated convolution, which is conducive to extracting multi-scale spectral-spatial joint features from plant leaves and further improves the performance of the 3D-CNN model without increasing network parameters and computational complexity.

[0044] (2) The present invention addresses the problems of high computational complexity of 3D-CNN networks, easy overfitting in limited sample conditions, and low model generalization performance. The present invention explores the optimal network structure of cascading 1D-CNN after multi-scale 3D-CNN, further reducing the number of network parameters of simple three-dimensional convolution, model computational complexity, and overfitting, thereby improving model generalization performance.

[0045] (3) The multi-scale cascade convolutional neural network model method for leaf hyperspectral image classification and regression proposed in the present invention is helpful for the classification and regression of leaf hyperspectral images, and can also provide new ideas and technical assistance for other image classification and regression methods in the field of agricultural informatization technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network;

[0047] Figure 2 Schematic diagram of a multi-scale spectral-spatial joint feature extraction module that integrates standard and dilated 3D convolutions;

[0048] Figure 3 The overall framework of the optimal multi-scale 3D-1D-CNN network for end-to-end modeling of hyperspectral images;

[0049] Figure 4 Schematic diagram of the performance comparison of multi-scale 3D-1D-CNN models for quantifying SPAD values ​​of basil leaf samples;

[0050] Figure 5 Schematic diagram of the performance comparison of multi-scale 3D-1D-CNN models for drought stress identification of pepper leaf samples;

[0051] Figure 6 This is the overall framework diagram of the multi-scale 3D-CNN network without cascading 1D-CNN;

[0052] Figure 7 Comparison of model prediction effects of baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, multi-scale 3D-CNN, and multi-scale 3D-1D-CNN for quantification of SPAD values ​​of basil leaf samples;

[0053] Figure 8 A comparison of the model prediction effects of baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, multi-scale 3D-CNN, and multi-scale 3D-1D-CNN for drought stress identification of pepper leaf samples. DETAILED DESCRIPTION

[0054] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network in conjunction with the accompanying drawings.

[0055] Firstly, in 3D-CNN, the present invention embeds dilated convolution to construct spectral-spatial feature extraction structures of different scales, realizes the fusion of multi-scale features, and further improves the performance of 3D-CNN model without increasing network parameters and computational complexity;

[0056] Secondly, based on the multi-scale 3D-CNN, an efficient multi-scale 3D-1D-CNN network structure is designed, that is, 1D-CNN is cascaded after 3D-CNN to further extract high-level abstract spectral features, and the optimal framework of the proposed multi-scale 3D-1D-CNN network is explored to reduce the computational complexity and overfitting degree of simple three-dimensional convolution;

[0057] Finally, the proposed multi-scale 3D-1D-CNN network model is compared with the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN and multi-scale 3D-CNN models on two facility crop leaf datasets with limited samples to verify the effectiveness of the proposed method.

[0058] like Figure 1 As shown, a leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network of the present invention specifically includes the following steps:

[0059] S1: Build a visible-near-infrared hyperspectral imaging system, obtain hyperspectral images of samples, set the spectral band range and spatial resolution, and correct the original hyperspectral images.

[0060] The visible-near infrared hyperspectral imaging system consists of a SNAPSCAN VNIR spectral camera (IMEC, Leuven, Belgium), a 35mm lens, a 150W annular halogen tungsten lamp light source, image acquisition software (HSImager), an image acquisition platform, etc.; the spectral band range of S1 is 470-900nm, with a total of 140 bands; the spectral camera of S1 uses a built-in scanning imaging method to quickly acquire the hyperspectral image of the sample, without the need for relative displacement of the camera or the sample, thus avoiding the problem of spatial deformation of the spectral image; the maximum spatial resolution of the spectral camera of S1 is 3650×2048, and the actual spatial resolution in data acquisition is adjusted according to the size of the leaf sample; the vertical distance between the 35mm lens of S1 and the sample is 34.5cm, and the exposure time is set to 30ms; the original hyperspectral image collected by the hyperspectral imaging system of S1 is subjected to the following black-and-white correction to calculate the reflectance of the sample and reduce the influence of illumination and camera dark current on the acquisition of spectra of different samples:

[0061]

[0062] Where R 0 and R are the hyperspectral images before and after correction, respectively. W and D are the white board reference image (diffuse reflectance close to 100%) and the dark current reference image, respectively.

[0063] S2: Collect hyperspectral images of basil leaf and pepper leaf samples, and preprocess the images to divide all images into a training set, a validation set, and a test set; the image preprocessing includes image background segmentation, noise point removal, and size adjustment.

[0064] Basil leaves were treated with three different light intensities under artificial LED lighting conditions: 200±5, 135±4, and 70±5 μmol m -2 s -1 , the red-blue light quality ratio was 3:1, the photoperiod was 16 hours / day, 40 pots were cultivated in each experimental group, 1 plant was planted in each pot, and a total of 120 basil plants were cultivated. Except for the different light intensities, the different experimental groups were managed with normal water, fertilizer, air, heat, disease, insects, and weeds. Data collection was carried out when the basil grew for 40 days. The SPAD value of each leaf was measured with a SPAD-502 chlorophyll meter (Minolta Camera Co., Osaka, Japan). The left and right sides of the mid-vein were measured 3 times respectively. The average of the 6 measurements was taken as the relative chlorophyll value of the sample. 540 hyperspectral images were collected, and the dimensions of each leaf sample hyperspectral image were 600×800×140;

[0065] Pepper leaves, two water treatments were applied to healthy pepper plants with consistent growth after 50 days of cultivation. The samples in the control group were irrigated normally, and the samples in the experimental group were irrigated normally (drip irrigation 600ml / day). The samples in the experimental group were subjected to 5 days of continuous drought (drip irrigation 50ml / day), and the drip irrigation time was 17:00 every day. There were 10 pots in each group, with 2 plants planted in each pot, for a total of 40 pepper plants. Except for the different water application, the different experimental groups were managed with normal fertilizer, air, heat, light, disease, insects, and weeds. 300 leaf samples were collected from each group, and a total of 600 hyperspectral images were collected. The dimension of each leaf sample hyperspectral image was 120×200×140;

[0066] Hyperspectral images and corresponding label data pairs are read through custom python;

[0067] The image background segmentation method is as follows: the 800nm ​​near-infrared band with the largest difference in spectral reflectance between leaves and background is used as the threshold band for background segmentation, and the threshold is set to 0.15;

[0068] The method of removing noise points is: using morphological transformation in opencv-python to achieve noise removal;

[0069] Resizing: Reduce the spatial dimensions to 120×120;

[0070] The image is divided into training set, validation set and test set by random sampling; 380 samples are randomly selected from basil leaves as training set, 80 as validation set and 80 as test set; 400 samples are randomly selected from pepper leaves as training set, 100 as validation set and 100 as test set, and the data set is divided into multiple batches, and the number of batch samples is set to 8.

[0071] S3: Introduce dilated convolution into 3D-CNN to expand the receptive field of the convolution kernel without increasing network parameters and computational complexity, and construct spectral-spatial feature extraction structures of different scales to achieve multi-scale feature fusion, and explore the optimal multi-scale 3D-CNN network structure to improve the performance of the 3D-CNN model;

[0072] 3D-CNN can extract continuous local spectral-spatial joint features end-to-end. The feature value of the jth feature of the i-th layer at the coordinates x, y, z is It can be expressed as:

[0073]

[0074] Where: P i , Q i and R i is the size of the 3D convolution kernel of the i-th layer, is the feature value of the kth feature at the (x+p)(y+q)(z+r) position of the (i-1)th layer, is the convolution kernel for the k-th feature of the (i-1) layer; m is the number of features in the (i-1) layer, b ij is the bias, and δ represents the activation function.

[0075] To construct a spectral-spatial feature extraction structure of different scales, standard 3D convolution and dilated 3D convolution are embedded in the same layer of the network in parallel. The two convolution kernels each occupy 50% of the number of channels. The feature maps output by different convolution kernels are batch normalized and then spliced ​​in the channel dimension. ReLU nonlinear activation is performed to achieve spectral-spatial feature extraction and fusion of different scales. The framework is as follows: Figure 2 As shown;

[0076] The dilated 3D convolution is implemented by inserting d-1 values ​​with a weight of 0 between adjacent weights of all rows and columns of the standard 3D convolution. d is the dilation factor, which expands the receptive field of the convolution kernel without increasing network parameters or losing information.

[0077] The convolution kernel of the standard 3D convolution is 3×3×3, d=1, and the receptive field is 3×3×3;

[0078] The convolution kernel of the expanded 3D convolution is 3×3×3, and the expanded 3D convolution kernel structures with d of 2, 3, and 4 are tested, corresponding to receptive fields of 5×5×5, 7×7×7, and 9×9×9, respectively. Convolution kernels with different receptive fields can extract spectral and spatial features of different scales in the feature map;

[0079] The optimal multi-scale 3D-CNN network structure takes the pre-processed hyperspectral image cube data block with a dimension of 120×120×140 as input, first uses a 3×3×3 convolution kernel with a step size of 2,2,1 to extract local spectral-spatial features, generates a 64-channel three-dimensional spectral image, and then inputs it into a continuous multi-scale spectral-spatial joint feature extraction module;

[0080] The results in Tables 1 and 2 show that when splicing a 3×3×3 dilated 3D convolution kernel with d=2, the model performance of the multi-scale 3D-CNN is better than other types.

[0081] Table 1 Comparison of performance of multi-scale 3D-CNN models for SPAD value quantification of basil leaf samples

[0082]

[0083] Table 2 Performance comparison of multi-scale 3D-CNN models for drought stress recognition in pepper leaf samples

[0084]

[0085]

[0086] S4: 1D-CNN is cascaded after the best multi-scale 3D-CNN, and three-dimensional data is used as input. More abstract spectral features are learned based on spectral-spatial joint feature extraction, and the optimal framework of the proposed multi-scale 3D-1D-CNN network is explored to reduce the number of network parameters, model calculation complexity and overfitting degree, and improve the generalization performance of the model.

[0087] The multi-scale 3D-1D-CNN network is the best multi-scale 3D-1D-CNN model obtained by testing the model performance of 3D-CNN to 1D-CNN at different locations (①-⑩) of the multi-scale 3D ResNet-18 network. Figure 3 As shown;

[0088] The basic components of the multi-scale 3D-CNN cascade 1D-CNN network include convolutional layer, batch normalization layer, ReLU nonlinear activation layer, mean pooling layer and fully connected layer. From the input layer to the output layer, the entire network is divided into 3D-CNN, 3D-CNN to 1D-CNN, and 1D-CNN stages;

[0089] In the 3D-CNN stage, the preprocessed hyperspectral image cube data block with a dimension of 120×120×140 is used as input. First, a 3×3×3 convolution kernel with a step size of 2, 2, 1 is used to extract local spectral-spatial features to generate a 64-channel three-dimensional spectral image, which is then input into a continuous multi-scale 3D-CNN module to achieve spectral-spatial feature extraction and fusion at different scales; Figure 3 As shown, the continuous multi-scale 3D-CNN module has 10 layers, and the number of output channels is 64, 64, 64, 64, 128, 128, 128, 128, 256, 256 respectively. The number of output channels of each layer is the number of input channels of the next layer.

[0090] 3D-CNN to 1D-CNN stage, Figure 4 and Figure 5 Indicates the conversion of position ⑦ in the multi-scale 3D ResNet-18 network, and the test set has the best effect. It can be seen that at the beginning, with the increase of multi-scale 3D convolution layers, the model training complexity of both data sets increased, but the model prediction performance increased significantly, indicating that multi-scale spectral spatial feature extraction on the input three-dimensional data block helps the model learn more and more rich information. However, when the multi-scale 3D convolution layer increases to position ⑦, the model complexity continues to increase, and the prediction generalization performance begins to decline, and overfitting occurs, mainly due to too many model parameters and limited sample data sets. Therefore, at position ⑦, the 256×15×15×35 (number of channels×spatial height×spatial width×number of bands) three-dimensional feature map output by 3D-CNN is converted into a one-dimensional feature of size 35 using a 15×15×1 3D convolution kernel.

[0091] In the 1D-CNN stage, a continuous one-dimensional convolution kernel of size 3 is used to extract deep spectral features to obtain a one-dimensional feature of 512×17. Then, the extracted high-level abstract features are compressed and aggregated based on mean pooling to obtain features with stronger representation capabilities, with a dimension of 512×1. Finally, the model prediction value is output through the fully connected layer;

[0092] The network structure of multi-scale 3D-CNN cascade 1D-CNN ( Figure 3 ) and the multi-scale 3D-CNN network without cascaded 1D-CNN ( Figure 6 ), the number of parameters of this model decreases by about 35.83%.

[0093] S5: Compare the optimal network framework of the multi-scale 3D-1D-CNN described in S4 with the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, and multi-scale 3D-CNN network models to verify the effectiveness of the proposed method.

[0094] Preferably, the benchmark 1D-CNN, benchmark 2D-CNN, benchmark 3D-CNN, and multi-scale 3D-CNN described in S5 mainly include a convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a mean pooling layer, and a fully connected layer;

[0095] The baseline 1D-CNN is a one-dimensional convolutional neural network based on the residual network framework (ResNet) that performs feature extraction in the spectral dimension;

[0096] The baseline 2D-CNN is a two-dimensional convolutional neural network based on ResNet, which performs feature extraction in the spatial dimension;

[0097] The baseline 3D-CNN is a three-dimensional convolutional neural network based on ResNet, which performs joint feature extraction in spectral + spatial dimensions;

[0098] Multiscale 3D-CNN is the best multiscale 3D-CNN network in S3, which can obtain spectral and spatial features at different scales and avoid incomplete expression of feature information at a single scale;

[0099] All models are trained from scratch, and the initial network weights are derived from random initialization values.

[0100] The loss function of the regression model is mean square error, and the loss function of the classification model is cross entropy;

[0101] The model was optimized using the gradient descent method, with the momentum set to 0.9 and the initial learning rate varying between {1×10 -2 ,1×10 -3 ,1×10 -4 ,1×10 -5}, the learning rate drops by one order of magnitude every 30 training rounds, and the total number of training rounds is 80;

[0102] The performance evaluation index of the regression model is the determination coefficient (R 2 ) and root mean square error (RMSE);

[0103] The evaluation indicators of the classification model are F1-score and accuracy.

[0104] Compared with the multi-scale 3D-CNN, the multi-scale 3D-1D-CNN optimal framework proposed in this paper not only improves the model performance under small sample conditions, but also significantly reduces the computational complexity, as shown in Tables 3 and 4:

[0105] Table 3 Performance comparison of multi-scale 3D-1D-CNN models for SPAD value quantification of basil leaf samples

[0106]

[0107] Table 4 Performance comparison of multi-scale 3D-1D-CNN models for drought stress recognition in pepper leaf samples

[0108]

[0109]

[0110] Figure 7 , Figure 8 The comparison of the prediction effects of the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, multi-scale 3D-CNN, and multi-scale 3D-1D-CNN models on two leaf datasets is intuitively demonstrated:

[0111] The baseline 1D-CNN model loses a lot of spectral and spatial information at the data input layer, resulting in insufficient model learning, i.e., underfitting;

[0112] The baseline 2D-CNN model simplifies the data preprocessing process and improves the automation of the hyperspectral image analysis process, but 2D-CNN only focuses on the extraction of spatial features;

[0113] The joint feature extraction of the benchmark 3D-CNN model in the spectral + spatial dimensions can obtain more effective and detailed local abstract features while retaining the information continuity of the spectral features, which is fully applicable to the data characteristics of the three-dimensional hyperspectral image blocks themselves. However, the 3D-CNN model has a large number of parameters and high computational complexity.

[0114] The multi-scale 3D-CNN model can improve the prediction effect of the model as a whole without increasing the number of model parameters and computational complexity. However, the test set results are significantly lower than the validation set results in the data set, indicating a high degree of overfitting.

[0115] It can be seen that compared with the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, and multi-scale 3D-CNN, the multi-scale 3D-1D-CNN model proposed in this paper has a higher R 2 There are improvements of 22.46%, 8.15%, 4.30% and 2% in the accuracy of the classification test set, and improvements of 28.56%, 16.59%, 8.49% and 4.13% in the accuracy of the classification test set, which proves the effectiveness of the designed network model under small sample conditions, and can reduce the network parameters of simple three-dimensional convolution, the model calculation complexity and the degree of overfitting, and improve the generalization performance of the model.

[0116] In summary, the technical effect of the present invention is remarkable, and it has made a good technical contribution to the development of leaf hyperspectral image classification and regression methods. It has broad application prospects in the field of plant science research and considerable economic benefits.

[0117] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network, characterized in that: The following steps are involved: S1: Build a hyperspectral imaging system; S2: Collect hyperspectral images of basil leaf and pepper leaf samples and preprocess the images; S3: Introducing dilated convolution into 3D-CNN, expanding the receptive field of the convolution kernel without increasing network parameters and computational complexity, and constructing spectral-spatial feature extraction structures of different scales to achieve multi-scale feature fusion and explore the optimal multi-scale 3D-CNN network structure; S4: A 1D-CNN network is cascaded after the optimal multi-scale 3D-CNN network described in S3, and 3D data is used as input to learn more abstract spectral features based on spectral-spatial joint feature extraction, and the optimal framework of the proposed multi-scale 3D-1D-CNN network is explored; The S4 cascades a 1D-CNN network after the best multi-scale 3D-CNN network, and performs a 3D-CNN to 1D-CNN model performance test at different locations of the 3D ResNet-18 network to obtain the best multi-scale 3D-1D-CNN model; The basic components of the best multi-scale 3D-CNN network cascade 1D-CNN network include convolution layer, batch normalization layer, ReLU nonlinear activation layer, mean pooling layer and fully connected layer. From the input layer to the output layer, the entire network is divided into 3D-CNN, 3D-CNN to 1D-CNN, and 1D-CNN stages; In the 3D-CNN stage, the preprocessed hyperspectral image cube data block with a dimension of 120×120×140 is used as input, and a 3×3×3 convolution kernel with a step size of 2, 2, 1 is first used to extract local spectral-spatial features to generate a 64-channel three-dimensional spectral image, which is then input into a continuous optimal multi-scale 3D-CNN network to achieve spectral-spatial feature extraction and fusion of different scales; In the 3D-CNN to 1D-CNN stage, after testing, the conversion effect of position ⑦ in the 3D ResNet-18 network is the best. The 3D feature map of the 3D-CNN output channel number × spatial height × spatial width × number of bands: 256 × 15 × 15 × 35 is converted into a one-dimensional feature of size 35 using a 15 × 15 × 1 3D convolution kernel; S5: Compare the optimal framework of the multi-scale 3D-1D-CNN network described in S4 with the baseline 1D-CNN, baseline 2D-CNN, baseline 3D-CNN, and multi-scale 3D-CNN network models to verify the effectiveness of the method.

2. The leaf hyperspectral image classification and regression method based on a multi-scale cascade convolutional neural network according to claim 1 is characterized in that: The S1: building a hyperspectral imaging system includes acquiring a hyperspectral image of a sample, setting a spectral band range and a spatial resolution, and correcting the original hyperspectral image.

3. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 2 is characterized in that: The hyperspectral imaging system includes a SNAPSCAN VNIR spectral camera, a 35 mm lens, a 150W annular halogen tungsten lamp light source, image acquisition software and an image acquisition platform; the spectral band range is 470-900nm, with a total of 140 bands; the SNAPSCAN VNIR spectral camera uses a built-in scanning imaging method to quickly acquire the hyperspectral image of the sample, without the need for relative displacement of the camera or the sample, the maximum spatial resolution of the SNAPSCAN VNIR spectral camera is 3650×2048, and the actual spatial resolution in data acquisition is adjusted according to the size of the leaf sample; the vertical distance between the 35mm lens and the sample is 34.5cm, and the exposure time is set to 30ms; the original hyperspectral image collected by the hyperspectral imaging system is subjected to the following black and white correction to calculate the reflectivity of the sample, while reducing the influence of illumination and camera dark current on the acquisition of spectra of different samples: ， Where R0 and R are the hyperspectral images before and after correction, respectively; W and D are the whiteboard reference image and the dark current reference image, respectively.

4. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1 is characterized in that: For the basil leaves described in S2, three different light intensity treatments were applied to the sweet basil cultivated under artificial LED lighting. Data collection was carried out when the basil grew for 40 days, the relative value of chlorophyll was measured, and 540 hyperspectral images were collected. For the pepper leaves described in S2, two water treatments were applied to healthy pepper plants with consistent growth after 50 days of cultivation. The samples in the control group were irrigated normally, and the samples in the experimental group were subjected to continuous drought for five days, and 600 hyperspectral images were collected. Hyperspectral images and corresponding label data pairs are read using custom python functions.

5. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1 is characterized in that: S2 The image preprocessing includes background segmentation, noise point removal, and size adjustment; The background segmentation method is: using the 800 nm near-infrared band with the largest difference in spectral reflectance between the leaves and the background as the threshold band for background segmentation, and the threshold is set to 0.15; The method for removing noise points is: using morphological transformation in opencv-python to achieve noise removal; The size adjustment is: reducing the spatial dimension uniformly to 120×120.

6. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1 is characterized in that: S2 includes dividing the preprocessed images into a training set, a validation set and a test set, wherein the method of dividing the training set, the validation set and the test set is a random sampling method; 380 samples are randomly selected from basil leaves as a training set, 80 as a validation set, and 80 as a test set; 400 samples are randomly selected from pepper leaves as a training set, 100 as a validation set, and 100 as a test set, and the data set is divided into multiple batches, and the number of batch samples is set to 8.

7. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1 is characterized in that: S3 describes constructing a spectral-spatial feature extraction structure of different scales, which is to embed standard 3D convolution and dilated 3D convolution in parallel in the same layer of the network. The two types of convolution kernels each occupy 50% of the number of channels. The feature maps output by different convolution kernels are then batch normalized and concatenated in the channel dimension, and ReLU nonlinear activation is performed.

8. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 7 is characterized in that: S3 The dilated 3D convolution is achieved by inserting d-1 values ​​with a weight of 0 between adjacent weights of all rows and columns of the standard 3D convolution, where d is the dilation factor, which expands the receptive field of the convolution kernel without increasing network parameters or losing information; the convolution kernel of the standard 3D convolution is 3×3×3, d=1, and the receptive field is 3×3×3; the convolution kernel of the dilated 3D convolution is 3×3×3, and the dilated 3D convolution kernel structures with d of 2, 3, and 4 are tested, corresponding to receptive fields of 5×5×5, 7×7×7, and 9×9×9, respectively. Convolution kernels with different receptive fields can extract spectral spatial features of different scales in the feature map; The optimal multi-scale 3D-CNN network structure is a model of splicing 3×3×3 dilated 3D convolution kernels with d=2.

9. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1 is characterized in that: In the 1D-CNN stage, a continuous one-dimensional convolution kernel of size 3 is used to extract deep spectral features to obtain a one-dimensional feature of 512×17. The extracted high-level abstract features are then compressed and aggregated based on mean pooling to obtain features with stronger representation capabilities with a dimension of 512×1. Finally, the model prediction value is output through the fully connected layer.

10. The leaf hyperspectral image classification and regression method based on multi-scale cascade convolutional neural network according to claim 1, characterized in that: The benchmark 1D-CNN, benchmark 2D-CNN, benchmark 3D-CNN, and multi-scale 3D-CNN described in S5 include a convolutional layer, a batch normalization layer, a ReLU nonlinear activation layer, a mean pooling layer, and a fully connected layer; The benchmark 1D-CNN is a one-dimensional convolutional neural network based on the residual network framework ResNet, which performs feature extraction in the spectral dimension; The benchmark 2D-CNN is a two-dimensional convolutional neural network based on ResNet, which performs feature extraction in the spatial dimension; The benchmark 3D-CNN is a three-dimensional convolutional neural network based on ResNet, which performs joint feature extraction in spectral + spatial dimensions; The multi-scale 3D-CNN is the best multi-scale 3D-CNN network in S3, which can obtain spectral and spatial features of different scales; All models are trained from scratch, and the initial network weights come from random initialization values; The loss function of the regression model is mean square error, and the loss function of the classification model is cross entropy; The model was optimized using the gradient descent method, with the momentum set to 0.

9. The initial learning rate varied between {1×10−2, 1×10−3, 1×10−4, 1×10−5}. The learning rate decreased by one order of magnitude every 30 training rounds, and the total number of training rounds was 80. The evaluation indicators of regression model performance are determination coefficient R2 and root mean square error RMSE; The evaluation indicators of the classification model are F1-score and accuracy.

Images