A method to improve soil iron oxide prediction accuracy based on augmented hyperspectral datasets

Abstract

The present invention discloses a method for improving the prediction accuracy of soil iron oxide based on an amplified hyperspectral dataset. The method comprises the following steps: amplifying the hyperspectral dataset and performing multiple fractional-order differential transformations on the original hyperspectral spectrum using a fractional-order differential algorithm; then combining the fractional-order differential spectra with the original spectrum to form different amplified hyperspectral databases to amplify the amount of original spectral data; establishing a 1D-CNN neural network model for each of the different amplified hyperspectral databases, where the independent variable is the hyperspectral reflectance value and the dependent variable is the soil iron oxide content; and selecting the optimal 1D-CNN model based on the models established for the different amplified databases. The present invention effectively solves the problem of low accuracy of deep learning models under small sample conditions, improves the applicability and effectiveness of deep learning models to a certain extent, effectively improves the existing prediction accuracy of soil iron oxide based on hyperspectral data, and provides new methods and ideas for hyperspectral prediction research of other soil properties.

Description

Technical Field

[0001] The present invention relates to soil property prediction technology, and in particular to a method for improving soil iron oxide prediction accuracy based on an amplified hyperspectral data set. Background Art

[0002] Soil iron oxide is a key indicator of soil fertility and water and soil conservation capacity. It also reflects soil leaching processes, weathering development, and zonal distribution. Therefore, soil iron oxide is one of the most useful indicators for describing soil development and soil classification. In summary, the rapid and accurate determination of soil iron oxide content is of great significance for agricultural production and soil genetic classification research.

[0003] Traditional methods for determining soil iron oxide content typically combine soil sampling and laboratory measurements. This intensive soil sampling and laboratory measurement process not only consumes significant manpower, material resources, and financial resources, but also prolongs the data acquisition cycle. In recent years, visible-near-infrared spectroscopy has been widely used for the rapid acquisition of soil properties and has also been shown to be effective for the rapid and accurate determination of soil iron oxide content. Furthermore, with the further development of artificial intelligence (AI) technology, deep learning eliminates the need for handcrafted feature extraction processes, such as preprocessing. Various deep learning convolutional neural network (CNN) models (i.e., AlexNet, VGGnet, GoogLeNet, and ResNet) have been developed and trained on large amounts of data, including over 10 million images. Recently, researchers have applied 1D CNN networks to the rapid acquisition of soil properties. Although CNNs often process images as input data, they have recently been successfully applied to vibration and reflectance spectroscopy. However, the scale of spectral data is often not as large as that of image data, and the collection of soil spectra is often limited by the number of soil samples. Other studies have shown that the size of the training set affects the ability of derived models to predict soil properties. The advantages of using CNNs on small samples are uncertain. When the size of the spectral dataset is less than 1,000, CNN modeling performance is inferior to partial least squares (PLSR) and Cubist models. A recent review of spectroscopy indicates that several studies have used small training samples for deep learning training, and increasing the training sample size can further improve model performance.

[0004] For image data, data amplification is often performed by horizontal flipping, vertical flipping, zooming in, zooming out, changing contrast and brightness, etc. However, no corresponding data amplification method has been proposed for hyperspectral data. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method for improving the prediction accuracy of soil iron oxide based on amplified hyperspectral datasets, thereby improving the existing prediction accuracy of soil iron oxide, fundamentally solving the problem of poor effect of applying CNN models to small-scale datasets, and improving the usability and universality of CNN models.

[0006] Technical solution: The method of improving soil iron oxide prediction accuracy based on an amplified hyperspectral dataset described in the present invention comprises the following steps:

[0007] (1) Amplify the hyperspectral dataset. Use the fractional differential algorithm to perform multiple fractional differential changes on the original hyperspectral data. Then combine the fractional differential spectra with the original spectra to form different amplified hyperspectral databases to amplify the amount of original spectral data.

[0008] The formula for performing multiple fractional-order differential changes on the original hyperspectral spectrum using the fractional-order differential algorithm in step (1) is:

[0009]

[0010] Where f(x) is the spectral reflectance of the corresponding band; v is the differential order; Γ(·) is the gamma function; m is the difference between the upper and lower limits of the differential, that is, the band range;

[0011] According to the above formula, v is taken as 0, 0.1, 0.2, 0.3, ..., 2 respectively. According to the above rule, v takes values ​​​​one by one corresponding to the original spectrum, 0.1-order differential spectrum, 0.2-order differential spectrum, 0.3-order differential spectrum, ..., 2-order differential spectrum, and constructs the amplified spectrum database, such as 0, 0+0.1-order differential spectrum, 0+0.1+0.2-order differential spectrum, ..., 0+0.1+0.2+...+2-order differential spectrum, where the 0-order differential represents the original spectrum data.

[0012] (2) A neural network 1D-CNN model was established for each of the different amplified hyperspectral databases formed. In this case, the independent variable was the hyperspectral reflectance value and the dependent variable was the soil iron oxide content.

[0013] (2.1) 1D-CNN modeling is performed on the amplified differential spectral library, with the input data being spectral data and the output data being soil iron oxide content;

[0014] (2.2) Randomly divide the dataset into a training set and a test set, and then randomly extract a part of the training set as a validation set for parameter tuning;

[0015] (2.3) To avoid overfitting, when the loss function of the validation set no longer decreases significantly, stop training and save the optimal model;

[0016] (2.4) Calculate the coefficient of determination R between the measured value of iron oxide content in the training set and the model prediction value 2 , root mean square error RMSE and relative analytical error RPD to evaluate the quality of the model.

[0017] (3) Select the optimal 1D-CNN model based on the models established by different amplified databases.

[0018] For different amplified data sets, 1DCNN models are established and evaluation indicators are calculated. R 2 The model with high RPD and lowest RMSE was the best model for predicting soil iron oxide content.

[0019] A computer storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for improving the prediction accuracy of soil iron oxide based on an amplified hyperspectral data set.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for improving the prediction accuracy of soil iron oxide based on an amplified hyperspectral data set is implemented.

[0021] Beneficial effects: Compared with the existing technology, the present invention has the following advantages: the present invention effectively solves the problem of low accuracy of deep learning models under small sample conditions, improves the applicability and effectiveness of deep learning models to a certain extent, effectively improves the existing prediction accuracy of soil iron oxide based on hyperspectral data, and provides new methods and ideas for hyperspectral prediction research of other soil properties. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the hyperspectral data amplification process;

[0023] Figure 2 Comparison of calculation results for different amplification databases;

[0024] Figure 3 This is a scatter plot of the measured and predicted values ​​of soil iron oxide in the test set under the optimal model;

[0025] Figure 4 The figure is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 4 As shown, a method for improving the prediction accuracy of soil iron oxide based on an amplified hyperspectral dataset includes the following steps:

[0028] (1) Amplify the hyperspectral dataset. Use the fractional differential algorithm to perform multiple fractional differential changes on the original hyperspectral data. Then combine the fractional differential spectra with the original spectra to form different amplified hyperspectral databases to amplify the amount of original spectral data.

[0029] The formula for performing multiple fractional-order differential changes on the original hyperspectral spectrum using the fractional-order differential algorithm in step (1) is:

[0030]

[0031] Where f(x) is the spectral reflectance of the corresponding band; v is the differential order; Γ(·) is the gamma function; m is the difference between the upper and lower limits of the differential, that is, the band range;

[0032] According to the above formula, v is taken as 0, 0.1, 0.2, 0.3, ..., 2 respectively. According to the above rule, v takes values ​​​​one by one corresponding to the original spectrum, 0.1-order differential spectrum, 0.2-order differential spectrum, 0.3-order differential spectrum, ..., 2-order differential spectrum, and constructs the amplified spectrum database, such as 0, 0+0.1-order differential spectrum, 0+0.1+0.2-order differential spectrum, ..., 0+0.1+0.2+...+2-order differential spectrum, where the 0-order differential represents the original spectrum data.

[0033] (2) A neural network 1D-CNN model was established for each of the different amplified hyperspectral databases formed. In this case, the independent variable was the hyperspectral reflectance value and the dependent variable was the soil iron oxide content.

[0034] (2.1) 1D-CNN modeling is performed on the amplified differential spectral library, with the input data being spectral data and the output data being soil iron oxide content;

[0035] (2.2) Randomly divide 70% of the data into a training set and 30% into a test set. Then, randomly divide 30% of the data in the training set into a validation set for parameter tuning.

[0036] (2.3) To avoid overfitting, when the loss function of the validation set does not decrease in 10 cycles, stop training and save the optimal model;

[0037] (2.4) Calculate the coefficient of determination R between the measured value of iron oxide content in the training set and the model prediction value 2 , root mean square error RMSE and relative analytical error RPD to evaluate the quality of the model.

[0038] (3) Select the optimal 1D-CNN model based on the models established by different amplified databases.

[0039] For different amplified data sets, 1DCNN models were established, and the evaluation indicators were calculated. The model with high R2 and RPD and the lowest RMSE was selected as the best model for predicting soil iron oxide content.

[0040] The present invention is further described in detail by taking 823 sets of spectral data and soil iron oxide data as examples. The spectral range is 400 to 2400 nm, but this is not intended to limit the present invention.

[0041] (1) 823 sets of data are read into Python. The data size is 823 × 2002, that is, 823 rows × 2002 columns, representing 823 samples, 2001 bands, and 1 iron oxide content. They are then stored as a two-dimensional matrix, denoted as matrix X0.

[0042] (2) Calculate the fractional differential spectrum of each sample row by row according to the following formula. The differential order ranges from 0 to 2, and the differential step is 0.1. The formula is:

[0043]

[0044] Where f(x) is the spectral reflectance of the corresponding band; v is the differential order; Γ(·) is the gamma function; and m is the difference between the upper and lower limits of the differential, that is, the band range.

[0045] The fractional-order differential spectrum matrix after differentiation is recorded as X0.1, X0.2, X0.3, …, X2.

[0046] (3) The differential spectrum matrices are combined into an amplified spectrum matrix in a gradually increasing manner, such as X0.1+X0.2, and the combination method is vertical splicing of the matrices.

[0047] (4) For each expanded spectral matrix, a 1D-CNN model was established, with the input data being the spectral data and the output data being the soil iron oxide content. 70% of the data was randomly divided into a training set and 30% of the data into a test set. Then, 30% of the data in the training set was randomly divided into a validation set for parameter tuning. To avoid overfitting, when the loss function of the validation set did not decrease within 10 cycles, the training was stopped and the optimal model was saved. The coefficient of determination R between the measured value of the iron oxide content in the test set and the model prediction value was calculated. 2 , root mean square error RMSE and relative analysis error RPD, such as Figure 2 As shown, to evaluate the quality of the model.

[0048] (5) Select the coefficient of determination R 2The model with the highest relative analysis error RPD and the lowest root mean square error RMSE is the optimal model for predicting soil iron oxide. This data set corresponds to an amplified hyperspectral data set composed of 0-1.2 order differential spectra. The test set R 2 =0.96, RMSE=4.29 g kg -1 and RPD = 5.01, the scatter plot of the measured and predicted values ​​of iron oxide content is as follows Figure 3 As shown, compared with the original spectrum R 2 =0.55, RMSE=14.58 g kg -1 and RPD = 1.47, the prediction accuracy of soil iron oxide was greatly improved.

Claims

1. A method for improving soil iron oxide prediction accuracy based on an amplified hyperspectral dataset, characterized in that: The following steps are involved: (1) Amplify the hyperspectral data set, and use the fractional differential algorithm to perform multiple fractional differential changes on the original hyperspectral data set; then combine the fractional differential spectra with the original spectra to form different amplified hyperspectral databases to amplify the amount of original spectral data; the formula for using the fractional differential algorithm to perform multiple fractional differential changes on the original hyperspectral data set in step (1) is: Where f(x) is the spectral reflectance of the corresponding band; v is the differential order; Γ(·) is the gamma function; m is the difference between the upper and lower limits of the differential, that is, the band range; According to the above formula, v takes values ​​of 0, 0.1, 0.2, 0.3, ..., 2, and v takes values ​​corresponding to the original spectrum, 0.1-order differential spectrum, 0.2-order differential spectrum, 0.3-order differential spectrum, ..., 2-order differential spectrum, respectively, according to the above rule, to construct the amplified spectrum database, where the 0-order differential represents the original spectrum data; (2) A one-dimensional neural network (1D-CNN) model was established for each of the different amplified hyperspectral databases. In this case, the independent variable was the hyperspectral reflectance value, and the dependent variable was the soil iron oxide content. (3) Select the optimal 1D-CNN model based on the models established by different augmented databases; The combination of the fractional-order differential spectrum and the original spectrum specifically includes: using differential algorithms with different differential orders to calculate the original hyperspectrum to obtain differential spectrum matrices with different differential orders, combining the differential spectrum matrices in a gradually increasing manner into an amplified spectrum matrix, and establishing a corresponding 1D-CNN model for each amplified spectrum matrix.

2. The method for improving soil iron oxide prediction accuracy based on amplified hyperspectral datasets according to claim 1, characterized in that: The step (2) is specifically as follows: (2.1) 1D-CNN modeling is performed on the amplified differential spectral library, with the input data being spectral data and the output data being soil iron oxide content; (2.2) Randomly divide the dataset into a training set and a test set, and then randomly extract a part of the training set as a validation set for parameter tuning; (2.3) To avoid overfitting, when the loss function of the validation set no longer decreases significantly, stop training and save the optimal model; (2.4) Calculate the coefficient of determination R2, root mean square error RMSE and relative analytical error RPD of the measured values ​​of iron oxide content in the training set and the model predicted values ​​to evaluate the quality of the model.

3. The method for improving soil iron oxide prediction accuracy based on amplified hyperspectral datasets according to claim 1, characterized in that: The step (3) is specifically as follows: For different amplified data sets, 1DCNN models were established, and the evaluation indicators were calculated. The model with high R2 and RPD and the lowest RMSE was selected as the best model for predicting soil iron oxide content.

4. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for improving soil iron oxide prediction accuracy based on an amplified hyperspectral dataset according to any one of claims 1 to 3 is implemented.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for improving soil iron oxide prediction accuracy based on amplified hyperspectral datasets according to any one of claims 1 to 3 is implemented.

Images