Soil organic matter prediction method and apparatus based on nonlinear memory-based learning and spectroscopy

Abstract

Disclosed in the present invention is a soil organic matter prediction method based on nonlinear memory-based learning and spectroscopy, comprising: constructing a sample set on the basis of the real soil organic matter content and spectral data of each soil sample, and dividing the sample set into a training set and a validation set; from the training set, screening for k samples having spectral data most similar to that of each soil sample to be tested to construct a training subset, and fitting a random forest model on the basis of the training subset and by means of ten-fold cross validation to obtain a nonlinear memory-based learning model; verifying the prediction accuracy of the nonlinear memory-based learning model by means of the validation set, when the prediction accuracy reaches a standard, a final nonlinear memory-based learning model being obtained; and inputting the spectral data of the soil sample to be tested into the corresponding final nonlinear memory-based learning model to obtain a predicted value of soil organic matter. According to the method, corresponding soil organic matter contents can be accurately obtained on the basis of spectral data of different soil samples to be tested. Also disclosed in the present invention is a soil organic matter prediction apparatus based on nonlinear memory-based learning and spectroscopy.

Description

Soil organic matter spectrum prediction method and device based on nonlinear memory learning Technical Field

[0001] The present invention belongs to the field of soil property prediction, and in particular relates to a soil organic matter spectrum prediction method and device based on nonlinear memory learning. Background Art

[0002] Soil organic matter refers to the organic matter contained in the soil. It is formed by decomposed or decomposing plant and animal remains and organic matter, and is an indispensable component of the solid phase of the soil. Soil organic matter plays an important role in plant growth and soil fertility. It provides the main source of plant nutrition and can release a large amount of nutrients during the mineralization process, promoting plant growth and development, improving the physical properties of the soil, and accelerating the activity of microorganisms and soil organisms, thereby speeding up the decomposition of nutrients in the soil, enhancing the soil's fertility retention and buffering capacity, and playing a significant role in soil formation, environmental protection, and sustainable agricultural development.

[0003] Soil organic matter helps maintain soil structure, provide nutrients, retain moisture, promote soil biological activity, and improve the soil environment. Furthermore, soil organic matter plays a vital role in the global carbon cycle, contributes to climate change mitigation, and has a certain carbon sequestration capacity. Therefore, soil organic matter is crucial for maintaining healthy soil ecosystems, crop production, and addressing climate change.

[0004] Soil organic matter determination based on traditional laboratory physical and chemical analysis has disadvantages such as cumbersome steps, high cost, poor timeliness and the possible use of environmentally harmful reagents. Therefore, the application of spectroscopic technology in the rapid and accurate determination of soil organic matter over a large area is rapidly expanding.

[0005] Prediction models based on spectral data are a key step in predicting soil organic matter using soil spectroscopy. The accuracy of these predictions is directly determined by the model's capabilities. Soil spectral prediction models have evolved from linear models such as multivariate linear regression, principal component regression, and partial least squares regression to current nonlinear models such as Cubist, random forests, support vector machines, and neural networks.

[0006] Currently, the vast majority of soil spectral prediction models employ a global training strategy, using all training data to establish a fixed prediction model, which is then used to predict all test samples. When using a large soil spectral database to predict test samples, prediction models based on a global training strategy ignore data heterogeneity, assigning uniform variable importance to all samples, and thus limiting the model's predictive power.

[0007] Furthermore, due to the high complexity of soil components and the multiple interactions between the spectra of different components, there is often a highly nonlinear relationship between soil spectra and target soil properties. Therefore, it is urgent to develop a nonlinear soil organic matter prediction model based on a local training strategy.

[0008] Summary of the Invention

[0009] The present invention provides a soil organic matter spectrum prediction method based on nonlinear memory learning, which can accurately obtain the corresponding soil organic matter content based on the spectral data of different soil samples to be tested.

[0010] A specific embodiment of the present invention provides a soil organic matter spectrum prediction method based on nonlinear memory learning, comprising:

[0011] Multiple soil samples were obtained, and a sample set was constructed based on the actual soil organic matter content and spectral data of each soil sample. The sample set was divided into a training set and a validation set using the Kennard-Stone algorithm.

[0012] Constructing a nonlinear memory learning model corresponding to each soil sample to be tested, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain a corresponding nonlinear memory learning model;

[0013] The prediction accuracy of the corresponding nonlinear memory learning model is verified through the validation set. When the prediction accuracy meets the standard, the corresponding final nonlinear memory learning model is obtained;

[0014] The spectral data of the soil sample to be tested is input into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter of the soil sample to be tested.

[0015] Furthermore, the Kennard-Stone algorithm is used to divide the sample set into a training set and a validation set, wherein the step of obtaining the training set includes:

[0016] S1, the sample with the largest spectral distance in the sample set is divided into the training set;

[0017] S2. From the remaining multiple samples, the sample with the largest spectral distance from the sample already included in the training set is included in the training set;

[0018] S3. Iterate step S2 until the number of samples in the training set reaches the sample number threshold, then stop the iteration to obtain the training set.

[0019] Furthermore, the step of assigning a sample having the largest spectral distance from a sample already assigned to the training set to the training set includes:

[0020] The principal component analysis of the spectral data of each sample was performed to select the first three spectral principal component data. The spectral distance between each sample was calculated using the Euclidean distance method, and the two samples with the largest spectral distance were included in the training set.

[0021] Furthermore, the k samples that are most similar to the spectral data of the soil sample to be tested are selected from the training set to form a training subset, wherein the number k of the most similar soil samples is obtained, including:

[0022] Based on the set value range of the most similar soil sample number and the interval of each optimization, the most similar soil sample number is traversed to construct different training subsets. Based on different training subsets, the random forest model is optimized by ten-fold cross validation to obtain the final most similar soil sample number k.

[0023] Furthermore, the prediction accuracy of the corresponding nonlinear memory learning model is verified by the validation set. If it does not meet the standard, the sample set including the training set and the validation set is reconstructed, and the nonlinear memory learning model corresponding to each soil sample to be tested is reconstructed based on the reconstructed training set, and the reconstructed nonlinear memory learning model is verified again by the reconstructed validation set.

[0024] Furthermore, the actual soil organic matter content of each soil sample was obtained, including:

[0025] Each soil sample collected was air-dried, ground, sieved, and then the soil organic matter content was determined by standard laboratory physical and chemical analysis methods.

[0026] Furthermore, spectral data of each soil sample was obtained, including:

[0027] The raw spectral data of the soil samples after air-drying, grinding and sieving were obtained by visible-near infrared spectrometer;

[0028] The original spectral data were spectrally preprocessed to remove the band data with low signal-to-noise ratio at both ends of the spectrum, and the retained spectral data were smoothed, denoised, signal enhanced and resampled to obtain the spectral data of each soil sample.

[0029] Furthermore, the 350-399 nm and 2451-2500 nm bands with low signal-to-noise ratios in the original spectral data were removed, and the spectral data in the 400-2450 nm band was retained.

[0030] Furthermore, the spectral data from 400 to 2450 nm were smoothed and denoised using the Savitzky–Golay method with a second-order polynomial 21 window number, the spectral signal characteristics were enhanced by first-order differentiation, and the spectral data were resampled to 5 nm resolution using a 5 nm moving average window.

[0031] A specific embodiment of the present invention further provides a soil organic matter spectrum prediction device based on nonlinear memory learning, comprising:

[0032] The dataset module is used to obtain multiple soil samples, construct a sample set based on the actual soil organic matter content and spectral data of each soil sample, and divide the sample set into a training set and a validation set using the Kennard-Stone algorithm;

[0033] A training module is used to train a corresponding nonlinear memory learning model based on each soil sample to be tested, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain the corresponding nonlinear memory learning model;

[0034] The verification module is used to verify the prediction accuracy of the corresponding nonlinear memory learning model through the verification set. When the prediction accuracy meets the standard, the corresponding final nonlinear memory learning model is obtained;

[0035] The output module is used to input the spectral data of the soil sample to be tested into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention constructs a training subset close to the spectral data of the soil sample to be tested based on its spectral data, and obtains a corresponding nonlinear memory learning model through training with the training subset. Since the training subset is constructed through the spectral data of the soil sample to be tested, the trained nonlinear memory learning model can reflect the heterogeneity of the corresponding soil sample, and thus can more accurately predict the organic matter content of the soil. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a flow chart of a soil organic matter spectrum prediction method based on nonlinear memory learning provided by an embodiment of the present invention;

[0039] FIG2 is a block diagram of a soil organic matter spectrum prediction device based on nonlinear memory learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings. In this embodiment, the soil organic matter content and soil visible-near infrared spectra of farmland surface soil samples from a city in Guangdong Province are selected for training, and a soil organic matter prediction model is finally obtained.

[0041] A specific embodiment of the present invention provides a soil organic matter spectrum prediction method based on nonlinear memory learning, as shown in FIG1 , comprising:

[0042] (1) Multiple soil samples were obtained, and a sample set was constructed based on the actual soil organic matter content and spectral data of each soil sample. The sample set was divided into a training set and a validation set using the Kennard-Stone algorithm.

[0043] The specific steps for obtaining a soil organic matter dataset provided by a specific embodiment of the present invention are as follows: 720 farmland surface soil samples (0-20 cm) were collected using a random stratified sampling design based on topography, soil type, and land use type. The samples were air-dried, ground, and sieved (2 mm). The samples were then subjected to laboratory physical and chemical analysis to determine their actual soil organic matter content, thus forming the soil organic matter dataset. The soil organic matter data were determined using the potassium dichromate oxidation method.

[0044] The specific steps of the soil spectral dataset provided by the specific embodiment of the present invention are as follows: the spectral data of each soil sample after passing through a 2mm sieve is measured by a visible-near-infrared spectrometer, and the original spectrum of the sample is obtained by averaging three sets of repeated spectra for each soil sample; the visible-near-infrared spectrometer is selected as the ASD FieldSpec 4 spectrometer, whose spectral wavelength range is 350-2500nm and the spectral resampling resolution is 1nm; after measuring every 10 soil samples, the spectrometer needs to be whiteboard calibrated; the 350-399 and 2451-2500nm bands with larger noise in the initial measured soil spectrum are removed, and the spectral data retained is the 400-2450nm band; the spectral data is smoothed, denoised, and the spectral signal characteristics are enhanced using the Savitzky–Golay method with a second-order polynomial 11 window number combined with first-order differential.

[0045] The specific steps for dividing a sample set into a training set and a validation set using the Kennard-Stone algorithm provided in a specific embodiment of the present invention are as follows: a soil organic matter dataset and a corresponding soil spectral dataset are used as the sample set, and the sample set is divided into a training set and a validation set. Using the Kennard-Stone algorithm, 70% of the data in the sample set is assigned to the training set, and the remaining 30% is assigned to the validation set. In this embodiment, the training set includes 504 soil samples, and the validation set includes 216 soil samples. These samples all contain soil organic matter content and corresponding spectral data that has undergone spectral preprocessing.

[0046] In one specific embodiment, the core steps of selecting training set samples using the Kennard-Stone algorithm include: (i) performing principal component analysis on the spectral data to select the first three spectral principal component data; (ii) calculating the pairwise spectral distances between all samples using Euclidean distance; (iii) assigning the two samples with the largest spectral distances to the training set; (iv) searching for the sample with the farthest spectral distance from the selected training set sample among the remaining samples and assigning it to the training set; (v) repeating step (iv) until the number of samples in the training set reaches a sample number threshold, stopping the iteration to obtain the training set. This method can include samples with a larger spectral range in the training set, thereby allowing the spectral data of different soil samples to be tested in subsequent steps to search for samples with similar spectral data from the training set. The samples in the sample set not selected as the training set are the validation set.

[0047] (2) Constructing a nonlinear memory learning model corresponding to each soil sample to be tested based on the training set, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain the corresponding nonlinear memory learning model.

[0048] In one embodiment, a local training strategy is employed to train the nonlinear memory learning model. Specifically, based on the spectral data of each soil sample to be tested, a training subset is formed by searching for the k samples with the most similar spectra in the training set. The corresponding nonlinear memory learning model is then established using this training subset. The specific steps of the nonlinear memory learning model are as follows: (i) principal component analysis is performed on the spectral data of the training set and the soil sample to be tested to select the first three principal components; (ii) the spectral distances between all training samples and the test sample are calculated using Euclidean distance; (iii) the k samples with the closest distance to the test sample are selected as the training subset; and (iv) a random forest prediction model is fitted to the training subset.

[0049] In a specific embodiment, the number of nearest neighbor samples k provided in this embodiment is optimized by performing ten-fold cross validation on the random forest model using a training subset, and the test range of k is 10-200 with an interval of 10. The final k value is optimized to 150.

[0050] The optimal number of trees and branch variables in the random forest in the nonlinear memory learning model provided in this embodiment is determined by ten-fold cross validation. The optimal number of trees and branch variables corresponding to each prediction sample is different.

[0051] (3) Verify the nonlinear memory learning model through the validation set: input the validation set into the established nonlinear memory learning model to obtain the predicted soil organic matter, compare the actual soil organic matter of the real soil samples in the validation set with the predicted soil organic matter, and evaluate the prediction accuracy of the nonlinear memory learning model; when the prediction accuracy meets the standard, the corresponding final nonlinear memory learning model is obtained. If it does not meet the standard, it is necessary to supplement more representative modeling set samples or optimize the spectral preprocessing method to reconstruct the sample set.

[0052] In this example, the coefficient of determination (R 2 ) and root mean square error (RMSE) are used to evaluate the prediction accuracy of the validation set. The accuracy evaluation results of the nonlinear memory learning model and the three prediction models based on all training set data are shown in Table 1. As shown in Table 1, the R 2 is 0.70, and the RMSE is 6.21 g kg -1 , which is better than partial least squares regression, Cubist and random forest models, and has a good prediction effect.

[0053] The specific embodiment of the present invention provides R 2 for:

[0054] The RMSE provided by the specific embodiment of the present invention is:

[0055] Among them, n is the number of samples, y i is the actual soil organic matter content of the i-th sample, is the predicted soil organic matter content of the i-th sample generated by the final nonlinear memory learning model.

[0056] (4) The spectral data of the soil sample to be tested is input into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter of the soil sample to be tested.

[0057] The present invention provides a soil organic matter spectrum prediction device based on nonlinear memory learning, as shown in FIG2 , comprising:

[0058] The dataset module is used to obtain multiple soil samples, construct a sample set based on the actual soil organic matter content and spectral data of each soil sample, and divide the sample set into a training set and a validation set using the Kennard-Stone algorithm;

[0059] A training module is used to train a corresponding nonlinear memory learning model based on each soil sample to be tested, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain the corresponding nonlinear memory learning model;

[0060] The verification module is used to verify the prediction accuracy of the corresponding nonlinear memory learning model through the verification set. When the prediction accuracy meets the standard, the corresponding final nonlinear memory learning model is obtained;

[0061] The output module is used to input the spectral data of the soil sample to be tested into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter.

[0062] In summary, the specific embodiment of the present invention searches for a sample subset closest to the visible-near-infrared spectrum of the sample to be tested from the soil visible-near-infrared spectrum database, fits the random forest prediction model to obtain a nonlinear memory learning model specific to each sample, and obtains a predicted value of soil organic matter content by inputting the soil visible-near-infrared spectrum of the sample to be tested into the final nonlinear memory learning model. Compared with the prior art of obtaining soil organic matter content through laboratory physical and chemical methods, the cost is lower and the efficiency is higher. Compared with the prior art of using a global prediction model of all soil spectral databases, the accuracy is higher.

[0063] The specific embodiment of the present invention establishes a specific prediction model for each sample to be tested through the soil visible-near-infrared spectral database and the nonlinear memory learning model. Therefore, the soil organic matter content of the sample to be tested is obtained based on the soil visible-near-infrared data through the specific nonlinear memory learning model constructed by the present invention, thereby improving the ability to quickly, efficiently and accurately predict the soil organic matter content through soil spectroscopy technology.

[0064] The nonlinear memory learning model proposed in the specific embodiment of the present invention can quickly and accurately predict the soil organic matter content based on soil visible-near infrared spectral data, greatly improving the efficiency of soil information perception while reducing the costs of soil sampling, transportation and testing, and providing a new idea for low-cost, high-efficiency, high-precision, large-scale soil resource surveys and ecological environment modeling.

[0065] Table 1 Comparison of soil organic matter prediction accuracy of different models based on the validation set

Claims

1. A soil organic matter spectrum prediction method based on nonlinear memory learning, characterized in that: include: Multiple soil samples were obtained, and a sample set was constructed based on the real soil organic matter content and spectral data of each soil sample. The sample set was divided into a training set and a validation set using the Kennard-Stone algorithm. Constructing a nonlinear memory learning model corresponding to each soil sample to be tested, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain a corresponding nonlinear memory learning model; The prediction accuracy of the corresponding nonlinear memory learning model is verified through the validation set. When the prediction accuracy reaches the standard, the corresponding final nonlinear memory learning model is obtained; The spectral data of the soil sample to be tested is input into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter of the soil sample to be tested.

2. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 1 is characterized in that: The Kennard-Stone algorithm is used to divide the sample set into a training set and a validation set, wherein the step of obtaining the training set includes: S1, the sample with the largest spectral distance in the sample set is included in the training set; S2, from the remaining multiple samples, the sample with the largest spectral distance from the sample already included in the training set is included in the training set; S3, iterate step S2 until the number of samples in the training set reaches the sample number threshold, then stop the iteration to obtain the training set.

3. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 2 is characterized in that: The step of assigning the sample having the largest spectral distance from the sample already assigned to the training set to the training set comprises: The principal component analysis of the spectral data of each sample was performed to select the first three spectral principal component data, the spectral distance between each sample was calculated by the Euclidean distance method, and the two samples with the largest spectral distance were included in the training set.

4. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 1 is characterized in that: The k samples that are most similar to the spectral data of the soil sample to be tested are selected from the training set to form a training subset, wherein the number of the most similar soil samples k is obtained, including: Based on the set value range of the most similar number of soil samples and the interval of each optimization, the most similar number of soil samples is traversed to construct different training subsets, and the random forest model is optimized by ten-fold cross validation based on different training subsets to obtain the final most similar number of soil samples k.

5. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 1 is characterized in that: The prediction accuracy of the corresponding nonlinear memory learning model is verified by the validation set. If it does not meet the standard, the sample set including the training set and the validation set is reconstructed, and the nonlinear memory learning model corresponding to each soil sample to be tested is reconstructed based on the reconstructed training set, and the reconstructed nonlinear memory learning model is verified again by the reconstructed validation set.

6. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 1 is characterized in that: Get the true soil organic matter content of each soil sample, including: Each soil sample collected was air-dried, ground, sieved, and then the soil organic matter content was determined by standard laboratory physical and chemical analysis methods.

7. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 1 is characterized in that: Spectral data were obtained for each soil sample, including: The raw spectral data of the soil samples after air-drying, grinding and sieving were obtained by visible-near infrared spectrometer; The original spectral data were spectrally preprocessed to remove the band data with low signal-to-noise ratio at both ends of the spectrum, and the retained spectral data were smoothed, denoised, signal enhanced and resampled to obtain the spectral data of each soil sample.

8. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 7, characterized in that: The 350-399nm and 2451-2500nm bands with low signal-to-noise ratios in the original spectral data were removed, and the spectral data in the 400-2450nm band was retained.

9. The soil organic matter spectrum prediction method based on nonlinear memory learning according to claim 8, characterized in that: The spectral data from 400 to 2450 nm were smoothed and denoised using the Savitzky–Golay method with a second-order polynomial 21 window number, the spectral signal characteristics were enhanced using the first-order differential, and the spectral data were resampled to a resolution of 5 nm using a 5 nm moving average window.

10. A soil organic matter spectrum prediction device based on nonlinear memory learning, characterized in that: include: The dataset module is used to obtain multiple soil samples, build a sample set based on the real soil organic matter content and spectral data of each soil sample, and divide the sample set into Divided into training set and validation set; A training module is used to train a corresponding nonlinear memory learning model based on each soil sample to be tested, including: selecting k samples that are most similar to the spectral data of the soil sample to be tested from the training set to construct a training subset, and fitting a random forest model based on the training subset through ten-fold cross validation to obtain the corresponding nonlinear memory learning model; The verification module is used to verify the prediction accuracy of the corresponding nonlinear memory learning model through the verification set, and when the prediction accuracy reaches the standard, the corresponding final nonlinear memory learning model is obtained; The output module is used to input the spectral data of the soil sample to be tested into the corresponding final nonlinear memory learning model to obtain the predicted value of soil organic matter.

Images