Soil organic carbon spectral prediction method and apparatus based on spectrum-guided ensemble learning

Abstract

Disclosed in the present invention are a soil organic carbon prediction method and apparatus based on spectrum-guided ensemble learning. The method comprises: obtaining soil samples and true organic carbon content and original soil spectra thereof, preprocessing the original soil spectra to obtain soil spectrum samples, constructing a sample set on the basis of the soil spectrum samples, dividing the sample set into a first training set and a validation set, and labelling with the true organic carbon content; training a partial least squares regression model, a Cubist model and a random forest model on the basis of the first training set and the corresponding label, and obtaining a carbon content prediction value set of the three models; and constructing a second training set on the basis of the carbon content prediction value set of the three models and soil spectrum principal component data, and training a second random forest model by means of the second training set and the corresponding label to obtain a spectrum-guided ensemble model. According to the method, the advantages of different prediction models are comprehensively utilized, and the carbon content of soil can be accurately predicted.

Description

Soil organic carbon spectral prediction method and device based on spectral guided ensemble learning Technical Field

[0001] The present invention relates to a soil property prediction method, and in particular to a soil organic carbon spectrum prediction method and device based on spectrum-guided ensemble learning. Background Art

[0002] Soil is the Earth's skin, providing numerous ecosystem services and functions that sustain life on Earth. Soil organic carbon (SOC), a key component of soil organic matter, is an important indicator of soil fertility. Estimating SOC facilitates optimal fertilization and is crucial for increasing crop yields, improving soil quality, and protecting the soil ecosystem.

[0003] Soil organic carbon accounts for over 70% of the carbon pool in terrestrial ecosystems, and the total organic carbon stored in soil exceeds the combined carbon pools in the atmosphere and vegetation. Therefore, even small changes in soil organic carbon can have a significant impact on the global carbon cycle. Given the importance of soil organic carbon to food security and climate change, it is necessary to monitor the dynamic changes in soil organic carbon in a timely manner.

[0004] Since soil organic carbon determination based on traditional laboratory physical and chemical analysis has disadvantages such as long cycle, high cost, and possible use of environmentally harmful reagents, the traditional method of estimating soil organic carbon is based on field sampling and laboratory chemical determination. Field sampling itself requires high manpower costs and is easily restricted by factors such as weather and terrain. Therefore, it is time-consuming and labor-intensive and difficult to accurately estimate soil organic carbon over large areas. Sampling personnel may even face life-threatening dangers. In addition, chemical reagents containing heavy metals such as potassium dichromate are often used during the determination process, which can easily cause environmental pollution or generate reagent processing costs.

[0005] Spectroscopic methods for measuring soil organic carbon are constantly evolving. Compared to mid-infrared spectroscopy, soil visible-near infrared spectroscopy offers advantages such as low cost, high portability, and minimal external interference, making it the most widely used method for measuring soil organic carbon.

[0006] Soil spectral prediction models are the core component of predicting soil organic carbon content using soil spectroscopy. The quality of the prediction model directly determines the reliability of the prediction results. Soil spectral prediction models have evolved from linear models such as multivariate linear regression and partial least squares regression to nonlinear models such as machine learning and deep learning. Due to differences in model structure, a single prediction model has varying applicability to different data, resulting in poor model stability.

[0007] Therefore, it is urgent to design a soil organic carbon prediction model that can comprehensively utilize the advantages of different prediction models, characterize the applicability of different prediction models through spectral guidance, and develop an integrated learning-based soil organic carbon prediction model.

[0008] Summary of the Invention

[0009] The present invention provides a soil organic carbon spectral prediction method based on spectral guided ensemble learning. This method comprehensively utilizes the advantages of different prediction models and can more accurately predict the carbon content of soil.

[0010] A specific embodiment of the present invention provides a soil organic carbon prediction method based on spectral-guided ensemble learning, comprising:

[0011] A plurality of soil samples are obtained, and a true organic carbon content and an original soil spectrum are obtained from each soil sample. Each original soil spectrum is preprocessed to obtain a soil spectrum sample, and a principal component analysis is performed on the soil spectrum sample to obtain soil spectrum principal component data. A sample set is constructed based on the soil spectrum sample, and the sample set is divided into a first training set and a validation set, with the true organic carbon content as a label;

[0012] Based on the first training set and the corresponding labels, a partial least squares regression model, a Cubist model, and a random forest model were trained respectively. Multiple soil spectral samples were input into the trained least squares regression model, the Cubist model, and the first random forest model to obtain the carbon content prediction value sets of the three models respectively.

[0013] A second training set was constructed based on the carbon content prediction value sets of the three models and the soil spectral principal component data. The second random forest model was trained using the second training set and the corresponding labels to obtain a spectrally guided ensemble model.

[0014] When applied, soil spectral samples are input into the spectrally guided ensemble model to obtain the predicted soil organic carbon content.

[0015] Furthermore, a second random forest model is trained using the second training set and corresponding labels to obtain a spectrally guided ensemble model, including:

[0016] Based on the second training set, ten-fold cross validation was used to determine the optimal number of trees and branch variables of the second random forest, and the spectral guided ensemble model was fitted based on the determined optimal number of trees and branch variables.

[0017] Furthermore, the actual organic carbon content and original soil spectrum corresponding to each soil sample were obtained, including:

[0018] Each soil sample was air-dried, ground, and sieved to obtain pretreated soil, which was then analyzed to obtain the true organic carbon content of each soil sample;

[0019] The spectral data of each pretreated soil was measured by visible-near infrared spectrometer to obtain the original soil spectrum.

[0020] Furthermore, each original soil spectrum is preprocessed to obtain a soil spectrum sample, including:

[0021] The band with larger noise in the original soil spectrum is removed, and the band of the retained original soil spectrum is smoothed, denoised and signal enhanced. The band with larger noise is 400-500nm, and the band of the retained original soil spectrum is 500-2450nm.

[0022] Furthermore, a partial least squares regression model is trained based on the first sample set and the corresponding labels, including:

[0023] Based on the first sample set, the optimal number of latent variables in the partial least squares regression model is determined through 10-fold cross validation, and the trained least squares regression model is obtained by fitting based on the optimal number of latent variables.

[0024] Furthermore, a Cubist model is trained based on the first sample set and the corresponding labels, including:

[0025] Based on the first sample set, the optimal number of committee trees in the Cubist model is determined through 10-fold cross validation, and the training of the Cubist model is completed based on the fitting of the optimal number of committee trees.

[0026] Furthermore, based on the first sample set and the corresponding labels, a first random forest model is constructed, including:

[0027] Based on the first sample set, the optimal number of trees and the number of branch variables in the first random forest model are determined through 10-fold cross validation, and the first random forest model is trained and fitted based on the optimal number of trees and the number of branch variables.

[0028] Furthermore, the validation set was input into the spectrally guided ensemble model to obtain the soil organic carbon prediction value set. The prediction accuracy was obtained based on the soil organic carbon prediction value and the corresponding label through the determination coefficient or mean square error. When the prediction accuracy reached the accuracy threshold, the second random forest verification was completed to obtain the final spectrally guided ensemble model.

[0029] Furthermore, the prediction accuracy R is obtained based on the predicted value of soil organic carbon and the corresponding label through the determination coefficient. 2 for:

[0030] Where n is the number of soil spectral samples in the validation set, y i is the label corresponding to the i-th soil spectrum sample, is the predicted value of soil organic carbon of the i-th soil spectral sample;

[0031] Or the prediction accuracy RMSE based on the predicted value of soil organic carbon and the corresponding label is obtained by the mean square error:

[0032] Where n is the number of soil spectral samples in the validation set, y i is the label corresponding to the i-th soil spectrum sample, is the predicted value of soil organic carbon for the i-th soil spectral sample.

[0033] The present invention also provides a soil organic carbon prediction device based on spectral guided ensemble learning, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that the computer memory adopts a spectral guided ensemble model constructed by the soil organic carbon spectral prediction method based on spectral guided ensemble learning;

[0034] When the computer processor executes the computer program, the following steps are implemented:

[0035] Soil spectra were input into the spectrally guided ensemble model to obtain the predicted soil organic carbon content.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention constructs three prediction models based on soil spectral data and the actual soil organic carbon content. The carbon content prediction value sets output by the three prediction models are combined with the principal component data of the soil spectrum as a training set to fit the second random forest model to obtain a spectrally guided ensemble learning model. This can retain the part of each prediction model in which the principal component of the spectrum and the soil carbon content are mapped more accurately, and then retain and combine the bands for accurately predicting the soil carbon content of each model, so that the soil carbon content can be predicted more accurately. Compared with the existing technology of obtaining soil organic carbon content through laboratory physical and chemical methods, the cost is lower and the efficiency is higher. Compared with the existing technology of using a single soil spectral prediction model, the accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a flowchart of a soil organic carbon prediction method based on spectral-guided ensemble learning according to a specific embodiment of the present invention;

[0039] FIG2 is a flow chart of a soil organic carbon prediction method based on spectral-guided ensemble learning according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and technical effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings. In this example, soil organic carbon content and soil visible-near infrared spectra of soil samples from Europe under different land cover and utilization conditions were selected for modeling, ultimately resulting in a soil organic carbon prediction model.

[0041] Since the characteristic spectra used by the single prediction models in the prior art are different, each model has a different optimal prediction interval, so the overall prediction accuracy of the single prediction model is not high. In the specific embodiment of the present invention, the prediction results of the single soil carbon content prediction model and the main components of the soil spectrum are used as training sets to train a new random forest model. The optimal prediction intervals of different single prediction models are mined through spectral guidance, thereby more accurately predicting the soil carbon content. The soil organic carbon prediction method based on spectral guided ensemble learning provided by the specific embodiment of the present invention is shown in Figures 1 and 2, and the specific steps are as follows:

[0042] (1) Obtain soil organic carbon dataset and soil spectral sample dataset:

[0043] In a specific embodiment of the present invention, a soil organic carbon dataset was obtained. The dataset consisted of 18,000 soil samples collected from farmland, woodland, and grassland in Europe according to land cover and use classification maps. The samples were air-dried, ground, and sieved to obtain pretreated soil. The pretreated soil was then subjected to laboratory physical and chemical analysis to determine the true organic carbon content of the soil samples. The soil organic carbon dataset was generated. The soil sampling depth ranged from 0 to 20 cm, and the true organic carbon content was determined using an elemental analyzer.

[0044] A specific embodiment of the present invention obtains a soil spectral sample dataset, including: measuring spectral data of each sieved pretreated soil using a visible-near-infrared spectrometer, and averaging three sets of repeated spectra for each pretreated soil to obtain the original soil spectrum of the sample; selecting a FOSS XDS Rapid Content Analyser spectrometer as the visible-near-infrared spectrometer, whose spectral wavelength range is 400-2500nm and the spectral sampling interval is 0.5nm; performing a whiteboard calibration on the spectrometer after every 10 spectral measurements; removing the 400-500nm band with relatively large noise in the initial spectrum, and retaining the spectral data in the 500-2450nm band; smoothing and denoising the original soil spectrum using the Savitzky–Golay method with a second-order polynomial 101 window number and enhancing the spectral signal characteristics to obtain a soil spectral sample; and selecting the first 20 principal component data of the soil spectral sample through principal component analysis to obtain soil spectral principal component data.

[0045] (2) Construct a single prediction model and a set of carbon content prediction values:

[0046] S1: Construct a sample set based on the soil spectral samples and divide the sample set into a first training set and a validation set. Using the actual organic carbon content as a label, randomly select 75% of the data in the sample set to the first training set, and the remaining 25% to the validation set. In this embodiment, the first training set contains 13,500 soil spectral samples, and the validation set contains 4,500 soil spectral samples.

[0047] S2: Based on the first training set and the corresponding labels, the partial least squares regression model, the Cubist model and the random forest model are trained respectively to obtain three single prediction models. Multiple soil spectral samples are input into the trained least squares regression model, the Cubist model and the first random forest model to obtain the carbon content prediction value sets of the three models respectively, thereby preliminarily obtaining the soil organic carbon prediction results.

[0048] In one specific embodiment, the optimal number of latent variables in the partial least squares regression model is determined through 10-fold cross-validation, and the partial least squares regression model is fitted based on the optimal number of latent variables to obtain the 10-fold cross-validation prediction value for each modeling set sample. In this embodiment, the optimal number of latent variables in the partial least squares regression model is optimized to 19.

[0049] In one specific embodiment, the optimal number of committee trees in the Cubist model is determined through ten-fold cross-validation, and the Cubist model is fitted based on the optimal number of committee trees to obtain the ten-fold cross-validation prediction value for each modeling set sample. In this embodiment, the optimal number of committee trees for the Cubist model is optimized to 5.

[0050] In one specific embodiment, the optimal number of trees and number of branch variables in the first random forest model is determined through ten-fold cross-validation. Based on the optimal number of trees and number of branch variables, the random forest model is fitted to obtain a ten-fold cross-validation prediction value for each modeling set sample. In this embodiment, the optimal number of trees and number of branch variables in the random forest model are optimized to 300 and 8, respectively.

[0051] (3) Constructing a spectrally guided ensemble learning model based on the carbon content prediction value set and soil spectral principal component data:

[0052] S3: Based on the carbon content prediction value sets of the three models and the soil spectral principal component data, a second training set is constructed. The second random forest model is trained using the second training set and the corresponding labels to obtain a spectrally guided ensemble model.

[0053] In one specific embodiment, the optimal number of trees and number of branching variables in the second random forest are determined through ten-fold cross-validation, and the second random forest model is fitted based on the optimal number of trees and number of branching variables to obtain the spectrally guided ensemble model. In this embodiment, the optimal number of trees in the second random forest is optimized to 500, and the optimal number of branching variables is optimized to 5.

[0054] (4) Verify the spectral-guided ensemble learning model: Use the established spectral-guided ensemble learning model to predict the validation set, compare the actual soil organic carbon of the real soil samples in the validation set with the predicted soil organic carbon, and evaluate the prediction accuracy of the spectral-guided ensemble learning model; when the prediction accuracy meets the standard, the established spectral-guided ensemble learning model can be used to predict soil organic carbon.

[0055] In a specific embodiment, the validation set is input into the spectrally guided integration model to obtain a soil organic carbon prediction value set, and the prediction accuracy is obtained based on the soil organic carbon prediction value and the corresponding label through the determination coefficient or the mean square error. When the prediction accuracy reaches the accuracy threshold, the second random forest verification is completed to obtain the final spectrally guided integration model.

[0056] In this example, the coefficient of determination (R 2 ) or root mean square error (RMSE) to evaluate the prediction accuracy of the validation set. The accuracy evaluation results of the spectral guided ensemble learning model and the three single models are shown in Table 1. As shown in Table 1, the R 2 is 0.76, and the RMSE is 9.55 g kg -1 , which is much better than single partial least squares regression, Cubist and random forest models, and has a good prediction effect.

[0057] The specific embodiment of the present invention provides R 2 for:

[0058] The RMSE provided by the specific embodiment of the present invention is:

[0059] Where n is the number of soil spectral samples in the validation set, y i is the label corresponding to the i-th soil spectrum sample, is the predicted value of soil organic carbon of the i-th soil spectral sample;

[0060] Table 1 Comparison of prediction accuracy of different models based on the validation set

[0061] When applied, the soil visible-near infrared spectrum of the soil sample to be tested is input into the final spectrum-guided ensemble learning model to obtain the predicted soil organic carbon content.

[0062] The present invention also provides a soil organic carbon spectrum prediction device based on spectral guided ensemble learning, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory adopts a spectral guided ensemble model constructed by a soil organic carbon spectrum prediction method based on spectral guided ensemble learning.

[0063] The present invention also provides a computer processor that implements the following steps when executing the computer program:

[0064] Soil spectra were input into the spectrally guided ensemble model to obtain the predicted soil organic carbon content.

[0065] The spectral-guided ensemble learning model proposed in this paper can quickly and accurately predict soil organic carbon content based on soil visible-near-infrared spectral data, greatly improving the acquisition efficiency while reducing the cost of soil information acquisition, and providing a new idea for low-cost, high-efficiency, high-precision, large-scale soil resource surveys and ecological environment modeling soil information collection.

[0066] The present invention establishes a mapping relationship between a single model prediction value, spectral principal components and soil organic carbon content through the final spectral-guided ensemble learning model. Therefore, the soil organic carbon content is obtained based on the soil visible-near infrared data through the final spectral-guided ensemble learning model constructed by the present invention, thereby improving the ability to quickly, efficiently and accurately predict soil organic carbon content through soil spectral technology.

Claims

1. A soil organic carbon prediction method based on spectral guided ensemble learning, characterized in that: include: A plurality of soil samples are obtained, and the true organic carbon content and the original soil spectrum are obtained from each soil sample, each original soil spectrum is preprocessed to obtain a soil spectrum sample, a principal component analysis is performed on the soil spectrum sample to obtain soil spectrum principal component data, a sample set is constructed based on the soil spectrum sample, the sample set is divided into a first training set and a validation set, and the true organic carbon content is used as a label; Based on the first training set and the corresponding labels, a partial least squares regression model, a Cubist model and a random forest model are trained respectively, and multiple soil spectral samples are input into the trained least squares regression model, the Cubist model and the first random forest model to obtain the carbon content prediction value sets of the three models respectively; A second training set is constructed based on the carbon content prediction value sets of the three models and the principal component data of soil spectra, and a second random forest model is trained by the second training set and the corresponding labels to obtain a spectral guided integration model; When applied, soil spectral samples are input into the spectrally guided ensemble model to obtain the predicted soil organic carbon content.

2. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1 is characterized in that: The second random forest model is trained by the second training set and the corresponding labels to obtain a spectrally guided ensemble model, including: Based on the second training set, ten-fold cross validation was used to determine the optimal number of trees and branch variables of the second random forest, and the spectral guided ensemble model was fitted based on the determined optimal number of trees and branch variables.

3. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1 is characterized in that: Get the true organic carbon content and original soil spectrum corresponding to each soil sample, including: Each soil sample was air-dried, ground, and sieved to obtain pretreated soil, and the pretreated soil was analyzed to obtain the true organic carbon content of each soil sample; The spectral data of each pretreated soil was measured by visible-near infrared spectrometer to obtain the original soil spectrum.

4. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1 is characterized in that: Each original soil spectrum is preprocessed to obtain a soil spectrum sample, including: The band with larger noise in the original soil spectrum is removed, and the band of the retained original soil spectrum is smoothed, denoised and signal enhanced. The band with larger noise is 400-500nm, and the band of the retained original soil spectrum is 500-2450nm.

5. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1 is characterized in that: A partial least squares regression model is trained based on the first sample set and the corresponding labels, including: Based on the first sample set, the optimal number of latent variables in the partial least squares regression model is determined through 10-fold cross validation, and the trained least squares regression model is fitted based on the optimal number of latent variables.

6. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1, characterized in that: The Cubist model is trained based on the first sample set and the corresponding labels, including: Based on the first sample set, the optimal number of committee trees in the Cubist model is determined through 10-fold cross validation, and the training of the Cubist model is completed based on the fitting of the optimal number of committee trees.

7. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1, characterized in that: A first random forest model based on the first sample set and the corresponding labels includes: Based on the first sample set, the optimal number of trees and the number of branch variables in the first random forest model are determined through 10-fold cross validation, and the first random forest model is trained and completed based on the fitting of the optimal number of trees and the number of branch variables.

8. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 1, characterized in that: The validation set is input into the spectrally guided integrated model to obtain the soil organic carbon prediction value set. The prediction accuracy is obtained by the determination coefficient or mean square error based on the soil organic carbon prediction value and the corresponding label. When the prediction accuracy reaches the accuracy threshold, the second random forest verification is completed to obtain the final spectrally guided integrated model.

9. The soil organic carbon prediction method based on spectral guided ensemble learning according to claim 8, characterized in that: The prediction accuracy R was obtained based on the predicted value of soil organic carbon and the corresponding label by using the determination coefficient. 2 for: Where n is the number of soil spectral samples in the validation set, y i is the label corresponding to the i-th soil spectrum sample, is the predicted value of soil organic carbon of the i-th soil spectral sample; Or the prediction accuracy RMSE based on the predicted value of soil organic carbon and the corresponding label is obtained by the mean square error: Where n is the number of soil spectral samples in the validation set, y i is the label corresponding to the i-th soil spectrum sample, is the predicted value of soil organic carbon for the i-th soil spectral sample.

10. A soil organic carbon prediction device based on spectral guided ensemble learning, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory uses a spectral guided integrated model constructed by the soil organic carbon spectral prediction method based on spectral guided integrated learning according to any one of claims 1 to 9; When the computer processor executes the computer program, the following steps are implemented: The soil spectra were input into the spectrally guided ensemble model to obtain the predicted soil organic carbon content.

Images