Soil organic matter spectrum prediction method based on texture perception residual superposition

Abstract

The invention discloses a soil organic matter spectrum prediction method based on texture perception residual superposition, and the method comprises the steps: converting the classified texture into a dummy variable for model training, thereby directly inputting the texture category and spectrum information into a trained model during application, and obtaining the soil organic matter content. The method does not need to test the texture, does not weaken the advantages of the in-situ spectrum of the soil, and can meet the demand of simply and efficiently predicting the organic matter content of the soil. According to the method, a mapping relation between different textures and residual error predicted values is established by training a second model, so that the residual error model provided by the invention can accurately capture difference values between baseline predicted values under different textures and real soil organic carbon content, and therefore, compared with the prior art, systematic errors can be displayed and corrected, and the accuracy and the reliability of the method are improved. And the model prediction performance is more stable, and the organic carbon content of the soil can be accurately predicted in a larger range of soil with strong heterogeneity.

Description

Technical Field

[0001] This invention belongs to the field of soil spectral quantitative analysis and soil quality monitoring, specifically involving a method for predicting soil organic matter spectrally based on the superposition of texture perception residuals. Background Technology

[0002] Soil spectral libraries are existing resources that have already been established. How to utilize the information from these existing spectral libraries to improve the accuracy of in-situ soil spectral prediction is a scientific problem that urgently needs to be solved. Existing in-situ soil organic matter quantification methods based on spectral libraries (such as PLS, SVR, RF, etc.) are easily affected by changes in soil texture (sand / silt / clay ratio): under the same spectral morphology, due to differences in mineralogical and structural properties, the model bias has a large systematic error in soils with different textures. Moreover, the interaction between in-situ soil moisture and soil texture affects the soil reflectance spectrum, making the differences in in-situ soil spectra more complex, the systematic error larger, and the prediction accuracy lower.

[0003] Patent application CN118169056A discloses a hyperspectral method for predicting soil organic matter content based on DBO-SVR, comprising the following steps: 1) soil sample collection and processing; 2) determination of available potassium content and hyperspectral data of soil samples in step 1); 3) preprocessing of hyperspectral data in step 2) and principal component analysis (PCA) for effective variable extraction; 4) construction of support vector machine regression and partial least squares regression models; 5) training of the DBO-SVR model on the spectral data; 6) establishment of a hyperspectral soil organic matter content prediction model; and 7) model accuracy evaluation. This method can detect soil organic matter content in real time, quickly, and accurately indoors, and has significant practical application value. However, the prediction method disclosed in the above patent application relies on indoor chemical analysis and classification for prediction, and does not fully consider the influence of soil texture on soil organic carbon content.

[0004] CN113436153A discloses a method for predicting carbon composition in undisturbed soil profiles based on hyperspectral imaging and support vector machine (SVM) technology. It obtains hyperspectral images of soil profile samples at preset depths at each sample location, using characteristic spectral bands corresponding to the soil carbon composition type in the target sample's spectral region as input and soil carbon composition data corresponding to that type as output. Through training, a prediction model for the corresponding soil carbon composition type is obtained, thereby predicting the carbon composition of the soil profile in the target area. The entire design can quickly and accurately predict the content of components such as organic carbon, soluble carbon, easily oxidizable carbon, and soil microbial biomass carbon in undisturbed soil profiles, and achieve refined mapping of their spatial distribution on the soil profile, thus overcoming the shortcomings of traditional laboratory chemical analysis methods. However, the prediction accuracy of the method disclosed in this patent application needs further improvement, and the soil carbon composition classification disclosed in the aforementioned patent application still requires laboratory chemical analysis, weakening the advantages of spectroscopy.

[0005] Traditional methods either utilize indoor drying and grinding spectroscopy or require laboratory analysis and classification to improve accuracy, making rapid measurement under in-situ field conditions difficult. They also either ignore soil texture or consider texture information but neglect residuals. Furthermore, even when using texture information, existing methods still require soil texture analysis (clay, sand, and silt content), which diminishes the advantages of in-situ soil spectroscopy. Therefore, there is an urgent need for a new method that can utilize rapidly acquired auxiliary information—texture correction—under in-situ field conditions to improve generalization accuracy and robustness across different texture scenarios without requiring soil texture analysis. Summary of the Invention

[0006] This invention provides a method for predicting soil organic matter spectrally based on the superposition of texture perception residuals. This method can predict soil organic matter content in a relatively simple, efficient and accurate manner.

[0007] This invention provides a method for predicting soil organic matter spectroscopically based on the superposition of texture-perceived residuals, comprising: Based on the sand, silt and clay content of soil samples in the soil spectral library, the texture is classified, and the classified texture is converted into dummy variables. Training sample set and validation sample set are constructed based on the spectral information of each soil sample in the soil spectral library, the corresponding texture dummy variables and the real soil organic carbon content. The first model is obtained. Using the training sample set, spectral information and dummy variables are used as independent variables, and the actual soil organic carbon content is used as the dependent variable. The first model is trained to obtain the global main effect model, and the in-process predicted value is obtained through the global main effect model. The second model is obtained by using the training sample set, taking spectral information and dummy variables as independent variables, and the difference between the in-process predicted value and the actual soil organic carbon content as the dependent variable. The second model is trained to obtain the residual model, and the residual predicted value is obtained through the residual model. When applying the method, the soil spectral information and texture category of the current soil sample are input into the global main effect model to obtain the baseline prediction value. The soil spectral information and texture category of the current soil sample and the baseline prediction value are input into the residual model to obtain the residual prediction value. The final soil organic carbon content prediction value is obtained based on the difference between the baseline prediction value and the residual prediction value.

[0008] Preferably, based on the sand, silt and clay content of soil samples in the soil spectral library, the corresponding textures are divided into three categories: coarse, medium and fine according to rules. Based on the classification results, the texture of each sample is converted into a dummy variable consisting of three columns of binary numbers.

[0009] Preferably, the corresponding textures are divided into three categories according to rules: coarse, medium, and fine, including: When the clay content in a soil sample is ≥35%, the soil sample is considered to have a fine texture. When the sand content in a soil sample is ≥65% and the clay content is ≤18%, the soil sample is coarse. The rest are medium.

[0010] Preferably, a feature matrix is ​​constructed by combining the spectral information of each soil sample with the corresponding dummy variables of texture, and the feature matrix is ​​used as the independent variable for training the first model and the second model.

[0011] Preferably, the step of training the first model includes: Using spectral information and dummy variables as independent variables and actual soil organic carbon content as the dependent variable, a first model is trained using a partial least squares regression model. Latent variables are selected through multi-fold cross-validation to obtain a global main effect model. Training samples are input into the global main effect model to obtain in-fold predicted values, and validation samples are input into the global main effect model to obtain baseline predicted values.

[0012] Preferably, the first model includes supervised learning-elastic network regression, least squares support vector regression, or kernel partial least squares regression.

[0013] Preferably, the second model includes Random Forest, Cubsit, XGBoost, LightGBM, or CatBoost.

[0014] Preferably, before training the first model, the spectral reflectance R is log(1 / R), and then Savitzky-Golay processing is performed. The processed spectrum and dummy variables are fused, and the fused features are used as the dependent variable for training the first model.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention converts the classified texture into dummy variables for model training. As a result, when applied, the soil organic matter content can be obtained by directly inputting the texture category and spectral information into the trained model. This eliminates the need for texture testing, does not diminish the advantages of in-situ soil spectroscopy, and meets the need for simple and efficient prediction of soil organic matter content.

[0016] This invention also establishes a mapping relationship between different soil textures and residual predicted values ​​by training a second model. This enables the residual model provided by this invention to capture the difference between the baseline predicted value and the actual soil organic carbon content under different textures more accurately. Therefore, compared with the prior art, this invention can show the correction of systematic errors, the model prediction performance is more robust, and it can more accurately predict the soil organic carbon content of soils with strong heterogeneity over a wider range. Attached Figure Description

[0017] Figure 1 A flowchart of a soil organic matter spectral prediction method based on texture-perceived residual superposition provided for a specific embodiment of the present invention; Figure 2 A flowchart illustrating the overall process of a soil organic matter spectral prediction method based on texture-perceived residual superposition, provided for a specific embodiment of the present invention. Figure 3 A comparison chart of prediction results provided for a specific embodiment of the present invention. Detailed Implementation

[0018] This invention proposes a texture-aware residual superposition correction scheme: First, a model is built using spectral library data. This model is then used for prediction to obtain the baseline predicted value of the region to be tested. Next, the residuals are modeled a second time using the spectra from the spectral library and the classified textures, resulting in a texture- and spectrum-based residual model. The residual predicted value is obtained through this residual model, treating texture (and spectrum) as "domain information" to learn and offset the systematic bias of the global baseline model in different texture spaces. This model, combined with the spectrum of the region to be tested, yields the residual predicted value (pre_res) for that region. The final predicted value is the difference between the baseline predicted value and the residual predicted value (pre_val - pre_res). This scheme provides targeted correction for systematic errors caused by different textures without sacrificing the global sample size.

[0019] A specific embodiment of the present invention provides a method for predicting soil organic matter spectroscopically based on the superposition of texture-perceived residuals, such as... Figure 1 and Figure 2 As shown, it includes: S1. Construct training and validation sample sets: Based on the texture content of soil samples in the soil spectral library, namely the content of sand, silt and clay (clay, slit and sand), the texture is classified, and the classified texture is converted into dummy variables. Based on the spectral information (Spe) of each soil sample in the soil spectral library, the corresponding texture dummy variables and the true soil organic carbon content (SOM_cal), the training and validation sample sets are constructed.

[0020] In one specific embodiment, the step of obtaining dummy variables of texture through texture encoding provided in this embodiment includes: In this embodiment, based on the sand / silt / clay content of the soil sample in the spectrum, the corresponding texture is divided into three categories: coarse, medium, and fine according to the Chinese three-part texture (coarse / medium / fine) rule, i.e., Texture∈ {coarse, medium, fine}, and converted into dummy variables (such as three columns). That is, based on the classification results, the texture of each sample is converted into a dummy variable consisting of three columns of binary numbers.

[0021] Specifically, the embodiments of this invention provide a method for classifying the corresponding textures into three categories according to rules: coarse, medium, and fine, including: When the clay content in a soil sample is ≥35%, the soil sample is considered to have a fine texture. When the sand content in a soil sample is ≥65% and the clay content is ≤18%, the soil sample is coarse. The rest are medium.

[0022] Using the above method, after obtaining the soil to be tested, there is no need to conduct soil analysis to accurately determine the texture content. Instead, the prediction result can be obtained by directly inputting the coarse, medium, and fine texture categories. In practical applications, the coarse, medium, and fine categories can be determined relatively accurately by hand grinding. Therefore, it is simple and efficient to use, and does not weaken the advantages of in-situ soil spectroscopy.

[0023] In this embodiment, the texture content of each sample is converted into a dummy variable and then concatenated with the spectral variables of the spectral information to form a feature matrix. This feature matrix is ​​then used as the input to the first model and the second model.

[0024] S2. Train the first model based on the training sample set to obtain the global main effect model (PLSR): Using the training sample set, spectral information and dummy variables are used as independent variables, and the actual soil organic carbon content is used as the dependent variable. Train the first model to obtain the global main effect model, and obtain the in-process predicted value through the global main effect model (PLSR).

[0025] In one specific embodiment, the step of training the first model includes: Using spectral information and texture as dummy variables and actual soil organic carbon (SOC) as the dependent variable, a first model is trained using a partial least squares regression model. Latent variables are selected through K-fold cross-validation. In one embodiment, K is 10. Validation can be performed using the root mean square least squares cross-validation (RMSECV) or one-sigma criteria to obtain a global main effects model. Training samples are input into the global main effects model to obtain in-fold predicted values, and validation samples are input into the global main effects model to obtain baseline predicted values.

[0026] Specifically, the first model provided in this embodiment includes supervised learning - Elastic Net regression, least squares support vector regression (LSVR), or kernel partial least squares regression (Kernel PLS).

[0027] S3. Based on the training sample set and the in-between predicted values, train the second model to obtain the residual model, i.e., the residual learning model (RF). Using the training sample set, use spectral information (Pre) and dummy variables (Tex) as independent variables X, and the difference between the in-between predicted values ​​and the actual soil organic carbon content as the dependent variable Y. Train the second model to obtain the residual learning model (RF). Obtain the residual estimate (res_train) through the residual learning model. In one embodiment, the number of trees in the second model is 800-1200, mtry≈ , p The number of independent variables used in the second-level "residual model" is where mtry controls how many feature variables are considered when splitting nodes in each tree.

[0028] In one specific embodiment, the second model provided in this embodiment includes Random Forest, Cubsit, XGBoost, LightGBM, or CatBoost.

[0029] In one specific embodiment, before training the first model, the spectral information is preprocessed with absorbance (log(1 / R)) and Savitzky-Golay smoothing (w = 11, p = 2, m = 0), where w is the window width, p is the polynomial order, and m is the derivative order. The preprocessed light and dummy variables are fused, and the fused features are used as the dependent variable for training the first model. In this embodiment, the absorption and Savitzky-Golay smoothing are introduced into the input, which can more effectively remove spectral noise and moisture information, thereby minimizing spectral noise and improving the accuracy of spectral prediction.

[0030] Before training the first and second models, the field method for judging soil texture requires touching the soil to determine the texture. The texture is classified into three categories: coarse, medium and fine according to the Chinese texture classification, and these three categories are coded. This can not only quickly determine the texture, but also use texture information to eliminate heterogeneity errors caused by texture.

[0031] S4. When applying the method, input the soil spectral information and texture category of the current soil sample into the global main effect model to obtain the baseline prediction value. Input the soil spectral information and texture category of the current soil sample and the baseline prediction value into the residual model to obtain the residual prediction value. Based on the difference between the baseline prediction value and the residual prediction value, the final predicted value of soil organic carbon content is obtained.

[0032] The validation process of the model using the validation set provided in this embodiment is as follows: The soil spectral library spectra (Spe_val) of the validation sample set (validation set) and the empirically perceived texture classification are input into the PLSR model to obtain the baseline prediction of organic matter content of the validation set (SOMval_pre).

[0033] The baseline prediction of organic matter content in the validation set and the soil spectral library spectra (Spe_val) of the validation sample set (validation set) are input into the RF model along with the empirically perceived texture classification to obtain the organic matter prediction residuals (Ses_val) of the validation set.

[0034] The final predicted value of soil organic carbon content (SOM_val_preSes_val) is obtained based on the difference between the baseline predicted value and the residual predicted value.

[0035] Then use RMSE, R 2 RPIQ and LCCC are used to evaluate the model output. A comparison is made between common methods of classifying soil texture using clay, silt, and sand derived from chemical analysis (texture classification in the figure, blue dots), texture perception, and incorporating this texture information (texture incorporation, red dots). The results are as follows: Figure 3 As shown in the diagram. Texture classification involves first dividing the data into three categories, then predicting the outcome for each category, and finally merging the predictions to obtain a statistical result. Texture merging involves standardizing the clay, silt, and sand values ​​and incorporating them into the spectral data for unified modeling and prediction.

Claims

1. A method for predicting soil organic matter spectroscopically based on the superposition of texture-perceived residuals, characterized in that, include: Based on the sand, silt and clay content of soil samples in the soil spectral library, the texture is classified, and the classified texture is converted into dummy variables. Training sample set and validation sample set are constructed based on the spectral information of each soil sample in the soil spectral library, the corresponding texture dummy variables and the real soil organic carbon content. The first model is obtained. Using the training sample set, spectral information and dummy variables are used as independent variables, and the actual soil organic carbon content is used as the dependent variable. The first model is trained to obtain the global main effect model, and the in-process predicted value is obtained through the global main effect model. The second model is obtained by using the training sample set, taking spectral information and dummy variables as independent variables, and the difference between the in-process predicted value and the actual soil organic carbon content as the dependent variable. The second model is trained to obtain the residual model, and the residual predicted value is obtained through the residual model. When applying the method, the soil spectral information and texture category of the current soil sample are input into the global main effect model to obtain the baseline prediction value. The soil spectral information and texture category of the current soil sample and the baseline prediction value are input into the residual model to obtain the residual prediction value. The final soil organic carbon content prediction value is obtained based on the difference between the baseline prediction value and the residual prediction value.

2. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1, characterized in that, Based on the sand, silt, and clay content of soil samples in the soil spectral library, the corresponding textures are divided into three categories: coarse, medium, and fine according to the rules. Based on the classification results, the texture of each sample is converted into a dummy variable consisting of three columns of binary numbers.

3. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 2, characterized in that, According to the rules, the corresponding textures are divided into three categories: coarse, medium, and fine, including: When the clay content in a soil sample is ≥35%, the soil sample is considered to have a fine texture. When the sand content in a soil sample is ≥65% and the clay content is ≤18%, the soil sample is coarse. The rest are medium.

4. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1, characterized in that, A feature matrix is ​​constructed by combining the spectral information of each soil sample with the corresponding dummy variables of texture. This feature matrix serves as the independent variable for training the first and second models.

5. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1, characterized in that, The steps for training the first model include: Using spectral information and dummy variables as independent variables and actual soil organic carbon content as the dependent variable, a first model is trained using a partial least squares regression model. Latent variables are selected through multi-fold cross-validation to obtain a global main effect model. Training samples are input into the global main effect model to obtain in-fold predicted values, and validation samples are input into the global main effect model to obtain baseline predicted values.

6. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1 or 5, characterized in that, The first model includes supervised learning-elastic network regression, least squares support vector regression, or kernel partial least squares regression.

7. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1, characterized in that, The second model includes Random Forest, Cubsit, XGBoost, LightGBM, or CatBoost.

8. The method for predicting soil organic matter spectroscopically based on texture-perceived residual superposition according to claim 1, characterized in that, Before training the first model, the spectral reflectance R is log(1 / R), and then Savitzky-Golay processing is performed. The processed spectrum and dummy variables are fused, and the fused features are used as the dependent variable for training the first model.

Images