A hyperspectral inversion method for soil organic matter content

Abstract

The present invention provides a hyperspectral inversion method for soil organic matter content, comprising: first, collecting soil samples, measuring the organic matter content and spectral data respectively, and completing data acquisition; then, transforming the soil spectrum using continuous wavelet transform to obtain wavelet coefficients of each band; using the SHAP method to calculate the Shaply value between the wavelet coefficients and organic matter content of each band to complete the selection of characteristic bands; finally, dividing the characteristic band wavelet coefficient data and organic matter content data into training sets and test sets respectively, constructing an AI model based on the training set and performing hyperparameter optimization, and verifying the model based on the test set to complete model construction and execution. The present invention can improve the reliability of the inversion model.

Description

Technical Field

[0001] The present invention relates to the field of soil environment monitoring, and in particular to a hyperspectral inversion method for soil organic matter content. Background Art

[0002] Soil organic matter is a general term for carbon-containing organic compounds in the soil. As the main source of nutrients for plant growth, its content directly affects the soil's water and fertilizer retention properties and crop yields. Carrying out large-scale and rapid monitoring of soil organic matter content is of great significance for promoting soil degradation prevention and improving arable land quality. Traditionally, soil organic matter content is mainly determined through chemical analysis of soil samples collected in the field. This method is highly accurate, but the operation is complex, the cycle is long, and the cost is high. It is difficult to quickly obtain organic matter content and distribution information over large areas. Hyperspectral remote sensing has the ability to quickly and macroscopically obtain ground object information, providing a new method for the efficient determination of soil organic matter content. In the hyperspectral inversion of soil organic matter content, the steps that have the greatest impact on accuracy include spectral transformation, feature band selection, and inversion model construction.

[0003] In terms of spectral transformation, conventional methods include differential transformation, reciprocal transformation, inverse logarithmic transformation, envelope removal, etc. These methods can improve the correlation between soil spectra and organic matter content, but cannot achieve spectral transformation of different resolutions. However, the higher the spectral resolution of the inverted soil organic matter content, the better. Moderately reducing the spectral resolution can improve the inversion accuracy of soil organic matter content. Secondly, when the soil organic matter content is low, its spectral characteristics are weak and unstable, and more effective signal processing methods are needed to highlight the spectral characteristics, which brings difficulties to the inversion of soil organic matter content.

[0004] For selecting characteristic bands, the correlation coefficient method and the feature importance method are widely used. The correlation coefficient measures the linear relationship between variables, but the relationship between spectral reflectance and organic matter content is not necessarily linear. The feature importance method accounts for nonlinear relationships but cannot reflect the positive or negative correlation between spectral reflectance and organic matter content. Both methods have shortcomings in selecting characteristic bands.

[0005] In terms of inversion model construction, commonly used models are mainly divided into two categories: statistical analysis models and machine learning models. Statistical analysis models include multivariate stepwise linear regression and partial least squares regression, but the relationship between spectral reflectance and organic matter content is complex, and it is not enough to use only linear models for inversion. Machine learning models are good at simulating nonlinear relationships and have been widely used for soil organic matter content inversion, including eXtreme Gradient Boosting (XGBoost), random forests, convolutional neural networks, support vector machines, etc. However, these models usually use a 7:3 ratio to randomly divide the data set into training and test sets to carry out model training and verification. This random division method is prone to accidental results and cannot guarantee the robustness of the model.

[0006] Therefore, establishing a hyperspectral inversion method for soil organic matter content that comprehensively considers the synergistic and efficient effects of spectral transformation, characteristic band selection, and inversion model construction will improve the accuracy of the inversion results and help provide technical support for soil environmental monitoring. Summary of the Invention

[0007] In view of the above technical problems, the technical solution adopted by the present invention is:

[0008] An embodiment of the present invention provides a hyperspectral inversion method for soil organic matter content, the method comprising the following steps:

[0009] S100, obtain soil organic matter content data set Y = {y1, y2, ..., y i ,…,y n} and soil spectral dataset V = {v1, v2, ..., v i ,…,v n}, the i-th data v i ={v i1 , v i2 ,…,v ij ,…,v im},y i is the organic matter content of the i-th soil sample, v ij is the reflectance of the jth band of the i-th soil sample, i ranges from 1 to n, n is the number of soil samples, j ranges from 1 to m, m is the number of bands;

[0010] S200, for any v ij , perform continuous wavelet transform based on the set k scales and obtain the corresponding wavelet coefficient set w ij ={w 1 ij , w 2 ij ,…,w rij ,…,w k ij},w r ij is the value of v at the rth scale ij The wavelet coefficients obtained by continuous wavelet transform, r ranges from 1 to k;

[0011] S300, based on S200, obtain k wavelet coefficient lists, wherein the rth wavelet coefficient list corresponding to the rth scale Wr={w r 1, w r 2,…,w r i ,…,w r n}, the wavelet coefficient set w of the i-th sample obtained at the r-th scale r i ={w r i1 , w r i2 ,…,w r ij ,…,w r im};

[0012] S400, respectively obtaining Shaply values between the wavelet coefficient and the soil organic matter content at each scale, and obtaining characteristic bands corresponding to each scale from the m bands based on the obtained Shaply values;

[0013] S500, divide the dataset Dr corresponding to the r-th scale into a training set and a test set; Dr = {(wt r 1,y1),…,(wt r i ,y i ),…,(wt r n ,y n )},wt r i is the wavelet coefficient set obtained by continuous wavelet transform of the reflectivity of the z characteristic bands in the i-th sample at the r-th scale, wt r i ={wt r i1 , wt r i2 ,…,wt r is ,…,wt r ip}, wt r isis the wavelet coefficient obtained by performing continuous wavelet transform on the reflectivity of the sth characteristic band in the ith sample at the rth scale, and the value of s ranges from 1 to z;

[0014] S600: Train the constructed rth AI model based on the training set corresponding to the rth scale to obtain a trained AI model r; obtain k trained AI models;

[0015] S700: Obtain the model accuracy of the trained AI model r based on the test set corresponding to the rth AI model, and obtain the model accuracy Pr; obtain k model accuracies;

[0016] S800, obtaining the AI model corresponding to max(P1, P2, ..., Pr, ..., Pk) as the target soil organic matter content inversion model;

[0017] S900: Invert the organic matter content of the target soil sample using the target soil organic matter content inversion model.

[0018] The present invention has at least the following beneficial effects:

[0019] The hyperspectral inversion method for soil organic matter content provided by the embodiment of the present invention comprehensively considers the impact of spectral transformation, characteristic band selection, and inversion model construction on hyperspectral inversion of soil organic matter content, which can improve the reliability of the inversion model. In addition, the present invention can perform spectral transformation according to different resolutions, taking into account the positive and negative correlation between spectral reflectance and organic matter content, as well as the nonlinear relationship between the two. It can more accurately use hyperspectral data to invert soil organic matter content, which can save time and economic costs for soil environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flowchart of the hyperspectral inversion method for soil organic matter content provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] Figure 1 This is a flowchart of the hyperspectral inversion method for soil organic matter content provided by an embodiment of the present invention.

[0024] like Figure 1 As shown, the soil organic matter content hyperspectral inversion method provided by the embodiment of the present invention includes the following steps:

[0025] S100, obtain soil organic matter content data set Y = {y1, y2, ..., y i ,…,y n} and soil spectral dataset V = {v1, v2, ..., v i ,…,v n}, the i-th data v i ={v i1 , v i2 ,…,v ij ,…,v im},y i is the organic matter content of the i-th soil sample, v ij is the reflectance of the jth band of the i-th soil sample, i ranges from 1 to n, n is the number of soil samples, j ranges from 1 to m, m is the number of bands.

[0026] In this embodiment, surface soil, for example, 0-5 cm thick, is primarily collected. The collected soil samples are placed in a cool, well-ventilated area indoors to air-dry, removed of plant debris and rocks, and then ground and passed through a 2 mm sieve. The soil samples are divided equally into two portions: one for chemical analysis to determine soil organic matter content, and the other for soil spectrometry. In one exemplary embodiment, a total of 194 soil samples are collected, so n = 194.

[0027] In the embodiment of the present invention, the soil organic matter content can be determined using existing methods, such as a tube furnace combustion infrared absorption method or a potassium dichromate external heating method.

[0028] In one illustrative embodiment, soil spectral data was collected in a darkroom using a portable spectrometer, the FieldSpec 4. The spectrometer has a wavelength range of 350 to 2500 nm, with a spectral sampling interval of 1 nm. Specifically, a soil sample was placed in a matte black container with a diameter of 12.5 cm and a depth of 2.3 cm. The container was gently shaken to restore the soil surface to its natural state. A 40W halogen lamp was used as the sole light source, with a 45° illumination angle, a probe distance of 22 cm, and a light source distance of 40 cm. Soil reflectance spectra were collected perpendicular to the soil surface. Calibration was performed using a 40 cm x 40 cm whiteboard to obtain absolute reflectance. Ten spectral curves were collected for each soil sample. ViewSpec Pro software was used to remove abnormal curves and average the spectral reflectance to obtain the raw spectral data for the soil sample. Since the signal-to-noise ratios of the 350-400 nm and 2401-2500 nm bands are relatively low, a total of 2000 bands from 401 to 2400 nm can be selected, so m = 2000.

[0029] S200, for any v ij , perform continuous wavelet transform based on the set k scales and obtain the corresponding wavelet coefficient set w ij ={w 1 ij , w 2 ij ,…,w r ij ,…,w k ij},w r ij is the value of v at the rth scale ij The wavelet coefficients obtained by continuous wavelet transform, r ranges from 1 to k.

[0030] In an embodiment of the present invention, Wavelet basis functions Wherein, a is a scale factor. In an exemplary embodiment, k=6. Specifically, the value of a can be 2, 4, 8, 16, 32, or 64. b is a translation factor. In an exemplary embodiment, b=0.

[0031] S300, based on S200, obtain k wavelet coefficient lists, wherein the rth wavelet coefficient list corresponding to the rth scale Wr={w r 1, w r 2,…,w r i ,…,w r n}, the wavelet coefficient set w of the i-th sample obtained at the r-th scale r i={w r i1 , w r i2 ,…,w r ij ,…,w r im}.

[0032] S400 , respectively obtaining Shaply values between the wavelet coefficient and the soil organic matter content at each scale, and obtaining characteristic bands corresponding to each scale from the m bands based on the obtained Shaply values.

[0033] Specifically, an AI model corresponding to each scale can be constructed to obtain k initial AI models. In an embodiment of the present invention, the AI model can be an XGBoost model. The essence of the XGBoost algorithm is the Boosting iterative method, which involves two key parts: the additive model (the strong learner is formed by linearly adding a series of weak learners) and the forward distribution algorithm (the new learner generated in the next round of iteration is trained on the basis of the previous round). Its base learner is a decision tree. Assuming that there are P decision trees in the model, the model adds the results of all decision trees as the final output. The inversion value of the organic matter content of the i-th sample Among them, fp represents the pth decision tree, F is the set of all decision trees, W i is the wavelet coefficient dataset of the i-th sample.

[0034] Among them, w r ij and y i Shaply value between Wherein, N represents the set of all bands, that is, the set of m bands, Q represents a subset of some bands, for example, a subset consisting of at least one band, q is the number of bands it contains, and Q∈N. Indicates the inversion value of the corresponding model when band j is not added, It represents the inversion value of the corresponding model after adding band j. Since the effect of retaining a certain band depends on the other bands that have been input to the model before, it is necessary to traverse all possible subsets Q∈N\{j} and calculate the difference between the model inversion values before and after adding band j. The weighted sum is used as the Shaply value of band j.

[0035] Those skilled in the art know that the specific method for obtaining the Shaply value between the wavelet coefficient and the soil organic matter content at each scale may be a well-known technique in the art.

[0036] Furthermore, the z characteristic bands corresponding to the r-th scale are obtained by the following steps:

[0037] S401, obtain the Shaply value Sr corresponding to the rth scale = {Sr1, Sr2, ..., Srj, ..., Srm}, where

[0038] S402, sort Sr in descending order according to the absolute value of Shaply value to obtain descending sort sequence S down r;

[0039] S403, obtain S down The bands corresponding to the first z Shaply absolute values in r are used as the z feature bands corresponding to the r-th scale, wherein the z-th Shaply absolute value is greater than a set threshold. In an exemplary embodiment, the set threshold is 0.01.

[0040] In an exemplary embodiment, 17, 15, 18, 21, 19, and 20 characteristic bands are selected for each scale, respectively.

[0041] S500, divide the dataset Dr corresponding to the r-th scale into a training set and a test set; Dr = {(wt r 1,y1),…,(wt r i ,y i ),…,(wt r n ,y n )},wt r i is the wavelet coefficient set obtained by continuous wavelet transform of the reflectivity of the z characteristic bands in the i-th sample at the r-th scale, wt r i ={wt r i1 , wt r i2 ,…,wt r is ,…,wt r ip}, wt r is is the wavelet coefficient obtained by performing continuous wavelet transform on the reflectivity of the sth characteristic band in the ith sample at the rth scale, and the value of s ranges from 1 to z.

[0042] In the embodiment of the present invention, the data set Dr corresponding to the rth scale is divided into a training set and a test set using a ten-fold cross validation method. Specifically, the data set Dr is randomly divided into 10 subsets of uniform size {D r1 ,D r2 ,D r3 ,D r4 ,Dr5 ,D r6 ,D r7 ,D r8 ,D r9 ,D r10}, and divide it 10 times; for each randomly divided data set, select 9 subsets as training sets and the remaining 1 subset as test set, and repeat 10 times.

[0043] S600: Train the constructed r-th AI model based on the training set corresponding to the r-th scale to obtain a trained AI model r; obtain k trained AI models.

[0044] During training, the objective function for each iteration can be in, represents the inversion value of the organic matter content of the i-th sample at the t-1th iteration, and l is the loss function, which measures the inversion value of the organic matter content. and the true value y i The error between Ω and Ω is the regularization function to prevent overfitting of the model. The optimal hyperparameter combination is selected by combining the loss function value and the loss function curve to obtain the optimal AI model.

[0045] In an embodiment of the present invention, the root mean square error (RMSE) can be selected as the loss function to optimize the model hyperparameters based on the training set. The selected hyperparameters may include: max_depth, min_child_weight, subsample, colsample_bytree, gamma, alpha, eta, and n_estimators.

[0046] S700, based on the test set corresponding to the rth AI model, obtain the model accuracy of the corresponding trained AI model r, and obtain the model accuracy Pr; obtain k model accuracy.

[0047] The accuracy of the trained model was verified using the test set. For each sample, the average of 10 results was taken as the final organic matter content inversion result. 2 ), mean squared error (MSE), mean absolute error (MAE), and RMSE are used as model accuracy evaluation methods.

[0048] Those skilled in the art will appreciate that the specific calculation model accuracy may be an existing method.

[0049] S800: Obtain the AI model corresponding to max(P1, P2, ..., Pr, ..., Pk) as the target soil organic matter content inversion model.

[0050] S900: Invert the organic matter content of the target soil sample using the target soil organic matter content inversion model.

[0051] The embodiment of the present invention is based on all bands and the characteristic bands selected by the SHAP method. The model accuracy obtained by using continuous wavelet transform and the model accuracy obtained by conventional spectral transformation methods such as reciprocal transform, reciprocal logarithmic transform, first-order differential transform, reciprocal first-order differential transform, and reciprocal logarithmic first-order differential transform can be shown in Tables 1 and 2:

[0052] Table 1 Modeling accuracy of all bands

[0053]

[0054]

[0055] Table 2 Modeling accuracy of characteristic bands selected by SHAP method

[0056]

[0057] From Table 1 and Table 2, we can see that:

[0058] (1) Compared with the original spectrum and conventional spectrum transformation methods, continuous wavelet transform is more conducive to improving the model accuracy;

[0059] (2) Modeling using the characteristic bands selected by the SHAP method has higher accuracy than using all bands;

[0060] (3) In continuous wavelet transform, the model accuracy is highest when the scale is 32.

[0061] In summary, the hyperspectral inversion method for soil organic matter content provided by the embodiment of the present invention comprehensively considers the effects of spectral transformation, characteristic band selection, and inversion model construction on hyperspectral inversion of soil organic matter content, and can improve the reliability of the inversion model. In addition, the present invention can perform spectral transformation according to different resolutions, taking into account the positive and negative correlation between spectral reflectance and organic matter content, as well as the nonlinear relationship between the two. It can more accurately use hyperspectral data to invert soil organic matter content, which can save time and economic costs for soil environmental monitoring.

[0062] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A hyperspectral inversion method for soil organic matter content, characterized in that: The method comprises the following steps: S100, obtain the soil organic matter content data set Y={y1, y2, ..., y i ,…,y n } and soil spectral dataset V = {v1, v2, …, v i ,…,v n }, the i-th data v i ={v i1 , v i2 ,…,v ij ,…,v im },y i is the organic matter content of the i-th soil sample, v ij is the reflectance of the jth band of the i-th soil sample, i ranges from 1 to n, n is the number of soil samples, j ranges from 1 to m, m is the number of bands; S200, for any v ij , perform continuous wavelet transform based on the set k scales and obtain the corresponding wavelet coefficient set w ij ={w 1 ij , w 2 ij ,…,w r ij ,…,w k ij },w r ij is the value of v at the rth scale ij The wavelet coefficients obtained by continuous wavelet transform, r ranges from 1 to k; S300, based on S200, obtain k wavelet coefficient lists, wherein the rth wavelet coefficient list Wr corresponding to the rth scale = {w r 1, w r 2,…,w r i ,…,w r n }, the wavelet coefficient set w of the i-th sample obtained at the r-th scale r i ={w r i1 , w r i2 ,…,w r ij ,…,w r im }; S400, respectively obtaining Shaply values between the wavelet coefficient and the soil organic matter content at each scale, and obtaining characteristic bands corresponding to each scale from the m bands based on the obtained Shaply values; S500, divide the dataset Dr corresponding to the r-th scale into a training set and a test set; Dr={(wt r 1,y1),…,(wt r i ,y i ),…,(wt r n ,y n )}, wt r i is the wavelet coefficient set obtained by continuous wavelet transform of the reflectivity of the z characteristic bands in the i-th sample at the r-th scale, wt r i ={wt r i1 , wt r i2 ,…,wt r is ,…,wt r iz }, wt r is is the wavelet coefficient obtained by continuous wavelet transform of the reflectivity of the sth characteristic band in the ith sample at the rth scale, and the value of s ranges from 1 to z; S600: Train the constructed rth AI model based on the training set corresponding to the rth scale to obtain a trained AI model r; obtain k trained AI models; S700: Obtain the model accuracy of the trained AI model r based on the test set corresponding to the rth AI model, and obtain the model accuracy Pr; obtain k model accuracies; S800, obtaining the AI model corresponding to max(P1, P2, ..., Pr, ..., Pk) as the target soil organic matter content inversion model; S900, inverting the organic matter content of the target soil sample using the target soil organic matter content inversion model; the AI model is an XGBoost model; The z characteristic bands corresponding to the r-th scale are obtained by the following steps: S401, obtain the Shaply value S corresponding to the rth scale r ={S r1 , S r2 ,…,S rj ,…,S rm },in, , w r ij and y i Shaply value between; S402, S r Sort by the absolute value of Shaply value from large to small to get descending sort sequence S down r; S403, obtain S down The bands corresponding to the first z Shaply absolute values in r are used as the z feature bands corresponding to the r-th scale, where the z-th Shaply absolute value is greater than the set threshold.

2. The method according to claim 1, characterized in that Set the threshold to 0~1.

3. The method according to claim 2, characterized in that The threshold value is set to 0.

01.

4. The method according to claim 1, wherein The range of m bands is 350~25000nm.

5. The method according to claim 4, characterized in that The m bands range from 401 to 2400 nm.

6. The method according to claim 1, characterized in that The spectrum sampling interval is 1~100nm.

7. The method according to claim 6, characterized in that The spectrum sampling interval is 1 nm.

8. The method according to claim 1, characterized in that In S500, a ten-fold cross validation method is used to divide the dataset Dr corresponding to the r-th scale into a training set and a test set.

Images