An in-situ spectral moisture removal method based on spectral absorption depth classification

Abstract

The invention discloses an in-situ spectral moisture removal method based on spectral absorption depth classification, including using a clustering method to classify field in-situ wet soil samples based on the difference between the maximum and minimum values ​​of the band range of the spectral absorption depth that can estimate the moisture content and identify the clay mineral content in the in-situ spectrum; randomly selecting the overall sample into a selected sample and a verification sample, and using the K-S method to obtain multiple subset samples from the selected sample; indoor air drying the selected sample, and spectroscopic measurement to obtain an indoor spectrum; performing an external parameter orthogonalization method on each type of indoor spectrum of each sample subset and the in-situ spectrum to obtain a correction matrix of the corresponding class of each subset, and removing moisture from the in-situ spectrum of the corresponding class in the verification sample through the correction matrix of each class of each subset. The method effectively removes the influence of moisture on the in-situ spectrum of different soil types.

Description

Technical Field

[0001] The present invention belongs to a method for quantitatively estimating soil property content, and in particular relates to an in-situ spectral moisture removal method based on spectral absorption depth classification. Background Art

[0002] Indoor soil vis-NIR technology uses spectral reflectance or absorption characteristics in the visible-near-infrared range to infer soil properties. This technology can quantitatively estimate multiple soil properties using a single spectral measurement. It requires the soil to be dried, ground, and sieved, thus avoiding interference from external environmental factors and achieving high prediction accuracy. Compared with traditional analytical methods, indoor vis-NIR technology is environmentally friendly and at least 10 times more efficient and cost-effective, especially for large-scale regional monitoring.

[0003] Consequently, regional, national, and even global spectral libraries have been established, featuring large sample sizes, a comprehensive range of soil types, and rich information content. Field in-situ vis-NIR spectroscopy eliminates the preprocessing required for indoor spectra, such as drying, grinding, and screening. Furthermore, it more closely resembles the imagery of aerial or space-based hyperspectral remote sensing data, laying the foundation for the transition of soil remote sensing from indoor environments to practical applications. Therefore, field in-situ soil spectral estimation techniques based on soil spectral libraries play an important role in large-scale monitoring.

[0004] The spectra in the soil spectral library are indoor spectra. Based on in-situ monitoring of the spectral library, moisture has a very significant impact on the spectra. The complex interaction between soil moisture and different soil types is the main factor affecting the accuracy of soil property prediction in field in-situ measurements. The specific impact includes the following aspects:

[0005] 1. Reflectivity Change: As moisture content increases, soil reflectivity gradually decreases. This is because increased moisture increases the refraction of light in the soil, resulting in a decrease in reflectivity. The effect of moisture on the spectrum is particularly significant around 1400nm, 1900nm, and 2200nm.

[0006] 2. The interactive effect of different moisture contents and different soil types on soil spectral reflectance: With the same moisture content, the spectral absorption of different soil types varies significantly. In the range of 1300-1440nm, the spectral absorption depth and absorption curve shape in the 1840-1940nm and 2100-2240nm regions are significantly different. However, the in situ spectra of soils formed by the same clay minerals have similar absorption characteristics in this region.

[0007] Therefore, using the spectral library built indoors to predict soil properties under in-situ conditions requires dehumidification of the in-situ spectra.

[0008] Current moisture removal methods only consider moisture content. When applied to in situ soil samples from diverse sources and soil types, these methods are ineffective and fail to meet the precision requirements for field monitoring. Therefore, developing effective spectral moisture removal methods for diverse soil types under in situ conditions in complex regions and establishing spectral prediction models using existing spectral library data to accurately predict soil property content in the field is of great significance for current research on arable land quality monitoring and rapid assessment. Summary of the Invention

[0009] The present invention provides an in-situ spectral moisture removal method based on spectral absorption depth classification, which has a good effect on removing moisture influence of in-situ spectra of different soil types.

[0010] A specific embodiment of the present invention provides an in-situ spectral moisture removal method based on spectral absorption depth classification, comprising:

[0011] Original field wet soil samples and their in-situ spectra were obtained, and clustering methods were used to classify the original field wet soil samples into n categories based on the difference between the maximum and minimum values ​​of the spectral bands in the in-situ spectra that can estimate the water content and identify the clay mineral content.

[0012] The classified original field wet soil samples are divided into selection samples and verification samples. Based on the selection samples, the KS method is used to obtain multiple subset samples, where each subset sample and verification sample has n categories.

[0013] The selected samples were air-dried and ground indoors and then subjected to spectral measurement to obtain indoor spectra;

[0014] Based on the indoor spectrum of each class of the selected subset samples and the in-situ spectrum of the corresponding class, the correction matrix of the corresponding class is obtained by the external parameter orthogonalization method. The in-situ spectrum of the corresponding class in the verification sample is dehydrated by the correction matrix of each class to obtain the n-class correction matrix corresponding to each selected subset sample, that is, the n-class correction spectrum.

[0015] Furthermore, the in-situ spectral moisture removal method based on spectral absorption depth classification provided by a specific embodiment of the present invention further includes:

[0016] The soil spectral library data were corrected by using the correction matrix of each class of the selected subset samples, and the partial least squares regression model of each class of each subset sample was obtained based on the corrected soil spectral library data by using the leave-one-out cross-validation method.

[0017] The corrected spectra after water removal in the verification samples based on each subset sample are input into the partial least squares regression model of each category to obtain the predicted soil attribute content based on each subset sample.

[0018] Furthermore, a clustering method was used to classify the original field wet soil samples based on the difference between the maximum and minimum values ​​of the spectral absorption depth band range in the in-situ spectrum, which can be used as an estimate of moisture content and identify clay mineral content, including:

[0019] The in-situ spectra were denoised using the Savitzky-Golay method, and the spectral reflectance was converted into spectral absorbance. The original field wet soil samples were clustered and classified based on the difference between the maximum and minimum values ​​of the bands in the in-situ spectra that can measure moisture content and identify clay minerals based on the spectral absorbance.

[0020] Furthermore, the clustering method is fuzzy clustering, and the number of clusters is determined by the fuzzy performance index, improved classification entropy and cluster independence index of fuzzy clustering, thereby achieving the classification of original field wet soil samples.

[0021] Furthermore, the wavelength range of the spectral absorption depth is 1300-1440nm or 1840-1940nm.

[0022] Furthermore, based on the indoor spectrum and in-situ spectrum of each category, the correction matrix of each category is obtained by the external parameter orthogonalization algorithm, where the calculation formula of the external parameter orthogonalization method is:

[0023] X=XP(p×p)+XQ(p×p)+R(n×p)

[0024] Where X is the in-situ spectrum, XP (p × p) is the spectral information in the in-situ spectrum that is useful for predicting soil properties, P is the correction matrix, Q (p × p) is the useless information in the in-situ spectrum affected by soil moisture content, R (p × p) is the residual matrix, p is the number of spectral bands, and n is the number of soil samples.

[0025] Furthermore, the in-situ spectra of the corresponding class in the validation sample are dehydrated by using the correction matrix of each class. The corrected spectrum V of the i-th class after dehydration is i for:

[0026] V i =S i ×P i

[0027] Among them, S i To verify the in-situ spectrum of the i-th type in the sample, P i is the correction matrix of the i-th class.

[0028] Furthermore, a high-density contact reflection probe was used to collect in-situ visible-near-infrared diffuse reflectance spectroscopy data of the original field wet soil sample to obtain the in-situ spectrum of the original field wet soil sample.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention classifies samples by using the differences in reflectance between wavelengths in the in-situ spectrum that reflect moisture content and clay minerals. It then uses an external parameter orthogonalization (EPO) algorithm to generate a correction matrix for each in-situ spectrum. This correction matrix removes moisture from each in-situ soil spectrum, addressing the ineffectiveness of conventional moisture removal methods under in-situ conditions due to the interaction between moisture and clay minerals. Furthermore, this moisture removal method only considers the in-situ soil spectrum, making it simple and convenient to use.

[0031] The in-situ spectral moisture removal method provided by the present invention corrects soil spectral library data, constructs a partial least squares regression model based on the corrected soil spectral library data, and inputs the corrected spectra of the in-situ spectra obtained by the moisture removal method in the validation set samples into the partial least squares regression model to obtain predicted soil property content. This can solve the problem of low precision caused by poor moisture removal effect when using existing spectral libraries for large-scale in-situ monitoring of soil properties, and provides technical support for precision agricultural soil digital mapping and rapid soil property monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of an in-situ spectral moisture removal method based on spectral absorption depth classification provided by a specific embodiment of the present invention;

[0033] Figure 2 A graph showing the root mean square error (RMSE) of the predictions of the validation samples using the PLSR model established after classification based on different numbers of corrected samples provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] To effectively remove moisture from in-situ spectra, the present invention utilizes the difference between the maximum and minimum values ​​of the spectral absorption band in the in-situ spectra, which can measure moisture content and identify clay minerals, to classify the original field wet soil samples using a clustering method. Then, a correction matrix is ​​obtained using the EPO algorithm, and the correction matrix is ​​used to effectively remove moisture from the in-situ spectra. The present invention thus provides an in-situ spectral moisture removal method based on spectral absorption depth classification, which specifically involves the following steps:

[0036] (1) Obtain the original field wet soil sample spectrum and classify the original field wet soil sample spectrum:

[0037] Original field wet soil samples and their in situ spectra were obtained, and a clustering method was used to classify the original field wet soil samples based on the difference between the maximum and minimum values ​​of the spectral absorption band range in the in situ spectra, which can be used as an estimate of moisture content and identify clay mineral content.

[0038] In a specific embodiment, this embodiment uses a high-density contact reflection probe to collect in-situ visible-near infrared diffuse reflectance spectroscopy data of an original field wet soil sample to obtain the original field wet soil sample, including:

[0039] A total of 298 in situ soil samples were collected from various counties in Jiangxi Province. This study used the ASD FieldSpec ProFR portable field spectrometer. To facilitate field measurements, the instrument was equipped with a high-density contact probe. Measurements were made without affecting crop roots, soil voids, rocks, and other surfaces that are not conducive to spectral measurement.

[0040] Soil samples were collected using a square soil sampler. The soil sample collected using this sampler was a soil sample cube with a depth of 20 cm and a cross-sectional area of ​​10 × 10 cm. Three positions were randomly selected in the upper, middle, and lower parts of the vertical section of the soil sample for field in-situ soil spectrum collection. Ten spectra were collected at each position, and the 30 spectra were averaged as the original field wet soil sample of the topsoil layer at the sampling point.

[0041] In a specific embodiment, the present embodiment classifies the original field wet soil sample, including:

[0042] The in-situ spectra were denoised using the Savitzky-Golay method, and the spectral reflectance was converted into spectral absorbance. Based on the difference between the maximum and minimum values ​​of the bands in the in-situ spectra that can measure moisture content and identify clay minerals, the original field wet soil samples were divided into n categories using a clustering method.

[0043] In a specific embodiment, the clustering method provided by the specific embodiment of the present invention is fuzzy clustering, and the number of clusters is determined by the fuzzy performance index, improved classification entropy and cluster independence index of fuzzy clustering. The number of clusters in this embodiment is 2, that is, the original field wet sample soil is divided into 2 categories.

[0044] In a specific embodiment, the wavelength range of the spectral absorption depth provided by this embodiment that can estimate the water content and identify the clay mineral content is 1300-1440 nm, or 1800-1940 nm.

[0045] (2) In a specific embodiment of the present invention, the original field wet soil sample after cluster classification is randomly divided into selection samples and verification samples, wherein the selection samples and verification samples both include samples of n cluster categories. Based on the selection samples, a series of subset samples are obtained using the KS algorithm: 20, 30, 40...N, where N is the total number of selection samples.

[0046] (3) In a specific embodiment of the present invention, a sample is selected and sent to a laboratory to measure the indoor spectrum, including:

[0047] Using a small aluminum box, 2 cm thick soil samples were collected from the three locations on the selected soil cubes used for the in-situ field measurements. The three soil samples were mixed and placed in a sampling bag, which served as the soil sample for that sampling point. This bag was labeled and brought back to the laboratory for spectral measurement. Each soil sample was ground and sieved, and then subjected to indoor spectral measurement to obtain an indoor spectrum. The indoor spectral measurement method was the same as the in-situ spectral measurement.

[0048] It can be understood that, by using the advanced classification method provided by the present invention, the optional subset samples provided by the specific embodiments of the present invention can be well used to remove the moisture effect of the in-situ spectrum of the verification sample.

[0049] (4) Based on the in-situ spectrum of each class of each subset sample and its indoor spectrum, the EPO correction matrix of each class of each subset is obtained by the external parameter orthogonalization method. The calculation formula of the external parameter orthogonalization method is:

[0050] X=XP(p×p)+XQ(p×p)+R(n×p)

[0051] Where X is the in-situ spectrum, P(p×p) is the spectral information in the in-situ spectrum that is useful for predicting soil properties, P is the correction matrix, Q(p×p) is the useless information of the in-situ spectrum affected by soil moisture content, R(p×p) is the residual matrix, p is the number of spectral bands, and n is the number of soil samples.

[0052] (5) The in-situ spectra of the corresponding class in the verification sample are dehydrated by using the correction matrix of each class of each subset sample to obtain n corrected verification samples of each subset, that is, the n-class correction matrix corresponding to each selected subset sample. The correction spectrum V of the i-th class after dehydration is i for:

[0053] V i =S i ×P i

[0054] Among them, S i To verify the in-situ spectrum of the i-th type in the sample, P i is the correction matrix of the i-th class.

[0055] (6) The soil spectral library data, i.e., the spectral library samples, are corrected using the EPO correction matrix of each category. Based on the corrected soil spectral library data, the partial least squares regression (PLSR) model of each category of its own sample is obtained through the leave-one-out cross-validation method; the corrected spectrum of the verification sample after moisture removal is input into the partial least squares regression model of each category to obtain the predicted soil attribute content.

[0056] In a specific embodiment, the parameters of each type of model are determined by a leave-one-out cross-validation method, and the parameters with the smallest root mean square error (RMSEcv) obtained by the leave-one-out cross-validation are selected as the validation model parameters, and the PLSR model is established using these parameters.

[0057] The accuracy evaluation method provided by the specific embodiment of the present invention uses the evaluation indicators root mean square error (RMSE) and correlation coefficient R 2 The smaller the RMSE of the validation sample, the larger the R, and the better the model effect.

[0058] In a specific embodiment, the present embodiment is classified by comparing the difference between the absorption values ​​at 1300nm and 1440nm, that is, corresponding to Figure 2 The 1440nm classification, and the 1800 and 1940nm absorption difference classification, which corresponds to Figure 2 The 1940nm classification was used to establish a PLSR model based on the in-situ spectrum after classification correction to evaluate the RMSE of the predicted soil attribute content obtained from the validation samples, such as Figure 2 As shown, this embodiment evaluates 17 subset samples, i.e., subset samples with sample sizes of 20, 30…, 170, respectively. The RMSE value corresponding to the predicted soil attribute content output by the PLSR model constructed after classification correction for each subset sample is lower than the RMSE value corresponding to the predicted soil attribute content output by the PLSR model constructed without classification correction. Therefore, no matter how many samples are used for in situ spectral correction, the PLSR model constructed by the method based on spectral absorption depth classification provided by the present invention can more accurately predict the soil attribute content.

Claims

1. An in-situ spectral moisture removal method based on spectral absorption depth classification, characterized in that: include: The original wet soil samples and their in-situ spectra were obtained, and the original wet soil samples were divided into n categories using a clustering method based on the difference between the maximum and minimum values ​​of the spectral band range that can estimate the water content and identify the clay mineral content in the in-situ spectrum. The classified original field wet soil samples are divided into selection samples and verification samples. Based on the selection samples, the KS method is used to obtain multiple subset samples, where each subset sample and verification sample has n categories. The selected samples are air-dried and ground indoors, and then the spectra are measured to obtain indoor spectra; Based on the indoor spectrum of each category of each selected subset sample and its in-situ spectrum, the correction matrix of the corresponding category is obtained by the external parameter orthogonalization method, and the in-situ spectrum of the corresponding category in the verification sample is dehydrated by the correction matrix of each category to obtain the n-category correction matrix corresponding to each selected subset sample; The clustering method is used to classify the original field wet soil samples based on the difference between the maximum and minimum values ​​of the band range of the spectral absorption depth in the in-situ spectrum, which can be used as an estimate of the moisture content and identify the clay mineral content, including: The in-situ spectra were denoised using the Savitzky-Golay method, and the spectral reflectance was converted into spectral absorbance. The original field wet soil samples were clustered and classified based on the difference between the maximum and minimum values ​​of the range of the bands in the in-situ spectra that can measure the moisture content and identify clay minerals based on the spectral absorbance.

2. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: Also includes: The soil spectral library data is corrected by using the correction matrix of each class of each selected subset sample, and the partial least squares regression model of each class of each subset sample is obtained by the leave-one-out cross validation method based on the corrected soil spectral library data; The corrected spectra after water removal in the validation samples based on each subset of samples were input into the partial least squares regression model of each category to obtain the predicted soil attribute content based on each subset of samples.

3. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: The clustering method is fuzzy clustering, and the number of clusters is determined by using the fuzzy performance index, improved classification entropy and cluster independence index of the fuzzy clustering, thereby achieving the classification of the original field wet soil sample.

4. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: The wavelength range of the spectral absorption depth is 1300-1440nm or 1840-1940nm.

5. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: Based on the indoor spectrum and the in-situ spectrum of each class of each subset sample, the correction matrix of each class of each subset sample is obtained by the external parameter orthogonalization algorithm, where the calculation formula of the external parameter orthogonalization method is: X=XP(p×p)+XQ(p×p)+R(n×p) Where X is the in-situ spectrum, XP (p × p) is the spectral information in the in-situ spectrum that is useful for predicting soil properties, P is the correction matrix, Q (p × p) is the useless information of the in-situ spectrum affected by soil moisture content, R (p × p) is the residual matrix, p is the number of spectral bands, and n is the number of soil samples.

6. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: The correction matrix of each class is used to remove moisture from the in-situ spectrum of the corresponding class in the verification sample. The correction spectrum V of the i-th class after moisture removal is i for: V i =S i ×P i Among them, S i To verify the in-situ spectrum of the i-th type in the sample, P i is the correction matrix of the i-th class.

7. The in-situ spectral moisture removal method based on spectral absorption depth classification according to claim 1 is characterized in that: The in-situ visible-near infrared diffuse reflectance spectroscopy data of the original field wet soil sample was collected using a high-density contact reflection probe to obtain the in-situ spectrum of the original field wet soil sample.

Images