A soil carbonate prediction method based on near infrared spectroscopy and transfer learning
By combining soil databases and local data from other regions, using migration component analysis and support vector machine model, the problems of soil carbonate prediction accuracy and applicability are solved, and rapid, accurate and low-cost soil carbonate content prediction is achieved.
Patent Information
- Application Number
- CN202311677514.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-07
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2043-12-07
AI Technical Summary
The prior art has small sample size and low model prediction accuracy in soil carbonate content prediction, and the model is only applicable in local areas, making it difficult to establish a reliable model in areas with scarce data.
Using existing large soil databases in other regions plus a small amount of local soil data, through migration component analysis and support vector machine regression model, a soil carbonate content prediction model suitable for this region is constructed to reduce data differences in different regions and achieve rapid and Accurate soil carbonate content prediction.
Fast, accurate and low-cost soil carbonate content prediction is achieved, suitable for a wide range of areas and reduces the demand for resources and time.
Smart Images
Figure CN117668476B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of soil analysis, and in particular to a soil carbonate prediction method based on near infrared spectroscopy and transfer learning. Background Art
[0002] Soil carbonate is an important component of the soil system and plays an important role in regulating soil pH and providing plant nutrients. In recent years, studies have shown that environmental changes such as the use of nitrogen fertilizers and atmospheric acid deposition have led to global soil acidification, especially in agricultural areas. Soil acidification has significantly accelerated the dissolution of soil carbonates and released carbon dioxide into the atmosphere. Therefore, in the context of global warming and soil acidification, rapid and accurate detection of carbonate content in soil is of vital importance for improving agricultural management and environmental assessment.
[0003] Traditional methods for measuring soil carbonate, such as industry standards, national standards, and international standards, rely on laboratory chemical analysis. Although these methods provide high accuracy, they are complex in operation and heavily rely on the expertise and practical skills of the personnel performing the measurements. Compared with traditional chemical measurement methods, soil spectroscopy provides a cost-effective method. It can estimate various physical and chemical properties of soil, such as soil texture classification and prediction of soil heavy metal content.
[0004] Compared with traditional chemical measurement methods, visible near-infrared spectroscopy detection technology provides a low-cost, economical and fast analysis and detection method. Although some people in this field have tried to use soil spectral data collected by spectrometers to predict soil carbonate content, the sample size is too small, the model prediction accuracy is not high, and the model is only applicable to local areas such as soil collection areas.
[0005] The establishment of near-infrared spectroscopy models depends on reliable databases, and the quality and quantity of databases are often positively correlated with the accuracy of the models. Therefore, it is challenging to establish reliable models in areas where data is scarce. Considering resource and time efficiency, it is often necessary to seek to build more robust and generalizable models with fewer samples. Summary of the invention
[0006] Based on the above problems, the purpose of the present invention is to provide a fast, accurate, low-cost and generally applicable method for predicting soil carbonate content. The method utilizes existing large-scale soil databases from other regions, plus a small amount of soil data collected locally, and uses a transfer component analysis, a transfer learning method, to reduce the difference between soil data from other regions and local soil data, and trains and constructs a support vector machine regression prediction model for soil spectral data and soil carbonate content suitable for the local area. The near-infrared spectral data of the soil to be tested is then processed by the same preprocessing method as the spectral data used for modeling, and then put into the above-mentioned model to predict its soil carbonate content.
[0007] The technical solution adopted by the present invention to achieve its invention object is a soil carbonate prediction method based on near infrared spectroscopy and transfer learning, the steps of which include:
[0008] S1. Obtain an existing public large-scale soil database containing soil carbonate data and corresponding soil near-infrared spectrum data as source domain data; measure and record the carbonate data and near-infrared spectrum data of the locally collected soil according to the method of measuring soil carbonate and near-infrared spectrum in the selected large-scale soil database, and construct a local small-scale soil database as target domain data, wherein the target domain data includes a training part of soil data for modeling, and a part of soil data to be tested;
[0009] Among them, the method for measuring soil carbonate and near-infrared spectrum in the selected large soil database is the existing traditional technology. The training part of the soil data used for modeling needs to measure its carbonate and near-infrared spectrum data, and the part of the soil data to be tested only measures its near-infrared spectrum data.
[0010] S2, preprocessing the source domain spectral data and the target domain spectral data to obtain preprocessed source domain spectral data and target domain spectral data;
[0011] S3, performing migration component analysis spectral transformation on the preprocessed source domain spectral data and target domain spectral data to obtain source domain spectral data and target domain spectral data after migration component analysis spectral transformation;
[0012] S4, performing support vector machine regression modeling on the source domain spectral data and the training part of the target domain spectral data after the migration component analysis spectral transformation, and the soil carbonate data corresponding to these soil spectra, to obtain a support vector machine regression prediction model of the trained soil spectral data and soil carbonate content;
[0013] S5. Input the near infrared spectrum data of the part of soil to be tested in the target domain data into the support vector machine regression prediction model obtained in step S4 to obtain the normalized soil carbonate prediction result of the test soil, and then perform data denormalization processing to obtain the carbonate prediction result of the soil to be tested.
[0014] Further, the preprocessing in step S2 specifically includes:
[0015] S21, remove the spectral data with measurement irregularities in the 400-499.5nm range known as “instrument artifacts”;
[0016] S22, using the downsampling method, retaining one value every 5.5nm, each sample finally contains 363 spectral features;
[0017] S23. Perform first-order derivative transformation on all spectral data.
[0018] Further, the migration component analysis spectrum transformation in step S3 specifically includes:
[0019] S31, input source domain spectral data X S and the target domain spectral data X T ;
[0020] The maximum mean difference distance MMD between the source domain and the target domain is constructed in the high-dimensional regenerated Hilbert space, and the distance formula is as follows:
[0021]
[0022] Among them, n1 and n2 are the number of samples in the source domain and the number of samples in the target domain respectively. and are the spectral feature vectors of the source domain and the target domain respectively, Φ is a kernel-induced feature map, ‖·‖ H Represents a high-dimensional regenerating Hilbert space;
[0023] S32, introduce the kernel matrix K, by minimizing the MMD between the source domain and the target domain data, while maximizing the data variance, using the kernel technique k, that is, k(x i x j )=Φ(x i ) T Φ(x j ), define the kernel matrix K on all spectral data:
[0024]
[0025] Where K S , K S,T , K T,S , K T,Trepresents the kernel matrix of two regions defined by kernel trick k;
[0026] S33, introduce parameter L, its calculation formula is as follows:
[0027]
[0028] S34. Utilization Transform the corresponding eigenvector to m dimension and further construct the kernel function The calculation formula is as follows:
[0029]
[0030] in, for (n s +n t )×m-dimensional matrix, m<<n s +n t ;
[0031] S35, further transform the above distance formula into:
[0032] Dist(X S ,X T )=tr((KWW T K)L)=tr(W T KLKW)
[0033] Among them, tr() means finding the trace of the matrix and minimizing the distance formula, that is:
[0034] min W tr(W T KLKW)+μtr(W T W)
[0035] Among them, μ is the equilibrium parameter, and the formula is subject to the following constraints:
[0036] W T KHKW=I m
[0037] Among them, I m ∈R m is the unit matrix, H is the center matrix, and its formula is as follows:
[0038]
[0039] in, is a matrix of all 1s
[0040] Finally, the calculated W is the spectral data after the migration component analysis spectrum transformation.
[0041] Further, the support vector machine regression modeling in step S4 specifically includes:
[0042] S41, using the training parts of the source domain spectral data and the target domain spectral data as training input data, and using the soil carbonate content data corresponding to these soil spectra as training output data;
[0043] S42, normalizing the input data and the output data respectively;
[0044] S43, transposing the input data and the output data respectively to adapt to the model;
[0045] S44, selecting a suitable kernel function type, and setting suitable penalty factors and radial basis function parameters;
[0046] S45. Construct a support vector machine regression prediction model for soil spectral data and soil carbonate content.
[0047] Transfer Component Analysis (TCA) is a classic feature-based transfer learning method that aims to learn a shared reproducible kernel Hilbert space in which the distance between the marginal distributions of the source and target domains is minimized while maximally preserving their respective intrinsic properties. This allows the feature distributions of the source and target domains to show similar patterns, enabling a model trained in one domain to better adapt to another different but related domain.
[0048] The beneficial effects of the present invention are:
[0049] By making full use of the existing large-scale soil databases in other regions and adding only a small amount of soil data collected locally, a support vector machine regression prediction model of soil spectral data and soil carbonate content suitable for the local area can be trained and constructed, which can realize soil carbonate content prediction quickly, accurately and at low cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0051] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0052] Figure 1 A specific embodiment of the present invention is shown, a soil carbonate prediction method based on near infrared spectroscopy and transfer learning, the steps of which include:
[0053] S1. Obtain an existing public large-scale soil database containing soil carbonate data and corresponding soil near-infrared spectrum data as source domain data; measure and record the carbonate data and near-infrared spectrum data of the locally collected soil according to the method of measuring soil carbonate and near-infrared spectrum in the selected large-scale soil database, and construct a local small-scale soil database as target domain data, wherein the target domain data includes a training part of soil data for modeling, and a part of soil data to be tested;
[0054] In this embodiment, in order to facilitate the establishment of a prediction model, the existing European LUCAS soil spectral library is used, wherein the source domain data uses data sets from other European regions except Austria, and the target domain data uses the Austrian data set;
[0055] S2, preprocessing the source domain spectral data and the target domain spectral data to obtain preprocessed source domain spectral data and target domain spectral data;
[0056] S3, performing migration component analysis spectral transformation on the preprocessed source domain spectral data and target domain spectral data to obtain source domain spectral data and target domain spectral data after migration component analysis spectral transformation;
[0057] S4, performing support vector machine regression modeling on the source domain spectral data and the training part of the target domain spectral data after the migration component analysis spectral transformation, and the soil carbonate data corresponding to these soil spectra, to obtain a support vector machine regression prediction model of the trained soil spectral data and soil carbonate content;
[0058] S5. Input the near infrared spectrum data of the part of soil to be tested in the target domain data into the support vector machine regression prediction model obtained in step S4 to obtain the normalized soil carbonate prediction result of the test soil, and then perform data denormalization processing to obtain the carbonate prediction result of the soil to be tested.
[0059] In this embodiment, the preprocessing in step S2 specifically includes:
[0060] S21, remove the spectral data with measurement irregularities in the 400-499.5nm range known as “instrument artifacts”;
[0061] S22, using the downsampling method, retaining one value every 5.5nm, each sample finally contains 363 spectral features;
[0062] S23. Perform first-order derivative transformation on all spectral data.
[0063] The migration component analysis spectrum transformation in step S3 specifically includes:
[0064] S31, input source domain spectral data XS and the target domain spectral data X T ;
[0065] The maximum mean difference distance MMD between the source domain and the target domain is constructed in the high-dimensional regenerated Hilbert space, and the distance formula is as follows:
[0066]
[0067] Among them, n1 and n2 are the number of samples in the source domain and the number of samples in the target domain respectively. and are the spectral feature vectors of the source domain and the target domain respectively, Φ is a kernel-induced feature map, ‖·‖ H Represents a high-dimensional regenerating Hilbert space;
[0068] S32, introduce the kernel matrix K, by minimizing the MMD between the source domain and the target domain data, while maximizing the data variance, using the kernel technique k, that is, k(x i x j )=Φ(x i ) T Φ(x j ), define the kernel matrix K on all spectral data:
[0069]
[0070] Where K S , K S,T , K T,S , K T,T represents the kernel matrix of two regions defined by kernel trick k;
[0071] S33, introduce parameter L, its calculation formula is as follows:
[0072]
[0073] S34. Utilization Transform the corresponding eigenvector to m dimension and further construct the kernel function The calculation formula is as follows:
[0074]
[0075] in, for (n s +n t )×m-dimensional matrix, m<<n s +n t ;
[0076] S35, further transform the above distance formula into:
[0077] Dist(X S ,X T )=tr((KWW T K)L)=tr(W T KLKW)
[0078] Among them, tr() means finding the trace of the matrix and minimizing the distance formula, that is:
[0079] min W tr(W T KLKW)+μtr(W T W)
[0080] Among them, μ is the equilibrium parameter, and the formula is subject to the following constraints:
[0081] W T KHKW=I m
[0082] Among them, I m ∈R m is the unit matrix, H is the center matrix, and its formula is as follows:
[0083]
[0084] in, is a matrix of all 1s
[0085] Finally, the calculated W is the spectral data after the migration component analysis spectrum transformation.
[0086] The support vector machine regression modeling in step S4 specifically includes:
[0087] S41, using the training parts of the source domain spectral data and the target domain spectral data as training input data, and using the soil carbonate content data corresponding to these soil spectra as training output data;
[0088] S42, normalizing the input data and the output data respectively;
[0089] S43, transposing the input data and the output data respectively to adapt to the model;
[0090] S44, selecting a suitable kernel function type, and setting suitable penalty factors and radial basis function parameters;
[0091] S45. Construct a support vector machine regression prediction model for soil spectral data and soil carbonate content.
Claims
1. A soil carbonate prediction method based on near infrared spectroscopy and transfer learning, characterized in that: It consists of the following steps: S1. Obtain an existing public large-scale soil database containing soil carbonate data and corresponding soil near-infrared spectrum data as source domain data; measure and record the carbonate data and near-infrared spectrum data of the locally collected soil according to the method of measuring soil carbonate and near-infrared spectrum in the selected large-scale soil database, and construct a local small-scale soil database as target domain data, wherein the target domain data includes a training part of soil data for modeling, and a part of soil data to be tested; S2, preprocessing the source domain spectral data and the target domain spectral data to obtain preprocessed source domain spectral data and target domain spectral data; S3, performing migration component analysis spectral transformation on the preprocessed source domain spectral data and target domain spectral data to obtain source domain spectral data and target domain spectral data after migration component analysis spectral transformation; S4, performing support vector machine regression modeling on the source domain spectral data and the training part of the target domain spectral data after the migration component analysis spectral transformation, and the soil carbonate data corresponding to these soil spectra, to obtain a support vector machine regression prediction model of the trained soil spectral data and soil carbonate content; S5, inputting the near infrared spectrum data of the part of the soil to be tested in the target domain data into the support vector machine regression prediction model obtained in step S4, obtaining the normalized soil carbonate prediction result of the test soil, and then performing data denormalization processing to obtain the carbonate prediction result of the soil to be tested; The preprocessing in step S2 specifically includes: S21, remove the spectral data with measurement irregularities in the 400 - 499.5 nm range known as "instrumental artifacts"; S22, using the downsampling method, retaining one value every 5.5 nm, each sample finally contains 363 spectral features; S23, performing first-order derivative transformation on all spectral data; The support vector machine regression modeling in step S4 specifically includes: S41, using the training parts of the source domain spectral data and the target domain spectral data as training input data, and using the soil carbonate content data corresponding to these soil spectra as training output data; S42, normalizing the input data and the output data respectively; S43, transposing the input data and the output data respectively to adapt to the model; S44, selecting a suitable kernel function type, and setting suitable penalty factors and radial basis function parameters; S45. Construct a support vector machine regression prediction model for soil spectral data and soil carbonate content; The migration component analysis spectrum transformation in step S3 specifically includes: S31. Input source domain spectral data and target domain spectral data ; The maximum mean difference distance MMD between the source domain and the target domain is constructed in the high-dimensional regenerated Hilbert space, and the distance formula is as follows: ； in, and are the number of samples in the source domain and the number of samples in the target domain, respectively. and are the spectral feature vector of the source domain and the spectral feature vector of the target domain, respectively. is a kernel-induced feature map, Represents a high-dimensional regenerating Hilbert space; S32, introduce the kernel matrix K, by minimizing the MMD between the source domain and the target domain data while maximizing the data variance, using the kernel technique k, that is, , define the kernel matrix K on all spectral data: ; in , , , represents the kernel matrix of two regions defined by kernel trick k; S33, introduce parameter L, its calculation formula is as follows: ； S34. Utilization Transform the corresponding eigenvector to m dimension and further construct the kernel function , and its calculation formula is as follows: ； in, for dimensional matrix, ; S35, further transform the above distance formula into: ； in, It means finding the trace of the matrix and minimizing the distance formula, that is: ； in, is the equilibrium parameter, and the formula is subject to the following constraints: ； in, is the unit matrix, H is the center matrix, and its formula is as follows: ； in, is a matrix of all 1s, Finally, the calculated W is the spectral data after the migration component analysis spectrum transformation.