A method, system, and medium for measuring soil available potassium
By introducing density clustering and weight selection strategies into the CARS algorithm to screen features of soil near-infrared spectral data, and combining Lasso regression and Bayesian optimization, the computational complexity and overfitting problems of the CARS algorithm in soil available potassium measurement are solved, thereby improving the model's predictive ability and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEILONGJIANG BAYI AGRICULTURAL UNIVERSITY
- Filing Date
- 2024-11-25
- Publication Date
- 2026-07-03
Smart Images
Figure CN119557752B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of soil data measurement technology, specifically to a method, system, and medium for measuring available potassium in soil. Background Technology
[0002] Available potassium in soil (potassium that can be absorbed by plants) is one of the essential nutrients required for plant growth and development. Measuring available potassium in soil helps determine the soil's potassium supply status and avoids over- or under-fertilization. Most potassium exists in exchangeable forms and minerals in the soil, with exchangeable potassium being the main source of available potassium and the form most easily absorbed by plants. By measuring the available potassium content in the soil, we can determine the soil's potassium fertilizer requirements and avoid environmental pollution and resource waste caused by excessive potassium accumulation. Furthermore, potassium deficiency can lead to various symptoms in plants, such as yellowing leaf tips, poor root development, and reduced resistance, seriously affecting crop growth and yield. Therefore, regularly testing the available potassium content in the soil not only improves fertilizer efficiency but also ensures healthy crop growth and reduces unnecessary economic losses.
[0003] Feature extraction is a crucial step in the identification and measurement of available potassium using near-infrared spectroscopy data. Appropriate variable subset selection offers advantages in model accuracy, stability, generality, and interpretability. Variable selection includes two types: variable interval selection and single variable selection. Optimal variable interval search methods include Moving Window Partial Least Squares (MWPLS), Search Combined Moving Window Partial Least Squares (SCMWPLS), and Interval Partial Least Squares (IPLS). Optimal single variable search is based on the principle of minimizing variable importance or collinearity, using methods such as Variable Importance Projection (VIP), Uninformative Variable Elimination (UVE), Principal Component Analysis (PCA), and Competitive Adaptive Reweighted Sampling (CARS). Among these, the CARS method has proven highly effective for spectral data.
[0004] CARS has advantages such as strong adaptability and the ability to handle imbalanced data, showing a strong competitive advantage when applied to spectral feature extraction. However, it also faces problems such as numerous and complex calculation steps, difficulty in parameter setting, and the risk of overfitting, which greatly affect the predictive performance and fitting effect of available potassium. Summary of the Invention
[0005] To address the aforementioned issues, this invention provides a method for measuring available potassium in soil. Based on the traditional CARS algorithm, this method utilizes density clustering principles to provide auxiliary tools for data feature selection by calculating density and weights. Through an adaptive model evaluation mechanism, the most relevant features are selected, ultimately establishing a more predictive model and improving the model's predictive performance and fitting effect for available potassium.
[0006] To achieve the above objectives, the present invention provides the following technical solution.
[0007] This invention provides a method for measuring available potassium in soil, comprising the following steps:
[0008] Extract near-infrared spectral data from the soil sample;
[0009] To solve the Euclidean distance between one row and other rows in the spectral matrix of near-infrared spectral data, the number of distances less than a preset threshold is taken as the nearest neighbor of that row. The spectral matrix is traversed to construct a nearest neighbor array. The density vector is normalized according to the nearest neighbor array, and the transpose of the density vector is assigned the initial weight of the competitive adaptive reweighted sampling method.
[0010] The initial value of the regularization parameter of the Lasso regularization term is determined. The mean square error between the predicted and actual values is used as the optimization objective. The optimal regularization parameter is determined by Bayesian optimization. The Lasso model is fitted based on the optimal regularization parameter and the near-infrared spectral data of the soil to be tested to obtain the Lasso model coefficient matrix and determine the regression coefficients of each spectral feature.
[0011] A competitive adaptive reweighted sampling method with initial weight assignment was used to screen spectral features with non-zero regression coefficients in the near-infrared spectral data of the soil to be tested, forming an effective feature matrix; based on the effective feature matrix, the PLS model was used to obtain the soil available potassium measurement results.
[0012] Preferably, constructing the nearest neighbor array includes the following steps:
[0013] Iterate through the spectral matrix X of the infrared spectral data. (m,n) Each line;
[0014] Calculate the Euclidean distance d between the i-th row and every other row. i :
[0015]
[0016] Where m is the number of samples; n is the number of spectral variables; and x is the element of each row of the spectral matrix.
[0017] Calculate the Euclidean distance d i The number of neighbors less than the threshold k represents the number of nearest neighbors in the i-th row, and the result is stored in the i-th element D of the nearest neighbor count array D. i middle.
[0018] Preferably, the step of assigning the transpose of the density vector as the initial weight of the competitive adaptive reweighted sampling method includes the following steps:
[0019] Calculate the normalized density vector D based on the nearest neighbor count array D. _norm :
[0020] D _norm =D i / ∑D i ;
[0021] The normalized density vector D _norm Transpose and assign the values to the initial weights of the competitive adaptive reweighted sampling method.
[0022] Preferably, obtaining the optimal regularization parameter includes the following steps:
[0023] Define the initial value of the regularization parameter as lambda = 0.1; define the maximum number of iterations for Bayesian optimization as nIterations = 100;
[0024] The mean squared error between the predicted and actual values is used as the optimization objective to construct the objective function:
[0025]
[0026] Where N is the number of samples, Let y be the predicted value for the i-th sample. i Let be the measured value of the i-th sample;
[0027] Based on the sample set, the Bayesian optimization method is used to iteratively find the optimal regularization parameter value, which is the value that minimizes the objective function after the maximum number of iterations.
[0028] The present invention also provides a soil available potassium measurement system, the system comprising:
[0029] processor;
[0030] A memory on which computer programs that can run on the processor are stored;
[0031] The computer program, when executed by the processor, implements the steps of the soil available potassium measurement method.
[0032] The present invention also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the soil available potassium measurement method.
[0033] The beneficial effects of this invention are:
[0034] This invention proposes a method, system, and medium for measuring available potassium in soil. By reforming the traditional CARS algorithm through two methods—initial weight selection and weight update strategy—this invention utilizes density-based clustering to provide auxiliary tools for data feature selection by calculating density and weights. An adaptive model evaluation mechanism selects the most relevant and important features for available potassium in soil, ultimately establishing a more predictive model. This method aims to optimize model performance by dynamically adjusting the feature set, thereby improving the model's predictive performance and fitting effect for available potassium in soil. Attached Figure Description
[0035] Figure 1 This is a flowchart of the soil available potassium measurement method according to an embodiment of the present invention;
[0036] Figure 2 This is the original spectrum of an embodiment of the present invention;
[0037] Figure 3 These are the spectral data after first-order derivative processing according to an embodiment of the present invention;
[0038] Figure 4 This refers to the RMSECV of the original data in this embodiment of the invention;
[0039] Figure 5 This is a graph showing the number of sampling variables in the original data of this invention embodiment;
[0040] Figure 6 This refers to the RMSECV of the data after first-order derivative processing in this embodiment of the invention.
[0041] Figure 7 This is a graph showing the number of data sampling variables after first-order derivative processing according to an embodiment of the present invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0043] Example
[0044] The CARS method filters key features by progressively removing unimportant variables. First, Monte Carlo sampling (MCS) is used to randomly select a certain proportion of data from the sample to build a multivariate calibration model, and the importance of variables is assessed based on the absolute value of the regression coefficients. Next, a forced wavelength selection is performed using an exponentially decreasing function (EDF), eliminating variables with low absolute coefficient values. Then, adaptive reweighted sampling (ARS) is applied, incorporating a roulette wheel selection strategy from genetic algorithms, to randomly select variables based on their probabilities and weights. After these steps, the remaining variables are used to construct new subsets, and the MCS, forced wavelength selection, and ARS calculations are repeated. Finally, the root mean square error (RMSE) of each subset is evaluated using cross-validation (RMSECV), and the subset with the lowest RMSE is selected as the optimal feature set.
[0045] CARS possesses advantages such as strong adaptability and the ability to handle imbalanced data, demonstrating a strong competitive advantage in spectral feature extraction. However, it also faces challenges such as numerous computational steps and complex processes, difficulties in parameter setting, and the risk of overfitting, especially when dealing with large amounts of data with low quality, particularly for measuring available potassium in soil. To address this, this invention proposes an IWCARS (Initial Weight and Weight, I&W) algorithm for measuring available potassium in soil, improved through two methods: initial weight selection and weight update strategies. The specific process is as follows... Figure 1 As shown, it includes the following steps:
[0046] S1: Extract near-infrared spectral data of the soil to be tested.
[0047] S2: Calculate the Euclidean distance between one row and other rows in the spectral matrix of near-infrared spectral data. The number of distances less than a preset threshold is taken as the number of nearest neighbors of that row. Traverse the spectral matrix to construct an array of nearest neighbor counts.
[0048] S3: Normalize the density vector based on the nearest neighbor count array, and assign the transpose of the density vector to the initial weights of the competitive adaptive reweighted sampling method.
[0049] S4: Determine the initial value of the regularization parameter of the Lasso regularization term, and use the mean square error between the predicted and true values as the optimization objective to determine the optimal regularization parameter through Bayesian optimization.
[0050] S5: Fit the Lasso model based on the optimal regularization parameter and the near-infrared spectral data of the soil to be tested, obtain the Lasso model coefficient matrix, and determine the regression coefficients of each spectral feature.
[0051] S6: A competitive adaptive reweighted sampling method is adopted after initial weight assignment to screen spectral features with non-zero regression coefficients in the near-infrared spectral data of the soil to be tested, forming an effective feature matrix.
[0052] S7: The PLS model is used to obtain the measurement results of available potassium in the soil based on the effective feature matrix.
[0053] Furthermore, constructing the nearest neighbor count array includes the following steps:
[0054] S2.1: Iterate through the spectral matrix X of the infrared spectral data. (m,n) Each line.
[0055] S2.2: Calculate the Euclidean distance d between the i-th row and every other row. i :
[0056]
[0057] Where m is the number of samples; n is the number of spectral variables; and x is the element of each row of the spectral matrix.
[0058] S2.3: Calculate the Euclidean distance d i The number of neighbors less than the threshold k represents the number of nearest neighbors in the i-th row, and the result is stored in the i-th element D of the nearest neighbor count array D. i middle.
[0059] S2.4: Calculate the normalized density vector D based on the nearest neighbor count array D. _norm :
[0060] D _norm =D i / ∑D i ;
[0061] The normalized density vector D _norm Transpose and assign the value to the initial_weights of the competitive adaptive reweighted sampling method.
[0062] Furthermore, obtaining the optimal regularization parameters includes the following steps:
[0063] S4.1: Define the initial value of the regularization parameter as lambda = 0.1. The strength of L1 regularization (Lasso regression) is mainly controlled by this parameter, and the optimal value will be found through subsequent optimization. Define the maximum number of iterations for Bayesian optimization as nIterations = 100.
[0064] S4.2: Calculate the mean square error between the predicted and actual values and construct the objective function using this as the optimization objective:
[0065]
[0066] Where N is the number of samples, Let y be the predicted value of the i-th sample. i Let be the measured value of the i-th sample.
[0067] S4.3: Based on the sample set, an iterative optimization method using Bayesian optimization is employed to find the optimal regularization parameter value, which minimizes the objective function after the maximum number of iterations. Ideally, an optimization option structure should be created, and information should not be displayed during optimization to improve computational efficiency or reduce user interface interference.
[0068] This invention proposes an improved IWCARS method. Classic CARS assigns an initial weight of 1 without differentiation, meaning that the relative importance of each sample is assumed to be consistent. This requires more iterations and computational resources. The IWCARS method attempts to provide the relative importance of each sample within a specific interval through a normalized density vector D_norm. The density of a sample reflects its representativeness, based on its density in the feature space. Samples that attract more neighbors are considered more representative in the feature space. By normalizing this density, this method effectively emphasizes important samples while mitigating the impact of unimportant samples on the model, thus facilitating better feature selection and modeling.
[0069] Furthermore, considering that PLS regression is suitable for handling multicollinearity problems and Lasso regression has relatively high computational efficiency, especially with large-scale data, this invention introduces an L1 regularization term into the model to quickly find sparse solutions by penalizing excessively large feature weights. Specifically, a Bayesian method is used to find the important Lasso regression parameter lambda. The L1 regularization optimization measure limits the model complexity, not only reducing the risk of overfitting but also further improving the model's stability and computational efficiency in high-dimensional data.
[0070] In this embodiment, soil samples were collected from a certain location, dried, ground, and sieved before being used as a sample set. The available potassium content of the soil in the sample set was measured using the national standard method.
[0071] When performing quantitative analysis on spectral data and sample physicochemical values, statistical parameters are needed to evaluate the effectiveness of the quantitative model and its predictive ability for unknown samples. Commonly used metrics for evaluating model performance include the coefficient of determination (R²). 2 ), root mean square error (RMSE), relative analysis error (RPD), mean absolute error (MAE), etc.
[0072] Soil spectral data preprocessing was performed using The Unscrambler X 10.3 software, including first derivative, multivariate scattering correction (MSC), standard normal transformation (SNV), Savitzky-Golay filtering (SG), and baseline removal. Feature extraction and regression algorithms were developed in PyCharm 2021.1.1. The feature extraction algorithms included the proposed IWCARS and CARA algorithms, while the regression algorithms included partial least squares (PLS), random forest (RF), gradient boosting machine (GBM), adaptive boosting (AdaBoost), and extreme random tree (ERT).
[0073] Near-infrared spectral data were acquired using a near-infrared spectrometer. The spectral range was 11520–4000 cm⁻¹. -1 The resolution is 8cm. -1 The number of scans was 32. Soil samples were placed in cylindrical quartz cups with a bottom diameter of 4.6 cm and a sample height of 5 mm. The cups were placed in the sample chamber, and the lids were closed. The average of five spectra was used to represent the near-infrared spectrum of the sample, resulting in a total of 95 near-infrared spectra. 76 spectra were used for calibration set training, and 19 spectra were used as validation set data. The original spectra are shown below. Figure 2 As shown.
[0074] In this embodiment, the original spectrum is calibrated using the first derivative, multivariate scattering correction (MSC), standard normal transformation (SNV), Savitzky-Golay filtering (SG), and baseline removal methods. The characteristic variables of the above five preprocessed data and the original data are screened using CARS and IWCARS algorithms, respectively, and the effect is evaluated.
[0075] The accompanying drawings in this embodiment use the original data and the first derivative as examples. Figure 3 The spectral data after processing with the first derivative. Figure 4 and Figure 5 A plot showing the RMSECV of the original data and the number of sampling variables. Figure 6 and Figure 7 The plot shows the RMSECV and the number of sampling variables for first-order derivative data.
[0076] in, Figure 4 and Figure 6 In the meantime, the CARS search is always above the IWCARS search, meaning that the RMSECV obtained by the IWCARS search (2.2089, 0.8637) is less than the RMSECV obtained by the CARS search (2.2299, 0.9036). Figure 5 and Figure 7In the comparison, IWCARS search achieved a greater number of variables (57, 65) with fewer iterations (25, 24), compared to CARS with (26, 25) iterations and (49, 49) variables. For both classes of data, IWCARS search yielded a smaller RMSECV and fewer iterations. This demonstrates that the IWCARS algorithm not only improves model fitting accuracy and reduces search complexity, but this optimization is also stable.
[0077] This invention uses CARS and IWCARS features to construct a partial least squares regression model (PLS) for available potassium content in soil, and further evaluates the effectiveness of IWCARS by measuring the model results.
[0078] Table 1 Comparison of regression model performance based on CARS and IWCARS feature variables
[0079]
[0080]
[0081] Table 1 lists a comparison of evaluation metrics for two feature variable models (IWCARS-PLS and CARS-PLS) with the same preprocessed data, except for Rp for the standard normal transformation data. 2 Rp of all IWCARS-PLS 2 and Rc 2 The IWCARS method is superior to CARS-PLS; the RMSECV, RMSEC, and RMSEP of IWCARS-PLS are all lower than the corresponding indices of CARS-PLS, indicating that the IWCARS method does indeed improve the accuracy of the measurement model compared to the classic CARS method.
[0082] Furthermore, the RMSEP of all six data types (original data, multivariate scattering correction, standard normal transformation, SG smoothing, baseline removal, and first derivative) in CARS-PLS was less than RMSEC, indicating overfitting. However, significantly different from CARS variables, the RMSEP and RMSECV values of the PLS model based on IWCARS variables were not significantly different for all six models, and were all higher than RMSEC. This indicates that IWCARS features effectively avoid the risk of overfitting in traditional CARS methods, and this algorithm is beneficial in overcoming overfitting.
[0083] This invention proposes a novel variable selection method, IWCARS, based on density clustering and combining it with the CARS algorithm by calculating density and weights. This method utilizes an adaptive model evaluation mechanism to select the most relevant features, ultimately building a more predictive model. Studies on soil available potassium measurement show that IWCARS further narrows the search range for variables compared to the traditional CARS algorithm, making variable selection more effective. The IWCARS method was successfully applied to near-infrared spectral analysis of available potassium in soil for variable selection. As the results show, IWCARS further narrows the search range for variables, making variable selection more effective. This indicates that IWCARS can effectively screen important variables related to available potassium in soil, resulting in more reasonable selection of feature variables in regression.
[0084] This embodiment also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for measuring available potassium in soil.
[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0086] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method of measuring available potassium in soil, characterized by, Includes the following steps: Extract near-infrared spectral data from the soil sample; To solve the Euclidean distance between one row and other rows in the spectral matrix of near-infrared spectral data, the number of distances less than a preset threshold is taken as the nearest neighbor of that row. The spectral matrix is traversed to construct a nearest neighbor array. The density vector is normalized according to the nearest neighbor array, and the transpose of the density vector is assigned the initial weight of the competitive adaptive reweighted sampling method. The initial value of the regularization parameter of the Lasso regularization term is determined. The mean square error between the predicted and actual values is used as the optimization objective. The optimal regularization parameter is determined by Bayesian optimization. The Lasso model is fitted based on the optimal regularization parameter and the near-infrared spectral data of the soil to be tested to obtain the Lasso model coefficient matrix and determine the regression coefficients of each spectral feature. A competitive adaptive reweighted sampling method with initial weights was used to screen spectral features with non-zero regression coefficients in the near-infrared spectral data of the soil to be tested, forming an effective feature matrix; based on the effective feature matrix, the PLS model was used to obtain the soil available potassium measurement results. Assigning the transpose of the density vector to the initial weights of the competitive adaptive reweighted sampling method includes the following steps: Based on a number of near neighbors D , compute normalized density vector : ； wherein, is the ith element of the array of neighborhood quantities D; normalizing the density vector transpose and assign as initial weights for the competitive adaptive reweighted sampling method.
2. The method of claim 1, wherein, The construction of the nearest neighbor count array includes the following steps: Looping through the spectral matrix of infrared spectral data of each row; Computing the first i Euclidean distance between rows and each row : ； wherein, is the number of samples; is the number of spectral variables; is the element of each row of the spectral matrix; statistical euclidean distance less than a threshold k the number of distances i of the first row, the result is stored in the number of neighbors array D of the first i element D i smaller than a threshold 3. The method of claim 1, wherein, The process of obtaining the optimal regularization parameter includes the following steps: Define the initial value of the regularization parameter as lambda = 0.1; define the maximum number of iterations for Bayesian optimization as nIterations = 100; The mean squared error between the predicted and actual values is used as the optimization objective to construct the objective function: ； in, N For the sample size, For the first i Predicted value for each sample For the first i Measurement values of each sample; Based on the sample set, the Bayesian optimization method is used to iteratively find the optimal regularization parameter value, which is the value that minimizes the objective function after the maximum number of iterations.
4. A soil available potassium measuring system characterized by, The system includes: processor; A memory on which computer programs that can run on the processor are stored; When the computer program is executed by the processor, it implements the steps of the soil available potassium measurement method as described in any one of claims 1 to 3.
5. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the soil available potassium measurement method as described in any one of claims 1 to 3.