Soil quick-acting potassium measuring method, system and medium

Abstract

The invention provides a soil quick-acting potassium measuring method and system and a medium, and relates to the technical field of soil data measurement, and the method comprises the following steps: extracting near infrared spectrum data of soil to be measured; solving Euclidean distances between one row and other rows in the spectrum matrix, and taking the number of distances smaller than a preset threshold value as the number of neighbors of the row; traversing the spectrum matrix, constructing a neighbor number array, normalizing a density vector, and transposing and assigning the density vector as an initial weight of a competitive adaptive reweighted sampling (CARS) method; determining an optimal regularization parameter through a Bayesian optimization method, fitting a Lasso model, obtaining a coefficient matrix of the Lasso model, and determining a regression coefficient of each spectral feature; and screening spectral features with non-zero regression coefficients in the near infrared spectral data of the soil to be detected to form an effective feature matrix. According to the method, the calculation complexity of the CARS method is reduced, parameter setting is simplified, overfitting is prevented, the model fitting precision is improved, and the prediction performance of soil available potassium measurement is improved.

Description

Technical Field

[0001] The invention relates to the technical field of soil data measurement, and in particular to a soil available potassium measurement method, system and medium. Background Art

[0002] Soil available potassium (potassium that can be absorbed by plants) is one of the important nutrients required for plant growth and development. The measurement of soil available potassium helps to determine the potassium supply status of the soil and avoid over-fertilization or under-fertilization. Most of the potassium element exists in the exchangeable state and minerals of the soil, among which exchangeable potassium is the main source of available potassium and the form most easily absorbed by plants. By measuring the available potassium content in the soil, the potassium fertilizer demand of the soil can be determined to avoid environmental pollution and waste of resources caused by excessive accumulation of potassium. In addition, potassium deficiency can cause plants to show various symptoms, such as yellowing of leaf tips, poor root development, reduced resistance, etc., which seriously affect the growth and yield of crops. Therefore, regular testing of the content of available potassium in the soil can not only improve the efficiency of fertilizer use, but also ensure the healthy growth of crops and reduce unnecessary economic losses.

[0003] In the process of identifying and measuring available potassium through near-infrared spectroscopy data, feature extraction is one of the key links. Appropriate variable subset screening has advantages in model accuracy, stability, versatility and interpretability. Variable selection includes two types: variable interval and single variable selection. The optimal variable interval search methods include moving window partial least squares (MWPLS), search combined moving window partial least squares (SCMWPLS) and interval partial least squares (IPLS). The optimal single variable search is based on the importance of the variable or the principle of minimum collinearity, such as variable importance projection (VIP), uninformative variable elimination (UVE) and principal component analysis (PCA), competitive adaptive reweighted sampling method (CARS), etc. Among them, the CARS method has been proven to be very effective when applied to spectral data.

[0004] CARS has the advantages of strong adaptability and ability to process unbalanced data, and shows strong competitive advantages when applied to spectral feature extraction. However, it also faces problems such as many calculation steps and complex processes, difficult parameter setting, and overfitting risks, which greatly affect the prediction performance and fitting effect of fast-acting potassium. Summary of the invention

[0005] To solve the above problems, the present invention provides a soil available potassium measurement method. Based on the traditional CARS algorithm and the principle of density clustering, the method provides an auxiliary tool for data feature screening by calculating density and weight, selects the most relevant features through an adaptive model evaluation mechanism, and finally establishes a model with better predictive ability, thereby improving the prediction performance and fitting effect of the model for available potassium.

[0006] To achieve the above object, the present invention provides the following technical solutions.

[0007] The present invention provides a method for measuring soil available potassium, comprising the following steps:

[0008] Extract near infrared spectrum data of the soil to be tested;

[0009] Solve the Euclidean distance between one row and other rows in the spectrum matrix of near-infrared spectrum data, take the number of distances less than the preset threshold as the number of neighbors of the row, traverse the spectrum matrix, and construct an array of the number of neighbors; normalize the density vector according to the array of the number of neighbors, and transpose the density vector and assign it as the initial weight of the competitive adaptive reweighted sampling method;

[0010] Determine the initial value of the regularization parameter of the Lasso regularization term, take the mean square error between the predicted value and the true value as the optimization target, and determine the optimal regularization parameter through the Bayesian optimization method; fit the Lasso model according to the optimal regularization parameter and the near-infrared spectrum data of the soil to be tested, obtain the Lasso model coefficient matrix, and determine the regression coefficient of each spectral feature;

[0011] The competitive adaptive reweighted sampling method after initial weight assignment was used to screen the spectral features with non-zero regression coefficients in the near-infrared spectral data of the tested soil to form an effective feature matrix. The PLS model was used based on the effective feature matrix to obtain the soil available potassium measurement results.

[0012] Preferably, the constructing of the neighbor quantity array comprises the following steps:

[0013] Loop through the spectrum matrix X of infrared spectrum data (m,n) Each row of

[0014] Calculate the Euclidean distance d between the i-th row and each row i :

[0015]

[0016] Where m is the number of samples; n is the number of spectral variables; x is the element of each row of the spectral matrix;

[0017] Statistical Euclidean distance d i The number of distances less than the threshold k in the ith row represents the number of neighbors in the ith row, and the result is stored in the ith element D of the neighbor number array D. i middle.

[0018] Preferably, the step of assigning the density vector transposition as the initial weight of the competitive adaptive reweighted sampling method comprises the following steps:

[0019] Based on the neighbor count array D, calculate the normalized density vector D _norm :

[0020] D _norm =D i / ∑D i ;

[0021] The normalized density vector D _norm Transpose and assign the initial weights to the competitive adaptive reweighted sampling method.

[0022] Preferably, the acquisition of the optimal regularization parameter comprises the following steps:

[0023] Define the initial value of the regularization parameter lambda = 0.1; define the maximum number of iterations of Bayesian optimization nIterations = 100;

[0024] The mean square error between the predicted value and the true value is calculated as the optimization target to construct the objective function:

[0025]

[0026] Where N is the number of samples, is the predicted value of the i-th sample, y i is the measured value of the i-th sample;

[0027] Based on the sample set, the Bayesian optimization method is used to iteratively search for the optimal solution. The regularization parameter value corresponding to the minimum value of the objective function after satisfying the maximum number of iterations is the optimal regularization parameter value.

[0028] The present invention also provides a soil available potassium measurement system, the system comprising:

[0029] processor;

[0030] a memory having stored thereon a computer program executable on the processor;

[0031] Wherein, when the computer program is executed by the processor, the steps of the soil available potassium measurement method are implemented.

[0032] The present invention also provides a computer-readable storage medium, on which a data processing program is stored, and when the data processing program is executed by a processor, the steps of the soil available potassium measurement method are implemented.

[0033] Beneficial effects of the present invention:

[0034] The present invention proposes a soil available potassium measurement method, system and medium. The present invention reforms the two methods of initial weight selection and weight update strategy. Based on the traditional CARS algorithm, the density-based clustering is used to calculate density and weight to provide an auxiliary tool for data feature screening. The most relevant important features of available potassium in soil are selected through an adaptive model evaluation mechanism, and finally a model with more predictive ability is established. The method aims to optimize the model performance by dynamically adjusting the feature set, thereby improving the model's predictive performance and fitting effect for available potassium in soil. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a method for measuring available potassium in soil according to an embodiment of the present invention;

[0036] Figure 2 is the original spectrum of the embodiment of the present invention;

[0037] Figure 3 is the spectral data after first-order derivative processing according to an embodiment of the present invention;

[0038] Figure 4 is the RMSECV of the original data of the embodiment of the present invention;

[0039] Figure 5 is a sampling variable quantity diagram of the original data of an embodiment of the present invention;

[0040] Figure 6 is the RMSECV of the data after the first-order derivative processing of the embodiment of the present invention;

[0041] Figure 7 It is a graph of the number of data sampling variables after first-order derivative processing according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] Example

[0044] The CARS method screens key features by gradually removing unimportant variables. First, a certain proportion of data is randomly selected from the sample using the Monte Carlo sampling (MCS) method to establish a multivariate calibration model, and the importance of the variables is evaluated based on the absolute value of the regression coefficient. Next, the exponential decreasing function (EDF) is used to force wavelength selection on the variables, and the variables with low absolute values ​​of the coefficients are eliminated. Then, the adaptive reweighted sampling (ARS) method is applied, which includes the roulette wheel selection strategy in the genetic algorithm to randomly select the variables according to their probability and weight. After these steps, the remaining variables are used to construct a new variable subset, and the MCS calculation, forced wavelength selection, and ARS calculation are repeated. Finally, the root mean square error of each variable subset is evaluated by cross-validation (RMSECV), and the variable subset with the lowest root mean square error is selected as the best feature set.

[0045] CARS has the advantages of strong adaptability and ability to process unbalanced data, and shows strong competitive advantages when applied to spectral feature extraction. However, it also faces challenges such as more calculation steps and complex processes, difficulty in parameter setting, and risk of overfitting, especially when processing large amounts of data and low data quality, which is very significant, especially for the measurement of soil available potassium. To this end, the present invention proposes an IWCARS (Initial weight and weight, I&W) algorithm to measure soil available potassium, which is improved by two methods: initial weight selection and weight update strategy. The specific process is as follows: Figure 1 As shown, the following steps are included:

[0046] S1: Extract near infrared spectrum data of the soil to be tested.

[0047] S2: Solve the Euclidean distance between one row and other rows in the spectral matrix of near-infrared spectral data, take the number of distances less than a preset threshold as the number of neighbors of the row, traverse the spectral matrix, and construct an array of the number of neighbors.

[0048] S3: Normalize the density vector according to the neighbor number array and transpose the density vector as the initial weight of the competitive adaptive reweighted sampling method.

[0049] S4: Determine the initial value of the regularization parameter of the Lasso regularization term, take the mean square error between the predicted value and the true value as the optimization target, and determine the optimal regularization parameter through the Bayesian optimization method.

[0050] S5: Fit the Lasso model according to the optimal regularization parameter and the near-infrared spectrum data of the soil to be tested, obtain the Lasso model coefficient matrix, and determine the regression coefficient of each spectral feature.

[0051] S6: A competitive adaptive reweighted sampling method after initial weight assignment is used to screen the spectral features with non-zero regression coefficients in the near-infrared spectral data of the soil to be tested to form an effective feature matrix.

[0052] S7: The PLS model was used to obtain the soil available potassium measurement results based on the effective feature matrix.

[0053] Furthermore, constructing the neighbor number array includes the following steps:

[0054] S2.1: Loop through the spectral matrix X of infrared spectral data (m,n) for each row.

[0055] S2.2: Calculate the Euclidean distance d between the i-th row and each row i :

[0056]

[0057] Where m is the number of samples; n is the number of spectral variables; and x is the element of each row of the spectral matrix.

[0058] S2.3: Statistical Euclidean distance d i The number of distances less than the threshold k in the ith row represents the number of neighbors in the ith row, and the result is stored in the ith element D of the neighbor number array D. i middle.

[0059] S2.4: Based on the neighbor count array D, calculate the normalized density vector D _norm :

[0060] D _norm =D i / ∑D i ;

[0061] The normalized density vector D _norm Transpose and assign the initial weights initial_weights of the competitive adaptive reweighted sampling method.

[0062] Furthermore, obtaining the optimal regularization parameter includes the following steps:

[0063] S4.1: Define the initial value of the regularization parameter lambda = 0.1. The strength of L1 regularization (Lasso regression) is mainly controlled by this parameter. The optimal value will be found through optimization later. Define the maximum number of iterations of Bayesian optimization nIterations = 100.

[0064] S4.2: Calculate the mean square error between the predicted value and the true value as the optimization target to construct the objective function:

[0065]

[0066] Where N is the number of samples, is the predicted value of the i-th sample, y i is the measured value of the i-th sample.

[0067] S4.3: Based on the sample set, the Bayesian optimization method is used to iteratively search for the optimal value. The regularization parameter value corresponding to the minimum value of the objective function after the maximum number of iterations is the optimal regularization parameter value. Preferably, by creating an optimization option structure, no information is displayed when performing the optimization, which is used to improve the calculation efficiency or reduce the interference of the user interface.

[0068] The present invention proposes an improved IWCARS method. The initial weight of the classic CARS is indifferently assigned to 1, that is, the relative importance of each sample is assumed to be consistent. This will require more iterations and computing resources. The IWCARS method attempts to provide the relative importance of each sample in a specific interval through a normalized density vector D_norm. According to the density of the sample in the feature space, the density of the sample reflects its representativeness. Samples that can attract more neighbors indicate that they are more representative in the feature space. By normalizing it, on the whole, this method can effectively emphasize important samples while weakening the influence of unimportant samples on the model, which is conducive to better feature selection and modeling.

[0069] In addition, the present invention takes into account that PLS regression itself is suitable for dealing with multicollinearity problems, and the computational efficiency of Lasso regression is relatively high, especially in large-scale data. The L1 regularization term is introduced into the model to quickly find a sparse solution by penalizing excessive feature weights. Among them, the Bayesian method is used to find the lambda value of the important parameter of Lasso regression. The optimization measures of the L1 regularization term limit the complexity of the model, which not only reduces the risk of overfitting but also further improves the stability of the model in high-dimensional data and improves the computational efficiency.

[0070] In this embodiment, soil samples were collected from a certain place, dried, ground, and sieved for later use as a sample set, and the available potassium content of the soil in the sample set was measured using the national standard method.

[0071] When conducting quantitative analysis on spectral data and sample physical and chemical values, some statistical parameters are needed to evaluate the effect of the quantitative model and its predictive ability for unknown samples. The indicators commonly used to evaluate model performance are: determination coefficient (R 2 ), root mean square error (RMSE), relative analytical error (RPD), mean absolute error (MAE), etc.

[0072] Soil spectral data preprocessing was completed in The Unscrambler X 10.3 software, including first-order derivative (1stDerivative), multivariate scattering correction (MSC), standard normal transformation (SNV), Savitzky-Golay filtering (SG), and baseline removal. Feature extraction and regression algorithms were developed in the PyCharm 2021.1.1 environment. The feature extraction algorithms include the IWCARS and CARA algorithms proposed in this paper, and the regression algorithms include partial least squares (PLS), random forest (RF), gradient boosting machine (GBM), adaptive boosting (AdaBoost) and extreme random tree (ERT) algorithms.

[0073] Near infrared spectrometer was used to collect near infrared spectrum data. The spectrum range is 11520~4000cm -1 , resolution 8cm -1 , the number of scans is 32 times. The soil is placed in a cylindrical quartz cup with a bottom diameter of 4.6 cm and a sample height of 5 mm. The cup is placed in the sample chamber and the lid is closed. The average of five spectra is used to represent the near-infrared spectrum of the sample. A total of 95 near-infrared spectra are obtained, of which 76 spectra are used for calibration set training and 19 spectra are used as validation set data. The original spectrum is shown in Figure 2 shown.

[0074] In this embodiment, the first-order derivative (1st Derivative), multivariate scattering correction (MSC), standard normal transformation (SNV), Savitzky-Golay filtering (SG), and baseline removal method are used to calibrate the original spectrum, and the CARS and IWCARS algorithms are used to screen the characteristic variables of the above five preprocessed data and original data, respectively, to evaluate the effects.

[0075] The accompanying drawings of this embodiment take the original data and the first-order derivative as examples. Figure 3 is the spectral data after first-order derivative processing, Figure 4 and Figure 5 is the RMSECV of the original data and the number of sampled variables. Figure 6 and Figure 7 Plot of RMSECV and number of sampled variables for the first derivative data.

[0076] in, Figure 4 and Figure 6 In the , CARS search is always above IWCARS, i.e., the RMSECV (2.2089, 0.8637) obtained by IWCARS search is smaller than the RMSECV (2.2299, 0.9036) obtained by CARS search; Figure 5 and Figure 7In the results, IWCARS search obtains more variables (57, 65) with fewer iterations (25, 24), while CARS has (26, 25) iterations and (49, 49) variables. For both types of data, IWCARS search obtains smaller RMSECV and fewer iterations. This shows that the IWCARS algorithm can not only improve the model fitting accuracy and reduce the search complexity, but also that this optimization is stable.

[0077] The present invention uses CARS characteristics and IWCARS characteristics to construct a partial least squares regression model (PLS) for available potassium content in soil, and further evaluates the IWCARS effect through the model measurement effect.

[0078] Table 1 Comparison of regression model performance based on CARS and IWCARS characteristic variables

[0079]

[0080]

[0081] Table 1 lists the comparison of the evaluation indicators of the two characteristic variable models (IWCARS-PLS and CARS-PLS) with the same preprocessed data, except for the Rp 2 , Rp of all IWCARS-PLS 2 and Rc 2 Higher than CARS-PLS; RMSECV, RMSEC and RMSEP of IWCARS-PLS are all smaller than the corresponding indicators of CARS-PLS, indicating that the IWCARS method does improve the accuracy of the measurement model compared with the classic CARS method.

[0082] In addition, the RMSEP of CARS-PLS for the six types of data (original data, multivariate scatter correction, standard normal transformation, SG smoothing, baseline removal, and first-order derivative) are all less than RMSEC, indicating overfitting. However, it is obviously different from the CARS variable. The RMSEP and RMSECV values ​​of the PLS model based on the IWCARS variable are not much different from those of the six types of models, and all are higher than RMSEC, indicating that the IWCARS feature effectively avoids the risk of overfitting in the traditional CARS method, and the algorithm is beneficial for overcoming overfitting.

[0083] The present invention proposes a new variable selection method IWCARS based on density clustering, which is combined with the CARS algorithm by calculating density and weight. The method uses an adaptive model evaluation mechanism to select the most relevant features so as to ultimately establish a model with greater predictive power. Studies on the measurement of soil available potassium have shown that IWCARS further narrows the search range of variables compared to the traditional CARS algorithm, making variable selection more effective. The IWCARS method has been successfully applied to near-infrared spectroscopy analysis of available potassium in soil for variable selection. As shown in the results, IWCARS further narrows the search range of variables, making variable selection more effective, which shows that IWCARS can effectively screen important variables related to available potassium in soil, and the selection of feature variables in regression is more reasonable.

[0084] This embodiment also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides a method for measuring available potassium in soil.

[0085] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for measuring soil available potassium, characterized in that: The following steps are involved: Extract near infrared spectrum data of the soil to be tested; Solve the Euclidean distance between one row and other rows in the spectrum matrix of near-infrared spectrum data, take the number of distances less than the preset threshold as the number of neighbors of the row, traverse the spectrum matrix, and construct an array of the number of neighbors; normalize the density vector according to the array of the number of neighbors, and transpose the density vector and assign it as the initial weight of the competitive adaptive reweighted sampling method; Determine the initial value of the regularization parameter of the Lasso regularization term, take the mean square error between the predicted value and the true value as the optimization target, and determine the optimal regularization parameter through the Bayesian optimization method; fit the Lasso model according to the optimal regularization parameter and the near-infrared spectrum data of the soil to be tested, obtain the Lasso model coefficient matrix, and determine the regression coefficient of each spectral feature; The competitive adaptive reweighted sampling method after initial weight assignment was used to screen the spectral features with non-zero regression coefficients in the near-infrared spectral data of the tested soil to form an effective feature matrix. The PLS model was used based on the effective feature matrix to obtain the soil available potassium measurement results.

2. The method for measuring soil available potassium according to claim 1, characterized in that: The step of constructing the neighbor quantity array comprises the following steps: Loop through the spectrum matrix X( m,n ) for each row; Calculate the Euclidean distance d between the i-th row and each row i : Where m is the number of samples; n is the number of spectral variables; x is the element of each row of the spectral matrix; Statistical Euclidean distance d i The number of distances less than the threshold k in the ith row represents the number of neighbors in the ith row, and the result is stored in the ith element D of the neighbor number array D. i middle.

3. The soil available potassium measurement method according to claim 2, characterized in that: The step of assigning the density vector transposition as the initial weight of the competitive adaptive reweighted sampling method comprises the following steps: Based on the neighbor count array D, calculate the normalized density vector D _norm : D _norm =D i / ∑D i ; The normalized density vector D _norm Transpose and assign the initial weights to the competitive adaptive reweighted sampling method.

4. The method for measuring soil available potassium according to claim 1, characterized in that: The acquisition of the optimal regularization parameter comprises the following steps: Define the initial value of the regularization parameter lambda = 0.1; define the maximum number of iterations of Bayesian optimization nIterations = 100; The mean square error between the predicted value and the true value is calculated as the optimization target to construct the objective function: Where N is the number of samples, is the predicted value of the i-th sample, y i is the measured value of the i-th sample; Based on the sample set, the Bayesian optimization method is used to iteratively search for the optimal solution. The regularization parameter value corresponding to the minimum value of the objective function after satisfying the maximum number of iterations is the optimal regularization parameter value.

5. A soil available potassium measurement system, characterized in that: The system comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the soil available potassium measurement method as described in any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, and when the data processing program is executed by a processor, the steps of the soil available potassium measurement method according to any one of claims 1 to 4 are implemented.

Images