Soil moisture prediction platform construction method based on feature screening

Abstract

The invention discloses a method for constructing a soil moisture prediction platform based on feature screening, which belongs to the technical field of quantitative remote sensing and comprises the following steps: acquiring environmental meteorological data, topographic data and soil moisture data; acquiring remote sensing data; matching the preprocessed remote sensing data, environmental meteorological data and topographic data with soil moisture data; establishing an importance analysis algorithm for analyzing the influence of the data on the soil moisture data, and performing feature screening according to the result of the importance analysis algorithm; and constructing a machine learning model, inputting the analysis data after feature screening into the trained machine learning model to obtain a predicted value of the soil moisture, and calculating a decision coefficient R2 and a root mean square error RMSE between the predicted value of the soil moisture and the soil moisture data to perform precision evaluation of the machine learning model. According to the feature screening-based soil moisture prediction platform construction method, the problem that the existing soil moisture can only be measured and is difficult to predict is solved.

Description

Technical Field

[0001] The invention relates to the field of quantitative remote sensing technology, and in particular to a method for constructing a soil moisture prediction platform based on feature screening. Background Art

[0002] Soil moisture is an important part of the global water cycle, affecting precipitation and temperature through the process of evapotranspiration. Its changes are not only related to plant growth and agricultural production, but also to global climate change and extreme climate events such as droughts, floods, and heat waves.

[0003] Soil moisture content can be obtained through a variety of methods. In the past, soil moisture data acquisition mainly relied on manual sampling and field observation. Traditional in-situ observation methods such as weight method, time domain reflectometry method and capacitance method measure at a single point. Although this method is easy to operate and provides the most accurate estimate of soil moisture at different soil depths, it is usually regarded as the "ground truth value", but it consumes a lot of manpower and material resources, has a high time cost, and cannot be used for large-scale surface measurements. Summary of the invention

[0004] In order to overcome the defects of the prior art, the present invention provides a method for constructing a soil moisture prediction platform based on feature screening to solve the above problems.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for constructing a soil moisture prediction platform based on feature screening, comprising the following steps: S1: Obtain environmental meteorological data, topographic data and soil moisture data; S2: Acquire remote sensing data and preprocess the remote sensing data; S3: matching the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data; wherein the pre-processed remote sensing data, environmental meteorological data and terrain data after the matching relationship is established are used as analysis data; S4: Taking soil moisture data as the dominant factor, establish an importance analysis algorithm for the impact of analysis data on soil moisture data, and perform feature screening based on the results of the importance analysis algorithm; S5: Build a machine learning model, randomly divide the analysis data after feature screening into a training set and a validation set, use the training set to train the machine learning model, and then input the analysis data after feature screening into the trained machine learning model to obtain the predicted value of soil moisture, calculate the determination coefficient R^2 and root mean square error RMSE between the predicted value of soil moisture and the soil moisture data to evaluate the accuracy of the machine learning model.

[0006] It is worth noting that in step S1, the environmental meteorological data includes air temperature, relative humidity, wind speed, wind direction, maximum wind speed, solar radiation, soil temperature, rainfall data and digital elevation model data; the digital elevation model data is analyzed and processed by geographic information system software to obtain elevation, slope and slope direction.

[0007] In step S2, remote sensing data of a corresponding date is acquired according to the observation time of the measured soil moisture data.

[0008] In step S2, the step of preprocessing the remote sensing data includes: performing radiation calibration and atmospheric correction on the remote sensing data to obtain GREEN data, RED data and NIR data; and then calculating the normalized vegetation index. ; Calculate the dual-band enhanced vegetation index ; Calculate the normalized water index ; Calculate the dual-band improved soil conditioning vegetation index , where sqrt() represents square root.

[0009] Specifically, in step S3, the matching method is to establish a connection between the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data with the same observation time and observation point.

[0010] It is worth noting that in step S4, the PC causal discovery algorithm is used as the importance analysis algorithm, and a directed acyclic graph is used to represent the causal structure of the analysis data and the soil moisture data. The importance ranking of the analysis data based on the pattern moisture data is obtained according to the causal structure, and then the analysis data is feature screened according to the importance ranking; in the directed acyclic graph, the nodes represent feature variables, and the feature variables include the analysis data and the soil moisture data. The directed edges between the nodes represent the causal relationship between the analysis data and the soil moisture data.

[0011] Preferably, the PC causal discovery algorithm in step S4 is executed including the following steps: S41: Use Fisher Z test to test the conditional independence of each pair of adjacent feature variables. If two of the feature variables are conditionally independent under a given factor K, remove the directed edge between the two feature variables. S42: Finally, the direction of each directed edge is determined by using d-separation to generate a directed acyclic graph; The formula of the Fisher Z test in step S41 is: Z(i,j|k) = Where Z(i,j|k) represents Fisher Z transform; = ; i and j represent the i-th and j-th characteristic variables, and the i-th and j-th are adjacent characteristic variables; k represents the remaining characteristic variable after removing the i-th and j-th characteristic variables; It represents the correlation between the i-th and j-th adjacent feature variables after correcting other feature variables k; The h in represents the variables other than the i-th characteristic variable. represents the variables other than the jth characteristic variable; \h represents excluding the influence of other h characteristic variables, h≤k-2; After Fisher Z transformation, the confidence interval of the correlation coefficient of adjacent characteristic variables was calculated using normal distribution to test the conditional independence of adjacent characteristic variables.

[0012] Optionally, in step S4, the Boruta-Shap algorithm is used as the importance analysis algorithm, and the steps of executing the Boruta-Shap algorithm include: S43: creating new copies of the analysis data, and then shuffling the new copies to remove the correlation between the new copies and the soil moisture to be predicted; wherein the new copies are shadow features in the Boruta-Shap algorithm; S44: running a classifier on the extended data including the original copy and the new copy, and then ranking the extended data based on the importance of the soil moisture data using a feature importance metric, and obtaining an importance score of the extended data based on the soil moisture data; wherein the original copy is the analysis data; S45: creating a threshold using the maximum importance score in the new replica, and assigning hits to the original replicas whose importance scores exceed the threshold; S46: performing a two-sided equality T test on each unassigned original copy to determine the importance relationship between each unassigned original copy and the soil moisture to be predicted, and obtaining the importance score of the unassigned original copy to the soil moisture to be predicted; S47: marking the unassigned original copies whose importance scores obtained by performing a two-sided equality T test are lower than the threshold value created in step S45 as unimportant features for the soil moisture to be predicted, and marking the unassigned original copies whose importance scores obtained by performing a two-sided equality T test are higher than the threshold value created in step S45 as important features for the soil moisture to be predicted; S48: Repeat steps S43 to S47 until each original copy is assigned importance or the Boruta-Shap algorithm reaches a preset operating limit.

[0013] Specifically, in step S4, Pearson correlation analysis is used as the importance analysis algorithm, and the calculation formula of the Pearson correlation coefficient R when performing Pearson correlation analysis is: R = ;in To analyze the data, is the soil moisture data, is the average value of the analyzed data, is the average value of soil moisture data, i represents the i-th sample, and n is the number of data samples; then the correlation coefficient threshold is set, and the data related to soil moisture data are screened in the analysis data according to the correlation coefficient threshold and the Pearson correlation coefficient R.

[0014] Specifically, in step S5, the determination coefficient R^2 = , root mean square error ; is the number of data samples; It is Soil moisture data; It is The predicted value of soil moisture; is the average value of soil moisture data; is the residual sum of squares; is the total sum of squares.

[0015] The beneficial effect of the present invention is that in the method for constructing a soil moisture prediction platform based on feature screening, after selecting an importance analysis algorithm for feature screening, the analysis data after feature screening is used to input into each machine learning model, and finally the accuracy of the machine learning model is evaluated, so as to select a suitable importance analysis algorithm and machine learning model to build a soil moisture prediction platform, and combine the importance analysis algorithm with the machine learning model to achieve high-precision prediction of soil moisture. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for constructing a soil moisture prediction platform based on feature screening in one embodiment of the present invention; Figure 2 Schematic diagram of three relationships of d-separation in one embodiment of the present invention; Figure 3 is a schematic diagram of a directed acyclic graph in one embodiment of the present invention; Figure 4 A schematic diagram of feature importance scores using the Boruta-Shap algorithm in one embodiment of the present invention; Figure 5 A Pearson correlation analysis heat map in one embodiment of the present invention; Figure 6 is a schematic diagram of the results of three importance analysis algorithms in one embodiment of the present invention; Figure 7 A schematic diagram of the results of R^2 and RMSE of soil moisture predicted by the AdaBoost model on the training set and the prediction set in one embodiment of the present invention; Figure 8 A schematic diagram of the R^2 and RMSE results of soil moisture prediction by XGBoost model on the training set and the prediction set in one embodiment of the present invention; Fig. 9 A schematic diagram of the results of R^2 and RMSE of soil moisture predicted by a random forest model on a training set and a prediction set in one embodiment of the present invention; Fig.10 A schematic diagram of the R^2 and RMSE results of the Stacking model predicting soil moisture on the training set and the prediction set in one embodiment of the present invention; Fig.11 A schematic diagram of the results of R^2 and RMSE of soil moisture predicted by a multivariate linear regression model on a training set and a prediction set in one embodiment of the present invention; Fig.12 A schematic diagram of comparison of prediction results of a validation set in one embodiment of the present invention; Fig.13 Schematic diagram of comparison between measured soil moisture value and predicted value in one embodiment of the present invention. DETAILED DESCRIPTION

[0017] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention. In particular, this solution is aimed at the soil moisture of tall grass prairie. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0018] like Figure 1-13 As shown, a method for constructing a soil moisture prediction platform based on feature screening includes the following steps: S1: Acquire environmental meteorological data, topographic data and soil moisture data; wherein the environmental meteorological data include air temperature, relative humidity, wind speed, wind direction, maximum wind speed, solar radiation, soil temperature, rainfall data and digital elevation model data; in step S1, the digital elevation model data is analyzed and processed by the geographic information system software ArcGIS 10.8.1 to obtain elevation, slope and slope direction; at the same time, the measured soil moisture data is downloaded for subsequent model accuracy verification analysis.

[0019] S2: Acquire remote sensing data and preprocess the remote sensing data; in step S2, the remote sensing data of the corresponding date is acquired according to the observation time of the measured soil moisture data. In this embodiment, the remote sensing data used is Landsat-8 optical satellite image data, specifically, Landsat-8 optical satellite image data is downloaded from the USGS website, and Landsat-8 optical satellite image data with less than 30% cloud cover is screened out as remote sensing data. In step S2, the step of preprocessing the remote sensing data includes: performing radiation calibration and atmospheric correction on the remote sensing data to obtain GREEN data, RED data and NIR data, and the preprocessing process is completed on the ENVI5.6 software; then the normalized vegetation index is calculated ; Calculate the dual-band enhanced vegetation index ; Calculate the normalized water index ; Calculate the dual-band improved soil conditioning vegetation index , where sqrt() represents square root.

[0020] In this scheme, the Konza tallgrass prairie with an area of ​​34.87 km2 in the Flynn Mountains of Manhattan, Kansas, USA is taken as an example. The environmental meteorological data, topographic data and soil moisture data of this example are provided by the Konza Prairie Biological Station. The main data include: (1) daily recorded environmental meteorological data, including air temperature (℃), relative air humidity (%), wind speed (m / s), wind direction (angle), maximum wind speed (m / s, recorded every ten seconds), solar radiation (joule / m^2), soil temperature (℃), the most recent rainfall (mm) and the time interval from the last rainfall (days), collected by the meteorological micro-recorder at the headquarters of the Konza Prairie Biological Station. (2) Topographic data is a 2-meter resolution digital elevation model (DEM) generated by LiDAR DEM data collected according to USGS standard specifications. ArcGIS 10.8.1 is used for data analysis to obtain elevation, slope and slope aspect. (3) Soil moisture data is daily soil moisture data (m^3 / m^3) at a depth of 30 cm, measured by the neutron probe method. (4) The remote sensing data are Landsat-8 optical satellite image data downloaded from the USGS website. The radiation calibration and atmospheric correction were completed on ENVI5.6 to obtain GREEN, RED and NIR data. The normalized difference vegetation index (NDVI), dual-band enhanced vegetation index (EVI2), normalized difference water index (NDWI) and dual-band modified soil adjusted vegetation index (MSAVI2) were calculated respectively. The remote sensing data, environmental meteorological data and terrain data of the same date after preprocessing are matched with the soil moisture data to establish a database. The final sorted air temperature (T_air), relative humidity (RHUM), wind speed (WSPEED), wind direction (WDIR), maximum wind speed (WMAX), solar radiation (SRAD), soil temperature (STEMP), the most recent rainfall (rain_gauges(mm)), the time interval from the last rainfall (rain_time(day)), elevation (elevation), slope (Slope), aspect (Aspect), GREEN, RED, NIR, NDVI, EVI2, NDWI and MSAVI2 data are used as input data for feature screening. The target variable is the daily soil moisture data at a depth of 30 cm (D30). The database has a total of 185 samples.

[0021] S3: Match the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data to establish a database; the pre-processed remote sensing data, environmental meteorological data and terrain data after the matching relationship is established are used as analysis data; the matching method is to establish a connection between the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data with the same observation time and observation point. In this embodiment, the matching of observation time means: the observation time of the whole set of data is a time interval, such as the data from May to October 2013, so there will be multiple measured environmental meteorological data and soil moisture data on different dates, so the time of soil moisture data is mainly used to find the pre-processed remote sensing data, environmental meteorological data and terrain data of the same time; the matching of observation points means: since each sample in the database needs to be measured at the same observation point, it is necessary to match the position.

[0022] S4: Taking soil moisture data as the dominant factor, establishing an importance analysis algorithm for the impact of analysis data on soil moisture data, and performing feature screening based on the results of the importance analysis algorithm; in step S4, the importance analysis algorithm is one of the PC causal discovery algorithm, the Boruta-Shap algorithm, or the Pearson correlation analysis; In the step S4, the PC causal discovery algorithm is executed to perform importance analysis, and a directed acyclic graph is used to represent the causal structure of the analysis data and the soil moisture data. The importance ranking of the analysis data based on the pattern moisture data is obtained according to the causal structure, and then the analysis data is feature screened according to the importance ranking; in the directed acyclic graph, the nodes represent feature variables, and the feature variables include the analysis data and the soil moisture data. The directed edges between the nodes represent the causal relationship between the analysis data and the soil moisture data. The more directed edges between the nodes, the higher the importance. When performing feature screening, the screening number a can be set, and then the top a analysis data with the highest importance are selected according to the importance ranking; The PC causal discovery algorithm mainly uses conditional independence to determine the dependency between factors, and its effectiveness depends on the following three conditions: causal sufficiency assumption: there are no unobserved confounding factors in any factor in the graph; causal Markov assumption: given a factor, the causality of the factor is conditionally independent; faithful assumption: two conditionally independent factors can be separated by another factor; since the causal Markov assumption shows that a factor is conditionally independent from all non-descendant factors given its direct cause, we assume that the obtained soil moisture data is sufficient to meet the causal sufficiency assumption; the PC causal discovery algorithm in step S4 includes the following steps: first start from a fully connected undirected graph, which contains all variables; S41: Use Fisher Z test to test the conditional independence of each pair of adjacent feature variables. If two of the feature variables are conditionally independent under a given factor K, remove the directed edge between the two feature variables. S42: Finally, the DAG is generated by using d-separation to determine the direction of each directed edge. For any three nodes in an undirected graph with dependencies, the following conditions must be satisfied: Figure 2 One of the three dependencies shown.

[0023] Specifically, the formula of the Fisher Z test in step S41 is: Z(i,j|k) = Where Z(i,j|k) represents Fisher Z transform, Converted to Z value, which approximately follows normal distribution; = ; i and j represent the i-th and j-th characteristic variables, and the i-th and j-th are adjacent characteristic variables; k represents the remaining characteristic variable after removing the i-th and j-th characteristic variables; It represents the correlation between the i-th and j-th adjacent feature variables after correcting other feature variables k. Also known as the partial correlation coefficient, the correction can be understood as assuming that the other characteristic variables k are all taken as the mean. When the i-th and j-th characteristic variables are conditionally independent, =0; The h in represents the variables other than the i-th characteristic variable, and represents the set of other variables other than the i-th characteristic variable. It represents the variables other than the j-th characteristic variable, and represents the set of other variables other than the j-th characteristic variable. When performing partial correlation analysis, we usually focus on the correlation between two characteristic variables, while controlling the influence of one or more other characteristic variables. h can contain one or more characteristic variables, which are excluded from the analysis to reduce their influence on the relationship between the main variables; \h means excluding the influence of other h characteristic variables, h≤k-2; after Fisher Z transformation, the confidence interval of the correlation coefficient of adjacent characteristic variables is calculated using normal distribution to test the conditional independence of adjacent characteristic variables, where in this scheme, it is a 95% confidence interval.

[0024] In step S4, the steps of executing the Boruta-Shap algorithm are as follows: S43: Create new copies of the analysis data and name them shadow + feature_name, and then shuffle these new copies to remove the correlation between these new copies and the soil moisture to be predicted; the new copies are shadow features in the Boruta-Shap algorithm; S44: running a classifier on the extended data including the original copy and the new copy, and then using a feature importance metric to rank the extended data based on its importance to the soil moisture data, and obtaining an importance score of the extended data based on the soil moisture data, wherein the classifier uses permutation importance as a metric for classifier selection; wherein the original copy is the analysis data; S45: creating a threshold using the maximum importance score in the new replica, and assigning hits to the original replicas whose importance scores exceed the threshold; that is, treating the original replicas whose importance scores exceed the threshold as hit items; S46: performing a two-sided equality T test on each unassigned original copy to determine the importance relationship between each unassigned original copy and the soil moisture to be predicted, and obtaining the importance score of the unassigned original copy to the soil moisture to be predicted; S47: The unassigned original copies whose importance scores obtained by performing a bilateral equality T test are lower than the threshold value created in the step S45 are marked as unimportant features for the soil moisture to be predicted (attributes with importance scores lower than the threshold value are considered unimportant), and the unassigned original copies whose importance scores obtained by performing a bilateral equality T test are higher than the threshold value created in the step S45 are marked as important features for the soil moisture to be predicted (attributes with importance scores higher than the threshold value are considered important); if the result of the bilateral equality T test shows that there is a significant importance relationship between the unassigned original copy and the soil moisture to be predicted, then this unassigned original copy is considered important; if the result of the bilateral equality T test shows that there is no significant importance relationship between the unassigned original copy and the soil moisture to be predicted, then this unassigned original copy is considered unimportant; if there are still unassigned original copies in the extended data that are equal to the threshold value after the number of runs initially set is reached, these original copies are identified as tentatively, that is, pending features, and therefore it is necessary to assign importance to all original copies by subsequently repeating steps S43 to S47.

[0025] S48: Repeat steps S43 to S47 until each original copy is assigned importance or the Boruta-Shap algorithm reaches a preset operating limit; then filter out the original copies marked as important features for the soil moisture to be predicted.

[0026] By combining the Boruta-Shap algorithm with the cooperative game theory of Shap analysis, the Boruta-Shap algorithm can automatically screen features that contribute to soil moisture data prediction. The Boruta-Shap algorithm is based on the core idea of ​​the Boruta algorithm: by creating shadow features, which are randomized versions of new copies, and by comparing the importance of shadow features with shadow features, the Boruta algorithm can identify features that are truly helpful for prediction tasks.

[0027] In step S4, the calculation formula of the Pearson correlation coefficient R when performing the Pearson correlation analysis is: R = ;in To analyze the data, is the soil moisture data, is the average value of the analyzed data, is the average value of soil moisture data, i represents the i-th sample, and n is the number of data samples; then the correlation coefficient threshold is set, and the data related to soil moisture data are screened in the analysis data according to the correlation coefficient threshold and the Pearson correlation coefficient R. Pearson correlation analysis is a statistical method used to evaluate the linear relationship between two continuous variables; it quantifies this relationship by calculating the Pearson correlation coefficient R, which is a value between -1 and 1, where 1 represents a perfect positive correlation, -1 represents a perfect negative correlation, and 0 represents no linear correlation. After setting the correlation coefficient threshold, the analysis data corresponding to the Pearson correlation coefficient R greater than the correlation coefficient threshold can be screened out as data related to soil moisture data.

[0028] In the step S4, the number of features screened by the PC causal discovery algorithm for the soil moisture data is used as the threshold for the number of features screened by the Boruta-Shap algorithm and the Pearson correlation analysis, so as to facilitate the subsequent comparison of the screening capabilities of the PC causal discovery algorithm, the Boruta-Shap algorithm and the Pearson correlation analysis. Preferably, the Boruta-Shap algorithm and the Pearson correlation analysis method select the importance or correlation ranking based on the number threshold to screen the features, and can add one or two additional features as a supplementary scheme, and compare the accuracy with the causal discovery feature screening scheme; for example, after the PC causal discovery algorithm obtains two feature numbers, two are used as the number threshold, and in the subsequent Boruta-Shap algorithm and the Pearson correlation analysis method, at least two features need to be screened, and one or two additional features are selected as supplementary schemes for comparison. The features with obvious causal orientation obtained by the PC causal discovery algorithm and the soil moisture data are used as the input features of the causal discovery feature screening scheme and input into the machine learning algorithm to predict soil moisture. In this embodiment, the features with obvious causal orientation refer to the features with obvious arrows pointing to the target variable in the directed acyclic graph (DAG).

[0029] This example first uses the PC causal discovery algorithm for feature screening. The algorithm is based on the Python language casual-learn platform experiment. The results are as follows Figure 3 and 6 As shown ( Figure 6 √ represents the selected features), soil temperature (STEMP) and slope aspect (Aspect) have a significant causal relationship with soil moisture data, which reveals the direct impact of soil temperature and slope aspect on soil moisture status, which means that when predicting soil moisture, changes in these two factors can directly predict changes in soil moisture. Therefore, soil temperature and slope aspect are selected and input into the machine learning model as the feature screening scheme of the PC causal discovery algorithm, as well as the minimum threshold for the Boruta-Shap algorithm and Pearson correlation analysis to screen features. Then, the Boruta-Shap algorithm and Pearson correlation analysis are used for feature screening respectively. The Boruta-Shap algorithm used in this example is combined with the XGBoost algorithm, and the Shap value is used to measure the contribution of features in the tree model, and then the importance of the features is ranked. The algorithm is implemented based on the Boruta-Shap library in the Python language. Through multiple experimental comparisons, the number of iterations (n_trials) needs to be set to 2000 times before the importance of each feature can be fully assigned. The feature screening results of the Boruta-Shap algorithm are as follows Figure 4 and 6As shown in the figure, the Shap value ranking is Aspect, elevation, Slope, soil temperature (STEMP), relative air humidity (RHUM), dual-band modified soil conditioned vegetation index (MSAVI2), RED, NIR, the most recent rainfall (rain_gauges(mm)), GREEN, normalized vegetation index (NDVI), dual-band enhanced vegetation index (EVI2), and time interval from the last rainfall (rain_time(day)). These features are the features that contribute to soil moisture prediction selected by the Boruta-Shap algorithm. At the same time, the Boruta-Shap algorithm screens out wind speed (WSPEED), air temperature (T_air), normalized water index (NDWI), solar radiation (SRAD), wind direction (WDIR), and maximum wind speed (WMAX). This example selects the top two features, aspect and elevation, as the feature screening scheme of the Boruta-Shap algorithm and inputs them into the machine learning model. In addition, we also additionally screen slope and soil temperature as comparison schemes. Therefore, the final solution for feature selection of Boruta-Shap algorithm is: 2 features selected by Boruta-Shap (slope aspect and elevation), 3 features selected by Boruta-Shap (slope aspect, elevation and slope), 4 features selected by Boruta-Shap (slope aspect, elevation, slope and soil temperature) and effective features selected by Boruta-Shap. Pearson correlation analysis is then used for feature selection. The results are as follows: Figure 5 and 6 As shown. In this example, we selected the two features with the greatest correlation with soil moisture, aspect (Aspect, R = -0.41) and RED (R = -0.33) as the Pearson correlation analysis feature screening scheme and input them into the machine learning model. At the same time, the third and fourth features of correlation, elevation (R = -0.32) and dual-band enhanced vegetation index (EVI2, R = 0.32), were selected as additional comparison schemes. The final scheme for Pearson correlation analysis feature screening is: 2 features selected by Pearson (aspect and RED) and 4 features selected by Pearson (aspect, RED, elevation and EVI2).

[0030] S5: Construct a machine learning model, randomly divide the feature-screened analysis data into a training set and a validation set at a ratio of 7:3, use the training set to train the machine learning model, then input the feature-screened analysis data into the trained machine learning model to obtain the predicted value of soil moisture, calculate the determination coefficient R^2 and root mean square error RMSE between the predicted value of soil moisture and the soil moisture data to evaluate the accuracy of the machine learning model. In step S5, the determination coefficient R^2 = , root mean square error ; is the number of data samples; It is Soil moisture data; It is The predicted value of soil moisture; is the average value of soil moisture data; is the residual sum of squares; is the total sum of squares. The coefficient of determination R^2 and the root mean square error RMSE are both commonly used indicators for evaluating the performance of regression models, but they provide information about the goodness of fit of the model from different perspectives. The coefficient of determination R^2 measures the proportion of the variability explained by the model to the total variability. The value ranges from 0 to 1. The higher the value, the better the model fits the data. The root mean square error RMSE measures the standard measure of the difference between the model's predicted values ​​and the actual observed values. Its value is always non-negative. The closer the value is to 0, the smaller the model's prediction error is and the better the model's performance is.

[0031] In this embodiment, the machine learning models are AdaBoost model, XGBoost model, random forest model, multiple linear regression model and Stacking model. For AdaBoost model, XGBoost model, random forest model and multiple linear regression model, the training set is directly input for training; for Stacking model, the top two models with the highest accuracy among AdaBoost model, XGBoost model, random forest model and multiple linear regression model are selected as base models, and then multiple linear regression model is selected as meta-model, and finally the model is trained on the training set by five-fold cross validation. In this embodiment, the XGBoost model and the random forest model are selected as the Stacking model as the base model, and a five-fold cross validation is used on the training set. The prediction results of the five tests form a data set A with the same size as the training set. During the cross validation process, the XGBoost model and the random forest model need to predict the entire test set. The final prediction result is the average value B of the prediction results of the five learners. The XGBoost model and the random forest model are both subjected to five-fold cross validation to obtain the training set Ai and the test set Bi (i<= 2); then the two features obtained by the base model XGBoost and the random forest are input into the meta-model multivariate linear regression model, and the trained results are used as the Stacking results.

[0032] Before building a machine learning model, hyperparameter tuning is performed to control the behavior and effect of the training algorithm. In this embodiment, the optimal hyperparameters of the model are automatically adjusted based on a five-fold cross validation and PSO particle swarm optimization method. Five-fold cross validation is a technique for evaluating the performance of a machine learning model. It divides the data set into five equal-sized subsets, using one of the subsets as a test set each time and the remaining four as training sets. This process is repeated five times, each time a different subset is selected as a test set, and the performance of the final model is obtained by the average of the five test results. This method helps to provide more reliable performance estimates and better understand the generalization ability of the model. The PSO particle swarm optimization algorithm is a group-based optimization technique that finds the optimal solution by simulating the social behavior of a flock of birds or a school of fish. Each particle in the PSO particle swarm optimization algorithm is a solution to the problem of optimization, and all particles are searched in a D-dimensional space. Each particle must be given a memory function to remember the best position found. Each particle also has a speed to determine the distance and direction of flight, which is dynamically adjusted according to its own flight experience and the flight experience of its companions. The best position experienced by each particle during its movement is the optimal solution found by the particle itself, called the individual extremum, usually represented by pBest. Through information sharing and cooperation among particles in the entire team, the best position experienced is the optimal solution currently found by the entire team, called the global extremum, usually represented by gBest. Particles update their speed and position according to the following formula: + ; + ;in represents the d-th dimension component of the flight velocity vector i of the particle in the k-th iteration; represents the d-th dimension component of the particle position vector i at the k-th iteration; and Represents the acceleration constant, which is used to adjust the maximum step size of learning. = 1.49618, =1.49618; and Represents two random functions, and the value range is usually [0,1] to increase the randomness of the search; is the inertia weight, a non-negative number, which adjusts the search range of the solution space, ω = 0.7298; the population size in this scheme is s = 100. Hyperparameter tuning can significantly improve the performance and generalization ability of the model, avoid overfitting, accelerate the convergence of the model training process, optimize the use of computing resources, ensure the stability and adaptability of the model on different data sets, and improve the quality of decision-making based on model prediction.

[0033] The machine learning prediction results under each machine learning model are as follows: The AdaBoost model results are as follows Figure 7 As shown in the figure, the PC causal discovery algorithm shows good performance both in the training set and the validation set, which indicates that the AdaBoost model is effective in feature screening. Compared with other feature screening methods, the PC causal discovery algorithm performs best on the validation set (R^2 = 0.7298, RMSE = 0.059), and there is no overfitting phenomenon. The second is the four features selected by the Botura-Shap algorithm (R^2 = 0.7169, RMSE = 0.061), but there is overfitting phenomenon. When the invalid features are greater than or equal to four or more, the AdaBoost model will overfit, and the prediction results of the effective features selected by the Botura-Shap algorithm have the most serious overfitting phenomenon. The Pearson correlation analysis shows that the overfitting phenomenon is stronger than that of the four features selected by the PC causal discovery algorithm and the Botura-Shap algorithm. Comprehensively evaluating the accuracy and overfitting effect, the PC causal discovery algorithm has the best prediction effect in the AdaBoost model.

[0034] The XGBoost model results are as follows Figure 8 As shown in the figure, the PC causal discovery algorithm performed best on the validation set (R^2 = 0.8944, RMSE = 0.037), but overfitting occurred. The next best result was the 4 features selected by the Botura-Shap algorithm (R^2 = 0.8853, RMSE = 0.039), which also showed overfitting. Although the prediction results of the 2 features and 3 features selected by the Boruta-Shap algorithm did not show overfitting, their accuracy was far lower than that of the PC causal discovery algorithm and the 4 features and effective features selected by the Botura-Shap algorithm. The two schemes for screening features by Pearson correlation analysis had the most serious overfitting effect compared with other schemes. Based on the comprehensive accuracy evaluation and overfitting effect, the PC causal discovery algorithm had the best prediction effect in the XGBoost model.

[0035] The results of the random forest model are as follows Fig. 9As shown in the figure, the PC causal discovery algorithm performed best on the validation set (R^2 = 0.7506, RMSE = 0.057), with overfitting, but the overfitting phenomenon was more serious than that of other algorithms. The next best was the effective features selected by the Botura-Shap algorithm (R^2 = 0.7394, RMSE = 0.058), which also showed overfitting. However, the prediction results of the 2 and 3 features selected by the Boruta-Shap algorithm did not show overfitting. Comprehensively evaluating the accuracy and overfitting effect, the effective features selected by the Botura-Shap algorithm had the best prediction effect in the random forest model.

[0036] The results of the multiple linear regression model are as follows Fig.10 As shown in the figure, the effective features selected by the Botura-Shap algorithm performed best on the validation set (R^2 = 0.6107, RMSE = 0.071), without overfitting, followed by the four features selected by the Botura-Shap algorithm (R^2 = 0.6000, RMSE = 0.072), which also did not show overfitting. Compared with the machine learning algorithm, the prediction effect of the PC causal discovery algorithm combined with the multivariate linear regression model is far inferior to that of the Botura-Shap algorithm and Pearson correlation analysis. This may be because the multivariate linear regression model cannot learn the complex nonlinear causal relationship between soil moisture, soil temperature and slope aspect, and the machine learning algorithm has obvious advantages in dealing with such complex nonlinear relationships. Moreover, although the multivariate linear regression model performed best among the effective feature schemes selected by the Botura-Shap algorithm, its prediction accuracy was far inferior to that of the machine learning model. Comprehensively considering the accuracy evaluation and overfitting effect, the effective features selected by the Botura-Shap algorithm have the best prediction effect in the multivariate linear regression model.

[0037] The results of the stacking model are as follows Fig.11 As shown in the figure, the PC causal discovery algorithm performed best on the validation set (R^2 = 0.8818, RMSE = 0.039), but overfitting occurred. The next best result was the four features selected by the Botura-Shap algorithm (R^2 = 0.8372, RMSE = 0.046), which also showed overfitting. Similarly, the prediction results of the two and three features selected by the Boruta-Shap algorithm did not show overfitting, and the accuracy on the validation set was close to the prediction results of the features screened by the Pearson correlation analysis, but the accuracy was far less than that of the PC causal discovery algorithm and the four features and effective features selected by the Botura-Shap algorithm. Based on the comprehensive accuracy evaluation and overfitting effect, the PC causal discovery algorithm has the best prediction effect in the Stacking model.

[0038] like Fig.12 As shown in the figure, a horizontal comparison of all prediction models in the validation set is listed. The XGBoost model performs best among all feature selection methods, especially when using the effective features selected by the PC causal discovery algorithm and Boruta-Shap, its R^2 value is the highest, close to 0.9, and the RMSE value is the lowest, indicating that the model has good predictive and generalization capabilities. The Stacking prediction model follows closely and performs quite well, especially when using the PC causal discovery algorithm, its R^2 value is high and the RMSE value is low. The performance of the random forest and AdaBoost models is moderate. The multivariate linear regression model performs the worst, especially when using the PC causal discovery algorithm, its R^2 value and RMSE value both show that the model has weak predictive ability. The feature subset selected by the PC causal discovery algorithm has the highest prediction accuracy among all machine learning algorithms, which can significantly improve the performance of the model. The feature selection of the Boruta-Shap algorithm can provide a better feature subset in most cases, which helps to improve the performance of the model. However, the features selected by the Pearson method do not bring performance improvements in most cases, and even in some cases, such as the multivariate linear regression model, the performance is not as good as other methods.

[0039] from Fig.13 The results in Figure 2 show that the PC causal discovery algorithm combined with the machine learning model can effectively improve the performance of the model. The PC causal discovery algorithm can discover the causal relationship between features and variables, and help remove features that have no direct causal relationship with the target variable; and the machine learning model has obvious advantages in dealing with complex and nonlinear problems. Therefore, the PC causal discovery algorithm combined with the machine learning model to predict soil moisture can effectively utilize limited data resources, focus on those factors that have a direct impact on soil moisture, further improve the accuracy of prediction, and help decision makers better understand and predict soil moisture changes, so as to make more reasonable resource allocation and management decisions. The Boruta-Shap algorithm can provide a better feature subset in most cases, but it will filter out redundant features, increase feature dimensions and noise, and thus lead to model overfitting. Pearson correlation analysis is based on the features selected by correlation ranking, and there may be a "false" high correlation between the features; in addition, Pearson correlation analysis is easily affected by outliers, which may distort the relationship between variables and affect the accuracy of prediction.

[0040] Compared with other technologies, the present invention combines the PC causal discovery algorithm with a machine learning model to achieve high-precision prediction of soil moisture. The present invention combines the characteristics of machine learning models that can capture and learn complex patterns and nonlinear relationships in data, and uses the PC causal discovery algorithm to explore the causal relationship between features and variables, helping to remove features that have no direct causal relationship with the target variable, eliminating the interference of "false" associations with some features that have high correlation with the variables, thereby reducing the model complexity and overfitting problems and enhancing its generalization ability, thereby improving the accuracy and interpretability of the model in predicting soil moisture.

[0041] The present invention compares and analyzes the effects of PC causal discovery algorithm, Boruta-SHAP algorithm and Pearson correlation analysis combined with machine learning model in soil moisture prediction, and finds that PC causal discovery algorithm has obvious advantages in feature screening, and this discovery is based on experimental results. Although Boruta-Shap algorithm and Pearson correlation analysis have certain application value in feature selection, they have limitations in dealing with complex soil moisture prediction problems. Although Boruta-Shap algorithm can identify important features, it is mainly based on the influence of features on model output, and may not fully reveal the causal relationship between features and target variables, resulting in the inability to exclude those features that are only indirectly related or have no causal relationship with the prediction target in some cases. Although Pearson correlation analysis can identify linear relationships between variables, it has limited recognition ability for nonlinear relationships and potential causal structures, and may not be able to fully capture the complex interactions between soil moisture and environmental meteorological characteristics and terrain characteristics. In contrast, PC causal discovery algorithm effectively removes features that are irrelevant or weakly related to soil moisture prediction by identifying and utilizing causal relationships between variables, thereby simplifying the model and reducing noise interference. This method not only improves the prediction accuracy of the model, but also enhances the generalization ability of the model, so that it can maintain high prediction performance under different conditions. In addition, the application of the PC causal discovery algorithm also improves the interpretability of the model, making the prediction results easier to understand and trust. Through this method of combining feature screening and machine learning, the present invention has achieved a technological breakthrough in the field of soil moisture prediction and provided a new and efficient tool for precision agriculture and water resources management.

[0042] Finally, the best solution (PC causal algorithm combined with XGBoost algorithm) was selected to predict the soil moisture in the database and compared with the measured values. The results are as follows Fig.12 As shown, the feature subset screened by the PC causal algorithm can effectively improve the model performance and the prediction accuracy of soil moisture.

[0043] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for constructing a soil moisture prediction platform based on feature screening, characterized in that: The following steps are involved: S1: Obtain environmental meteorological data, topographic data and soil moisture data; S2: Acquire remote sensing data and preprocess the remote sensing data; S3: matching the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data; wherein the pre-processed remote sensing data, environmental meteorological data and terrain data after the matching relationship is established are used as analysis data; S4: Taking soil moisture data as the dominant factor, establish an importance analysis algorithm for the impact of analysis data on soil moisture data, and perform feature screening based on the results of the importance analysis algorithm; S5: Build a machine learning model, randomly divide the analysis data after feature screening into a training set and a validation set, use the training set to train the machine learning model, and then input the analysis data after feature screening into the trained machine learning model to obtain the predicted value of soil moisture, calculate the determination coefficient R^2 and root mean square error RMSE between the predicted value of soil moisture and the soil moisture data to evaluate the accuracy of the machine learning model.

2. The method for constructing a soil moisture prediction platform based on feature screening according to claim 1, characterized in that: In step S1, the environmental meteorological data includes air temperature, relative humidity, wind speed, wind direction, maximum wind speed, solar radiation, soil temperature, rainfall data and digital elevation model data; the digital elevation model data is analyzed and processed by geographic information system software to obtain elevation, slope and slope direction.

3. The method for constructing a soil moisture prediction platform based on feature screening according to claim 2, characterized in that: In step S2, remote sensing data of a corresponding date is acquired according to the observation time of the measured soil moisture data.

4. The method for constructing a soil moisture prediction platform based on feature screening according to claim 3 is characterized in that: In step S2, the step of preprocessing the remote sensing data includes: performing radiation calibration and atmospheric correction on the remote sensing data to obtain GREEN data, RED data and NIR data; and then calculating the normalized vegetation index. ; Calculate the dual-band enhanced vegetation index ; Calculate the normalized water index ; Calculate the dual-band improved soil conditioning vegetation index , where sqrt() represents square root.

5. The method for constructing a soil moisture prediction platform based on feature screening according to claim 4, characterized in that: In step S3, the matching method is to establish a connection between the pre-processed remote sensing data, environmental meteorological data, terrain data and soil moisture data with the same observation time and observation point.

6. The method for constructing a soil moisture prediction platform based on feature screening according to claim 5, characterized in that: In step S4, the PC causal discovery algorithm is used as the importance analysis algorithm, and a directed acyclic graph is used to represent the causal structure of the analysis data and the soil moisture data. The importance ranking of the analysis data based on the pattern moisture data is obtained according to the causal structure, and then the analysis data is feature screened according to the importance ranking; in the directed acyclic graph, the nodes represent feature variables, and the feature variables include the analysis data and the soil moisture data. The directed edges between the nodes represent the causal relationship between the analysis data and the soil moisture data.

7. The method for constructing a soil moisture prediction platform based on feature screening according to claim 6, characterized in that: The PC causal discovery algorithm in step S4 is executed, comprising the following steps: S41: Use Fisher Z test to test the conditional independence of each pair of adjacent feature variables. If two of the feature variables are conditionally independent under a given factor K, remove the directed edge between the two feature variables. S42: Finally, the direction of each directed edge is determined by using d-separation to generate a directed acyclic graph; The formula of the Fisher Z test in step S41 is: Z(i,j|k) = Where Z(i,j|k) represents Fisher Z transform; = ; i and j represent the i-th and j-th characteristic variables, and the i-th and j-th are adjacent characteristic variables; k represents the remaining characteristic variable after removing the i-th and j-th characteristic variables; It represents the correlation between the i-th and j-th adjacent feature variables after correcting other feature variables k; The h in represents the variables other than the i-th characteristic variable. represents the variables other than the jth characteristic variable; \h represents excluding the influence of other h characteristic variables, h≤k-2; After Fisher Z transformation, the confidence interval of the correlation coefficient of adjacent characteristic variables was calculated using normal distribution to test the conditional independence of adjacent characteristic variables.

8. The method for constructing a soil moisture prediction platform based on feature screening according to claim 7, characterized in that: In step S4, the Boruta-Shap algorithm is used as the importance analysis algorithm, and the steps of executing the Boruta-Shap algorithm include: S43: creating new copies of the analysis data, and then shuffling the new copies to remove the correlation between the new copies and the soil moisture to be predicted; wherein the new copies are shadow features in the Boruta-Shap algorithm; S44: running a classifier on the extended data including the original copy and the new copy, and then ranking the extended data based on the importance of the soil moisture data using a feature importance metric, and obtaining an importance score of the extended data based on the soil moisture data; wherein the original copy is the analysis data; S45: creating a threshold using the maximum importance score in the new replica, and assigning hits to the original replicas whose importance scores exceed the threshold; S46: performing a two-sided equality T test on each unassigned original copy to determine the importance relationship between each unassigned original copy and the soil moisture to be predicted, and obtaining the importance score of the unassigned original copy to the soil moisture to be predicted; S47: marking the unassigned original copies whose importance scores obtained by performing a two-sided equality T test are lower than the threshold value created in step S45 as unimportant features for the soil moisture to be predicted, and marking the unassigned original copies whose importance scores obtained by performing a two-sided equality T test are higher than the threshold value created in step S45 as important features for the soil moisture to be predicted; S48: Repeat steps S43 to S47 until each original copy is assigned importance or the Boruta-Shap algorithm reaches a preset operating limit.

9. The method for constructing a soil moisture prediction platform based on feature screening according to claim 8, characterized in that: In step S4, Pearson correlation analysis is used as the importance analysis algorithm, and the calculation formula of the Pearson correlation coefficient R when performing Pearson correlation analysis is: R = ;in To analyze the data, is the soil moisture data, is the average value of the analyzed data, is the average value of soil moisture data, i represents the i-th sample, and n is the number of data samples; then the correlation coefficient threshold is set, and the data related to soil moisture data are screened in the analysis data according to the correlation coefficient threshold and the Pearson correlation coefficient R.

10. The method for constructing a soil moisture prediction platform based on feature screening according to claim 9, characterized in that: In step S5, the determination coefficient R^2 = , root mean square error ; is the number of data samples; It is Soil moisture data; It is The predicted value of soil moisture; is the average value of soil moisture data; is the residual sum of squares; is the total sum of squares.

Images