A method and system for environmental data analysis based on big data

Abstract

The present invention discloses an environmental data analysis method and system based on big data, including: collecting environmental data and its environmental basic data and geographical location; obtaining first data according to the correlation of the environmental basic data to the environmental data, dividing the environmental data according to the geographical information to obtain regional data; calculating the spatial correlation of the environmental data according to the regional data to obtain second data, and obtaining impact data based on the first data and the second data; establishing a prediction model according to the impact data and the environmental data, inputting the data to be analyzed into the prediction model, and obtaining the prediction result of the environmental data. The present invention adopts, reduces, and significantly reduces costs. The present invention realizes a more accurate and comprehensive prediction of environmental data by comprehensively applying big data technology and advanced machine learning algorithms, and provides an efficient and reliable technical means for environmental monitoring and management.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental data analysis, and in particular to an environmental data analysis method and system based on big data. Background Art

[0002] Environmental data analysis is a key technology for addressing global environmental challenges. However, traditional methods are mainly based on small-scale data sets and simple statistical analysis, which are incapable of processing large data sets and multi-source data sets. They are usually unable to fully reveal the inherent complexity and dynamic characteristics of environmental data, resulting in limited accuracy and practicality of prediction models. In addition, existing technologies are also insufficient in automated data processing and real-time analysis, making it difficult to meet the growing demand for efficient and accurate monitoring. Therefore, it is particularly important to develop an environmental data analysis method that can process large-scale data, capture the complexity of environmental data, and provide real-time and accurate predictions. Summary of the invention

[0003] The purpose of the present invention is to provide an environmental data analysis method and system based on big data.

[0004] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0005] The present invention comprises the following steps:

[0006] Collect environmental data and its basic environmental data and geographical location;

[0007] Obtaining first data according to the relevance of the environmental basic data to the environmental data, and dividing the environmental data according to the geographic information to obtain regional data;

[0008] Calculate the spatial correlation of the environmental data according to the regional data to obtain second data, and obtain impact data based on the first data and the second data;

[0009] A prediction model is established according to the impact data and the environmental data, and the data to be analyzed is input into the prediction model to obtain a prediction result of the environmental data.

[0010] Furthermore, the basic environmental data and the geographical location include:

[0011] Basic environmental data include geological types and human activities. Human activities include building density and forest coverage. Geographical location includes longitude, latitude and altitude.

[0012] Furthermore, the method for obtaining the first data according to the correlation degree of the environmental basic data to the environmental data includes:

[0013] Perform unique hot encoding on the topography, use basic environmental data and environmental data to build a data set, use basic environmental data as the independent variable and environmental data as the dependent variable, divide the data set into a training set and a test set, and use the training set to train a regression model based on the random forest algorithm:

[0014] The decision tree performs feature selection and splitting based on the information gain rate. The formula for the information gain rate is:

[0015]

[0016] Where IGR(f) is the information gain rate of feature f, q j is the sample ratio of the jth category in the data set R, n is the number of categories in the data set R, is the sum of the weighted entropy of all possible values ​​w of feature f, E represents the conditional entropy, R w is a subset of feature f with value w, |R w | is R w is the number of samples in the dataset R, |R| is the number of samples in the dataset R, Var(I f ) is the feature importance I f The variance of ’ To control Var(I f ),

[0017] The objective function of a single decision tree is:

[0018]

[0019] Where N left and N right are the number of samples in the left and right child nodes after the decision tree splits, Left and Right represent the sample index sets in the left and right child nodes, respectively. i is sample i, representing the i-th environmental data, w i is the weight of sample i, and are the average target values ​​of samples in the left and right child nodes, respectively. is the regularization weight of O, O is the regularization term of the decision tree,

[0020] Use the hyperparameters of the regression model as particles and optimize the particles using the test set:

[0021]

[0022] in, is the particle speed, represents the number of iterations, j is the index of the particle, d represents the input of the regression model, is the dynamic inertia weight, s is the position of the particle, and is a dynamically adjusted learning factor, r1 and r2 are random factors, is the best historical position of the particle, is the global optimal position, ζ is a constant, υ is a random number representing random oscillation, ω max and ω min are the maximum and minimum values ​​of the inertia weight, and are the maximum and minimum values ​​of the learning factor c1, and are the maximum and minimum values ​​of the learning factor c2, is the preset number of iterations,

[0023] The SHAP value is used to check the contribution of the feature data to the regression model, and the feature data whose contribution to the regression model is less than a preset contribution threshold is removed to obtain the first data.

[0024] Furthermore, the method of dividing the environmental data according to the geographic information to obtain regional data includes:

[0025] The geographic location is converted into a data vector, and the Euclidean distance between two environmental data is calculated to obtain distance data. Two environmental data whose distance data is less than a preset distance threshold are adjacent data to each other. One environmental data is taken as the center point of the area, and the adjacent data of the center point and the distance data between the center point and its adjacent data are taken as the area data of this environmental data.

[0026] Furthermore, the method of calculating the spatial correlation of the environmental data according to the regional data to obtain the second data includes:

[0027] Normalize the distance data in the regional data and establish a spatial weight matrix.

[0028] Spatial regression model:

[0029]

[0030] Among them, y i is the i-th observed value of the response variable, the response variable is the environmental data of the center point, ρ is the spatial autocorrelation parameter, w il is the element in the i-th row and l-th column of the spatial weight matrix W, y l for y i The adjacent data, N' is the number of adjacent data, is the spatial lag term, x i is the i-th covariate in the covariate matrix X, which is the first data. represents the matrix transpose, α is the parameter vector, g j (z ij ) is a nonparametric smooth function, z ij Represents the value of the non-parametric term of the i-th observation in the j-th dimension, ∈ i is a random error term, a spline approximation of a nonparametric smooth function:

[0031]

[0032] Among them, B jl (z ij )β jl is the B-spline basis function, K j is the number of spline basis functions, β jl are the spline coefficients,

[0033] The acceptance rate of updates to model parameters and nodes is calculated using the reverse jump Markov chain Monte Carlo algorithm:

[0034]

[0035] Where R′ represents the acceptance rate, Π represents the posterior distribution, q represents the proposed distribution, θ represents the model parameters, θ includes α and β, k represents the number of nodes, ζ represents the position of the node, and Y is y i A collection of

[0036] Predicted values ​​from the spatial regression model:

[0037]

[0038] Where β is the set of spline coefficients, σ ​​is the standard deviation parameter, and is the scaling factor of the error distribution, ξ i for y i Auxiliary parameters, η i for y i The standard normal distribution random error term, η i ~N(0,1),

[0039] Likelihood function:

[0040]

[0041] in for A collection of A(ρ) is the spatial autocorrelation matrix, A(ρ)=I N -ρW,I N is the unit matrix, φ(X,Z)=(X T ,B T(Z)), Z is the set of non-parametric terms, ξ is the auxiliary parameter vector,

[0042] The joint posterior distribution and the conditional posterior distribution on α:

[0043] p(ρ,θ,k,ζ,σ,ξ,φ∣Y)∝L(Y∣ρ,θ,σ,ξ)Π(ρ)Π(θ)Π(σ)Π(ξ)Π(φ),

[0044] Π(α|ρ,β,k,ξ,φ,Y)∝N(μ α ,Σ α )

[0045] Among them, μ α is the mean vector, Σ α is the covariance matrix, with the values ​​of the spatial lag terms as the second data.

[0046] Furthermore, the method of fusing the first data and the second data to obtain the influence data includes:

[0047] The first data and the second data are used to establish a data matrix, the data matrix is ​​divided into a training matrix and a test matrix, and the training matrix is ​​used to train the elastic network model. The loss function of the elastic network model is:

[0048]

[0049] Where v is the parameter vector of the elastic network model, v0 is the intercept term, and s t,j is the jth feature data at time point t in the first data, v j For t,j The feature weight of , T is the number of time points, J is the number of features in the first data, represents the second data, λ1, λ2 and λ3 are regularization parameters, ρ is the smoothing coefficient, ρ∈[0,1], F represents the norm, and the optimal parameter vector is:

[0050]

[0051] The elastic network model is trained using the optimal parameter vector, and the accuracy of the elastic network model is verified using the test matrix. The first data whose absolute value of feature weight is less than the preset weight threshold is removed, and the second data and the filtered first data are used to calculate the influence data:

[0052]

[0053] where s t,i is the i-th first data at time point t after screening, and I is the number of features of the first data after screening.

[0054] Furthermore, the method for establishing a prediction model based on the impact data and the environmental data includes:

[0055] The impact data and environmental data are normalized and preprocessed, and the time series are obtained by sorting them in chronological order, and the stationarity test of the time series is performed:

[0056]

[0057] Where S t is the time series data at time point t, ΔS t YesS t The first-order difference of is a constant term, β ‘ is the coefficient of the time trend term, γ is the coefficient of the autoregressive term, φ ’ i is the coefficient of the i-th order lag term, p is the order of the autoregressive term, δ j is the coefficient of the jth order difference term, q is the order of the difference term, ∈ t is the error term,

[0058] The ADF statistic and p-value are calculated and compared with the critical value. The critical value is set to 5%. If the time series is less than the critical value, it is stable. If it is greater than the critical value, it is non-stationary. The difference method is used to stabilize the non-stationary time series.

[0059] Using time series to build forecasting models:

[0060]

[0061] in is the predicted value of the prediction model for the environmental data at time point t, c is a constant term, φ'(B) is an autoregressive term, B is a backward operator, θ(B) is a moving average term, ∈ t is the error term at time point t, X mt is the mth influencing data at time point t, M is the number of influencing data, β ‘ m is the coefficient affecting the data in the mth order,

[0062] Comprehensive error correction of residuals:

[0063]

[0064] in, for The weight of For the polynomial basis functions, is the number of polynomial basis functions, γ k′ for e t-k′ The weight oft-k′ is the error term with a lag of k′ steps, is the adjustment factor based on lag k′ and time point t, l is the maximum number of lags, ζ t,residual is the noise term of the residual,

[0065] The predicted values ​​of the non-stationary time series are de-differentiated and de-normalized, and the predicted values ​​of the stationary time series are de-normalized to obtain the predicted data of the environmental data.

[0066] An environmental data analysis system based on big data, characterized by comprising:

[0067] Collection module, collecting environmental data and its basic environmental data and geographical location;

[0068] a calculation module, which obtains first data according to the correlation degree of the environmental basic data to the environmental data, divides the environmental data according to the geographic information to obtain regional data, calculates the spatial correlation of the environmental data according to the regional data to obtain second data, and establishes a prediction model according to the impact data and the environmental data;

[0069] A data fusion module, which obtains impact data based on the first data and the second data;

[0070] The output module inputs the data to be analyzed into the prediction model to obtain the prediction results of the environmental data.

[0071] The beneficial effects of the present invention are:

[0072] The present invention can effectively improve the accuracy and efficiency of environmental data prediction by combining GIS technology and machine learning algorithms. By combining environmental data with geographic location information, the system can identify and quantify the mutual influence of environmental factors between different geographical locations, thereby more accurately predicting environmental change trends. Automated data processing and intelligent prediction models not only improve prediction efficiency, but also enhance the generalization ability of the model, providing strong technical support for environmental monitoring and management, and having significant social and environmental benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a flowchart of the steps of an environmental data analysis method and system based on big data of the present invention. DETAILED DESCRIPTION

[0074] The present invention is further described below by means of specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0075] The present invention provides an environmental data analysis method and system based on big data, comprising the following steps:

[0076] like Figure 1 As shown, in this embodiment, the following steps are included:

[0077] Collect environmental data and its basic environmental data and geographical location;

[0078] In the actual evaluation, the collected data include:

[0079] Environmental basic data and geographical location:

[0080]

[0081]

[0082] Three sets of environmental data at different times:

[0083] Group 1: PM2.5 (μg / m 3 ): Site A, 50, Site B, 75, Site C, 30, PM10 (μg / m 3 ): Site A, 100, Site B, 120, Site C, 80,

[0084] Group 2: PM2.5 (μg / m 3 ): Site A, 51, Site B, 76, Site C, 31, PM10 (μg / m 3 ): Site A, 101, Site B, 121, Site C, 81,

[0085] Group 1: PM2.5 (μg / m 3 ): Site A, 52, Site B, 77, Site C, 32, PM10 (μg / m 3 ): Site A, 102, Site B, 122, Site C, 82;

[0086] Obtaining first data according to the relevance of the environmental basic data to the environmental data, and dividing the environmental data according to the geographic information to obtain regional data;

[0087] In the actual assessment, the first data: geological type, building density and forest coverage,

[0088] Area data: A, B and C are adjacent to each other, and the distance data are: AB, 149.15m, AC, 298.35m, BC, 149.43m;

[0089] Calculate the spatial correlation of the environmental data according to the regional data to obtain second data, and obtain impact data based on the first data and the second data;

[0090] In the actual evaluation, the second data: A: PM2.5, 42.5, PM10, 90, B: PM2.5, 62.5, PM10, 110, C: PM2.5, 22.5, PM10, 60,

[0091] Impact data: A: PM2.5, 47.5, PM10, 95, B: PM2.5, 67.5, PM10, 115, C: PM2.5, 27.5, PM10, 65;

[0092] A prediction model is established according to the impact data and the environmental data, and the data to be analyzed is input into the prediction model to obtain a prediction result of the environmental data.

[0093] In this embodiment, the basic environmental data and the geographical location include:

[0094] Basic environmental data include geological types and human activities. Human activities include building density and forest coverage. Geographical location includes longitude, latitude and altitude.

[0095] In this embodiment, the method for obtaining the first data according to the relevance of the environmental basic data to the environmental data includes:

[0096] Perform unique hot encoding on the topography, use basic environmental data and environmental data to build a data set, use basic environmental data as the independent variable and environmental data as the dependent variable, divide the data set into a training set and a test set, and use the training set to train a regression model based on the random forest algorithm:

[0097] The decision tree performs feature selection and splitting based on the information gain rate. The formula for the information gain rate is:

[0098]

[0099] Where IGR(f) is the information gain rate of feature f, q j is the sample ratio of the jth category in the data set R, n is the number of categories in the data set R, is the sum of the weighted entropy of all possible values ​​w of feature f, E represents the conditional entropy, R w is a subset of feature f with value w, |R w | is R w is the number of samples in the dataset R, |R| is the number of samples in the dataset R, Var(I f ) is the feature importance I f The variance of ’ To control Var(I f ),

[0100] The objective function of a single decision tree is:

[0101]

[0102] Where N left and N right are the number of samples in the left and right child nodes after the decision tree splits, Left and Right represent the sample index sets in the left and right child nodes, respectively. i is sample i, representing the i-th environmental data, w i is the weight of sample i, and are the average target values ​​of samples in the left and right child nodes, respectively. is the regularization weight of O, O is the regularization term of the decision tree,

[0103] Use the hyperparameters of the regression model as particles and optimize the particles using the test set:

[0104]

[0105] in, is the particle speed, represents the number of iterations, j is the index of the particle, d represents the input of the regression model, is the dynamic inertia weight, s is the position of the particle, and is a dynamically adjusted learning factor, r1 and r2 are random factors, is the best historical position of the particle, is the global optimal position, ζ is a constant, υ is a random number representing random oscillation, ω max and ω min are the maximum and minimum values ​​of the inertia weight, and are the maximum and minimum values ​​of the learning factor c1, and are the maximum and minimum values ​​of the learning factor c2, is the preset number of iterations, which is 100.

[0106] The contribution of the feature data to the regression model is tested using the SHAP value, and the feature data whose contribution to the regression model is less than a preset contribution threshold is removed, where the preset contribution threshold is 5%, to obtain the first data.

[0107] In this embodiment, the method of dividing the environmental data according to the geographic information to obtain regional data includes:

[0108] The geographic location is converted into a data vector, and the Euclidean distance between two environmental data is calculated to obtain the distance data. Two environmental data whose distance data is less than a preset distance threshold are adjacent data to each other. The preset distance threshold is 1 km. One environmental data is taken as the center point of the area, and the adjacent data of the center point and the distance data between the center point and its adjacent data are taken as the area data of this environmental data.

[0109] In this embodiment, the method of calculating the spatial correlation of the environmental data according to the regional data to obtain the second data includes:

[0110] Normalize the distance data in the regional data and establish a spatial weight matrix.

[0111] Spatial regression model:

[0112]

[0113] Among them, y i is the i-th observed value of the response variable, the response variable is the environmental data of the center point, ρ is the spatial autocorrelation parameter, w il is the element in the i-th row and l-th column of the spatial weight matrix W, y l for y i The adjacent data, N' is the number of adjacent data, is the spatial lag term, x i is the i-th covariate in the covariate matrix X, which is the first data. represents the matrix transpose, α is the parameter vector, g j (z ij ) is a nonparametric smooth function, z ij Represents the value of the non-parametric term of the i-th observation in the j-th dimension, ∈ i is a random error term, a spline approximation of a nonparametric smooth function:

[0114]

[0115] Among them, B jl (z ij )β jl is the B-spline basis function, K j is the number of spline basis functions, β jl are the spline coefficients,

[0116] The acceptance rate of updates to model parameters and nodes is calculated using the reverse jump Markov chain Monte Carlo algorithm:

[0117]

[0118] Where R′ represents the acceptance rate, Π represents the posterior distribution, q represents the proposed distribution, θ represents the model parameters, θ includes α and β, k represents the number of nodes, ζ represents the position of the node, and Y is y i A collection of

[0119] Predicted values ​​from the spatial regression model:

[0120]

[0121] Where β is the set of spline coefficients, σ ​​is the standard deviation parameter, and is the scaling factor of the error distribution, ξ i for y i Auxiliary parameters, η i for y i The standard normal distribution random error term, η i ~N(0,1),

[0122] Likelihood function:

[0123]

[0124] in for A collection of A(ρ) is the spatial autocorrelation matrix, A(ρ)=I N -ρW,I N is the unit matrix, φ(X,Z)=(X T ,B T (Z)), Z is the set of non-parametric terms, ξ is the auxiliary parameter vector,

[0125] The joint posterior distribution and the conditional posterior distribution on α:

[0126] p(ρ,θ,k,ζ,σ,ξ,φ∣Y)∝L(Y∣ρ,θ,σ,ξ)Π(ρ)Π(θ)Π(σ)Π(ξ)Π(φ),

[0127] Π(α|ρ,β,k,ξ,φ,Y)∝N(μ α ,Σ α )

[0128] Among them, μ α is the mean vector, Σ α is the covariance matrix, with the values ​​of the spatial lag terms as the second data.

[0129] In this embodiment, the method of fusing the first data and the second data to obtain the influencing data includes:

[0130] The first data and the second data are used to establish a data matrix, the data matrix is ​​divided into a training matrix and a test matrix, and the training matrix is ​​used to train the elastic network model. The loss function of the elastic network model is:

[0131]

[0132] Where v is the parameter vector of the elastic network model, v0 is the intercept term, and s t,j is the jth feature data at time point t in the first data, v j For t,j The feature weight of , T is the number of time points, J is the number of features in the first data, represents the second data, λ1, λ2 and λ3 are regularization parameters, ρ is the smoothing coefficient, ρ∈[0,1], F represents the norm, and the optimal parameter vector is:

[0133]

[0134] The elastic network model is trained using the optimal parameter vector, and the accuracy of the elastic network model is verified using the test matrix. The first data whose absolute value of feature weight is less than the preset weight threshold is removed, and the preset weight threshold is 0.05. The second data and the filtered first data are used to calculate the impact data:

[0135]

[0136] where s t,i is the i-th first data at time point t after screening, and I is the number of features of the first data after screening.

[0137] In this embodiment, the method for establishing a prediction model based on the impact data and the environmental data includes:

[0138] The impact data and environmental data are normalized and preprocessed, and the time series are obtained by sorting them in chronological order, and the stationarity test of the time series is performed:

[0139]

[0140] Where S t is the time series data at time point t, ΔS t YesS t The first-order difference of is a constant term, β ‘ is the coefficient of the time trend term, γ is the coefficient of the autoregressive term, φ i is the coefficient of the i-th order lag term, p is the order of the autoregressive term, δ j is the coefficient of the jth order difference term, q is the order of the difference term, ∈ t is the error term,

[0141] The ADF statistic and p-value are calculated and compared with the critical value. The critical value is set to 5%. If the time series is less than the critical value, it is stable. If it is greater than the critical value, it is non-stationary. The difference method is used to stabilize the non-stationary time series.

[0142] Using time series to build forecasting models:

[0143]

[0144] in is the predicted value of the prediction model for the environmental data at time point t, c is a constant term, φ(B) is an autoregressive term, B is a backward operator, θ(B) is a moving average term, ∈ t is the error term at time point t, X mt is the mth influencing data at time point t, M is the number of influencing data, β ‘ m is the coefficient affecting the data in the mth order,

[0145] Comprehensive error correction of residuals:

[0146]

[0147] in, for The weight of For the polynomial basis functions, is the number of polynomial basis functions, γ k′ for e t-k′ The weight of t-k′ is the error term with a lag of k′ steps, is the adjustment factor based on lag k′ and time point t, l is the maximum number of lags, ζ t,residual is the noise term of the residual,

[0148] The predicted values ​​of the non-stationary time series are de-differentiated and de-normalized, and the predicted values ​​of the stationary time series are de-normalized to obtain the predicted data of the environmental data.

[0149] An environmental data analysis system based on big data, characterized by comprising:

[0150] Collection module, collecting environmental data and its basic environmental data and geographical location;

[0151] a calculation module, which obtains first data according to the correlation degree of the environmental basic data to the environmental data, divides the environmental data according to the geographic information to obtain regional data, calculates the spatial correlation of the environmental data according to the regional data to obtain second data, and establishes a prediction model according to the impact data and the environmental data;

[0152] A data fusion module, which obtains impact data based on the first data and the second data;

[0153] The output module inputs the data to be analyzed into the prediction model to obtain the prediction results of the environmental data.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for analyzing environmental data based on big data, characterized in that: The following steps are involved: Collect environmental data and its basic environmental data and geographical location, Environmental basic data include geological types and human activities. Human activities include building density and forest coverage. Geographical location includes longitude, latitude and altitude. Obtaining first data according to the relevance of the environmental basic data to the environmental data, The topography was encoded with one-hot encoding, and the environmental basic data and environmental data were used to establish a data set. The environmental basic data was used as the independent variable and the environmental data was used as the dependent variable. The data set was divided into a training set and a test set. The training set was used to train a regression model based on the random forest algorithm. Use the SHAP value to test the contribution of the feature data to the regression model, remove the feature data whose contribution to the regression model is less than a preset contribution threshold, and obtain the first data. The environmental data is divided according to the geographic information to obtain regional data; the geographic location is converted into a data vector, and the Euclidean distance between two environmental data is calculated to obtain distance data. Two environmental data with a distance data less than a preset distance threshold are mutually adjacent data, and one environmental data is taken as the center point of the region, and the adjacent data of the center point and the distance data between the center point and its adjacent data are taken as the regional data of this environmental data; Calculate the spatial correlation of the environmental data according to the regional data to obtain second data, and use the value of the spatial lag term as the second data. Normalize the distance data in the regional data and establish a spatial weight matrix. Spatial regression model: Among them, y i is the i-th observed value of the response variable, the response variable is the environmental data of the center point, ρ is the spatial autocorrelation parameter, w il is the element in the ith row and lth column of the spatial weight matrix W, y l for y i The adjacent data, N' is the number of adjacent data, is the spatial lag term, x i is the i-th covariate in the covariate matrix X, which is the first data. represents the matrix transpose, α is the parameter vector, g j (z ij ) is a nonparametric smooth function, z ij Represents the value of the non-parametric term of the i-th observation in the j-th dimension, ∈ i is a random error term, a spline approximation of a nonparametric smooth function: Among them, B jl (z ij )β jl is the B-spline basis function, K j is the number of spline basis functions, β jl are the spline coefficients, Obtaining influence data based on the first data and the second data; A prediction model is established according to the impact data and the environmental data, and the data to be analyzed is input into the prediction model to obtain a prediction result of the environmental data.

2. According to the big data-based environmental data analysis method of claim 1, it is characterized in that: The decision tree performs feature selection and splitting based on the information gain rate. The formula for the information gain rate is: Where IGR(f) is the information gain rate of feature f, q j is the sample ratio of the jth category in the data set R, n is the number of categories in the data set R, is the sum of the weighted entropy of all possible values ​​w of feature f, E represents the conditional entropy, R w is a subset of feature f with value w, |R w | is R w is the number of samples in the dataset R, |R| is the number of samples in the dataset R, Var(I f ) is the feature importance I f The variance of α' is the control Var(I f ), The objective function of a single decision tree is: Where N left and N right are the number of samples in the left and right child nodes after the decision tree splits, Left and Right represent the sample index sets in the left and right child nodes, respectively. i is sample i, representing the i-th environmental data, w i is the weight of sample i, and are the average target values ​​of samples in the left and right child nodes, respectively. is the regularization weight of O, O is the regularization term of the decision tree, Use the hyperparameters of the regression model as particles and optimize the particles using the test set: in, is the velocity of the particle, represents the number of iterations, j is the index of the particle, d represents the input of the regression model, is the dynamic inertia weight, s is the position of the particle, and is a dynamically adjusted learning factor, r1 and r2 are random factors, is the best historical position of the particle, is the global optimal position, ζ is a constant, υ is a random number representing random oscillation, ω max and ω min are the maximum and minimum values ​​of the inertia weight, and are the maximum and minimum values ​​of the learning factor c1, and are the maximum and minimum values ​​of the learning factor c2, is the preset number of iterations.

3. The environmental data analysis method based on big data according to claim 1, characterized in that: The acceptance rate of updates to model parameters and nodes is calculated using the reverse jump Markov chain Monte Carlo algorithm: Where R' represents the acceptance rate, Π represents the posterior distribution, q represents the proposed distribution, θ represents the model parameters, θ includes α and β, k represents the number of nodes, ζ represents the position of the node, and Y is y i A collection of Predicted values ​​from the spatial regression model: Where β is the set of spline coefficients, σ ​​is the standard deviation parameter, and is the scaling factor of the error distribution, ξ i for y i Auxiliary parameters, η i for y i The standard normal distribution random error term, η i ~N(0,1), Likelihood function: in for A collection of A(ρ) is the spatial autocorrelation matrix, A(ρ)=I N -ρW,I N is the unit matrix, Ф(X,Z)=(X T ,B T (Z)), Z is the set of non-parametric terms, ξ is the auxiliary parameter vector, The joint posterior distribution and the conditional posterior distribution on α: p(ρ,θ,k,ζ,σ,ξ,φ∣Y)∝L(Y∣ρ,θ,σ,ξ)Π(ρ)Π(θ)Π(σ)Π(ξ)Π(φ), Π(α|ρ,β,k,ξ,φ,Y)∝N(μ α ,S α ) Among them, μ α is the mean vector, Σ α is the covariance matrix.

4. The environmental data analysis method based on big data according to claim 1, characterized in that: The method of fusing the first data and the second data to obtain influence data includes: The first data and the second data are used to establish a data matrix, the data matrix is ​​divided into a training matrix and a test matrix, and the training matrix is ​​used to train the elastic network model. The loss function of the elastic network model is: Where v is the parameter vector of the elastic network model, v0 is the intercept term, and s t,j is the jth feature data at time point t in the first data, v j For t,j The feature weight of , T is the number of time points, J is the number of features in the first data, represents the second data, λ1, λ2 and λ3 are regularization parameters, ρ' is the smoothing coefficient, ρ'∈[0,1], F represents the norm, and the optimal parameter vector is: The elastic network model is trained using the optimal parameter vector, and the accuracy of the elastic network model is verified using the test matrix. The first data whose absolute value of feature weight is less than the preset weight threshold is removed, and the second data and the filtered first data are used to calculate the influence data: where s t,i is the i-th first data at time point t after screening, and I is the number of features of the first data after screening.

5. The environmental data analysis method based on big data according to claim 1, characterized in that: The method for establishing a prediction model according to the impact data and the environmental data comprises: The impact data and environmental data are normalized and preprocessed, and the time series are obtained by sorting them in chronological order, and the stationarity test of the time series is performed: Where S t is the time series data at time point t, ΔS t YesS t The first-order difference of is the constant term, β is the coefficient of the time trend term, γ is the coefficient of the autoregressive term, Ф' i is the coefficient of the i-th order lag term, p is the order of the autoregressive term, δ j is the coefficient of the jth order difference term, q is the order of the difference term, ∈ t is the error term, The ADF statistic and p-value are calculated and compared with the critical value. The critical value is set to 5%. If the time series is less than the critical value, it is stable. If it is greater than the critical value, it is non-stationary. The difference method is used to stabilize the non-stationary time series. Using time series to build forecasting models: in is the predicted value of the prediction model for the environmental data at time point t, c is a constant term, Ф'(B) is an autoregressive term, B is a backward operator, θ(B) is a moving average term, ∈ t is the error term at time point t, X mt is the mth influencing data at time point t, M is the number of influencing data, β ‘ m is the coefficient affecting the data in the mth order, Comprehensive error correction of residuals: in, for The weight of For the polynomial basis functions, is the number of polynomial basis functions, γ k' for e t-k' The weight of t-k' is the error term with a lag of k' steps, is the adjustment factor based on lag k' and time point t, l is the maximum number of lags, ζ t,residual is the noise term of the residual, The predicted values ​​of the non-stationary time series are de-differentiated and de-normalized, and the predicted values ​​of the stationary time series are de-normalized to obtain the predicted data of the environmental data.

6. An environmental data analysis system based on big data, used to execute the method according to any one of claims 1 to 5, characterized in that: include: Collection module, collecting environmental data and its basic environmental data and geographical location; A calculation module, which obtains first data according to the correlation degree of the environmental basic data to the environmental data, divides the environmental data according to the geographic information to obtain regional data, calculates the spatial correlation of the environmental data according to the regional data to obtain second data, and establishes a prediction model according to the impact data and the environmental data; A data fusion module, which obtains impact data based on the first data and the second data; The output module inputs the data to be analyzed into the prediction model to obtain the prediction results of the environmental data.

Images