Big data-based ecological environment real-time monitoring system and method

Abstract

The invention relates to the technical field of ecological environment monitoring, in particular to an ecological environment real-time monitoring system and method based on big data, and the system comprises a data collection module, a data transmission module, a data processing and analysis module, a data storage module, an early warning module, and a mobile terminal. By carrying out linear correlation analysis, nonlinear correlation analysis and dynamic correlation analysis on multi-source data and carrying out trend prediction on ecological environment parameters by adopting a machine learning algorithm, accurate environment early warning and decision support can be realized. The abnormal value is detected, the worker remeasures the abnormal value according to the area positioning information and compares the remeasured value with the original measured value to determine whether the abnormal value is continuously adopted, and through repeated measurement and comparison, the situation that the abnormal value is identified as the abnormal value due to sudden change of the environmental parameters can be avoided; and the optimal opportunity of environmental governance is delayed.

Description

Technical Field

[0001] The present invention relates to the technical field of ecological environment monitoring, and more particularly to a real-time ecological environment monitoring system based on big data. Background Art

[0002] With the acceleration of the industrialization process, ecological environment problems have become increasingly prominent. Traditional ecological environment monitoring methods usually rely on manual sampling and laboratory analysis, which have problems such as long data collection cycles, poor real-time performance, and limited coverage. In addition, existing monitoring systems lack the ability to effectively integrate and analyze multi-source data, making it difficult to achieve accurate environmental early warning and decision support. Therefore, there is an urgent need for a real-time ecological environment monitoring system and method based on big data to improve monitoring efficiency and data analysis capabilities. In view of this, we propose a real-time ecological environment monitoring system and method based on big data. Summary of the Invention

[0003] The purpose of the present invention is to provide a real-time ecological environment monitoring system and method based on big data to solve the problems raised in the above background art.

[0004] To achieve the above purpose, the present invention provides the following technical solutions:

[0005] A real-time ecological environment monitoring system based on big data, including a data collection module, a data transmission module, a data processing and analysis module, a data storage module, an early warning module, and a mobile terminal.

[0006] The data collection module includes sensor nodes distributed in different regions, and each node is equipped with multiple sensors for real-time collection of ecological environment parameters.

[0007] The data transmission module is communicatively connected to the data collection module and the data processing and analysis module respectively, and is used to transmit the ecological environment parameters collected by the data collection module to the data processing and analysis module through a wireless network.

[0008] The data processing and analysis module is used to perform data cleaning, integration, and data analysis on the collected ecological environment parameters, and use machine learning algorithms to predict the trends of ecological environment parameters.

[0009] The data storage module is communicatively connected to the data processing and analysis module, and is used to store and access the collected ecological environment parameters and analysis results.

[0010] The early warning module is communicatively connected to the data processing and analysis module, and is used to generate early warning information according to the analysis results.

[0011] The mobile terminal is wirelessly connected to the data processing and analysis module and the early warning module respectively, and includes a user display interface for receiving data analysis and processing results and early warning information.

[0012] Preferably, the sensor nodes are evenly distributed in different areas, and each area and sensor node are numbered individually, and each sensor node is also provided with a GPS positioning component.

[0013] Preferably, the data cleaning includes processing missing values ​​and outliers, wherein:

[0014] When there are missing values, the data processing and analysis module sends a signal to the early warning module, which generates early warning information and sends it to the mobile terminal. The staff checks whether the sensor and transmission network are normal based on the missing values ​​and fills in the missing values.

[0015] Z-score or IQR is used to detect outliers. When an outlier exists, the data processing and analysis module sends a signal to the early warning module. The early warning module generates early warning information and sends it to the mobile terminal. The staff re-measures the outlier based on the regional positioning information and compares the re-measured value with the original measured value to determine whether the outlier should continue to be used.

[0016] Preferably, filling the missing values ​​includes filling the missing values ​​with the mean, median or mode.

[0017] Preferably, when the relative deviation between the remeasured value and the original measured value is greater than or equal to a set threshold, the remeasured value is used to fill the original measured value, and when the relative deviation between the remeasured value and the original measured value is less than the set threshold, the original measured value is retained.

[0018] Preferably, the data analysis includes linear correlation analysis, nonlinear correlation analysis and dynamic correlation analysis. The linear correlation analysis adopts Pearson correlation coefficient and Spearman correlation coefficient, the nonlinear correlation analysis adopts mutual information and Granger causality test, and the dynamic correlation analysis adopts VAR model and LSTM model.

[0019] Preferably, the data processing and analysis module also includes visualizing the data analysis results through heat maps, network maps, and time series maps, and sending them to the mobile terminal.

[0020] Preferably, the machine learning algorithm includes regression analysis, cluster analysis, and time series prediction.

[0021] Preferably, the different areas are evenly divided into multiple sampling points, and different sampling points are provided with unique positioning coordinates according to GPS positioning information, and the sampling points of multiple sensors need to be changed at regular intervals.

[0022] In addition, this technical solution also provides a real-time monitoring method of the ecological environment based on big data, including the following steps:

[0023] S1. Real-time collection of ecological environment parameters through a variety of sensors distributed in different areas;

[0024] S2, transmitting the ecological environment parameters collected by the data collection module to the data processing and analysis module through a wireless network;

[0025] S3. Clean, integrate and analyze the collected ecological environment parameters through the data processing and analysis module, and use machine learning algorithms to predict the trend of ecological environment parameters;

[0026] S4, generating warning information according to the analysis results through the warning module;

[0027] S5. Transmit the received data analysis and processing results and warning information to the mobile terminal.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The real-time ecological environment monitoring system based on big data includes data acquisition module, data transmission module, data processing and analysis module, data storage module, early warning module, and mobile terminal. It can realize accurate environmental early warning and decision support by performing linear correlation analysis, nonlinear correlation analysis, and dynamic correlation analysis on multi-source data, and using machine learning algorithms to predict the trend of ecological environment parameters.

[0030] 2. The real-time ecological environment monitoring system based on big data detects outliers. The staff re-measures the outliers according to the regional positioning information and compares the re-measured values ​​with the original measured values ​​to determine whether the outliers should continue to be used. Through repeated measurements and comparisons, it can avoid being identified as outliers due to sudden changes in environmental parameters, thereby delaying the best time for environmental governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the real-time monitoring system of the ecological environment based on big data in the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See alsoFigure 1 As shown in the figure, the present invention provides a technical solution:

[0034] An ecological environment real-time monitoring system based on big data, including a data acquisition module, a data transmission module, a data processing and analysis module, a data storage module, an early warning module, and a mobile terminal.

[0035] The data acquisition module includes sensor nodes distributed in different regions, and each node is equipped with multiple sensors for real-time acquisition of ecological environment parameters.

[0036] The data transmission module is communicatively connected to the data acquisition module and the data processing and analysis module respectively, and is used to transmit the ecological environment parameters collected by the data acquisition module to the data processing and analysis module through a wireless network.

[0037] The data processing and analysis module is used to perform data cleaning, integration, and data analysis on the collected ecological environment parameters, and use machine learning algorithms to predict the trends of ecological environment parameters.

[0038] The data storage module is communicatively connected to the data processing and analysis module, and is used to store and access the collected ecological environment parameters and analysis results.

[0039] The early warning module is communicatively connected to the data processing and analysis module, and is used to generate early warning information according to the analysis results.

[0040] The mobile terminal is wirelessly connected to the data processing and analysis module and the early warning module respectively, and includes a user display interface for receiving data analysis processing results and early warning information.

[0041] In this embodiment, the sensor nodes are evenly distributed in different regions, and each region and sensor node are numbered separately. Each sensor node is also provided with a GPS positioning component. The monitoring data of different regions are stored separately in the data storage module, which is convenient for data classification and access. Each different region is evenly divided into multiple sampling points, and each sampling point has a unique positioning coordinate according to the GPS positioning information. Multiple sensors need to change the sampling points at regular intervals.

[0042] In this embodiment, the data cleaning includes processing missing values and outliers, where

[0043] When there are missing values, the data processing and analysis module sends a signal to the early warning module, and the early warning module generates early warning information and sends it to the mobile terminal. The staff checks whether the sensors and the transmission network are normal according to the missing values. When the sensors and the transmission network fail, the staff can discover and handle them in time, and fill in the missing values, including filling the missing values with the mean, median, or mode.

[0044] Use Z-score or IQR to detect outliers. Z-score is a commonly used statistical method that determines whether a data point is abnormal by measuring the standard deviation distance between the data point and the mean of the data set. The calculation formula is as follows:

[0045]

[0046] Where X is the value of the data point, μ is the mean of the data set, and σ is the standard deviation of the data set. When the Z-score is 0, the value of the data point is equal to the mean of the data set; when the Z-score is greater than 0, the value of the data point is higher than the mean of the data set; when the Z-score is less than 0, the value of the data point is lower than the mean of the data set.

[0047] When the absolute value of the Z-score is greater than the set threshold, the data point is judged as an outlier.

[0048] IQR (Interquartile Range) is based on the quartiles of the data and can effectively identify extreme values ​​in the data. It includes the following steps:

[0049] 1) Calculate quartiles

[0050] Q1 (First Quartile): 25% of the data points in the data set are less than or equal to Q1.

[0051] Q3 (third quartile): 75% of the data points in the data set are less than or equal to Q3.

[0052] 2) Calculate IQR

[0053] IQR is the difference between Q3 and Q1:

[0054] IQR=Q3-Q1

[0055] 3) Determine the range of outliers

[0056] Lower bound: Q1-1.5×IQR

[0057] Upper bound: Q3 + 1.5 × IQR

[0058] 4) Identify outliers

[0059] Any data point that is smaller than the lower bound or larger than the upper bound is considered an outlier.

[0060] When there are abnormal values, the data processing and analysis module sends a signal to the early warning module. The early warning module generates early warning information and sends it to the mobile terminal. The staff re-measures the abnormal value based on the regional positioning information and compares the re-measured value with the original measured value to determine whether the abnormal value should continue to be used.

[0061] Specifically, when the relative deviation between the retest value and the original measurement value is greater than or equal to the set threshold, it proves that the original measurement value is abnormal and cannot be used. The original measurement value is filled with the retest value. When the relative deviation between the retest value and the original measurement value is less than the set threshold, it proves that the original measurement value is correct, and the original measurement value is retained. At the same time, whether the measured value is too large or too small is related to the sudden change of environmental parameters. Through repeated measurement and comparison, it is possible to avoid being identified as an outlier due to the sudden change of environmental parameters and delay the best time for environmental governance. Subsequently, the staff still need to continuously monitor this area.

[0062] Furthermore, the data analysis includes linear correlation analysis, non-linear correlation analysis, and dynamic correlation analysis. The linear correlation analysis uses Pearson correlation coefficient and Spearman correlation coefficient. The non-linear correlation analysis uses mutual information and Granger causality test. The dynamic correlation analysis uses VAR model and LSTM model, specifically as follows:

[0063] The Pearson correlation coefficient is used to measure the linear relationship between two continuous variables. Its value range is between -1 and 1, which can reflect the correlation strength and direction between variables. The calculation formula for the Pearson correlation coefficient of two variables X and Y is:

[0064]

[0065] When γ = 1, it is a perfect positive linear correlation, and variables X and Y show a completely consistent linear growth relationship;

[0066] When γ = -1, it is a perfect negative linear correlation, and variables X and Y show a completely opposite linear relationship;

[0067] When γ = 0, there is no linear correlation, and there is no linear relationship between variables X and Y;

[0068] When 0 < γ < 1, it is a positive correlation, and variables X and Y show a positive correlation relationship;

[0069] When -1 < γ < 0, it is a negative correlation, and variables X and Y show a negative correlation relationship.

[0070] The Spearman correlation coefficient is a non-parametric statistical method used to measure the monotonic relationship between two variables. It does not require the data to satisfy the normal distribution and is applicable to ordinal data or interval data that do not satisfy the linear relationship. The calculation formula is:

[0071]

[0072] where d i is the rank difference between the two variables, and n is the sample size. The calculation steps are as follows:

[0073] 1) Convert the data to ranks: Sort the data of the two variables respectively and assign ranks.

[0074] 2) Calculate the rank difference: Calculate the rank difference d for each pair of data points. i ;

[0075] 3) Square the rank difference: Calculate the square of each d. i square

[0076] 4) Sum: Sum all. add them up;

[0077] 5) Substitute the result into the formula to calculate the Spearman correlation coefficient.

[0078] ρ ranges between -1 and 1. ρ = 1: Perfect positive correlation; ρ = -1: Perfect negative correlation; ρ = 0:

[0079] No correlation.

[0080] Mutual information measures the amount of shared information between two random variables. Mutual information can capture complex dependencies between variables, including non-linear and non-monotonic relationships. For two random variables X and Y, their mutual information is:

[0081]

[0082] where p(x,y) is the joint probability distribution of X and Y; p(x) and p(y) are the marginal probability distributions of X and Y respectively.

[0083] Granger causality test is based on the vector autoregressive model. Suppose there are two time series X t and Y t , to test whether X is the Granger cause of Y, it can be carried out through the following steps:

[0084] 1) Construct an unconstrained model:

[0085]

[0086] where p is the lag order; β i and γ i are regression coefficients; ∈ t is the error term.

[0087] 2) Construct a constrained model:

[0088] In the constrained model, assume that X is not the Granger cause of Y, that is, γ i = 0 holds for all i:

[0089]

[0090] 3) Hypothesis Testing

[0091] Null hypothesis H0: X is not the Granger cause of Y (i.e., γ i = 0 for all i);

[0092] Alternative hypothesis H1: X is the Granger cause of Y (i.e., there exists at least one γ i ≠ 0).

[0093] By comparing the goodness of fit of the unconstrained model and the constrained model (using the F-test or likelihood ratio test), determine whether to reject the null hypothesis.

[0094] The VAR model is a multivariate time series model, where each variable is a linear function of its own lagged values and the lagged values of other variables. For a system with k variables, the VAR(p) model can be expressed as:

[0095]

[0096] where, Y t is a k×1 vector representing the k variables at time t; c is a k×1 constant vector; is a k×k coefficient matrix representing the influence of the i-th lag; ∈ t is a k×1 error vector, usually assumed to be white noise; p is the lag order.

[0097] The core idea of LSTM is to control the flow of information through memory cells and three gating mechanisms (input gate, forget gate, output gate). For a time step t, the calculation process of LSTM is as follows:

[0098] 1) Forget gate:

[0099] f t = σ(W f · [h t-1 , x t + b f )

[0100] where, h t-1 is the hidden state of the previous time step; x t is the input of the current time step; W f and b f are the weights and biases of the forget gate.

[0101] 2) Input gate:

[0102] i t = σ(w i · [h t-1 , x t + b i )

[0103]

[0104] Among them, i t is the part that determines the update; is the candidate memory cell value.

[0105] 3) Update the memory cell:

[0106]

[0107] Among them, C t-1 is the memory cell of the previous time step; C t is the updated memory cell.

[0108] 4) Output gate:

[0109] o t = σ(W o · [h t-1 , x t + b o )

[0110] h t = o t · tanh(C t )

[0111] Among them, o t is the part that determines the output; h t is the hidden state of the current time step.

[0112] By performing linear correlation analysis, non - linear correlation analysis, and dynamic correlation analysis on different data, it is convenient to understand the variation relationship between different environmental parameters. When a certain environmental parameter in the region changes, the change state of another related environmental parameter can be timely monitored, which is convenient for the timeliness of staff handling, and potential pollution sources can be discovered in a timely manner.

[0113] In addition, the data processing and analysis module also includes visually visualizing the data analysis results through heat maps, network diagrams, and time - series diagrams, and sending them to the mobile terminal for convenient visual observation by the staff.

[0114] In this embodiment, the machine learning algorithm includes regression analysis, clustering analysis, and time - series prediction.

[0115] Regression analysis is mainly used to predict continuous values. By establishing a relationship model between independent variables and dependent variables for prediction, it mainly includes various methods such as linear regression, polynomial regression, ridge regression, logistic regression, and decision - tree regression.

[0116] Clustering analysis belongs to unsupervised learning methods. Its goal is to divide the samples in the dataset into several groups (clusters) so that the samples within the same group are as similar as possible, while the samples between different groups are as different as possible. It mainly includes various methods such as K-means clustering and hierarchical clustering.

[0117] Time series prediction: A time series measures a variable or a set of variables in an objective process and obtains an ordered set discretized with time as the independent variable at different moments.

[0118] By using machine learning algorithms to perform real-time analysis and trend prediction on different ecological parameters, it is possible to process and analyze large-scale and multi-dimensional ecological data. Through the real-time analysis of these data, the change characteristics and trends of the ecosystem can be quickly identified, thus greatly improving the efficiency of ecological monitoring. At the same time, machine learning algorithms can automatically learn the laws and patterns in the data, avoiding errors caused by human factors and improving the accuracy of monitoring. In addition, the results of real-time analysis and trend prediction can provide a scientific basis for ecological management. For example, in water resource management, by predicting the changes in river flow and water quality, a more scientific water resource scheduling plan can be formulated to avoid water shortages or pollution incidents. In biodiversity conservation, by predicting the changes in species distribution and quantity, corresponding protection measures can be formulated to prevent species extinction and ecosystem imbalance. In addition, machine learning algorithms can also assist in formulating ecological restoration plans, evaluating the effects of ecological restoration measures, providing scientific guidance for ecological restoration, and promoting ecological environmental protection and sustainable development.

[0119] In addition, this technical solution also provides a real-time ecological environment monitoring method based on big data, including the following steps:

[0120] S1. Real-time collect ecological environment parameters through a variety of sensors distributed in different regions;

[0121] S2. Transmit the ecological environment parameters collected by the data collection module to the data processing and analysis module through a wireless network;

[0122] S3. Through the data processing and analysis module, perform data cleaning, integration and data analysis on the collected ecological environment parameters, and use machine learning algorithms to predict the trends of ecological environment parameters;

[0123] S4. Generate warning information through the warning module according to the analysis results;

[0124] S5. Transmit the received data analysis and processing results and warning information to the mobile terminal.

[0125] The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. The real-time monitoring system of ecological environment based on big data is characterized by: It includes data acquisition module, data transmission module, data processing and analysis module, data storage module, early warning module, and mobile terminal. The data acquisition module includes sensor nodes distributed in different areas, each node is equipped with multiple sensors for real-time acquisition of ecological environment parameters; The data transmission module is respectively connected to the data acquisition module and the data processing and analysis module for transmitting the ecological environment parameters collected by the data acquisition module to the data processing and analysis module via a wireless network; The data processing and analysis module is used to clean, integrate and analyze the collected ecological environment parameters, and use machine learning algorithms to predict the trend of ecological environment parameters; The data storage module is in communication with the data processing and analysis module and is used to store and access the collected ecological environment parameters and analysis results; The early warning module is in communication with the data processing and analysis module and is used to generate early warning information according to the analysis results; The mobile terminal is wirelessly connected to the data processing and analysis module and the early warning module respectively, and includes a user display interface for receiving data analysis and processing results and early warning information.

2. The real-time ecological environment monitoring system based on big data according to claim 1 is characterized in that: The sensor nodes are evenly distributed in different areas, and each area and sensor node are numbered separately. Each sensor node is also provided with a GPS positioning component.

3. The real-time ecological environment monitoring system based on big data according to claim 1 is characterized in that: The data cleaning includes processing missing values ​​and outliers, wherein: When there are missing values, the data processing and analysis module sends a signal to the early warning module, which generates early warning information and sends it to the mobile terminal. The staff checks whether the sensor and transmission network are normal based on the missing values ​​and fills in the missing values. Z-score or IQR is used to detect outliers. When an outlier exists, the data processing and analysis module sends a signal to the early warning module. The early warning module generates early warning information and sends it to the mobile terminal. The staff re-measures the outlier based on the regional positioning information and compares the re-measured value with the original measured value to determine whether the outlier should continue to be used.

4. The real-time ecological environment monitoring system based on big data according to claim 3 is characterized in that: The filling of missing values ​​includes filling the missing values ​​with the mean, median or mode.

5. The real-time ecological environment monitoring system based on big data according to claim 3 is characterized in that: When the relative deviation between the re-measured value and the original measured value is greater than or equal to the set threshold, the re-measured value is used to fill the original measured value. When the relative deviation between the re-measured value and the original measured value is less than the set threshold, the original measured value is retained.

6. The real-time ecological environment monitoring system based on big data according to claim 1 is characterized in that: The data analysis includes linear correlation analysis, nonlinear correlation analysis and dynamic correlation analysis. The linear correlation analysis adopts Pearson correlation coefficient and Spearman correlation coefficient, the nonlinear correlation analysis adopts mutual information and Granger causality test, and the dynamic correlation analysis adopts VAR model and LSTM model.

7. The real-time ecological environment monitoring system based on big data according to claim 6 is characterized in that: The data processing and analysis module also includes visualizing the data analysis results through heat maps, network maps, and time series maps, and sending them to the mobile terminal.

8. The real-time ecological environment monitoring system based on big data according to claim 1 is characterized in that: The machine learning algorithms include regression analysis, cluster analysis, and time series prediction.

9. The real-time ecological environment monitoring system based on big data according to claim 1 is characterized in that: The different areas are evenly divided into multiple sampling points, and different sampling points are provided with unique positioning coordinates according to GPS positioning information. Multiple sensors need to change the sampling points at regular intervals.

10. A real-time monitoring method of ecological environment based on big data, characterized in that: The following steps are involved: S1. Real-time collection of ecological environment parameters through a variety of sensors distributed in different areas; S2, transmitting the ecological environment parameters collected by the data collection module to the data processing and analysis module through a wireless network; S3. Clean, integrate and analyze the collected ecological environment parameters through the data processing and analysis module, and use machine learning algorithms to predict the trend of ecological environment parameters; S4, generating warning information according to the analysis results through the warning module; S5. Transmit the received data analysis and processing results and warning information to the mobile terminal.

Images