Air pollution real-time early warning method and system based on intelligent algorithm and Flink

Abstract

The invention provides an air pollution real-time early warning method based on an intelligent algorithm and Flink, and relates to the technical field of environment detection, and the method comprises the steps: collecting pollution gas data, preprocessing the pollution gas data, and storing the preprocessed pollution gas data in a MySQL database; marking unlabeled historical data in the MySQL database by using the self-attention time domain convolutional network model to obtain a marked data set; performing automatic rule extraction on the marked data set by using a decision tree algorithm PART to obtain a rule set; the rule set is converted into a complex event mode rule through Flink CEP; capturing changed data of the MySQL database in real time through Maxwell, and synchronizing the data to a theme partition of Kafka; the data flow of the theme partition in the Kafka is consumed in real time through the Flink, the predefined complex event mode rule is matched through the Flink CEP, and warning information of the air pollution level is generated when matching succeeds and displayed on a control console in real time. The air pollution level change is monitored in real time, the complex event processing rule is automatically extracted, the rule is automatically generated, and the labor cost is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of environmental detection technology, and in particular to a real-time air pollution early warning method and system based on intelligent algorithm and Flink. Background Art

[0002] The Air Quality Index (AQI) is an important indicator for assessing the degree of air pollution, but traditional monitoring methods often rely on manual detection and rule definition, which are inefficient and slow to respond.

[0003] Therefore, it is necessary to improve the existing real-time early warning technology for air pollution to overcome the defects of the existing technology. Summary of the invention

[0004] In order to overcome the problems existing in the related art, one of the purposes of the present invention is to provide a real-time early warning method for air pollution based on intelligent algorithms and Flink. The real-time early warning method for air pollution based on intelligent algorithms and Flink automatically extracts CEP rules through intelligent algorithms to overcome the inefficiency of relying on manually defined rules in air quality monitoring in the prior art, and realizes the automation of rule generation. At the same time, combined with real-time stream processing technology, it can capture and process changes in AQI data in a timely manner to ensure the accuracy and timeliness of early warning information.

[0005] A real-time air pollution warning method based on intelligent algorithms and Flink, including:

[0006] Collect polluted gas data and store them in MySQL database after preprocessing;

[0007] The self-attention time-domain convolutional network model is used to label the unlabeled historical data in the MySQL database to obtain a labeled dataset;

[0008] Using a decision tree algorithm PART to automatically extract rules from the labeled data set to obtain a rule set;

[0009] Convert the rule set into complex event pattern rules through Flink CEP;

[0010] Maxwell is used to capture data changes in the MySQL database in real time and synchronize it to the topic partition of Kafka;

[0011] Flink is used to consume the data stream of topic partitions in Kafka in real time, and Flink CEP is used to match predefined complex event pattern rules. When the match is successful, a warning message of the air pollution level is generated and displayed on the console in real time.

[0012] In some embodiments, collecting polluted gas data and storing it in a MySQL database after preprocessing specifically includes:

[0013] The polluted gas data is collected every five minutes through sensors;

[0014] Preprocess the polluted gas data, calculate the air pollution index value AQI and store it in the MySQL database;

[0015] The pollutant gas data includes ozone, particulate matter, carbon monoxide, sulfur dioxide and nitrogen dioxide.

[0016] In some implementations, labeling unlabeled historical data using a self-attention temporal convolutional network model specifically includes:

[0017] Constructing an error vector using the self-attention temporal convolutional network model;

[0018] Anomaly scores are calculated based on the error vector, multivariate Gaussian function, and Mahalanobis distance formula, and the data set is labeled.

[0019] In some embodiments, the self-attention time-domain convolutional network model includes a self-attention module and a time-domain convolutional network module, and the self-attention time-domain convolutional network model is trained using a historical data set in a MySQL database.

[0020] In some embodiments, the self-attention temporal convolutional network includes an input layer, a linear layer, a self-attention temporal convolutional layer, and an output layer;

[0021] The self-attention temporal convolution layer comprises a self-attention layer, a normalization layer and a temporal convolution layer;

[0022] The temporal convolution layer includes a causal convolution 1D layer, a weight normalization layer, an activation layer and a Dropout layer.

[0023] In some implementations, calculating anomaly scores based on the error vector, multivariate Gaussian function, and Mahalanobis distance formula, and marking data specifically includes:

[0024] The multivariate Gaussian distribution is fitted by the error vector, and the multivariate Gaussian distribution function formula is as follows:

[0025]

[0026] Among them, μ is the mean of the error vector, ∑ is the covariance matrix, ∑ -1 is the inverse matrix of the covariance matrix;

[0027] Using maximum likelihood estimation to obtain the estimated value of μ Estimated value of ∑

[0028] The parameters obtained by the maximum likelihood estimation method and

[0029] The Mahalanobis distance is calculated using the obtained μ and ∑. The Mahalanobis distance represents the distance of the sample from the center point. The Mahalanobis distance formula is as follows:

[0030] a t =(e t -μ) T ∑ -1 (e t -μ)

[0031] Among them, e t is the error vector of the sample point, a t is the distance scalar of the sample point from the center point;

[0032] Set the threshold ε and confidence interval, the a t The value of is in the confidence interval [0,ε] for 95% and from ε to infinity for 10%;

[0033] when a t >ε when the data is marked as abnormal data;

[0034] when a t When ≤ε, the data is marked as normal data.

[0035] In some implementations, after obtaining the labeled data set, the method further includes:

[0036] The labeled data set is divided into a training set and a test set. The training set is used for classification training of the decision tree PART model, and the test set is used to evaluate the classification results of the model. The evaluation indicators of the evaluation model include accuracy, recall rate, F1-score and ROC area.

[0037] In some embodiments, the Maxwell configuration file includes a data producer type, a Kafka cluster address, a Kafka topic name, a MySQL database server address, and a MySQL database user name and password.

[0038] The second object of the present invention is to provide a real-time early warning system for air pollution based on intelligent algorithms and Flink, which is used to execute the real-time early warning method for air pollution based on intelligent algorithms and Flink as mentioned above. The system includes a collection module for collecting polluted gas data, a data storage module, a data processing module, and a data monitoring and alarm module.

[0039] The beneficial effects of the present invention are:

[0040] The present invention provides a real-time air pollution early warning method based on intelligent algorithm and Flink, which realizes accurate pollution event recognition and automatic rule generation through anomaly detection of self-attention time-domain convolutional network model and rule extraction of decision tree algorithm; utilizes Flink's real-time computing capability and CEP technology, combined with Maxwell's data synchronization function, realizes real-time data processing and early warning triggering, and enhances the real-time nature of early warning; has higher accuracy, real-time nature, degree of automation, flexibility and maintainability, effectively improves the efficiency and effect of air pollution early warning, and more effectively protects public health and environmental safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a brief flow chart of the real-time early warning method for air pollution based on intelligent algorithm and Flink provided in the embodiment of the present application;

[0042] Figure 2 This is a diagram of the architecture of the self-attention time-domain convolutional network model provided in an embodiment of the present application;

[0043] Figure 3 A histogram of Mahalanobis distance results provided in an embodiment of the present application;

[0044] Figure 4 A sample diagram of rules extracted from the decision tree PART model provided in the embodiments of the present application;

[0045] Figure 5 This is a sample diagram of the health concern levels of the Air Pollution Index (AQI) provided in the embodiments of the present application. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0047] Example

[0048] In recent years, the rapid development of intelligent algorithms has brought new opportunities for environmental monitoring. Deep learning and data mining algorithms can automatically identify abnormal patterns and trends from large amounts of data, thereby improving the efficiency and accuracy of data processing.

[0049] At the same time, the rise of stream processing technology, especially tools such as Apache Flink, has greatly enhanced the ability to process real-time data streams. Flink provides complex event processing (CEP) capabilities, enabling the system to match and analyze complex events in data streams in real time. This capability is of great significance in air pollution warning systems, which can identify potential air quality problems in a timely manner.

[0050] As a real-time database capture tool, Maxwell can effectively write database changes to message queues (such as Kafka), laying the foundation for subsequent real-time data processing.

[0051] By combining traditional data monitoring methods with advanced intelligent algorithms and stream processing technologies, an efficient and automated real-time early warning system for air pollution can be realized.

[0052] This system can not only reduce the cost of manual intervention, but also improve the response speed to changes in air quality, thereby reducing the impact of air pollution on human health and protecting the health of the public, especially vulnerable groups.

[0053] like Figures 1 to 5 As shown, this embodiment provides a real-time air pollution warning method based on intelligent algorithm and Flink. The brief process of the real-time air pollution warning method is as follows Figure 1 As shown, specifically including:

[0054] Step I: The sensor stores the collected polluted gas data in the MySQL database after preprocessing.

[0055] Step II. The self-attention temporal convolutional network model marks the unlabeled historical data as abnormal or normal. The dataset is automatically labeled through deep learning technology to obtain a labeled dataset.

[0056] Step III. Use the decision tree algorithm PART to automatically extract rules from the labeled data set. Each leaf node to the root node represents a rule, and in this way, a set of rules will be obtained.

[0057] Step IV. Flink CEP converts the obtained rule set into complex event pattern rules. After that, Flink is used to consume the data of the topic partitions in Kafka in real time. The data of the Kafka topic partitions comes from the database change data captured in real time by Maxwell.

[0058] Step V. Flink reads the data stream from the Kafka topic partition in real time and applies complex event processing (CEP) technology to match predefined pattern rules. Once a qualifying event is detected, Flink CEP will generate a warning message about the air pollution level and display it on the console in real time.

[0059] Specifically, in step I, the sensor collects harmful gas data such as ozone, particulate matter, carbon monoxide, sulfur dioxide and nitrogen dioxide every five minutes. These gas data are pre-processed to obtain the air pollution index (AQI) and stored in the database. The database will store historical data, current data and data to be collected in the future.

[0060] Air pollution warning levels are divided into 6 levels. The higher the value of the Air Pollution Index (AQI), the worse the air quality. The specific health concern levels of AQI are as follows: Figure 5 shown.

[0061] Furthermore, in step II, the historical data in the database is used to divide the training set and the test set. The training set is used to train the self-attention temporal convolutional network model, and the test set is input into the model for regression prediction, and then the error vector is constructed. The error vector is used to fit the Gaussian distribution, and then the mean and covariance are estimated using the maximum likelihood. Finally, the abnormal score is calculated according to the Mahalanobis distance formula to determine the threshold. Finally, the data is marked as normal or abnormal by the threshold.

[0062] Furthermore, in step III, the obtained labeled data set is divided into a training set and a test set. The training set is used to train the decision tree PART model, and the test set is used to determine the classification results. Finally, event rules are extracted based on the classification results.

[0063] Furthermore, in steps IV and V, Flink CEP converts the rule set extracted by the decision tree algorithm into complex event pattern rules. After that, Maxwell will capture the data stream of database changes in real time and send it to the Kafka topic partition. Flink reads the data stream of the topic partition in Kafka in real time. Once the qualified data stream is detected, Flink CEP will identify these data streams and generate complex events.

[0064] First, the historical data set in the database will be used to train the model, which consists of a self-attention module and a time-domain convolutional network module. After the deep learning model is trained, the model is used to construct an error vector, which is then used to calculate the anomaly score using the multivariate Gaussian function and the Mahalanobis distance formula to label the data set.

[0065] The error vector will fit a multivariate Gaussian distribution, and the multivariate Gaussian distribution function formula is as follows:

[0066]

[0067] Where μ is the mean of the error vector, ∑ is the covariance matrix, ∑ -1 is the inverse matrix of the covariance matrix. The estimated value of μ can be obtained using the Maximum Likelihood Estimation (MLE) Estimated value of ∑ The parameters obtained by the maximum likelihood estimation method and They are equal to μ and ∑ of our sample.

[0068] The obtained μ and ∑ are used to calculate the Mahalanobis distance, which represents the distance of the sample from the center point. The Mahalanobis distance formula is as follows:

[0069] a t =(e t -μ) T ∑ -1 (e t -μ)

[0070] Among them, e t is the error vector of the sample point, a t is the distance scalar of the sample point from the center point. Set the threshold ε and confidence interval, the a t The value of is in the confidence interval [0,ε] for 95% and from ε to infinity for 10%;

[0071] when a t >ε when the data is marked as abnormal data;

[0072] when a t When ≤ε, the data is marked as normal data.

[0073] The labeled data set is divided into a training set and a test set. The training set is used for classification training of the decision tree PART model, and the test set is used to evaluate the classification results of the model. A set of rules from the root node to the leaf node will be obtained. The evaluation indicators include accuracy, recall rate, F1-score and ROC area.

[0074] Maxwell needs to specify a series of parameters through the configuration file to capture database change data in real time, including data producer type (producer), Kafka cluster address (kafka.bootstrap.servers), database server address (host), database user name (user), password (password) and Kafka topic name (kafka_topic) to ensure that it can connect correctly and send change data.

[0075] The real-time early warning method for air pollution based on intelligent algorithms and Flink provided in this embodiment realizes effective monitoring and rapid response to air pollution through steps such as sensor acquisition, data preprocessing, intelligent algorithm analysis, rule extraction and real-time early warning. By using Maxwell to capture database changes in real time and combining Flink's stream processing capabilities, real-time monitoring and early warning of air quality data are realized, shortening the time from data collection to early warning issuance, and responding to air pollution incidents more promptly. Using the self-attention time-domain convolutional network model for anomaly detection can better capture complex patterns and nonlinear relationships in the data, improve the accuracy of abnormal event identification, and reduce false alarms and missed reports. The early warning rules are automatically extracted through the decision tree algorithm PART, avoiding the subjectivity and cumbersomeness of manual rule making, and improving the efficiency and accuracy of rule making.

[0076] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a restriction. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in the subsequent accompanying drawings.

[0077] In addition, it should be noted that the use of terms such as "first" and "second" to limit is only for the convenience of distinction. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of this application.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time early warning method for air pollution based on intelligent algorithm and Flink, characterized in that: include: Collect polluted gas data and store them in MySQL database after preprocessing; The self-attention time-domain convolutional network model is used to label the unlabeled historical data in the MySQL database to obtain a labeled dataset; Using a decision tree algorithm PART to automatically extract rules from the labeled data set to obtain a rule set; Convert the rule set into complex event pattern rules through Flink CEP; Maxwell is used to capture data changes in the MySQL database in real time and synchronize it to the topic partition of Kafka; Flink is used to consume the data stream of topic partitions in Kafka in real time, and Flink CEP is used to match predefined complex event pattern rules. When the match is successful, a warning message of the air pollution level is generated and displayed on the console in real time.

2. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 1 is characterized in that: The collecting of polluted gas data and storing it in the MySQL database after preprocessing specifically includes: The polluted gas data is collected every five minutes through sensors; Preprocess the polluted gas data, calculate the air pollution index value AQI and store it in the MySQL database; The pollutant gas data includes ozone, particulate matter, carbon monoxide, sulfur dioxide and nitrogen dioxide.

3. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 1 is characterized in that: The method of labeling unlabeled historical data using the self-attention time-domain convolutional network model specifically includes: Constructing an error vector using the self-attention temporal convolutional network model; Anomaly scores are calculated based on the error vector, multivariate Gaussian function, and Mahalanobis distance formula, and the data set is labeled.

4. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 3 is characterized in that: The self-attention time-domain convolutional network model includes a self-attention module and a time-domain convolutional network module, and the self-attention time-domain convolutional network model is trained using historical data sets in a MySQL database.

5. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 4 is characterized in that: The self-attention temporal convolutional network includes an input layer, a linear layer, a self-attention temporal convolutional layer and an output layer; The self-attention temporal convolution layer comprises a self-attention layer, a normalization layer and a temporal convolution layer; The temporal convolution layer includes a causal convolution 1D layer, a weight normalization layer, an activation layer and a Dropout layer.

6. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 3 is characterized in that: The calculating of the anomaly score based on the error vector, the multivariate Gaussian function and the Mahalanobis distance formula and marking the data specifically includes: The multivariate Gaussian distribution is fitted by the error vector, and the multivariate Gaussian distribution function formula is as follows: Where η is the mean of the error vector, ∑ is the covariance matrix, ∑ -1 is the inverse matrix of the covariance matrix; Using maximum likelihood estimation to obtain the estimated value of μ Estimated value of ∑ The parameters obtained by the maximum likelihood estimation method and The Mahalanobis distance is calculated using the obtained μ and ∑. The Mahalanobis distance represents the distance of the sample from the center point. The Mahalanobis distance formula is as follows: to t =(and t -μ) T ∑ -1 (and t -μ) Among them, e t is the error vector of the sample point, a t is the distance scalar of the sample point from the center point; Set the threshold ε and confidence interval, the a t The value of is in the confidence interval [0,ε] for 95% and from ε to infinity for 10%; when a t >ε when the data is marked as abnormal data; when a t When ≤ε, the data is marked as normal data.

7. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 1 is characterized in that: After obtaining the labeled data set, the method further includes: The labeled data set is divided into a training set and a test set. The training set is used for classification training of the decision tree PART model, and the test set is used to evaluate the classification results of the model. The evaluation indicators of the evaluation model include accuracy, recall rate, F1-score and ROC area.

8. The real-time early warning method for air pollution based on intelligent algorithm and Flink according to claim 1 is characterized in that: The Maxwell configuration file includes the data producer type, Kafka cluster address, Kafka topic name, MySQL database server address, and MySQL database user name and password.

9. A real-time air pollution warning system based on intelligent algorithms and Flink, characterized in that: The real-time early warning method for air pollution based on intelligent algorithm and Flink as described in any one of claims 1 to 8 includes a collection module for collecting polluted gas data, a data storage module, a data processing module, and a data monitoring and alarm module.

Images