Pollutant property analysis method and system based on machine learning

Abstract

The invention discloses a pollutant property analysis method and system based on machine learning, and belongs to the technical field of pollutant analysis. The method comprises the following steps: S1, collecting historical pollutant data of a monitoring point; preprocessing the historical pollutant data to obtain pollutant historical processing data; the historical pollutant data comprises historical pollutant monitoring data, geographic data and meteorological data; s2, performing feature extraction on the historical pollutant processing data to obtain key features; s3, constructing a machine model, and performing property analysis training on the key features by adopting the machine model to obtain a first analysis result; the machine model is optimized based on the first analysis result, and the optimized machine model is obtained; and S4, collecting real-time pollutant data of a monitoring point, and analyzing the real-time pollutant data by adopting the optimized machine model to obtain a second analysis result.

Description

Technical Field

[0001] The present invention belongs to the technical field of pollutant analysis, and specifically relates to a pollutant property analysis method and system based on machine learning. Background Art

[0002] Commonly used pollution prediction methods mainly include Gaussian model calculation method, which requires that the pollutant concentration is normally Gaussian distributed on the Y and Z axes, and that the wind speed is uniform, the source strength of the pollution source is continuous and uniform, and the surface is sufficiently flat. However, due to the different concentrations of reaction substrates, light intensity and time in different places, it is impossible to meet such idealized conditions. Therefore, the Gaussian model calculation method is not very effective in actual pollution prediction. Summary of the invention

[0003] The present invention aims to solve the deficiencies of the prior art and proposes a pollutant property analysis method and system based on machine learning. By analyzing the relationship between pollutants and geographical, meteorological and other factors, the impact of pollutants on air quality is analyzed. By multiple screening of features, key features that have a greater impact on the properties of pollutants are obtained, thereby improving the model prediction accuracy. At the same time, the use of machine learning to analyze the properties of pollutants can not only automatically process large amounts of data, but also more accurately and quickly identify the properties of pollutants, providing data reference for environmental quality management.

[0004] To achieve the above object, the present invention provides the following scheme: a pollutant property analysis method based on machine learning, comprising the following steps:

[0005] S1. Collect historical pollutant data of monitoring points; and pre-process the historical pollutant data to obtain historical pollutant processing data; the historical pollutant data includes: historical pollutant monitoring data, geographical data and meteorological data;

[0006] S2. Extracting features from the historical pollutant treatment data to obtain key features;

[0007] S3, constructing a machine model, using the machine model to perform property analysis training on the key features to obtain a first analysis result; optimizing the machine model based on the first analysis result to obtain an optimized machine model;

[0008] S4. Collect real-time pollutant data of the monitoring point, and use the optimized machine model to analyze the real-time pollutant data to obtain a second analysis result.

[0009] Further preferably, the method for preprocessing the historical pollutant data includes: outlier processing, missing value processing and normalization processing.

[0010] Further preferably, S2 comprises the following steps:

[0011] S21, extracting features from the pollutant historical processing data to obtain parameter features;

[0012] S22. Based on the parameter features, construct a feature matrix; calculate the covariance matrix and eigenvector of the feature matrix;

[0013] S23. Decompose the feature matrix based on the feature vector to obtain the key feature.

[0014] Further preferably, the machine model includes: a periodic model, a time series model and a recognition model;

[0015] The periodic model is used to perform seasonal analysis on the key features to obtain a time series analysis result based on seasonal terms;

[0016] The time series model is used to perform medium- and long-term forecasting on the key features to obtain medium- and long-term forecasting analysis results;

[0017] The recognition model is used to perform forward and reverse propagation on the key features to obtain recognition analysis results.

[0018] The present invention also provides a pollutant property analysis system based on machine learning, comprising: a historical data acquisition module, a feature extraction module, an optimization module and an analysis module;

[0019] The historical data collection module is used to collect historical pollutant data of monitoring points; and pre-process the historical pollutant data to obtain historical pollutant processing data; the historical pollutant data includes: historical pollutant monitoring data, geographical data and meteorological data;

[0020] The feature extraction module is used to extract features from the pollutant historical processing data to obtain key features;

[0021] The optimization module is used to construct a machine model, use the machine model to perform property analysis training on the key features to obtain a first analysis result; optimize the machine model based on the first analysis result to obtain the optimized machine model;

[0022] The analysis module is used to analyze the collected real-time pollutant data of the monitoring points using the optimized machine model to obtain a second analysis result.

[0023] Further preferably, the process of preprocessing the historical pollutant data by the historical data acquisition module includes: outlier processing, missing value processing and normalization processing.

[0024] Further preferably, the feature extraction module includes: an extraction unit, a calculation unit and a decomposition unit;

[0025] The extraction unit is used to extract features from the pollutant historical processing data to obtain parameter features;

[0026] The calculation unit is used to construct a feature matrix based on the parameter features; calculate the covariance matrix and the eigenvector of the feature matrix;

[0027] The decomposition unit is used to decompose the feature matrix based on the feature vector to obtain the key feature.

[0028] Further preferably, the machine model includes: a periodic model, a time series model and a recognition model;

[0029] The periodic model is used to perform seasonal analysis on the key features to obtain a time series analysis result based on seasonal terms;

[0030] The time series model is used to perform medium- and long-term forecasting on the key features to obtain medium- and long-term forecasting analysis results;

[0031] The recognition model is used to perform forward and reverse propagation on the key features to obtain recognition analysis results.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention analyzes the impact of pollutants on air quality by analyzing the relationship between pollutants and geographical, meteorological and other factors. By performing multiple screening of features, key features that have a greater impact on the properties of pollutants are obtained, thereby improving the model prediction accuracy. At the same time, the use of machine learning to analyze the properties of pollutants can not only automatically process large amounts of data, but also more accurately and quickly identify the properties of pollutants, providing data reference for environmental quality management. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 Schematic diagram of the process of the pollutant property analysis method based on machine learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, this embodiment provides a pollutant property analysis method based on machine learning, comprising the following steps:

[0040] S1. Collect historical pollutant data of monitoring points; and pre-process the historical pollutant data to obtain historical pollutant processing data; the historical pollutant data includes: historical pollutant monitoring data, geographical data and meteorological data.

[0041] In this embodiment, the method for preprocessing historical pollutant data includes: outlier processing, missing value processing and normalization processing.

[0042] In this embodiment, the box plot method is used to analyze the outliers of the collected historical pollutant data, and the values ​​less than Q1-1.5IQR or greater than Q3+1.5IQR are defined as outliers. At the same time, the data analyzed as outliers are regarded as missing values. The moving average method is used to take the average value of several adjacent items before and after the missing data as the missing value, which can eliminate the ups and downs of the original data series caused by accidental factors without changing the rules of the original data series. Afterwards, the z-score normalization method is used to normalize the data for data with different dimensions, so that the normalized data distribution satisfies the mean of 0 and the variance of 1.

[0043] Subsequently, the accuracy of the model was evaluated by using historical pollutant data after preprocessing and historical pollutant data without preprocessing. The model using the preprocessed historical pollutant data had the highest accuracy, the predicted values ​​and actual values ​​fit well, and had good generalization ability.

[0044] S2. Extract features from historical pollutant treatment data to obtain key features.

[0045] In this embodiment, S2 includes the following steps:

[0046] S21. Extract features from historical pollutant treatment data to obtain parameter features.

[0047] First, the random forest algorithm is used to analyze the historical pollutant treatment data. From the many collected parameters, the features that have a greater impact on the analysis results are selected, namely the parameter features, thereby reducing the number of features and the complexity of calculation.

[0048] The process of obtaining parameter characteristics includes:

[0049] Select the corresponding out-of-bag data for each decision tree and calculate the out-of-bag data error once; randomly add noise interference to the features of all samples of the out-of-bag data and calculate the out-of-bag data error again. For a random forest with N trees, the importance of its features is the average difference between the two out-of-bag data errors; sort the importance of each feature in descending order, determine the proportion that needs to be eliminated this time, and obtain the parameter feature set.

[0050] S22. Construct a feature matrix based on the parameter features; calculate the covariance matrix and eigenvector of the feature matrix.

[0051] The parameter features and parameter feature data after feature extraction are constructed into a feature matrix. Then the covariance matrix, eigenvalues ​​and corresponding eigenvectors are calculated. Specifically, the eigenvalues ​​are arranged in descending order and the eigenvectors corresponding to the eigenvalues ​​are obtained.

[0052] S23. Perform singular value decomposition on the feature matrix based on the eigenvector to obtain key features.

[0053] The number of key features is determined by the cumulative variance contribution rate method, which is based on the proportion of the cumulative sum of the selected key feature variances in the comprehensive variance of all parameter features to determine the number of key features. In this embodiment, more than 85% is selected.

[0054] S3. Build a machine model, use the machine model to perform property analysis training on key features, and obtain a first analysis result; optimize the machine model based on the first analysis result to obtain an optimized machine model.

[0055] In this embodiment, the machine model includes: a periodic model, a time series model and an identification model; the periodic model is used to perform seasonal analysis on key features to obtain time series analysis results based on seasonal terms; the time series model is used to perform medium- and long-term predictions on key features to obtain medium- and long-term prediction analysis results; the identification model is used to perform forward and reverse propagation on key features to obtain identification analysis results. The machine model uses historical pollutant monitoring data, geographic data and meteorological data as input to predict air quality.

[0056] Specifically, the seasonal term data is considered in the periodic model, which can reflect the seasonal trend. The expression of the model is as follows:

[0057]

[0058] In the formula, Φ(L), Φ s (L), Θ(L), and Θ s (L) is an expression related to parameter L; Differentiation operation to indicate trend; Represents seasonal differencing.

[0059] The time series model includes: cell state and input gate, forget gate and output gate; among them, the forget gate is used to calculate the network input x at time t. t and the network output h at time t-1 t-1 The information of cell status is retained and deleted. The calculation methods include:

[0060] f t =σ(W f *[h t-1 , x t ])+b f ,

[0061] In the formula, f t represents the output of the forget gate at time t; W f represents the weight matrix of the forget gate; b f represents the bias matrix of the forget gate, and σ() represents the sigmoid activation function.

[0062] The input gate is used to decide what information to add to the cell state based on the network input at time t and the network output at time t-1. First, the sigmoid neural network layer is used to decide what information to update; then the tanh neural network layer is used to create a new candidate cell state and update the information of the candidate cell state to the new cell state.

[0063] The output gate is used to determine which information of the cell state to output based on the network input at time t and the network output at time t-1. First, the sigmiod neural network layer is used to determine the part that needs to be output; then the updated cell state is sorted into the [-1,1] interval through the tanh neural network layer. The output of the sigmiod neural network layer is multiplied with the output of the tanh neural network layer to obtain the output of the time series model.

[0064] The recognition model uses BP neural network, which includes forward propagation and back propagation. In the forward propagation process, the input data is processed layer by layer from the input layer to the hidden layer, and then transmitted to the output layer to obtain the output. The state of each layer of neurons only affects the state of the neurons in the next layer. When the actual output does not match the expected output, the back propagation process is performed. In the back propagation process, the error between the actual output and the expected output is propagated from the output layer to the input layer through each hidden layer layer by layer, and the error is apportioned to all neurons in each layer. By adjusting the connection weights and thresholds between neurons in each layer of the neural network, the error is continuously reduced. After repeated training and learning, the weights and thresholds are continuously adjusted to minimize the mean square error, so that the actual output continues to approach the expected output.

[0065] After that, the weight of each model is calculated, and the calculation method includes:

[0066]

[0067] In the formula, ni represents the number of times the true prediction effect of the i-th model is better than that of other models; Indicates the total number of predictions.

[0068] Multiply the prediction value of each model by its corresponding weight and add them up to get the prediction value of the final machine model.

[0069] S4. Collect real-time pollutant data at the monitoring point, and use the optimized machine model to analyze the real-time pollutant data to obtain a second analysis result.

[0070] Embodiment 2:

[0071] This embodiment provides a pollutant property analysis system based on machine learning, including: a historical data acquisition module, a feature extraction module, an optimization module and an analysis module.

[0072] The historical data collection module is used to collect historical pollutant data of monitoring points; and pre-process the historical pollutant data to obtain historical pollutant processing data; the historical pollutant data includes: historical pollutant monitoring data, geographical data and meteorological data.

[0073] Furthermore, the process of preprocessing the historical pollutant data by the historical data collection module includes: outlier processing, missing value processing and normalization processing.

[0074] In this embodiment, the box plot method is used to analyze the outliers of the collected historical pollutant data, and the values ​​less than Q1-1.5IQR or greater than Q3+1.5IQR are defined as outliers. At the same time, the data analyzed as outliers are regarded as missing values. The moving average method is used to take the average value of several adjacent items before and after the missing data as the missing value, which can eliminate the ups and downs of the original data series caused by accidental factors without changing the rules of the original data series. Afterwards, the z-score normalization method is used to normalize the data for data with different dimensions, so that the normalized data distribution satisfies the mean of 0 and the variance of 1.

[0075] The feature extraction module is used to extract features from historical pollutant treatment data to obtain key features.

[0076] Furthermore, the feature extraction module includes: an extraction unit, a calculation unit and a decomposition unit.

[0077] The extraction unit is used to extract features from historical pollutant treatment data to obtain parameter features.

[0078] First, the random forest algorithm is used to analyze the historical pollutant treatment data. From the many collected parameters, the features that have a greater impact on the analysis results are selected, namely the parameter features, thereby reducing the number of features and the complexity of calculation.

[0079] The process of obtaining parameter characteristics includes:

[0080] Select the corresponding out-of-bag data for each decision tree and calculate the out-of-bag data error once; randomly add noise interference to the features of all samples of the out-of-bag data and calculate the out-of-bag data error again. For a random forest with N trees, the importance of its features is the average difference between the two out-of-bag data errors; sort the importance of each feature in descending order, determine the proportion that needs to be eliminated this time, and obtain the parameter feature set.

[0081] The calculation unit is used to construct a feature matrix based on parameter features; and calculate the covariance matrix and eigenvector of the feature matrix.

[0082] The parameter features and parameter feature data after feature extraction are constructed into a feature matrix. Then the covariance matrix, eigenvalues ​​and corresponding eigenvectors are calculated. Specifically, the eigenvalues ​​are arranged in descending order and the eigenvectors corresponding to the eigenvalues ​​are obtained.

[0083] The decomposition unit is used to decompose the feature matrix based on the feature vector to obtain key features.

[0084] The number of key features is determined by the cumulative variance contribution rate method, which is based on the proportion of the cumulative sum of the selected key feature variances in the comprehensive variance of all parameter features to determine the number of key features. In this embodiment, more than 85% is selected.

[0085] The optimization module is used to build a machine model, use the machine model to perform property analysis training on key features, and obtain a first analysis result; based on the first analysis result, the machine model is optimized to obtain an optimized machine model.

[0086] In this embodiment, the machine model includes: a periodic model, a time series model and a recognition model; the periodic model is used to perform seasonal analysis on key features to obtain time series analysis results based on seasonal terms; the time series model is used to perform medium- and long-term predictions on key features to obtain medium- and long-term prediction analysis results; the recognition model is used to perform forward and backward propagation on key features to obtain recognition analysis results.

[0087] The machine model uses historical pollutant monitoring data and geographic data as input to predict meteorological data.

[0088] Specifically, the seasonal term data is considered in the periodic model, which can reflect the seasonal trend. The expression of the model is as follows:

[0089]

[0090] In the formula, Φ(L), Φ s (L), Θ(L), and Θ s (L) is an expression related to parameter L; Differentiation operation to indicate trend; Represents seasonal differencing.

[0091] The time series model includes: cell state and input gate, forget gate and output gate; among them, the forget gate is used to calculate the network input x at time t. t and the network output h at time t-1 t-1 The information of cell status is retained and deleted. The calculation methods include:

[0092] f t =σ(W f *[h t-1 x t )+b f ,

[0093] In the formula, f t represents the output of the forget gate at time t; W f represents the weight matrix of the forget gate; b f represents the bias matrix of the forget gate, and σ() represents the sigmoid activation function.

[0094] The input gate is used to decide what information to add to the cell state based on the network input at time t and the network output at time t-1. First, the sigmoid neural network layer is used to decide what information to update; then the tanh neural network layer is used to create a new candidate cell state and update the information of the candidate cell state to the new cell state.

[0095] The output gate is used to determine which information of the cell state to output based on the network input at time t and the network output at time t-1. First, the sigmiod neural network layer is used to determine the part that needs to be output; then the updated cell state is sorted into the [-1,1] interval through the tanh neural network layer. The output of the sigmiod neural network layer is multiplied with the output of the tanh neural network layer to obtain the output of the time series model.

[0096] The recognition model uses BP neural network, which includes forward propagation and back propagation. In the forward propagation process, the input data is processed layer by layer from the input layer to the hidden layer, and then transmitted to the output layer to obtain the output. The state of each layer of neurons only affects the state of the neurons in the next layer. When the actual output does not match the expected output, the back propagation process is performed. In the back propagation process, the error between the actual output and the expected output is propagated from the output layer to the input layer through each hidden layer layer by layer, and the error is apportioned to all neurons in each layer. By adjusting the connection weights and thresholds between neurons in each layer of the neural network, the error is continuously reduced. After repeated training and learning, the weights and thresholds are continuously adjusted to minimize the mean square error, so that the actual output continues to approach the expected output.

[0097] After that, the weight of each model is calculated, and the calculation method includes:

[0098]

[0099] In the formula, ni represents the number of times the true prediction effect of the i-th model is better than that of other models; Indicates the total number of predictions.

[0100] Multiply the prediction value of each model by its corresponding weight and add them up to get the prediction value of the final machine model.

[0101] The analysis module is used to analyze the real-time pollutant data collected from the monitoring points using an optimized machine model to obtain a second analysis result.

[0102] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A pollutant property analysis method based on machine learning, characterized in that: The following steps are involved: S1. Collect historical pollutant data of monitoring points; and preprocessing the historical pollutant data to obtain historical pollutant processing data; The historical pollutant data include: historical pollutant monitoring data, geographical data and meteorological data; S2. Extracting features from the historical pollutant treatment data to obtain key features; S3, constructing a machine model, using the machine model to perform property analysis training on the key features to obtain a first analysis result; optimizing the machine model based on the first analysis result to obtain an optimized machine model; S4. Collect real-time pollutant data of the monitoring point, and use the optimized machine model to analyze the real-time pollutant data to obtain a second analysis result.

2. The pollutant property analysis method based on machine learning according to claim 1, characterized in that: The method for preprocessing the historical pollutant data includes: outlier processing, missing value processing and normalization processing.

3. The pollutant property analysis method based on machine learning according to claim 1, characterized in that: S2 includes the following steps: S21, extracting features from the pollutant historical processing data to obtain parameter features; S22, constructing a feature matrix based on the parameter features; Calculating the covariance matrix and eigenvectors of the feature matrix; S23. Decompose the feature matrix based on the feature vector to obtain the key feature.

4. The pollutant property analysis method based on machine learning according to claim 1, characterized in that: The machine model includes: a periodic model, a time series model and a recognition model; The periodic model is used to perform seasonal analysis on the key features to obtain a time series analysis result based on seasonal terms; The time series model is used to perform medium- and long-term forecasting on the key features to obtain medium- and long-term forecasting analysis results; The recognition model is used to perform forward and reverse propagation on the key features to obtain recognition analysis results.

5. A pollutant property analysis system based on machine learning, the analysis system is used to implement the analysis method according to any one of claims 1 to 4, characterized in that: include: Historical data collection module, feature extraction module, optimization module and analysis module; The historical data collection module is used to collect historical pollutant data of monitoring points; and preprocessing the historical pollutant data to obtain historical pollutant processing data; The historical pollutant data include: historical pollutant monitoring data, geographical data and meteorological data; The feature extraction module is used to extract features from the pollutant historical processing data to obtain key features; The optimization module is used to construct a machine model, use the machine model to perform property analysis training on the key features to obtain a first analysis result; optimize the machine model based on the first analysis result to obtain the optimized machine model; The analysis module is used to analyze the collected real-time pollutant data of the monitoring points using the optimized machine model to obtain a second analysis result.

6. The pollutant property analysis system based on machine learning according to claim 5, characterized in that: The process of preprocessing the historical pollutant data by the historical data acquisition module includes: outlier processing, missing value processing and normalization processing.

7. The pollutant property analysis system based on machine learning according to claim 5, characterized in that: The feature extraction module includes: an extraction unit, a calculation unit and a decomposition unit; The extraction unit is used to extract features from the pollutant historical processing data to obtain parameter features; The calculation unit is used to construct a feature matrix based on the parameter features; calculate the covariance matrix and the eigenvector of the feature matrix; The decomposition unit is used to decompose the feature matrix based on the feature vector to obtain the key feature.

8. The pollutant property analysis system based on machine learning according to claim 5, characterized in that: The machine model includes: a periodic model, a time series model and a recognition model; The periodic model is used to perform seasonal analysis on the key features to obtain a time series analysis result based on seasonal terms; The time series model is used to perform medium- and long-term forecasting on the key features to obtain medium- and long-term forecasting analysis results; The recognition model is used to perform forward and reverse propagation on the key features to obtain recognition analysis results.

Images