An environment and rating system based on big data analysis

Abstract

The present invention belongs to the field of air quality detection application technology, and in particular, relates to an environment and rating system based on big data analysis, including obtaining air quality levels, air quality evaluation parameter indicators and corresponding sky image data sets, and standardizing them. Based on the improved principal component analysis algorithm, feature extraction is performed on the air quality evaluation parameter indicators, a stacked noise reduction autoencoder is used to learn the evaluation parameter indicator characteristics and sky image characteristics of air quality samples, and a restricted Boltzmann machine is used to analyze and model the relationship between various features to obtain fused features, and a multi-feature fused air quality rating LSTM model is established based on the fused features to quickly and efficiently analyze and rate air quality in real time. The method proposed in the present invention fully takes into account the influence of various influencing factors on the air quality level, and integrates the sky image features to rate the air quality in real time, reliably and efficiently.

Description

Technical Field

[0001] The present invention belongs to the technical field of air quality detection applications, and in particular relates to an environment and rating system based on big data analysis. Background Art

[0002] Existing air quality rating methods focus on studying the relationship between a single factor and air quality level. Air quality rating is a complex dynamic process. Geographical features, climate characteristics, and pollutant emissions will all affect air quality ratings. An air quality rating model built using only a single factor cannot solve the problem of feature sharing between various types of data, and has no significant effect on improving the effectiveness, convenience, and real-time performance of detection. Summary of the invention

[0003] In view of the existing technical problems, the present invention proposes an environment and rating system based on big data analysis, which has a reasonable design, simple method, strong theoretical basis, and can clearly define evaluation parameter indicators with high correlation with air quality levels, and solve the problems of difficulty in real-time analysis and great difficulty in practical application.

[0004] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: an environment and rating system based on big data analysis, the system is used to perform the following steps:

[0005] S1. Obtain the data set of air quality level, air quality evaluation parameter index and sky image of the target area through the data information system;

[0006] S2. Standardization: standardize the air quality evaluation parameters to eliminate differences in the parameters.

[0007] The dimension and magnitude of the standard interval are affected by the standard formula:

[0008]

[0009] Among them, X ij is the original value of the jth air quality evaluation parameter index of the ith existing air quality sample, X′ ij is the standardized value of the jth air quality evaluation parameter index of the i-th air quality sample, and S j are the mean and standard deviation of the jth air quality evaluation parameter index;

[0010] S3, feature extraction, based on the improved principal component analysis algorithm, feature extraction is performed on the standardized existing air quality evaluation parameter indicators to obtain two sets of feature variables. The improved principal component analysis algorithm is described as:

[0011] S31. Conduct correlation analysis on the air quality evaluation parameter indicators. Analyze the air quality evaluation parameter indicators based on the Pearson correlation coefficient method and determine the correlation between the indicators. The correlation coefficient is used for measurement. The calculation formula is:

[0012]

[0013] Where cov(X,Y) is the covariance between any two evaluation parameter indicators X and Y, σ X Expressed as the standard deviation of the air quality evaluation parameter index X, σ Y Expressed as the standard deviation of air quality evaluation parameter Y, μ X and μ Y are the means of X and Y, respectively, XY The value range of is -1 to 1. The larger the absolute value of the correlation coefficient is, the stronger the correlation between the two indicators is. The indicators with poor correlation are linearized.

[0014] S32. Calculate the characteristic weights of all air quality evaluation parameter indicators, and the calculation formula is:

[0015]

[0016] Among them, p ij is the characteristic weight of the jth air quality evaluation parameter index of the ith air quality sample, and m is the total number of air quality samples;

[0017] S33 calculates the weight of the air quality evaluation parameter index according to the characteristic proportion of each air quality evaluation parameter index, and the calculation formula is:

[0018]

[0019] Among them, e j is the entropy value corresponding to the j-th air quality evaluation parameter index, n represents the total number of air quality evaluation parameter indicators, W j is the weight of the j-th air quality evaluation parameter index;

[0020] S34. Based on the weights of various air quality evaluation parameter indicators and combined with local impact normalization, the original data is centered. The calculation formula is:

[0021]

[0022] in, is the mean of all air quality evaluation parameter indicators, X' is the set of air quality sample values ​​after centering, X j is the value of the jth air quality evaluation parameter index;

[0023] S35, calculating and solving the eigenvalues ​​and eigenvectors of the covariance matrix, the first K eigenvalues ​​after descending order are the K principal components after principal component analysis;

[0024] S4, feature fusion, performing feature layer fusion on the two sets of feature variables obtained by feature extraction and the image features extracted from the sky image to obtain fused feature data, and dividing the feature data into training set data and test set data;

[0025] S5. Build an air quality rating LSTM model with multi-feature fusion, train the LSTM model with training set data, test the accuracy of the LSTM model with test set data, and obtain an air quality rating LSTM model with multi-feature fusion;

[0026] S6. Collect air quality evaluation parameter indicators and sky images of air quality samples of the area to be evaluated and pre-process them with existing air quality samples, including standardization, feature extraction and feature fusion;

[0027] S7. Import the processed data into the multi-feature fused air quality rating LSTM model to obtain the air quality level of the area to be evaluated.

[0028] Preferably, the air quality parameter indicators of step S1 include: SO2, NO2, CO, O3, PM2.5, PM10, and the air quality levels include: level one, level two, level three, level four, level five and level six.

[0029] Preferably, the number of the two groups of characteristic variables in step S3 is respectively smaller than the number of evaluation parameter indicators related to the existing air quality samples.

[0030] Preferably, the improved principal component analysis algorithm of step S3 analyzes the correlation between the air quality evaluation parameter indicators before data processing based on the principal component analysis algorithm, and performs linear processing on the air quality evaluation parameter indicators with poor correlation; when centralizing the data, the parameter weights of the various air quality evaluation parameter indicators are comprehensively considered.

[0031] Preferably, the LSTM model of step S5 is fused with multiple layers of LSTM, and the error back propagation method is used to adjust the parameters of the multi-layer LSTM model, so as to obtain a stable and convergent multi-layer LSTM model.

[0032] Compared with the prior art, the advantages and positive effects of the present invention are:

[0033] 1. The improved principal component analysis algorithm proposed in the present invention takes into account the correlation between various evaluation parameter indicators on the basis of the principal component analysis algorithm, analyzes the correlation between various air quality evaluation parameter indicators before data processing, and performs linear processing on indicators with poor correlation; when centralizing the data, the weights of various air quality evaluation parameter indicators are comprehensively considered, while the local influence of a single air quality evaluation parameter indicator is considered, and the local influence is normalized, which alleviates the overfitting of the model to a certain extent; greatly improves the interpretability of various features of the principal component, and ensures the maximization and optimization of the amount of retained information while avoiding a large amount of data.

[0034] 2. The feature fusion method adopted by the present invention uses a sky image taken in real time by a mobile phone, uses a stacked denoising autoencoder to learn the air quality evaluation parameter indicators and sky image features of the air quality samples, and uses a restricted Boltzmann machine to model their relationship to obtain the fused features; subsequently, it is only necessary to take the sky image of the target area in real time and combine it with the air quality evaluation parameter indicators, and establish a multi-feature fused air quality rating LSTM model based on the fused features, so as to evaluate the air quality levels and their dynamic changes at different times and locations simply, in real time, effectively and dynamically. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A model structure diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0039] Embodiment, considering that the existing air quality rating method focuses on studying the relationship between a single factor and the air quality grade. The air quality grade is a complex dynamic process, and the geographical features, climate characteristics and pollutant emissions will affect the air quality rating. The air quality rating model established using only a single factor cannot solve the feature sharing between various types of data well, and has no significant effect on improving the effectiveness, convenience and real-time performance of detection. To this end, the present invention provides an environment and rating system based on big data analysis. First, the air quality grade, air quality evaluation parameter index and sky image data set of the target area are obtained through a data information system. According to the "Technical Regulations for Environmental Air Quality Index (AQI) (Trial)", the evaluation parameter index includes: SO2, NO2, CO, O3, PM2.5, PM10, and the air quality grade includes: Level 1, Level 2, Level 3, Level 4, Level 5 and Level 6.

[0040] In order to eliminate the impact of dimensions and magnitudes between different air quality evaluation parameter indicators, the air quality evaluation parameter indicators are first standardized. The standardized formula is:

[0041]

[0042] Among them, X ij is the original value of the jth air quality evaluation parameter index of the ith existing air quality sample, X′ ij is the standardized value of the jth air quality evaluation parameter index of the i-th air quality sample, and S j are the mean and standard deviation of the jth air quality evaluation parameter index;

[0043] There are many air quality evaluation parameter indicators, and the regularity is not strong. It is necessary to extract the difference information of air quality of different levels from the complex data. Principal component analysis maps high-dimensional data to low-dimensional space through linear changes, and these low-dimensional data can retain the information of the original data as much as possible. Principal component analysis is a "linear" dimensionality reduction technology, and the degree of correlation between variables will affect the dimensionality reduction effect. When the traditional principal component analysis algorithm directly calculates the correlation coefficient of the standardized variables, if the correlation between the variables is not high, the dimensionality reduction effect of the principal component analysis will not be obvious. At the same time, the traditional principal component analysis algorithm directly centers the standardized data when performing feature analysis, without considering the weight of each feature and the local influence of a single feature. The weight distribution between each feature and the local influence of a single feature are important factors that affect the evaluation of air quality levels and prevent the model from overfitting. The present invention cites Pearson correlation analysis and local response normalization to improve the traditional principal component analysis algorithm to ensure the maximization and optimization of the amount of information retained. Before principal component analysis, the Pearson correlation coefficient method was introduced to analyze the air quality evaluation parameter indicators and determine the degree of correlation between them, which was measured by the correlation coefficient. The calculation formula is:

[0044]

[0045] Where cov(X,Y) is the covariance between any two air quality evaluation parameter indicators X and Y, σ X Expressed as the standard deviation of the air quality evaluation parameter index X, σ Y Expressed as the standard deviation of air quality evaluation parameter Y, μ X and μ Y are the means of X and Y respectively. XY The value range of is -1 to 1. The larger the absolute value of the correlation coefficient is, the stronger the correlation between the two indicators is. The indicators with poor correlation are linearized.

[0046] Weighted normalization is a commonly used data fusion technology. It weights the features of multiple data sources. The importance of each feature needs to be considered during processing, and higher weights are given to features with high importance, thereby improving the prediction accuracy and stability of the model. It is widely used in various engineering scenarios.

[0047] The traditional principal component analysis algorithm centralizes the original data in a centralized data set. Since there is no artificial setting of parameters to ensure that the results are relevant to the data, the data in the data set are easily affected by extreme values. After dimensionality reduction, the explanatory power of each feature of the principal component is reduced, and the unselected principal components may also contain important information about sample differences, resulting in incomplete data. The present invention introduces weighted normalization to weight and normalize the data before centralization. First, the characteristic weights of all evaluation parameter indicators are calculated, and the calculation formula is:

[0048]

[0049] Among them, p ij is the characteristic weight of the jth air quality evaluation parameter index of the ith air quality sample, and m is the total number of air quality samples. The entropy value and weight of the air quality evaluation parameter index are calculated based on the characteristic weight of each air quality evaluation parameter index. The calculation formula is:

[0050]

[0051]

[0052] Among them, e j is the entropy value corresponding to the jth air quality evaluation parameter index, W j is the weight of the jth air quality evaluation parameter index, and n is the total number of air quality evaluation parameter indexes. The original data is centered according to the weight of each air quality evaluation parameter index and the local response normalization. The calculation formula is:

[0053]

[0054] in, is the mean of all air quality evaluation parameter indicators. X' is the set of air quality sample values ​​after centering. j is the value of the jth air quality evaluation parameter index; calculate the covariance matrix of the air quality evaluation parameter index:

[0055]

[0056] Finally, by solving the eigenvalues ​​and eigenvectors of the covariance matrix, the first K eigenvalues ​​after descending order are the K principal components after principal component analysis.

[0057] At present, mainstream air quality rating models are generally based on machine learning, supervised learning and other technologies. These methods extract features from air quality evaluation parameter indicators as representation vectors of air quality levels, and use air quality level labels as labeled data to train air quality rating models. There is a technology that establishes a database by acquiring satellite remote sensing images and environmental data of the target area, but remote sensing images are relatively complex, data processing is relatively cumbersome, and acquisition is relatively difficult. These methods do not fully consider aspects such as timeliness, dynamics, and integration of image information. In response to the deficiencies of the existing prior art, the present invention only needs to acquire the sky image of the target area on the basis of acquiring the base of the air quality evaluation parameter indicators of the target area, and fuse the image features with the air quality evaluation parameter indicator features, and proposes a multi-feature fused air quality rating LSTM model that can reliably, quickly, and with sufficient accuracy predict the current air quality level. Figure 1 As shown, the model first uses a stacked denoising autoencoder to learn the expression of air quality evaluation parameter index features and sky image features in the original data, and then uses a restricted Boltzmann machine to analyze the relationship between each feature and use their relationship to model, and obtain the fused data. Finally, the fused data is imported into the multi-feature fused air quality rating LSTM model. Unlike the traditional LSTM model, the LSTM model in the present invention takes the basic information of multiple air quality grade samples to be evaluated; then a multi-layer LSTM model is constructed, and the labeled training set data is input into the multi-layer LSTM model, and the minimum square error is used as the cost function to calculate the error between the result output by the multi-layer LSTM model and the labeled result, and the error back propagation method is used to adjust the parameters of the multi-layer LSTM model, thereby obtaining a stable and convergent multi-layer LSTM model; finally, for the unevaluated data, it is pre-processed and input into the multi-layer LSTM model to obtain the classification result, and finally an environment and rating system based on big data analysis is obtained.

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An environment and rating system based on big data analysis, characterized in that: The system is used to perform the following steps: S1. Obtain the data set of air quality level, air quality evaluation parameter index and sky image of the target area through the data information system; S2. Standardization: standardize the air quality evaluation parameter indicators to eliminate the impact of dimensions and magnitudes between different indicators. The standardization formula is: Among them, X ij is the original value of the jth air quality evaluation parameter index of the ith existing air quality sample, X′ ij is the standardized value of the jth air quality evaluation parameter index of the i-th air quality sample, and S j are the mean and standard deviation of the jth air quality evaluation parameter index; S3, feature extraction, based on the improved principal component analysis algorithm, feature extraction is performed on the standardized existing air quality evaluation parameter indicators to obtain two sets of feature variables. The improved principal component analysis algorithm is described as: S31. Conduct correlation analysis on the air quality evaluation parameter indicators. Analyze the air quality evaluation parameter indicators based on the Pearson correlation coefficient method and determine the correlation between the indicators. The correlation coefficient is used for measurement. The calculation formula is: Where cov(X,Y) is the covariance between any two air quality evaluation parameter indicators X and Y, σ X Expressed as the standard deviation of the air quality evaluation parameter index X, σ Y Expressed as the standard deviation of air quality evaluation parameter Y, μ X and μ Y are the means of X and Y respectively; ρ XY The value range of is -1 to 1. The larger the absolute value of the correlation coefficient is, the stronger the correlation between the two indicators is. The indicators with poor correlation are linearized. S32. Calculate the characteristic weights of all air quality evaluation parameter indicators, and the calculation formula is: Among them, p ij is the characteristic weight of the jth air quality evaluation parameter index of the ith air quality sample, and m is the total number of air quality samples; S33 calculates the weight of the air quality evaluation parameter index according to the characteristic proportion of each air quality evaluation parameter index, and the calculation formula is: Among them, e j is the entropy value corresponding to the j-th air quality evaluation parameter index, n represents the total number of air quality evaluation parameter indicators, W j is the weight of the j-th air quality evaluation parameter index; S34. Based on the weights of various air quality evaluation parameter indicators and combined with local response normalization, the original data is centered. The calculation formula is: in, is the mean of all air quality evaluation parameter indicators, X' is the set of air quality sample values ​​after centering, X j is the value of the jth air quality evaluation parameter index; S35, calculating and solving the eigenvalues ​​and eigenvectors of the covariance matrix, the first K eigenvalues ​​after descending order are the K principal components after principal component analysis; S4, feature fusion, performing feature layer fusion on the two sets of feature variables obtained by feature extraction and the image features extracted from the sky image to obtain fused feature data, and dividing the feature data into training set data and test set data; S5. Build an air quality rating LSTM model with multi-feature fusion, train the LSTM model with training set data, test the accuracy of the LSTM model with test set data, and obtain an air quality rating LSTM model with multi-feature fusion; S6. Collect air quality evaluation parameter indicators and sky images of air quality samples of the area to be evaluated and pre-process them with existing air quality samples, including standardization, feature extraction and feature fusion; S7. Import the processed data into the multi-feature fused air quality rating LSTM model to obtain the air quality level of the area to be evaluated.

2. The environment and rating system based on big data analysis according to claim 1, characterized in that: The air quality evaluation parameter indicators of step S1 include: SO2, NO2, CO, O3, PM2.5, PM10, and the air quality levels include: level one, level two, level three, level four, level five and level six.

3. The environment and rating system based on big data analysis according to claim 1, characterized in that: The number of the two groups of characteristic variables in step S3 is less than the number of existing air quality evaluation parameter indicators.

4. The environment and rating system based on big data analysis according to claim 1, characterized in that: The feature fusion of step S4 uses a stacked denoising autoencoder to learn the air quality evaluation parameter indicators of air quality samples and the features of sky images, and uses a restricted Boltzmann machine to model their relationship to obtain fused features. Finally, the fused features are used to establish a multi-feature fused air quality rating LSTM model.

Images