A method, apparatus, device, and medium for evaluating an air quality forecasting model

Abstract

The application relates to a method, device, equipment and medium for evaluating an air quality prediction model, comprising the following steps: acquiring prediction data and measured data output by each air quality prediction model, wherein the prediction data comprises pollutant concentration prediction values and air quality index prediction values, and the measured data comprises pollutant concentration actual values and air quality index actual values; determining a primary pollutant according to the measured data; calculating the values of each model with respect to each first index in a first evaluation index system according to the concentration prediction values and the actual values of the primary pollutant; calculating the values of each model with respect to each second index in a second evaluation index system according to the air quality index prediction values and the actual values; calculating the standard scores of each model with respect to each first index; performing importance evaluation on the first evaluation index system according to the values of the first index and the second index, so as to obtain the weights of each first index; and performing weighted summation on the standard scores and the weights of the first index, so as to obtain the final scores of each model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of air quality prediction, and in particular relates to a method, device, equipment and medium for evaluating an air quality prediction model. BACKGROUND

[0002] In recent years, the field of air quality prediction has developed rapidly. Relevant business departments mainly rely on air quality multi-model prediction business systems to carry out air quality prediction work. Forecasters make manual corrections based on air quality prediction numerical models, statistical models and multi-model ensemble prediction models, on the basis of comprehensive analysis of pollution source emission data, real-time air quality monitoring data and meteorological prediction results, and finally give the air quality prediction results after manual correction.

[0003] However, the prediction results of various air quality prediction models are usually uneven. When the prediction results differ greatly, manual correction based on experience cannot be accurate, which in turn affects the subsequent air quality prediction results. Therefore, there is an urgent need for a method that can accurately evaluate the performance of various air quality prediction models. SUMMARY

[0004] In view of the above analysis, the embodiments of the present application aim to provide a method, device, equipment and medium for evaluating an air quality prediction model, to solve the technical problem of low accuracy of straw burning proportion estimation results in the prior art.

[0005] The purpose of the present application is achieved as follows:

[0006] The first aspect of the present application provides a method for evaluating an air quality prediction model, comprising:

[0007] obtaining prediction data and measured data output by each air quality prediction model, the prediction data including pollutant concentration prediction values and air quality index prediction values, and the measured data including pollutant concentration actual values and air quality index actual values;

[0008] determining a primary pollutant according to the measured data;

[0009] calculating the values of each model with respect to each first index in the first evaluation index system according to the concentration prediction values and actual values of the primary pollutant, and calculating the values of each model with respect to each second index in the second evaluation index system according to the air quality index prediction values and actual values;

[0010] calculating the standard scores of each model with respect to each of the first indexes;

[0011] performing importance evaluation on the first evaluation index system according to the values of the first and second indexes, to obtain the weights of each of the first indexes;

[0012] The standard scores and weights of the first indicators are weighted and summed to obtain the final score of each model.

[0013] Further, the first evaluation index system includes a correlation coefficient, a mean error, a root mean square error, a mean deviation, and a mean absolute error, and the second evaluation index system includes a range prediction accuracy and a level prediction accuracy of an air quality index.

[0014] Further, the method further includes: aligning the prediction data and the measured data in a time scale.

[0015] Further, the method further includes:

[0016] The air quality sub-index of each pollutant is calculated according to the actual value of the concentration of the pollutant, and is expressed as:

[0017]

[0018] wherein IAQI p represents an air quality sub-index of the pollutant p, represents a high value of a concentration limit value of the pollutant p, represents a low value of the concentration limit value of the pollutant p, represents a corresponding air quality sub-index, represents a corresponding air quality sub-index, C p represents a concentration value of the pollutant p in the measured data;

[0019] The pollutant with the highest air quality sub-index is set as the primary pollutant.

[0020] Further, the method further includes:

[0021] The standard score of each model with respect to a first indicator that is a positive indicator is expressed as:

[0022]

[0023] wherein the positive indicator includes a correlation coefficient, represents a standard score of model i with respect to the first indicator j, represents a value of model i with respect to the first indicator j, and respectively represent a maximum value and a minimum value of the first indicator j;

[0024] The standard score of each model with respect to a first indicator that is a negative indicator is expressed as:

[0025]

[0026] The negative indicators include mean error, root mean square error, mean deviation and mean absolute error.

[0027] Further, according to the values of the first indicators and the second indicators, the importance of the first evaluation index system is evaluated to obtain the weights of the first indicators, including:

[0028] A feature vector is constructed according to the values of the first indicators and the second indicators;

[0029] The model prediction performance is rated according to the values of the second indicators, and the rating result is taken as a training label;

[0030] The feature vector and the training label are taken as training samples and input into a random forest classification model for training, and the importance contribution values of the first indicators are output, which are taken as the weights of the first indicators after normalization.

[0031] Further, the standard scores and the weights of the first indicators are weighted and summed to obtain the final scores of the models, which are represented as:

[0032]

[0033] wherein Score i represents the final score of the model i, w j represents the weight of the first indicator j, and n represents the number of the first indicators in the first evaluation index system. i j

[0034] The second aspect embodiment of the present application provides a device for evaluating air quality prediction models, including:

[0035] A data acquisition module is configured to acquire prediction data and measured data output by each air quality prediction model, wherein the prediction data includes pollutant concentration prediction values and air quality index prediction values, and the measured data includes pollutant concentration actual values and air quality index actual values.

[0036] A primary pollutant identification module is configured to determine a primary pollutant according to the measured data.

[0037] An index calculation module is configured to calculate the values of each model with respect to each first indicator in the first evaluation index system according to the concentration prediction values and the actual values of the primary pollutant, and to calculate the values of each model with respect to each second indicator in the second evaluation index system according to the air quality index prediction values and the actual values.

[0038] A standard score calculation module is configured to calculate the standard scores of each model with respect to each first indicator. ​​

[0039] an importance evaluation module configured to perform importance evaluation on the first evaluation index system according to the values of the first index and the second index, and obtain the weight of each of the first indexes;

[0040] a final score module configured to perform weighted summation on the standard scores of the first indexes and the weights to obtain the final scores of the models.

[0041] In a third aspect, an electronic device is provided, which includes a memory and a processor, the memory storing a computer program, and the computer program being executed by the processor to implement the method for evaluating air quality prediction models according to any of the embodiments.

[0042] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method for evaluating air quality prediction models according to any of the embodiments.

[0043] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0044] The method for evaluating air quality prediction models provided by the present application comprehensively considers multiple pollutants and multiple evaluation indexes, and thus accurately evaluates the prediction results, i.e., the model performance, of the multiple air quality prediction models, so as to provide accurate air quality prediction for users in the subsequent process. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0046] Figure 1 a flowchart of the method for evaluating air quality prediction models provided by the present embodiment 1;

[0047] Figure 2 a schematic diagram of the device for evaluating air quality prediction models provided by the present embodiment 2;

[0048] Figure 3An electronic device architecture schematic diagram is provided for the embodiment 3 of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. It should be noted that the embodiments and the features in the embodiments in the present disclosure can be combined, separated, interchanged and / or rearranged without conflict. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0050] Embodiment 1

[0051] As shown in the specific embodiment of the present application, Figure 1 a method for evaluating an air quality prediction model is disclosed, comprising the following steps:

[0052] S1, obtaining prediction data and measured data output by each air quality prediction model, the prediction data comprising pollutant concentration prediction values and air quality index prediction values, and the measured data comprising pollutant concentration actual values and air quality index actual values.

[0053] For example, the air quality prediction model comprises CAMx, CMAQ, NAQPMS, WRF-CHem, LightGBM, LSTM and other commonly used models in the industry. The pollutant concentration data in the prediction data output by these models comprises the concentration average value of carbon monoxide, sulfur dioxide, nitrogen dioxide, ozone, PM 2.5 , PM 10 in a certain time period and the concentration value at each time point; the air quality index AQI is consistent with the time scale of the pollutant concentration data; and the measured data is also consistent with the time scale of the pollutant concentration data.

[0054] In order to ensure the consistency of the time scale, the prediction data and the measured data are also subjected to alignment processing in the time scale, specifically, the original data is resampled according to a set time interval (such as hourly, daily), all model output and measured data are converted to the same time dimension, and the effective sample data set with a common timestamp is screened out by using the timestamp alignment method. For the missing time points, interpolation method or rejection processing can be used to ensure that the calculation of subsequent evaluation indexes is based on a unified time series data basis.

[0055] S2, determining a primary pollutant according to the measured data.

[0056] In this embodiment, step S2 specifically comprises:

[0057] S201, calculating air quality sub-indexes of each pollutant according to actual values of pollutant concentrations, denoted as:

[0058]

[0059] wherein IAQI p denotes an air quality sub-index of the pollutant p, denotes a high value of a concentration limit value of the pollutant p, denotes a low value of the concentration limit value of the pollutant p, denotes a corresponding air quality sub-index, denotes a corresponding air quality sub-index, C p denotes a concentration value of the pollutant p in the measured data; the concentration limit of each pollutant and the corresponding air quality sub-index are obtained from the Ambient Air Quality Standard (GB395-2012).

[0060] S202, setting the pollutant with the highest air quality sub-index as the primary pollutant.

[0061] Specifically, the air quality index AQI is the maximum value of IAQI p , and when the AQI is greater than 50, the primary pollutant is the pollutant p corresponding to the maximum value of IAQI p .

[0062] S3, calculating values of each model with respect to each first index in the first evaluation index system according to the actual primary pollutant and the measured data; and calculating values of each model with respect to each second index in the second evaluation index system according to the air quality index and the measured data.

[0063] In this embodiment, the first evaluation index system comprises correlation coefficient, mean error, root mean square error, mean deviation and mean absolute error, and the second evaluation index system comprises range prediction accuracy and level prediction accuracy of the air quality index.

[0064] Specifically, for each model, the concentration value of the most representative actual primary pollutant in the pollutant concentration data and the corresponding measured data are selected as the basis for calculation, and each first index is calculated, including:

[0065] Supposing that the actual primary pollutant obtained in step S2 is q, the concentration of the pollutant q predicted by the model i at time t is denoted as , the average concentration of the pollutant q predicted by the model i in the period of 1 to m is denoted as , and the average concentration of the pollutant q in the measured data is denoted as The actual value of the concentration of the pollutant q at time t is represented by The actual average value of the concentration of the pollutant q in the period from 1 to m is represented by

[0066] (1) The correlation coefficient is represented by

[0067]

[0068] (2) The average error is represented by

[0069]

[0070] (3) The root mean square error is represented by

[0071]

[0072] (4) The average deviation is represented by

[0073]

[0074] (5) The average absolute error is represented by

[0075]

[0076] Specifically, the range prediction accuracy rate and the level prediction accuracy rate of the air quality index are calculated by the air quality index AQI output by the model and the actual AQI; for the range prediction accuracy rate, when the AQI prediction median is less than or equal to 50 (the air quality is excellent), the median is floated by 10 (the lower limit value should be greater than or equal to 0) up and down, when the AQI prediction median exceeds 50, the median is floated by 20% up and down, to obtain the floated AQI prediction range (all up to the nearest integer), if the actual AQI of the city on the day falls within the floated AQI prediction range, it is recorded as a prediction accuracy, and the range prediction accuracy rate is the number of accurate times divided by the total number of times; for the level prediction accuracy rate, if the actual air quality level of the city on the day falls within the level prediction range corresponding to the floated AQI (the floating standard is the same as the AQI range prediction accuracy rate), it is recorded as a prediction accuracy, otherwise it is a prediction high or low, and the level prediction accuracy rate is the number of accurate times divided by the total number of times.

[0077] S4, calculating the standard scores of each model with respect to each of the first indexes.

[0078] In this embodiment, considering that some index values indicate that the lower the value, the more accurate the model, and some index values indicate that the higher the value, the more accurate the model, the first indexes are divided into positive indexes and negative indexes, the positive indexes include the correlation coefficient, and the negative indexes include the average error, the root mean square error, the average deviation, and the average absolute error, and when calculating the standard scores, the following formulas are used respectively:

[0079] The standard score of each model with respect to the first index is represented as:

[0080]

[0081] wherein, represents the standard score of model i with respect to the first index j, represents the value of model i with respect to the first index j, and respectively represent the maximum value and the minimum value of the first index j;

[0082] The standard score of each model with respect to the first index is represented as:

[0083]

[0084] S5, performing importance evaluation on the first evaluation index system according to the values of the first index and the second index, to obtain the weight of each first index.

[0085] In this embodiment, step S5 comprises:

[0086] S501, constructing a feature vector according to the values of the first index and the second index;

[0087] S502, performing rating on the model prediction performance according to the value of the second index, and taking the rating result as a training label;

[0088] For example, when the AQI range prediction is accurate, the model prediction performance is recorded as “excellent”, when the actual AQI value falls within the range of ±20% to ±40% of the median of the model predicted AQI, the model prediction performance is recorded as “good”, and when the actual AQI value falls outside the range of ±40% of the median of the model predicted AQI, the model prediction performance is recorded as “unqualified”.

[0089] S503, inputting the feature vector and the training label into a random forest classification model as training samples for training, and outputting the importance contribution value of each first index, which, after normalization, is taken as the weight of each first index.

[0090] Specifically, all feature vectors and training labels are normalized, and are divided into a training set and a test set in a ratio of 3:1, the hyperparameters of the random forest are set, and then the training is performed; during the training, the importance of the first index can be evaluated by the average impurity reduction or the average accuracy reduction method, or the evaluation results obtained by the two methods are weighted and fused to form the importance score of the first index, and then the importance score is standardized as the weight.

[0091] S6. The standard scores and weights of the first indicator are weighted and summed to obtain the final scores of each model.

[0092] In this embodiment, step S6 is represented as:

[0093]

[0094] Among them, Score i w represents the final score of model i. j represents the weight of the first indicator j, and n represents the number of first indicators in the first evaluation indicator system.

[0095] In some embodiments, the weights of the first indicator can be sorted, and the top 80% of the indicators can be selected to calculate the final score, in order to save computing resources.

[0096] In some embodiments, after calculating the final score of each model in step S6, the optimal model is selected, and the GIS heat map and time series error curve of the past few days are presented through the QT interface based on the measured data of the past few days and the forecast data of the optimal model.

[0097] Compared with existing technologies, the method for evaluating air quality forecasting models provided in this embodiment calculates the values ​​of a first indicator and a second indicator by using the forecast data and measured data output by each air quality forecasting model. Then, it calculates the standard score of each model with respect to the first indicator to obtain a preliminary model performance evaluation result. Next, it uses the values ​​of the first and second indicators to evaluate the importance of the first evaluation index system to obtain the weight of each first indicator. The weighted sum of the first indicator and the standard score is then used to obtain the final score of each model. This method comprehensively considers multiple pollutants and multiple evaluation indicators, thereby achieving an accurate evaluation of the forecast results output by multiple air quality forecasting models, i.e., model performance, so as to provide users with accurate air quality forecasts in the future.

[0098] Example 2

[0099] This embodiment provides an apparatus for evaluating air quality forecasting models, such as... Figure 2 As shown, it includes:

[0100] The data acquisition module is used to acquire forecast data and measured data output by various air quality forecast models. The forecast data includes predicted values ​​of pollutant concentrations and predicted values ​​of air quality index, and the measured data includes actual values ​​of pollutant concentrations and actual values ​​of air quality index.

[0101] A primary pollutant identification module is used to determine the primary pollutant based on the measured data.

[0102] The index calculation module is configured to calculate the value of each model with respect to each first index in the first evaluation index system according to the predicted value and the actual value of the concentration of the primary pollutant; and calculate the value of each model with respect to each second index in the second evaluation index system according to the predicted value and the actual value of the air quality index.

[0103] The standard score calculation module is configured to calculate the standard score of each model with respect to each first index.

[0104] The importance evaluation module is configured to perform importance evaluation on the first evaluation index system according to the value of each first index and each second index, and obtain the weight of each first index.

[0105] The final score module is configured to perform weighted summation on the standard score and the weight of each first index to obtain the final score of each model.

[0106] Embodiment 3

[0107] The embodiment provides an electronic device, such as Figure 3 as shown, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to implement the method for evaluating the air quality prediction model according to any one of the above embodiments.

[0108] Embodiment 4

[0109] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for evaluating the air quality prediction model according to any one of the above embodiments.

[0110] The computer readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0111] Those skilled in the art should further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0112] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0113] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for evaluating air quality forecasting models, characterized in that, include: Acquire forecast data and measured data output by various air quality forecast models. The forecast data includes predicted values ​​of pollutant concentrations and predicted values ​​of air quality index. The measured data includes actual values ​​of pollutant concentrations and actual values ​​of air quality index. The primary pollutant was determined based on the measured data. The values ​​of each model for each primary indicator in the first evaluation index system are calculated based on the predicted and actual concentrations of the primary pollutants. The values ​​of each model for each secondary indicator in the second evaluation index system are calculated based on the predicted and actual air quality index. The first evaluation index system includes correlation coefficient, average error, root mean square error, average deviation, and average absolute error. The second evaluation index system includes the accuracy of air quality index range forecast and the accuracy of air quality index level forecast. Calculate the standard score for each model with respect to the first indicator for each item; Based on the values ​​of the first and second indicators, the importance of the first evaluation indicator system is assessed to obtain the weight of each of the first indicators. The final score of each model is obtained by weighting and summing the standard scores and weights of the first indicator.

2. The method for evaluating air quality forecasting models according to claim 1, characterized in that, Also includes: The forecast data and the measured data are aligned on a time scale.

3. The method for evaluating air quality forecasting models according to claim 1, characterized in that, The determination of the primary pollutant based on the measured data includes: The air quality index for each pollutant is calculated based on its actual concentration and expressed as follows: in, Indicates pollutants p Air quality index, Indicates pollutants p The higher value of the concentration limit, Indicates pollutants p The lower value of the concentration limit, express The corresponding air quality sub-index, express The corresponding air quality sub-index, Indicates pollutants in the measured data p Concentration value; The pollutant with the highest air quality index is designated as the primary pollutant.

4. The method for evaluating air quality forecasting models according to claim 1, characterized in that, The calculation of the standard score of each model with respect to each of the first indicators includes: The standard scores for each model with the first indicator being a positive indicator are expressed as follows: The positive indicators include the correlation coefficient. Representation Model i Regarding the first indicator j Standard score, Representation Model i Regarding the first indicator j The value, and These represent the first indicator. j The maximum and minimum values; The standard scores for each model when the first indicator is negative are expressed as follows: The negative indicators include average error, root mean square error, average deviation, and average absolute error.

5. The method for evaluating air quality forecasting models according to claim 4, characterized in that, Based on the values ​​of the first and second indicators, the importance of the first evaluation indicator system is assessed to obtain the weights of each of the first indicators, including: Construct a feature vector based on the values ​​of the first and second indicators; The model's prediction performance is rated based on the value of the second indicator, and the rating result is used as the training label. The feature vector and training labels are input as training samples into the random forest classification model for training. The importance contribution value of each of the first indicators is output, and after normalization, it is used as the weight of each of the first indicators.

6. The method for evaluating air quality forecasting models according to claim 5, characterized in that, The final score for each model is obtained by weighted summation of the standard score and weight of the first indicator, as shown below: in, Representation Model i The final score, Indicates the first indicator j The weight, n This indicates the number of first indicators in the first evaluation indicator system.

7. An apparatus for evaluating air quality forecasting models, characterized in that, The device includes: The data acquisition module is used to acquire forecast data and measured data output by various air quality forecast models. The forecast data includes predicted values ​​of pollutant concentrations and predicted values ​​of air quality index, and the measured data includes actual values ​​of pollutant concentrations and actual values ​​of air quality index. A primary pollutant identification module is used to determine the primary pollutant based on the measured data. The indicator calculation module is used to calculate the values ​​of each model for each first indicator in the first evaluation indicator system based on the predicted and actual concentration values ​​of the primary pollutant; and to calculate the values ​​of each model for each second indicator in the second evaluation indicator system based on the predicted and actual air quality index values; the first evaluation indicator system includes correlation coefficient, average error, root mean square error, average deviation, and average absolute error, and the second evaluation indicator system includes the range forecast accuracy and level forecast accuracy of the air quality index; The standard score calculation module is used to calculate the standard score of each model with respect to the first indicator for each item; The importance assessment module is used to assess the importance of the first evaluation index system based on the values ​​of the first index and the second index, and to obtain the weight of each of the first indexes. The final scoring module calculates the final score for each model by weighting and summing the standard scores and weights of the first indicator.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for evaluating an air quality forecasting model as described in any one of claims 1-6.

9. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for evaluating an air quality forecasting model as described in any one of claims 1-6.

Images