Monitoring data quality control method and device based on multi-outlier detection algorithm

Abstract

The invention discloses a monitoring data quality control method and device based on a multi-outlier detection algorithm, and belongs to the field of environmental monitoring data processing, and the method comprises the following steps: step 1, data screening and preprocessing, step 2, data quality control, and step 3, grid data and regional data generation. According to the invention, by combining three machine learning algorithms and utilizing STL decomposition, misjudgment is effectively avoided, and the accuracy of anomaly detection is improved. According to the improved IDW + algorithm, by introducing a variation function and a parameter estimation point screening mechanism, the precision of an interpolation result is ensured, and interference of invalid parameter estimation points is reduced. IDW interpolation is firstly carried out on the missing points, it can be ensured that an interpolated basic data set is more complete, and instability caused by the missing data points does not need to be considered too much during Kriging interpolation. The overall interpolation result obtains double guarantee on the accuracy of local data and the smoothness of global space grid data.

Description

Technical Field

[0001] The present invention belongs to the field of environmental monitoring data processing, and in particular relates to a monitoring data quality control method and device based on a multi-outlier detection algorithm. Background Art

[0002] With the accelerated development of urbanization and industrialization, air pollution has become a global concern. Countries have established air quality monitoring networks to collect real-time pollutant concentration data through monitoring stations and obtain more extensive regional air quality information with the help of remote sensing technology. However, existing technologies still have some limitations in monitoring data processing and spatial analysis.

[0003] At present, air quality data mainly comes from two channels: ground monitoring stations and satellite remote sensing inversion. Satellite remote sensing can invert aerosol optical thickness (AOD) over a large area and estimate ground particulate matter concentration (such as PM2.5). However, due to the limitations of satellite platforms, sensor resolution and inversion algorithms, the temporal and spatial resolution of satellite remote sensing data is relatively low, and data in some areas may be missing due to reasons such as cloud cover. The difficulty of satellite data processing is also greater than that of ground monitoring stations. First, the inversion process not only depends on the optical or thermal infrared data observed by satellites, but also needs to be combined with auxiliary information such as meteorological data, surface albedo, and aerosol patterns. The acquisition and integration process of these data sources is relatively complex. Secondly, the inversion algorithm itself is also difficult. It is necessary to consider the characteristics of different sensors, band combinations, and changes in atmospheric conditions. These factors will have a great impact on the accuracy of the inversion results. In addition, interfering substances such as aerosols, clouds, and water vapor in the atmosphere further increase the difficulty of satellite data processing and bring uncertainty to air quality inversion. Finally, since satellites acquire large-scale, continuous spatiotemporal data, the amount of data is huge, which places high demands on computing and storage capabilities. The differences in resolution, bands, and time intervals between different satellite platforms and sensors make the integration and fusion of multi-platform data more complicated.

[0004] Although the resolution of satellite inversion data has improved in recent years, local air quality assessment still mainly relies on high-precision pollutant concentration data provided by ground monitoring stations. The amount of raw monitoring data generated by ground monitoring stations is huge, including multiple pollutants and other related indicators, and the data is usually generated hourly. During the collection and transmission process, the monitoring equipment may be missing or outliers due to external factors such as failure or extreme weather. If these outliers are not effectively processed, they may affect the accuracy of subsequent data analysis. Traditional anomaly detection methods (such as three sigma criteria, box plots, etc.) are difficult to adapt to multidimensional, nonlinear and complex data due to their reliance on data distribution assumptions. Therefore, more accurate anomaly detection methods have become a key link in data processing. At the same time, in order to achieve regional air quality assessment, it is usually necessary to perform spatial interpolation on the local data of the monitoring points to generate a continuous distribution map of regional pollutant concentrations. Although the traditional inverse distance weighted (IDW) method is simple to calculate, it fails to consider the spatial correlation between monitoring points, resulting in low interpolation accuracy, especially when the stations are unevenly distributed, it is easy to produce interpolation errors. The Kriging interpolation method takes into account the spatial autocorrelation between monitoring points and can generate smoother and more accurate regional pollutant concentration distribution maps, but its calculation process is complex and the processing efficiency of large-scale data is not high, especially in areas with sparse station distribution. The interpolation effect is poor, and the interpolation result cannot retain the original attribute values ​​of the reference points. Therefore, how to optimize the spatial interpolation method to improve the interpolation accuracy and reduce the calculation complexity, while retaining the original attribute values ​​of the reference points during the interpolation process to produce a pollutant concentration data set whose original values ​​of the monitoring points are not changed, is also an important issue in air quality data processing.

[0005] The disadvantages of the prior art are as follows:

[0006] (1) Limitations of satellite remote sensing inversion: Satellite remote sensing inversion of air quality data not only faces problems such as low temporal and spatial resolution and cloud cover, but also relies on a variety of auxiliary data (such as meteorological data, aerosol patterns, etc.). The inversion algorithm is complex and needs to be calibrated with ground monitoring data. In addition, atmospheric interference, massive data processing, and the integration of different satellite platforms and sensors also increase the difficulty of inversion. This makes the inversion process not only time-consuming and labor-intensive, but also full of uncertainty and technical challenges.

[0007] (2) Limitations of ground-based monitoring data: Although ground-based monitoring stations can provide high-precision local pollution data, due to their limited coverage, it is difficult to fully reflect the spatial distribution characteristics of regional air quality.

[0008] (3) Imperfect processing of missing data and outliers: Ground-based air quality monitoring data is prone to missing values ​​and outliers due to the influence of various factors. Traditional statistical outlier detection methods such as three sigma and box plots are not robust enough for complex and multidimensional data, and are prone to misjudgment or missed detection. Existing methods for processing missing data also mainly rely on simple rules or single algorithms, and the data accuracy after missing value processing is not high.

[0009] (4) Insufficient accuracy of a single spatial interpolation algorithm: Existing interpolation algorithms, such as the IDW or Kriging method, have their own shortcomings when used alone. Although the IDW method is highly efficient, it only considers distance, resulting in low local interpolation accuracy; although the Kriging method has high accuracy, it is complex to calculate and cannot retain the original attribute values ​​of the reference points. Summary of the invention

[0010] In view of the technical problems in the background technology, the first aspect of the present invention solves the problem of imperfect processing of outliers and missing values ​​in monitoring data: the existing methods have limitations in processing outliers and missing values ​​in monitoring data, resulting in inaccurate data analysis results. The present invention will detect and repair outliers by combining multiple algorithms (such as Isolation Forest, LOF, DBSCAN), and at the same time use improved interpolation methods to improve the accuracy of missing value completion to solve the problem of incompleteness of ground monitoring point data.

[0011] In view of the technical problems in the background technology, the second aspect of the present invention solves the problem of insufficient coverage of ground monitoring point data: due to the uneven distribution of ground monitoring sites, it is difficult for existing air quality assessments to fully cover the distribution of pollutants in a large area. The present invention proposes to expand limited monitoring data to a larger area through an improved interpolation algorithm, solve the problem of sparse site distribution, and improve the coverage and accuracy of regional air quality assessment.

[0012] In view of the technical problems in the background technology, the third aspect of the present invention solves the problem of insufficient precision of a single spatial interpolation algorithm: the current single IDW or Kriging interpolation algorithm has limitations and cannot take into account both interpolation efficiency and precision. The present invention will optimize the local interpolation precision through a multi-level interpolation method combining improved IDW+ and Kriging, and achieve a balance between computational efficiency and global interpolation effect, thereby solving the interpolation precision problem in regional air quality assessment.

[0013] The technical solution of the present invention is as follows:

[0014] The monitoring data quality control method based on the multi-outlier detection algorithm includes the following steps:

[0015] Step 1: Data screening and preprocessing

[0016] Step 1.1: Data screening

[0017] Unique codes are assigned to monitoring sites, monitored pollutants and monitoring times.

[0018] Step 1.2: Data validity conversion

[0019] Convert hourly data into daily data, and calculate the arithmetic mean of existing data except invalid data: for the five pollutants SO2, NO2, CO, PM2.5 and PM10, traverse the hourly monitoring data daily. If the cumulative missing data is greater than 4 hours, the daily average value is regarded as invalid data and represented by NaN. Otherwise, calculate the arithmetic mean of the existing data. For O3, traverse the hourly monitoring data daily. If the cumulative missing data is greater than 2 hours in every consecutive 8 hours, the daily average value is regarded as invalid data and represented by NaN. Otherwise, calculate the arithmetic mean of the existing data.

[0020] Step 2: Data quality control

[0021] Step 2.1: Outlier processing

[0022] ① Trend and seasonal decomposition of data:

[0023] The time series data is decomposed using STL (Seasonal-Trend decomposition using Loess) to break the data into trend, seasonality and residual parts, and anomaly detection is performed only on the residual.

[0024] ②Outlier detection combining multiple algorithms:

[0025] Three different algorithms, Isolation Forest, LOF and DBSCAN, are used for outlier detection, and each algorithm runs independently.

[0026] ③Outlier classification processing:

[0027] Extreme outliers: Points that are unanimously considered to be abnormal by all algorithms are directly marked as NaN and are not repaired; Mild outliers: Points that are abnormal detected by two algorithms are repaired using linear interpolation; Non-outliers: No more than one algorithm detects an abnormality and no modification is made.

[0028] Step 2.2: Missing value processing

[0029] The traditional inverse distance weighted (IDW) method only considers the spatial distance when calculating the weights. However, in practical applications, distance alone is not enough to fully reflect the correlation between spatial points, especially in the presence of spatial heterogeneity, and the interpolation accuracy is low. To overcome this shortcoming, the improved IDW+ algorithm introduces the spatial variation function into the weight calculation, replacing the single distance factor, so as to better characterize the degree of variation between spatial points, and adds distance and height difference parameters when screening reference points, further improving the accuracy of the interpolation results and reducing the deviation.

[0030] ①Use the variation function to replace the power of distance in the traditional IDW method, and the calculation formula is as follows:

[0031] ,

[0032] ,

[0033] In the formula, Represents the i-th known sample point To target point The weighting coefficient of Indicates the target point to be interpolated; Represents the position of the i-th known sample point; Represents the variogram, which reflects the sample points With target point The degree of spatial variation between them; n is the number of known sample points involved in the interpolation calculation; represents the estimated value at the target point; Represents the attribute value of the i-th known sample point.

[0034] ② Improved parameter selection

[0035] The validity of the reference points is screened based on the distance and height difference: the upper quartile of the distance is used as the threshold to screen out the sites with closer distance as valid reference points, and the upper quartile of the height difference is used as the threshold to screen out the sites with smaller height difference as valid reference points. If either the distance or the height difference between two points does not meet the validity condition, the point will not be used as a reference point. Finally, the site with both valid distance and height difference will be used as a reference point.

[0036] Step 3: Generation of grid data and regional data

[0037] After converting the hourly air quality data into daily data and controlling the data quality, the monthly and annual averages are calculated in the time dimension, and the kilometer network data of the target city or urban agglomeration Ci is obtained by Kriging spatial interpolation in the spatial dimension. The daily, monthly and annual average data of the six major pollutants representing Ci are then obtained by averaging the kilometer network data.

[0038] A monitoring data quality control device comprises one or more electronic devices, wherein the one or more electronic devices are used to implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0039] An electronic device comprises: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0040] A computer-readable storage medium stores executable instructions, which, when executed by a processor, enable the processor to implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0041] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0042] Beneficial effects:

[0043] 1. The existing technology usually adopts the traditional three sigma method, box plot method or single machine learning algorithm, which is easy to misjudge data, especially for time series data with trends and seasonal fluctuations. The present invention effectively avoids misjudgment and improves the accuracy of anomaly detection by combining three machine learning algorithms and using STL decomposition.

[0044] 2. The traditional IDW interpolation method only considers the distance between monitoring points, but does not consider spatial correlation, resulting in low interpolation accuracy. Although the Kring method considers spatial correlation, the interpolation result cannot retain the original value of the reference point attribute. The improved IDW+ algorithm of the present invention ensures that the interpolation result retains the original value of the monitoring point attribute by introducing the variation function and the reference point screening mechanism, reduces the interference of invalid reference points, and improves the interpolation accuracy.

[0045] 3. Compared with using Kriging interpolation directly, IDW+ interpolation of missing points can ensure that the basic data set for interpolation is more complete, and there is no need to consider the instability caused by missing data points when using Kriging interpolation. The overall interpolation result is doubly guaranteed in terms of the accuracy of local data and the smoothness of global spatial grid data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the key coding process in the present invention;

[0047] Figure 2 It is a schematic diagram of the process of data validity conversion in the present invention;

[0048] Figure 3 It is a schematic diagram of the process of outlier processing in the present invention;

[0049] Figure 4 It is a schematic diagram of the process of missing value processing in the present invention;

[0050] Figure 5 It is a schematic diagram of the whole process from monitoring point data to regional data in the present invention;

[0051] Figure 6 Direct visualization of monitoring points;

[0052] Figure 7 This is a visualization of the 10-kilometer grid data;

[0053] Figure 8 A zoomed-in view of the 10-kilometer grid data visualization;

[0054] Fig. 9 This is a visualization of one-kilometer grid data. DETAILED DESCRIPTION

[0055] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. However, the following embodiments are limited to explaining the present invention, and the protection scope of the present invention should include the entire contents of the claims, and through the description of the following embodiments, those skilled in the art can fully implement the entire contents of the claims of the present invention.

[0056] Example 1

[0057] The monitoring data quality control method based on the multi-outlier detection algorithm includes the following steps:

[0058] Step 1: Data screening and preprocessing

[0059] (1) Data screening

[0060] Monitoring data coding is crucial for the collection, exchange, processing and use of data from ground monitoring sites, and is the basis for the information management of ambient air quality monitoring. The amount of raw data generated by ground monitoring points is huge, covering multiple dimensions, including monitoring data of multiple pollutants and other indicators, and these data are usually generated on a daily basis. First, select the most critical coding: monitoring site, monitoring pollutant, and monitoring time. These three elements are the minimum items required in data processing, and their relationship is as follows: Figure 1 As shown in the figure, data filtering can reduce the running memory required for subsequent data processing and improve the overall efficiency.

[0061] Site coding: Assign unique codes to monitoring sites to identify monitoring locations and ensure accurate correspondence of the geographical locations of monitoring data.

[0062] Pollutant coding: Assign unique codes to different types of pollutants to identify the types of pollutants being monitored and clarify the pollutant attributes of the monitoring data.

[0063] Monitoring time code: Assign a unique time code to each piece of monitoring data to record the time when the monitoring data was generated.

[0064] (2) Data validity conversion

[0065] The missing and abnormal hourly data are greatly affected by the monitoring instruments, so in the data preprocessing stage, the hourly data is first converted into daily data according to the data validity regulations. The "Ambient Air Quality Standard (GB3095)" implemented in my country in 2016 clearly stipulates the validity of pollutant monitoring data, as shown in Table 1:

[0066] Table 1 Pollutant data validity specifications

[0067] Therefore, when processing data standardization, for the five pollutants SO2, NO2, CO, PM2.5 and PM10, the hourly monitoring data is traversed daily. If the cumulative missing data is greater than 4 hours, the daily average value is considered invalid data and represented by NaN. Otherwise, the arithmetic mean of the existing data is calculated. For O3, the hourly monitoring data is traversed daily. If the cumulative missing data is greater than 2 hours in every consecutive 8 hours, the daily average value is considered invalid data and represented by NaN. Otherwise, the arithmetic mean of the existing data is calculated. The algorithm flow is as follows: Figure 2 shown.

[0068] Step 2: Data quality control

[0069] (1) Outlier processing

[0070] The present invention proposes a time series data anomaly detection and repair method based on the combination of multiple algorithms.

[0071] ① Trend and seasonal decomposition of data:

[0072] STL (Seasonal-Trend decomposition using Loess) is used to decompose time series data into trend, seasonality, and residual parts. Anomaly detection is only performed on residuals to prevent normal trends and seasonal fluctuations from being misjudged as outliers.

[0073] ②Outlier detection combining multiple algorithms:

[0074] The present invention uses three different algorithms, Isolation Forest, LOF and DBSCAN, to detect outliers. Isolation Forest is an isolation forest algorithm based on decision trees. It isolates outliers through tree structures and can effectively process high-dimensional data and isolated outliers. LOF is a local outlier factor that calculates sample density and is used to identify points with abnormal density in local areas. DBSCAN is a density-based clustering algorithm that is suitable for identifying outliers in sparse areas. Each algorithm runs independently, and the final comprehensive result is used to classify outliers.

[0075] ③Outlier classification processing:

[0076] Extreme outliers: Points that are unanimously considered to be abnormal by all algorithms are directly marked as NaN and are not repaired; Mild outliers: Points detected as abnormal by two algorithms are repaired using linear interpolation, and the repair range is limited to adjacent points to prevent excessive interpolation from affecting the overall data; Non-outliers: No more than one algorithm detects an abnormality, and no modification is made to maintain data integrity. The algorithm flow is as follows Figure 3 shown.

[0077] (2) Missing value processing

[0078] The present invention proposes a strategy of "local completion first, then global interpolation". In the data quality control step, the improved IDW+ algorithm is used to interpolate the sites with missing values. An important feature of this algorithm is that it can retain the values ​​of the original monitoring sites. This ensures that the original data of the monitoring sites are not affected, and at the same time, the missing data are reasonably supplemented, which is independent of the subsequent global interpolation algorithm.

[0079] ① Improvement of IDW method

[0080] Although the IDW algorithm is simple to use and has high execution efficiency, since it only considers the distance between two points, the interpolation accuracy is not high and the deviation cannot be estimated. Specifically, for any point in space Properties , its value can be obtained by i ) attributes of known sample points ) is estimated, and the weighting coefficient of the traditional IDW method is Only the distance between two points As shown in formula (2).

[0081] (1)

[0082] (2)

[0083] This technique introduces the variogram , which reflects the sample point With target point The spherical model is used as the theoretical model of the variation function, as shown in formula (3). The estimation results of this model are more robust than those of other commonly used models. Sill value, is the arch height, a is the range, which is obtained by data fitting in the algorithm.

[0084] (3)

[0085] The distance weight in the IDW algorithm Replace with variogram-based weights ,As shown in Equation (4) and Equation (5), the improved IDW+ method takes spatial correlation into account, and the interpolation result is more accurate.

[0086] (4)

[0087] (5)

[0088] The letters in the formula represent: and is the weighting coefficient used to characterize the i-th sample point Interpolation points the extent of the impact; Indicates interpolation points and sample points The spatial distance; k represents the power exponent of the distance, which controls the speed at which the weight changes with distance, and is usually taken as 2; n is the number of known sample points involved in the interpolation calculation.

[0089] ② Improvement of reference point screening

[0090] Traditional IDW usually considers all monitoring points with data as reference points. However, the spatial distribution of these sites is not uniform, and the distances and height differences between different monitoring points vary significantly. In order to improve the efficiency and accuracy of interpolation, this technology introduces the effectiveness screening of distance and height difference: using the upper quartile of the distance as the threshold, sites with closer distances are screened out as valid reference points; similarly, using the upper quartile of the height difference as the threshold, sites with smaller height differences are screened out as valid reference points. As long as one of the distance or height difference between two points does not meet the validity condition, the point will not be used as a reference point to reduce the interference of irrelevant points. Finally, sites with both valid distance and height difference are used as reference points to improve the accuracy and computational efficiency of interpolation. With this method, the number of reference points is significantly reduced, but the accuracy of spatial interpolation is improved. The algorithm flow is as follows Figure 4 shown.

[0091] Step 3: Generation of grid data and regional data

[0092] After converting the hourly air quality data into daily data and controlling the data quality, the monthly and annual averages are calculated in the time dimension, and the kilometer network data of the target city / city cluster Ci is obtained by kriging spatial interpolation in the spatial dimension. Then, the daily, monthly and annual average values ​​of the six major pollutants representing Ci are obtained by averaging the kilometer network data. The algorithm flow is as follows: Figure 5 shown.

[0093] Example 2

[0094] The specific implementation of the method in Example 1 and the rationality of the strategy are verified, and the technical solution is further elaborated in detail by taking the Chengdu-Chongqing urban agglomeration as an example.

[0095] Step 1: Data screening and preprocessing

[0096] The study area is determined to be the Chengdu-Chongqing urban agglomeration, and only the hourly concentration data of six major pollutants from 133 national monitoring points in 16 cities in the Chengdu-Chongqing urban agglomeration are screened and used, with the time range being 2015-2021.

[0097] Step 2: Data quality control

[0098] Identification and handling of outliers

[0099] This technology uses STL to decompose the time series and a flexible repair strategy to prevent normal seasonal fluctuations or trend changes from being mistaken for outliers. It then uses the ISO, LOF, and DBSCAN algorithms to identify pollutant concentration outliers, and then uses the three algorithms to vote for identification. The results are shown in Table 2.

[0100] The BoxPlot method detects outliers based on quartiles and IQR (interquartile range), which is prone to produce more outlier detections for data whose distribution deviates from the median; the 3 Sigma method is also based on mean and standard deviation, but for complex data distribution, it may miss some local anomalies; the Isolation Forest and LOF methods perform well in capturing local or isolated points; DBSCAN detects a lot of outliers on O3, PM2.5 and PM10 pollutants. This is because DBSCAN is a density-based clustering algorithm, which may produce a large number of outlier detections in sparse areas (such as pollutant peaks), indicating that it is very sensitive to density changes in data, but it may produce too many "outliers" and lead to misjudgment. The STL + Voting method used in this technology detects a moderate number of outliers on all pollutants and has a low anomaly rate. It combines the advantages of multiple algorithms and reduces false detections and missed detections through a voting mechanism, so it is more reliable in detecting real outliers. Different repair strategies are selected according to the severity of the outliers to ensure the integrity and rationality of the data. Extreme anomalies are removed and mild anomalies are smoothly repaired. Finally, the daily data anomaly values ​​of the monitoring points of the Chengdu-Chongqing urban agglomeration are reduced to zero.

[0101] Table 2 Effects of different algorithms on outlier identification

[0102] Identification and treatment of missing values

[0103] Taking PM2.5 concentration data as an example, three interpolation methods, IDW, IDW+ and Kriging, are used to interpolate missing values. Experiments show that all three methods can complete missing values. 5-fold cross validation is used to evaluate the effect of each interpolation algorithm. The results are shown in Table 3.

[0104] IDW+ obviously performs best in most evaluation indicators, with the lowest mean square error (MSE), standard deviation (STD) and absolute bias mean (Abs Bias Mean). This shows that the IDW+ method after introducing the variogram and parameter point screening is superior to the traditional IDW and Kriging methods in interpolation accuracy and stability. The IDW method performs well, but is not as stable and accurate as IDW+. The Kriging method performs the worst, especially in MSE, standard deviation and absolute deviation maximum values, with large error fluctuations, especially in some areas where the interpolation accuracy is very poor, indicating that it has no advantage in interpolation with a large number of missing values.

[0105] Table 3 Cross-validation results of missing value interpolation

[0106] Step 3: Convert monitoring point data into regional data

[0107] ① Grid data interpolation algorithm: first use IDW+ to complete the missing values ​​of the station, then use Kriging interpolation

[0108] Taking the PM2.5 data of the Chengdu-Chongqing urban agglomeration from 2015 to 2021 as an example, the processing effects of the three algorithms of IDW, Kriging direct interpolation, and IDW+local completion followed by global Kriging interpolation are verified as shown in Table 4. The results of IDW direct interpolation show a larger standard deviation and a wider data range. This shows that IDW interpolation produces large fluctuations in data distribution, especially in areas where monitoring points are sparse. IDW is more dependent on local point values ​​and is prone to produce large interpolation errors in places where data is sparse. The grid data obtained by Kriging direct interpolation is relatively smooth, with a narrow range of values ​​and a small standard deviation. The mean of IDW+Kriging is similar to that of direct Kriging, and the mean square error MSE between them is 0.42, which proves that the two schemes have similar effects in handling missing values, but the former has a lower standard deviation, indicating that the data obtained by IDW+Kriging is more concentrated and smoother, and the effect is better.

[0109] Table 4 Verification results of grid data interpolation algorithm

[0110] ②Grid division value: one kilometer

[0111] Taking the PM2.5 concentration data of the Chengdu-Chongqing urban agglomeration in 2015 as an example, the interpolation effect using only the monitoring point data is as follows: Figure 6 As shown in the figure, the rendering cannot cover the area and is severely pixelated. The interpolated contour lines are also very stiff and cannot accurately reflect the air quality of the area. Therefore, it is necessary to interpolate to grid data. The closer the unit distance, the greater the density of the grid points, and the smoother the image after rendering. However, this will also increase the amount of calculation and bring computing burden. The effect of a grid division value of ten kilometers is shown below. Figure 7 As shown in , it can be clearly observed that the visualization effect of pollutant concentration is greatly improved, the rendering covers the entire area, and the contour lines are smoother, but pixels can still be found after zooming in on the image, such as Figure 8 As shown. Construct kilometer network data and then interpolate, the effect is as follows Fig. 9 Therefore, this technology sets the grid resolution to one kilometer, which is sufficient for spatial analysis.

[0112] A monitoring data quality control device comprises one or more electronic devices, wherein the one or more electronic devices are used to implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0113] An electronic device comprises: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0114] A computer-readable storage medium stores executable instructions, which, when executed by a processor, enable the processor to implement a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0115] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements a monitoring data quality control method and device based on a multi-outlier detection algorithm.

[0116] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A monitoring data quality control method based on a multi-outlier detection algorithm, characterized in that: The steps include: Step 1: Data screening and preprocessing; Step 2: Data quality control; Step 3: Generation of grid data and regional data.

2. The monitoring data quality control method based on the multi-outlier detection algorithm according to claim 1 is characterized in that: The step 1 includes the following steps: Step 1.1: Data screening Assign unique codes to monitoring sites, monitored pollutants and monitoring time; Step 1.2: Data validity conversion Convert hourly data into daily data, traverse the hourly monitoring data every day, and calculate the arithmetic average of the existing data except the invalid data.

3. The monitoring data quality control method based on the multi-outlier detection algorithm according to claim 1 is characterized in that: The step 2 includes the following steps: Step 2.1: Outlier processing ① Trend and seasonal decomposition of data: Use STL to decompose time series data into trend, seasonality and residual parts, and anomaly detection is only performed on residuals; ②Outlier detection combining multiple algorithms: Three different machine learning algorithms, Isolation Forest, LOF, and DBSCAN, were used for outlier detection, and each algorithm ran independently; ③Outlier classification processing: Extreme outliers: points that are unanimously considered to be abnormal by all algorithms are directly marked as NaN and are not repaired; Mild outliers: points that are abnormal detected by two algorithms are repaired using linear interpolation; Non-outliers: points that are detected as abnormal by no more than one algorithm are not modified; Step 2.2: Missing value processing The improved IDW+ algorithm was used to interpolate sites with missing values; ①The improved IDW+ algorithm is shown as follows: ， ， In the formula, Represents the i-th known sample point To target point The weighting coefficient of Indicates the target point to be interpolated; Represents the position of the i-th known sample point; Represents the variogram, which reflects the sample points With target point The degree of spatial variation between them; n is the number of known sample points involved in the interpolation calculation; represents the estimated value at the target point; Represents the attribute value of the i-th known sample point; ② Improved parameter selection The validity of the reference points is screened based on the distance and height difference: the upper quartile of the distance is used as the threshold to screen out the sites with closer distance as the valid reference points. Similarly, the upper quartile of the height difference is used as the threshold to screen out the sites with smaller height difference as the valid reference points. If either the distance or the height difference between two points does not meet the validity condition, the point will not be used as a reference point. Finally, the sites with both valid distance and height difference will be used as reference points.

4. The monitoring data quality control method based on the multi-outlier detection algorithm according to claim 1 is characterized in that: The step 3 includes the following steps: after converting the hourly air quality data into daily data and controlling the data quality, the monthly and annual averages are calculated in the time dimension, and the kilometer network data of the target city or urban agglomeration Ci is obtained by using Kriging spatial interpolation in the spatial dimension, and then the daily average, monthly average, and annual average data of the pollutants representing Ci are obtained by averaging the kilometer network data.

5. The monitoring data quality control method based on the multi-outlier detection algorithm according to claim 2 is characterized in that: In step 1.2, for the five pollutants SO2, NO2, CO, PM2.5 and PM10, the hourly monitoring data are traversed daily. When the cumulative missing data is greater than 4 hours, the daily average value is regarded as invalid data and represented by NaN. Otherwise, the arithmetic mean of the existing data is calculated.

6. The monitoring data quality control method based on the multi-outlier detection algorithm according to claim 2 is characterized in that: In step 1.2, for O3, the hourly monitoring data is traversed daily. If the cumulative missing data is greater than 2 hours in every 8 consecutive hours, the daily average value is regarded as invalid data and represented by NaN. Otherwise, the arithmetic mean of the existing data is calculated.

7. A monitoring data quality control device, characterized in that: The method comprises one or more electronic devices, wherein the one or more electronic devices are used to implement the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor implements the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images