Abnormality identification method and equipment for air data and medium

Abstract

The invention provides an air data anomaly identification method and device and a medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring air data in a target monitoring period; the occurrence frequency of different first digits in the air data is calculated, and a first anomaly in the air data is recognized based on the Benford's law according to the occurrence frequency; performing time sequence analysis on the air data, and identifying a second anomaly in the air data; correlation among the air indexes is calculated, and a third anomaly in the air data is recognized according to the correlation; and according to the weather data in the target monitoring period, predicting reference air data, and according to the reference air data, identifying a fourth anomaly in the air data. According to the invention, the abnormity in the air data can be comprehensively and accurately identified.

Description

Methods, equipment and media for identifying anomalies in air data Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device and medium for anomaly identification of air data. Background Technology

[0002] In recent years, the ecological and environmental monitoring system has been continuously improved, and the air quality monitoring network has gradually extended from national and provincial control stations to cities, counties and a large number of micro-monitoring stations, with the scale of monitoring data growing exponentially.

[0003] However, in actual monitoring, air monitoring data anomalies occur frequently due to factors such as differences in monitoring equipment performance, meteorological disturbances, human intervention, and data transmission errors. This severely restricts the scientific rigor and accuracy of environmental supervision, pollution source tracing, and air quality assessment. Therefore, constructing an intelligent screening system capable of automatically identifying abnormal data across models and scenarios is of great significance. Summary of the Invention

[0004] This invention provides a method, device, and medium for identifying anomalies in air data, so as to comprehensively and accurately identify anomalies in air data.

[0005] In a first aspect, embodiments of the present invention provide a method for identifying anomalies in air data, comprising: acquiring air data within a target monitoring period; calculating the frequency of occurrence of each different first digit in the air data, and identifying a first anomaly in the air data based on the frequency of occurrence and Benford's law; performing time-series analysis on the air data to identify a second anomaly in the air data; calculating the correlation between various air indicators, and identifying a third anomaly in the air data based on the correlation; predicting reference air data based on weather data within the target monitoring period, and identifying a fourth anomaly in the air data based on the reference air data.

[0006] In one possible implementation, identifying the first anomaly in the air data based on the occurrence frequency and Benford's Law includes: for each digit, calculating the occurrence frequency based on the occurrence frequency and the amount of data; and identifying the first anomaly in the air data based on the occurrence frequency and historical standard frequencies.

[0007] In one possible implementation, identifying the first anomaly in the air data based on the frequency of occurrence, according to Benford's Law, includes: for each digit, calculating the expected frequency based on historical standard frequencies and data volume; calculating the chi-square statistic based on the frequency of occurrence and the expected frequency of each digit; and identifying the first anomaly in the air data based on the chi-square statistic.

[0008] In one possible implementation, the step of performing time-series analysis on the air data to identify a second anomaly in the air data includes: obtaining the time-series variation patterns of various air indicators, identifying the second anomaly in the air data based on the time-series variation patterns; and identifying the second anomaly in the air data using one or more time-series analysis models selected from statistical models, machine learning models, and deep learning models.

[0009] In one possible implementation, calculating the correlation between various air quality indicators and identifying a third anomaly in the air data based on the correlation includes: calculating the correlation between various air quality indicators to generate a correlation coefficient matrix; identifying a third anomaly in the air data based on the correlation coefficient matrix; and performing dimensionality reduction on the air quality indicators using principal component analysis and calculating T. 2 The actual values ​​of the statistic and the Q statistic; based on T 2 The actual values ​​of the statistic and the Q statistic, and T 2 Historical standard values ​​of the statistic and Q statistic are used to identify a third anomaly in the air data.

[0010] In one possible implementation, the step of predicting reference air data based on weather data within the target monitoring period, and identifying a fourth anomaly in the air data based on the reference air data, includes: extracting weather features from the weather data, inputting the weather features into a pre-trained prediction model to obtain the reference air data; calculating the residual distribution between the reference air data and the air data; calculating the residual standard deviation based on the residual distribution; and identifying the fourth anomaly in the air data based on the residual standard deviation.

[0011] In one possible implementation, the statistical model includes: a Z-score model, a quartile range model, and a generalized ESD model; the machine learning model is an isolated forest model; and the deep learning model is a long short-term memory network model.

[0012] In one possible implementation, the pre-trained prediction model includes: a multiple linear regression model, a generalized additive model, an XGboost model, and a machine learning model.

[0013] In a second aspect, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0014] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0015] The beneficial effects of this invention compared to existing technologies are as follows: This invention identifies frequency deviations in large-scale air data caused by human intervention or systematic anomalies by calculating the frequency of occurrence of different first digits in air data and utilizing Benford's law, thus effectively identifying overall data structure anomalies. It also identifies anomalous data points through time-series analysis of air data, enabling rapid screening of structural anomalies in large datasets. Furthermore, it constructs a multi-indicator anomaly identification technology driven by pollutant synergistic patterns, identifying anomalous data points based on the correlation between various air indicators. Finally, it acquires weather data within the target monitoring period, predicts reference air data based on the weather's influence on air quality, and identifies anomalous points deviating from the established patterns. This invention forms a multi-perspective, multi-scale, and multi-mechanism small model system, covering key dimensions such as statistical characteristics, temporal patterns, spatial continuity, and physicochemical logic. Through parallel detection using multiple models, it can capture different types of anomalies from different angles, achieving comprehensive coverage of various data anomalies, including short-term mutations, slow drifts, structural anomalies, spatial jumps, and meteorological inconsistencies. Compared to traditional single-method approaches, this invention offers higher stability and efficiency when processing massive amounts of monitoring data, significantly improving the data cleaning and quality control capabilities of regulatory departments. Attached Figure Description

[0016] Figure 1 is a flowchart of the air data anomaly identification method provided in an embodiment of the present invention; Figure 2 is a structural diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0018] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0019] Referring to Figure 1, which shows the implementation flowchart of the air data anomaly identification method provided by the embodiment of the present invention, detailed as follows: Step 101, acquire air data within the target monitoring period.

[0020] For example, air quality data from the past 24 hours can be obtained, including but not limited to: PM2.5, ... 10 Concentrations of pollutants such as SO2, NO2, CO, and O3.

[0021] Step 102: Calculate the frequency of occurrence of each different first digit in the air data, and based on the frequency of occurrence, identify the first anomaly in the air data according to Benford's Law.

[0022] Ford's Law applies to datasets spanning multiple orders of magnitude, with large sample sizes, and without strong truncation or artificial adjustment. Analysis of air quality data reveals that PM2.5 and PM2.5... 10 Pollutants such as SO2, NO2, CO, and O3 exhibit a range from single digits to hundreds or even thousands of µg / m³ on both hourly and daily average scales, demonstrating a typical "order-of-magnitude" characteristic. Simultaneously, the data volumes for each category exceed one million, meeting the sample size requirements of Benford's Law. Stability analysis of data distribution across different years, seasons, and stations further confirms that these monitoring sequences do not exhibit significant artificial truncation or fixed range effects. Therefore, they generally meet the application prerequisites of Benford's Law and are suitable as a data foundation for model development.

[0023] After confirming its applicability, a large-scale extraction and statistical analysis of the first digit distribution of historical monitoring data was conducted. The study processed over 20 million data points, analyzing the first digit of pollutant concentrations for each data point and counting the frequency of occurrence of digits 1–9. Subsequently, the data was grouped and statistically analyzed by pollutant type, year, quarter, and monitoring station, constructing a feature library of first digit distribution across time and space dimensions. The comparison results show that the first digit distribution of most pollutants is largely consistent with the theoretical frequency of Benford's Law, while significant deviations occur in some years, specific stations, or individual time periods. These deviations are typically related to equipment malfunctions, sensor drift, data transmission errors, or human error, providing important evidence and validation samples for the subsequent construction of anomaly detection rules.

[0024] In one possible implementation, based on Benford's Law, the first anomaly in the air data is identified according to the frequency of occurrence, including: for each digit, calculating the occurrence frequency based on the frequency of occurrence and the amount of data; and identifying the first anomaly in the air data based on the occurrence frequency and historical standard frequencies.

[0025] For example, in this embodiment, the frequency of occurrence of all data from the previous two years (more than 20 million records) was calculated, specifically the frequency of each of the digits 1 to 9 appearing as the first digit, to obtain the historical standard frequency. The frequency of occurrence is calculated as (number of times the digit appears) / (total number of data points). Then, a bar chart can be created, displaying the frequencies of occurrence of digits 1-9 side-by-side or superimposed with the historical standard frequency to visually observe the differences between the two. Obvious deviations (such as abnormally low frequencies of digits 1 or 9, or abnormally high frequencies of digits 5 or 6) may indicate abnormal data.

[0026] In one possible implementation, based on Benford's Law, the first anomaly in the air data is identified according to the frequency of occurrence, including: for each digit, calculating the expected frequency based on historical standard frequencies and data volume; calculating the chi-square statistic based on the frequency of occurrence and expected frequency of each digit; and identifying the first anomaly in the air data based on the chi-square statistic.

[0027] To quantify the degree of deviation, this embodiment uses chi-square goodness-of-fit for statistical testing: ; where, expected frequency = historical standard frequency × total number of data points.

[0028] Determine significance: Degrees of freedom are 8 (because there are 9 numbers, and the sum is fixed). Select a significance level α (usually 0.05 or 0.01). Consult the chi-square distribution table to find the critical value. If the chi-square statistic is greater than the critical value, then at significance level α, the data is considered to be outlier.

[0029] Step 102: Perform time-series analysis on the air data to identify the second anomaly in the air data.

[0030] In one possible implementation, the temporal variation patterns of various air quality indicators can be obtained, and based on these patterns, a second anomaly in the air data can be identified.

[0031] For example: PM 2.5 and PM 10 Diurnal variation of the boundary layer: Due to the diurnal variation of the boundary layer, the boundary layer is low at night, and pollutants are compressed near the ground; after sunrise, the boundary layer rises, vertical diffusion is enhanced, and the concentration decreases; around sunset, the boundary layer lowers again, and pollutants accumulate. This respiration effect leads to the accumulation of primary pollutants such as PM2.5. 2.5 and PM 10 It often exhibits a bimodal distribution (morning peak, evening peak). If the PM levels at 6:00 and 18:00... 2.5 and PM 10 Values ​​below 12:00 and 0:00 are considered to indicate a possible anomaly.

[0032] PM 2.5 and PM 10 Seasonal changes in PM2.52.5 PM2.5 concentrations show a clear seasonal variation, with the highest concentrations in winter, followed by autumn and spring, and the lowest in summer. However, PM2.5 concentrations... 10 The peak levels differ between the north and south. In the south, where dust storms have a smaller impact, winter may see higher PM2.5 levels due to local dust and regional pollution transport. 10 The season with the highest PM2.5 concentration. If PM2.5 concentrations are highest in summer... 2.5 If the value is higher than the average of the previous winter, it is considered that there is a possibility of an anomaly.

[0033] Diurnal variation of O3: O3 is a secondary pollutant, and its precursors are nitrogen oxides (NOx). x and volatile organic compounds (VOCs) s Its formation depends on solar radiation. Therefore, its concentration usually peaks in the afternoon (13:00-17:00), in stark contrast to the trough of primary pollutants. If the O3 value is higher than the peak value of the previous peak period (16:00-19:00), it is considered to be possibly abnormal; if the O3 peak occurs during the peak period, it is considered normal.

[0034] The seasonal variation of O3 is characterized by lower levels in winter and higher levels in summer. This is because O3 is not directly emitted, but rather generated through photochemical reactions of nitrogen oxides and volatile organic compounds (VOCs) under strong sunlight and high temperatures. Summer's strong solar radiation and long hours of sunshine provide ample ultraviolet radiation to drive these photochemical reactions, while the high temperatures significantly accelerate the reaction rate and promote the volatilization of more VOCs, resulting in substantial Ozone formation. Winter, on the other hand, presents the opposite scenario. Weak sunlight and low temperatures cause photochemical reactions to almost cease, resulting in extremely slow reaction rates. Furthermore, the stable atmospheric stratification in winter, coupled with the large amounts of fresh nitrogen oxides generated at night by heating, further depletes the already limited Ozone reserves, causing O3 concentrations to remain consistently low throughout the winter. If summer O3 levels are lower than the average of the previous winter, an anomaly is considered possible; similarly, if winter O3 levels are higher than the average of the previous summer, an anomaly is also considered possible.

[0035] Diurnal variations of NO2 and CO: Human activity cycles, particularly the morning and evening rush hours, lead to increased vehicle emissions, causing double peaks in NO2 and CO concentrations during these times. If NO2 and CO levels at 9:00 and 18:00 are below 0:00, an anomaly is considered possible. If NO2 and CO levels between 21:00 and 7:00 are higher than those at 9:00 and 18:00, an anomaly is also considered possible.

[0036] In one possible implementation, a second anomaly in air data can also be identified using one or more time-series analysis models, including statistical models, machine learning models, and deep learning models.

[0037] For example, statistical models include the Z-score model, interquartile range model, and generalized ESD model; machine learning models include the Isolation Forest model; and deep learning models include the Long Short-Term Memory Network model.

[0038] Step 103: Calculate the correlation between various air quality indicators and identify the third anomaly in the air data based on the correlation.

[0039] The correlation detection of indicators is based on the physicochemical relationships and synergistic changes among air pollutants. Different pollutants have inherent interactions and transformation relationships in the atmospheric environment, and these relationships exhibit stable correlation patterns under normal conditions. Based on previous monitoring data, the correlations between various indicators can be identified, and subsequent data deviating from the previous patterns are considered abnormal data.

[0040] In one possible implementation, the correlation between various air quality indicators is calculated, and based on the correlation, a third anomaly in the air data is identified. This includes: calculating the correlation between various air quality indicators to generate a correlation coefficient matrix; identifying the third anomaly in the air data based on the correlation coefficient matrix; and performing dimensionality reduction on the air quality indicators using principal component analysis and calculating T. 2 The actual values ​​of the statistic and the Q statistic; based on T 2 The actual values ​​of the statistic and the Q statistic, and T 2 Historical standard values ​​of the statistic and Q statistic are used to identify the third anomaly in air data.

[0041] For example: (1) Collect historical data for a sufficiently long period of time (at least 1-2 years is recommended) to cover all relevant indicators and ensure that the data represents the normal operating status. Data sources include sensors, databases, log files, etc., and must include timestamps and indicator values.

[0042] (2) Preprocess the data, including data cleaning and standardization.

[0043] (3) Use Pearson correlation coefficient (linear relationship), Spearman rank correlation coefficient (monotonic relationship) or Kendall Tau (ordered data) to calculate the correlation between pairs of indicators and generate a correlation coefficient matrix.

[0044] (4) Use principal component analysis or factor analysis to identify the potential structure between key influencing factors and indicators. Specifically, based on correlation analysis, construct a principal component analysis model of the indicator relationships. Calculate T. 2 The statistic measures the degree of variation of the sample in the principal component space and detects abnormal deviations in the principal component directions; the Q statistic measures the boundary of the normal relationship defined by the sample reconstruction error and detects abnormal patterns in the residual space.

[0045] (5) Anomaly detection: Using T data from the previous two years 2 Using Q as a baseline, the new input data is arbitrary T. 2 If Q exceeds the standard, it is considered that there may be an anomaly.

[0046] General pattern detection, such as PM 2.5 It is PM 10 O3 and NO2 are physical subsets of each other, and are often released simultaneously in a single emission (such as coal combustion, dust, and vehicle exhaust). Therefore, in most cases, they exhibit a strong positive correlation. O3 and NO2 have one of the most classic relationships in atmospheric chemistry, namely the titration effect; O3 and NO2 are negatively correlated, and if O3 and NO2 are positively correlated, it is considered an anomaly.

[0047] Step 104: Based on the weather data within the target monitoring period, predict the reference air data, and based on the reference air data, identify the fourth anomaly in the air data.

[0048] By acquiring historical air and weather data (temperature, humidity, wind speed and direction, precipitation, etc.), we can identify the patterns in which weather affects air quality, and any subsequent data that deviates from these patterns is considered abnormal.

[0049] In one possible implementation, reference air data is predicted based on weather data within the target monitoring period, and a fourth anomaly is identified in the air data based on the reference air data. This includes: extracting weather features from the weather data, inputting the weather features into a pre-trained prediction model to obtain reference air data; calculating the residual distribution between the reference air data and the air data; calculating the residual standard deviation based on the residual distribution; and identifying the fourth anomaly in the air data based on the residual standard deviation.

[0050] For example, (1) Data preparation and preprocessing of air data: historical time series data obtained from environmental monitoring stations, including but not limited to: PM 2.5 PM 10 Hourly or daily average concentrations of SO2, NO2, CO, O3, etc.

[0051] Meteorological data: Data obtained from meteorological departments or professional meteorological service platforms for the same region and time scale, including: Temperature: daily average temperature, maximum / minimum temperature, and temperature difference.

[0052] Humidity: Relative humidity.

[0053] Wind: wind direction, wind speed, and wind force level.

[0054] Precipitation: amount of precipitation, duration of precipitation.

[0055] Other factors include air pressure, visibility, and cloud cover.

[0056] Ensure that air quality and meteorological data are fully synchronized in timestamps, matching data from the nearest and most representative meteorological station with air quality monitoring stations. Use methods such as interpolation, averaging of preceding and following times, or direct elimination to conduct preliminary statistical screening, removing obvious instrumental errors, and standardizing / normalizing the data.

[0057] (2) Pattern Discovery and Model Building The goal of this stage is to establish a functional relationship: Air Quality = f (Meteorological Conditions). Depending on the data characteristics and complexity, one or more models can be selected, such as: Multiple Linear Regression: For example, PM2.5... 2.5 The linear model is: a*temperature + b*humidity + c*wind speed + d*wind direction index + constant.

[0058] The generalized additive model is a better choice because it can better handle the nonlinear relationship between meteorological factors and pollutant concentrations.

[0059] The XGboos model can automatically capture complex nonlinear relationships and feature interaction effects, has high prediction accuracy, and can output feature importance, clearly indicating the contribution of each meteorological factor.

[0060] Neural networks: capable of fitting extremely complex relationships for massive amounts of data.

[0061] With a well-trained model, given any set of meteorological input conditions, the expected value (or expected range) of air quality indicators under normal conditions can be predicted.

[0062] (3) Screening for abnormal data: Define the abnormal threshold: Use the trained model to calculate the predicted value for historical normal data. Calculate the residual between the predicted value and the actual value: Residual = Actual concentration - Predicted concentration. Analyze the distribution of the residual, which is usually considered to follow a normal distribution.

[0063] This embodiment employs a four-model ensemble + consensus mechanism for screening. Four models make predictions independently, and the residuals of each model are calculated. The standard deviation of the residuals calculated during the training phase is used, with three times the standard deviation as a threshold. Any model whose residual exceeds the threshold is marked as an anomaly. The number of models marking each data point as an anomaly is counted, and the final anomaly is determined based on the consensus threshold. A possibility of an anomaly is judged if 50% of the models agree.

[0064] This invention, through calculating the frequency of occurrence of different first digits in air data and utilizing Benford's Law, identifies frequency deviations in large-scale air data caused by human intervention or systematic anomalies, effectively identifying overall data structure anomalies. By performing time-series analysis on air data to identify anomalous data points, it can quickly screen for structural anomalies in large datasets. Furthermore, by constructing a multi-indicator anomaly identification technology driven by pollutant synergistic patterns, it identifies anomalous data points based on the correlation between various air indicators. Finally, by acquiring weather data within the target monitoring period and predicting reference air data based on the weather's influence on air quality, it identifies anomalous points deviating from the established patterns. This invention forms a multi-perspective, multi-scale, and multi-mechanism small-model system covering key dimensions such as statistical characteristics, temporal patterns, spatial continuity, and physicochemical logic. Through parallel detection using multiple models, it can capture different types of anomalies from different angles, achieving comprehensive coverage of various data anomalies, including short-term mutations, slow drifts, structural anomalies, spatial jumps, and meteorological inconsistencies. Compared to traditional single-method approaches, this invention offers higher stability and efficiency when processing massive amounts of monitoring data, significantly improving the data cleaning and quality control capabilities of regulatory departments.

[0065] The statistical model used in step 103 of the above embodiment will be explained below.

[0066] Statistical models were first used for anomaly detection. Most of them are based on comparing statistical properties to check for outliers. During the training phase, the distribution parameters are optimized according to specific evaluation criteria. The resulting distribution is then bounded by probability thresholds.

[0067] Statistical models include: (1) Z-score model. The Z-score model is used to measure the deviation between data points and the mean of the dataset, and is standardized in units of standard deviation. Its core is to convert data of different scales into a uniform Z-score value through standardization, which is convenient for comparison and identification of anomalies.

[0068] The formula is: Where X is a single data point, μ is the mean of the dataset, and σ is the standard deviation.

[0069] Detection steps: Calculate the mean and standard deviation, then calculate the Z-score; set a threshold, and data points with Z-scores exceeding the threshold are considered abnormal.

[0070] (2) Interquartile range model The interquartile range method identifies outliers or analyzes the dispersion of data based on the quartiles of the data and the range of the data distribution.

[0071] The steps are as follows: Data sorting: Arrange the environmental monitoring data in ascending order; Calculate quartiles: Q1 (first quartile): 25% of the values ​​in the data are less than or equal to Q1. Q3 (third quartile): 75% of the values ​​in the data are less than or equal to Q3; Calculate IQR: IQR = Q3 - Q1; Determine the outlier range: Lower limit Q1 - 1.5 × IQR, upper limit Q3 + 1.5 × IQR. Values ​​below the lower limit or above the upper limit are considered outliers. Outliers may be measurement errors or extreme environmental events, and their removal or retention should be determined based on the actual situation. This method is not sensitive to outliers because IQR only depends on the middle 50% of the data, avoiding the influence of extreme values.

[0072] (3) Generalized ESD model; The generalized ESD test is applicable to normally distributed data. Its core idea is to determine whether the extreme values ​​in the data are outliers by iteratively detecting them.

[0073] The steps are as follows: Determine the test parameters: significance level (α): usually set to 0.05; maximum number of outliers (k): set according to the amount of data and the expected number of outliers; calculate the ESD statistic: for each data point x i The ESD statistic is the absolute deviation from the mean divided by the standard deviation; find the maximum ESD value: determine the maximum ESD value in the current dataset, and denot it as ESD. max Calculate the critical value: Use the t-distribution to calculate the critical value. ,in, It is the quantile of the t-distribution; comparison and judgment: if ESD max > If a point is found to be an outlier, it is identified as an outlier and removed. Repeat the above steps until there are no outliers or the maximum number of outliers, k, is reached.

[0074] The machine learning model used in step 103 of the above embodiment will be described below.

[0075] With breakthroughs in computer technology, machine learning algorithms have been increasingly applied to anomaly detection, such as support vector machines, decision trees, and random forests. A classic definition of machine learning is: if a computer program's performance on a task improves with experience, then it can be said that the program has learned from experience. These algorithms learn the characteristics of monitored data within a normal range through computer models and identify data that does not conform to normal characteristics as anomalies. These methods can handle more complex data patterns and adapt to data changes to some extent. Machine learning methods are more flexible when dealing with complex data and can adapt to various data patterns, but they often require longer training times, are sensitive to parameter tuning, and may experience overfitting with noisy data.

[0076] This embodiment uses an isolated forest. Isolated forests perform exceptionally well on large datasets, especially when dealing with high-dimensional data, achieving rapid convergence while maintaining high detection performance. Through unsupervised learning, they do not require known anomaly labels and can effectively handle multi-feature contaminant data.

[0077] Isolation forests leverage two characteristics of outliers: their small number and significant attribute value differences from normal points. Outliers are easier to isolate, resulting in shorter paths within the tree structure. The algorithm detects outliers by constructing multiple isolated trees, where outliers have shorter paths than normal points. Outlier scores in isolation forests typically range from -1 to 1 or 0 to 1. Scores close to -1 are highly likely to be outliers, close to 0 are boundary cases, and close to 1 are highly likely to be normal data. In the program, the modified threshold is set as follows: scores ≤ -0.2 indicate an outlier, and scores > -0.2 indicate normal data.

[0078] The following describes the deep learning model used in step 103 of the above embodiment.

[0079] Traditional methods have limited performance on high-dimensional or complex data, while deep learning, with its powerful feature learning capabilities, has become a new direction for solving anomaly detection problems. Deep learning methods train neural networks to learn the latent relationships in data, reconstruct input data or predict future data, and calculate reconstruction or prediction errors as anomaly indicators, thereby achieving anomaly detection. Traditional machine learning relies on manually designed feature extractors, while deep learning can automatically learn features from raw data, avoiding human intervention. Deep learning can handle high-dimensional, nonlinear, and complex data patterns and has strong generalization capabilities, but it requires a large amount of data and computational resources, and its performance on small datasets may not be as good as traditional methods. With the improvement of computing power and the popularization of deep learning technology, these methods have demonstrated powerful capabilities in handling high-dimensional, nonlinear, and complex data, becoming a current research hotspot in the field of anomaly detection.

[0080] This type of algorithm uses contextual information from time series or sequence data to predict future data points. This embodiment employs a Long Short-Term Memory (LSTM) network.

[0081] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network capable of handling long-term dependencies in time-series data. They capture patterns in time series data through memory units and gating mechanisms (input gate, forget gate, output gate) and truncate gradients without compromising their integrity.

[0082] This method detects outlier data by using a trained model to predict time series data and comparing the predicted values ​​with the actual values. If the difference exceeds a preset threshold, the data is considered outlier.

[0083] Based on the above, this application addresses the prominent issues in air monitoring data, such as diverse anomaly types, complex spatiotemporal patterns, significant equipment errors, coupled pollutant-related mechanisms, and difficulties in integrating multi-source data. It constructs a technical system and designs a large-scale model screening technology system for air monitoring, including anomaly identification technology based on data distribution patterns, intelligent screening technology based on time-space coupling characteristics, correlation discrimination technology based on physical and chemical laws and expert knowledge, and monitoring equipment error identification technology based on mechanistic models. This enables intelligent screening of monitoring data throughout the entire process and across multiple dimensions.

[0084] Specifically, this application is mainly divided into three technical parts: (1) Data distribution and statistical law-driven anomaly identification technology. Air monitoring data are affected by factors such as human interference, extreme weather, and equipment malfunction, and often show abnormal values ​​that deviate from the normal distribution characteristics. In order to overcome the problems of traditional statistical methods being sensitive to distribution assumptions and insufficient response to complex disturbances, this application constructs an anomaly identification system driven by data distribution characteristics based on Benford's law, statistical distribution analysis and composite anomaly testing methods.

[0085] First, a probability distribution model of the first digit is constructed using Benford's Law to identify frequency deviations in large-scale air monitoring data caused by human intervention or systematic anomalies, effectively identifying anomalies in the overall data structure. Second, statistical anomaly detection algorithms such as Z-score, interquartile range model, and generalized ESD are employed to identify outliers from three dimensions: outliers, distribution deviations, and extreme fluctuations, respectively, constructing a multi-statistic joint discrimination framework. This technology does not rely on specific scenario assumptions and can quickly screen for structural anomalies in large datasets, providing high-quality input for subsequent time series, correlation, and equipment modeling.

[0086] Furthermore, this application combines statistical regularity with machine learning models, utilizes algorithms such as isolated forests to identify abnormal patterns with high-dimensional nonlinear characteristics, and constructs abnormal boundaries through indicators such as model path length and abnormal scores. This effectively compensates for the insufficient adaptability of traditional statistical methods to complex patterns, forming a statistical-machine learning integrated anomaly identification technology system with broad applicability.

[0087] (2) Dynamic Anomaly Identification Technology Driven by Time Series Patterns: Air pollutants exhibit significant diurnal, seasonal, and long-term trend changes due to emission cycles, meteorological variations, and human activities. Traditional time series models struggle to simultaneously capture periodicity, abrupt changes, and lag effects. This application constructs a time series screening technology that integrates the advantages of statistics, machine learning, and deep learning. This technology introduces a long short-term memory network autoencoder deep learning time series prediction model for anomaly detection. Its core principle is based on sequence reconstruction error analysis, training the autoencoder to accurately reconstruct the time series of normal data. Abnormal sequences cannot be accurately reconstructed by the model, resulting in significant reconstruction errors, thereby screening out abnormal data. Simultaneously, a triple pattern library based on the diurnal variation, seasonal variation, and activity cycle of factors is constructed, such as PM2.5. 2.5 / PM 10 By comparing real-time data with a database of patterns such as the bimodal diurnal variation of pollutants, the afternoon peak cycle of O3, and the commuter peak patterns of NO2 / CO, the system enables the interpretation of time-series anomalies based on physical law matching. This system achieves multi-scale identification of long-term trends, short-term fluctuations, and temporary mutations in pollutants, forming an intelligent screening technology for time-series anomalies in air monitoring that is adaptable to various scenarios.

[0088] (3) Multi-indicator anomaly identification technology driven by pollutant physicochemical correlation and indicator synergy: There are clear physical, chemical and emission common correlations among atmospheric pollutants, such as PM2.5. 2.5 With PM 10 The particle size nesting relationship and the photochemical inverse correlation mechanism between O3 and NO2 are studied. Disruption of the synergistic change pattern of pollutants often indicates abnormal records, equipment malfunction, or human interference. This application constructs a multi-indicator anomaly identification technology driven by pollutant synergistic laws based on correlation analysis, principal component analysis, and residual space modeling. First, the correlation matrix between pollutants is calculated using Pearson, Spearman, and Kendall Tau methods, and principal component analysis is combined to extract major pollutant factors and potential structures, constructing a comprehensive indicator synergistic model. Second, T... 2 The Q statistic and Q-statistic respectively characterize the degree of deviation of monitoring data in the principal component space and residual space. When new data deviates from the historical baseline limit, it is characterized as a structural anomaly. This technique effectively overcomes the limitations of single-index models and realizes quantitative anomaly identification and interpretation based on the co-evolution law of pollutants.

[0089] This application has the following advantages over the prior art: (1) Compared with the traditional rule-based method: the transformation from single threshold judgment to multi-dimensional evidence fusion. Traditional air quality data screening relies heavily on manual experience rules (such as fixed thresholds, graded judgments, upper and lower limits of anomalies, etc.). Such methods are simple in structure, but cannot adapt to complex and ever-changing environmental scenarios, and are also difficult to handle anomalies caused by the superposition of multiple pollutants, multiple scenarios, and multiple factors. This application adopts a multi-model system (time series, correlation, weather, space, equipment mechanism, Benford's law, etc.) coupled with a large model for fusion judgment, realizing the transformation from "single-point threshold judgment" to "multi-source evidence reasoning", which greatly improves the accuracy, comprehensiveness and robustness of anomaly identification.

[0090] (2) Compared with single statistical models: Shift from univariate prediction to multi-structure collaborative monitoring. Existing technologies often use seasonal models or trend prediction to identify anomalies, but such single models are not sensitive to nonlinear, sudden, and structural anomalies, and are difficult to explain the source of anomalies. This application introduces statistical methods, deep learning methods, and structural models (such as correlation models and spatial distance models) to form a multi-dimensional anomaly capture capability, and achieves structured interpretation through large models, so that anomalies can not only be identified, but also traced and explained, greatly enhancing the usability of actual business.

[0091] (3) Compared with local analysis based on a single site: Shift from local judgment to regional spatial consistency judgment. Traditional methods usually rely on the site's own data to judge anomalies, lacking spatial coordination reference, which can easily misjudge real pollution processes as anomalies or ignore single-point data drift. This application realizes the detection of synchronization between sites, the construction of spatial residual fields, and the verification of spatial consistency through spatial distance model and analysis of the collaborative behavior of neighboring sites, so that anomaly identification is improved from "looking at a point" to "looking at the whole area", which significantly reduces misjudgment and missed judgment.

[0092] (4) Compared with quality control systems that rely on manual review: Shifting from experience-driven to intelligent and automated. Currently, data quality control in the industry relies heavily on manual review, which is labor-intensive, subjective, and unstable. This application realizes full-chain automation from data cleaning, initial screening, evidence generation, anomaly interpretation to final conclusion, which not only significantly reduces labor costs but also reduces human interference, making the quality control process more standardized, repeatable, and quantifiable.

[0093] (5) Compared with black-box models that rely solely on deep learning: From unexplainable to explainable and traceable. Although deep learning models have strong fitting capabilities, the black-box problem makes them difficult to implement in regulatory scenarios. This application realizes explainable anomaly discrimination logic through an architecture in which a small model is responsible for identification and a large model is responsible for interpretation, including: why it is abnormal, where the anomaly is, which rules it conflicts with, and what the possible sources are, thereby meeting the needs of environmental supervision for traceability, review and accountability.

[0094] (6) Compared with static quality control systems: The traditional quality control system is rigid and difficult to update, shifting from a one-time construction to an iterative and scalable system. The model system designed in this application supports: continuous addition or replacement of small models, dynamic expansion of the knowledge base of large models, rolling updates of the database over time, and rapid integration of new pollutants and monitoring factors. This gives the entire system long-term vitality and evolutionary capability, allowing it to continuously adapt to future weather changes, site expansion, and increases in monitoring factors.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0096] Figure 2 is a schematic diagram of an electronic device provided in an embodiment of the present invention. As shown in Figure 2, the electronic device 2 of this embodiment includes a processor 20 and a memory 21. The memory 21 stores a computer program 22. When the processor 20 executes the computer program 22, it implements the steps in the various method embodiments described above.

[0097] For example, computer program 22 may be divided into one or more modules / units, which are stored in memory 21 and executed by processor 20 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 22 in electronic device 2.

[0098] Electronic device 2 may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that FIG2 is merely an example of electronic device 2 and does not constitute a limitation on electronic device 2. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 2 may also include input / output devices, network access devices, buses, etc.

[0099] The processor 20 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0100] The memory 21 can be an internal storage unit of the electronic device 2, such as a hard disk or RAM. The memory 21 can also be an external storage device of the electronic device 2, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 21 can include both internal and external storage units of the electronic device 2. The memory 21 is used to store the computer program 22 and other programs and data required by the electronic device 2. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0101] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0102] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0103] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0104] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0105] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for anomaly identification in air data, characterized in that, include: Acquire air data within the target monitoring period; Calculate the frequency of occurrence of each different first digit in the air data, and based on the frequency of occurrence, identify the first anomaly in the air data according to Benford's Law; Perform time-series analysis on the air data to identify a second anomaly in the air data; Calculate the correlation between various air quality indicators, and identify a third anomaly in the air quality data based on the correlation. Based on the weather data within the target monitoring period, predict reference air data, and based on the reference air data, identify a fourth anomaly in the air data.

2. The method for anomaly identification of air data according to claim 1, characterized in that, The step of identifying the first anomaly in the air data based on the occurrence frequency and Benford's Law includes: for each digit, calculating the occurrence frequency based on the occurrence frequency and the amount of data; and identifying the first anomaly in the air data based on the occurrence frequency and historical standard frequencies.

3. The method for anomaly identification of air data according to claim 1, characterized in that, The step of identifying the first anomaly in the air data based on the occurrence frequency and Benford's Law includes: for each number, calculating the expected frequency based on historical standard frequencies and data volume; calculating the chi-square statistic based on the occurrence frequency and expected frequency of each number; and identifying the first anomaly in the air data based on the chi-square statistic.

4. The method for anomaly identification of air data according to any one of claims 1 to 3, characterized in that, The step of performing time-series analysis on the air data to identify a second anomaly in the air data includes: obtaining the time-series variation patterns of various air indicators, identifying the second anomaly in the air data based on the time-series variation patterns; and identifying the second anomaly in the air data using one or more time-series analysis models selected from statistical models, machine learning models, and deep learning models.

5. The method for anomaly identification of air data according to any one of claims 1 to 3, characterized in that, The step of calculating the correlation between various air quality indicators and identifying a third anomaly in the air data based on the correlation includes: calculating the correlation between various air quality indicators to generate a correlation coefficient matrix; identifying a third anomaly in the air data based on the correlation coefficient matrix; and performing dimensionality reduction on the air quality indicators using principal component analysis and calculating T. 2 The actual values ​​of the statistic and the Q statistic; based on T 2 The actual values ​​of the statistic and the Q statistic, and T 2 Historical standard values ​​of the statistic and Q statistic are used to identify a third anomaly in the air data.

6. The method for anomaly identification of air data according to any one of claims 1 to 3, characterized in that, The step of predicting reference air data based on weather data within the target monitoring period, and identifying a fourth anomaly in the air data based on the reference air data, includes: extracting weather features from the weather data, inputting the weather features into a pre-trained prediction model to obtain the reference air data; calculating the residual distribution between the reference air data and the air data; calculating the residual standard deviation based on the residual distribution; and identifying a fourth anomaly in the air data based on the residual standard deviation.

7. The method for anomaly identification of air data according to claim 4, characterized in that, The statistical models include: Z-score model, interquartile range model, and generalized ESD model; the machine learning model is the isolated forest model; and the deep learning model is the long short-term memory network model.

8. The method for anomaly identification of air data according to claim 6, characterized in that, The pre-trained prediction models include: multiple linear regression models, generalized additive models, XGboost models, and machine learning models.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images