An environmental protection terminal anomaly identification method based on clustering and data stream features

Abstract

This invention discloses an environmental protection terminal anomaly identification method based on clustering and data flow features. It adopts a two-layer serial architecture. The first layer, dynamic micro-cluster attenuation clustering layer, performs online clustering of multi-source sensor data streams through time-decayed quadruple cluster feature vectors, so that the cluster boundaries adaptively follow the normal concept drift and mark sufficiently isolated data points as candidate anomalies. The second layer, physical mechanism residual decoupling layer, embeds the material conservation constraint equation into the discrimination framework, combines the normalized physical linkage residual and feature smoothness penalty term, and performs three-class discrimination of candidate anomalies according to the joint dual-index decision rule: human active fraud, equipment passive failure, or normal concept drift. The discrimination result of normal drift is fed back to the first layer to update the micro-cluster boundary, realizing two-layer closed-loop adaptation. This invention is unsupervised throughout the process, does not require fraud labels, has low memory consumption, and the single-point update time complexity is O(1), which is fully adaptable to resource-constrained edge terminals.

Description

Technical Field

[0001] This invention belongs to the field of data anomaly detection technology, specifically involving a multi-type anomaly identification method for environmental monitoring edge terminals based on online data stream clustering and physical mechanism residual decoupling. It can be applied to online identification and classification of data anomalies in multi-sensor fusion environmental monitoring scenarios such as continuous emission monitoring systems (CEMS). Background Technology

[0002] With the deepening implementation of the "Regulations on the Administration of Discharge Permits", environmental automatic monitoring data has become the core basis for administrative law enforcement and discharge accounting. Discharging enterprises generally deploy continuous emission monitoring systems (CEMS), which simultaneously collect multiple signals through various sensors such as pollutant concentration analyzers, flue gas flow meters, oxygen content sensors, and flue gas temperature sensors. These signals have a strict linkage and conservation relationship in terms of physical and chemical mechanisms, and different variables cannot change independently or arbitrarily.

[0003] Existing anomaly detection technologies mainly suffer from the following shortcomings: First, both the static threshold rule-based method and the global static clustering-based method determine the decision boundary using offline historical data, which cannot adapt to the concept drift caused by seasonal changes and operating condition adjustments in the environmental data stream. This results in a large number of normal operating condition changes being falsely reported as abnormal, seriously undermining the credibility of the regulatory system.

[0004] Second, deliberate falsification by humans (such as elevating the sensor to compress the values ​​to an extremely stable level) and passive aging drift of the sensor (systematic data shift) are highly similar in the traditional time-series statistical feature space. Existing pure data-driven methods cannot distinguish them by utilizing multivariate physical linkages, resulting in the inability to assign blame for falsification or accurately locate the type of equipment failure.

[0005] Third, although deep learning methods have certain feature learning capabilities, their memory and computing power requirements far exceed the capacity limits of edge terminal gateways; moreover, real fake samples are extremely scarce in the industry, and supervised training cannot establish effective classification decision boundaries, making it impossible to deploy the model.

[0006] Fourth, existing univariate statistical process control methods (such as CUSUM and EWMA) cannot utilize the physical and chemical conservation relationships between multiple signals such as pollutant concentration, flue gas flow rate, and oxygen content to verify cross-modal anomalies, and are powerless against highly concealed multi-sensor collaborative fraud. Summary of the Invention

[0007] The purpose of this invention is to provide an environmental protection terminal anomaly identification method based on clustering and data flow characteristics, so as to solve the technical problem that existing technologies cannot accurately distinguish between three types of heterogeneous events—normal concept drift, passive equipment failure, and deliberate falsification—on resource-constrained edge terminals in an unsupervised manner.

[0008] This invention is implemented as follows: a method for identifying anomalies in environmental protection terminals based on clustering and data flow characteristics, employing a two-layer serial architecture, including a first-layer dynamic micro-cluster attenuation clustering layer and a second-layer physical mechanism residual decoupling layer; the method includes the following processing steps: The first layer performs online preprocessing and dynamic micro-cluster clustering on the multi-dimensional sensor data streams collected by the CEMS terminal. For each active micro-cluster, a quadruple cluster feature vector is maintained, which includes weighted sample counts, weighted linear sums of each dimension, weighted sums of squares of each dimension, and the most recently updated timestamp. An exponential time decay weight is introduced for the contribution of historical data points, so that the cluster boundaries are automatically updated as normal operating conditions change. When a newly arrived data point cannot be integrated into the extended neighborhood of any existing active micro-cluster, the data point is marked as a candidate anomaly and output to the second layer. The second layer receives the candidate anomalies, substitutes the original observed values ​​of the auxiliary variables at the corresponding time into the pre-constructed physicochemical conservation constraint equation, calculates the theoretical predicted value of the target variable, and then calculates the normalized physical linkage residual and the feature smoothness penalty term; based on the joint dual-index decision rule of the two, each candidate anomaly is judged as deliberate falsification, passive equipment failure or normal concept drift. For candidate points identified as normal concept drift, the second layer feeds back the identification result to the first layer, triggering the expansion update of the corresponding microcluster boundary, thus achieving closed-loop adaptation between the two layers.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: By introducing a time-decay micro-cluster clustering mechanism, the cluster boundary is automatically updated as the normal concept drifts, significantly reducing the false alarm rate caused by changes in normal operating conditions. By embedding the physical and chemical conservation constraint equation as a residual decoupler, automatic identification of human-initiated fraud and passive equipment failure is achieved for the first time under unsupervised conditions, effectively making up for the causal identification defects of pure data-driven methods. The micro-cluster summary structure only needs to store quaternary statistics, and the memory usage is normally maintained within 10MB. The time complexity of single-point update is O(1), which is fully compatible with resource-constrained edge terminals. No fake label data is required throughout the process. Unsupervised anomaly type identification is achieved through the counter-evidence logic of the physical mechanism of normal operating conditions, which has high industrial feasibility. Attached Figure Description

[0010] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2The flowchart shows the online clustering algorithm for the first layer of dynamic micro-cluster decay clustering. Figure 3 The flowchart for the discrimination of the second-layer physical mechanism residual decoupling layer is shown below. Figure 4 This is a timing diagram of the end-to-end data flow in a two-layer architecture; Figure 5 Architecture diagram for system edge deployment. Detailed Implementation

[0011] To further understand the invention's content, features, and effects, the following embodiments are provided, and detailed descriptions are given in conjunction with the accompanying drawings.

[0012] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0013] System Overall Architecture The environmental protection terminal anomaly identification method proposed in this invention adopts a two-layer serial architecture (see...). Figure 1 The first layer is the Dynamic Micro-cluster Decay Clustering Layer (hereinafter referred to as the MCAD layer), which is responsible for discovering potential candidate anomalies at the data flow level; the second layer is the Physical Mechanism Residual Decoupling Layer (hereinafter referred to as the PMRD layer), which is responsible for identifying the types of candidate anomalies output by the MCAD layer, and finally outputting three types of labels: normal operating condition drift, passive equipment failure, and deliberate human fraud.

[0014] Data preprocessing The multidimensional sensor data stream collected by the CEMS terminal includes multiple channels such as NOx concentration, SO2 concentration, flue gas flow rate, oxygen content, and flue gas temperature. The number of channel dimensions is denoted as d. For each newly arrived multidimensional sampling vector, Z-Score standardization is performed on each dimension. That is, the original observation value of each dimension is subtracted from the mean of the historical samples in the initialization window and then divided by the standard deviation of the corresponding historical samples to obtain the standardized multidimensional feature vector, which is used for subsequent clustering processing.

[0015] First layer: Dynamic micro-cluster decay clustering layer (MCAD layer) The MCAD layer maintains a quadruple cluster feature vector for each active micro-cluster in the data stream, which includes: the weighted total number N of aggregated data points in the micro-cluster, the weighted linear sum vector LS of each dimension, the weighted square sum vector SS of each dimension, and the timestamp of the most recent update T_last. Based on the above quadruple, the weighted centroid and root mean square radius of each dimension of the micro-cluster can be derived in real time without storing the original data, achieving incremental updates with O(1) time complexity.

[0016] To achieve automatic tracking of concept drift, an exponentially decaying weight is introduced for the contribution of each data point to the microcluster:

[0017] in, For the current moment During the evaluation, at any time The attenuation weight of the generated data points to their respective microclusters ranges from (0, 1]. The decay rate parameter controls the rate at which historical data is forgotten. The larger the value, the faster historical data is forgotten. The recommended value range is 0.01 to 0.1, and it should be calibrated according to the cycle of environmental and working conditions. This is the time interval between the current moment and the moment the data point was generated, with the unit consistent with the sampling frequency.

[0018] Based on the aforementioned attenuation weights, the effective weighted number of a micro-cluster is defined as the sum of the attenuation weights of all historical data points in that micro-cluster at the current moment. When the effective weighted number of a micro-cluster is lower than a preset sparsity threshold... At that time, the micro-cluster is marked as a sparse micro-cluster and removed from the active cluster set, allowing the model to automatically forget the old distribution, and the sparsity threshold is applied. The recommended value range is 1.0 to 5.0.

[0019] The online clustering process of the MCAD layer is as follows (see...) Figure 2 ): The initialization phase sets the upper limit for the number of active micro-clusters. (Recommended value: 50 to 200) Microcluster expansion coefficient (Recommended value: 1.5 to 3.0, used to control the absorption neighborhood range of microclusters), peripheral isolation judgment coefficient. (Recommended 3.0 to 5.0, used to distinguish edge-off points from significantly isolated points, must meet the following requirements) ), sparsity threshold Attenuation rate and periodic maintenance intervals (Recommended sampling points: 100 to 500) Initialize both the active micro-cluster set and the candidate anomaly set to empty sets, and set the internal time counter to zero.

[0020] For each newly arrived standardized multidimensional data point, calculate its Euclidean distance to the centroids of all active microclusters. Select the microcluster with the smallest distance as the nearest microcluster. If this distance does not exceed the mean root-mean-square radius of each dimension of the nearest microcluster and the microcluster expansion coefficient... The product of these factors is then used to merge the data point into the micro-cluster. The quadruple cluster feature vector is updated incrementally with a decay weight of 1 at the current time (N increases by 1, LS is added to the current normalized vector, SS is added to the element-wise square of the current normalized vector, and T_last is updated to the current time), and the normal label is output.

[0021] If this distance exceeds the mean root mean square radius of the nearest microcluster in each dimension and If the product of the two data points is used, a new micro-cluster is created centered on that data point, and its four-element feature vector is initialized (N is 1, LS and SS are initialized to the current normalized vector and their element-wise squares respectively, and T_last is the current time). If the number of currently active micro-clusters has reached the upper limit, then... If the data point is located at the lowest effective weighted cluster, the cluster with the lowest number of weighted points will be removed before a new cluster is added. Additionally, if the distance from the data point to the nearest cluster exceeds the mean of the root mean square radius of the nearest cluster in each dimension, then... If the product is calculated, the data point is added to the candidate anomaly set, assigned a verification label, and waited to be passed to the PMRD layer for type identification.

[0022] Each cumulative For each data point, periodic decay maintenance is performed on all active micro-clusters, ensuring the effective weighted number of points is below the sparsity threshold. Sparse microclusters are removed from the active cluster set.

[0023] Second layer: Physical mechanism residual decoupling layer (PMRD layer) The PMRD layer processing procedure is as follows (see...) Figure 3 ): First, a physicochemical conservation constraint equation is constructed. Taking NOx emission monitoring in CEMS as the core example, a theoretical prediction equation for NOx concentration is established based on the mass conservation and stoichiometric relationship of flue gas emissions. The theoretical prediction value is calculated based on the original measured value of NOx concentration, the measured value of oxygen content, and the oxygen content conversion relationship of standard reference oxygen content (a fixed value, such as 6%, is taken according to national standards such as GB 13271). Combined with the standard state temperature and pressure correction function of flue gas flow rate and flue gas temperature, the theoretical prediction value is calculated. The fixed parameters of this constraint equation are read from the equipment factory configuration file or national standard parameter library. In the initial observation window after system startup (it is recommended to collect at least 24 hours of normal operating data), the reference statistics of each variable are obtained.

[0024] For each candidate outlier output by the MCAD layer, the original observed values ​​of the auxiliary variables (flue gas flow rate, oxygen content, and flue gas temperature) at the corresponding time are substituted into the above physical constraint equations to calculate the theoretical predicted value. Then, the absolute value of the difference between the theoretical predicted value and the actual observed value of the target variable is taken, and divided by the mean of the observed values ​​of the target variable within the recent sliding window to obtain the normalized relative physical linkage residual.

[0025] in, for The dimensionless relative physical linkage residual at any given moment; for Actual observed NOx concentration at time; These are theoretical predictions derived from flue gas flow rate, oxygen content, and flue gas temperature using physical constraint equations. This is the mean of NOx concentration observations within the recent sliding window, used to eliminate the influence of dimensions. The larger this residual is, the more serious the deviation in the physical linkage between the target variable and the auxiliary variable.

[0026] Meanwhile, the feature smoothness penalty term is calculated and defined as the ratio of the standard deviation of the target variable within a local sliding window of length W with the current time as the endpoint to the reference standard deviation corresponding to the same window length under historical normal operating conditions. The smaller the ratio, the more abnormally stable the target variable is compressed within the local time period. The recommended sliding window length W is 30 to 120 sampling points. The reference standard deviation is obtained from historical data under normal operating conditions during the system initialization phase.

[0027] Based on the above two indicators, the PMRD layer classifies each candidate point according to the following joint dual-indicator decision rule (see...). Figure 3 ): When the relative physical linkage residual exceeds the physical residual discrimination threshold Furthermore, the feature smoothness penalty term is lower than the smoothness penalty discrimination threshold. When the physical linkage has been decoupled and the local fluctuations of the target variable have been abnormally compressed, it is determined to be fraudulent. When the relative physical linkage residual does not exceed And the feature smoothness penalty term is not less than When the physical linkage relationship is not obviously decoupled, the local standard difference of the target variable often increases (high frequency noise is enhanced) or a systematic covariance shift occurs, which is judged as an increase in equipment noise or an aging fault. When the relative physical linkage residual does not exceed Furthermore, the feature smoothness penalty term is at the lower bound threshold of smoothness. to When the candidate point is within the closed interval, it is re-marked as a normal concept drift. Although the MCAD layer marks it as a candidate anomaly deviating from the current microcluster, the physical residual and smoothness penalty are within the normal range, which is a normal environmental concept drift. The current microcluster boundary should be updated to absorb this type of drift. .

[0028] It can be calibrated based on the residual distribution under normal operating conditions using the principle of three times the standard deviation. This is the threshold for smoothness penalty; values ​​below this indicate that the data has been abnormally flattened.

[0029] Closed-loop feedback and threshold adaptive calibration For candidate points identified as normal concept drift, the PMRD layer feeds the results back to the MCAD layer, triggering an expansion update of the corresponding microcluster boundary. This allows the model to adaptively follow normal concept drift, achieving closed-loop adaptation between the two layers (see [link to relevant documentation]). Figure 1 (Mid-loop feedback interface).

[0030] For points determined to be deliberately falsified by humans, a falsification event record with a timestamp is generated, reported to the environmental protection supervision cloud platform via the MQTT protocol, and the evidence log is persistently stored locally; for points determined to be passive equipment failures, an automatic push of equipment maintenance work orders is triggered.

[0031] The system supports threshold adaptive calibration: after a fixed recalibration period... (Recommended to be done on a daily basis) Recalculate the distribution of relative physical linkage residuals and feature smoothness penalty terms on historical data that is deemed normal, and automatically update the data based on a 3-standard-deviation principle. and This avoids threshold failure due to environmental baseline drift after long-term operation.

[0032] System Deployment Architecture The algorithm of this invention runs on the edge monitoring terminal gateway of the sewage discharge enterprise, without relying on real-time inference computing power in the cloud (see...). Figure 5 The CEMS terminal sensors synchronously access multiple signals to the edge gateway via the RS485 / Modbus protocol. The edge gateway is based on an ARM Cortex-A series processor, configured with 256MB to 1GB of RAM, and runs a dual-layer algorithm module of MCAD and PMRD. The algorithm is deployed in the form of microservices and supports technical transformation through OTA firmware upgrades without the need to add new hardware. The final identification results and event logs are reported to the environmental supervision cloud platform via 4G / 5G network, while the timestamped and tamper-proof evidence records are stored locally.

Claims

1. A method for identifying anomalies in environmental protection terminals based on clustering and data flow features, characterized in that, It adopts a two-layer serial architecture, including a first layer of dynamic micro-cluster attenuation clustering and a second layer of physical mechanism residual decoupling; The first layer performs online clustering of multidimensional sensor data streams, maintains a quadruple cluster feature vector for each active micro-cluster, and introduces an exponential time decay weight for the contribution of historical data points. When a newly arrived data point cannot be integrated into the extended neighborhood of any existing active microcluster and is sufficiently isolated from all active microclusters, the data point is marked as a candidate outlier and output to the second layer; The second layer receives candidate outliers, substitutes the original observed values ​​of the auxiliary variables at the corresponding time into the pre-constructed physical and chemical conservation constraint equation, calculates the theoretical predicted value of the target variable, and then calculates the normalized physical linkage residual and feature smoothness penalty term. Based on the joint dual-index decision rule of the two, the candidate outliers are respectively judged as human-induced fraud, passive equipment failure or normal concept drift. For candidate points identified as normal concept drift, the second layer feeds back the identification result to the first layer, triggering the expansion update of the corresponding microcluster boundary.

2. The method according to claim 1, characterized in that, The quadruple cluster feature vector includes: the weighted total number N of aggregated data points in the micro-cluster, the weighted linear sum vector LS for each dimension, the weighted sum of squares vector SS for each dimension, and the timestamp T_last of the micro-cluster's most recent update; the weighted centroid and root mean square radius of each dimension of the micro-cluster are derived in real time based on the quadruple cluster feature vector. The exponential time decay weight is calculated as follows: at the current time... For time Weights are assigned to the generated data points. ,in This is the decay rate parameter; When the effective weighted points of a microcluster are lower than the preset sparsity threshold When this happens, the microcluster is removed from the active cluster set.

3. The method according to claim 1, characterized in that, The first layer processes each newly arrived standardized data point as follows: Calculate the Euclidean distance from the data point to the centroids of all active microclusters, and take the microcluster with the smallest distance as the nearest microcluster; If the distance does not exceed the mean root mean square radius of each dimension of the nearest microcluster and the microcluster expansion coefficient The product of the two values ​​is then used to merge the data point into the micro-cluster and incrementally update the quadruple cluster feature vector, outputting a normal label. If this distance exceeds the mean root mean square radius of each dimension of the nearest microcluster and If the distance exceeds the product of the root mean square radius of the nearest micro-cluster and the isolation judgment coefficient of the outer perimeter, then a new micro-cluster is created with the data point as the center and its four-element feature vector is initialized; if the distance still exceeds the product of the root mean square radius of the nearest micro-cluster and the outer isolation judgment coefficient, then the new micro-cluster is created with the data point as the center and its four-element feature vector is initialized. The product of these factors is then added to the candidate anomaly set, where... Greater than ; When the number of active microclusters has reached its limit When adding new microclusters, remove the microcluster with the lowest effective weighted points.

4. The method according to claim 1, characterized in that, Each cumulative For each data point, periodic decay maintenance is performed on all active micro-clusters, ensuring the effective weighted number of points is below the sparsity threshold. The sparse micro-clusters are removed from the active cluster set, and the effective weighted number is the sum of the decay weights of all historical data points in the micro-cluster at the current moment.

5. The method according to claim 1, characterized in that, The physicochemical conservation constraint equation is established based on the mass conservation and stoichiometric relationship of flue gas emissions. By converting the measured values ​​of flue gas flow rate and oxygen content with the standard reference oxygen content, and combining the standard state temperature and pressure correction function, the theoretical predicted value of the target pollutant concentration is calculated. The standard reference oxygen content is a fixed value according to the corresponding emission standard. The fixed parameters of the physicochemical conservation constraint equation are read from the equipment factory configuration file or the national standard parameter library, and the reference statistics are obtained from historical data under normal operating conditions in the initial observation window after system startup.

6. The method according to claim 1, characterized in that, The normalized physical linkage residual is the absolute value of the difference between the actual observed value and the theoretical predicted value of the target variable divided by the mean of the observed values ​​of the target variable within the recent sliding window; the larger the normalized physical linkage residual, the more serious the deviation in the physical linkage relationship between the target variable and the auxiliary variable. The feature smoothness penalty term is the ratio of the standard deviation of the target variable within a local sliding window ending at the current time to the reference standard deviation corresponding to the same window length under historical normal operating conditions. The reference standard deviation is statistically obtained during the system initialization phase. The smaller the feature smoothness penalty term, the more abnormally stable the target variable is compressed within a local time period.

7. The method according to claim 1, characterized in that, The joint dual-indicator decision-making rule includes the following three rules: When the normalized physical linkage residual exceeds the physical residual discrimination threshold Furthermore, the feature smoothness penalty term is lower than the smoothness penalty discrimination threshold. At that time, it was determined to be a deliberate act of falsification by a person; When the normalized physical linkage residual does not exceed And the feature smoothness penalty term is not less than When this occurs, it is determined to be a passive equipment failure; When the normalized physical linkage residual does not exceed And the feature smoothness penalty term is at the lower bound threshold of smoothness. to When within a closed interval, it is relabeled as a normal concept drift, where Less than .

8. The method according to claim 7, characterized in that, After each fixed recalibration period, the distribution of the normalized physical linkage residual and the feature smoothness penalty term is recalculated on historical data that is judged to be normal, and the physical residual discrimination threshold is automatically updated according to the principle of 3 times the standard deviation. and the smoothness penalty discrimination threshold .

9. The method according to claim 1, characterized in that, Before the first layer of processing, Z-Score standardization is performed on each dimension of the multidimensional sensor data stream. This involves subtracting the mean of the historical samples in the initialization window from the original observation value of each dimension and then dividing by the standard deviation of the corresponding historical samples to obtain the standardized multidimensional feature vector.

10. An environmental protection terminal anomaly identification system implementing the method of any one of claims 1 to 9, characterized in that, include: The physical sensing layer contains various types of sensors in the continuous emission monitoring system for flue gas, and outputs multiple sensor signals to the edge computing layer via the RS485 / Modbus protocol; The edge computing layer includes a data access and preprocessing module, a dynamic micro-cluster attenuation clustering engine, a physical mechanism residual decoupling engine, a result post-processing and reporting module, and a local evidence log storage module. A closed-loop feedback interface is provided between the dynamic micro-cluster attenuation clustering engine and the physical mechanism residual decoupling engine. The cloud-based monitoring layer receives the identification results and event logs reported by the edge computing layer via a 4G / 5G network, and sends algorithm updates to the edge computing layer via the OTA firmware upgrade channel.

Images