Drainage pipe network sensor anomaly detection method based on self-adaptive soft measurement strategy

Abstract

The invention discloses a drainage pipe network sensor anomaly detection method based on an adaptive soft measurement strategy, and the method comprises the steps: screening upstream and downstream sensor data which strongly depends on a target sensor based on the analysis of a pipe network topological structure and the mutual information analysis of sensor data, and enabling the data to serve as the input characteristics of a soft measurement model; initial division of normal / abnormal data sets is carried out; a KAN-Attention deep learning algorithm is applied, a target sensor index soft measurement model is constructed and trained, and target sensor accurate soft measurement is realized; constructing a confidence interval historical data set based on actual measurement and soft measurement results of the sensor, judging the abnormal state of a single measurement result of the sensor by applying a confidence interval dynamic search method, and judging the abnormal state of the sensor according to the proportion of abnormal occurrence of continuous multiple measurements of the sensor; based on a dynamic moving window method of a probability density function, a confidence interval data set and a soft measurement model are adaptively updated to ensure long-term effectiveness of the method.

Description

Technical Field

[0001] This invention relates to the field of urban drainage, and more specifically to a method for detecting sensor anomalies in drainage pipe networks based on an adaptive soft measurement strategy. Background Technology

[0002] Sensors are the eyes of smart water systems, serving not only as the foundation for data collection but also as a core element in data-driven intelligent management and decision support. However, existing sensor systems still suffer from critical issues such as frequent equipment failures and data instability. For example, hardware and network malfunctions can easily lead to zero or constant sensor outputs; environmental factors, signal interference, and power supply failures can cause abnormal fluctuations in sensor output; and equipment aging and surface contamination can lead to sensor output drift and deviation. Therefore, it is urgent to implement sensor anomaly detection to ensure data reliability and overall system stability.

[0003] The following existing technologies were found through a search: The patent specification with publication number CN120602902A discloses a method and system for detecting anomalies in farmland wireless sensors based on a self-encoder, which is not applicable to the detection of anomalies in drainage pipe network sensors.

[0004] Patent specification CN112819087A discloses a method for anomaly detection of effluent BOD sensors in wastewater treatment based on a modular neural network. It uses density-based feature clustering to cluster wastewater quality variables and employs a mutual information-based method to select auxiliary variables to improve the detection accuracy of the effluent BOD sensor. Simultaneously, it designs a modular neural network with a sub-network structure based on error correction to perform real-time soft measurement of the effluent BOD sensor values ​​during wastewater treatment, thereby detecting anomalies in the effluent BOD sensor and aiming to promptly identify sensor abnormalities and improve the quality of wastewater treatment. However, this patented technical solution is not specifically designed for drainage pipe networks and does not utilize the sensor deployment characteristics of the drainage pipe network itself for sensor anomaly detection.

[0005] Currently, sensor anomaly detection methods still suffer from core problems such as strong reliance on fixed thresholds, poor model adaptability, and frequent false alarms and missed alarms. Specifically, these methods often rely on preset values ​​to determine whether sensor data is abnormal. However, drainage systems operate under complex conditions and are subject to changing environmental factors, making it difficult for fixed thresholds to adapt to data characteristics in different scenarios. At the same time, many detection models lack flexibility and are unable to cope with changes in system operating modes, leading to false alarms (identifying normal data as abnormal) and missed alarms (failing to detect actual anomalies) in practical applications. Summary of the Invention

[0006] To address the aforementioned technical problems and shortcomings in this field, this invention provides a drainage network sensor anomaly detection method based on an adaptive soft measurement strategy. This method enables accurate and dynamic detection of abnormal states of drainage network sensors, facilitating refined and intelligent management of drainage systems, improving the anomaly detection and alarm capabilities of drainage network sensors, and solving core problems of existing drainage network sensor anomaly detection methods, such as strong dependence on fixed thresholds, poor model adaptability, and frequent false alarms and missed alarms.

[0007] The specific technical solution is as follows: A method for detecting sensor anomalies in drainage pipe networks based on an adaptive soft-sensing strategy, comprising: Based on pipeline topology analysis and sensor data mutual information analysis, upstream and downstream sensor data strongly dependent on the target sensor are selected as input features for the soft measurement model, and an initial division of the normal / abnormal dataset is performed. The KAN-Attention deep learning algorithm is applied to construct and train the soft measurement model for the target sensor indicators, achieving accurate soft measurement of the target sensor. Based on the sensor's actual measurements and soft measurement results, a historical dataset of confidence intervals is constructed, and a dynamic confidence interval search method is applied to determine the abnormal state of a single sensor measurement result. The abnormal state of the sensor is determined by the proportion of abnormalities in multiple consecutive measurements (e.g., if the proportion of abnormalities in 100 consecutive measurements exceeds 20%, the sensor is considered abnormal). A dynamic moving window method based on the probability density function is used to adaptively update the confidence interval dataset and the soft measurement model to ensure the long-term effectiveness of the method.

[0008] In the drainage network sensor anomaly detection method based on adaptive soft measurement strategy of the present invention, the target sensor for anomaly detection can be any sensor in the drainage network.

[0009] Furthermore, the aforementioned method for detecting sensor anomalies in drainage pipe networks based on an adaptive soft measurement strategy specifically includes the following steps: (1) For the selected drainage network, analyze the network topology and sensor layout, select the target sensor to be detected for anomalies, identify the upstream and downstream sensors related to the target sensor in spatial location, and obtain the time series data of each sensor; based on the mutual information analysis method, further identify the upstream and downstream sensors that are strongly dependent on the target sensor as input features of the subsequent soft measurement model. (2) Based on the historical operating status of the target sensor, the normal state dataset and the abnormal state dataset are initially divided; based on the Kolmogorov-Arnold network coupled attention mechanism algorithm (KAN-Attention) deep learning algorithm, a soft measurement model of the target sensor index is constructed, and the model is trained and tested with the normal state dataset to achieve accurate soft measurement of the target sensor index; (3) Calculate the absolute difference between the measured results of the target sensor and the output results of the soft measurement model based on the normal state dataset, and construct a historical dataset of confidence intervals; after each measurement, apply the confidence interval dynamic search method to determine whether the sensor monitoring result is abnormal; further determine whether the sensor is abnormal by the proportion of abnormalities that occur in multiple consecutive monitoring of the sensor. (4) In practical applications, the normal state dataset and the historical confidence interval dataset are updated according to the anomaly detection results; based on the performance of the soft measurement model, the dynamic moving window method based on the probability density function is applied to adaptively update the soft measurement model to ensure the long-term effectiveness of the method.

[0010] Furthermore, in step (1), the upstream and downstream sensors related to the target sensor in terms of spatial location include any one or more of the following: water level sensor, water quantity sensor, and water quality sensor.

[0011] Furthermore, in step (1), the time resolution of the time series data acquired from each sensor is consistent.

[0012] The aforementioned anomaly detection method for drainage network sensors based on an adaptive soft measurement strategy can initially divide the normal and abnormal datasets primarily based on expert experience methods to identify zero or constant output values ​​caused by hardware failures, as well as obvious abnormal fluctuations, drift deviations, and abrupt changes. The total dataset duration should ideally be no less than one year.

[0013] Furthermore, the aforementioned drainage network sensor anomaly detection method based on adaptive soft measurement strategy applies mutual information analysis to screen the input features of the soft measurement model. First, it is necessary to obtain the probability distribution of time series data of each sensor, then calculate the mutual information between the time series data of each sensor and the target sensor, and screen the model features by evaluating the magnitude of the mutual information. Finally, it is preferable that the number of features does not exceed 10.

[0014] Furthermore, the aforementioned drainage network sensor anomaly detection method based on an adaptive soft sensing strategy applies Bayesian optimization to adjust hyperparameters of the soft sensing model, including the number of hidden layers, hidden layer units, attention heads, and learning rate, and trains the model to obtain the optimal soft sensing model. The coefficient of determination (R²) of the soft sensing model is then determined. 2 A score above 90% is ideal.

[0015] Furthermore, in the aforementioned drainage network sensor anomaly detection method based on an adaptive soft measurement strategy, the confidence interval historical dataset is stored in key-value pair format.

[0016] Furthermore, the aforementioned confidence interval dynamic search method calculates the distance between the new measurement value and each key in the historical confidence interval dataset after each sensor measurement, sorts these distances, selects several key-value pairs with the smallest distance (e.g., 100 pairs), and selects the largest value among these key-value pairs as the confidence interval for this measurement. If the absolute difference between the actual measurement result and the output result of the soft measurement model exceeds the confidence interval, the sensor measurement result is determined to be abnormal.

[0017] Furthermore, in practical applications, the anomaly detection method for drainage network sensors based on an adaptive soft measurement strategy can update the normal state dataset and the historical confidence interval dataset every month. Specifically, the aforementioned expert experience methods can be used for outlier removal.

[0018] Furthermore, in the aforementioned drainage network sensor anomaly detection method based on adaptive soft sensing strategy, the dynamic moving window method based on probability density function calculates the expected value of the target sensor soft sensing model output value within the moving window, further calculates the relative rate of change of the expected value, and when the relative rate of change of the expected value is higher than a set threshold (e.g., 5%), the soft sensing model is retrained with a new dataset to ensure the model accuracy and long-term effectiveness of the method.

[0019] Compared with the prior art, the beneficial effects of this invention are as follows: 1) Based on the analysis of pipeline topology and sensor data mutual information, this invention selects upstream and downstream sensor data that are strongly dependent on the target sensor as model input features, and applies the KAN-Attention deep learning algorithm to construct and train a soft measurement model of the target sensor index. Compared with other prediction models, it can improve the accuracy, generalization ability and robustness of the target sensor soft measurement.

[0020] 2) This invention constructs a historical dataset of confidence intervals and applies a dynamic search method for confidence intervals to determine the abnormal state of a single measurement result of a sensor, which can solve the problem of dependence on fixed static thresholds through big data.

[0021] 3) In practical applications, this invention will dynamically update the confidence interval dataset and, based on the dynamic moving window method of the probability density function, adaptively update the soft measurement model, thus ensuring the long-term effectiveness of the method. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a drainage network sensor anomaly detection method based on an adaptive soft measurement strategy according to the present invention.

[0023] Figure 2 This is a performance diagram of the soft measurement model in a specific implementation.

[0024] Figure 3 This is a graph of historical datasets for confidence intervals in a specific implementation method.

[0025] Figure 4 This is a diagram of a sensor malfunction during a specific implementation. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] See Figure 1 A method for detecting sensor anomalies in drainage pipe networks based on an adaptive soft measurement strategy, comprising the following steps: (1) For the selected drainage network, analyze the network topology and sensor layout, select the target sensor to be detected for anomalies, identify the upstream and downstream sensors related to the target sensor in spatial location, and obtain the time series data of each sensor; based on the mutual information analysis method, further identify the upstream and downstream sensors that are strongly dependent on the target sensor as input features of the subsequent soft measurement model. (2) Based on the historical operating status of the target sensor, the normal state dataset and the abnormal state dataset are initially divided; based on the KAN-Attention deep learning algorithm, a soft measurement model of the target sensor index is constructed, and the model is trained and tested with the normal state dataset to achieve accurate soft measurement of the target sensor index; (3) Calculate the absolute difference between the measured results of the target sensor and the output results of the soft measurement model based on the normal state dataset, and construct a historical dataset of confidence intervals; after each measurement, apply the confidence interval dynamic search method to determine whether the sensor monitoring result is abnormal; further determine whether the sensor is abnormal by the proportion of abnormalities that occur in multiple consecutive monitoring of the sensor. (4) In practical applications, the normal state dataset and the historical confidence interval dataset are updated according to the anomaly detection results; based on the performance of the soft measurement model, the dynamic moving window method based on the probability density function is applied to adaptively update the soft measurement model to ensure the long-term effectiveness of the method.

[0028] In the above-mentioned anomaly detection method for drainage network sensors based on adaptive soft measurement strategy, the target sensor for anomaly detection can be any sensor in the drainage network.

[0029] In step (1), the upstream and downstream sensors related to the target sensor in terms of spatial location include any one or more of the following: water level sensor, water quantity sensor, and water quality sensor.

[0030] In step (1), the time resolution of the time series data of each sensor is consistent.

[0031] In step (1), the mutual information analysis method is used to screen the input features of the soft sensor model. First, the probability distribution of the time series data of each sensor needs to be obtained. Then, the mutual information between the time series data of each sensor and the target sensor is calculated, as follows:

[0032] In the formula, It is the probability distribution function of the time series data of the target sensor. It is the probability distribution function of the time series data of the upstream and downstream sensors related to it. The probability distribution function is expressed by Gaussian kernel density estimation. It is the mutual information between sensors. Indicates the time step of the data. This indicates the total time step of the acquired data; model features are selected by evaluating the magnitude of mutual information, with the final number of features not exceeding 10.

[0033] In step (2), the initial division of the normal and abnormal datasets is mainly based on expert experience methods to identify zero or constant outputs caused by hardware failures, as well as obvious abnormal fluctuations, drift deviations, and mutation characteristics. The total duration of the datasets is no less than one year.

[0034] The target sensor index soft measurement model is built based on the KAN-Attention deep learning algorithm. Using the selected features as input and target sensor data as output, Bayesian optimization is applied to adjust hyperparameters such as the number of hidden layers, hidden layer units, attention heads, and learning rate, and the model is trained to obtain the optimal model. The soft measurement model's R-value is [not specified]. 2 It should be higher than 90%, as shown below:

[0035]

[0036] ...

[0037]

[0038] In the formula, It is the input vector of the soft measurement model. Time series data representing different characteristic indicators, Indicates the number of features. It is the output of the first layer KAN. and It is a univariate continuous function of the first-level KAN. It is the first The output of layer KAN, and It is the first A univariate continuous function of layer KAN. It is the first The output of layer KAN, , ; It is the attention mechanism layer. It is the output value of the soft measurement model.

[0039] Furthermore, in the above-mentioned anomaly detection method for drainage network sensors based on an adaptive soft measurement strategy, the historical dataset of confidence intervals is stored in key-value pair format, as shown below:

[0040] In the formula, It is a historical dataset of confidence intervals. These are the actual measurement results from the target sensor. It is the absolute difference between the measured results of the target sensor and the output results of the soft measurement model. It is the proportionality coefficient, with a value of 1.1. This indicates the total time step of the obtained data.

[0041] The aforementioned confidence interval dynamic search method calculates a new measurement value after each sensor measurement. Each key in the historical dataset with confidence intervals The distances are calculated and sorted. The 100 key-value pairs with the smallest distances are selected, and the largest value among these 100 key-value pairs is selected as the confidence interval for this measurement. If the absolute difference between the actual measurement result and the output result of the soft measurement model exceeds the confidence interval, the sensor measurement result is determined to be abnormal.

[0042]

[0043]

[0044]

[0045] This measurement was determined to be abnormal;

[0046] In the formula, For new measurement value Each key in the historical dataset with confidence intervals distance, The 100 key-value pairs with the smallest distance in the historical dataset of the confidence interval. Indicates the new measurement value The value data among the 100 key-value pairs with the smallest distance. This represents the confidence interval for the measurement results. This indicates the output of the soft measurement model.

[0047] In step (3), the sensor is judged to be abnormal by the proportion of abnormalities observed in multiple consecutive monitoring. Specifically, 100 consecutive monitoring results are selected. If the proportion of abnormalities in these 100 measurement results exceeds 20%, the sensor is judged to be abnormal.

[0048] Furthermore, in practical applications, the above-mentioned anomaly detection method for drainage network sensors based on an adaptive soft measurement strategy updates the normal state dataset and the historical confidence interval dataset every month. Specifically, the aforementioned expert experience methods can be used for outlier removal.

[0049] Furthermore, in the above-mentioned drainage network sensor anomaly detection method based on an adaptive soft measurement strategy, the dynamic moving window method based on the probability density function calculates the expected value of the target sensor soft measurement model output value within the moving window, and further calculates the relative rate of change of the expected value, as shown below:

[0050]

[0051] In the formula, It is the expected value of the target sensor soft measurement model output value within the current moving window. It is the expected value of the target sensor soft measurement model output value within the previous moving window. It is the expected relative rate of change. This refers to the size of the movable window, with a length of 1 month.

[0052] When the expected relative rate of change is higher than 5%, the soft sensor model is retrained with a new dataset to ensure model accuracy and long-term effectiveness of the method.

[0053] This case study is based on the IoT monitoring system of a combined sewer system in a certain city. A flow meter upstream of a booster pump station was selected as the target sensor for anomaly detection. The network topology and sensor deployment were analyzed to identify the upstream and downstream sensors spatially related to the target flow meter. The time resolution of the acquired sensor data was 1 hour, and the total data duration was 1 year. Using mutual information analysis, the upstream and downstream sensors strongly dependent on the target flow meter were identified as 2 flow meters, 5 level gauges, 2 conductivity sensors, and 1 ammonia nitrogen sensor. Furthermore, based on expert experience, zero or constant output values ​​caused by hardware failures, as well as obvious abnormal fluctuations, drift deviations, and abrupt changes, were identified, leading to the initial division of the normal and abnormal datasets.

[0054] The KAN-Attention deep learning algorithm was applied, with the above-selected features standardized as the model input and the target sensor data as the output. The hyperparameters were adjusted using Bayesian optimization, and the optimal hyperparameters are shown in Table 1 below.

[0055] Table 1

[0056] The prediction results obtained from the soft measurement model are as follows: Figure 2 As shown, R 2 The value can be higher than 90%.

[0057] Further, based on the normal state dataset, the absolute difference between the measured results of the target sensor and the output results of the soft measurement model is calculated, and multiplied by a scaling factor to construct a historical dataset of confidence intervals in key-value pair form, such as... Figure 3 As shown in the figure. During the application of the method, the normal state dataset and the historical confidence interval dataset are dynamically updated every month. At the same time, a dynamic moving window method based on the probability density function is applied to calculate the expected value of the target sensor soft measurement model output value and its relative rate of change within the moving window, and to adaptively update the soft measurement model to ensure the effectiveness of the method.

[0058] After each measurement, a dynamic confidence interval search method is applied to determine whether the sensor monitoring result is abnormal. Furthermore, the proportion of abnormalities observed in multiple consecutive monitoring cycles is used to determine if the sensor itself is malfunctioning. In this embodiment, a dynamic confidence interval search is performed on nearly 100 measurement results; if the proportion of abnormalities is as high as 90%, the sensor is determined to be malfunctioning.

[0059] Furthermore, it should be understood that after reading the above description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims.

Claims

1. A method for detecting sensor anomalies in drainage pipe networks based on an adaptive soft-sensing strategy, characterized in that, include: Based on pipeline topology analysis and sensor data mutual information analysis, upstream and downstream sensor data that are strongly dependent on the target sensor are selected as input features for the soft measurement model, and the normal / abnormal datasets are initially divided. The KAN-Attention deep learning algorithm is applied to construct and train the soft measurement model of the target sensor index to achieve accurate soft measurement of the target sensor. Based on the sensor's actual measurement and soft measurement results, a historical dataset of confidence intervals is constructed, and a dynamic search method for confidence intervals is applied to determine the abnormal state of a single sensor measurement result. The abnormal state of the sensor is determined by the proportion of abnormalities in multiple consecutive measurements. The dynamic moving window method based on the probability density function adaptively updates the confidence interval dataset and the soft measurement model to ensure the long-term effectiveness of the method.

2. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 1, characterized in that, The specific steps include: (1) For the selected drainage network, analyze the network topology and sensor layout, select the target sensor to be detected for anomalies, identify the upstream and downstream sensors related to the target sensor in spatial location, and obtain the time series data of each sensor; based on the mutual information analysis method, further identify the upstream and downstream sensors that are strongly dependent on the target sensor as input features of the subsequent soft measurement model. (2) Based on the historical operating status of the target sensor, the normal state dataset and the abnormal state dataset are initially divided; based on the KAN-Attention deep learning algorithm, a soft measurement model of the target sensor index is constructed, and the model is trained and tested with the normal state dataset to achieve accurate soft measurement of the target sensor index; (3) Calculate the absolute difference between the measured results of the target sensor and the output results of the soft measurement model based on the normal state dataset, and construct a historical dataset of confidence intervals; after each measurement, apply the confidence interval dynamic search method to determine whether the sensor monitoring result is abnormal; further determine whether the sensor is abnormal by the proportion of abnormalities that occur in multiple consecutive monitoring of the sensor. (4) In practical applications, the normal state dataset and the historical confidence interval dataset are updated according to the anomaly detection results; based on the performance of the soft measurement model, the dynamic moving window method based on the probability density function is applied to adaptively update the soft measurement model to ensure the long-term effectiveness of the method.

3. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 2, characterized in that, In step (1): The upstream and downstream sensors related to the target sensor in terms of spatial location include any one or more of water level sensors, water quantity sensors, and water quality sensors. The time resolution of the acquired time-series data from each sensor is consistent.

4. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 1 or 2, characterized in that, To screen input features for soft measurement models using mutual information analysis, the probability distribution of time-series data for each sensor must first be obtained. Then, the mutual information between the time-series data of each sensor and the target sensor is calculated. The model features are screened by evaluating the magnitude of the mutual information, with the final number of features not exceeding 10.

5. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 1 or 2, characterized in that, Bayesian optimization was applied to tune hyperparameters of the soft sensor model, including the number of hidden layers, hidden layer units, attention heads, and learning rate, and the model was trained to obtain the optimal soft sensor model. The test R of the soft sensor model was then performed. 2 Above 90%.

6. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 1 or 2, characterized in that, The confidence interval historical dataset is stored in key-value pairs.

7. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 6, characterized in that, The aforementioned confidence interval dynamic search method calculates the distance between the new measurement value and each key in the historical confidence interval dataset after each sensor measurement, sorts these distances, selects the key-value pairs with the smallest distance, and selects the largest value among these key-value pairs as the confidence interval for this measurement. If the absolute difference between the actual measurement result and the output result of the soft measurement model exceeds the confidence interval, the sensor measurement result is determined to be abnormal.

8. The drainage network sensor anomaly detection method based on adaptive soft measurement strategy according to claim 1 or 2, characterized in that, The dynamic moving window method based on probability density function calculates the expected value of the target sensor soft measurement model output value within the moving window, and further calculates the relative rate of change of the expected value. When the relative rate of change of the expected value is higher than a set threshold, the soft measurement model is retrained with a new dataset to ensure the model accuracy and long-term effectiveness of the method.

Images