A data-driven method for anomaly detection and recovery of water network sensors

Abstract

This invention discloses a data-driven method for anomaly detection and recovery of water network sensors. It constructs minimum-order observers for sensors in the water network, solves the parameter matrix and online update rules of each minimum-order observer using historical data of the water network, and in practical use, inputs real-time input and output data of the water network into the minimum-order observers to obtain a set of real-time estimates. The absolute error of the real-time estimates is used to determine whether the sensor is abnormal. For abnormal sensors, the output data of normal sensors and real-time input data are used to reconstruct their measurement data, thus achieving anomaly recovery. Without needing to obtain a precise dynamic model of the water network and edge weight information, rapid sensor anomaly detection and data recovery can be achieved simply by constructing observers using historical input and output data, providing an efficient and robust solution for water network sensor monitoring.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of water network monitoring, sensors and data-driven control, and specifically relates to a data-driven method for anomaly detection and recovery of water network sensors. Background Technology

[0002] Water networks, as a core component of urban infrastructure and industrial production, are a key part of the national infrastructure system, playing a fundamental and strategic supporting role in urban operation and industrial development. They not only ensure a stable supply of water for urban residents and industrial production, but also undertake multiple functions such as flood control and drainage, water ecosystem restoration, and optimal allocation of water resources, serving as a crucial guarantee for achieving sustainable urban development and efficient industrial operation. Water networks are composed of natural river and lake systems and artificial water conveyance and distribution projects (such as reservoirs, pumping stations, and water pipelines), forming a three-tiered system of backbone networks, regional networks, and local networks, covering urban and rural areas and connecting regions, breaking the bottleneck of uneven spatial and temporal distribution of water resources.

[0003] The stable operation of a water network depends on the real-time monitoring of key physical quantities (such as node water pressure, pipeline flow, and water quality parameters) by sensors. However, sensors in a water network are susceptible to factors such as harsh environments, equipment aging, and malicious interference, resulting in abnormal situations such as data offset, interruption, and distortion. If abnormal data cannot be detected and recovered in a timely manner, it will lead to serious problems such as scheduling decision errors, missed reporting of leakage accidents, and decreased system operating efficiency.

[0004] Traditional methods for detecting anomalies in water network sensors can be broadly categorized into two types: One type is model-based methods, which establish a hydraulic dynamic model of the water network and combine it with techniques such as Kalman filtering and state estimation to determine anomalies. However, these methods rely on precise system parameters (such as pipe length, diameter, and roughness) and a complete network topology. In reality, the edge weights of water networks (such as pipe hydraulic resistance) are often difficult to obtain accurately, and the network structure may dynamically change according to operating conditions, leading to decreased model accuracy and insufficient reliability in anomaly detection. The other type is data-statistical methods, which analyze the temporal characteristics of sensor data (such as mean, variance, and trend) or the correlation of multi-sensor data to achieve anomaly detection. However, these methods are sensitive to missing data and struggle to distinguish between sensor anomalies and actual changes in the water network's operating conditions, resulting in a high false detection rate.

[0005] In recent years, the development of data-driven control technology has provided new avenues for model-independent anomaly detection. The minimum-order function observer, as a core tool for data-driven state estimation, offers advantages such as no system identification required, direct construction from input and output data, the lowest order (equal to the objective function's state dimension), and high computational efficiency. It can achieve real-time estimation of target sensor data without relying on a precise water network model. However, there is currently no technical solution for applying the minimum-order function observer to anomaly detection and recovery of water network sensors. There is an urgent need to develop a targeted method to address the problems of model dependence, weak anti-interference capabilities, and poor data recovery performance in existing technologies. Summary of the Invention

[0006] In view of this, the present invention provides a data-driven method for anomaly detection and recovery of water network sensors, which achieves rapid detection and data recovery of anomalies of water network sensors without obtaining the precise dynamic model and edge weight information of the water network.

[0007] This invention provides a data-driven method for anomaly detection and recovery of water network sensors, comprising the following steps:

[0008] Step 1: Sensor arrays and control devices deployed in the water network collect historical data of the water network. After removing measurement noise, the historical data is divided into past data segments and future data segments, and a Hankel matrix of input, output and target state is constructed.

[0009] Step 2: Establish the minimum-order observer of the sensor and solve for the parameter matrix of the minimum-order observer using historical data. Make it satisfy , solution ,in, , , These are the data weights for past inputs, past outputs, and future outputs, respectively. For recursive static parameters, , for Moore-Penrose pseudo-reverse, For any matrix, For future sensor data matrices, For the past input data matrix, For the past output data matrix, To output a data matrix for the future, This is a matrix of past sensor data;

[0010] The online update rule for the least-order observer of the sensor is as follows:

[0011]

[0012] in Let be the estimated value of the sensor data at time t. For the real-time input data at time t, This represents the real-time output data at time t. for Real-time output data at all times For sensor data in The estimated value of the time;

[0013] Step 3: In actual use, input the real-time input data of the water network and the output data of different sensors into the corresponding minimum-order observer to obtain multiple real-time estimated values ​​of the sensor measurement data. Calculate the absolute error between the estimated values ​​to identify abnormal sensors. Retain the output data and real-time input data of normal sensors. Based on this data, use the minimum-order observer of the abnormal sensor to reconstruct its actual measurement data. Output the sensor anomaly detection results and the complete monitoring data after recovery.

[0014] Furthermore, the historical data of the water network includes input data and output data. The input data is controllable data generated by the control equipment, including water pump operating pressure, valve opening adjustment signals, and reservoir water level replenishment. The output data is sensor measurement data, including water pressure, pipeline flow rate, and cumulative flow rate at each monitoring node.

[0015] Furthermore, the recursive static parameters in the parameter matrix are verified. Does it satisfy Schur stability? If Schur stability is not satisfied, adjust any matrix. Optimize the parameter matrix.

[0016] Furthermore, the recursive static parameters in the verification parameter matrix The way to determine whether Schur stability is satisfied is by: detection The detectability, among which, for Pseudo-inverse correspondence submatrix, for In the left null basis matrix Corresponding part.

[0017] Furthermore, the anomaly threshold is based on the statistical distribution of observation errors in historical normal data, using... Criterion setting.

[0018] Furthermore, the abnormal threshold is dynamically adjusted using an adaptive threshold mechanism.

[0019] Furthermore, the method for reconstructing the actual measurement data based on this data using a minimum-order observer of the anomaly sensor is as follows:

[0020]

[0021] in, For reconstruction Real-time target status sensor data, for Real-time output data from a constantly functioning sensor. for Real-time output data from a constantly functioning sensor. for Real-time target status sensor data.

[0022] Furthermore, verify the reconstructed data. The rationality of the data includes: checking the consistency between the reconstructed data and the hydraulic constraints of the water network; calculating the similarity between the reconstructed data and the historical normal data of the same period; and adjusting the observer parameters or supplementing historical data for retraining if the reconstruction error exceeds the preset threshold, until the recovery accuracy requirements are met.

[0023] Furthermore, the method for determining the abnormal sensor by the absolute error between the calculated estimated values ​​is as follows: if the absolute error is greater than the abnormal threshold and the duration is greater than the preset duration, the sensor corresponding to the lowest-order observer where the abnormal data is located is abnormal and an alarm is triggered; if the absolute error is greater than the abnormal threshold and the number of occurrences within the set time is less than the set value, it is an instantaneous interference and no alarm is triggered.

[0024] Furthermore, truncated singular value decomposition is used to remove measurement noise from historical water network data.

[0025] Beneficial effects:

[0026] This invention acquires historical data of a water network, including input and output data, to construct minimum-order observers for sensors within the network. The historical data is used to solve for the parameter matrices and online update rules of each minimum-order observer. In practical use, the acquired real-time input data and a set of output data from the water network are used as inputs to the minimum-order observers of the sensors to obtain a set of real-time estimates. The relationship between the absolute error of the real-time estimates and the error threshold is used to determine whether the sensors are abnormal. For abnormal sensors, the output data of normal sensors and real-time input data are used as inputs to the minimum-order observers of those abnormal sensors, reconstructing the measurement data of the abnormal sensors and achieving anomaly recovery. Without needing to acquire a precise dynamic model of the water network and edge weight information, rapid detection and data recovery of sensor anomalies can be achieved solely through the construction of observers using historical input and output data, providing an efficient and robust solution for water network sensor monitoring. Attached Figure Description

[0027] Figure 1This is a schematic diagram of the topology of a water network containing 97 nodes.

[0028] Figure 2 This is a graph showing the water level under normal operating conditions and the estimated values ​​from four observers in the example.

[0029] Figure 3 This is a graph showing the estimated values ​​of the observer after a sensor malfunction, using a data-driven water network sensor anomaly detection and recovery method provided by the present invention in this embodiment. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] This invention provides a data-driven method for anomaly detection and recovery of water network sensors. The core idea is to acquire historical data of the water network, including input and output data, to construct minimum-order observers for the sensors in the water network, and to use historical data to solve for the parameter matrices and online update rules of each minimum-order observer. In practical use, the acquired real-time input data of the water network and a set of output data are used as inputs to the minimum-order observers of the sensors to obtain a set of real-time estimated values. The relationship between the absolute error of the real-time estimated values ​​and the error threshold is used to determine whether the sensors are abnormal. For abnormal sensors, the output data and real-time input data of normal sensors are used as inputs to the minimum-order observers of the abnormal sensors to reconstruct the measurement data of the abnormal sensors, thereby achieving anomaly recovery.

[0032] This invention provides a data-driven method for anomaly detection and recovery of water network sensors, specifically including the following steps:

[0033] Step 1: Collect historical data of the water network, including input and output data, through sensor arrays and control equipment deployed in the water network. Input data includes controllable data generated by control equipment such as water pump operating pressure, valve opening adjustment signals, and reservoir water level replenishment. Output data includes sensor measurement data such as water pressure, pipeline flow, and cumulative flow value at each monitoring node. Truncate singular value decomposition (SVD) is used to remove measurement noise from the collected historical data. The historical data is then divided into past data segments and future data segments to construct a Hankel matrix for the input, output, and target states of the observer design, thus completing the preprocessing of the historical data.

[0034] The output data must include multiple sets of time-series data under normal operating conditions, and the data length must satisfy the continuous excitation (PE) condition, i.e., the data length T satisfies... , where n is the water network state dimension and m is the input dimension.

[0035] In the preprocessing of historical data, the historical data is divided into past data segments and future data segments. The past data segment includes the past input data matrix. Past output data matrix Past target sensor data matrix The future data segment includes the future output data matrix. Future target sensor data matrix The constructed Hankel matrices for the input, output, and target states are respectively... , , and verify If the conditions are met, it means a function observer can be constructed; otherwise, the data is changed and the observer is constructed again. This indicates that two Hankel matrices are stacked column-wise.

[0036] Step 2: Establish a minimum-order observer for the sensors in the water network sensor array. Construct a data matrix using preprocessed historical data, and solve for the parameter matrix of the minimum-order observer using the Moore-Penrose pseudo-inverse. To satisfy ,in , , These are the data weights for past inputs, past outputs, and future outputs, respectively. For the recursive static parameters of the least-order observer, The future target sensor data matrix; the solved parameter matrix is ,in, , for Moore-Penrose pseudo-reverse, It is any matrix;

[0037] Verification parameter matrix Does it satisfy Schur stability, i.e., the magnitude of all eigenvalues ​​is less than 1? If Schur stability is not satisfied, adjust any matrix. Optimize the parameter matrix to ensure asymptotic convergence of the observer error;

[0038] The minimum-order observer of the target sensor is generated based on the obtained parameter matrix, and its online update rule is as follows:

[0039]

[0040] in Let be the estimated value of the target sensor data at time t. For the real-time input data at time t, Let be the real-time output data of the target sensor at time t. for Real-time output data at all times For target sensor data in The estimated value of the time.

[0041] Furthermore, in the parameter matrix The verification method for whether Schur stability is satisfied is: detection The detectability, among which, for Pseudo-inverse correspondence submatrix, for In the left null basis matrix Corresponding part.

[0042] Step 3: In actual use, obtain real-time input data of the water network. and output data from different target sensors By inputting this data into the least-order observer of the corresponding target sensor, multiple real-time estimates of the target sensor's measurement data can be obtained. Calculate the estimated value absolute error between ;like If the duration exceeds a preset time, it is determined that the target sensor corresponding to the lowest-order observer where the abnormal data is located is abnormal and an abnormality alarm is triggered. If an event occurs only once within the set time range, it is considered a transient disturbance and no abnormal alarm is triggered. This is the abnormal threshold.

[0043] Furthermore, the abnormal threshold It can be based on the statistical distribution of observation errors from historical normal data, and adopt... The criteria can be set, or dynamically adjusted using an adaptive threshold mechanism. The preset duration can be 3 sampling periods.

[0044] Step 4: After identifying the abnormal sensor, retain the observation data from other normal sensors and the real-time input data from the water network, isolate the data from the abnormal sensor, and reconstruct the actual measurement data of the abnormal sensor based on the output data and real-time input data of the normal sensors using the constructed minimum-order observer of the abnormal sensor. Specifically:

[0045]

[0046] in, To reconstruct the obtained target state sensor data, This is the real-time output data of a normal sensor.

[0047] Furthermore, verify the reconstructed data. The rationality of the data includes: checking the consistency between the reconstructed data and the hydraulic constraints of the water network (such as the conservation of nodal flow); calculating the similarity between the reconstructed data and the normal data of the same period in history; if the reconstruction error exceeds the preset threshold, adjusting the observer parameters or supplementing historical data for retraining until the recovery accuracy requirements are met.

[0048] Step 5: Output the sensor anomaly detection results and the complete monitoring data after recovery, providing reliable data support for water network scheduling and fault diagnosis.

[0049] The Yangtze River basin constitutes a vast network of waterways in my country, comprised of the Yangtze River itself, the Yalong River, the Min River, the Jialing River, the Wujiang River, and numerous other rivers, forming a complex, tree-like drainage system. It is an organic combination of natural waterways and man-made water conservancy projects. Its source is the Tuotuo River, originating from the Qinghai-Tibet Plateau, serving as the primary source of the entire Yangtze River system. The water flows along the main channel through numerous measurable water level nodes, primarily large-scale water conservancy projects such as the Three Gorges Dam, Danjiangkou Dam, Gezhouba Dam, and the cascade reservoirs of the lower Jinsha River. These reservoirs are crucial control hubs, equipped with sophisticated water level and flow monitoring systems, whose real-time data provides strong support for core functions such as flood control, power generation, and navigation. Simultaneously, numerous tributaries and the main channel converge at their points, forming a vast number of natural confluence nodes within the network, such as Cuntan in Chongqing (where the Yangtze and Jialing Rivers meet) and Wuhan (where the Yangtze and Han Rivers meet). These nodes are mostly naturally formed complex river network structures, and usually do not have the precise water level monitoring capabilities of reservoirs.

[0050] The water network integrates a single natural source, precisely controllable key nodes, and complex natural confluence points into a dynamically balanced whole. However, the Yangtze River's main stream is long and its basin is vast, with intricate tributaries. Although hydrological monitoring systems have been deployed at key locations such as the confluence of important tributaries and reservoir dams, achieving real-time, comprehensive monitoring of water levels across the entire basin remains a significant challenge.

[0051] This invention focuses on the water network of the Yangtze River basin. It uses the water flow of the Tuotuo River on the Qinghai-Tibet Plateau as the input to the water network, and the water level measurements of important tributary confluence points and reservoir dams as known output information. The water level conditions of some remote, difficult-to-measure, and potentially unsafe tributary confluence points within the Yangtze River basin are considered as unknown target output information. The aim is to achieve real-time estimation of water levels at unknown nodes, thereby effectively preventing floods.

[0052] Example:

[0053] This embodiment takes four sensors in a water network as an example to verify the multi-vehicle distributed continuous coverage trajectory planning method based on particle swarm optimization algorithm provided by the present invention. The specific process is as follows:

[0054] In setting up a water network, the topology of the water network is as follows: Figure 1 As shown, minimum-order observers are constructed for the four selected sensors (Sensors 1-4) to estimate the water level at the target node. When all four sensors are functioning normally, the estimates from the four minimum-order observers are as follows: Figure 2 As shown in the figure, the unmarked curve represents the actual water level, the square-marked curve represents the estimate of the target state based on the observer based on sensor 1, the circular-marked curve represents the estimate of the target state based on the observer based on sensor 2, the triangular-marked curve represents the estimate of the target state based on the observer based on sensor 3, and the diamond-marked curve represents the estimate of the target state based on the observer based on sensor 4. All of them can correctly estimate the actual water level. If sensor 1 malfunctions, but the other sensors continue to work normally, the estimates of the four observers will be as follows: Figure 3 As shown (with the same markings) Figure 2 When sensor 1 malfunctions, its observer estimate deviates (marked by a square curve), while the estimates of the other observers remain stable and correct. Therefore, while maintaining a correct estimate of the target water level, the abnormal sensor 1 is identified.

[0055] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data-driven method for anomaly detection and recovery of water network sensors, characterized in that, Includes the following steps: Step 1: Sensor arrays and control devices deployed in the water network collect historical data of the water network. After removing measurement noise, the historical data is divided into past data segments and future data segments, and a Hankel matrix of input, output and target state is constructed. Step 2: Establish the minimum-order observer of the sensor and solve for the parameter matrix of the minimum-order observer using historical data. Make it satisfy , solution ,in, , , These are the data weights for past inputs, past outputs, and future outputs, respectively. For recursive static parameters, , for Moore-Penrose pseudo-reverse, For any matrix, For future sensor data matrices, For the past input data matrix, For the past output data matrix, To output a data matrix for the future, This is a matrix of past sensor data; The online update rule for the least-order observer of the sensor is as follows: ， in Let be the estimated value of the sensor data at time t. For the real-time input data at time t, This represents the real-time output data at time t. for Real-time output data at all times For sensor data in The estimated value of the time; Step 3: In actual use, input the real-time input data of the water network and the output data of different sensors into the corresponding minimum-order observer to obtain multiple real-time estimated values ​​of the sensor measurement data. Calculate the absolute error between the estimated values ​​to identify abnormal sensors. Retain the output data and real-time input data of normal sensors. Based on this data, use the minimum-order observer of the abnormal sensor to reconstruct its actual measurement data. Output the sensor anomaly detection results and the complete monitoring data after recovery.

2. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, The historical data of the water network includes input data and output data. The input data is controllable data generated by the control equipment, including water pump operating pressure, valve opening adjustment signals and reservoir water level replenishment. The output data is sensor measurement data, including water pressure, pipeline flow and cumulative flow value of each monitoring node.

3. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, Verify the recursive static parameters in the parameter matrix Does it satisfy Schur stability? If Schur stability is not satisfied, adjust any matrix. Optimize the parameter matrix.

4. The method for anomaly detection and recovery of water network sensors according to claim 3, characterized in that, The recursive static parameters in the verification parameter matrix The way to determine whether Schur stability is satisfied is by: detection The detectability, among which, for Pseudo-inverse correspondence submatrix, for In the left null basis matrix Corresponding part.

5. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, The anomaly threshold is based on the statistical distribution of observation errors in historical normal data, using... Criterion setting.

6. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, The abnormal threshold is dynamically adjusted using an adaptive threshold mechanism.

7. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, The method for reconstructing the actual measurement data using the least-order observer of the anomaly sensor based on this data is as follows: ， in, For reconstruction Real-time target status sensor data, for Real-time output data from a constantly functioning sensor. for Real-time output data from a constantly functioning sensor. for Real-time target status sensor data.

8. The method for anomaly detection and recovery of water network sensors according to claim 7, characterized in that, Validate reconstructed data The rationality of the data includes: checking the consistency between the reconstructed data and the hydraulic constraints of the water network; calculating the similarity between the reconstructed data and the historical normal data of the same period; and adjusting the observer parameters or supplementing historical data for retraining if the reconstruction error exceeds the preset threshold, until the recovery accuracy requirements are met.

9. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, The method for determining the abnormal sensor by the absolute error between the calculated estimates is as follows: if the absolute error is greater than the abnormal threshold and the duration is greater than the preset duration, the sensor corresponding to the lowest-order observer where the abnormal data is located is abnormal and an alarm is triggered; if the absolute error is greater than the abnormal threshold but the number of occurrences within the set time is less than the set value, it is an instantaneous interference and no alarm is triggered.

10. The method for anomaly detection and recovery of water network sensors according to claim 1, characterized in that, Truncation singular value decomposition is used to remove measurement noise from historical water network data.

Images