Drainage pipe network data cleaning and intelligent repairing method fusing multiple models

Abstract

The invention discloses a drainage pipe network data cleaning and intelligent repairing method fusing multiple models. The method comprises the following steps that S1, original monitoring data flow is collected; s2, carrying out the preliminary detection and elimination of the abnormity based on IQR; s3, performing upstream and downstream multi-source feature extraction and XGBoost weight analysis; step S4, multivariable depth autoregression prediction based on PatchTST is carried out; s5, generating and publishing continuous high-quality data; according to the method, the key technical problems of various abnormal types, incomplete abnormal detection, low cleaning and repairing precision, poor data continuity and trend consistency, high system integration and real-time processing difficulty and the like commonly existing in the actual collection and transmission process of the online monitoring data of the drainage pipe network are solved.

Description

Technical Field

[0001] This invention relates to the field of urban drainage pipe network technology, and in particular to a method for cleaning and intelligent repairing drainage pipe network data that integrates multiple models. Background Technology

[0002] Urban drainage networks are crucial infrastructure for ensuring urban flood control, drainage, and water environment management. With the advancement of smart water management and digital network management, online monitoring data has become the core data support for drainage network operation analysis, scheduling decisions, and risk early warning. However, due to the complex environment of monitoring equipment, the dispersed distribution of monitoring points, and the long data transmission links, online monitoring data of drainage networks (such as liquid level, flow rate, and rainfall) often exhibits various anomalies during actual collection and transmission, including sudden changes, outliers, data drift, periodic errors, long-term missing data, and systematic shifts. These anomalies not only affect the accuracy and timeliness of the data but also directly restrict the implementation and effectiveness of smart drainage services such as intelligent analysis, model prediction, and automatic scheduling.

[0003] Currently, the mainstream data cleaning and anomaly repair technologies mainly include the following categories:

[0004] Statistical anomaly detection methods, such as the mean ± 3σ method, interquartile range (IQR) method, Z-score, and sliding window, can perform preliminary screening and removal of extreme values ​​and outliers in the data. These methods are simple to implement and computationally efficient, but their ability to handle dynamic data distribution and complex anomalies (such as drift and long-term failures) is limited, and they are difficult to cover diverse anomaly types.

[0005] Anomaly identification and imputation methods based on time-series modeling, represented by autoregressive models such as AR, MA, and ARIMA, predict and impute missing or outlier segments by analyzing the temporal structure, periodicity, and trend of the data. These methods are effective in repairing anomalies with strong periodicity and trends, but their performance is insufficient when encountering data with drastic fluctuations, significant nonlinear characteristics, or complex operating conditions, and they are also quite sensitive to parameter and model selection.

[0006] Anomaly detection and repair methods based on machine learning or deep learning: In recent years, an increasing number of scholars and engineers have attempted to use decision trees, support vector machines, and neural networks (such as LSTM, GRU, temporal convolutional networks, Transformer, etc.) to intelligently identify and repair complex anomalies. Deep models can automatically learn anomaly patterns from multi-dimensional data features and have better generalization ability for data with varied patterns and complex structures. However, single deep learning models still suffer from false positives and false negatives in scenarios with limited data samples or simple anomaly types, and the model training and deployment complexity is also high.

[0007] Multi-model fusion and hierarchical processing methods: To overcome the limitations of single methods, some scholars have proposed multi-algorithm cascade or integrated schemes (such as initial statistical screening, time series prediction, and finally neural network completion), which improves the comprehensiveness and accuracy of data cleaning to a certain extent. However, most existing solutions are limited to the field of general time series data. For scenarios such as drainage pipe networks with hydraulic and physical constraints, significant upstream and downstream correlations, and diverse anomaly types, there is still a lack of integrated data cleaning and intelligent repair technologies with engineering usability, high adaptability, and minute-level online processing capabilities.

[0008] Currently, the closest technical solutions for cleaning and repairing anomalies in online monitoring data of drainage pipe networks mainly involve data anomaly detection and interpolation based on a single model or a simple multi-model concatenation. A typical solution generally includes: first, using statistical methods such as IQR or Z-score to initially screen outomas in the data stream; then, using time-series prediction models such as ARIMA to interpolate some short-term missing or abnormal segments; and finally, using simple machine learning models (such as random forests or single-layer neural networks) for secondary discrimination and repair of specific anomaly patterns.

[0009] Such technical solutions have certain engineering applicability and can achieve good results in small-scale data cleaning scenarios with relatively simple anomaly types. However, when faced with large-scale monitoring data of drainage pipe networks with complex actual operating conditions, diverse data anomaly types, and significant physical coupling relationships, existing solutions generally have the following limitations:

[0010] 1. Insufficient modeling of physical constraints

[0011] Traditional methods mostly treat the data from each monitoring point as an independent time series, ignoring the hydraulic and physical relationships between upstream and downstream areas and between nodes and pipes in the drainage network. This results in insufficient ability to distinguish between "structural anomalies" and "trend anomalies," and is prone to misjudgment or omission.

[0012] 2. Incomplete coverage of exception types

[0013] Statistical detection and simple temporal interpolation can only handle primary anomalies such as sudden changes and outliers, and have limited ability to identify complex anomalies such as drift, periodicity disorder, systematic shift, and persistent missing values.

[0014] 3. Limited precision in cleaning and repair

[0015] Because a single algorithm is insufficient to fully extract abnormal features, the repair results are prone to deviating from the actual trend, affecting the continuity and reliability of data, and failing to meet the demand for high-quality data in smart water management.

[0016] 4. Weak intelligent self-adaptation and online response capabilities.

[0017] Traditional solutions rely on manual setting of algorithm parameters and model structure, lacking adaptive adjustment to dynamic changes in data; in real-world business scenarios such as rapid online data stream processing and minute-level response, the system response is lagging, making it difficult to guarantee data quality in real time.

[0018] 5. Insufficient integration and scalability

[0019] Most existing cleaning solutions exist as standalone tools or single-point algorithms, making it difficult to seamlessly integrate them into smart drainage platforms and hindering their expansion into large-scale, multi-point, and multi-regional automated applications.

[0020] In summary, existing technologies for cleaning and repairing online monitoring data in drainage pipe networks have shortcomings such as limited coverage of anomaly types, low cleaning and repair accuracy, insufficient modeling of physical correlations, untimely algorithm response, and weak intelligent adaptive capabilities. There is an urgent need for an innovative technical solution that integrates multiple models, combines physical constraints with intelligent learning, and has the ability to identify multiple types of anomalies in a hierarchical manner and perform online repair within minutes. Summary of the Invention

[0021] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method for cleaning and intelligent repairing drainage network data that integrates multiple models. This method aims to solve key technical problems commonly encountered in the actual collection and transmission of online monitoring data of drainage networks, such as diverse anomaly types, incomplete anomaly detection, low cleaning and repair accuracy, poor data continuity and trend consistency, and difficulty in system integration and real-time processing.

[0022] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for cleaning and intelligent repairing drainage pipe network data integrating multiple models, which includes the following steps:

[0023] Step S1: Acquisition of raw monitoring data stream;

[0024] Step S2: Preliminary anomaly detection and removal based on IQR;

[0025] Step S3: Upstream and downstream multi-source feature extraction and XGBoost weight analysis;

[0026] Step S4: Multivariate deep autoregressive prediction based on PatchTST;

[0027] Step S5: Continuous generation and publication of high-quality data.

[0028] Preferably, step S1 includes the following sub-steps:

[0029] Step S101: Real-time acquisition of liquid level, flow rate, and rainfall data: Obtain raw time-series data of liquid level, flow rate, and rainfall at each monitoring point in the drainage network to form a raw monitoring data stream;

[0030] Step S102, Data Timestamp Unification and Caching: The collected data is timestamped and formatted, and written into the cache / storage in chronological order to provide a complete and orderly sequence for subsequent anomaly detection.

[0031] Preferably, step S2 includes the following sub-steps:

[0032] Step S201, Calculation of interquartile range of sliding window: In window w k Calculate the interquartiles Q1, Q3, and interquartile range: IRQ = Q3 - Q1;

[0033] Step S202, Outlier Marking and Removal: Threshold τ min =Q1-k·IRQ,τ max =Q3 + k·IRQ, when at some time T k The sample x(T) k )<τ min or x(T) k )>τ max If a mutation or outlier is detected, it is marked as a mutation / outlier and removed; the adaptive threshold coefficient k = k0(1 + αCV) k ), where k0 is the base multiple, α is the adjustment parameter, and CV k =σ k / μ k σ is the window coefficient of variation. k μ is the standard deviation. k The mean;

[0034] Step S203, Preliminary Cleaning Sequence Output: Output a preliminary cleaned data sequence that has eliminated extreme outliers to reduce the complexity of subsequent modeling and suppress error propagation.

[0035] Preferably, step S3 includes the following sub-steps:

[0036] Step S301, Multi-source feature construction: For target monitoring point i * Multidimensional feature vectors of upstream and downstream liquid levels, flow rates, and rainfall are extracted, and a lag set L = {0, ..., L} is introduced. l};

[0037] Step S302, XGBoost model training: Using the cleaned sequence from step S2 as labels, train the boosting tree model to fit and learn the multi-source features;

[0038] Training tree:

[0039]

[0040] in Indicates at time point T k The predicted value, f tThis represents the t-th sub-model or function. Indicates at time T k The input feature vector, F represents the set of functions, and M represents the total number of sub-models in the ensemble;

[0041] Objective function:

[0042]

[0043] In the formula, L is the objective function. l represents the experience loss, and l represents the quadratic loss. For the true value, For predicted values, Ω represents the regularization term, and Ω represents the complexity regularization term.

[0044] Step S303, Key Feature Locking: Based on the gain / split frequency output feature importance, automatically select the set of key features that have the greatest impact on the target point for use as input to the deep model.

[0045] Preferably, step S4 includes the following sub-steps:

[0046] Step S401, Training Sample Construction: Combine the historical data of the target point with the key features of S303, and generate (input sequence, prediction sequence) pairs using a sliding window; the input tensor shape is L. x ·d, where L x Let H be the length of the input sequence, d be the feature dimension, and H be the prediction step size.

[0047] Step 402, PatchTST model training: The input sequence is divided into patches for embedding and encoding. The local-global pattern is modeled by the Transformer encoder, and the future H-step prediction is obtained by decoding. The loss function can be weighted MSE, which only assigns higher weights to abnormal / missing positions.

[0048] Step S403, Trend Reconstruction and Missing Segment Imputation: For drifting, periodic disorder, or missing segments, the trained PatchTST model is used to reconstruct the trend and output the structured imputation results.

[0049] Step S404, Continuity and Consistency Verification: Combine the sliding statistical indicators RMSE, MAE and physical constraints to perform consistency verification on the repair results; if the threshold is not met, return to S401 for fine-tuning and retraining.

[0050] Preferably, in step S404,

[0051]

[0052] In the formula, M is the total number of sample points, x ref (Tk ) is at time T k The true value of x clean (T k ) is at time T k The predicted value.

[0053] Preferably, in step S404,

[0054]

[0055] In the formula, M is the total number of sample points, x ref (T k ) is at time T k The true value of x clean (T k ) is at time T k The predicted value.

[0056] Preferably, in step S404, the physical constraints include the following processes:

[0057] Physical consistency:

[0058]

[0059] To simplify the continuity constraint, we approximate it using the water level-discharge curve q = f(h):

[0060]

[0061] In the formula, ρ(,) represents the Pearson correlation coefficient. This indicates that the upstream monitoring point is at time T. k-τ The "cleaned" data value, where τ represents the time delay. This indicates that the downstream monitoring point is at time T. k The "cleaned" data value, ρ min This represents the preset minimum relevance threshold. This represents the rate of change of the system's storage capacity over time, also known as the "capacity change rate".

[0062] Preferably, in step S404, if the threshold is not met, returning to S401 for fine-tuning and retraining specifically means: if the threshold is not met, i.e., RMSE≤ε1, ρ≥ρ min If ε1 is the preset error threshold, then return to S401 to fine-tune the model or adjust the feature set S.

[0063] Preferably, step S5 specifically includes the following sub-steps:

[0064] Step S501, Data splicing and complete sequence reconstruction: Fill the positions of outliers removed in S2 with the interpolation results of S4 to maintain the continuity of the time axis and the sequence;

[0065] Step S502, High-quality data release: Output continuous high-quality data after multi-model cleaning and repair, providing reliable input for drainage network analysis, early warning and scheduling.

[0066] Beneficial effects of this invention:

[0067] 1. This invention is the first to integrate multiple algorithms, including statistical detection, physical constraint multivariate autoregressive modeling, machine learning and deep learning neural networks, in a hierarchical manner, covering multiple types of anomalies such as mutation, drift, missing, and periodic disorder.

[0068] 2. This invention introduces physical constraint modeling of upstream and downstream liquid levels / flow rates to achieve trend discrimination and source tracing of abnormal data, significantly improving the accuracy of complex anomaly detection and repair.

[0069] 3. This invention uses a deep temporal series model (PatchTST) to reconstruct trends and imputate high-quality data for structural missing data and complex pattern anomalies, ensuring data continuity and spatiotemporal consistency.

[0070] 4. This invention supports online processing of minute-level data streams and dynamic adaptation of algorithm parameters, making it suitable for real-time management of large-scale drainage network data.

[0071] 5. This invention features an open platform interface and modular architecture, facilitating deep integration and horizontal expansion with various smart water systems.

[0072] 6. This invention will effectively solve the pain points of incomplete anomaly detection, low repair accuracy, and difficulty in platform integration in the online monitoring data cleaning of drainage pipe networks, significantly improve data quality and business intelligence level, and has broad engineering application prospects and industrialization value.

[0073] 7. This invention is not a simple superposition of algorithms, but a systematic design of "noise screening first, weighting second, reconstruction third, and final verification": front-end IQR suppresses the negative transfer of noise to downstream models; XGBoost transforms upstream and downstream causality / delay into interpretable key feature subsets; PatchTST captures local and global patterns simultaneously, enabling structured trend reconstruction of drift, phase loss, and long missing segments; the output end uses "double consistency" to control and trigger fine-tuning loops to ensure that both numerical and physical standards are met.

[0074] 8. The technical advantage of this invention does not stem from a single algorithm, but from a systematic design that combines hierarchical collaboration and unified constraints: prior noise reduction lowers variance, causal weighting enhances interpretability and transferability, deep reconstruction improves interpolation quality and continuity, and closed-loop verification ensures that the output data is both "numerically compliant" and "physically reliable." This enables the continuous output of high-quality, traceable, and maintainable data sequences for analysis, early warning, and scheduling under various operating conditions, seasons, and rainfall patterns—something that is difficult to reliably achieve with existing single-path or loosely integrated systems.

[0075] 9. This invention solves the key technical problems that are common in the actual collection and transmission of online monitoring data for drainage pipe networks, such as the variety of anomaly types, incomplete anomaly detection, low cleaning and repair accuracy, poor data continuity and trend consistency, and great difficulty in system integration and real-time processing. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating a multi-model-integrated method for cleaning and intelligent repairing drainage network data.

[0077] Figure 2 A diagram illustrating the expected effect of predictive completion for large, consecutively missing data segments;

[0078] Figure 3 A diagram illustrating the expected effect of interpolating and completing data that changes rapidly over a short period of time. Detailed Implementation

[0079] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0080] First, the shortcomings of the existing technology are derived using causal reasoning as follows:

[0081] (1) Reason: Existing methods mostly rely on a single statistical threshold or a single variable time series model (such as fixed IQR, Z-score, ARIMA).

[0082] Results: It can only identify isolated extreme points relatively well, and has poor adaptability to complex anomalies such as drift, periodic disorder, systematic shift, and long segment missing.

[0083] Defects: Incomplete anomaly detection coverage, significant false positives and false negatives, and subsequent analysis and early warning are hampered by noise.

[0084] (2) Reasons: Most of them process the data of each monitoring point independently, ignore the hydraulic coupling and time delay between upstream and downstream, and rely on experience for feature selection.

[0085] Results: It is impossible to quantify the causal impact and optimal lag of upstream and downstream liquid levels / flow rates and rainfall on the target point. Drawbacks: Characteristic engineering is highly subjective, has poor interpretability, and is prone to failure when operating conditions change.

[0086] (3) Reason: The repair is done directly "end-to-end" using only a single depth model, lacking hierarchical progression and prior noise screening.

[0087] Results: The model was "dragged off" by abnormal noise and was unstable in reconstructing long-term missing data and complex trends.

[0088] Defects: The repair results have poor continuity and inconsistent trends, making it difficult to reliably support business operations.

[0089] (4) Reasons: Most schemes lack statistical and physical consistency verification and closed-loop retraining mechanism.

[0090] Results: Even if the output data meets the standards, it may still violate the hydraulic relationship between upstream and downstream; if the model is not calibrated for a long time, its performance will drift.

[0091] Defects: Insufficient data reliability and availability, making it difficult to integrate into the core scheduling / early warning process.

[0092] (5) Reason: The parameter threshold and feature set lack adaptive update and scene transfer design.

[0093] Result: The detection and repair capabilities deteriorate rapidly when the season, rain pattern, or operating conditions change.

[0094] Disadvantages: Insufficient generalization and robustness, high maintenance costs.

[0095] To address the aforementioned shortcomings, the objective of Embodiment 1 of the present invention is as follows:

[0096] Objective 1: To improve the coverage and detection accuracy of abnormality types.

[0097] Prior noise reduction is performed using IQR's dynamic sliding window and adaptive threshold to remove extreme outliers and reduce the burden of subsequent modeling. Then, the layered collaboration of XGBoost and PatchTST covers complex anomalies such as drift, periodic disorder, systematic shift, and long segment missing.

[0098] Objective 2: To "bring in" physical coupling, making feature selection rational and well-founded.

[0099] XGBoost is used to perform data-driven weight evaluation of the feature importance and optimal lag of upstream and downstream liquid level / flow rate and rainfall, mapping the upstream and downstream causality / time lag of hydraulics into an interpretable set of key features, objectively replacing empirical feature selection.

[0100] Objective 3: To achieve structural trend reconstruction and high-quality interpolation at a deep level.

[0101] Using PatchTST multivariate autoregression as the core, local-global pattern joint modeling is performed on key features and historical sequences of target points to achieve structured reconstruction of complex trends and consistent imputation of missing segments, ensuring that the output sequence is smooth, continuous and conforms to physical logic.

[0102] Objective 4: To ensure long-term reliability through "dual consistency + closed loop".

[0103] At the output end, a dual threshold verification is introduced, which includes statistical consistency (such as MAE / RMSE thresholds) and physical consistency (such as upstream and downstream hysteresis correlation / hydraulic constraints). If the threshold is not met, automatic backflow fine-tuning (adjusting the feature set or retraining) is performed to form a robust closed loop of detection → repair → verification → retraining.

[0104] Objective 5: To achieve adaptive maintenance of parameters and features, thereby enhancing generalization robustness.

[0105] The IQR threshold, hysteresis, and feature weights are adaptively updated as the data distribution changes, maintaining stable performance under seasonal / rainy / operating condition shifts and reducing manual maintenance.

[0106] In summary, Embodiment 1 of this invention aims to systematically address the core shortcomings of existing technologies in terms of anomaly coverage, interpretability, repair quality, and long-term robustness using an integrated approach of "statistical noise screening, machine learning weighting of causality, deep model trend reconstruction, and statistical + physical dual verification closed loop," thereby continuously outputting continuous, highly reliable drainage network monitoring data that can be used for scheduling and early warning.

[0107] Example 1: As Figure 1 As shown, a method for cleaning and intelligent repairing drainage network data by integrating multiple models includes the following steps:

[0108] Step S1: Acquisition of raw monitoring data stream.

[0109] S1 includes the following sub-steps:

[0110] Step S101, Real-time acquisition of liquid level / flow rate / rainfall data: Obtain raw time-series data such as liquid level, flow rate, and rainfall at each monitoring point in the drainage network to form a raw monitoring data stream.

[0111] Step S102, Data Timestamp Unification and Caching: The collected data is timestamped and formatted, and written into the cache / storage in chronological order to provide a complete and orderly sequence for subsequent anomaly detection.

[0112] Step S2: Preliminary anomaly detection and removal based on IQR

[0113] S2 includes the following sub-steps:

[0114] Step S201, Calculation of interquartile range of sliding window: In window wk Calculate the interquartiles (Q1, Q3) and the interquartile range (IRQ = Q3 - Q1).

[0115] Step S202, Outlier Marking and Removal: Threshold τ min =Q1-k·IRQ,τ max =Q3 + k·IRQ, when at some time T k The sample x(T) k )<τ min or x(T) k )>τ max If a mutation or outlier is detected, it is marked as a mutation / outlier and removed; the adaptive threshold coefficient k = k0(1 + αCV) k ), where k0 is the base multiple, α is the adjustment parameter, and CV k =σ k / μ k σ is the window coefficient of variation. k μ is the standard deviation. k This is the mean.

[0116] Step S203, Preliminary Cleaning Sequence Output: Output a preliminary cleaned data sequence that has eliminated extreme outliers to reduce the complexity of subsequent modeling and suppress error propagation.

[0117] Step S3: Upstream and downstream multi-source feature extraction and XGBoost weight analysis

[0118] S3 includes the following sub-steps:

[0119] Step S301, Multi-source feature construction: For target monitoring point i * Multidimensional feature vectors such as upstream and downstream liquid levels, flow rates, and rainfall are extracted, and a lag set L = {0, ..., L} is introduced. l}

[0120] Step S302, XGBoost model training: Using the cleaned sequence from step S2 as labels, train the boosting tree model to fit and learn the multi-source features.

[0121] Training tree:

[0122]

[0123] Objective function:

[0124]

[0125] In the formula, L is the objective function. l represents the experience loss, and l represents the quadratic loss. For the true value, For predicted values, Ω represents the regularization term, and Ω represents the complexity regularization term.

[0126] Step S303, Key Feature Locking: Based on the gain / split frequency output feature importance, automatically select the set of key features that have the greatest impact on the target point for use as input to the deep model.

[0127] Step S4: Multivariate Deep Autoregressive Prediction Based on PatchTST

[0128] S4 includes the following sub-steps:

[0129] Step S401, Training Sample Construction: Combine the historical data of the target point with the key features of S303, and generate (input sequence, prediction sequence) pairs using a sliding window; the input tensor shape is L. x ·d, where L x d is the length of the input sequence, d is the feature dimension, and H is the prediction step size.

[0130] Step 402, PatchTST Model Training: The input sequence is segmented into patches for embedding and encoding. A Transformer encoder models the local-global pattern, and decoding yields the prediction for the next H steps. The loss function can be a weighted MSE, assigning higher weights only to outlier / missing locations.

[0131] It is suggested to take Δt = 5min, and input window L. x The prediction step size is 1h, the prediction step size H is 1h; the patch length P = 8 (optional 4-16), the step size S = 4; the Transformer encoder d model =256 (optional 128–512), number of layers 3 (optional 2–4), number of attention heads 8 (optional 4–8), dropout = 0.2, optimizer AdamW (Ir≈2×10 -4 ).

[0132] Weighted MSE is defined as in (Recommended) miss =3,λ anom =2, W max =6), assigning higher weight only to missing / abnormal locations.

[0133] Step S403, Trend Reconstruction and Missing Segment Imputation: For drifting, periodic disorder, or missing segments, the trained PatchTST model is used to reconstruct the trend and output the structured imputation results.

[0134] Step S404, Continuity and Consistency Verification: Combine sliding statistical indicators (such as RMSE, MAE) and physical constraints (such as upstream and downstream hydraulic balance) to perform consistency verification on the repair results; if the threshold is not met, return to S401 for fine-tuning and retraining.

[0135]

[0136] Physical consistency (upstream and downstream lag correlation):

[0137]

[0138] Simplified continuity constraints—approximated by the water level-discharge curve q = f(h):

[0139]

[0140] If the threshold (RMSE≤ε1, ρ≥ρ) is not met min If the condition is met, then return to the S401 fine-tuning model or adjust the feature set S.

[0141] Step S5: Continuous generation and publication of high-quality data.

[0142] S5 includes the following sub-steps:

[0143] Step S501, Data splicing and complete sequence reconstruction: Fill the positions of outliers removed in S2 with the interpolation results of S4 to maintain the continuity of the time axis and sequence.

[0144] Step S502, High-quality data release: Output continuous high-quality data after multi-model cleaning and repair, providing reliable input for drainage network analysis, early warning and scheduling.

[0145] The embodiments of this invention solve key technical problems commonly encountered in the actual collection and transmission of online monitoring data for drainage pipe networks, such as diverse anomaly types, incomplete anomaly detection, low cleaning and repair accuracy, poor data continuity and trend consistency, and high difficulty in system integration and real-time processing. These problems are mainly reflected in the following aspects:

[0146] Diverse anomaly types and incomplete detection: Existing methods mostly target primary anomalies such as isolated outliers, making it difficult to comprehensively and accurately identify and judge complex anomaly types such as drift, periodic disorder, long-term missing data, and systematic shifts.

[0147] Insufficient accuracy in data repair and interpolation: Single statistical methods or traditional time series models cannot effectively reconstruct the true trend of drainage network monitoring data with drastic fluctuations, significant physical constraints, and strong correlations. They also have poor interpolation effects for long-term missing or complex abnormal segments, resulting in data continuity and reliability that cannot meet the requirements of smart drainage systems.

[0148] Feature selection is subjective and lacks physical correlation: Existing cleaning and restoration solutions mostly rely on manual experience to select features, making it difficult to automatically mine the physical and statistical correlations between multiple sources of information such as upstream and downstream monitoring points and rainfall, which limits the applicability and engineering versatility of data governance.

[0149] Lack of intelligent adaptive and efficient online processing capabilities: Existing technologies mostly rely on batch processing, which cannot achieve online anomaly cleaning and intelligent repair of minute-level, real-time data streams. Furthermore, the algorithm parameters do not have adaptive dynamic optimization capabilities, making it difficult to adapt to changes in the actual operating environment.

[0150] Low integration and weak scalability:

[0151] Most data cleaning solutions exist as standalone tools, lacking deep integration and horizontal expansion capabilities with large systems such as smart water management platforms, making it difficult to support the global automated data governance needs of large-scale drainage networks.

[0152] In summary, this embodiment achieves accurate detection and structural trend repair of multiple types of anomalies through multi-level model fusion, automatic feature selection, and deep time series modeling, comprehensively improving the accuracy, real-time performance, and engineering usability of drainage network data cleaning, and providing a high-quality data foundation for scenarios such as smart drainage and digital twin networks.

[0153] The advantages of this embodiment can be derived from its hierarchical collaborative mechanism: First, a sliding window IQR is used for "pre-noise reduction" to eliminate extreme outliers and reduce the variance of subsequent modeling; then, XGBoost is used to automatically screen key features and optimal lags from multi-source information such as upstream and downstream liquid levels / flow rates and rainfall, explicitly quantifying and interpreting the physical coupling relationship; finally, PatchTST is used to perform multivariate autoregressive prediction on a subset of key features, realizing structured trend reconstruction and imputation of drift, periodic disorder, and long-segment missing data. This constitutes a progressive closed loop of "statistical screening - causal weighting - deep reconstruction," which, compared to single statistical or single deep methods, covers more anomaly types and is more stable in complex operating conditions.

[0154] This reasoning chain directly brings three advantages: First, the accuracy and continuity of detection and repair are significantly improved (noise is filtered out before modeling, making it less prone to error amplification); second, the results are more consistent with the physical network (upstream and downstream lags and key factors are captured and constrained by data-driven approaches), reducing false positives and false negatives; third, thresholds and feature sets can be adaptively maintained according to changes in the scenario, enhancing generalization and long-term robustness, and reducing the cost of manual rule maintenance. The final output of high-quality continuous data more reliably supports subsequent core business operations such as analysis, early warning, and scheduling.

[0155] Example 2: Steps one through five are the same as in Example 1. Specifically, the sampling interval is recommended to be 1–15 minutes (preferably 5 minutes); the IQR sliding window is recommended to cover 2–24 hours; the adaptive threshold is adjusted according to the window's coefficient of variation; XGBoost uses importance ranking to filter key features and lags; PatchTST input is recommended to be 6–24 hours, and prediction is recommended to be 0.5–2 hours; the loss uses MSE weighted by the anomaly / missing location; the validation is jointly determined by the RMSE / MAE threshold and the upstream and downstream lag correlation thresholds, and retraining or feature set update is triggered if the criteria are not met. The specific expected results are shown in the figure below. Figure 2 and 3 As shown in the figure, predictive completion of large, continuous missing data and interpolation completion of data that changes abruptly within a short period of time both have excellent data cleaning effects.

[0156] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for cleaning and intelligent repairing drainage network data by integrating multiple models, characterized in that: It includes the following steps: Step S1: Acquisition of raw monitoring data stream; Step S2: Preliminary anomaly detection and removal based on IQR; Step S3: Upstream and downstream multi-source feature extraction and XGBoost weight analysis; Step S4: Multivariate deep autoregressive prediction based on PatchTST; Step S5: Continuous generation and publication of high-quality data.

2. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 1, characterized in that: Step S1 includes the following sub-steps: Step S101: Real-time acquisition of liquid level, flow rate, and rainfall data: Obtain raw time-series data of liquid level, flow rate, and rainfall at each monitoring point in the drainage network to form a raw monitoring data stream; Step S102, Data Timestamp Unification and Caching: The collected data is timestamped and formatted, and written into the cache / storage in chronological order to provide a complete and orderly sequence for subsequent anomaly detection.

3. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 1, characterized in that: Step S2 includes the following sub-steps: Step S201, Calculation of interquartile range of sliding window: In window w k Calculate the interquartiles Q1, Q3, and interquartile range: IRQ = Q3 - Q1; Step S202, Outlier Marking and Removal: Threshold τ min =Q1-k·IRQ,τ max =Q3 + k·IRQ, when at some time T k The sample x(T) k )<τ min or x(T) k )>τ max If so, mark it as a mutation / outlier and remove it; Adaptive threshold coefficient k = k0(1 + αCV) k ), where k0 is the base multiple, α is the adjustment parameter, and CV k =σ k / μ k σ is the window coefficient of variation. k μ is the standard deviation. k The mean; Step S203, Preliminary Cleaning Sequence Output: Output a preliminary cleaned data sequence that has eliminated extreme outliers to reduce the complexity of subsequent modeling and suppress error propagation.

4. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 1, characterized in that: Step S3 includes the following sub-steps: Step S301, Multi-source feature construction: For target monitoring point i * Multidimensional feature vectors of upstream and downstream liquid levels, flow rates, and rainfall are extracted, and a lag set L = {0, ..., L} is introduced. l }; Step S302, XGBoost model training: Using the cleaned sequence from step S2 as labels, train the boosting tree model to fit and learn the multi-source features; train Tree: in Indicates at time point T k The predicted value, f t This represents the t-th sub-model or function. Indicates at time T k The input feature vector, F represents the set of functions, and M represents the total number of sub-models in the ensemble; Objective function: In the formula, L is the objective function. l represents the experience loss, and l represents the quadratic loss. For the true value, For predicted values, Ω represents the regularization term, and Ω represents the complexity regularization term. Step S303, Key Feature Locking: Based on the gain / split frequency output feature importance, automatically select the set of key features that have the greatest impact on the target point for use as input to the deep model.

5. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 4, characterized in that: Step S4 includes the following sub-steps: Step S401, Training Sample Construction: Combine the historical data of the target point with the key features of S303, and generate (input sequence, prediction sequence) pairs using a sliding window; the input tensor shape is L. x ·d, where L x Let H be the length of the input sequence, d be the feature dimension, and H be the prediction step size. Step 402, PatchTST model training: The input sequence is divided into patches for embedding and encoding. The local-global pattern is modeled by the Transformer encoder and decoded to obtain the prediction of the next H steps. The loss function can be a weighted MSE, which assigns higher weights only to outlier / missing locations; Step S403, Trend Reconstruction and Missing Segment Imputation: For drifting, periodic disorder, or missing segments, the trained PatchTST model is used to reconstruct the trend and output the structured imputation results. Step S404, Continuity and Consistency Verification: Combine the sliding statistical indicators RMSE, MAE and physical constraints to perform consistency verification on the repair results; if the threshold is not met, return to S401 for fine-tuning and retraining.

6. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 5, characterized in that: In step S404, In the formula, M is the total number of sample points, x ref (T k ) is at time T k The true value of x clean (T k ) is at time T k The predicted value.

7. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 5, characterized in that: In step S404, In the formula, M is the total number of sample points, x ref (T k ) is at time T k The true value of x clean (T k ) is at time T k The predicted value.

8. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 5, characterized in that: In step S404, the physical constraints include the following processes: Physical consistency: To simplify the continuity constraint, we approximate it using the water level-discharge curve q = f(h): In the formula, ρ(,) represents the Pearson correlation coefficient. This indicates that the upstream monitoring point is at time T. k-τ The "cleaned" data value, where τ represents the time delay. This indicates that the downstream monitoring point is at time T. k The "cleaned" data value, ρ min This represents the preset minimum relevance threshold. This represents the rate of change of the system's storage capacity over time, also known as the "capacity change rate".

9. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 8, characterized in that: In step S404, if the threshold is not met, return to S401 for fine-tuning and retraining. Specifically, if the threshold is not met, i.e., RMSE≤ε1, ρ≥ρ min If ε1 is the preset error threshold, then return to S401 to fine-tune the model or adjust the feature set S.

10. The method for cleaning and intelligent repairing drainage network data integrating multiple models according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: Step S501, Data splicing and complete sequence reconstruction: Fill the positions of outliers removed in S2 with the interpolation results of S4 to maintain the continuity of the time axis and the sequence; Step S502, High-quality data release: Output continuous high-quality data after multi-model cleaning and repair, providing reliable input for drainage network analysis, early warning and scheduling.

Images