Urban drainage pipe network monitoring data cleaning and intelligent prediction method

Abstract

The invention provides an urban drainage pipe network monitoring data cleaning and intelligent prediction method, and the method comprises the steps: firstly obtaining pipe network monitoring data, and carrying out the classification tracking and repairing of missing values; adopting a dynamic IQR algorithm based on a sliding window to adaptively identify abnormal candidate points; secondly, introducing a pipe network topological relation, comparing upstream and downstream data change trends, eliminating non-physical anomalies caused by equipment faults, and reserving real hydraulic events; calculating the physical delay time between the nodes by using the cross correlation coefficient; and finally, constructing a random forest model, taking upstream historical data after delay alignment as feature input, and realizing accurate prediction of a future water level and quantification of a feature contribution degree. According to the method, a physical mechanism and machine learning are fused, the problems that data cleaning lacks adaptivity and a deep learning model lacks interpretability are effectively solved, and the accuracy of waterlogging early warning is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data cleaning and intelligent prediction, and particularly relates to a method for cleaning and intelligently predicting urban drainage pipe network monitoring data. BACKGROUND

[0002] As the "vein" system of the city, the operation state of the urban drainage pipe network is directly related to the safety of urban flood control and water environment quality. With the development of smart water, real-time monitoring of the pipe network through the deployment of liquid level, flow and other sensors has become the mainstream means. However, in actual operation, monitoring data is easily affected by equipment failure, signal interference, bad weather and complex pipe network environment, often appearing problems such as data missing, noise interference and abnormal mutation, which seriously affect subsequent analysis and dispatching decision.

[0003] In the existing data processing and prediction technology, there are mainly two categories of technical routes. The first category focuses on physical models or static topology. For example, the prior art CN 115795122B proposes a method for combing the topology relationship of urban drainage pipe network, which constructs a directed graph and uses DAG test to correct the flow direction and ring structure of the pipe network. Although it can solve the topological logical error of the basic data of the pipe network, it focuses on the checking of static data and is difficult to meet the dynamic cleaning needs of real-time monitoring data. Another prior art CN 117892560B proposes a high-geographic-precision drainage pipe network simulation method, which simulates the flow through three-dimensional modeling and surface runoff calculation. Although this kind of method has clear physical mechanism, it often relies on complex parameter settings and heavy computing resources, and is mainly used in the planning stage, which is difficult to meet the timeliness requirements of real-time online monitoring and rapid early warning.

[0004] The second type of technical route is the data-driven method that has emerged in recent years, especially the prediction model based on deep learning. For example, the prior art CN 116227362A discloses a prediction method based on graph and deep learning, which uses a graph convolutional neural network (GCN) and a gated recurrent unit (GRU) to predict the state of the pipe network; the prior art CN 120069791A proposes an intelligent monitoring device using a spatio-temporal convolutional network (ST-CNN+) and an attention mechanism. Although these technologies improve the prediction accuracy to some extent through the powerful fitting ability of neural networks and implicitly learn the spatio-temporal features in an "end-to-end" manner, they generally have the "black box" problem: the internal decision logic of the model is difficult to explain, and management personnel cannot intuitively know which upstream node fluctuation caused the downstream warning. More importantly, data-driven deep learning models often have difficulty distinguishing between "non-physical faults" (such as drift, communication errors) of sensors and real "hydraulic conduction events" (such as flood propagation) when facing data anomalies, which can easily lead to false positives or false negatives. In addition, traditional data cleaning methods mostly use fixed statistical thresholds, which cannot adapt to the non-stationary characteristics of the drainage pipe network, such as the wide range of water level fluctuations in dry seasons, rainy seasons, and different rainfall intensities. Therefore, there is an urgent need for a method that can explicitly incorporate the physical mechanisms of hydraulics (such as the lag effect and topological conduction) into the data processing flow, dynamically adapt to environmental changes, and provide highly interpretable decision-making basis. SUMMARY

[0005] The main purpose of the present application is to provide a city drainage pipe network monitoring data cleaning and intelligent prediction method, which solves the technical problems of lack of dynamic self-adaptive ability of traditional cleaning methods, lack of explainability of deep learning models, and difficulty in distinguishing between sensor faults and real hydraulic events using physical topological logic.

[0006] To solve the above technical problems, the technical solution adopted by the present application is: a city drainage pipe network monitoring data cleaning and intelligent prediction method, the method comprising: S1, obtaining original spatio-temporal data of city drainage pipe network monitoring points, aligning time stamps and checking continuity of the original spatio-temporal data, classifying and tracking missing values in the data and supplementing them to obtain preliminary repaired data; S2, using a dynamic IQR detection method to identify outliers of the preliminary repaired data obtained in S1, setting a sliding time window and calculating the dynamic quartiles in the window to determine the dynamic threshold value varying with time, and marking the abnormal candidate points; S3, introduce the drainage pipe network topology relationship to determine the S2 marked abnormal candidate points, distinguish non-physical abnormalities from real event abnormalities by comparing the data change trend of the current monitoring point and its upstream monitoring point, eliminate non-physical abnormal data, and retain real event abnormal data to complete preliminary cleaning; S4, lag analysis is performed on the data of each monitoring point after S3 preliminary cleaning, the cross-correlation coefficient of the monitoring point and its upstream monitoring point water level sequence is calculated, and the lag time of the upstream monitoring point propagation to the current monitoring point is determined; S5, a random forest prediction model is constructed, the historical data of the current monitoring point and the historical data of the upstream monitoring point after alignment with the lag time are input into the model as features, the future water level of the current monitoring point is predicted, and the contribution of each feature to the prediction result is quantified.

[0007] In the preferred scheme, in step S1, the specific method for classifying and tracking missing values is: For short-term missing values with missing time within a preset short-time threshold, linear interpolation or spline interpolation method is used to restore the short-term trend of the data based on the smooth and continuous characteristics of the water level in a very short time; For medium-long term missing values or data with periodic characteristics with missing time exceeding the preset short-time threshold, Kriging interpolation method or ARIMA time series model is used to fill in the missing water level values in the missing period by combining the periodic law of the site water level and the recent trend.

[0008] In the preferred scheme, in step S1, the preset short-time threshold is 5 minutes, spline interpolation is used for missing values within 5 minutes, and Kriging interpolation method is used for missing values exceeding 5 minutes by introducing spatial correlation.

[0009] In the preferred scheme, in step S2, the specific calculation process of the dynamic IQR detection method includes: Define a sliding time window, and calculate the window time sliding average value Q2, the first quartile Q1 and the third quartile Q3 in the sliding time window; According to the formula Calculate the dynamic interquartile range; Set the lower limit of the dynamic threshold as , and the upper limit as , if the data exceeds the upper and lower limits of the dynamic threshold, it is marked as an abnormal candidate point.

[0010] In the preferred scheme, in step S2, the algorithm logic of the dynamic IQR detection and adaptive threshold updating specifically further includes: Initialize a first-in-first-out queue with a length of L as a sliding window, push the latest collected water level data to the tail of the queue, and remove the oldest data from the head of the queue; Introduce a trend factor calculating linear regression slope of data in current window dynamically adjusting scale factor according to slope when is greater than preset trend threshold, increasing value to tolerate fast rising flood trend, preventing normal sharp water level rise from being misjudged as abnormal; calculating dynamic threshold interval according to adjusted value only when real-time data falls outside the interval, triggering abnormal marking signal.

[0011] In the preferred solution, in step S3, the specific logic for determining abnormality using topological relationship is as follows: if only the current monitoring point data shows abnormal fluctuation, while the upstream monitoring point data all show normal and stable changes, it is determined that the abnormality of the current monitoring point is caused by non-physical abnormality due to self-failure of the local liquid level meter, and is excluded; if the current monitoring point shows abnormality, and the upstream monitoring point also shows similar abnormal pattern in the corresponding time period, it is determined that it is a real event abnormality caused by pipeline rupture or flood in flood season, and is retained.

[0012] In the preferred solution, in step S3, the abnormality classification determination algorithm based on topological causal logic further includes: a local directed acyclic graph (DAG) structure is constructed, with the current monitoring point as a child node and the directly upstream monitoring point as a parent node; when the data of the current monitoring point at time t is marked as abnormal in step S2, a backtracking verification mechanism is triggered: the data state of all parent nodes of the node in the DAG within the time interval is retrieved, where is a preset maximum water flow propagation time; a logical judgment is performed: if the data of all parent nodes within the corresponding backtracking interval are all marked as “normal”, and the data of the current monitoring point shows isolated sharp jump, the logical output is determined as “sensor drift or communication error code”, and data exclusion and interpolation repair instructions are executed; a logical judgment is performed: if at least one parent node has abnormal fluctuation consistent with trend within the corresponding backtracking interval, the logical output is determined as “hydraulic conduction event”, and the abnormal data is retained as a real working condition record.

[0013] In the preferred solution, in step S4, the formula for calculating the cross-correlation coefficient is: ; where A and B represent water level sequences of two monitoring points, toffset and are mean and standard deviation, respectively, and N is the sample number; the maximum cross-correlation coefficient is selected as the time offset between the monitoring point A and the monitoring point B. , the maximum cross-correlation coefficient is selected as the time offset between the monitoring point A and the monitoring point B.

[0014] In the preferred solution, in step S4, the specific algorithm steps for constructing the alignment feature matrix based on the time lag are as follows: Define the monitoring point A as the target prediction point and the monitoring point B as the upstream correlation point, and obtain the time series data vectors of the two points and . Based on the calculated optimal time lag , perform a time axis translation operation on the upstream monitoring point sequence to generate the lag feature vector , so that the first element in the original sequence corresponds to the first element in the original sequence . Iterate through all upstream monitoring points of the monitoring point A , calculate the respective time lags , and generate the corresponding lag feature vectors; Construct a fusion feature matrix , where each row of the matrix represents a time step, and the columns of the matrix include the historical observation values of the target point A, the lag feature vectors of all upstream points after time lag translation , and the periodic encoding features corresponding to the time step. Divide the fusion feature matrix into a training set and a test set, input the random forest model, and force the model to learn the causal relationship based on the hydraulic propagation law.

[0015] In the preferred solution, in step S5, the random forest prediction model is constructed and the feature input method is as follows: The input feature variables include the historical time series data of the current monitoring point and the time series data of the upstream monitoring points after adjustment according to the time lag; The input feature variables also include 5-minute and 10-minute sequence data of the time lag sequence, forming multiple groups of feature variables. Calculate the baseline error of the out-of-bag sample predicted by each tree of the random forest, calculate the new error by randomly disturbing a feature of the out-of-bag sample, and determine the feature importance score using the difference between the disturbed error and the baseline error.

[0016] In the preferred solution, in step S5, the specific algorithm steps for quantifying the importance of features using the random forest algorithm are "out-of-bag data permutation test", which includes: When constructing each decision tree of the random forest, a training subset is generated using Bootstrap sampling, and the remaining samples that are not sampled form the out-of-bag data OOB; The established decision tree is used to predict the out-of-bag data OOB, and the classification accuracy or mean square error is calculated, denoted as the baseline accuracy ; For the feature variable to be evaluated , keeping other features in the out-of-bag data OOB unchanged, only the values of the feature are randomly shuffled and rearranged among all samples, destroying the true correspondence between the feature and the label, obtaining the perturbed out-of-bag data OOB'; The perturbed out-of-bag data OOB' is predicted again using the same decision tree, and the new prediction error is calculated, denoted as the perturbed accuracy ; The importance score of the feature is calculated , and if the score is significantly greater than zero, it is determined that the upstream station and the lag time have a real physical driving effect on the downstream water level.

[0017] In the preferred solution, in step S5, quantifying the contribution of each feature to the prediction result is also used for model pruning: The importance scores of all decision trees in the random forest are averaged to generate the final feature importance ranking map; When the feature importance score of a certain upstream node is consistently below a preset threshold, the feature is automatically excluded in the next model training period, achieving automatic pruning of the feature space and lightweight iteration of the model, thereby reducing the computational load while ensuring accuracy.

[0018] In the preferred solution, in step S5, the random forest algorithm is used instead of deep learning methods. Through explicit lag feature alignment and out-of-bag data feature importance evaluation, the model achieves interpretability, and can directly output the physical weight of each upstream station on the downstream station water level, rather than just outputting the predicted value.

[0019] In the preferred solution, the method further includes robustness evaluation in the prediction phase: during the random forest prediction process, simulate extreme data missing conditions, mask part of the upstream monitoring point data, and only use the remaining valid features for prediction, and dynamically adjust the weight of the remaining features according to the feature importance score to maintain the prediction stability of the model in the case of partial sensor failure.

[0020] A city drainage pipe network monitoring data cleaning and intelligent prediction system, the system comprises: a data acquisition and preliminary repair module configured to perform step S1, acquire monitoring data and interpolate and repair missing values; a dynamic anomaly detection module configured to perform step S2, identify abnormal candidate points based on a dynamic IQR algorithm adjusted by a sliding window, a first-in-first-out queue and a trend factor; a topological correlation determination module configured to perform step S3, filter non-physical anomalies based on a local directed acyclic graph structure combined with upstream monitoring point state backtracking; a lag analysis module configured to perform step S4, periodically recalculate the cross-correlation coefficient matrix between nodes and update the optimal lag time ; an intelligent prediction and evaluation module configured to perform step S5, a feature construction unit dynamically adjusts the input feature matrix structure of the random forest model using the latest parameters, runs the random forest model to predict future water levels and outputs feature importance ranking; an internal feedback mechanism of the system: when the intelligent prediction and evaluation module identifies low importance features, it feeds back to the data acquisition module to reduce the sampling priority of the corresponding sensor or performs feature elimination in the model.

[0021] In a preferred embodiment, the data acquisition and preliminary repair module transmits the repaired data to the dynamic anomaly detection module; The dynamic anomaly detection module sends the identified abnormal candidate point index to the topological correlation determination module; The topological correlation determination module retrieves historical data for backtracking verification according to the local directed acyclic graph structure, generates a cleaned data set and transmits it to the lag analysis module and the intelligent prediction and evaluation module respectively; The lag analysis module calculates the lag time parameter according to the cleaned data set and sends it to the intelligent prediction and evaluation module to dynamically adjust the feature matrix structure; The intelligent prediction and evaluation module makes predictions based on the adjusted feature matrix and feeds back the generated feature importance ranking to the data acquisition and preliminary repair module to guide the optimization of data acquisition strategy.

[0022] The application provides a kind of urban drainage pipe network monitoring data cleaning and intelligent prediction method, first, the quality and reliability of monitoring data are significantly improved.The spline interpolation and Kriging interpolation are used for missing data of different lengths respectively, which retains the short-time smoothing characteristics and spatial correlation of water level while filling the data.Especially the proposed dynamic IQR detection method introduces a trend factor to dynamically adjust the threshold coefficient, combined with the sliding window mechanism, so that the abnormal detection threshold can be adaptively adjusted with the trend changes caused by day, season and rainfall, effectively preventing the misjudgment of normal flood peak rising sharply as abnormal in flood season, while also avoiding the missed detection of small fluctuations in dry season, greatly improving the robustness of abnormal identification.

[0023] Secondly, the application realizes intelligent abnormality judgment based on physical logic.Different from simply relying on statistical characteristics for abnormality removal, the application constructs a local directed acyclic graph (DAG) and uses the topological relationship of the pipe network for backtracking verification.By comparing the data change trend of the current node and the upstream parent node, the system can accurately distinguish between isolated "non-physical anomalies" caused by equipment failure and associated "real event anomalies" caused by flood propagation.This mechanism gives the data cleaning process "hydraulic wisdom", ensuring the integrity of the real waterlogging warning signal while removing false signals, avoiding the deletion of valuable disaster information.

[0024] Thirdly, the application improves the interpretability and scientificity of the prediction model through explicit feature engineering.The accurate lag time is calculated through cross-correlation coefficient, and the upstream data is physically aligned on the time axis, so that the features input into the model are calibrated by "physical laws" rather than blindly fitted by the model.On this basis, the random forest model is used in combination with the out-of-bag data permutation test method, which can quantify the contribution of each upstream station and lag time to the prediction result.Management personnel can intuitively see which pipe segment and how many minutes ago the water level change dominated the current prediction result, breaking the "black box" of deep learning and providing reliable decision support for emergency dispatch.

[0025] Finally, the application constructs a closed-loop system with self-optimization capability.The system can automatically identify and remove redundant features or low-quality sensor data that have no contribution for a long time based on feedback of feature importance, realizing automatic pruning of feature space.This not only reduces the computational load of the system, but also enables the model to maintain the stability of prediction by dynamically adjusting the weights of remaining features in the extreme case of partial sensor failure, significantly enhancing the practical application value of the system in complex and harsh working conditions. BRIEF DESCRIPTION OF DRAWINGS

[0026] The application will be further described below in conjunction with the drawings and examples: Figure 1 is the key technology roadmap of the application; Figure 2 is a schematic diagram of the monitoring points WS1-WS10 of the present application; Figure 3 is an example diagram of abnormal water level detection of the monitoring point WS1 of the present application; Figure 4 is a disassembled structural view of the cover plate of the total protective shell of the present application; Figure 5 is a non-physical anomaly diagram of the monitoring point WS8 of the present application; Figure 6 is a 5-minute future prediction diagram of the monitoring well WS9 of the present application; Figure 7 is an important feature diagram of the 5-minute future prediction of the monitoring well WS9 of the present application; Figure 8 is a 60-minute future prediction diagram of the monitoring well WS9 of the present application. DETAILED DESCRIPTION

[0027] Example 1 As shown in Figures 1-8 , a city drainage pipe network monitoring data cleaning and intelligent prediction method, the method comprises: S1, obtaining the original space-time data of the city drainage pipe network monitoring point, aligning the time stamp and verifying the continuity of the original space-time data, classifying and tracking the missing values in the data and supplementing them to obtain the preliminary repaired data; S2, using a dynamic IQR detection method to identify outliers in the preliminary repaired data obtained in S1, setting a sliding time window and calculating the dynamic quartiles in the window to determine the dynamic threshold value changing with time, thereby marking the abnormal candidate points; S3, introducing the topological relationship of the drainage pipe network to determine the abnormal candidate points marked in S2, comparing the data change trend of the current monitoring point with that of its upstream monitoring point to distinguish between non-physical anomalies and real event anomalies, eliminating non-physical anomaly data and retaining real event anomaly data, and completing the preliminary cleaning; S4, performing lag analysis on the data of each monitoring point after the preliminary cleaning in S3, calculating the cross-correlation coefficient of the water level sequence of the monitoring point and its upstream monitoring point to determine the lag time of the upstream monitoring point propagation to the current monitoring point; S5, constructing a random forest prediction model, inputting the historical data of the current monitoring point and the historical data of the upstream monitoring point after alignment with the lag time into the model as features, predicting the future water level of the current monitoring point, and quantifying the contribution of each feature to the prediction result.

[0028] The embodiment discloses a kind of urban drainage pipe network monitoring data cleaning and intelligent prediction method, this method realizes the whole process processing of monitoring data by five closely linked steps.Firstly, step S1 is executed, obtains the original space-time data of each monitoring point of urban drainage pipe network.Due to the clock drift or transmission delay of different equipment in monitoring network, the system first carries out time stamp alignment to original space-time data, it is unified to standard time grid, and carries out continuity check to identify data flow breakpoint.For the missing value found in data, the system does not use single filling mode, but carries out classification tracking and supplementing.This means that the system will be according to the characteristics of missing data, such as the length of missing period or the regularity of missing mode, match corresponding interpolation strategy, to obtain preliminary repair data.The beneficial effect of this step is that the misplacement of multi-source heterogeneous data in time dimension is eliminated by standardization preprocessing, and the data break problem is effectively solved, so that the subsequent processing step receives time-continuous and structure-intact data stream, and lays a foundation for high-precision cleaning and prediction.

[0029] Then step S2 is executed, and the preliminary repair data obtained in step S1 is identified by using dynamic IQR detection method.This process discards the traditional fixed threshold determination method, and sets a sliding time window moving along the time axis.In each sliding time window, the system calculates the statistical characteristics of data, including dynamic first quartile Q1 and dynamic third quartile Q3 in the window.The first quartile Q1 is the value at the 25th percentile after arranging the data in the window from small to large, and the third quartile Q3 is the value at the 75th percentile.On this basis, the system calculates the dynamic interquartile range according to the formula Subsequently, the system determines the dynamic threshold value varying with time, the upper limit of the threshold value is set as , and the lower limit is set as Wherein k is the proportion coefficient.Any data point exceeding the upper and lower limits of the dynamic threshold is marked as an abnormal candidate point.The beneficial effect of this step is that the abnormal detection standard is no longer rigid fixed value, but can adaptively adjust with the periodic fluctuation and trend change of water level data, thereby significantly reducing the false positive rate and false negative rate caused by environmental background change, and improving the adaptability to non-stationary time series data.

[0030] Subsequently, step S3 is performed to introduce the topological relationship of the drainage pipe network to make a secondary determination on the abnormal candidate points marked in step S2. The system uses the physical connection structure information of the pipe network to compare the data variation trends of the current monitoring point and the upstream monitoring point. The determination logic is that if the current monitoring point has abnormal fluctuations and the upstream monitoring point has stable data, which violates the continuity principle of hydraulic conduction, it is determined as a non-physical abnormality, that is, caused by sensor failure or drift, and is removed. On the contrary, if the current monitoring point has an abnormality and the upstream monitoring point also has a related variation trend within the corresponding time range, it is determined as a real event abnormality, that is, a real water level change caused by rainfall or pipe confluence, and is retained. Through this step, preliminary cleaning is completed, and the beneficial effect is that the causal logic of the physical topology replaces the simple statistical judgment, giving the algorithm the ability to identify device failure and real water conditions, effectively preventing important disaster signal from being cleaned incorrectly, and ensuring the authenticity and physical meaning of the retained data.

[0031] In step S4, lag analysis is performed on the data of each monitoring point after preliminary cleaning in step S3. Since water flow needs time to propagate in the pipe network, changes upstream will not be immediately reflected downstream, so it is necessary to calculate the cross-correlation coefficient of the water level sequence of the monitoring point and its upstream monitoring point. The calculation formula is: ; In the formula, represents the cross-correlation coefficient of monitoring point A and upstream monitoring point B with a time lag of , represents the mathematical expectation, and represent the water level values of monitoring point A at time t and monitoring point B at time , and are the mean values of the two sequences, and are the standard deviations of the two sequences. The system calculates the cross-correlation coefficients at a series of different values, and selects the value corresponding to the maximum coefficient to determine the lag time of the upstream monitoring point propagating to the current monitoring point. The beneficial effect of this step is to concretize the abstract hydraulic propagation law into a calculable physical parameter, accurately quantifying the delay characteristics of water flow propagation, and providing a key basis for subsequent construction of a space-time alignment feature that conforms to physical reality.

[0032] Finally, step S5 is performed to construct the random forest prediction model. The system inputs the historical data of the current monitoring point as a set of features, and simultaneously, according to the lag time determined in step S4, the historical data of the upstream monitoring point is time-axis aligned and input as another set of features. The model uses these features to predict the future water level of the current monitoring point. The random forest algorithm can handle high-dimensional features and prevent overfitting by constructing multiple decision trees and aggregating the results. More importantly, this step also quantifies the contribution of each feature to the prediction result using the characteristics of the random forest algorithm, for example, by calculating the average impurity reduction of the feature in the decision tree split. The beneficial effect of this step is that, by explicitly aligning the input of physical features, the model is forced to learn the causal relationship based on the law of hydraulic propagation, rather than spurious statistical correlations; at the same time, quantifying the contribution of features breaks the limitations of traditional black box models, allowing management personnel to intuitively understand which upstream nodes have the greatest impact on the current water level, thereby providing accurate and reliable decision support for urban drainage scheduling and waterlogging cause analysis.

[0033] In the preferred scheme, the specific method for classifying and tracking missing values in step S1 is as follows: For short-term missing values with a missing time within a preset short-time threshold, linear interpolation or spline interpolation method is used to restore the short-term trend of the data based on the smooth and continuous characteristics of the water level in a very short time. For medium-term missing values or data with periodic characteristics with a missing time exceeding the preset short-time threshold, Kriging interpolation method or ARIMA time series model is used to predict the water level value in the missing period based on the periodicity and recent trend of the site water level.

[0034] The preset short-time threshold in step S1 is 5 minutes. Spline interpolation is used for missing values within 5 minutes, and Kriging interpolation method is used for missing values exceeding 5 minutes.

[0035] The specific implementation of classifying and tracking missing values in step S1. The core of this step is to adopt different mathematical repair strategies according to the time span characteristics of data missing, to ensure the physical reasonableness of data repair. Specifically, the system first judges the duration of missing data. When the missing time is within the preset short-time threshold, the system determines it as short-term missing. For such missing values, the system uses linear interpolation or spline interpolation method. The application of this method is based on the continuity principle of fluid mechanics, that is, in a very short time scale, the water level change of urban drainage network shows a smooth and continuous curve, and does not have the physical condition of sudden change. Therefore, by constructing a linear function or a low-order piecewise polynomial using the known observation points before and after the missing interval, the short-term trend of the data can be restored with high fidelity based on the smooth and continuous characteristics of the water level in a very short time.

[0036] When the missing time exceeds the preset short-time threshold, or the data shows a periodic characteristic missing with a certain rule, the system determines it as a medium and long-term missing. For such cases, the single time neighborhood interpolation cannot accurately capture the fluctuation rule of the data, and the system turns to use the Kriging interpolation method or the ARIMA time series model. The Kriging interpolation method is an optimal unbiased estimation method based on spatial autocorrelation. Its core logic is to quantify the spatial correlation by using the variogram, and to estimate the missing point value by calculating the weighted linear combination of the known monitoring point data in the spatial neighborhood. Its estimation formula can be expressed as , wherein represents the predicted value at the position to be estimated, , wherein represents the known measurement value of the th adjacent observation position, represents the weight coefficient assigned to the th observation value, and the sum of all weight coefficients is 1. The ARIMA model is an autoregressive integrated moving average model, which uses the historical time series data of the station itself, combined with the periodic rule and recent trend of the water level to predict the water level value in the missing period.

[0037] In the preferred embodiment, the preset short-time threshold in step S1 is accurately set to 5 minutes. For missing data with a duration of less than 5 minutes, the system specially uses the spline interpolation method. The spline interpolation method, especially the cubic spline interpolation, constructs a cubic polynomial in each segment interval and requires the first and second derivatives of the curve at the nodes to be continuous, thereby ensuring that the repaired water level curve has excellent smoothness in geometry, avoiding the breakpoints produced by simple linear connection. For missing data with a duration of more than 5 minutes, the system introduces spatial correlation and uses the Kriging interpolation method. At this time, the algorithm is no longer limited to the time dimension of a single point, but horizontally calls the data of adjacent nodes in the network topology, and uses hydraulic connectivity to infer the missing value.

[0038] In the preferred scheme, in step S2, the specific calculation process of the dynamic IQR detection method includes: Define a sliding time window, and calculate the window time sliding average Q2, the first quartile Q1 and the third quartile Q3 within the sliding time window; Calculate the dynamic interquartile range according to the formula Set the lower limit of the dynamic threshold to , and the upper limit to , if the data exceeds the upper and lower limits of the dynamic threshold, it is marked as an abnormal candidate point.

[0039] ​The embodiment details the specific calculation process of the dynamic IQR detection method in step S2. The system first defines a sliding time window with a fixed time length, which continuously moves forward on the time axis as real-time monitoring data is collected. At each specific time step, the system extracts all historical water level data falling within the current sliding time window as a sample set. For this sample set, the system performs statistical analysis to calculate the window time sliding average Q2, which reflects the central tendency or reference height of the water level within the current window. At the same time, the system locates the first quartile Q1 and the third quartile Q3 by sorting the sample set. The first quartile Q1 is the value at the 25th percentile after arranging the data in the window from small to large, representing the statistical boundary of the lower water level; the third quartile Q3 is the value at the 75th percentile, representing the statistical boundary of the higher water level.

[0040] On this basis, the system calculates the dynamic interquartile range according to the formula In this formula, $IQR$ represents the interquartile range, which measures the dispersion or fluctuation range of the data distribution within the current sliding window, effectively eliminating the influence of extreme values on the fluctuation amplitude. Subsequently, the system sets a dynamically changing abnormality judgment threshold. Specifically, the system sets the lower limit of the dynamic threshold as and the upper limit of the dynamic threshold as In this calculation formula, is a proportionality coefficient used to adjust the sensitivity of abnormality detection, which is usually preset or adaptively adjusted according to the noise level of historical data. When the real-time collected monitoring data value is higher than the calculated upper limit or lower than the calculated lower limit, the system determines that the data point is outside the normal water level fluctuation range, and thus marks it as an abnormal candidate point.

[0041] The beneficial effects of the above dynamic IQR detection method are that it overcomes the limitations of traditional fixed threshold methods in dealing with non-stationary time series. Since the water level of the drainage network is affected by various factors such as residents' water usage patterns, seasonal changes in rainfall, etc., its reference value and fluctuation amplitude change over time. By introducing a sliding window and dynamic quartile calculation, the upper and lower limits of abnormality judgment are no longer rigid constants, but can adaptively adjust in real time to follow the periodic changes and trend drift of the data. This forms a dynamic envelope line that closely follows the normal fluctuation range of the data, significantly reducing the risk of misjudging normal flood peaks as abnormalities during flood season, while also reducing the missed detection of small abnormal fluctuations during dry season, greatly improving the accuracy and robustness of abnormality identification.

[0042] In the preferred scheme, in step S2, the algorithm logic of the dynamic IQR detection and adaptive threshold updating specifically further includes: Initialize a first-in-first-out queue of length L as a sliding window, and store the latest collected water level data. Push to the end of the queue and remove the oldest data from the head of the queue. ; Introducing trend factors Calculate the linear regression slope of the data within the current window. The proportional coefficient is dynamically adjusted according to the slope. ,when When the trend exceeds the preset threshold, increase The goal is to tolerate the rapidly rising flood peak and prevent the normal, sharp rise in water level from being misjudged as abnormal. According to the adjusted Value calculation dynamic threshold range Only when real-time data If the value falls outside this range, an anomaly flag signal is triggered.

[0043] The algorithm logic for dynamic IQR detection and adaptive threshold update described in step S2 is explained in detail. To achieve real-time tracking of non-stationary water level data, the system first initializes a first-in-first-out queue of length L as a sliding window. This queue serves as a temporary data storage container, strictly managing the data flow according to time order. Whenever the sensor acquires a new water level data... When the time comes, the system pushes it to the tail of the queue, while the oldest data stored at the head of the queue is added. Removed. This mechanism ensures that the data within the window always consists of the L most recent samples, thus guaranteeing the timeliness of subsequent statistical analysis and enabling it to keenly capture the local data characteristics at the current moment.

[0044] Based on this, the algorithm introduces a trend factor. And calculate the linear regression slope of the data within the current window. The slope is obtained by linearly fitting the water level data within the window to time, and its physical meaning lies in representing the rate and direction of change of the current water level. In actual operating conditions, such as during heavy rain or when drainage pumping stations are activated, the water level in the pipe network will change drastically in a short period of time. To prevent such normal rapid changes from being misjudged as abnormal, the system uses the calculated slope to dynamically adjust the proportional coefficient. The specific logic is as follows: the system sets a trend threshold, and when the absolute value of the slope reaches a certain threshold... When the water level exceeds the preset trend threshold, it indicates that the current water level is in a phase of rapid change, and the system will automatically increase the threshold. Value. Increase The operation of the value essentially expands the judgment range of normal data to "tolerate" the rapid rising flood trend or rapid falling drainage trend, so that the algorithm can distinguish between physical sharp changes and sudden changes caused by sensor failure.

[0045] Subsequently, the system calculates a dynamic threshold interval according to the adjusted value. The calculation formula is interval In this formula, represents the first quartile of the data in the current window, represents the third quartile, is the quartile range of the current window, that is, , is the slope-corrected proportionality coefficient. The system compares the real-time collected data with the dynamic threshold interval, and only when falls outside the interval, an abnormal flag signal is triggered, identifying it as an abnormal candidate point.

[0046] The beneficial effect of using the above adaptive threshold updating logic is that it greatly enhances the robustness of the abnormal detection algorithm to complex hydrological working conditions. The traditional fixed value method often produces a large number of false positives when facing the rapid rise of water level caused by flood in flood season, resulting in the real water regime data being incorrectly cleaned. This embodiment realizes the dynamic adjustment of the threshold width by introducing a slope feedback mechanism, that is, a relatively narrow threshold is maintained during the data stable period to improve the sensitivity to small faults, and the threshold is automatically widened during the data drastic change period to adapt to the real physical process. This design not only reduces the false positive rate, but also ensures the integrity and credibility of the monitoring data under various flow conditions, providing accurate data support for subsequent flood control scheduling.

[0047] In the preferred scheme, the specific logic for determining abnormalities in step S3 using topological relationships is as follows: If only the current monitoring point data shows abnormal fluctuations, while the upstream monitoring point data all show normal and stable changes, it is determined that the current monitoring point abnormality is a non-physical abnormality caused by the local water level gauge itself failure, and is excluded; If the current monitoring point shows an abnormality, and the upstream monitoring point also shows a similar abnormal pattern in the corresponding time period, it is determined to be a real event abnormality caused by pipeline rupture or flood in flood season, and is retained.

[0048] The specific logic of determining the abnormality in step S3 by using the topological relationship. After detecting that the data of the current monitoring point appears abnormal fluctuation, the system does not immediately characterize it, but starts the topological correlation analysis program. The system reads the pre-stored topological structure diagram of the drainage pipe network, and identifies all the direct upstream monitoring points of the current monitoring point. Subsequently, the system calls the data sequence of these upstream monitoring points in the same time period or the corresponding time period considering the flow velocity lag. The system executes the first kind of determination logic: if the comparison result shows that only the data of the current monitoring point presents a large mutation or shock, while the data of all upstream monitoring points remains within the normal fluctuation range, showing a smooth change trend, then according to the principle of hydraulic continuity, the system determines that the abnormality of the current monitoring point is not caused by the upstream inflow, but is a non-physical abnormality caused by the fault of the local liquid level meter. Such faults may be caused by the sensor probe being entangled by floating objects, poor circuit contact or signal transmission interference. For such non-physical abnormalities, the system performs the rejection operation, and can optionally use the interpolation method to repair the data at the time point, or directly mark it as invalid data.

[0049] Then the second kind of determination logic is executed: if the comparison result shows that the current monitoring point appears abnormal at the same time, one or more upstream monitoring points also show similar abnormal patterns in the corresponding time period, such as synchronous water level sharp rise or fall. At this time, the system determines that such abnormal fluctuation conforms to the propagation law of water flow in the pipe network, and belongs to real event abnormality caused by pipe rupture or floodwater inflow in flood season. For such real event abnormality, the system is retained without cleaning and rejection, and is marked as high-value working condition data for subsequent model training or disaster warning.

[0050] The beneficial effect of using the above topological determination logic is that it successfully introduces the causal constraints of the physical world into the data cleaning process. The traditional cleaning method based on single-point statistics often has difficulty in distinguishing between the spikes caused by device failure and the spikes caused by flood, which can easily lead to false deletion of real danger signals or false alarm of device failure. The present embodiment introduces upstream and downstream correlation verification, simulates the logic of manual investigation, and can intelligently identify the physical properties of the data. This not only greatly reduces the false alarm rate caused by device aging or environmental interference, but more importantly, ensures the integrity of the real waterlogging data in extreme weather, providing accurate and error-free basic data for subsequent water accumulation prediction and emergency dispatch, thereby improving the credibility and practical value of the entire monitoring system.

[0051] In a preferred embodiment, in step S3, the abnormal classification and determination algorithm based on the topological causal logic further comprises: constructing a local directed acyclic graph (DAG) structure with the current monitoring point as the child node and the direct upstream monitoring point as the parent node; When the step S2 marks the current monitoring point as abnormal at time t , the system triggers a backtracking verification mechanism to retrieve the data status of all parent nodes of the current monitoring point in the DAG within the time interval . The maximum water flow propagation time is predefined. The system performs a logical judgment: if all parent nodes have normal data within the corresponding backtracking interval, and the current monitoring point data shows an isolated sharp jump, the logical output is "sensor drift or communication error code", and the system executes data rejection and interpolation repair instructions. The system performs a logical judgment: if at least one parent node has abnormal fluctuations consistent with the trend within the corresponding backtracking interval , the logical output is "hydraulic conduction event", and the system retains the abnormal data as a real working condition record.

[0052] The step S3 details the abnormal classification determination algorithm based on topological causal logic. The system first constructs a local directed acyclic graph structure that clearly defines the hydraulic upstream and downstream relationships between monitoring points. The current monitoring point to be detected is set as the child node, and the direct upstream monitoring point connected physically is set as the parent node. When the dynamic IQR detection algorithm in step S2 marks the data of the current monitoring point at time t as an abnormal candidate point, the system automatically triggers a backtracking verification mechanism. The running logic of this mechanism is to retrieve the historical data status of all parent nodes of the child node in the directed acyclic graph structure within the time interval . The maximum water flow propagation time is defined as the preset maximum water flow propagation time, which is determined based on the hydraulic residence time of the longest pipe segment in the pipe network, ensuring that the causal verification covers the entire time range that may have a physical impact.

[0053] Subsequently, the system performs the first type of logical judgment. If within the specified backtracking interval , the data status of all parent nodes is marked as normal and smooth by the system, and only the data of the current monitoring point shows a large isolated sharp jump, the system determines that the abnormality lacks upstream hydraulic support. Accordingly, the determination logic outputs the conclusion of sensor drift or communication error code, etc. device side failure, and immediately executes the data rejection instruction, while calling the interpolation algorithm in step S1 to repair the position to eliminate the impact of noise on the subsequent model.

[0054] Next, the system performs the second type of logical judgment. If within the corresponding backtracking interval , at least one parent node is detected to have abnormal fluctuations consistent with the current data The trend-consistent abnormal fluctuation, for example, the water level of the upstream node also rises sharply a few minutes ago, the system will determine that the anomaly conforms to the physical propagation law of water flow from high to low. According to this, the judgment logic outputs the conclusion that it is a hydraulic conduction event, indicating that it is caused by real working conditions such as rainfall confluence or gate opening. The system will keep this abnormal data as a real working condition record and will not clean it up, so as to ensure that the waterlogging early warning model can capture this important peak signal.

[0055] The above-mentioned abnormal judgment algorithm based on topological causal logic has significant beneficial effects. First, by constructing a local directed acyclic graph and setting a physical backtracking window , the method successfully upgrades the simple time series anomaly detection to space-time joint verification, solving the problem that traditional statistical methods cannot distinguish between data glitches and real flood peaks. Second, using the state of the upstream parent node as the basis for decision, the false positive rate is greatly reduced, avoiding the deletion of real disaster precursor signals due to sensor failure, thereby ensuring the physical authenticity and business security of the drainage pipe network monitoring data. This cleaning strategy based on causal chain provides a high-quality data set verified by rigorous physical logic for subsequent intelligent prediction.

[0056] In the preferred scheme, in step S4, the cross-correlation coefficient formula is calculated as: ; Where A and B represent the water level sequences of two monitoring points, is the time offset, and are the mean and standard deviation, respectively, and N is the sample size; by calculating a series of cross-correlation coefficients , the maximum cross-correlation coefficient corresponding to the time offset is selected as the lag time between monitoring point A and monitoring point B.

[0057] The specific mathematical model for calculating the cross-correlation coefficient in step S4 and its physical meaning are described. In this step, the system uses the normalized cross-correlation function to quantify the linear correlation degree of the water level time series of two monitoring points at different time offsets. The specific calculation formula is as follows: ; In the formula, the symbol represents the cross-correlation coefficient of monitoring point A and monitoring point B with a time lag of , and its value ranges from negative 1 to positive 1. The symbol represents the number of samples involved in the calculation, i.e. the length of the time series. The term is a normalization coefficient for unbiased estimation. The cumulative term in the formula is composed of two products, where represents the water level of monitoring point A at time Water level observations Subtract the mean of the sequence Then divide by the standard deviation of the sequence. This process, statistically known as data standardization or Z-score standardization, physically eliminates the differences in the absolute magnitude of water levels caused by variations in bottom elevation or pipe diameter among different monitoring wells, transforming water level fluctuations into dimensionless relative fluctuation intensities. Similarly, This indicates that monitoring point B is at time [time missing]. Standardized water level value. Cumulative symbol. This means that within the entire observation time window, the offset of the two standardized sequences is calculated. The degree of co-variation. If both deviate from the mean simultaneously and in the same direction most of the time, the cumulative result will be a large positive number, indicating a strong positive correlation.

[0058] Based on the above formula, the system performs the process of determining the lag time. The system sets a time offset range, for example, from 0 minutes to 90 minutes, and changes it sequentially within this range in steps (e.g., 1 minute). The possible values. For each specific The system substitutes the above formula to calculate a corresponding cross-correlation coefficient. After iterative calculations, the system obtains a sequence of cross-correlation coefficients. The system then searches for the largest cross-correlation coefficient within this sequence and identifies the value corresponding to that maximum value. Value. This specific This value is defined as the physical lag time between monitoring point A and monitoring point B. From a hydraulic perspective, this means that the water level fluctuation characteristics at upstream monitoring point A, after [a certain period of time], [the time elapses]. After minutes of fluid propagation, the effect was most strongly reflected in the water level changes at downstream monitoring point B.

[0059] The beneficial effects of using the above calculation formula and strategy are mainly reflected in three aspects. First, by introducing the mean... and standard deviation After standardization, the algorithm exhibits strong robustness, automatically eliminating systematic errors caused by differences in sensor zero-point calibration or pipe burial depth. This ensures that correlation calculations focus solely on water level fluctuations rather than absolute values, accurately reflecting the essence of hydraulic conduction. Secondly, by iteratively searching for the maximum cross-correlation coefficient, a data-driven, non-invasive method is provided to measure actual flow velocity and propagation time within the pipe network. Compared to theoretical hydraulic model calculations, this method better reflects the true rheological characteristics under actual operating conditions such as pipe siltation and changes in pipe wall roughness. Finally, the precisely determined lag time... This provides a strict time alignment basis for subsequent prediction models, ensuring that the input models are "upstream historical moment" data with strong causal relationships, rather than irrelevant "upstream current moment" data, thereby significantly improving the physical interpretability and prediction accuracy of the prediction models.

[0060] In the preferred scheme, the specific algorithm steps for constructing the aligned feature matrix based on the hysteresis time in step S4 are as follows: Define monitoring point A as the target prediction point and monitoring point B as the upstream correlation point, and obtain the time series data vectors of both. and ; Based on the calculated optimal delay time For the upstream monitoring point sequence Perform a time-axis translation operation to generate a lag feature vector. , making The first in Each element corresponds to the original sequence. The first in One element; Traverse all upstream monitoring points of monitoring point A Calculate their respective delay times. And generate the corresponding lagged feature vector; Constructing a fusion feature matrix Each row of the matrix represents a time step, and the columns of the matrix include: the historical observations of target point A, and the lag eigenvectors of all upstream points after lag shift. And the periodic coding features corresponding to the time steps; Fuse feature matrix The dataset is divided into training and testing sets, which are then input into a random forest model, forcing the model to learn causal relationships based on the laws of water propagation.

[0061] The specific implementation of constructing the aligned feature matrix based on lag time in step S4 is as follows: To transform multi-source spatiotemporal data into an input format directly understandable by machine learning models, the system first establishes the relationship between the prediction target and the input source. Monitoring point A is defined as the target monitoring point to be predicted, and monitoring point B is defined as an upstream associated point with hydraulic connection to monitoring point A. The system extracts the water level time series data of both from the historical database within the same time period and denotes them as vectors. and .

[0062] Next, the system performs a crucial timeline translation operation. This utilizes the optimal lag time calculated through cross-correlation analysis in the preceding steps. The system monitors the sequence of upstream monitoring points. a new lag feature vector is generated , and the element value at the th time step in this vector is strictly corresponding to the value at the th time step in the original sequence . The physical essence of this operation is to delay the upstream water level fluctuation event in time axis, so that it is aligned with the time point when the fluctuation propagates to the downstream monitoring point A in data structure, thus realizing the synchronization of causal events.

[0063] Subsequently, the system extends this alignment logic to the entire upstream influence domain of monitoring point A. The system traverses all upstream monitoring points of monitoring point A, and for each upstream monitoring point, it calls its specific lag time between it and monitoring point A. The system repeats the above translation operation for each upstream monitoring point's sequence, thus generating a series of corresponding lag feature vectors . This ensures that the influence of upstream nodes at different distances on the downstream is captured within the correct time window.

[0064] On this basis, the system constructs a fusion feature matrix . This matrix serves as the core input structure for model training, with each row representing a specific observation time step. The column structure of the matrix is carefully designed to include the target point A's own historical observation values, all upstream point lag feature vectors to after lag delay alignment, and the periodic encoding features corresponding to this time step, such as hour encoding or week encoding to represent human water usage patterns.

[0065] Finally, the system divides the constructed fusion feature matrix into training set and test set according to the preset proportion, and inputs it into the random forest model for training. This data construction method forces the model to learn causal relationships based on hydraulic propagation rules, rather than simply statistical correlations.

[0066] The beneficial effects of using the above method to construct the alignment feature matrix mainly lie in three aspects. First, by introducing the physical lag time The feature alignment method effectively solves the problem of time misalignment between input features and prediction targets in traditional time series prediction. This enables the model to directly capture the real physical process of upstream water level changes leading to downstream water level changes, avoiding the decline in prediction accuracy caused by time bias. Secondly, the method compresses the spatial topological information (i.e. multiple upstream nodes) and time propagation information (i.e. different lag times) of the entire network into a unified matrix structure, greatly reducing the difficulty of model learning. Compared with deep learning models that need to implicitly learn the spatio-temporal dependence through a large amount of data, this explicit feature engineering significantly improves the convergence speed and prediction stability of the random forest model. Finally, the periodic encoding features contained in the fusion matrix enable the model to consider both the suddenness of water transmission and the regularity of urban water use, thereby maintaining high prediction accuracy in different operating conditions such as dry season and rainy season.

[0067] In the preferred scheme, in step S5, the random forest prediction model is specifically constructed and the feature input method is as follows: The input feature variables include the current monitoring point historical time series data and the time series data of each upstream monitoring point adjusted according to the lag time; The input feature variables also include 5-minute and 10-minute sequence data of the lag time series adjacent to the upper and lower, forming multiple groups of feature variables; The baseline error is calculated by using the random forest to predict the out-of-bag samples, the new error is calculated by randomly disturbing a feature of the out-of-bag samples, and the feature importance score is determined by the difference between the disturbed error and the baseline error.

[0068] The specific construction process of the random forest prediction model in step S5 and the feature input method. When constructing the input feature space of the prediction model, the system not only considers the longitudinal correlation in the time dimension, but also integrates the horizontal causal relationship in the spatial dimension. Specifically, the input feature variables first include the historical time series data of the current monitoring point itself, such as the water level readings every minute in the past hour, which is used to capture the autoregressive trend of the current node water level, i.e. to infer the future state from the past state. At the same time, the input feature variables include the best lag time The time series data of each upstream monitoring point adjusted according to the lag time. This means that if the lag time from upstream node B to the current node A is 30 minutes, the system will extract the historical water level data of upstream node B 30 minutes ago as the key input for predicting the current node A water level, thereby achieving accurate alignment of physical causality on the time axis.

[0069] More importantly, in order to deal with the uncertainty caused by the change of flow rate in the pipe network and the dispersion effect of water flow during the propagation in the pipe, the embodiment introduces a fault-tolerant window mechanism in the feature construction. The input feature variables are further expanded to include the sequence data of the upper and lower adjacent 5 minutes and 10 minutes based on the optimal lag time sequence. Specifically, in addition to extracting the data at the moment of the upstream node , the system also extracts the data at , , and , forming multiple groups of feature variables around the optimal lag point. This multi-time scale feature input method can completely cover the wave front and wave back information of the water flow fluctuation, forming a feature envelope with a time width. The beneficial effect of this implementation is that it significantly enhances the robustness of the model to flow rate fluctuations. In actual working conditions, the size of the rainfall will change the pipe fullness and then fine-tune the flow rate, resulting in a small deviation between the actual arrival time and the calculated average lag time . The introduction of minute and minute adjacent sequence data is equivalent to giving the model a "fuzzy search" ability in the time dimension, ensuring that even if the flood peak arrives early or late, the model can still capture this precursor signal through the features of adjacent time points, thereby avoiding the prediction error caused by the too rigid time alignment.

[0070] In terms of model training and explanatory analysis, the embodiment uses the out-of-bag data permutation test mechanism unique to the random forest algorithm to quantify feature importance. Random forest is an ensemble of multiple decision trees. When building each decision tree, self-sampling method is used to extract samples from the original training set with replacement, and the samples not extracted constitute out-of-bag data OOB. The system first uses the trained decision tree to predict the out-of-bag data OOB and calculates the baseline error, usually using mean square error MSE as the evaluation index. Subsequently, in order to evaluate the importance of a certain feature (such as the lag water level of a certain upstream node), the system keeps all other features in the out-of-bag data unchanged, and only randomly shuffles the values of the specific feature in all samples. This shuffling operation breaks the real physical connection between the feature and the target label. The system uses the same decision tree to predict the shuffled data again and calculates the new error. Finally, the difference between the shuffled error and the baseline error is used to determine the feature importance score.

[0071] The beneficial effect of this feature importance assessment method lies in providing transparent physical interpretability for black-box machine learning models. If randomly shuffling a feature of an upstream monitoring point causes a sharp increase in the model's prediction error (i.e., a large difference), it indicates that the information contained in that feature is crucial for accurately predicting downstream water levels, and there is a strong physical causal relationship between the two. Conversely, if shuffling a feature leaves almost no difference in error, it indicates that the feature is redundant or irrelevant noise. In this way, the system can output a quantitative feature contribution report, clearly indicating which upstream pipe sections are the main risk sources causing downstream flooding, as well as the main time paths of water propagation. This not only verifies the correctness of the lag analysis in step S4, but also provides a scientific basis for the emergency dispatch of drainage networks. For example, managers can prioritize checking the operating conditions of key upstream nodes based on the indications of high-importance features, achieving a leap from "data prediction" to "causal analysis."

[0072] In the preferred scheme, step S5, which uses the random forest algorithm to quantify feature importance, involves the "out-of-bag data permutation test," including: When constructing each decision tree in the random forest, Bootstrap sampling is used to generate a training subset, and the remaining unsampled samples form out-of-bag (OOB) data. The established decision tree is used to predict out-of-bag (OOB) data, and the classification accuracy or mean squared error is statistically analyzed and denoted as the baseline accuracy. ; For the feature variables to be evaluated Keeping other features in the out-of-bag (OOB) data unchanged, only the features are changed. The values ​​are randomly shuffled and rearranged among all samples to destroy the true correspondence between the feature and the label, resulting in perturbed out-of-bag data OOB'. Using the same decision tree, predict the perturbed out-of-bag (OOB) data again, calculate the new prediction error, and denot it as the perturbed accuracy. ; Calculation features Importance score If the score is significantly greater than zero, it is determined that the upstream station and the delay time have a real physical driving effect on the downstream water level.

[0073] The embodiment details the specific implementation of quantifying the importance of features in step S5 using the random forest algorithm, namely the out-of-bag data permutation test method. This method uses the inherent integration characteristics of the random forest algorithm to strictly quantify and evaluate the predictive contribution of each input feature through statistical means. First, during the construction of the random forest model, the system uses Bootstrap self-sampling method to randomly sample with replacement from the original training set for each decision tree in the forest. This process generates a training subset of the same size as the original data set for building the decision tree. Since it is a sampling with replacement, about 36.8% of the samples in the original data set are not sampled. This part of the remaining samples that do not participate in the training of the current decision tree is called out-of-bag data OOB.

[0074] Subsequently, the system uses the established current decision tree to predict the out-of-bag data OOB corresponding to the tree. Since the OOB data does not participate in the training process of the model, it can be regarded as a natural validation set. The system statistics the prediction results of the decision tree on the OOB data. For regression problems, the mean square error is usually calculated, and for classification problems, the classification accuracy is calculated, and this initial error index is recorded as the baseline accuracy . This index reflects the baseline prediction ability of the model for unknown data without interference of features.

[0075] Next, in order to evaluate the importance of a certain feature variable to the prediction results, the system performs feature perturbation operation. The system keeps all other feature values in the out-of-bag data OOB unchanged except for the feature , and only randomly rearranges the values of the feature among all OOB samples. This operation preserves the statistical distribution characteristics of the feature data while completely destroying the real correspondence and physical connection between the feature and the target label (i.e. downstream water level), thereby constructing the perturbed out-of-bag data OOB'.

[0076] On this basis, the system uses the same decision tree to predict the perturbed out-of-bag data OOB' again. Since the information of the key feature has been disturbed, if the feature is crucial to the model prediction, the prediction performance of the model will inevitably decrease significantly. The system calculates the perturbed prediction error and records it as the perturbed accuracy .

[0077] Finally, the system calculates the importance score of the feature according to the formula . The physical meaning of this formula is to quantify the loss of model performance caused by destroying the information of the feature . If the calculated score significantly greater than zero, indicating that disrupting the feature results in a significant increase in prediction error, thus determining that the upstream station and the corresponding lag time feature contain critical prediction information and have a real physical driving effect on the downstream water level. The system averages the feature importance scores calculated by all decision trees in the forest to obtain the final evaluation result.

[0078] The beneficial effects of the above-mentioned out-of-bag data permutation test are that it provides an embedded feature evaluation mechanism without the need for an additional validation set, maximizing the use of limited monitoring data resources. More importantly, this method converts the data-driven black box model into a white box model with physical interpretability. By quantifying the scores, the system can scientifically identify which upstream monitoring points are key factors affecting the downstream water level and the specific lag time of hydraulic propagation, thereby verifying the physical correctness of the lag analysis in the previous step. This not only helps technicians to troubleshoot irrelevant redundant features, achieve automatic pruning and optimization of the model, but also provides intuitive and reliable decision-making basis for emergency dispatch of the urban drainage system, ensuring that the flood control instructions are based on real hydraulic causal relationships rather than false statistical correlations.

[0079] In the preferred scheme, in step S5, quantifying the contribution of each feature to the prediction result is also used for model pruning: averaging the importance scores of all decision trees in the random forest to generate a final feature importance ranking graph; When the feature importance score of a certain upstream node is consistently below a preset threshold, the feature is automatically excluded in the next model training cycle, achieving automatic pruning of the feature space and lightweight iteration of the model, thereby reducing the computational load while ensuring accuracy.

[0080] This embodiment details the specific implementation of model pruning and lightweight iteration based on feature contribution quantification in step S5. After completing the training of the random forest model and the feature importance evaluation, the system does not stop at a single evaluation result, but further performs integrated statistical operations. The system extracts the importance scores calculated by each decision tree in the random forest for each feature and performs arithmetic averaging operations on these scores. Through this averaging process, the system can eliminate the random bias that may exist in a single decision tree, thereby generating a robust and objective final feature importance ranking graph. The graph arranges the importance scores from high to low, intuitively showing the contribution of each input feature in the current model, including various upstream monitoring points and their different lag time series, to the prediction of the downstream water level.

[0081] On this basis, the system introduces a dynamic feature space automatic pruning mechanism. The system sets a preset importance threshold, and long-term tracks and monitors the importance scores of each feature. When the system detects that the feature importance score of a certain upstream node is always below the preset threshold in continuous multiple model training cycles or in a long time window, the system determines that the feature is redundant information or invalid noise for the current prediction task. This may mean that the hydraulic connection between the upstream node and the current prediction point is weak, or the quality of the sensor data of the node is low for a long time, which cannot provide effective information. Based on this determination, the system automatically triggers a pruning instruction, and in the next model training cycle, the column of the low-score feature is directly removed from the input data, and no longer participates in the calculation.

[0082] The above model pruning and lightweight iteration strategy has significant beneficial effects. First, the strategy realizes efficient use of computing resources. By removing redundant features that have little contribution to the prediction result, the system significantly reduces the dimension of the input data. When dealing with massive high-frequency urban drainage network monitoring data, the reduction of dimension directly translates into a significant reduction in model training time and memory consumption, enabling complex machine learning algorithms to run smoothly on edge computing devices or industrial control machines with limited computing power, thereby reducing hardware deployment costs. Second, the strategy improves the generalization ability and robustness of the model. Statistical principles show that too many irrelevant features often introduce noise, leading to overfitting of the model. By automatic pruning, the system is actually purifying the input space and retaining truly physical causal features, which in turn helps to improve the prediction accuracy of the model under unknown conditions. Finally, the mechanism gives the system the ability to evolve. With the adjustment of the pipe network structure or the change of the sedimentation situation, the hydraulic connection between nodes may change. The automatic pruning mechanism enables the model to adapt to such dynamic changes, automatically eliminate outdated feature associations, and ensure that the prediction model always maintains the optimal structure.

[0083] In the preferred scheme, the random forest algorithm is used in step S5, which realizes the interpretability of the model through explicit lag feature alignment and out-of-bag data feature importance evaluation, and can directly output the physical weight of each upstream site on the downstream site water level, rather than only outputting the predicted value.

[0084] The method further includes robustness evaluation in the prediction phase: in the random forest prediction process, simulate extreme data missing conditions, mask part of the upstream monitoring point data, and only use the remaining valid features for prediction, and dynamically adjust the weights of the remaining features according to the feature importance scores to maintain the prediction stability of the model under the condition of partial sensor failure.

[0085] The embodiment details the strategy of model selection in step S5 and its technical advantages, especially why the random forest algorithm is chosen instead of the current popular deep learning method, and how this choice gives the model physical interpretability. At the same time, the embodiment also details the specific mechanism of implementing robustness evaluation in the prediction stage.

[0086] At the model construction level, the embodiment uses the random forest algorithm combined with the lag features calculated in the previous step S4 to build a prediction system with explicit physical meaning. The deep learning methods in the prior art, such as long short-term memory network LSTM or gated recurrent unit GRU, usually use an end-to-end data input method, hiding the spatio-temporal feature extraction process in the multi-layer nonlinear mapping of the neural network, resulting in a black box inside the model, which cannot intuitively explain the logical relationship between input variables and output results. Unlike this, the embodiment first calculates the exact lag time through physical methods, completes the explicit alignment of features, and then inputs these features with clear physical direction into the random forest. On this basis, the out-of-bag data permutation test method built-in random forest is used to directly calculate and output the importance scores of each upstream station feature. Under this architecture, the importance score is no longer just a statistical indicator, but is directly translated into the physical weight of the upstream station on the downstream station water level. For example, if the feature importance score of a certain upstream monitoring point is the highest, it clearly indicates that this pipe section is the main hydraulic source causing the fluctuation of the downstream water level.

[0087] The technical scheme has the beneficial effect of overcoming the interpretability problem of artificial intelligence algorithms in industrial engineering applications. By directly outputting physical weights, the system can clearly show the cause and effect logic of the prediction results to the city drainage dispatch personnel, i.e., clearly indicating which upstream pipeline's inflow caused the current warning. This transparent decision support significantly enhances the trust of artificial intelligence systems, making the flood control dispatch instructions have a basis, and avoiding the dispatch risks that may be brought by blindly relying on black box models.

[0088] Further, the embodiment details the robustness evaluation and dynamic adjustment mechanism in the prediction phase. In the actual operation of the random forest model, an anti-interference simulation program is built into the system. This program simulates the data loss under extreme working conditions, that is, it is assumed that in severe environments such as heavy rain, some upstream monitoring points completely fail due to communication interruption or equipment damage. In this situation, the system performs shielding operations to cut off the data stream of the failed sensors and only uses the remaining valid feature data to perform prediction calculations. In order to ensure that the prediction accuracy does not drop sharply, the system performs dynamic weight adjustment strategy according to the pre-calculated feature importance score map. Specifically, when a high-weight key sensor fails, the system automatically searches for secondary associated nodes that have high mutual correlation with the failed node in historical data, and temporarily increases the weight of these remaining valid features in the voting decision, using redundant information to compensate for missing information.

[0089] The beneficial effect of implementing this robustness evaluation and adjustment mechanism is that it greatly improves the survivability and business continuity of the monitoring system in the common failure scenario of partial sensor failure. Traditional prediction models often rely on complete input vectors, and once data is missing, they cannot run or output incorrect results. This embodiment gives the system reasoning ability under incomplete information through dynamic reorganization based on feature importance, ensuring that at the moment when data is most needed in a disaster, the system can still provide water level prediction results with reference value, ensuring the safety bottom line of the urban drainage system.

[0090] Embodiment 2 In combination with Embodiment 1, a kind of urban drainage pipe network monitoring data cleaning and intelligent prediction system, the system includes: Data acquisition and preliminary repair module, for executing step S1, obtains monitoring data and carries out interpolation repair to missing value; Dynamic anomaly detection module, for executing step S2, based on sliding window, first-in first-out queue and trend factor adjustment dynamic IQR algorithm identifies abnormal candidate points; Topological correlation determination module, for executing step S3, based on local directed acyclic graph structure combined with upstream monitoring point state backtracking, filter non-physical anomaly; Delay analysis module, for executing step S4, periodically recalculates the mutual correlation coefficient matrix between nodes, updates optimal delay time ; Intelligent prediction and evaluation module, for executing step S5, feature construction unit uses the latest Parameter dynamic adjustment random forest model input feature matrix structure, runs random forest model to predict future water level and outputs feature importance ranking; Internal feedback mechanism: when the intelligent prediction and evaluation module identifies low importance features, it feeds back to the data acquisition module to reduce the sampling priority of the corresponding sensor or removes the features in the model.

[0091] In the preferred embodiment, the data acquisition and preliminary repair module transmits the repaired data to the dynamic anomaly detection module; The dynamic anomaly detection module sends the identified abnormal candidate point index to the topology correlation determination module; The topology correlation determination module retrieves historical data for backtracking verification based on the local directed acyclic graph structure, generates a cleaned data set, and transmits it to the lag analysis module and the intelligent prediction and evaluation module respectively; The lag analysis module calculates the lag time parameter based on the cleaned data set and sends it to the intelligent prediction and evaluation module to dynamically adjust the feature matrix structure; The intelligent prediction and evaluation module makes predictions based on the adjusted feature matrix and feeds back the generated feature importance ranking to the data acquisition and preliminary repair module to guide the optimization of data collection strategies.

[0092] A city drainage pipe network monitoring data cleaning and intelligent prediction system, which is constructed by multiple functional modules, realizes full-automatic closed-loop processing from bottom-layer data collection to high-layer intelligent decision-making. The system mainly includes data acquisition and preliminary repair module, dynamic anomaly detection module, topology correlation determination module, lag analysis module and intelligent prediction and evaluation module, and each module is tightly coupled through specific data flow and control flow.

[0093] First, the data acquisition and preliminary repair module as the input front end of the system is mainly responsible for performing the tasks in step S1. This module communicates with the sensor network distributed at each key node of the drainage pipe network through the Internet of Things interface and pulls real-time monitoring data such as liquid level and flow rate. Since the original data often has time asynchronization or packet loss phenomenon, this module has built-in time alignment algorithm and interpolation repair engine. For the identified missing values, it automatically matches spline interpolation or Kriging interpolation strategy for repair according to the missing duration. The beneficial effect of this module is that it provides a standardized, time-continuous high-quality data basis for downstream analysis, effectively shielding the instability of the bottom-layer hardware from interfering with the upper-layer algorithm.

[0094] Secondly, the dynamic anomaly detection module is used to perform step S2, which is the first line of defense for data cleaning. The module maintains a sliding window and a first-in-first-out queue for each monitoring point. The module not only calculates the quartile in the window, but also calculates the trend factor, i.e. linear slope, of the data in real time. Based on these statistics, the module uses the dynamic IQR algorithm to generate a time-varying anomaly judgment threshold in real time, thereby identifying potential abnormal candidate points. The beneficial effect of the module is that it discards the traditional static threshold alarm mechanism and can adaptively handle the rapid rise of water level caused by flood peak in the rainy season, significantly reducing the false positive rate caused by environmental background changes.

[0095] Thirdly, the topological correlation determination module is used to perform step S3, which is the core unit for realizing physical mechanism cleaning. The module stores local directed acyclic graph structure data, which clearly defines the upstream and downstream hydraulic connection relationship of the whole network. When receiving the index of the abnormal candidate point, the module will automatically trigger the backtracking verification program to call the historical state of the parent node upstream of the node within a certain backtracking window for comparison. By distinguishing isolated single-point jumps from linked multi-point fluctuations, the module can accurately filter out non-physical anomalies, i.e. sensor failures, while retaining real event anomalies. The beneficial effect of the module is that it solidifies the expert's causal inference logic into a software algorithm, ensuring that the cleaned data set only contains hydrological fluctuations with real physical significance, avoiding the false deletion of important disaster signals.

[0096] Fourthly, the lag analysis module is used to perform step S4, which is a bridge connecting data and physical laws. The module is designed as a periodic background service, such as recalculating the cross-correlation coefficient matrix between nodes every 24 hours. Through this periodic calculation, the module continuously updates the best lag time parameter of the upstream node propagating to the downstream node. The beneficial effect of the module is that it can dynamically capture changes in the running state of the pipe network, such as the increase in lag time caused by slower flow due to pipe sedimentation, ensuring that the subsequent prediction model always uses the time alignment parameter that best matches the current physical working condition.

[0097] Fifthly, the intelligent prediction and evaluation module is used to perform step S5, which is the decision core of the system. The feature construction unit in the module receives the latest lag time parameter from the lag analysis module and dynamically adjusts the column structure of the input feature matrix accordingly, ensuring that the data input into the random forest model is strictly aligned in time and causality. The module runs the random forest algorithm to predict future water levels and simultaneously outputs the feature importance ranking. The beneficial effect of the module is that it provides white-boxed prediction results, allowing managers to not only see water level prediction values, but also intuitively understand which upstream pipeline's inflow dominates the current water level change through the feature importance ranking.

[0098] Furthermore, the system includes a crucial internal feedback mechanism. When the intelligent prediction and evaluation module identifies that the feature importance scores of certain upstream nodes are consistently below a threshold, it generates a feedback signal and sends it to the data acquisition and preliminary repair module. Upon receiving the feedback, the acquisition module can either reduce the sampling priority of the corresponding sensor to save bandwidth or automatically remove the feature during model training. The beneficial effect of this mechanism is that it endows the system with self-optimization capabilities, enabling on-demand allocation of computing and sensing resources and reducing long-term maintenance costs.

[0099] In the preferred implementation of the system, data flow between modules follows strict physical logic. The data acquisition and preliminary repair module transmits the repaired standardized data stream to the dynamic anomaly detection module for initial screening. The dynamic anomaly detection module sends the indexes of identified anomaly candidates to the topology association determination module for fine screening. The topology association determination module retrieves historical data based on the local directed acyclic graph structure for backtracking verification, generating the final cleaned dataset. This dataset is split for transmission: one path is used for real-time prediction and transmitted to the intelligent prediction and evaluation module; the other path is used for parameter calibration and transmitted to the hysteresis analysis module. The hysteresis analysis module calculates the physical hysteresis parameters based on the cleaned dataset and sends these parameters to the intelligent prediction and evaluation module, directly controlling the construction method of its feature matrix. Finally, the intelligent prediction and evaluation module completes the prediction based on the adjusted feature matrix and feeds back the generated feature importance ranking to the data acquisition and preliminary repair module to guide the optimization of the data collection strategy. The beneficial effect of this connection is that it constructs a closed-loop intelligent control circuit, ensuring efficient coordination of data flow, parameter flow, and control flow within the system, and realizing a complete technical chain from data perception to physical cognition to feedback control.

[0100] Example 3 Further explanation in conjunction with Example 1, such as Figures 1-8 As shown, the present invention aims to develop a multi-level cleaning method that integrates rule filtering, statistical testing, and machine learning algorithms.

[0101] (1) In the algorithm flow, first, based on the rules of hydraulics and the laws of statistical physics, the reasonable range threshold and variation law of the data are dynamically set, and the data are preliminarily screened. For the problem of data missing, an adaptive interpolation method based on time series analysis is developed. For short-term missing data, spline interpolation is used. For long-term missing data, spatial correlation is introduced, and Kriging interpolation method is used. Then, a sliding window statistical method is used to develop dynamic IQR detection. Based on the correlation within the period and the change trend between periods, the abnormality is identified according to the local range variation law. And through the calculation of the cross-correlation coefficient of the water level sequence of the upstream measuring point and the downstream measuring point, combined with theoretical analysis, the lag analysis is carried out, and the results are embedded in the random forest machine learning in the multi-feature input prediction task. Through the feature importance index and SHAP index of the numerical model, the contribution degree of different feature data to the prediction result is quantified, the "black box" of the model is broken, and the decision basis is provided. The specific process is shown in Figure 1 .

[0102] (2) Before intelligent prediction, the data are first cleaned, which mainly includes three core steps, namely "filling missing data, dynamic IQR coarse screening, and intelligent abnormality judgment". The intelligent system first strictly aligns the time stamp and continuity of the original data stream, and after finding the missing position, this study discards the simple mean or pre-value filling, and adopts the interpolation method which is more consistent with the hydraulic characteristics. For short-term missing data (such as a few minutes, 5 minutes), linear interpolation or spline interpolation is used. This method assumes that within a very short time, the water level change is smooth and continuous, which can better restore the short-term trend of the data. For medium and long-term missing data (more than 5 minutes) or data with obvious periodic characteristics, time series model-based prediction filling (such as Kriging interpolation method or ARIMA) is used. The system learns the periodicity of the water level of the site, combines with the recent trend, and predicts the most possible water level value in the missing period to fill in.

[0103] (3) Then, dynamic IQR detection is carried out to find abnormal points. Most static threshold methods (such as fixed upper and lower limits) or traditional IQR (interquartile range) cannot adapt to the characteristics of the periodic change of water level with time, weather, and water consumption, which is easy to produce a large number of misjudgments. This project proposes a dynamic IQR method to realize the adaptive adjustment of the abnormal threshold. The system defines a sliding time window, calculates the window time sliding average Q2 (also the second quartile) within the sliding window, and calculates the first quartile Q1 and the third quartile Q3. The dynamic is obtained. The upper and lower limits of the abnormal value are usually defined as: Lower limit: ; Upper limit: ; wherein is the proportion coefficient, usually taking 1.5, which can be adjusted according to the actual situation.

[0104] (4) The "abnormal candidate points" monitored by dynamic IQR include real fault abnormalities and real physical abnormalities (caused by rainwater leading to sudden increase of flood peak), which must be distinguished. The project introduces the topological relationship of water system to realize intelligent judgment. If only the data of the monitoring point is abnormal, and the data of the upstream monitoring points are all normal and stable, it strongly indicates that the abnormality of this time is probably the fault of the liquid level meter at this point (such as temporary disturbance of probe by bubbles and silt). The system marks it as "non-physical abnormality" and performs data rejection. Similar abnormalities (such as simultaneous sudden drop) occur at the upstream monitoring points, forming an associated abnormal mode. This is extremely likely to indicate the occurrence of real physical events such as pipeline rupture and flood in flood season, and the system marks it as "event abnormality".

[0105] (5) Before entering deep cleaning, the lag analysis is performed on each monitoring point. First, the original liquid level time series data of each monitoring point is preliminarily cleaned, and the obvious abnormal points are rejected, and the missing values are reasonably interpolated. And the cross-correlation coefficient is calculated by the following formula:

[0106] Among them, and are the mean and standard deviation. is the time offset, and the cross-correlation coefficient of monitoring point A and monitoring point B at time offset is calculated by the above formula. When the time offset is exactly equal to the time required for the fluctuation to propagate in the actual pipe network, the synchronization of the change trend of the two signals is the highest, and the correlation is the strongest. Therefore, a series of cross-correlation coefficients under different time offsets are calculated, and the time offset corresponding to the maximum value is the lag time between monitoring point A and monitoring point B.

[0107] (6) The powerful integrated machine learning algorithm of random forest is used to comprehensively utilize the historical data of the monitoring point and the time series of the upstream monitoring points after lag time adjustment to predict the liquid level change process of the monitoring point in a period of time. First, each tree is used to predict the out-of-bag sample to calculate a baseline error; a feature of the out-of-bag sample is randomly disturbed to destroy the relationship between the feature and the label; the disturbed data is predicted again to calculate a new error, and the importance of the feature is the difference between the error after disturbance and the baseline error. If a certain feature is very important, it will cause the prediction error of the model to rise sharply after disturbance. Finally, the importance of the feature is calculated for all trees in the forest, and the average value is taken to obtain the final feature importance score.

[0108] Example 4 Further illustrated in combination with Example 4, as Figures 1-8Further illustrated, further illustrate the present application.

[0109] Example: Real-time anomaly identification and cleaning of well level data in some pump station area WS1~WS10. Here, the liquid level data of nine monitoring wells WS1~WS9 from 00:00 on November 1, 2024 to 00:00 on June 15, 2025, a total of 325,441 time points, are analyzed. And the WS9 monitoring well is predicted. As Figure 2 illustrated.

[0110] Dynamic IQR monitoring is adopted. Taking all data of WS1 monitoring well in this time period as an example, dynamic IQR monitoring Figure 3 is shown. The black line is the initial data, the red line represents the window time sliding average Q2, and the purple area marks the area surrounded by the dynamic first quartile Q1 and the dynamic third quartile Q3, which is the conventional water level threshold. We can find that the threshold is statistically higher in the flood season than in the non-flood season.

[0111] On this basis, the system will make intelligent anomaly judgment. The topological relationship of water system is introduced to realize intelligent judgment. If only the data of this monitoring point is abnormal, and the data of its upstream monitoring points are all normal and stable, it strongly indicates that this abnormality is probably the fault of the liquid level meter itself (such as temporary disturbance of the probe by bubbles and silt). The system marks it as "non-physical anomaly" and removes the data. As Figure 4 illustrated, the two abnormal points detected by monitoring well WS8 are not found in its upstream monitoring well WS7, so they are diagnosed and marked as "non-physical anomaly".

[0112] At this point, the system has completed the preliminary cleaning of the data, and then the preliminary cleaned data is analyzed for lag. As Figure 5 illustrated, the correlation coefficients of WS9 monitoring well and its eight upstream monitoring wells under different lags are shown. We find that the correlation of WS9 with all monitoring wells is higher than 0.5 except WS3, indicating that most of the monitoring well data have good correlation; in addition, the correlation of each monitoring well first increases and then decreases: this means that there is a most relevant time, that is, the propagation time of the monitoring well water flow to WS9, which is marked by a dashed line, which represents the corresponding lag time.

[0113] Then, random forest learning and prediction are carried out: here we use the lag time series of WS1~WS8 and its five minutes and ten minutes above and below, that is, 8 monitoring wells, 5 feature variables for each monitoring well, plus the current time series of WS9 monitoring well, a total of 41 feature variables, as shown in the table below. In the following description, we use WS i-k to represent the sequence of the i th monitoring well k minutes ago, such as WS3-10 representing the liquid level sequence of monitoring well WS3 10 minutes ago.

[0114] The data from 2024-11-01 00:00 to 2025-04-30 00:00 is used for lag analysis and model training, and the data from 2024-05-01 00:00 to 2025-06-01 00:00 is used for testing. Figure 6 The prediction results of the future 5 minutes of the WS9 monitoring well after the random forest algorithm is learned are shown.

[0115] The RMSE of the results is 0.097, the MSE is 0.0094, the R2 is 0.952, the MAE is 0.03, and the MAPE is 5.06%, and the prediction error is small, and the top ten features are shown in Figure 7 .

[0116] The application can also predict the water level in the next 60 minutes, and has good accuracy, as shown in Figure 8 . The RMSE of the results is 0.163, the MSE is 0.0265, the R2 is 0.864, the MAE is 0.05, and the MAPE is 7.69%.

[0117] The application creates a multi-level, multi-strategy fusion technical solution based on its innovation, which plays a crucial and multi-dimensional beneficial effect in the field of smart water affairs, especially in the cleaning and prediction of urban drainage network monitoring data, which is specifically reflected in: (1) The data quality and reliability are comprehensively improved: For short-term and long-term missing data, different intelligent interpolation strategies (spline interpolation and Kriging interpolation method) are used, and the inherent periodicity and trend of the data are restored, avoiding the destruction of the statistical characteristics of the data by simple filling methods, providing continuous, complete, and consistent high-quality data flow for subsequent accurate analysis.

[0118] (2) The dynamic IQR method completely changes the limitations of static threshold. The sliding window mechanism makes the anomaly detection threshold dynamically adjust with the day, season, and weather cycle, significantly reducing the probability of misjudging normal flood peaks as anomalies in flood season or missing anomalies in dry season, improving the accuracy and adaptability of anomaly identification.

[0119] (3) By introducing the spatial topological relationship of the network, the system can intelligently distinguish between equipment failure and real physical events. This not only avoids the risk of deleting real waterlogging signals, but also reduces the excessive alarm of temporary equipment failure, making the data cleaning process have hydraulic logic wisdom and improving the credibility and business value of the data.

[0120] (4) Based on high-quality data, the propagation time of water flow is accurately quantified through lag analysis, and a spatio-temporal aligned feature set is constructed, which is input into the random forest model for training. This model can comprehensively utilize the information of the current point and upstream to accurately predict the water level in the next 5 to 60 minutes (e.g., the R² of 5-minute prediction can reach 0.952), which can help to gain valuable decision-making time for waterlogging risk warning and emergency dispatch.

[0121] (5) The feature importance evaluation function built into the random forest model can clearly quantify the contribution of each upstream station and its different lag time features to the prediction results. This provides intuitive and reliable decision-making basis for managers, enhances their trust in automated prediction results, and greatly improves the interpretability and decision support ability of the model.

[0122] (6) The automatic rule generation process developed can quickly evaluate data quality and generate cleaning rules, replacing the time-consuming and labor-intensive manual analysis and rule-making process, greatly improving the efficiency of data processing. The feature importance ranking and other functions provided by the model are no longer vague empirical judgments, but provide data-driven and quantitative decision-making basis. This not only liberates technical personnel from tedious and repetitive work, but also improves the scientificity and refinement level of decision-making.

[0123] The above embodiments are only preferred technical solutions of the present application, and should not be regarded as limiting the present application. The protection scope of the present application should be based on the technical solutions recited in the claims, including equivalent replacement solutions of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present application.

Claims

1. A method for cleaning and intelligent prediction of urban drainage network monitoring data, characterized by: The method includes: S1. Obtain the original spatiotemporal data of urban drainage network monitoring points, perform timestamp alignment and continuity verification on the original spatiotemporal data, classify, track and supplement missing values ​​in the data, and obtain preliminary repair data; S2. The dynamic IQR detection method is used to identify outliers in the preliminary repair data obtained in S1. By setting a sliding time window and calculating the dynamic quartiles within the window, the dynamic threshold that changes over time is determined, thereby marking outlier candidate points. S3. Introduce the drainage network topology relationship to determine the abnormal candidate points marked in S2. By comparing the data change trends of the current monitoring point and its upstream monitoring point, distinguish between non-physical anomalies and real event anomalies, eliminate non-physical anomaly data, retain real event anomaly data, and complete the initial cleaning. S4. Perform lag analysis on the data of each monitoring point after the initial cleaning in S3, calculate the cross-correlation coefficient between the water level sequence of the monitoring point and its upstream monitoring point, and determine the lag time from the upstream monitoring point to the current monitoring point. S5. Construct a random forest prediction model, using historical data of the current monitoring point and historical data of the upstream monitoring point after lag time alignment as features input to the model, predict the future water level of the current monitoring point, and quantify the contribution of each feature to the prediction result.

2. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S1, the specific method for classifying, tracking, and supplementing missing values ​​is as follows: For short-term missing time within a preset short-term threshold, linear interpolation or spline interpolation methods are used to restore the short-term trend of the data based on the smooth and continuous characteristics of water level within a very short time. For medium- to long-term missing data or data with periodic characteristics that are missing for a period exceeding a preset short-term threshold, Kriging interpolation or ARIMA time series model is used to fill in the missing water level values ​​by combining the station's water level cycle pattern and recent trend prediction.

3. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 2, characterized in that: step S1 has a preset short-term threshold of 5 minutes. For missing data within 5 minutes, spline interpolation is used, and for missing data longer than 5 minutes, spatial correlation is introduced and Kriging interpolation is used.

4. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S2, the specific calculation process of the dynamic IQR detection method includes: Define a sliding time window, and calculate the window time sliding average Q2, the first quartile Q1, and the third quartile Q3 within the sliding time window; According to the formula Calculate the dynamic interquartile range; Set the lower limit of the dynamic threshold to The upper limit is If the data exceeds the upper or lower limit of the dynamic threshold, it is marked as an abnormal candidate point.

5. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 4, characterized in that: In step S2, the algorithm logic for dynamic IQR detection and adaptive threshold update specifically includes: Initialize a first-in-first-out queue of length L as a sliding window, and store the latest collected water level data. Push to the end of the queue and remove the oldest data from the head of the queue. ; Introducing trend factors Calculate the linear regression slope of the data within the current window. The proportional coefficient is dynamically adjusted according to the slope. ,when When the trend exceeds the preset threshold, increase The goal is to tolerate the rapidly rising flood peak and prevent the normal, sharp rise in water level from being misjudged as abnormal. According to the adjusted Value calculation dynamic threshold range Only when real-time data If the value falls outside this range, an anomaly flag signal is triggered.

6. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S3, the specific logic for determining anomalies using topological relationships is as follows: If only the data at the current monitoring point shows abnormal fluctuations, while the data at its upstream monitoring points all show normal and stable changes, the abnormality at the current monitoring point is determined to be a non-physical abnormality caused by a malfunction of the local level gauge itself, and it is removed from the list. If an anomaly is detected at the current monitoring point, and a similar anomaly pattern is detected at its upstream monitoring point during the corresponding time period, it is determined that the anomaly is caused by a pipeline rupture or flooding during the flood season, and the anomaly is retained.

7. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 6, characterized in that: In step S3, the anomaly classification and determination algorithm based on topological causal logic further includes: Construct a local directed acyclic graph (DAG) structure with the current monitoring point as the child node and the directly upstream monitoring point as the parent node; When step S2 marks the data of the current monitoring point at time t When an anomaly occurs, a backtracking verification mechanism is triggered: all parent nodes of this node in the DAG are retrieved within the specified time interval. The data status within, where The preset maximum water flow propagation time; Execution logic judgment: If all parent nodes' data within the corresponding backtracking interval are marked as "normal", and the current monitoring point data... If the behavior is an isolated spike jump, the logic output is determined to be "sensor drift or communication error", and data removal and interpolation repair instructions are executed. Execution logic judgment: If at least one parent node exists within the corresponding backtracking interval, it is related to... If the abnormal fluctuations show a consistent trend, the logic output will be "hydraulic transmission event", and the abnormal data will be retained as a record of the actual working conditions.

8. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S4, the formula for calculating the cross-correlation coefficient is: ； Where A and B represent the water level sequences of two monitoring points, For time offset, and These represent the mean and standard deviation, respectively, where N is the sample size; cross-correlation coefficients are calculated at a series of different time offsets. The time offset corresponding to the largest cross-correlation coefficient is selected as the lag time between monitoring point A and monitoring point B.

9. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 8, characterized in that: In step S4, the specific algorithm steps for constructing the aligned feature matrix based on the hysteresis time are as follows: Define monitoring point A as the target prediction point and monitoring point B as the upstream correlation point, and obtain the time series data vectors of both. and ; Based on the calculated optimal delay time For the upstream monitoring point sequence Perform a time-axis translation operation to generate a lag feature vector. , making The first in Each element corresponds to the original sequence. The first in One element; Traverse all upstream monitoring points of monitoring point A Calculate their respective delay times. And generate the corresponding lagged feature vector; Constructing a fusion feature matrix Each row of the matrix represents a time step, and the columns of the matrix include: the historical observations of target point A, and the lag eigenvectors of all upstream points after lag shift. And the periodic coding features corresponding to the time steps; Fuse feature matrix The dataset is divided into training and testing sets, which are then input into a random forest model, forcing the model to learn causal relationships based on the laws of water propagation.

10. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S5, the specific construction and feature input method of the random forest prediction model are as follows: The input feature variables include historical time series data of the current monitoring point, and time series data of each upstream monitoring point after adjustment according to the lag time; The input feature variables also include adjacent 5-minute and 10-minute time series data of the lag time series, forming multiple sets of feature variables; The baseline error is calculated by predicting out-of-bag samples using each tree in a random forest. A new error is calculated by randomly shuffling a feature of the out-of-bag samples. The feature importance score is determined by the difference between the shuffled error and the baseline error.

11. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 10, characterized in that: In step S5, the specific algorithmic steps for quantifying feature importance using the random forest algorithm are the "out-of-bag data permutation test," which includes: When constructing each decision tree in the random forest, Bootstrap sampling is used to generate a training subset, and the remaining unsampled samples form out-of-bag (OOB) data. The established decision tree is used to predict out-of-bag (OOB) data, and the classification accuracy or mean squared error is statistically analyzed and denoted as the baseline accuracy. ; For the feature variables to be evaluated Keeping other features in the out-of-bag (OOB) data unchanged, only the features are changed. The values ​​are randomly shuffled and rearranged among all samples to destroy the true correspondence between the feature and the label, resulting in perturbed out-of-bag data OOB'. Using the same decision tree, predict the perturbed out-of-bag (OOB) data again, calculate the new prediction error, and denot it as the perturbed accuracy. ; Calculation features Importance score If the score is significantly greater than zero, it is determined that the upstream station and the delay time have a real physical driving effect on the downstream water level.

12. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 11, characterized in that: In step S5, quantifying the contribution of each feature to the prediction result is also used for model pruning: The importance scores of all decision trees in the random forest are averaged to generate the final feature importance ranking map. When the feature importance score of an upstream node is consistently below a preset threshold, the feature is automatically removed in the next model training cycle, achieving automated pruning of the feature space and lightweight model iteration, thereby reducing the computational load while ensuring accuracy.

13. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: In step S5, the random forest algorithm, compared with deep learning methods, achieves model interpretability through explicit hysteresis feature alignment and out-of-bag data feature importance assessment. It can directly output the physical weights of the influence of each upstream station on the water level of the downstream station, rather than just outputting the predicted values.

14. The method for cleaning and intelligent prediction of urban drainage network monitoring data according to claim 1, characterized in that: The method also includes robustness assessment in the prediction phase: during the random forest prediction process, extreme data missing situations are simulated, some upstream monitoring point data are masked, prediction is made using only the remaining effective features, and the weights of the remaining features are dynamically adjusted according to the feature importance scores to maintain the prediction stability of the model in the case of partial sensor failure.

15. A system for cleaning and intelligent prediction of urban drainage network monitoring data, characterized in that: The system includes: The data acquisition and preliminary repair module is used to perform step S1, acquire monitoring data and interpolate and repair missing values; The dynamic anomaly detection module is used to execute step S2, which identifies anomaly candidate points based on the dynamic IQR algorithm of sliding window, first-in-first-out queue and trend factor adjustment. The topology association determination module is used to execute step S3, which filters out non-physical anomalies based on the local directed acyclic graph structure and the status backtracking of upstream monitoring points. The lag analysis module is used to execute step S4, periodically recalculating the cross-correlation matrix between nodes and updating the optimal lag time. ; The intelligent prediction and evaluation module is used to execute step S5. The feature construction unit utilizes the latest... The parameters are dynamically adjusted to adjust the input feature matrix structure of the random forest model, and the random forest model is run to predict future water levels and output the feature importance ranking. Internal feedback mechanism: When the intelligent prediction and evaluation module identifies low-importance features, it feeds back to the data acquisition module to reduce the sampling priority of the corresponding sensor or to remove features in the model.

16. The urban drainage network monitoring data cleaning and intelligent prediction system according to claim 15, characterized in that: The data acquisition and preliminary repair module transmits the repaired data to the dynamic anomaly detection module; The dynamic anomaly detection module sends the index of the identified anomaly candidate points to the topology association determination module; The topology association determination module retrieves historical data based on the local directed acyclic graph structure for backtracking verification, generates a cleaned dataset, and transmits it to the hysteresis analysis module and the intelligent prediction and evaluation module respectively. The lag analysis module calculates the lag time parameter based on the cleaned dataset and sends the parameter to the intelligent prediction and evaluation module to dynamically adjust the feature matrix structure. The intelligent prediction and evaluation module makes predictions based on the adjusted feature matrix and feeds back the generated feature importance ranking to the data acquisition and preliminary repair module to guide the optimization of the data collection strategy.

Images