Non-contact urban floodwater monitoring abnormal data correction method

Abstract

The application aims to provide a non-contact urban floodwater monitoring abnormal data correction method. First, it is judged whether the short-time monitoring data missing is caused by the data transmission failure of the floodwater monitoring equipment or whether the hardware failure leads to the inability to monitor. The interpolation algorithm is used to interpolate the short-time floodwater monitoring data missing value. The floodwater point data that cannot be monitored for a long time due to equipment damage is generated based on the floodwater monitoring equipment value of other similar areas. Then, the abnormal fluctuation data caused by the passing or parking of vehicles is corrected. The floodwater period monitoring data is intercepted. Finally, the oscillation process caused by the automobile driving is corrected using the set Kalman filtering algorithm. The application solves the problem of abnormal floodwater data caused by vehicles or pedestrians in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of quality technology of urban flood waterlogging monitoring data, and specifically relates to a non-contact method for correcting abnormal data in urban flood waterlogging monitoring. Background Technology

[0002] Urban flooding is a frequent occurrence due to extreme rainfall caused by global climate change, localized heavy rainfall caused by the urban "rain island effect," and changes in underlying surface characteristics caused by rapid urbanization. It has become a prominent problem restricting urban development and management. With the rapid advancement of new urbanization, it is foreseeable that urban flooding will persist for a considerable period in the future.

[0003] Waterlogging monitoring is a fundamental component of urban flood prevention and mitigation systems, serving as a crucial basis for flood early warning and forecasting, flood control plan development, and risk assessment. As urban flooding becomes increasingly severe, many cities have established waterlogging monitoring systems to monitor water depth changes at key waterlogging points in real time and issue early warnings based on the monitoring data. However, with the accumulation of monitoring data, numerous anomalies have been identified. While non-contact waterlogging monitoring devices, such as ultrasonic level gauges, offer high safety due to their lack of contact with the road surface, they are highly susceptible to data fluctuations caused by vehicle movement, vehicles parking under the gauge, or pedestrians crossing, resulting in inconsistent data quality that cannot effectively support urban flood prevention and control. Summary of the Invention

[0004] The purpose of this invention is to provide a non-contact method for correcting abnormal data in urban flood waterlogging monitoring, which solves the problem in the prior art where waterlogging monitoring data is abnormal due to vehicles or pedestrians.

[0005] The technical solution adopted in this invention is a non-contact method for correcting abnormal data in urban flooding monitoring, which is implemented according to the following steps:

[0006] Step 1: Determine whether the short-term missing monitoring data is caused by a data transmission failure of the water accumulation monitoring equipment, or by a hardware failure that prevents monitoring.

[0007] Step 2: Use an interpolation algorithm to interpolate missing values ​​in the short-term water accumulation monitoring data;

[0008] Step 3: Generate data for water accumulation points that cannot be monitored for a long time due to equipment failure, based on water accumulation monitoring equipment values ​​from other similar areas;

[0009] Step 4: Correct any abnormal fluctuations in data caused by vehicles passing by or stopping;

[0010] Step 5: Extract monitoring data during the period of water accumulation;

[0011] Step 6: Correct the oscillation process caused by the car's movement using an ensemble Kalman filter algorithm.

[0012] The invention is further characterized in that,

[0013] Step 1 is implemented in the following steps:

[0014] The calculation formula is as follows:

[0015] t q =t j -t i

[0016] Among them, t q t represents the time interval for missing data. j For the latest missing data time, t i For the moment preceding the first missing data point, when t q When t < α, α is a threshold, which represents short-term missing monitoring data caused by transmission failure; when t < α, α represents the threshold value. q When ≥α, it means that the device cannot monitor due to a hardware failure; i is the data sequence number before the first missing data; j is the latest missing data sequence number; q represents the time interval of the missing data.

[0017] Step 2 is as follows:

[0018]

[0019] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3...; d n d represents the last measured data point before the missing data point was determined. m This is the first measured data point after the missing data point, and Δt is the time interval between monitoring data points. m-n For monitoring data d m and d n The time interval; n is the data sequence number of the last measured data before the missing data; m is the data sequence number of the first measured data after the missing data; q indicates that the currently calculated water level data is missing data.

[0020] Step 3 is implemented in the following steps:

[0021] Using the faulty device as the center, calculate the distance between each neighboring device and the faulty device, further obtain the shortest distance, and determine the one normal device closest to the faulty device.

[0022] Calculate the distance l between each normally operating device and the faulty device. b :

[0023]

[0024] l min =min(l1,l2,l3,...,l b )

[0025] In the formula, l b y is the distance between normal and faulty equipment; b is the equipment number, b = 1, 2, 3...; x f y f These are the relative coordinates of the faulty device in the x and y directions, respectively; x b y b These are the relative coordinates of each normal device in the x and y directions; min The minimum distance is given; f represents the coordinates of the faulty device.

[0026] Calculate the ratio of monitoring data from the two devices during the most recent flooding event:

[0027]

[0028]

[0029] In the formula, m z The ratio of faulty equipment data to normal equipment data is given by z, where z represents the time sequence of water accumulation data, z = 1, 2, 3...; m average This is the average ratio of all water accumulation data. These are the monitoring values ​​for normal equipment; These are the monitoring values ​​for faulty equipment;

[0030] The monitoring data for the faulty equipment is generated based on the ratio and the monitoring data of the normal equipment.

[0031]

[0032] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3, ...; This is monitoring data from normal equipment.

[0033] Step 4 is implemented in the following steps:

[0034] First, identify and handle any abnormal fluctuations caused by passing vehicles:

[0035] For the water depth d at time i i ≠0, and the water depth d before time i is ? i-1 The water depth d after time i i+1 In the case where both are equal to 0, let d i =0;

[0036] Secondly, abnormal fluctuation data caused by vehicle parking is identified and processed:

[0037] For the water depth d at time i i ≥γ, where γ is a pre-set water depth threshold, and the water depth d at the time preceding time i is ≥γ. i-1 =0, the water depth d one time after time i i+1 =d i And let d i =d i+1 =0; if d c c = i + 2, i + 3, ... are all equal to d before correction. i Let d c =0, until the monitored value is 0, then the cycle correction ends.

[0038] Step 5 is implemented in the following steps:

[0039] To determine the start and end times of a period of water accumulation, if there is no water accumulation for the five minutes preceding a non-zero value, and water accumulation begins in the next moment, then the moment preceding the non-zero value is considered the start time t of this water accumulation period. star If there is no water accumulation for five consecutive minutes after the last non-zero value, the moment t after the non-zero value is considered the end of this water accumulation period. end ,

[0040] The time period t was determined star ~t end Then, the water accumulation monitoring data value d for that period is filtered from the data storage database. i ,i=star,stari+1,…,end.

[0041] Step 6 is implemented in the following steps:

[0042] The EnKF ensemble Kalman filter algorithm was used to correct abnormal oscillation data caused by vehicle movement.

[0043] The specific process of the filtering algorithm is as follows:

[0044] Step 6.1: Initialize the background field, given N random variables X that conform to a Gaussian distribution. i Let i = 1, ..., N, and the state variable be the water depth, i.e., X. i =d i The initial sequence data is perturbed using a Gaussian distribution, and N new sets of sequence data are randomly generated.

[0045] Step 6.2: Calculate the predicted values ​​of each variable at time k+1.

[0046]

[0047] in, It is a variable The predicted value at time k+1; It is a variable The analysis value at time k, where the superscript 'a' indicates that this value is an observed value, M k (·) is a nonlinear model operator; Q k It is the model error variance matrix; w i,k It is Gaussian white noise with an expected value of 0 and a variance of Q. k ;

[0048] Step 6.3: Calculate the Kalman gain matrix at time k+1:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] in, It is the predicted value of the i-th random variable at time k+1, where the subscript k indicates the time. Let be the predicted value of the i-th state variable at time k+1; R is the predicted average value of the state variable at time k+1. k It is the observation error variance matrix; H k (·) is the observation operator. In this method, both the state variable and the observation are the water depth, so H k (·) represents a scalar; the superscript T denotes the transpose matrix; the superscript f indicates that the current variable value is the predicted value; K k+1 The Kalman gain matrix; H is the state transition matrix; H is the state variable; N is the number of generated sequence data sets.

[0055] Step 6.4: Calculate the average value of the state variables of the analysis field at time k+1. and background field error variance matrix

[0056]

[0057]

[0058]

[0059] in, It is the i-th variable The analysis value at time k+1; It is a state variable The average value at time k+1; v i,k It is Gaussian white noise with an expected value of 0 and a variance of R. k The superscript f indicates that the current variable value is a predicted value; the superscript a indicates that the value is an observed value.

[0060] Step 6.5: The calculation process returns to step 6.2 to perform the calculation for the next time step;

[0061] Step 6.6: Calculate repeatedly until the preset time, then output the corrected water depth data.

[0062] The beneficial effects of this invention are that the non-contact method for correcting abnormal data in urban flooding monitoring adopts a "layer-by-layer analysis and graded correction" approach to diagnose whether the flooding data is "missing", "fluctuating", or "oscillating". Data that is diagnosed as normal is directly stored in the database, while abnormal data is corrected by interpolation, construction, filtering, etc. before storage. This can effectively improve the quality of stored and displayed data and efficiently support urban flood prevention and mitigation work. Attached Figure Description

[0063] Figure 1 A flowchart for a non-contact method for correcting abnormal data in urban flooding monitoring;

[0064] Figure 2 Measured water depth data for a non-contact urban flood waterlogging monitoring anomaly data correction method;

[0065] Figure 3 This is the corrected water depth data for a non-contact urban flood waterlogging monitoring anomaly data correction method. Detailed Implementation

[0066] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0067] The flowchart of the non-contact urban flooding monitoring abnormal data correction method of the present invention is as follows: Figure 1 As shown, please follow these steps:

[0068] Step 1: Determine whether the short-term missing monitoring data is caused by a data transmission failure of the water accumulation monitoring equipment, or by a hardware failure that prevents monitoring.

[0069] Step 1 is implemented in the following steps:

[0070] The calculation formula is as follows:

[0071] t q =tj -t i

[0072] Among them, t q t represents the time interval for missing data. j For the latest missing data time, t i For the moment preceding the first missing data point, when t q When t < α, α is a threshold, which represents short-term missing monitoring data caused by transmission failure; when t < α, α represents the threshold value. q When ≥α, it means that the device cannot monitor due to a hardware failure; i is the data sequence number before the first missing data; j is the latest missing data sequence number; q represents the time interval of the missing data.

[0073] Step 2: Use an interpolation algorithm to interpolate missing values ​​in the short-term water accumulation monitoring data;

[0074] Step 2 is as follows:

[0075]

[0076] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3...; d n d represents the last measured data point before the missing data point was determined. m This is the first measured data point after the missing data point, and Δt is the time interval between monitoring data points. m-n For monitoring data d m and d n The time interval; n is the data sequence number of the last measured data before the missing data; m is the data sequence number of the first measured data after the missing data; q indicates that the currently calculated water level data is missing data.

[0077] Step 3: Generate data for water accumulation points that cannot be monitored for a long time due to equipment failure, based on water accumulation monitoring equipment values ​​from other similar areas;

[0078] Step 3 is implemented in the following steps:

[0079] Using the faulty device as the center, calculate the distance between each neighboring device and the faulty device, further obtain the shortest distance, and determine the one normal device closest to the faulty device.

[0080] Calculate the distance l between each normally operating device and the faulty device. b :

[0081]

[0082] l min =min(l1,l2,l3,...,l b )

[0083] In the formula, l b y is the distance between normal and faulty equipment; b is the equipment number, b = 1, 2, 3...; x f y f These are the relative coordinates of the faulty device in the x and y directions, respectively; x b y b These are the relative coordinates of each normal device in the x and y directions; min The minimum distance is given; f represents the coordinates of the faulty device.

[0084] Calculate the ratio of monitoring data from the two devices during the most recent flooding event:

[0085]

[0086]

[0087] In the formula, m z The ratio of faulty equipment data to normal equipment data is given by z, where z represents the time sequence of water accumulation data, z = 1, 2, 3...; m average This is the average ratio of all water accumulation data. These are the monitoring values ​​for normal equipment; These are the monitoring values ​​for faulty equipment;

[0088] The monitoring data for the faulty equipment is generated based on the ratio and the monitoring data of the normal equipment.

[0089]

[0090] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3, ...; This is monitoring data from normal equipment.

[0091] Step 4: Correct any abnormal fluctuations in data caused by vehicles passing by or stopping;

[0092] Step 4 is implemented in the following steps:

[0093] In urban areas, where roads are heavily paved, surface runoff from rainfall flows along road slopes to low-lying areas, forming water accumulation. During periods of water accumulation, this water can be absorbed through evaporation, infiltration, or gravity flow into the city's drainage network, gradually reducing the water depth. These are all normal variations in urban water depth. However, because commonly used non-contact water accumulation monitoring equipment is installed on poles along the roadside, it is significantly affected by vehicle traffic. This results in abnormal fluctuations in the monitoring data, characterized by suddenness, large amplitude, and high frequency. As traffic volume increases, continuous data oscillations may occur.

[0094] Therefore, a second diagnostic step is used for the identification and correction of "outliers." First, abnormal fluctuations caused by passing vehicles are identified and addressed:

[0095] For the water depth d at time i i ≠0, and the water depth d before time i is ? i-1 The water depth d after time i i+1 In the case where both are equal to 0, let d i =0;

[0096] Secondly, abnormal fluctuation data caused by vehicle parking is identified and processed:

[0097] For the water depth d at time i i ≥γ, where γ is a pre-set water depth threshold, and the water depth d at the time preceding time i is ≥γ. i-1 =0, the water depth d one time after time i i+1 =d i And let d i =d i+1 =0; if d c c = i + 2, i + 3, ... are all equal to d before correction. i Let d c =0, until the monitored value is 0, then the cycle correction ends.

[0098] Step 5: Extract monitoring data during the period of water accumulation;

[0099] Step 5 is implemented in the following steps:

[0100] Due to the large amount of monitoring data, in order to save data correction time, the water accumulation period is identified in step 5 and the water accumulation data within the period is extracted as the input data for step 6.

[0101] To determine the start and end times of a period of water accumulation, if there is no water accumulation for the five minutes preceding a non-zero value, and water accumulation begins in the next moment, then the moment preceding the non-zero value is considered the start time t of this water accumulation period. star If there is no water accumulation for five consecutive minutes after the last non-zero value, the moment t after the non-zero value is considered the end of this water accumulation period. end ,

[0102] The time period t was determined star ~t end Then, the water accumulation monitoring data value d for that period is filtered from the data storage database. i ,i=star,stari+1,…,end.

[0103] Step 6: Correct the oscillation process caused by the car's movement using an ensemble Kalman filter algorithm.

[0104] Step 6 is implemented in the following steps:

[0105] The Ensemble Kalman filter (EnKF) algorithm is used to correct abnormal oscillation data caused by vehicle movement.

[0106] In 1994, influenced by the sequential data assimilation algorithm proposed in stochastic dynamic forecasting theory, the ensemble Kalman filter was proposed. It utilizes the Monte Carlo method, using a set of N random variables conforming to a Gaussian distribution to represent the probability density function in stochastic dynamic forecasting. Through integration, the probability density function of the overall state at the next time step is calculated, and the statistical characteristics (such as mean and covariance) at that time are obtained. This method includes two steps: prediction and analysis. In the prediction stage, the predicted value of the model state at the current time step is generated based on the model state at the previous time step. In the analysis stage, multi-source data is introduced. In this invention, the data used are the original water accumulation monitoring data and its data after moving average processing. Finally, the minimum variance estimation method is used to re-analyze the model state to generate data that eliminates oscillations. In the ensemble Kalman filter algorithm, the Monte Carlo method can generate multiple water accumulation data with different oscillation characteristics based on existing water accumulation data through random perturbation, effectively avoiding random errors caused by using a single water accumulation data for filtering. The ensemble Kalman filter algorithm includes two steps: prediction and analysis. In the prediction phase, filtering generates a predicted value for the current mode state based on the mode state at the previous time step. In the analysis phase, the minimum variance estimation method is used to reanalyze the mode state to generate data that eliminates oscillations.

[0107] The specific process of the filtering algorithm is as follows:

[0108] Step 6.1: Initialize the background field, given N random variables X that conform to a Gaussian distribution. i In this invention, i = 1, ..., N, the state variable is the water depth, i.e., X. i =d i The initial sequence data is perturbed using a Gaussian distribution, and N new sets of sequence data are randomly generated.

[0109] Step 6.2: Calculate the predicted values ​​of each variable at time k+1.

[0110]

[0111] in, It is a variable The predicted value at time k+1; It is a variable The analysis value at time k, where the superscript 'a' indicates that this value is an observed value, M k(·) is a nonlinear model operator; Q k It is the model error variance matrix; w i,k It is Gaussian white noise with an expected value of 0 and a variance of Q. k ;

[0112] Step 6.3: Calculate the Kalman gain matrix at time k+1:

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] in, It is the predicted value of the i-th random variable at time k+1, where the subscript k indicates the time. Let be the predicted value of the i-th state variable at time k+1; R is the predicted average value of the state variable at time k+1. k It is the observation error variance matrix; H k (·) is the observation operator. In this method, both the state variable and the observation are the water depth, so H k (·) represents a scalar; the superscript T denotes the transpose matrix; the superscript f indicates that the current variable value is the predicted value; K k+1 The Kalman gain matrix; H is the state transition matrix; H is the state variable; N is the number of generated sequence data sets.

[0119] Step 6.4: Calculate the average value of the state variables of the analysis field at time k+1. and background field error variance matrix

[0120]

[0121]

[0122]

[0123] in, It is the i-th variable X i a The analysis value at time k+1; It is a state variable The average value at time k+1; v i,k It is Gaussian white noise with an expected value of 0 and a variance of R.k The superscript f indicates that the current variable value is a predicted value; the superscript a indicates that the value is an observed value.

[0124] Step 6.5: The calculation process returns to step 6.2 to perform the calculation for the next time step.

[0125] Step 6.6: Calculate repeatedly until the preset time, then output the corrected water depth data.

[0126] Example 1

[0127] A city has implemented a smart stormwater management system, establishing over 30 urban flooding monitoring points, which are an important component of the system. Under normal operating conditions, each flooding monitoring device continuously uploads flood depth data with minute-level resolution. Due to the massive volume of uploaded data, it is difficult to manually check the data quality one by one. In this situation, the abnormal data correction algorithm provided by this invention can be used for correction.

[0128] Table 1. Water depth monitoring data

[0129]

[0130] Table 2. Water depth monitoring data from normal monitoring equipment.

[0131]

[0132]

[0133] The water depth monitoring data is used as input data and substituted into the algorithm for correction processing. First, the missing measurement duration is calculated through steps 1 to 3. Based on the missing measurement duration, the type of missing value is determined, and different methods are used to imputate or generate missing data according to the type. After the data filling work is completed, the abnormal fluctuation data and oscillation data are corrected through the progressive processing flow of secondary diagnosis, namely steps 4 to 6.

[0134] Table 3. Corrected water depth data

[0135]

[0136] Table 3 shows the water depth data after correction by the proposed correction algorithm. Compared with the original water depth monitoring data shown in Table 1, it can be seen that the corrected water depth data fills in the missing values ​​in the original data and smooths out the abnormal fluctuations and oscillations in the original water depth monitoring data. This indicates that the algorithm can effectively fill in the missing values ​​in the water depth monitoring data and correct abnormal fluctuations and oscillations.

[0137] Figure 2 This represents the process of change in the original water depth. Figure 3 The image shows the water depth change process after correction by the algorithm proposed in this paper. Comparing the two images, it can be seen that the corrected water depth change process is more reasonable and closer to the actual water depth change pattern at urban waterlogging points.

[0138] Example 2

[0139] The flowchart of the non-contact urban flooding monitoring abnormal data correction method of the present invention is as follows: Figure 1 As shown, please follow these steps:

[0140] Step 1: Determine whether the short-term missing monitoring data is caused by a data transmission failure of the water accumulation monitoring equipment, or by a hardware failure that prevents monitoring.

[0141] Step 1 is implemented in the following steps:

[0142] The calculation formula is as follows:

[0143] t q =t j -t i

[0144] Among them, t q t represents the time interval for missing data. j For the latest missing data time, t i For the moment preceding the first missing data point, when t q When t < α, α is a threshold, which represents short-term missing monitoring data caused by transmission failure; when t < α, α represents the threshold value. q When ≥α, it means that the device cannot monitor due to a hardware failure; i is the data sequence number before the first missing data; j is the latest missing data sequence number; q represents the time interval of the missing data.

[0145] Step 2: Use an interpolation algorithm to interpolate missing values ​​in the short-term water accumulation monitoring data;

[0146] Step 2 is as follows:

[0147]

[0148] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3...; d n d represents the last measured data point before the missing data point was determined. m This is the first measured data point after the missing data point, and Δt is the time interval between monitoring data points. m-n For monitoring data d m and d nThe time interval; n is the data sequence number of the last measured data before the missing data; m is the data sequence number of the first measured data after the missing data; q indicates that the currently calculated water level data is missing data.

[0149] Step 3: Generate data for water accumulation points that cannot be monitored for a long time due to equipment failure, based on water accumulation monitoring equipment values ​​from other similar areas;

[0150] Step 4: Correct any abnormal fluctuations in data caused by vehicles passing by or stopping;

[0151] Step 5: Extract monitoring data during the period of water accumulation;

[0152] Step 5 is implemented in the following steps:

[0153] Due to the large amount of monitoring data, in order to save data correction time, the water accumulation period is identified in step 5 and the water accumulation data within the period is extracted as the input data for step 6.

[0154] To determine the start and end times of a period of water accumulation, if there is no water accumulation for the five minutes preceding a non-zero value, and water accumulation begins in the next moment, then the moment preceding the non-zero value is considered the start time t of this water accumulation period. star If there is no water accumulation for five consecutive minutes after the last non-zero value, the moment t after the non-zero value is considered the end of this water accumulation period. end ,

[0155] The time period t was determined star ~t end Then, the water accumulation monitoring data value d for that period is filtered from the data storage database. i ,i=star,stari+1,…,end.

[0156] Step 6: Correct the oscillation process caused by the car's movement using an ensemble Kalman filter algorithm.

[0157] Step 6 is implemented in the following steps:

[0158] The Ensemble Kalman filter (EnKF) algorithm is used to correct abnormal oscillation data caused by vehicle movement.

[0159] In 1994, influenced by the sequential data assimilation algorithm proposed in stochastic dynamic forecasting theory, the ensemble Kalman filter was proposed. It utilizes the Monte Carlo method, using a set of N random variables conforming to a Gaussian distribution to represent the probability density function in stochastic dynamic forecasting. Through integration, the probability density function of the overall state at the next time step is calculated, and the statistical characteristics (such as mean and covariance) at that time are obtained. This method includes two steps: prediction and analysis. In the prediction stage, the predicted value of the model state at the current time step is generated based on the model state at the previous time step. In the analysis stage, multi-source data is introduced. In this invention, the data used are the original water accumulation monitoring data and its data after moving average processing. Finally, the minimum variance estimation method is used to re-analyze the model state to generate data that eliminates oscillations. In the ensemble Kalman filter algorithm, the Monte Carlo method can generate multiple water accumulation data with different oscillation characteristics based on existing water accumulation data through random perturbation, effectively avoiding random errors caused by using a single water accumulation data for filtering. The ensemble Kalman filter algorithm includes two steps: prediction and analysis. In the prediction phase, filtering generates a predicted value for the current mode state based on the mode state at the previous time step. In the analysis phase, the minimum variance estimation method is used to reanalyze the mode state to generate data that eliminates oscillations.

[0160] The specific process of the filtering algorithm is as follows:

[0161] Step 6.1: Initialize the background field, given N random variables X that conform to a Gaussian distribution. i In this invention, i = 1, ..., N, the state variable is the water depth, i.e., X. i =d i The initial sequence data is perturbed using a Gaussian distribution, and N new sets of sequence data are randomly generated.

[0162] Step 6.2: Calculate the predicted values ​​of each variable at time k+1.

[0163]

[0164] in, It is a variable The predicted value at time k+1; It is a variable The analysis value at time k, where the superscript 'a' indicates that this value is an observed value, M k (·) is a nonlinear model operator; Q k It is the model error variance matrix; w i,k It is Gaussian white noise with an expected value of 0 and a variance of Q. k ;

[0165] Step 6.3: Calculate the Kalman gain matrix at time k+1:

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] in, It is the predicted value of the i-th random variable at time k+1, where the subscript k indicates the time. Let be the predicted value of the i-th state variable at time k+1; R is the predicted average value of the state variable at time k+1. k It is the observation error variance matrix; H k (·) is the observation operator. In this method, both the state variable and the observation are the water depth, so H k (·) represents a scalar; the superscript T denotes the transpose matrix; the superscript f indicates that the current variable value is the predicted value; K k+1 The Kalman gain matrix; H is the state transition matrix; H is the state variable; N is the number of generated sequence data sets.

[0172] Step 6.4: Calculate the average value of the state variables of the analysis field at time k+1. and background field error variance matrix

[0173]

[0174]

[0175]

[0176] in, It is the i-th variable The analysis value at time k+1; It is a state variable The average value at time k+1; v i,k It is Gaussian white noise with an expected value of 0 and a variance of R. k The superscript f indicates that the current variable value is a predicted value; the superscript a indicates that the value is an observed value.

[0177] Step 6.5: The calculation process returns to step 6.2 to perform the calculation for the next time step.

[0178] Step 6.6: Calculate repeatedly until the preset time, then output the corrected water depth data.

[0179] Example 3

[0180] The flowchart of the non-contact urban flooding monitoring abnormal data correction method of the present invention is as follows: Figure 1 As shown, please follow these steps:

[0181] Step 1: Determine whether the short-term missing monitoring data is caused by a data transmission failure of the water accumulation monitoring equipment, or by a hardware failure that prevents monitoring.

[0182] Step 1 is implemented in the following steps:

[0183] The calculation formula is as follows:

[0184] t q =t j -t i

[0185] Among them, t q t represents the time interval for missing data. j For the latest missing data time, t i For the moment preceding the first missing data point, when t q When t < α, α is a threshold, which represents short-term missing monitoring data caused by transmission failure; when t < α, α represents the threshold value. q When ≥α, it means that the device cannot monitor due to a hardware failure; i is the data sequence number before the first missing data; j is the latest missing data sequence number; q represents the time interval of the missing data.

[0186] Step 2: Use an interpolation algorithm to interpolate missing values ​​in the short-term water accumulation monitoring data;

[0187] Step 2 is as follows:

[0188]

[0189] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3...; d n d represents the last measured data point before the missing data point was determined. m This is the first measured data point after the missing data point, and Δt is the time interval between monitoring data points. m-n For monitoring data d m and d n The time interval; n is the data sequence number of the last measured data before the missing data; m is the data sequence number of the first measured data after the missing data; q indicates that the currently calculated water level data is missing data.

[0190] Step 3: Generate data for water accumulation points that cannot be monitored for a long time due to equipment failure, based on water accumulation monitoring equipment values ​​from other similar areas;

[0191] Step 3 is implemented in the following steps:

[0192] Using the faulty device as the center, calculate the distance between each neighboring device and the faulty device, further obtain the shortest distance, and determine the one normal device closest to the faulty device.

[0193] Calculate the distance l between each normally operating device and the faulty device. b :

[0194]

[0195] l min =min(l1,l2,l3,...,l b )

[0196] In the formula, l b y is the distance between normal and faulty equipment; b is the equipment number, b = 1, 2, 3...; x f y f These are the relative coordinates of the faulty device in the x and y directions, respectively; x b y b These are the relative coordinates of each normal device in the x and y directions; min The minimum distance is given; f represents the coordinates of the faulty device.

[0197] Calculate the ratio of monitoring data from the two devices during the most recent flooding event:

[0198]

[0199]

[0200] In the formula, m z The ratio of faulty equipment data to normal equipment data is given by z, where z represents the time sequence of water accumulation data, z = 1, 2, 3...; m average This is the average ratio of all water accumulation data. These are the monitoring values ​​for normal equipment; These are the monitoring values ​​for faulty equipment;

[0201] The monitoring data for the faulty equipment is generated based on the ratio and the monitoring data of the normal equipment.

[0202]

[0203] In the formula, For missing data; i represents the time sequence of the missing data, i = 1, 2, 3, ...; This is monitoring data from normal equipment.

[0204] Step 4: Correct any abnormal fluctuations in data caused by vehicles passing by or stopping;

[0205] Step 4 is implemented in the following steps:

[0206] In urban areas, where roads are heavily paved, surface runoff from rainfall flows along road slopes to low-lying areas, forming water accumulation. During periods of water accumulation, this water can be absorbed through evaporation, infiltration, or gravity flow into the city's drainage network, gradually reducing the water depth. These are all normal variations in urban water depth. However, because commonly used non-contact water accumulation monitoring equipment is installed on poles along the roadside, it is significantly affected by vehicle traffic. This results in abnormal fluctuations in the monitoring data, characterized by suddenness, large amplitude, and high frequency. As traffic volume increases, continuous data oscillations may occur.

[0207] Therefore, a second diagnostic step is used for the identification and correction of "outliers." First, abnormal fluctuations caused by passing vehicles are identified and addressed:

[0208] For the water depth d at time i i ≠0, and the water depth d before time i is ? i-1 The water depth d after time i i+1 In the case where both are equal to 0, let d i =0;

[0209] Secondly, abnormal fluctuation data caused by vehicle parking is identified and processed:

[0210] For the water depth d at time i i ≥γ, where γ is a pre-set water depth threshold, and the water depth d at the time preceding time i is ≥γ. i-1 =0, the water depth d one time after time i i+1 =d i And let d i =d i+1 =0; if d c c = i + 2, i + 3, ... are all equal to d before correction. i Let d c =0, until the monitored value is 0, then the cycle correction ends.

[0211] Step 5: Extract monitoring data during the period of water accumulation;

[0212] Step 5 is implemented in the following steps:

[0213] Due to the large amount of monitoring data, in order to save data correction time, the water accumulation period is identified in step 5 and the water accumulation data within the period is extracted as the input data for step 6.

[0214] To determine the start and end times of a period of water accumulation, if there is no water accumulation for the five minutes preceding a non-zero value, and water accumulation begins in the next moment, then the moment preceding the non-zero value is considered the start time t of this water accumulation period. star If there is no water accumulation for five consecutive minutes after the last non-zero value, the moment t after the non-zero value is considered the end of this water accumulation period. end ,

[0215] The time period t was determined star ~t end Then, the water accumulation monitoring data value d for that period is filtered from the data storage database. i ,i=star,stari+1,…,end.

[0216] Step 6: Correct the oscillation process caused by the car's movement using an ensemble Kalman filter algorithm.

Claims

1. A non-contact method for correcting abnormal data in urban flooding monitoring, characterized in that: The specific steps are as follows: Step 1: Determine whether the short-term missing monitoring data is caused by a data transmission failure of the water accumulation monitoring equipment, or by a hardware failure that prevents monitoring. Step 2: Use an interpolation algorithm to interpolate missing values ​​in the short-term water accumulation monitoring data; Step 3: Generate data for water accumulation points that have been unable to be monitored for a long time due to equipment failure, based on water accumulation monitoring equipment values ​​from other similar areas; Step 3 is implemented in the following steps: Using the faulty device as the center, calculate the distance between each neighboring device and the faulty device, further obtain the shortest distance, and determine the one normal device closest to the faulty device. Calculate the distance between each normally operating device and the faulty device. : In the formula, The distance between normal equipment and faulty equipment; b For equipment number, b =1, 2, 3…; , These are the relative coordinates of the faulty device in the x and y directions, respectively; , These are the relative coordinates of each normal device in the x and y directions; The minimum distance; f represents the coordinates of the faulty device; Calculate the ratio of monitoring data from the two devices during the most recent flooding event: In the formula, This is the ratio of data from faulty equipment to data from normal equipment. z The data on water accumulation is presented in chronological order. z =1,2,3……; This is the average ratio of all water accumulation data. These are the monitoring values ​​for normal equipment; These are the monitoring values ​​for faulty equipment; The monitoring data for the faulty equipment is generated based on the ratio and the monitoring data of the normal equipment. In the formula, Data missing; i The time sequence of the missing data. i =1,2,3……; This is monitoring data from normal equipment; Step 4: Correct any abnormal fluctuations in data caused by vehicles passing by or stopping; Step 4 is implemented in the following steps: First, identify and handle any abnormal fluctuations caused by passing vehicles: for i The water is deep at all times ,and i water depth at the moment before , i water depth at the next moment In the case where both are equal to 0, let =0; Secondly, abnormal fluctuation data caused by vehicle parking is identified and processed: for i The water is deep at all times , The water depth threshold is set in advance, and i water depth at the moment before =0, i water depth at the next moment = And let = =0; if , c = i +2, i +3, ... are all equal to the original value before correction. Then let =0, until the monitored value is 0, then the cycle correction ends; Step 5: Extract monitoring data during the period of water accumulation; Step 6: Correct the oscillation process caused by the car's movement using an ensemble Kalman filter algorithm.

2. The non-contact urban flooding monitoring abnormal data correction method according to claim 1, characterized in that, Step 1 is implemented in the following steps: The calculation formula is as follows: in, t q For the time interval of missing data, For the latest missing data time, For the moment preceding the first missing data point, when hour, The threshold is the short-term missing monitoring data caused by transmission failure; when In other words, it is a hardware failure that prevents the device from monitoring. i The data sequence number preceding the first missing data point; j q represents the latest missing data sequence number; q represents the missing data number.

3. The non-contact urban flooding monitoring abnormal data correction method according to claim 2, characterized in that, Step 2 is described in detail below: In the formula, Data missing; i The time sequence of the missing data. i =1,2,3……; This is the last measured data point before the missing data was obtained. This is the first measured data point after the missing data was obtained. The time interval for monitoring data, For monitoring data and The time interval; n The data sequence number of the last measured data point before the missing data point; m This is the data sequence number of the first measured data after the missing measurement.

4. The non-contact urban flooding monitoring abnormal data correction method according to claim 3, characterized in that, Step 5 is implemented in the following steps: To determine the start and end times of a period of water accumulation, if there is no water accumulation for five minutes before a non-zero value, and water accumulation begins in the next moment, then the moment before the non-zero value is considered the start time of this water accumulation period. t star If there is no water accumulation for five consecutive minutes after the last non-zero value, the moment following the non-zero value is considered the end of this water accumulation period. t end , The time period was determined. t star ~ t end Then, waterlogging monitoring data values ​​for that period are filtered from the data storage database. , i =star, stari+1,…, end.

5. The non-contact urban flooding monitoring abnormal data correction method according to claim 4, characterized in that, Step 6 is implemented in the following steps: The EnKF ensemble Kalman filter algorithm was used to correct abnormal oscillation data caused by vehicle movement. The specific process of the filtering algorithm is as follows: Step 6.1: Initialize the background field and provide... N A random variable that follows a Gaussian distribution X i , i =1, ..., N, where the state variable is the water depth, i.e. The initial sequence data was perturbed using a Gaussian distribution, and then randomly generated. N A new set of sequence data; Step 6.2: Calculate the values ​​of each variable. k Predicted value at time +1 : ， in, It is a variable exist k The predicted value at time +1; It is a variable exist k The analysis value at time 'a' indicates that this value is an observed value. It is a nonlinear model operator; It is the model error variance matrix; It is Gaussian white noise with an expected value of 0 and a variance of . ; Step 6.3, Calculation k Kalman gain matrix at time +1: in, It is the first i A random variable in k The predicted value at time +1, index k Indicates time; For the first i The state variables are in k The predicted value at time +1; Is the state variable in k The predicted average at time +1, R k It is the observation error variance matrix; It is an observation operator; in this method, both the state variable and the observation are the water depth. The variable is a scalar; the superscript T indicates that it is a transpose matrix; the superscript f indicates that the current variable value is the predicted value. The Kalman gain matrix; This is the state transition matrix; H For state variables; N The number of sequence data sets generated; Step 6.4, Calculate k Average value of the analytical field state variables at time +1 and background field error variance matrix : ， in, It is the first i Variables exist k Analysis value at time +1; It is a state variable exist k The average value at time +1; It is Gaussian white noise with an expected value of 0 and a variance of . R k The superscript f indicates that the current variable value is a predicted value; the superscript a indicates that the value is an observed value. Step 6.5: The calculation process returns to step 6.2 to perform the calculation for the next time step; Step 6.6: Calculate repeatedly until the preset time, then output the corrected water depth data.

Images