Sewage pipe network rainfall-inflow event identification method based on semi-supervised iterative training

Abstract

The invention discloses a sewage pipe network rainfall-inflow event identification method based on semi-supervised iterative training, and the method comprises the following steps: S1, obtaining real-time liquid level data, historical liquid level data and real-time rainfall data; s2, the two kinds of liquid level data are preprocessed; s3, obtaining a rainfall sequence, a real-time liquid level sequence and a historical liquid level sequence of a unified time scale; s4, obtaining a trend term by adopting STL; s5, obtaining a state recognition model; s6, obtaining a candidate rainfall-inflow event; s7, judging whether a time interval threshold value is reached or not, and if not, entering the next step; if yes, whether the period is adjusted or not is judged based on the candidate rainfall-inflow event frequency, if not, the next step is executed, and if yes, the fourth step is executed; and S8, outputting a result in real time. According to the method, the hidden state sequence of the liquid level change is recognized in combination with the STL model and the HMM model, and long-term, stable and low-labor-cost recognition of the rainfall inflow event of the sewage pipe network is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pipe network monitoring, and in particular relates to a sewage pipe network rainfall-inflow event recognition method based on semi-supervised iterative training. Background Art

[0002] Urban drainage systems are critical infrastructure for ensuring safe urban operations and the quality of life for residents. With the continuous advancement of urbanization, problems such as mixed rainwater and sewage connections, aging pipes, and illegal connections have become increasingly prominent, leading to significant inflows of external water into the drainage network during rainfall. These inflows not only increase the processing load on sewage treatment plants but can also cause environmental risks such as sewage overflows and water pollution. Therefore, timely and accurate identification of abnormally high pipe level increases caused by rainfall is crucial for urban water management, drainage system maintenance, and risk warning.

[0003] Currently, methods for identifying rainfall inflow events primarily rely on manual experience or threshold-based judgment methods. For example, preliminary judgments are made by observing whether a sudden increase in liquid level coincides with a rainfall period or whether flow rate changes exceed empirical limits. However, these methods have significant limitations. On the one hand, manual judgment is inefficient and highly subjective, making it difficult to adapt to large-scale, multi-site online monitoring data. On the other hand, static thresholds cannot cope with diverse drainage characteristics and dynamically changing environmental conditions, resulting in high misjudgment rates and poor adaptability. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method for identifying rainfall-inflow events in sewage pipe networks with semi-supervised iterative training, which has high accuracy, strong adaptability and can realize automatic identification.

[0005] Technical solution: The present invention discloses a method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training, comprising the following steps: S1: Obtain the real-time liquid level data and historical liquid level data of the target monitoring site, as well as the real-time rainfall data of the area corresponding to the target monitoring site; S2: Preprocess the real-time liquid level data and historical liquid level data by removing outliers and filling missing values ​​respectively; S3: resampling the real-time rainfall data, pre-processed real-time liquid level data and historical liquid level data to obtain rainfall series, real-time liquid level series and historical liquid level series with a unified time scale; S4: Use the STL time series decomposition method to decompose the real-time liquid level sequence and the historical liquid level sequence respectively, and obtain the trend item T of the real-time liquid level sequence respectively. t实 and the trend term T of the historical liquid level series t历 ; S5: Construct a hidden Markov model and use the trend term T t历 The time series is used as the observation sequence, and the Baum-Welch expectation maximization algorithm is used to perform unsupervised training on the hidden Markov model to obtain a state recognition model for predicting the liquid level hidden state sequence; S6: The trend item T of the real-time liquid level data t实 The state recognition model is input, and the state recognition model outputs a liquid level hidden state sequence; the "high liquid level" state segment in the liquid level hidden state sequence is screened, and the "high liquid level" state segment that meets the preset conditions is identified as a candidate rainfall-inflow event; S7: Set a time interval threshold for state recognition model calibration. If the time interval threshold is not reached, directly proceed to the next step. When the time interval threshold is reached, calculate the occurrence frequency of the candidate rainfall-inflow event within the specified period, and compare it with the artificial experience frequency or the artificially labeled sample frequency. If the difference is less than the preset frequency error threshold, proceed to the next step. If the difference is greater than or equal to the frequency error threshold, increment the seasonal period parameter period value in the STL time series decomposition method at equal intervals, and re-execute steps S4 to S7 to achieve adaptive adjustment and dynamic iterative training of the state recognition model until the difference is less than the preset frequency error threshold. S8: Output the final identified candidate rainfall-inflow events in a structured record format in real time for subsequent display, analysis, or decision support.

[0006] Furthermore, in step S2, the real-time liquid level data and the historical liquid level data are preprocessed as follows: an adaptive 3σ anomaly detection algorithm based on a sliding time window is used to identify outliers for the two types of liquid level data respectively, and after removing the identified outliers, the linear interpolation method within the adjacent time window is used to fill in the outliers and missing values.

[0007] Furthermore, in step S3, the adaptive 3σ anomaly detection algorithm based on a sliding time window is used to identify outliers in the liquid level data as follows: , where LB represents the lower limit of the normal value of the liquid level data in the past week, UB represents the upper limit of the normal value of the liquid level data in the past week, μ represents the mean of the liquid level data of the target monitoring station in the past week, and σ represents the standard deviation of the liquid level data of the target monitoring station in the past week; When the monitoring value of the liquid level data is lower than LB or higher than UB, it is judged as an abnormal value; The linear interpolation method in the adjacent time window is used to fill the outliers and missing values ​​as follows: , where S tIndicates the missing value or abnormal value of the liquid level data at time t, D t Indicates S t Timestamp; S t-1 Indicates the distance S in the liquid level data t The normal value of the most recent moment, D t-1 Indicates S t-1 Timestamp; S t+1 Indicates the distance S in the liquid level data t The normal value at the most recent moment, D t+1 Indicates S t+1 timestamp.

[0008] Furthermore, the resampling method in step S3 is the sliding window averaging method, and the sampling method is as follows: determine the sampling frequency, the initial period of the sampling time series, and the terminal period of the sampling time series, and calculate the regularized time point sequence based on this; for each target sampling point, take its data within the forward T minutes range as a sliding window, and calculate the average liquid level in the window, and use the average as the resampling value of the target sampling point.

[0009] Furthermore, in step S4, the STL time series decomposition method decomposes the two liquid level sequences into the following formulas based on the local weighted regression algorithm: , where y t Represents liquid level sequence data, S t represents the seasonal term, T t represents the trend term, R t represents the residual term.

[0010] Furthermore, the steps for training the hidden Markov model are as follows: S51: The double-hidden-state Gaussian HMM in the hidden Markov model is selected for training, and the number of hidden states of the HMM is set to 2. The hidden state S1 represents the "normal fluctuation" state of the liquid level under normal operation, and the hidden state S2 represents the "high liquid level" state under abnormal operation. The hidden state set is S={S1,S2}; Use the trend item T of historical level data t历 The liquid level monitoring value represents the observation state O, and the observation state set is ; S52: Use K-means clustering method to analyze the trend item T of historical liquid level data t历 Perform clustering and divide it into two data clusters; Based on the clustering results, the initial observation probability distribution parameter set {μ1, σ1, μ2, σ2} is constructed, where μ1 is the observed mean of the hidden state S1, σ1 is the observed standard deviation σ1 of the hidden state S1, μ2 is the observed mean of the hidden state S2, and σ2 is the observed standard deviation σ2 of the hidden state S2; Based on the clustering results, the initial state probability vector is obtained and the trend items T contained in the two data clusters are calculated respectively. t历 The number of trend items in the total T t历 The proportion π in the initial state probability vector π=[p1,p2] is obtained, where p1 refers to a trend item T t历 The probability of belonging to S1, p2 refers to the trend term T t历 The probability of belonging to S2, and p1+p2=1; S53: Construct the initial state transition probability matrix A, , and set A to a high self-transition probability structure; where a ij It represents the probability that the hidden state at time t+1 changes to j when the hidden state at time t is i; S54: Calculate the observation state probability matrix B, , the observation probability density function under each hidden state obeys the one-dimensional normal distribution , Indicates that in the hidden state S i At time t, the liquid level value o is observed N The observation probability density of μ i is the hidden state S i The mean parameter, σ i is the hidden state S i The standard deviation parameter of S55: Use the Baum-Welch expectation maximization algorithm to perform unsupervised iterative training on HMM, by maximizing the trend term T t历 The log-likelihood function of the corresponding observation sequence is used to optimize the HMM model parameters; S56: Setting the stopping condition of the unsupervised iterative training: the log-likelihood increment is lower than the preset convergence threshold or the number of iterations reaches the preset upper limit; S57: After the unsupervised iterative training stops, the trained HMM obtained is the intermediate model, and the intermediate model outputs the corresponding trend term T t历 The hidden state time series of the observation sequence is then compared with the difference in the observation means of the two hidden states. If the difference is greater than or equal to the preset threshold, the intermediate model converges and is used as the state recognition model; if the difference is less than the preset threshold, the intermediate model is optimized and trained again until the difference is greater than or equal to the preset threshold.

[0011] Furthermore, the preset threshold in step S57 is set as follows: Assume that the trend term T of the historical liquid level sequence is t历 The sequence of liquid level monitoring values ​​is {x1,x2,x3,…,x Q}; Calculate {x1,x2,x3,…,x Q The mean of ; Based on the mean Calculate {x1,x2,x3,…,x Q The global standard deviation σ of global , and the global standard deviation σ global As the preset threshold, , where q∈[1,Q].

[0012] Furthermore, the process of optimizing and training the intermediate model again in step S57 is as follows: S571: If the difference between the observed means of the two hidden states is less than the preset threshold, adjust the elements in the initial state transition probability matrix A; the automatic adjustment method is: increase The conversion probability is reduced The conversion probability is maintained The conversion probability of S572: Obtain historical liquid level data without heavy rainfall interference, and use the STL time series decomposition method to obtain the trend item of the historical liquid level data. Then, use the probability of the trend item belonging to the hidden state S1 and the hidden state S2 to reset the initial state probability vector π: If the proportion of continuous high liquid level segments in the training set exceeds 20%, increase p2 to the range of 0.1-0.2; S573: After adjusting A and π, return to step S55 to perform unsupervised iterative training on the HMM again until the difference between the observed means of the two hidden states in step S57 is greater than or equal to the preset threshold; if the condition that the difference between the observed means of the two hidden states is greater than or equal to the preset threshold is still not met after three cycles, the intermediate model obtained in the last cycle is used as the state recognition model.

[0013] Furthermore, the preset conditions in step S6 are: ① the duration of the hidden state time series of the "high liquid level" state is not less than T1 hours and the cumulative rainfall of a single rainfall is not less than R mm; ② the interval between the hidden state time series of the "high liquid level" state and the previous rainfall is not less than T2 hours; if conditions ① and ② are met at the same time, it is identified as a candidate rainfall-inflow event.

[0014] In step S7, the seasonal period parameter period value of the STL time series decomposition method is adjusted as follows: in the initial state, the seasonal period parameter period is set to period0, and in the nth iteration, the seasonal period parameter period is period0+nΔ period , where Δ period is the increment step of the period parameter.

[0015] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: the present invention can effectively separate the trend, period and noise components in the liquid level data through the STL time series decomposition algorithm, and then combine with the Gaussian hidden Markov model (HMM) to identify the hidden state sequence of liquid level changes, which can accurately determine whether the liquid level anomaly is caused by rainfall, thereby improving the recognition accuracy of rainfall inflow events; the hidden Markov model training in the present invention constitutes an inner layer iteration, and each training of the hidden Markov model ensures its full convergence; the seasonal period parameter period of the STL time series decomposition method is adjusted according to the matching degree to constitute an outer layer iteration, and the double iteration ensures the robustness and accuracy of the state recognition model. In addition, the present invention combines the STL and HMM models to identify the hidden state sequence of liquid level changes, without the need for manual labeling one by one, and has self-learning capabilities. It is suitable for deployment on an intelligent monitoring platform to achieve long-term, stable, and low-labor-cost recognition of rainfall inflow events.

[0016] The present invention gradually expands the seasonal cycle parameter period of the STL time series decomposition method, which can suppress the influence of atypical short-term fluctuations, realize multi-scale trend extraction from the short term to the medium and long term, and introduces an adjustable and adaptive time scale dimension for the Markov model to identify hidden states; different from the existing technology of statically decomposing time series with a fixed period, the present invention dynamically adjusts the seasonal cycle parameter period of the STL time series decomposition method to adapt to the differences in liquid level response scales of urban drainage systems in different seasons, weather or working conditions, and avoids the situation where abnormal signals are buried or misjudged due to static decomposition of time series with a fixed period. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the present invention; Figure 2 This is a trend item comparison chart after adjusting the seasonal cycle parameter period in an embodiment of the present invention; Figure 3 This is a diagram showing the real-time liquid level data recognition result according to an embodiment of the present invention; Figure 4 This is a result output diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0019] Example 1

[0020] The present invention discloses a method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training. Figure 1 As shown, the following steps are included: S1: Obtain the real-time liquid level data and historical liquid level data of the target monitoring site, as well as the real-time rainfall data of the area corresponding to the target monitoring site.

[0021] The sampling frequency of real-time liquid level data shall not be less than five minutes each time; the real-time rainfall data shall be the rainfall data measured at the target monitoring station or the grid rainfall data corrected by inversion.

[0022] S2: Preprocess the real-time liquid level data and historical liquid level data by removing outliers and filling missing values ​​respectively.

[0023] The preprocessing method for real-time liquid level data and historical liquid level data is as follows: an adaptive 3σ anomaly detection algorithm based on a sliding time window is used to identify outliers for the two types of liquid level data respectively. After removing the identified outliers, the linear interpolation method within the adjacent time window is used to fill in the outliers and missing values.

[0024] The adaptive 3σ anomaly detection algorithm based on sliding time window is used to identify outliers in liquid level data as follows: , where LB represents the lower limit of the normal value of the liquid level data in the past week, UB represents the upper limit of the normal value of the liquid level data in the past week, μ represents the mean of the liquid level data of the target monitoring station in the past week, and σ represents the standard deviation of the liquid level data of the target monitoring station in the past week; When the monitored value of the liquid level data is lower than LB or higher than UB, it is judged as an abnormal value. To address the common problems of missing values ​​and abnormal fluctuations in drainage monitoring data, the present invention adopts the 3σ method of sliding window to dynamically identify abnormal points and uses a time interpolation algorithm to repair them, which significantly improves the data availability and the input quality of the subsequent HMM model and avoids misjudgment due to abnormal data.

[0025] The linear interpolation method in the adjacent time window is used to fill the outliers and missing values ​​as follows: , where S t Indicates the missing value or abnormal value of the liquid level data at time t, D t Indicates S t Timestamp; S t-1 Indicates the distance S in the liquid level data t The normal value of the most recent moment, D t-1 Indicates S t-1 Timestamp; S t+1 Indicates the distance S in the liquid level data t The normal value at the most recent moment, D t+1 Indicates S t+1 Timestamp. Timestamp is represented as 2023-11-30 11:00:00.

[0026] S3: Resample the real-time rainfall data, pre-processed real-time liquid level data and historical liquid level data to obtain rainfall series, real-time liquid level series and historical liquid level series with a unified time scale.

[0027] The resampling method is a sliding window averaging method, and the sampling method is as follows: the sampling frequency, the initial period of the sampling time series, and the final period of the sampling time series are determined, and based on this, a regularized time point sequence is calculated. For each target sampling point, the data within the range of T minutes before is used as a sliding window, and the mean liquid level within this window is calculated, and this mean is used as the resampled value for the target sampling point. Using the window mean as the value at this time point can condense high-frequency data, thereby obtaining rainfall series, real-time liquid level series, and historical liquid level series at regular time intervals.

[0028] S4: Use the STL time series decomposition method to decompose the real-time liquid level sequence and the historical liquid level sequence respectively, and obtain the trend item T of the real-time liquid level sequence respectively. t实 and the trend term T of the historical liquid level series t历 .

[0029] The STL time series decomposition method decomposes the liquid level sequence into the following formula based on the local weighted regression algorithm: , where y t Represents liquid level sequence data, S t represents the seasonal term, T t represents the trend term, R t represents the residual term. The trend term reflects long-term changes in liquid level, the seasonal term captures periodic fluctuations (such as diurnal or seasonal variations), and the residual term retains random fluctuations and anomalies. The STL time series decomposition method can remove normal periodic fluctuations, making abnormal changes more prominent in the residual. In subsequent steps, the trend term is used instead of the liquid level series as the input to the state recognition model, which can reduce the adverse effects of fluctuations and noise in the pipeline network liquid level data on the recognition effect.

[0030] S5: Construct a hidden Markov model and use the trend term T t历 The time series of liquid level is used as the observation sequence, and the Baum-Welch expectation maximization algorithm is used to perform unsupervised training on the hidden Markov model to obtain a state recognition model for predicting the liquid level hidden state sequence.

[0031] The present invention constructs an automated recognition process through the STL time series decomposition method and the hidden Markov model, which can achieve accurate judgment of potential rainfall inflow events in multi-source monitoring data, and is conducive to improving the overall recognition efficiency and engineering adaptability.

[0032] The steps to train a hidden Markov model are as follows: S51: Double-hidden-state Gaussian HMM in the hidden Markov model is selected for training. The characteristics of double-hidden-state Gaussian HMM are more suitable for the continuity and fluctuation characteristics of liquid level change data. The number of hidden states of HMM is set to 2, and the hidden state S1 represents the "normal fluctuation" state of the liquid level under normal operation, and the hidden state S2 represents the "high liquid level" state under abnormal operation. The resulting hidden state set is S={S1,S2}; the "high liquid level" state is often seen in the "continuous rise" or "high liquid level" state caused by external interference.

[0033] Use the trend item T of historical level data t历 The liquid level monitoring value represents the observation state O, and the observation state set is .

[0034] S52: Use K-means clustering method to analyze the trend item T of historical liquid level data t历 Clustering is performed to divide the data into two data clusters; the purpose of this step is to pre-segment the data based on the global statistical characteristics of the historical liquid level data using the unsupervised K-means clustering method (setting the number of categories to 2) to obtain a preliminary state division.

[0035] Based on the clustering results, the initial observation probability distribution parameter set {μ1, σ1, μ2, σ2} is constructed, where μ1 is the observed mean of hidden state S1, σ1 is the observed standard deviation of hidden state S1, μ2 is the observed mean of hidden state S2, and σ2 is the observed standard deviation of hidden state S2. In addition, μ1 is the mean of the data cluster corresponding to hidden state S1, σ1 is the variance of the data cluster corresponding to hidden state S1, μ2 is the mean of the data cluster corresponding to hidden state S2, and σ2 is the variance of the data cluster corresponding to hidden state S2.

[0036] Based on the clustering results, the initial state probability vector is obtained and the trend items T contained in the two data clusters are calculated respectively. t历 Quantity in the total trend item T t历 The proportion π in the initial state probability vector π=[p1,p2] is obtained, where p1 refers to a trend item T t历 The probability of belonging to S1, p2 refers to the trend term T t历 The probability of belonging to S2, and p1+p2=1.

[0037] S53: Construct the initial state transition probability matrix A, , and set A to a high self-transition probability structure; where a ij It indicates the probability that the hidden state at time t+1 changes to j when the hidden state at time t is i. The high self-transition probability structure means that there are some states i such that a ii Much larger than a ij(and j≠i), that is, the probability of staying in the current state is much greater than the probability of transferring to other states; if the diagonal elements of A are > 0.8, such as , to enhance the temporal continuity and stability of the state.

[0038] S54: Calculate the observation state probability matrix B, , the observation probability density function under each hidden state obeys the one-dimensional normal distribution , Indicates that in the hidden state S i At time t, the liquid level value o is observed N The observation probability density is defined by the above observation probability density function (Gaussian distribution, also known as one-dimensional normal distribution), where μ i is the hidden state S i The mean parameter, σ i is the hidden state S i The standard deviation parameter of ; Through this function, the likelihood of any observation value in a given hidden state can be calculated, which is used for the inference and model training of the hidden state sequence, that is, after determining o N After the hidden state is obtained, the observation value is generated according to the observation probability density function (Gaussian distribution, also known as one-dimensional normal distribution) corresponding to the hidden state.

[0039] S55: Use the Baum-Welch expectation maximization algorithm to perform unsupervised iterative training on HMM, by maximizing the trend term T t历 The log-likelihood function of the corresponding observation sequence is used to optimize the HMM model parameters.

[0040] S56: Set the stopping condition for unsupervised iterative training: the log-likelihood increment is lower than the preset convergence threshold or the number of iterations reaches the preset upper limit; in this embodiment, the convergence threshold is set to 10⁻ 4 , set the upper limit of the iteration rounds to 100 rounds.

[0041] S57: After the unsupervised iterative training stops, the trained HMM obtained is the intermediate model, and the intermediate model outputs the corresponding trend term T t历 The hidden state time series of the observation sequence is then compared with the difference between the observed means of the two hidden states. If the difference is greater than or equal to the preset threshold, the intermediate model has converged and is used as the state recognition model. If the difference is less than the preset threshold, the intermediate model is optimized and trained again until the difference is greater than or equal to the preset threshold. If the difference between the observed means of the two hidden states is less than the preset threshold, it indicates that the "normal state" and "high liquid level" are difficult to separate, and the state recognition model structure needs to be adjusted.

[0042] Preferably, the preset threshold is set as follows: the trend item T of the historical liquid level sequence is set as t历The sequence of liquid level monitoring values ​​is {x1,x2,x3,…,x Q}; Calculate {x1,x2,x3,…,x Q The mean of ; Based on the mean Calculate {x1,x2,x3,…,x Q The global standard deviation σ of global , and the global standard deviation σ global As the preset threshold, , where q∈[1,Q].

[0043] The process of optimizing and training the intermediate model again is as follows: S571: If the difference between the observed means of the two hidden states is less than the preset threshold, adjust the elements in the initial state transfer probability matrix A; the elements in the initial state transfer probability matrix A are adjusted to enhance the discrimination ability of the model. The automatic adjustment method is: increase The conversion probability is reduced The conversion probability is maintained Increase the conversion probability. To enhance the self-transition probability of the high liquid level state, for example, to increase S2 to 0.95 to enhance the stability of the "high liquid level" state; to reduce It is to reduce the conversion probability from normal state to high liquid level, for example, to 0.05, to avoid slight fluctuations being misjudged as abnormalities; maintain The conversion probability is to maintain the natural continuity of the "normal state", for example, to control it at around 0.9.

[0044] S572: Obtain historical liquid level data without heavy rainfall interference and use the STL time series decomposition method to obtain the trend term of this historical liquid level data. The initial state probability vector π is reset using the probability of this trend term belonging to hidden states S1 and S2. Without heavy rainfall interference, the liquid level time series exhibits a low-frequency trend dominated by daily periodic fluctuations, with smooth changes and no sudden increases or decreases. Decomposing this liquid level series using the STL time series decomposition method and extracting this trend component as a modeling basis can eliminate the interference of short-term fluctuations on hidden state recognition.

[0045] In step S52, the initial state probability vector π is set by default to π=[0.95, 0.05], which means that the model is more likely to start from the "normal state" initially; If the proportion of continuous high liquid level segments in the training set exceeds 20%, increase p2 to the range of 0.1-0.2; After adjusting A and π, the process returns to step S55 and iterates the HMM again, unsupervised, until the difference between the observed means of the two hidden states in step S57 is greater than or equal to the preset threshold. If the difference between the observed means of the two hidden states is still not greater than or equal to the preset threshold after three cycles, the intermediate model obtained in the last cycle is used as the state recognition model. Setting an upper limit of three cycles helps to ensure a balance between efficiency and performance in HMM training.

[0046] S6: The trend item T of the real-time liquid level data t实 The state recognition model is input and outputs a liquid level latent state sequence. The "high liquid level" state segments in the liquid level latent state sequence are screened, and the "high liquid level" state segments that meet the preset conditions are identified as candidate rainfall-inflow events.

[0047] When the trend item T of the real-time liquid level data is t实 By inputting the observation sequence into the state recognition model, we can obtain the corresponding latent state time series. In the context of liquid level data, this latent state time series represents the current liquid level status, such as high, normal, or low. In the case of rainfall inflow into a pipeline, the liquid level typically remains persistently high. Therefore, the HMM model can be used to identify the latent state of the liquid level data and automatically isolate the time periods of high liquid level values.

[0048] The preset conditions in step S6 are: ① The duration of the hidden state time series of the "high liquid level" state is not less than T1 hours and the cumulative rainfall of a single rainfall is not less than R mm; ② The interval between the hidden state time series of the "high liquid level" state and the previous rainfall is not less than T2 hours; if conditions ① and ② are met at the same time, it is identified as a candidate rainfall-inflow event.

[0049] In this embodiment, T1 is set to 3 hours, R is set to 2 mm, and T2 is set to 12 hours. In actual application, T1, R, and T2 can be adjusted according to the regional hydrological characteristics. For the rainfall time series in the target monitoring point area, the effective rainfall events are divided according to the two conditions that the cumulative rainfall of a single rainfall is greater than 2 mm and the interval with the previous rainfall is more than 12 hours; for the hidden state time series of the high liquid level identified by the state recognition model, the liquid level continuous high value period with the continuous repetition time of the hidden state time series of the "high liquid level" state for 3 hours or more is screened out; based on the effective rainfall period and the liquid level continuous high value period obtained in the above two steps, the overlapping period of the two periods is first screened out, and then the intersection of the effective rainfall period and the liquid level continuous high value period is the identified candidate rainfall-inflow event.

[0050] S7: Set the time interval threshold for state recognition model calibration. If the time interval threshold is not reached, proceed directly to the next step. When the time interval threshold is reached, calculate the occurrence frequency of the candidate rainfall-inflow event within the specified period, and compare it with the artificial experience frequency or the artificially labeled sample frequency. If the difference is less than the preset frequency error threshold, proceed to the next step. If the difference is greater than or equal to the frequency error threshold, increase the seasonal period parameter period value in the STL time series decomposition method in an equidistant manner, and re-execute steps S4 to S7 to achieve adaptive adjustment and dynamic iterative training of the state recognition model until the difference is less than the preset frequency error threshold.

[0051] Preferably, the frequency error threshold is set to 20%.

[0052] Preferably, the time interval threshold is set to half a month, that is, the frequency of occurrence of candidate rainfall-inflow events within a specified period and the artificial experience frequency or the artificial annotation sample frequency are evaluated once every half a month, and it is determined whether the seasonal period parameter period of the STL time series decomposition method needs to be adjusted. In actual application, the user can set the time interval threshold according to actual needs. Setting the time interval threshold for state recognition model calibration can not only avoid the problem of low operating efficiency of the state recognition model caused by real-time evaluation of the frequency of occurrence of a candidate rainfall-inflow event within a specified period and the artificial experience frequency or the artificial annotation sample frequency, but also does not affect the normal application of the state recognition model.

[0053] Artificial empirical frequency: By acquiring a small amount of artificial prior knowledge, we construct empirical frequencies of inflow events during the dry and rainy seasons. For example, based on years of experience and three years of historical data, we determine that inflow events occur on average once every 20 days during a typical dry season and once every five days during the rainy season. This frequency can be used as a reference benchmark for the event recognition accuracy of the state recognition model.

[0054] Manually labeled sample frequency refers to the process of manually identifying and labeling some rainfall-inflow events under certain conditions, combined with the output of the preliminary model, to establish a real sample library for further evaluation of the accuracy of the state recognition model. The preliminary model refers to the state recognition model trained in step S5 and not yet fed with real-time liquid level data.

[0055] For example, the first iteration of the state recognition model showed an event approximately every five days in the dry season and every two days in the rainy season. However, historical experience indicates that the actual frequency of events is approximately every 30 days in the dry season and every 10 days in the rainy season. The overestimation of the state recognition model's results may be due to oversensitivity caused by setting the STL parameter too low. Therefore, it is necessary to subsequently increase the seasonal period parameter, i.e., the seasonal smoothing window.

[0056] The way to adjust the seasonal cycle parameter period of the STL time series decomposition method is: in the initial state, the seasonal cycle parameter period is set to period0, and when performing the nth iteration, the seasonal cycle parameter period is period0+nΔ. In this embodiment, period0 is set to 24 hours, and Δ is set to 1 hour, that is, the seasonal cycle parameter period increases by 1 hour for each iteration. Gradually expand the seasonal cycle parameter period of the STL time series decomposition method, that is, gradually expand the smoothing window width of the trend term in the STL time series decomposition method, suppress the influence of atypical short-term fluctuations, and thus enhance the expression ability of the "persistent liquid level rise" feature. Gradually expanding the seasonal cycle parameter period of the STL time series decomposition method not only realizes multi-scale trend extraction from short-term to medium- and long-term, but also introduces an adjustable and adaptive time scale dimension for the Markov model to identify hidden states, which is significantly different from the existing technology that only statically decomposes time series with a fixed period. As Figure 2 As shown in the figure, the seasonal period parameter period of the STL time series decomposition method is adjusted to 24 hours and 96 hours respectively. It can be seen from the figure that after adjusting the seasonal period parameter period, the trend time series of the same original liquid level is different. Figure 2 The “original liquid level” here refers to the liquid level data, and the “trend time series” refers to the trend item obtained after the liquid level data is decomposed by STL.

[0057] In this way, steps S4-S7 are iterated repeatedly, and the state recognition model parameters are continuously adjusted so that the frequencies of identified dry season and rainy season events gradually approach the empirical values. When the difference between the candidate rainfall-inflow event density output by the state recognition model and the artificial empirical frequency or the artificially labeled sample frequency is less than the frequency error threshold, it is considered that the parameter selection is reasonable and the state recognition model has reached calibration balance. In this embodiment, the hidden Markov model training constitutes an inner iteration, ensuring full convergence each time; the seasonal period parameter period of the STL time series decomposition method is adjusted according to the matching degree to constitute an outer iteration. This dual iteration ensures the robustness and accuracy of the state recognition model.

[0058] Traditional STL time series decomposition methods are mainly applied with fixed periods, which makes it difficult to adapt to the differences in liquid level response scales in urban drainage systems under different seasons, weather or working conditions, and easily causes abnormal signals to be buried or misjudged. The present invention combines "equally spaced periodic increments + outer trend density feedback iteration + inner HMM state recognition retraining" to construct a "trend density-aware time series decomposition strategy" that can achieve the following effects: (1) Dynamic matching trend scale: The gradual increase of the seasonal period parameter period can achieve a smooth evolution of the trend item from daily changes to weekly scales, so that the HMM model automatically captures the real response cycle of the liquid level affected by rainfall; (2) Prevent overfitting and oversensitivity: A smaller seasonal period parameter period may mistakenly identify occasional fluctuations as events. The increment strategy can effectively filter out short-term anomalies and improve recognition stability. Qualitative; (3) Enhance the adaptive ability of the HMM model: Without manual intervention, the HMM model automatically completes the self-calibration of the trend extraction scale; (4) Introduce a structural feedback loop: Through the feedback of the matching deviation between the candidate rainfall-inflow event density and the prior frequency, the cycle is guided to increase, and the outer trend adjustment participates in the coordinated optimization of the inner model training; (5) Break through the conventional static setting paradigm: The existing STL time series decomposition method is mostly based on empirical setting cycles. The present invention proposes a clear dynamic adjustment logic and automatic convergence mechanism, which can adapt to the differences in the liquid level response scale of the urban drainage system in different seasons, weather or working conditions.

[0059] During the iterative process of steps S4-S7, the STL time series decomposition method in step S4 is as follows: the liquid level data is re-decomposed based on the updated period value, and a ternary structure of seasonal term, trend term and residual term is adopted, where the trend term T t It is used for HMM modeling in the next step. STL uses LOESS local regression method to estimate the trend curve. The trend bandwidth affected by period determines the trend response time scale. Step S5 uses the updated trend sequence T t , retraining the hidden Markov model (HMM). The Baum-Welch algorithm is used to update the state transition matrix A, the initial observation probability distribution parameter set {μ1, σ1, μ2, σ2}, and the initial state probability vector π in each iteration until the log-likelihood function converges, ensuring that the HMM achieves a local optimal fit under each new trend. In steps S6 and S7, event recognition rules are applied to the "high liquid level" state sequence predicted by the state recognition model, re-extracting rainwater inflow events and statistically analyzing the event density during the dry and wet seasons.

[0060] S8: Output the final identified candidate rainfall-inflow events in a structured record format in real time for subsequent display, analysis, or decision support.

[0061] Preferably, the results are finally output in real time in a structured record format, including the start and end time of the rainfall-inflow event, the corresponding rainfall amount and the liquid level change amplitude, etc., and are presented in a structured format, which is convenient for connecting to the smart water system for automatic early warning, operation and maintenance scheduling or model calibration during actual use. These results can be visually displayed on the monitoring platform, or provided for further analysis and decision-making by operation and maintenance personnel. At the same time, since the rainfall inflow characteristics (the start and end time of the rainfall-inflow event, the corresponding rainfall amount and the liquid level change amplitude, etc.) may change over time (such as the impact of climate change or pipeline network reconstruction), and the time interval threshold is set, the present invention supports regular retraining: after accumulating more unlabeled data, the above-mentioned iterative calibration process is repeated periodically to update the state recognition model parameters.

[0062] The process of identifying inflow events in the present invention can be fully automatically executed without manual intervention. It is suitable for deployment in urban drainage information platforms or water dispatching systems, achieving real-time response at the minute level, and is suitable for automatic processing of large-scale multi-site liquid level data. In addition, the key identification parameters (such as T1, T2, R) in the present invention can be flexibly configured according to the characteristics of different cities or regions, have strong scene adaptability and promotion capabilities, and support refined management needs. The present invention can achieve accurate identification and early warning of rainfall inflow events, effectively prevent urban risks such as pipe network overflow and water pollution, reduce unnecessary loads on sewage treatment systems, reduce manual inspection costs and operation and maintenance manpower investment, and help urban drainage systems transform towards digitalization and intelligence.

[0063] Example 2

[0064] The present invention discloses a method for identifying rainfall-inflow events in a sewage pipe network using semi-supervised iterative training, and the method is applied to a SCADA00182 station in a drainage system in a certain urban area of ​​Beijing, comprising the following steps: The SCADA00182 site is used as the target monitoring site, and step S1 in Example 1 is executed, wherein the time resolution of the real-time liquid level data is 5 minutes, the collection period is not less than 30 days, and the time resolution of the real-time rainfall data is 5 minutes. The real-time liquid level data collected in this embodiment is as follows Figure 3 As shown in b and c in FIG, the real-time rainfall data collected in this embodiment is as follows Figure 3 As shown in a in .

[0065] Execute step S2 in Example 1 to pre-process the real-time liquid level data and the historical liquid level data by removing outliers and filling missing values.

[0066] Execute step S3 in Example 1, and regularize the rainfall data, preprocessed real-time liquid level data and historical liquid level data into a time series of 1 data point per hour, slidingly calculate the window mean of each time point and its previous 60 minutes, and use the window mean as the value of the time point.

[0067] Execute step S4 in Example 1, use the STL time series decomposition method to decompose the real-time liquid level sequence and the historical liquid level sequence respectively, and obtain the trend item T of the real-time liquid level sequence. t实 and the trend term T of the historical liquid level series t历 .

[0068] Execute step S5 in Example 1 to construct a hidden Markov model and use the trend term T t历 The time series of liquid level is used as the observation sequence, and the Baum-Welch expectation maximization algorithm is used to perform unsupervised training on the hidden Markov model to obtain a state recognition model for predicting the liquid level hidden state sequence.

[0069] S6: The trend item T of the real-time liquid level data t实 Input the state recognition model obtained in step S5 to obtain the hidden state of the corresponding observation sequence as follows Figure 3 As shown in b.

[0070] The state recognition model outputs a hidden state time series with a high liquid level as the recognition result. Filtering conditions are set, and the hidden state time series with a "high liquid level" state as the recognition result is filtered. The hidden state time series of the "high liquid level" state that meets the filtering conditions is determined as a candidate rainfall-inflow event. For the rainfall time series in the target monitoring point area, effective rainfall events are divided according to the two conditions that the cumulative rainfall of a single rainfall is greater than 2mm and the interval with the previous rainfall is more than 12h. For the hidden state time series of the "high liquid level" state identified by the state recognition model, the liquid level continuous high value period with a continuous repetition time of 3h or more of the hidden state time series of the "high liquid level" state is filtered out. Based on the effective rainfall period and the liquid level continuous high value period obtained in the above two steps, the overlapping period of the two periods is first filtered out, and then the intersection of the effective rainfall period and the liquid level continuous high value period is the identified candidate rainfall-inflow event. The candidate rainfall-inflow event finally obtained in this embodiment is as follows: Figure 3 As shown in c. Figure 3 As shown in a and b, the proposed method effectively eliminates fluctuations and avoids misidentifying transient high values ​​caused by fluctuations as high values. When continuous high values ​​are detected over the past day, real-time rainfall data is further combined to identify the time period of inflow events caused by rainfall, achieving results very close to those of manual identification.

[0071] Step S7 in Example 1 is executed to obtain the historical frequency of inflow events, and the following historical frequencies are obtained: dry season (November to March of the following year): on average, a significant rainwater inflow event occurs approximately once every 30 days; rainy season (June to September): on average, it occurs approximately once every 10 days.

[0072] Based on the state recognition model output recognition results, the frequency of candidate rainfall-inflow events is obtained: dry season: 1 every 5 days on average; rainy season: 1 every 2.3 days on average.

[0073] The frequencies of the candidate rainfall-inflow events deviate significantly from the historical frequencies obtained based on manual experience. The difference between the two is too large, indicating that there are too many misjudgments. It is preliminarily speculated that the STL smoothing period is too small, resulting in too many short-term fluctuations being identified as high liquid level states. It is necessary to adjust the seasonal period parameter period of the STL time series decomposition method, return to step S4, and loop through steps S4-S7 until the difference between the two is less than the preset value of 5%.

[0074] In this embodiment, a total of three cycles are performed, that is, the seasonal cycle parameter period is adjusted three times, and the conditions of the three adjustments of the seasonal cycle parameter period are as follows: First round of adjustment: Increase the period to 48 hours, execute steps S4-S7, and the state recognition model outputs the recognition results to obtain the candidate rainfall-inflow event frequency of 1 time / 10 days in the dry season and 1 time / 5 days in the rainy season; Second round of adjustment: Increase the period to 72 hours, execute steps S4-S7, and the state recognition model outputs the recognition results to obtain the candidate rainfall-inflow event frequency of 1 / 25 days in the dry season and 1 / 11 days in the rainy season; The third round of fine-tuning: Increase the period to 96 hours, execute steps S4-S7, and the state recognition model outputs the recognition results, and the frequency of candidate rainfall-inflow events is 1 time / 29 days in the dry season and 1 time / 9.5 days in the rainy season.

[0075] The output recognition result of the state recognition model in the third cycle differs from the historical frequency by less than 5%. It is considered that the parameters of the state recognition model have converged and the semi-supervised calibration is completed.

[0076] The final state recognition model was permanently deployed in the monitoring station's real-time data stream monitoring system with a 96-hour period, running the recognition algorithm on a rolling 5-minute basis. Field tests over the subsequent six months demonstrated that the state recognition model achieved an average recognition accuracy of 91.4%, significantly reducing false alarms and significantly lowering manual review costs.

[0077] S8: Output the candidate rainfall-inflow events finally identified in a structured record format in real time. Specifically, it includes: outputting the identified rainfall inflow event period in JSON, CSV or GeoJSON format, including the site number, event start and end time, high liquid level duration and corresponding rainfall information, for external information system to call. The output results of this embodiment are as follows: Figure 4 shown.

Claims

1. A method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training, characterized by: The following steps are included: S1: Obtain the real-time liquid level data and historical liquid level data of the target monitoring site, as well as the real-time rainfall data of the area corresponding to the target monitoring site; S2: Preprocess the real-time liquid level data and historical liquid level data by removing outliers and filling missing values ​​respectively; S3: resampling the real-time rainfall data, pre-processed real-time liquid level data and historical liquid level data to obtain rainfall series, real-time liquid level series and historical liquid level series with a unified time scale; S4: Use the STL time series decomposition method to decompose the real-time liquid level sequence and the historical liquid level sequence respectively, and obtain the trend item T of the real-time liquid level sequence respectively. t实 and the trend term T of the historical liquid level series t历 ; S5: Construct a hidden Markov model and use the trend term T t历 The time series is used as the observation sequence, and the Baum-Welch expectation maximization algorithm is used to perform unsupervised training on the hidden Markov model to obtain a state recognition model for predicting the liquid level hidden state sequence; S6: The trend item T of the real-time liquid level data t实 The state recognition model is input, and the state recognition model outputs a liquid level hidden state sequence. The "high liquid level" state segments in the liquid level hidden state sequence are screened, and the "high liquid level" state segments that meet the preset conditions are identified as candidate rainfall-inflow events. S7: Setting a time interval threshold for state recognition model calibration. If the time interval threshold is not reached, directly proceed to the next step. When the time interval threshold is reached, calculate the occurrence frequency of the candidate rainfall-inflow event within the specified period, and compare the difference with the artificial experience frequency or the artificially labeled sample frequency. If the difference is less than a preset frequency error threshold, proceed to the next step. If the difference is greater than or equal to the frequency error threshold, increment the seasonal period parameter period value in the STL time series decomposition method at equal intervals, and re-execute steps S4 to S7 until the difference is less than the preset frequency error threshold. S8: Output the finally identified candidate rainfall-inflow events in a structured record format in real time.

2. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: In step S2, the real-time liquid level data and the historical liquid level data are preprocessed as follows: an adaptive 3σ anomaly detection algorithm based on a sliding time window is used to identify outliers for the two types of liquid level data respectively. After the identified outliers are eliminated, the linear interpolation method within the adjacent time window is used to fill in the outliers and missing values.

3. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 2, characterized in that: In step S3, the adaptive 3σ anomaly detection algorithm based on a sliding time window is used to identify outliers in the liquid level data as follows: , where LB represents the lower limit of the normal value of the liquid level data in the past week, UB represents the upper limit of the normal value of the liquid level data in the past week, μ represents the mean of the liquid level data of the target monitoring station in the past week, and σ represents the standard deviation of the liquid level data of the target monitoring station in the past week; When the monitoring value of the liquid level data is lower than LB or higher than UB, it is judged as an abnormal value; The linear interpolation method in the adjacent time window is used to fill the outliers and missing values ​​as follows: , where S t Indicates the missing value or abnormal value of the liquid level data at time t, D t Indicates S t Timestamp; S t-1 Indicates the distance S in the liquid level data t The normal value of the most recent moment, D t-1 Indicates S t-1 Timestamp; S t+1 Indicates the distance S in the liquid level data t The normal value at the most recent moment, D t+1 Indicates S t+1 timestamp.

4. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: The resampling method in step S3 is the sliding window averaging method, and the sampling method is as follows: determine the sampling frequency, the initial period of the sampling time series, and the terminal period of the sampling time series, and calculate the regularized time point sequence based on this; for each target sampling point, take the data within the forward T minutes as the sliding window, and calculate the average liquid level in the window, and use the average as the resampling value of the target sampling point.

5. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: In step S4, the STL time series decomposition method decomposes the two liquid level sequences into the following formulas based on the local weighted regression algorithm: , where y t Represents liquid level sequence data, S t represents the seasonal term, T t represents the trend term, R t represents the residual term.

6. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: The steps to train a hidden Markov model are as follows: S51: The double-hidden-state Gaussian HMM in the hidden Markov model is selected for training, and the number of hidden states of the HMM is set to 2. The hidden state S1 represents the "normal fluctuation" state of the liquid level under normal operation, and the hidden state S2 represents the "high liquid level" state under abnormal operation. The resulting hidden state set is S={S1,S2}; Use the trend item T of historical level data t历 The liquid level monitoring value represents the observation state O, and the observation state set is ; S52: Use K-means clustering method to analyze the trend item T of historical liquid level data t历 Perform clustering and divide it into two data clusters; Based on the clustering results, the initial observation probability distribution parameter set {μ1, σ1, μ2, σ2} is constructed, where μ1 is the observed mean of the hidden state S1, σ1 is the observed standard deviation σ1 of the hidden state S1, μ2 is the observed mean of the hidden state S2, and σ2 is the observed standard deviation σ2 of the hidden state S2; Based on the clustering results, the initial state probability vector is obtained and the trend items T contained in the two data clusters are calculated respectively. t历 The number of trend items in the total T t历 The proportion π in the initial state probability vector π=[p1,p2] is obtained, where p1 refers to a trend item T t历 The probability of belonging to S1, p2 refers to the trend term T t历 The probability of belonging to S2, and p1+p2=1; S53: Construct the initial state transition probability matrix A, , and set A to a high self-transition probability structure; where a ij It represents the probability that the hidden state at time t+1 changes to j when the hidden state at time t is i; S54: Calculate the observation state probability matrix B, , the observation probability density function under each hidden state obeys the one-dimensional normal distribution , Indicates that in the hidden state S i At time t, the liquid level value o is observed N The observation probability density of μ i is the hidden state S i The mean parameter, σ i is the hidden state S i The standard deviation parameter of S55: Use the Baum-Welch expectation maximization algorithm to perform unsupervised iterative training on HMM, by maximizing the trend term T t历 The log-likelihood function of the corresponding observation sequence is used to optimize the HMM model parameters; S56: Setting the stopping condition of the unsupervised iterative training: the log-likelihood increment is lower than the preset convergence threshold or the number of iterations reaches the preset upper limit; S57: After the unsupervised iterative training stops, the trained HMM obtained is the intermediate model, and the intermediate model outputs the corresponding trend term T t历 The hidden state time series of the observation sequence is then compared with the difference in the observation means of the two hidden states. If the difference is greater than or equal to the preset threshold, the intermediate model converges and is used as the state recognition model; if the difference is less than the preset threshold, the intermediate model is optimized and trained again until the difference is greater than or equal to the preset threshold.

7. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 6, characterized in that: The preset threshold value in step S57 is set as follows: Assume that the trend term T of the historical liquid level sequence is t历 The sequence of liquid level monitoring values ​​is {x1,x2,x3,…,x Q }; Calculate {x1,x2,x3,…,x Q The mean of ; Based on the mean Calculate {x1,x2,x3,…,x Q The global standard deviation σ of global , and the global standard deviation σ global As the preset threshold, , where q∈[1,Q].

8. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 6, characterized in that: The process of optimizing and training the intermediate model again in step S57 is as follows: S571: If the difference between the observed means of the two hidden states is less than the preset threshold, adjust the elements in the initial state transition probability matrix A; the automatic adjustment method is: increase The conversion probability is reduced The conversion probability is maintained The conversion probability of S572: Obtain historical liquid level data without heavy rainfall interference, and use the STL time series decomposition method to obtain the trend item of the historical liquid level data. Then, use the probability of the trend item belonging to the hidden state S1 and the hidden state S2 to reset the initial state probability vector π: If the proportion of continuous high liquid level segments in the training set exceeds 20%, increase p2 to the range of 0.1-0.2; S573: After adjusting A and π, return to step S55 and perform unsupervised iterative training on the HMM again until the difference between the observed means of the two hidden states in step S57 is greater than or equal to a preset threshold; If the condition that the difference between the observed means of the two hidden states is greater than or equal to the preset threshold is not met after three cycles, the intermediate model obtained in the last cycle is used as the state recognition model.

9. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: The preset conditions in step S6 are: ① The duration of the hidden state time series of the "high liquid level" state is not less than T1 hours and the cumulative rainfall of a single rainfall event is not less than R mm; ② The interval between the hidden state time series of the "high liquid level" state and the previous rainfall event is not less than T2 hours. If both conditions ① and ② are met, the event is identified as a candidate rainfall-inflow event.

10. The method for identifying rainfall-inflow events in sewage pipe networks using semi-supervised iterative training according to claim 1, characterized in that: In step S7, the seasonal period parameter period value of the STL time series decomposition method is adjusted as follows: in the initial state, the seasonal period parameter period is set to period0, and in the nth iteration, the seasonal period parameter period is period0+nΔ period , where Δ period is the increment step of the period parameter.

Images