Water level delay anomaly detection method for time series of water conservancy big data

Abstract

The water level delay anomaly detection method for water conservancy big data in this application first pre-processes the water level data. By standardizing the water level data, the water level data is made uniform in time density and uniform in numerical measurement standards. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their changing patterns consistent. Then, the time series with the highest similarity is found in the matched water level data by the longest common substring method. The water level delay is obtained by averaging the difference in the time attributes of the upstream and downstream water level data in the common substring. The relationship between the upstream water level value and the previous downstream water level value and the current downstream water level value is used to establish a water level relationship model through support vector regression. The water level prediction value is obtained using the water level relationship model. Then, the residual between the actual value and the predicted value is calculated. The water level anomaly is obtained by analyzing the distribution model of the residual. The accuracy and precision of water level delay anomaly detection are greatly improved.

Description

Technical Field

[0001] The present application relates to a method for detecting time-series water level delay anomalies, and in particular to a method for detecting time-series water level delay anomalies using water conservancy big data, belonging to the technical field of water conservancy big data detection. Background Art

[0002] Water conservancy projects benefit both the nation and its people by regulating and controlling natural surface and groundwater flows to achieve flood control and drought relief. With the rapid development of computer and communication technologies and the advent of the Internet+ era, the informatization of water conservancy is becoming increasingly urgent. Hydrological monitoring stations utilize wireless communication modules to transmit real-time monitoring data, providing timely insights into the hydrological characteristics of different water bodies and significantly improving the efficiency of hydrological and water conservancy departments. Furthermore, water conservancy departments utilize collected hydrological data to make water resource allocation arrangements and prevent flood disasters. The digitization of hydrological information provides a theoretical basis for key decisions by water conservancy departments, providing intuitive data on water conditions and ensuring scientific, forward-looking, and accurate management of water conservancy projects.

[0003] While the rapid development of wireless sensing technology has enabled the automated collection and transmission of hydrological data, the fact that hydrological monitoring equipment is located outdoors and the collected data must be transmitted over the network can lead to inaccurate data. Factors such as network instability, signal interference, and aging and damage to monitoring equipment can cause deviations from actual values ​​during the collection and transmission process. Fluctuations in water conditions at monitoring stations are caused by natural factors such as rainfall, equipment failures, and network instability. The accuracy of hydrological data provides data support for water conservancy departments in developing effective water resource scheduling plans. Effective flood and drought prevention and control decisions are crucial for the livelihoods, safety, and personal safety of surrounding residents.

[0004] Outlier detection can identify observed values ​​that deviate significantly or significantly from the actual values, allowing for preliminary corrections. It can also prompt staff to monitor water level values ​​during specific periods in real time and manually correct them, ensuring the accuracy of water level data and preventing flooding during these periods. Inaccuracies in water level data collection and transmission, such as those caused by network instability or aging equipment, can jeopardize water conservancy workers' assessment of reservoir water conditions and impact subsequent analysis and research of the site's water conditions. Outlier detection can alert staff to identify the source of data errors, allowing them to repair or replace monitoring equipment or check for network faults. Outlier detection can also provide preliminary corrections for abnormal data, enabling water conservancy workers to accurately assess water conditions and providing valuable guidance for future water conservancy work.

[0005] The problems that need to be solved by the existing water level abnormal point detection method and the key technical difficulties of this application include:

[0006] (1) Existing data anomaly detection schemes generally study anomalies from two aspects. One aspect is to establish a relationship model between current data and historical data, and detect anomaly data through probability calculation, distance, density or clustering methods. These methods only detect anomalies from a numerical perspective. The change of water level data is a gradual process, and the water level data at adjacent times at the same station are closely related. The other part of the method ignores the time attribute of water level data and the correlation of adjacent data, and only establishes a simple relationship between data, which is not suitable for complex and changeable water level data. In addition, due to the inconsistent rate of change of upstream and downstream water levels and the large difference in slope, it is not suitable for direct similarity measurement. Therefore, the existing technology methods are not suitable for water level delay anomaly detection.

[0007] (2) Due to the lack of data and redundancy in water level data during certain periods, and the different collection frequencies and times of water level data at different stations, the water level data collected by monitoring equipment cannot be directly used for data analysis and processing. Existing technologies lack standardization of water level data, lack the removal of redundant data, and are unable to complete missing data. The data format is not uniform and cannot meet the requirements of data mining. Due to the distance between upstream and downstream stations and the particularity of water resources, there is a certain delay in the water level change trend between upstream and downstream. Existing technologies cannot find the section with the most similar water level change trend between upstream and downstream based on the similarity of water level changes, and cannot calculate the upstream and downstream water level delay. Before similarity measurement, there is a lack of standardization of upstream and downstream water levels, making it impossible to measure similarity. After finding the upstream and downstream water level delay, it is impossible to organize the upstream water level data according to the collection time of the downstream water level, and it is impossible to model the water level relationship. The different time density of water level data makes it difficult to perform similarity measurement and subsequent data analysis and processing. Data mining is costly and inefficient, and the practicality and effectiveness of the mined information are low, which is not conducive to the detection of water level delay anomalies.

[0008] (3) The existing technology cannot solve the problem of abnormal collection and transmission of water level data. It lacks analysis of the influencing factors related to the real-time water level of the downstream. It cannot use support vector regression to establish the relationship between water levels. It lacks explanation of the characteristics of the kernel function and important parameters in the support vector regression. It lacks analysis of the impact of the parameters in the kernel function on the water level relationship model. It lacks the use of support vector regression to establish a relationship model between the real-time water level of the site and the water level of the upstream site at the corresponding moment and the water level of the site at the previous moment. It lacks the setting of important parameters in the support vector regression based on the actual characteristics of the water level data set. It cannot use the support vector regression method to establish a water level relationship model. It cannot predict the water level value based on the established water level relationship model. It cannot calculate the residual between the predicted water level value and the actual value. It lacks the method of detecting abnormal residuals through probability distribution based on the residual conforming to the normal distribution characteristics. It cannot derive abnormal water levels based on the time attributes of the residual sequence. Summary of the Invention

[0009] This application is based on the characteristics of hydrological data and is developed and improved from two processes: water level data preprocessing and downstream water level anomaly point detection. First, the water level data is preprocessed, and the water level data is standardized to make the density uniform in time and the numerical measurement standard uniform. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their change patterns consistent. Then, the time series with the highest similarity is found in the matched water level data by the longest common substring method. The time attribute of the upstream and downstream water level data in the common substring is used to calculate the difference and average the water level delay. The upstream and downstream water level data are aligned by time delay, and the aligned upstream and downstream data are used to detect anomalies. The relationship between the upstream water level value and the previous downstream water level value and the current downstream water level value is used to establish a water level relationship model through support vector regression. The water level relationship model is used to obtain the water level prediction value. Then, the residual between the actual value and the predicted value is calculated. The water level anomaly point is obtained by analyzing the distribution model of the residual. The accuracy and precision of water level time delay anomaly point detection are greatly improved.

[0010] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] The water level time delay anomaly detection method for water conservancy big data is improved from two processes: water level data preprocessing and downstream water level anomaly detection based on hydrological data characteristics:

[0012] A- Establish a water level data preprocessing method including upstream and downstream water level delays:

[0013] A1 constructs a preprocessing method for irregular data. Based on the uneven temporal density and inconsistent variation range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals to improve data availability while reducing data redundancy. Normalization is used to convert the upstream and downstream water level data from absolute values ​​to relative values.

[0014] Based on the similarity of upstream and downstream water level change trends and different change rates, A2 uses a dynamic time warping algorithm to compress and stretch the water level data on the time axis according to the similarity of upstream and downstream water level change trends, so that the upstream and downstream water level data correspond one-to-one under the same change trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring. The upstream and downstream water level delay is calculated based on the time attribute of the digital data of the longest common substring.

[0015] A3 uses the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and uses the upstream water level data to calculate the water level data value corresponding to that time, and obtains the upstream and downstream water level data at the same time;

[0016] B- Establishing a water level anomaly detection method based on support vector regression: Based on the correlation between the water level value at a certain moment of the measuring station and the water level value at the previous moment, and the similar change trend of the water level upstream and downstream of the station, these two water level factors are used to assist in the detection of downstream water level anomalies. The water level anomaly detection method is established on the basis of solving the upstream and downstream water level delay:

[0017] B1 analyzes the characteristics of water level data, selects upstream water level data related to the station water level data and the water level data at the previous moment as auxiliary factors for real-time water level anomaly detection, and analyzes the kernel function and its parameters in support vector regression;

[0018] B2 uses support vector machine regression to establish a water level relationship model between the upstream water level and the site water level at the previous moment and the site real-time water level, and verifies the evaluation indicators of the established water level model effect;

[0019] B3 predicts the water level value based on the established water level relationship model. By analyzing the residuals between the predicted water level value and the measured value, the normal range is obtained based on the distribution characteristics of the residuals, and then the abnormal residual values ​​are found. Then, the water level anomaly points are obtained based on the time attributes of the residual sequence.

[0020] Preferably, water level data is preprocessed: based on the similarity of water level changes, the section with the most similar upstream and downstream water level change trends is found, and then the upstream and downstream water level time delay is calculated. Before similarity measurement, the upstream and downstream water levels are normalized, and then similarity measurement can be performed. After finding the upstream and downstream water level time delay, the upstream water level data is sorted according to the downstream water level collection time, so as to facilitate the subsequent modeling of the water level relationship;

[0021] First, the water level data is standardized, and the standardized upstream and downstream water level data are matched using the dynamic time difference bending algorithm. The upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data are found through the longest common substring method in the similarity measurement. The upper and lower water level delays are calculated according to the time attributes of the water level data. According to the delay, the corresponding upstream water level data is found according to the collection time of the downstream water level data for water level relationship modeling.

[0022] Preferably, the water level data is evenly spaced: the upstream and downstream water level data have the same time interval, the upstream and downstream water level data are interpolated and supplemented, and the upstream and downstream water level data after the even interval are measured for similarity at the same time to obtain the time delay of the upstream and downstream water levels;

[0023] Select appropriate water level data time interval: The water level time series data interval selected in this application is △t. The water level data does not change significantly within △t, which reduces the data redundancy caused by fine-grained time intervals.

[0024] Assume that the upstream water level dataset is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i ,t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n}, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i ),u i-1 ≤u i, the upstream and downstream water levels are normalized into water level time series data starting at 0:00 every day with an interval of △t. The water level data will not change suddenly within △t. When the time point in the time series is not an integer multiple of the time interval △t, the two collection points closest to the required time point are selected, and the water level change pattern during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows:

[0025] Select the start time startTime and end time endTime of the upstream and downstream data collection, and generate a time set with an interval of △t as R={r1 ， r2, L, r n-1 , r n}, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n The calculation of upstream water level data at time r is an integer multiple of the time interval △t:

[0026] First, search for a value greater than or equal to r in the upstream initial data. i The minimum time t i , if r i =t i The upstream water level is r i Water level value v at the moment i , which is equal to the initial upstream water level data v i , if r i ≠ti, that is, r i >t i , then there exists r i ∈[t i-1 , t i ], in a relatively short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then the water level value v′ corresponding to time ri is i Calculate as follows:

[0027]

[0028] The time series data of the upstream water level data after normalization is expressed as:

[0029] D′ n = <d′1,d′2,d′3,…,d′ n-1 ,d′ n >where d′ i =(v′ i , r i )

[0030] The downstream water level data is processed in the same way. The processed water level data is:

[0031] G′ n = <g′1,g′2,g′3,…,g′ n-1 ,g′ n >where g′ i =(h′ i ,r i )

[0032] where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data after the above processing are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one. The water level data with the same time arrangement are more reasonably compared for similarity.

[0033] Preferably, the water level data is normalized:

[0034] After the water level data is processed with uniform intervals, the upstream and downstream water level data are arranged correspondingly with the same time interval △t, and the absolute value of the data is converted into an overall relative value. The normalized data ignores the specific value of the data, which is convenient for depicting the change trend of the water level data. The linear normalization method is used to normalize the water level data within the range of (0,1);

[0035] The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the interval (O, 1). First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >, where d′ i =(v′ i , r i ), the water level data is normalized as follows:

[0036]

[0037] After normalization, the upstream and downstream water level time series data are all in the interval (0,1), with the same water level measurement standard, and at the same time prepare basic data for the similarity comparison of upstream and downstream water levels.

[0038] Preferably, the upstream and downstream water level peaks are determined: the upstream and downstream water level peaks are used to assist in calculating the delay, that is, how long does it take for the peak of the upstream water level to be transmitted to the downstream? This transmission time is a rough value of the upstream and downstream water level delay. Then, similarity measurement is performed on the water level data adjacent to the peak to accurately solve the upstream and downstream water level delay;

[0039] Before calculating the upstream and downstream water level delay, the peak values ​​of the upstream and downstream water levels are first determined. First, a moment is randomly selected in the downstream water level time series to be detected, and the moment with the largest peak value is searched within the time range of one day after the moment. If there are multiple peak values, the peak value with the largest water level value is selected. If there is no peak value within the time period, the trend of the water level change within the time period is judged. If it is a downward trend, the water level peak value is searched before the time period. If it is an upward trend, the downstream water level peak value is searched after the time period. After finding the downstream water level peak value, the peak value of the upstream water level is searched within three days before the downstream peak value. If there are multiple peak values, the moment with the larger peak value is selected to correspond to the peak value of the upstream water level. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, the similarity measurement of the water level data adjacent to the peak value is performed to obtain the upstream and downstream water level delay.

[0040] Preferably, the upstream and downstream water level matching based on dynamic time difference bending is as follows: using the calculated upstream and downstream peak values, selecting the upstream and downstream water level data near the peak values, and matching the upstream and downstream water level data using the dynamic time difference bending method, assuming that the upstream water level time series is UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >, the downstream water level time series is DW n = <dw1,dw2,dw3,…,dw m-1 , dw m >, construct a distance matrix, the element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment, and the difference between the upstream and downstream water levels is the distance;

[0041] The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m*n Select a line from the upper left corner to the lower right corner. This line is the matching data pair of upstream and downstream water levels. The points along the line are the corresponding matching points of the upstream and downstream water level data.

[0042] When matching upstream and downstream water level data, the selected matching path must meet the following requirements:

[0043] Boundary conditions: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start to the lower right corner a mn End, you cannot arbitrarily start matching from the middle data;

[0044] Continuity: If the upstream water point uwi and the downstream water point dw j If the match is successful, the next upstream matching point uw i+1 Only with downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points;

[0045] Monotonicity: The points on the path must be aligned from left to right, and data cross-matching is not allowed;

[0046] The path selection for each grid point can only be performed in three directions. If the grid point coordinates are (i, j), the coordinates of the next grid point can only be (i+1, j), (i, j+1), or (i+1, j+l). The dynamic time difference warping algorithm always selects the path with the smallest cumulative sum in the three directions when selecting the matching path. The matching distance matrix of the upstream and downstream water levels is defined as:

[0047]

[0048] Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the minimum neighboring element at that point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows:

[0049]

[0050] where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distances reach the minimum. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md is used. mn The element starts at the upper left corner of its adjacent md m-1n-1 、The above element md m-1n 、Left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when

[0051] The upstream water level sequence generated by the dynamic time warping algorithm is:

[0052]

[0053] The downstream water level sequence is:

[0054]

[0055] Preferably, the water level delay is determined based on the longest common substring: the longest common subsequence algorithm is used in the matched upstream and downstream water level data to find the upstream and downstream water level data substrings with the highest similarity, and the time attributes corresponding to the upstream and downstream water level data of the longest common substring are compared. The mean of the difference between the upstream and downstream water level sequence times is the time delay of the upstream and downstream water levels;

[0056] Assume that the longest common substrings are:

[0057] Upstream water level subsequence: S′ k = <s′1,s′2,s′3…,s′ k >where s′ i =(se i ,sr i );

[0058] Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…f′ k >where f i ′=(fo i ,fr i )

[0059] The longest common substring algorithm is improved according to the characteristics of the water level dataset. The improved algorithm meets the following requirements:

[0060] 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 The formula is established;

[0061] 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant;

[0062] If the water level sequence meets the above requirements, the length of the longest common substring of the upstream and downstream water level sequences is k;

[0063] The longest common substring is solved by dynamic programming. First, the common substring matrix DM of the upstream and downstream water level sequences is established. nFirst, calculate the first row and first column of the common substring matrix. The value of the first row element is whether the upstream water level data is approximately equal to the first downstream water level data. If equal, it is 1 otherwise 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If equal, it is 1 otherwise 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If equal, the value of the node is the value of the upper left element plus 1, otherwise it is 0. The calculation formula of the common substring matrix element dm(i,j) is as follows:

[0064]

[0065] As long as the absolute value of the difference between the upstream and downstream water levels is not higher than c, the two are considered equal in theory, and the data corresponding to the longest common substring is the water level data with the closest water level change trend.

[0066] Preferably, upstream water level data selection: using upstream and downstream water level data to detect abnormal points, when detecting abnormal points in downstream water level data, it is necessary to calculate the time corresponding to the upstream water level according to the upstream and downstream water level delay according to the downstream water level acquisition time, and then calculate the upstream water level value at that time;

[0067] The initial upstream and downstream water level time series are:

[0068] Upstream water level time series: D n =<d1=(v1,t1), d2=(v2,t2),…,d n-1 =(v n-1 , t n-1 ), d n =(v n , t n )>

[0069] Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ), g n =(h n ,u n )>

[0070] The method of calculating the upstream water level according to the downstream water level is to set the upstream time set as T = {t1, t2, L, t n-1 , t n}, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i-deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 ,t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT is i Calculate as follows:

[0071]

[0072] Upstream water level time series considering the upstream and downstream water level delay problem:

[0073] UD n = <ud1,ud2,ud3…,ud n-1 ,ud n > where ud i =(z i ,nu i )

[0074] At this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend. The upstream data and the downstream data at the same time are closely related.

[0075] Preferably, a regression model and prediction of water level-related factors are established: Support vector regression establishes a linear regression model so that as many data set samples as possible fall on the regression model, i.e., the residual is as small as possible. When calculating the loss in support vector regression, data deviations within a certain range are considered acceptable and are not counted in the loss. The loss distance constant ε>0 is defined, and support vector regression introduces an insensitive loss function to implement a strategy of not calculating losses within a certain error range.

[0076] By introducing two relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the introduction of the relaxation factor, the distance between the actual value and the regression model is converted to be less than or equal to the loss distance constant ε after adding the relaxation variable;

[0077] When establishing a water level relationship model, the standard deviation, mean square error, and r-square of the actual water level values ​​and the predicted water level values ​​in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the more the water level relationship model conforms to the changing trend of the actual water level, indicating that the model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0,1] interval. When r_square is approximately close to 1, it means that the function fits better, that is, the water level relationship model is suitable for the water level data set. If the evaluation index obtained by the water level relationship model is poor, it is necessary to readjust the parameters in the support vector regression until the evaluation index meets the actual requirements.

[0078] Preferably, residual-based outlier detection:

[0079] The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level detection data are obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation is based on the selection of unknown parameters so that the probability of the sample occurring is maximized when it is used as the estimated value. Assume that the residual sequence is: Where: τi is the absolute value of the difference between the actual value and the predicted value;

[0080] The abnormal points of water level data are detected by using the characteristics of residual conforming to normal distribution, and the mean and variance of the residual are obtained by maximum likelihood estimation. Conforms to the normal distribution The probability density function of the distribution is:

[0081]

[0082] Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is:

[0083]

[0084] Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative:

[0085]

[0086] According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is:

[0087]

[0088] The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that is impossible to occur under normal circumstances, that is, the residual data is considered to be an outlier.

[0089] Compared with the existing technology, the innovation and advantages of this application are:

[0090] (1) This application is based on the characteristics of hydrological data and is developed and improved from two processes: water level data preprocessing and downstream water level anomaly point detection. First, the water level data is preprocessed, and the water level data is standardized to make the time density uniform and the numerical measurement standard uniform. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their change patterns consistent. Then, the time series with the highest similarity is found in the matched water level data by the longest common substring method. The time attribute of the upstream and downstream water level data in the common substring is used to calculate the difference and average to obtain the water level delay. Then, the upstream and downstream water level data are aligned by time delay, and the aligned upstream and downstream data are used to detect anomalies. The relationship between the upstream water level value and the previous downstream water level value and the current downstream water level value is used to establish a water level relationship model through support vector regression. The water level prediction value is obtained using the water level relationship model. Then, the residual between the actual value and the predicted value is calculated. The water level anomaly point is obtained by analyzing the distribution model of the residual. The accuracy and precision of water level delay anomaly point detection are greatly improved.

[0091] (2) When utilizing the relationship between upstream and downstream water levels, the present application noticed that there is a certain time delay in the changing trend of the upstream and downstream water levels, and established a water level data preprocessing method including upstream and downstream water level delay. Based on the characteristics of similar changing trends and different changing rates of upstream and downstream water levels, the dynamic time difference bending algorithm is used to compress and stretch the water level data on the time axis according to the similarity of the upstream and downstream water level changing trends, so that the upstream and downstream water level data correspond one by one under the same changing trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring. The upstream and downstream water level delay is calculated based on the time attribute of the longest common substring digital data, and the upstream collection time corresponding to the downstream water level collection time is found. The upstream water level data is used to calculate the water level data value corresponding to the time, and the upstream and downstream water level data at the same time are obtained. The relationship between the upstream and downstream water levels is cleverly utilized to improve the quality of water level delay anomaly detection.

[0092] (3) When detecting water level anomalies, the present application considers the relationship between the upstream and downstream water levels and the relationship between the current water level value and the water level value at the adjacent historical moments, and uses the relationship between these variables and the current water level to assist in the detection of water level anomalies. Since water level data is a type of data with low precision but large sample size, a water level anomaly detection method based on support vector regression is established to process inaccurate data, analyze the characteristics of water level data, and use support vector machine regression to establish a water level relationship model between the upstream water level and the site water level at the previous moment and the real-time water level of the site. The evaluation indicators of the established water level model are verified, and the water level value is predicted based on the established water level relationship model, which greatly improves the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is the symbols used in this application and their description diagrams.

[0094] Figure 2 This is a flow chart for determining the peak water levels upstream and downstream.

[0095] Figure 3 It is a flow chart of establishing a real-time water level relationship model using the upstream water level and the water level at the previous moment.

[0096] Figure 4 This is a schematic diagram of the downstream data before water level interval processing.

[0097] Figure 5 It is a schematic diagram of the actual data points of the downstream water level in the data preprocessing experiment.

[0098] Figure 6 This is the upstream and downstream water level data graph before normalization in the data preprocessing experiment.

[0099] Figure 7 This is a schematic diagram of the water level data segment before matching by the dynamic time difference bending algorithm.

[0100] Figure 8 This is a schematic diagram of the upstream and downstream water level data after matching with the dynamic time difference bending algorithm.

[0101] Figure 9 This is a comparison chart of the accuracy of the proposed method and the outlier solution based on the sliding window. DETAILED DESCRIPTION

[0102] Below, in conjunction with the accompanying drawings, the technical solution of the water level delay anomaly detection method for water conservancy big data time series provided by this application is further described so that technical personnel in this field can better understand this application and implement it.

[0103] Water level data is a special type of time-dependent sequence, and its temporal variations are unpredictable. Given the close relationship between the current water level, the upstream water level, and nearby historical water levels, a support vector regression algorithm is used to establish a relationship model between these two factors and the current water level to assist in detecting downstream water level anomalies. In reality, while upstream and downstream water levels exhibit similar trends, there is a certain time delay between their changes. This application addresses this time delay before detecting water level anomalies.

[0104] First, the water level data is preprocessed and standardized to make the water level data uniform in time density and unified in numerical measurement standards. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their change patterns consistent. Then, the longest common substring method is used to find the time series with the highest similarity in the matched water level data. The time attributes of the upstream and downstream water level data in the common substring are used to calculate the difference and average the water level delay. The upstream and downstream water level data are then aligned through the time delay, and the aligned upstream and downstream data are used to detect anomalies.

[0105] Using the relationship between the upstream water level value and the downstream water level value at the previous moment and the current downstream water level value, a water level relationship model is established through support vector regression. The water level prediction value is obtained using the water level relationship model, and then the residual between the actual value and the predicted value is calculated. The water level anomaly point is obtained by analyzing the distribution model of the residual.

[0106] A-Establish a water level data preprocessing method including upstream and downstream water level time delay: A1 constructs a preprocessing method for irregular data. Based on the characteristics of uneven temporal density and inconsistent variation range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals to improve data availability while reducing data redundancy. Normalization is used to convert the upstream and downstream water level data from absolute values ​​to relative values. A2 Based on the characteristics of similar upstream and downstream water level variation trends and different variation rates, the dynamic time difference warping algorithm is used to compress and stretch the water level data on the time axis according to the similarity of the upstream and downstream water level variation trends, so that the upstream and downstream water level data correspond one to one under the same variation trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring, and the upstream and downstream water level time delay is calculated based on the time attribute of the longest common substring digital data. A3 uses the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and uses the upstream water level data to calculate the water level data value corresponding to that time, thereby obtaining the upstream and downstream water level data at the same time.

[0107] B- Establishing a water level anomaly detection method based on support vector regression:

[0108] Based on the correlation between the water level value at a certain moment of the measuring station and the water level value at the previous moment, and the similar change trend between the water level upstream and downstream of the station, these two water level factors are used to assist in the detection of downstream water level anomalies. On the basis of solving the time delay of upstream and downstream water levels, a water level anomaly detection method based on support vector regression is established: B1 analyzes the characteristics of water level data, selects upstream water level data and water level data at the previous moment related to the station water level data as auxiliary factors for real-time water level anomaly detection, and analyzes the kernel function and its parameters in support vector regression; B2 uses support vector machine regression to establish a water level relationship model between the upstream water level and the station water level at the previous moment and the real-time water level of the station, and verifies the evaluation indicators of the effect of the established water level model;

[0109] B3 predicts the water level value based on the established water level relationship model. By analyzing the residuals between the predicted water level value and the measured value, the normal range is obtained based on the distribution characteristics of the residuals, and then the abnormal residual values ​​are found. Then, the water level anomaly points are obtained based on the time attributes of the residual sequence.

[0110] 1. Water level data preprocessing

[0111] Because water level data can contain missing and redundant data during certain periods, and because water level data collection frequencies and times vary across different sites, water level data standardization is necessary before detecting outliers. Data processing can remove redundant data and complete missing data according to specific rules, unifying the data format and meeting the requirements of data mining. Preprocessing water level data ensures consistent temporal density, facilitating similarity measurement and subsequent data analysis and processing. Data preprocessing makes data mining possible, reducing the cost of data mining and improving its efficiency. Furthermore, the mined information is both more practical and more effective.

[0112] The purpose of this application is to detect abnormal points in water level data. However, the water level data collected by the monitoring equipment cannot be directly used for data analysis and processing. Therefore, the water level data needs to be pre-processed before the detection of abnormal water level data can be performed. The upstream water level and the downstream water level have similar changing trends. However, due to the distance between the upstream and downstream stations and the particularity of water resources, there is a certain delay in the changing trends of the upstream and downstream water levels. This application finds the section with the closest changing trends of the upstream and downstream water levels based on the similarity of the water level changes, and then calculates the upstream and downstream water level delays. Before performing the similarity measurement, the upstream and downstream water levels are standardized, and then the similarity measurement can be performed. After finding the time delay of the upstream and downstream water levels, the upstream water level data is sorted according to the collection time of the downstream water level to facilitate the subsequent modeling of the water level relationship.

[0113] This application first standardizes the water level data, matches the standardized upstream and downstream water level data using a dynamic time difference bending algorithm, and uses the longest common substring method in the similarity measurement to find the upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data. The upper and lower water level delays are calculated based on the time attributes of the water level data, and the corresponding upstream water level data is found according to the time delay and the collection time of the downstream water level data for water level relationship modeling.

[0114] (1) Water level preprocessing scenarios and objectives

[0115] 1. Scenario Description

[0116] Because upstream and downstream water level data are collected at different times and intervals, and the collected data is redundant and missing, water level data needs to be standardized. Although the upstream and downstream water levels show similar trends, the water needs to pass through a longer section of river to flow from the upstream station to the downstream station. Therefore, when the upstream water level rises, the downstream water level does not rise immediately, but lags behind. Although the upstream and downstream water levels show similar patterns of change, the downstream water level changes later. When using the upstream water level to assist in downstream water level anomaly detection, the time delay between the upstream and downstream water levels must be addressed first.

[0117] 2. Preprocessing objectives

[0118] In response to the above problems, this application sets the following goals: (1) Water level data standardization: Evenly space and normalize the water level data to make it more consistent with the changing characteristics of the water level, and also make the upstream and downstream water level data within the same measurement range. (2) According to the changing trend of the upstream and downstream water levels, the upstream and downstream water level data with the highest similarity are obtained by comparing the similarity of the upstream and downstream water levels, and the upstream and downstream water level delays are obtained by using the time attributes of the water level data. (3) According to the time attributes of the water level value of the station to be measured, the water level value at the corresponding time upstream is selected using the collection time of the downstream station. The symbols used in this application and their descriptions are shown in Figure 1 .

[0119] (2) Water level data standardization

[0120] 1. Water level data is evenly spaced

[0121] Because upstream and downstream water level data are collected by two different monitoring devices with inconsistent sampling frequencies and times, the time series water level data has an uneven temporal density distribution, making it difficult to compare the similarity between upstream and downstream water levels. Therefore, it is necessary to ensure that the upstream and downstream water level data have the same time interval. Since the upstream and downstream water levels are monitored at different time intervals, the uniform interval process requires interpolation and supplementation of the upstream and downstream water level data according to certain rules. After uniform intervals, the upstream and downstream water level data can be measured for similarity at the same time, thereby determining the time delay of the upstream and downstream water levels.

[0122] Select appropriate water level data time interval: The water level time series data interval selected in this application is △t, and the water level data does not change significantly within △t, reducing the data redundancy caused by fine-grained time intervals; suppose the upstream water level dataset is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i ,t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n}, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i ),u i-1 ≤u i , the upstream and downstream water levels are normalized into water level time series data starting at 0:00 every day with an interval of △t. The water level data will not change suddenly within △t. When the time point in the time series is not an integer multiple of the time interval △t, the two collection points closest to the required time point are selected, and the water level change law during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows: select the start time startTime and end time endTime of the upstream and downstream collection data, and generate a time set with an interval of △t as R={r1 ， r2, L, r n-1 , r n}, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n The calculation of the upstream water level data at time r is an integer multiple of the time interval △t: First, find the value greater than or equal to r in the upstream initial data. i The minimum time ti , if r i =t i The upstream water level is r i Water level value v at the moment i , which is equal to the initial upstream water level data v i , if r i ≠t i , that is, r i >t i , then there exists r i ∈[t i-1 , t i ], in a relatively short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then r i The water level value v′ corresponding to the moment i Calculate as follows:

[0123]

[0124] The time series data of the upstream water level data after normalization is expressed as:

[0125] D′ n = <d′1,d′2,d′3,…,d′ n-1 ,d′ n >where d' i =(v′ i ,r i )

[0126] The downstream water level data is processed in the same way. The processed water level data is:

[0127] G′ n = <g′1,g′2,g′3,…,g′ n-1 , g′ n >where g′ i =(h′ i ,r i )

[0128] where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data after the above processing are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one. The water level data with the same time arrangement are more reasonably compared for similarity.

[0129] 2. Water level data normalization

[0130] After the water level data is processed with uniform spacing, the upstream and downstream water level data are arranged in the same time interval Δt. However, the upstream and downstream water level values ​​are within different ranges, which is not conducive to measuring the similarity of water level values ​​at different scales. Normalization converts the absolute value of the data into an overall relative value. The normalized data ignores the specific data values ​​and focuses on the data's changing relationship, which facilitates the description of the changing trend of the water level data. The linear normalization method is used to normalize the water level data to the range of (0,1).

[0131] The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the interval (O, 1). First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >, where d′ i =(v′ i , r i ), the water level data is normalized as follows:

[0132]

[0133] After normalization, the upstream and downstream water level time series data are all in the interval (0,1), with the same water level measurement standard, and at the same time prepare basic data for the similarity comparison of upstream and downstream water levels.

[0134] (3) Determination of water level delay based on similarity

[0135] 1. Determination of upstream and downstream water level peaks

[0136] The upstream and downstream water level peaks are used to assist in the calculation of delay. That is, how long does it take for the peak of the upstream water level to be transmitted to the downstream? This transmission time is a rough value of the upstream and downstream water level delay. Then, the similarity measurement of the water level data adjacent to the peak is performed to accurately solve the upstream and downstream water level delay.

[0137] Before calculating the upstream and downstream water level delay, the peak values ​​of the upstream and downstream water levels are first determined. First, a moment is randomly selected in the downstream water level time series to be detected, and the moment with the largest peak value is searched within the time range of one day after the moment. If there are multiple peak values, the peak value with the largest water level value is selected. If there is no peak value within the time period, the trend of the water level change within the time period is determined. If it is a downward trend, the water level peak value is searched before the time period. If it is an upward trend, the downstream water level peak value is searched after the time period. After finding the downstream water level peak value, the peak value of the upstream water level is searched within the three days before the downstream peak value (i.e., 1.5 times the empirical value of the water level delay). If there are multiple peak values, the peak value of the upstream water level corresponding to the moment with the larger peak value is selected. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, the water level data adjacent to the peak values ​​are similarly measured to obtain the upstream and downstream water level delays. The flowchart for determining the peak values ​​of the upstream and downstream water levels is shown in the figure below. Figure 2 .

[0138] 2. Upstream and downstream water level matching based on dynamic time difference bending

[0139] Using the calculated upstream and downstream peak values, we select the upstream and downstream water level data near the peak values, and use the dynamic time difference bending method to match the upstream and downstream water level data. Let the upstream water level time series be UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >, the downstream water level time series is DW n = <dw1,dw2,dw3,…,dw m-1 , dw m >, construct a distance matrix. The element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment. The difference between the upstream and downstream water levels is its distance. The upstream and downstream water level distance matrix is:

[0140] The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m*n Select a line connecting the upper left corner to the lower right corner. This line is the matching data pair of upstream and downstream water levels. The points along the line are the corresponding matching points of the upstream and downstream water level data.

[0141] When matching upstream and downstream water level data, the selected matching path must meet the following requirements:

[0142] Boundary conditions: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start to the lower right corner a mn End, you cannot arbitrarily start matching from the middle data;

[0143] Continuity: If the upstream water point uwi and the downstream water point dw jIf the match is successful, the next upstream matching point uw i+1 Only with downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points;

[0144] Monotonicity: The points on the path must be aligned from left to right, and cross-matching of data is not allowed, otherwise the order will be reversed;

[0145] Due to the monotonicity and continuity of the upstream and downstream water level matching paths, the path selection for each grid point can only be performed in three directions. If the grid point coordinates are (i, j), then the coordinates of the next grid point can only be (i+1, j), (i, j+1), or (i+1, j+l). When selecting the matching path, the dynamic time difference warping algorithm always chooses the path with the smallest cumulative sum in the three directions. The matching distance matrix of the upstream and downstream water levels is defined as:

[0146]

[0147] Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the minimum neighboring element at that point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows:

[0148]

[0149] where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distances reach the minimum. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md is used. mn The element starts at the upper left corner of its adjacent md m-1n-1 、The above element md m-1n 、Left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when

[0150] The upstream water level sequence generated by the dynamic time warping algorithm is:

[0151]

[0152] The downstream water level sequence is:

[0153]

[0154] 3. Water level delay determination based on the longest common substring

[0155] Due to the large difference in the change rate of upstream and downstream data in certain time periods, there are still large differences between the matched data. This application uses the longest common subsequence algorithm in the matched upstream and downstream water level data to find the upstream and downstream water level data substrings with the highest similarity, and compares the time attributes corresponding to the upstream and downstream water level data of the longest common substring. The mean of the difference in the time of the upstream and downstream water level sequences is the time delay of the upstream and downstream water levels.

[0156] The present application searches for the longest common substring with the highest similarity in the matched upstream and downstream water level data, that is, searches for the upstream and downstream water level segments with the closest water level change trends. The longest common substring algorithm is improved according to the characteristics of the water level data set. When judging the common substring, it is determined that the difference between the upstream water level data and the downstream water level data is within a certain interval c, that is, the two are determined to be equal. This solves the problem of amplitude scaling of the upstream and downstream water level data when comparing similarity. The water level data near the upstream and downstream peak values ​​are selected and matched using the dynamic time difference bending algorithm. Then, the longest common substring with the closest change trend is found in the matched upstream and downstream water level data values. Then, the time delay of the upstream and downstream water levels is obtained through the time attribute of the water level data in the longest common substring.

[0157] Assume that the longest common substrings are:

[0158] Upstream water level subsequence: S′ k = <s′1,s′2,s′3…,s′ k >where s′ i =(se i ,sr i );

[0159] Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…,f′ k >where f′ i =(fo i ,fr i )

[0160] The longest common substring algorithm is improved according to the characteristics of the water level dataset. The improved algorithm meets the following requirements:

[0161] 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 The formula is established;

[0162] 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant;

[0163] If the water level sequence meets the above requirements, the length of the longest common substring of the upstream and downstream water level sequences is k;

[0164] The longest common substring is solved by dynamic programming. First, the common substring matrix DM of the upstream and downstream water level sequences is established. n First, calculate the first row and first column of the common substring matrix. The value of the first row element is whether the upstream water level data is approximately equal to the first downstream water level data. If equal, it is 1 otherwise 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If equal, it is 1 otherwise 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If equal, the value of the node is the value of the upper left element plus 1, otherwise it is 0. The calculation formula of the common substring matrix element dm(i,j) is as follows:

[0165]

[0166] As long as the absolute value of the difference between the upstream and downstream water levels is not higher than c, the two are considered equal in theory, and the data corresponding to the longest common substring is the water level data with the closest water level change trend.

[0167] The longest common subsequence in the upstream and downstream water level sequences is the segment of data with the most similar change trends in the upstream and downstream water level data, that is, the upstream and downstream water level data sequences corresponding to this period of time have similar changes. The upstream and downstream water level time delay is calculated based on the time attributes of this segment of water level time series data.

[0168] Using the longest common substring of the upstream and downstream water level data, calculate the time difference of the corresponding moments according to the time attribute of the common substring water level data, and then average the obtained time differences:

[0169]

[0170] The longest common substring of the upstream and downstream water level change trends is a time series with the same length, and the upstream and downstream substring pairs are continuous in time. The upstream and downstream water level delay deltatime is the mean of the time difference of the longest common substring.

[0171] (4) Upstream water level data selection

[0172] Use upstream and downstream water level data to detect anomalies. When detecting anomalies in downstream water level data, it is necessary to calculate the time corresponding to the upstream water level according to the upstream and downstream water level delay according to the downstream water level collection time, and then calculate the upstream water level value at that time;

[0173] The initial upstream and downstream water level time series are:

[0174] Upstream water level time series: D n =<d1=(v1,t1),d2=(v2,t2),…,d n-1 =(v n-1 ,t n-1 ),d n =(v n ,t n )>

[0175] Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ),g n =(h n ,u n )>

[0176] The method of calculating the upstream water level according to the downstream water level is to set the upstream time set as T = {t1, t2, L, t n-1 , t n}, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i -deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 ,t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT is i Calculate as follows:

[0177]

[0178] Upstream water level time series considering the upstream and downstream water level delay problem:

[0179] UD n = <ud1,ud2,ud3…,ud n-1 ,ud n > where ud i =(z i ,nu i )

[0180] At this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend. The upstream data and the downstream data at the same time are closely related.

[0181] 2. Water Level Anomaly Detection Using Support Vector Regression

[0182] In order to solve the abnormal problems of water level data collection and transmission, this application proposes a support vector regression water level anomaly point detection method that takes into account the influence of time lag. After solving the upstream and downstream water level delay problem, the support vector regression algorithm is used to establish a relationship model between the downstream real-time water level and the upstream water level and the downstream water level at the previous moment. The future water level value is predicted by the established water level model, and the residual between the actual value and the predicted value is analyzed using the probability calculation method to obtain the abnormal residual value. Then, the water level value corresponding to the time at that moment is found through the time corresponding to the abnormal residual value. This water level value is the abnormal data.

[0183] (1) Support Vector Regression Kernel Function

[0184] The upstream water level and the water level at the previous moment, which are closely related to the current moment, are used to establish a relationship model with the current moment water level. The kernel function is introduced to reduce the dimension of the inner product of the water level and the complexity of the support vector regression algorithm. The relationship model between the upstream water level, the water level at the previous moment and the current moment water level is established. The Gaussian kernel function K(x, z) = exp(γ||xz|| 2 ) The parameter γ measures the influence of a single water level sample on the water level relationship model. The parameter γ is set according to the characteristics of the water level dataset. The parameter γ defines the influence of a single sample on the regression function. When the parameter γ is small, the influence of a single sample on the regression function is greater, and the support regression vector fits this sample by increasing the complexity of the regression function. Conversely, when γ is large, the influence of a single data on the regression function is small, and the complexity of the regression function is also high. In the Gaussian kernel function, the value of the parameter γ is 1 / the number of sample features.

[0185] (2) Establishing a regression model and prediction of water level related factors

[0186] The relationship model between the upstream corresponding water level and the water level at the previous moment of the point is established. The historical water level data is used for learning, and then the real-time water level data is predicted. The abnormal points are found by comparing the residuals of the water level data using the probability distribution method.

[0187] Support vector regression builds a linear regression model to ensure that as many data set samples as possible fall on the regression model, i.e., the residual is as small as possible. When calculating the loss in support vector regression, data deviations within a certain range are considered acceptable and are not counted in the loss. The loss distance constant ε>0 is defined. Support vector regression introduces an insensitive loss function to achieve a strategy of not calculating losses within a certain error range.

[0188] On the basis of the insensitive function, the objective function of the support vector regression problem is transformed into the objective function structural risk. The former is a measure of the complexity of the water level regression function, and the latter is the empirical risk used to describe the prediction effect between the prediction function and the actual value. The prediction error is defined by the combined results of the two. According to the principle of structural risk minimization, the objective function expects the complexity of the water level relationship model to be low, and expects the model to be as small as possible in order to minimize misclassification. The penalty factor C is the coordination of the relationship between the two. When C is large, the abnormal points are modeled as normal points. When C is small, the abnormal points will be excluded from the normal points and the slack variables will be used to penalize them.

[0189] By introducing two relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the introduction of the relaxation factor, the distance between the actual value and the regression model is converted to be less than or equal to the loss distance constant ε after adding the relaxation variable;

[0190] When establishing the regression model, the loss is calculated for the values ​​whose errors are not within the range of the loss distance constant ε. When modeling the water level relationship, the loss needs to be as small as possible. However, the small loss comes at the cost of increasing the complexity of the regression model. At this time, the penalty factor C is used to adjust the prediction effect and complexity of the water level relationship model, and the Lagrange multiplier is introduced. The Lagrange multiplier method can convert the objective function into a Lagrangian function.

[0191] Figure 3 A flow chart of the real-time water level relationship model established by the support vector regression algorithm using the upstream water level and the water level at the previous moment.

[0192] When establishing a water level relationship model, the standard deviation, mean square error, and r-square of the actual water level values ​​and the predicted water level values ​​in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the more the water level relationship model conforms to the changing trend of the actual water level, indicating that the model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0,1] interval. When r_square is approximately close to 1, it means that the function fits better, that is, the water level relationship model is suitable for the water level data set. If the evaluation index obtained by the water level relationship model is poor, it is necessary to readjust the parameters in the support vector regression until the evaluation index meets the actual requirements.

[0193] Support vector regression utilizes an insensitive loss function to improve the regression function's tolerance for outliers, helping the regression function learn more about data trends rather than focusing on the precise fit of individual data points. Support vector regression abandons fitting data that deviates significantly from other sample points in the dataset and directly calculates the loss. The introduction of an insensitive loss function facilitates the detection of water level anomalies. Furthermore, the kernel function in support vector regression solves the complex inner product calculation problem when mapping linearly inseparable data to high dimensions, avoiding the curse of data dimensionality and simplifying computational complexity. Support vector regression models have excellent generalization capabilities, can learn complex water level relationship models, and effectively fit the changing trends of water level data.

[0194] (3) Outlier Detection Based on Residuals

[0195] The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level detection data are obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation is based on the selection of unknown parameters so that the probability of the sample occurring is maximized when it is used as the estimated value. Assume that the residual sequence is: Where: τ i is the absolute value of the difference between the actual value and the predicted value;

[0196] The abnormal points of water level data are detected by using the characteristics of residual conforming to normal distribution, and the mean and variance of the residual are obtained by maximum likelihood estimation. Conforms to the normal distribution The probability density function of the distribution is:

[0197]

[0198] Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is:

[0199]

[0200] Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative:

[0201]

[0202] According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is:

[0203]

[0204] The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that is impossible to occur under normal circumstances, that is, the residual data is considered to be an outlier.

[0205] Before solving the normal residual range, the normal range of the standard normal distribution is first calculated at the significance level, and then the normal interval of the water level residual is deduced based on the relationship between the general normal distribution and the standard normal distribution. The probability distribution density function of the standard (0,1) normal distribution is And the distribution function φ(x) is:

[0206]

[0207] Formula 13 is the probability distribution density function;

[0208]

[0209] Equation 13 is the distribution function;

[0210] According to the properties of the standard normal distribution: φ(-x)=1-φ(x);

[0211] Normal distribution is Its probability density function is:

[0212]

[0213] The data set η follows a (0,1) normal distribution, that is,

[0214]

[0215]

[0216] Under a given significance level α', find the maximum value of η;

[0217]

[0218] In the standard normal distribution, P(η>za )=a,0<a<1that is, P(n<z a )=1-a;

[0219] a and Z a The corresponding relationship is obtained by querying the formula of the normal distribution table, and the corresponding quantile z is obtained by querying the normal distribution table using α' α' The values ​​of are:

[0220]

[0221] The normal value of the residual in the water level relationship model is obtained in (0, τ max ), if the data falls outside the normal range, it is considered that the data is a low-probability event that will not occur under normal circumstances, and the water level data that produces the residual at that moment is abnormal data.

[0222] 3. Experimental plan and results analysis

[0223] The proposed solution is analyzed, experiments are conducted on real data sets and simulated data sets, and the experimental results are analyzed.

[0224] (1) Experimental plan analysis

[0225] When detecting water level anomalies, this application fully analyzed the characteristics of water level data and found that there is a close relationship between upstream and downstream water levels. However, due to the certain distance between the upstream and downstream collection sites and the particularity of water resources, although the upstream and downstream water level data have similar change trends, there is a time delay problem. The anomaly point detection scheme proposed in this application first matches the upstream and downstream water level data using a dynamic time difference bending algorithm, and then uses the longest common substring method to find the water level segments with the closest change trends, and uses the time attributes of the water level data to calculate the delay.

[0226] The support vector regression method is suitable for detecting anomalies in water level data. Since the time intervals for collecting water level data by the liquid level gauge are inconsistent and the time density of collected data is uneven, it is reasonable to allow a certain deviation between the model's predicted value and the actual value when performing data modeling under such irregular water level data. The established water level relationship model pays more attention to the changing trend of water level data, and will not cause the blurring of the changing trend due to overfitting in order to ensure the accuracy of individual data predictions, thereby affecting the detection of water level anomalies. Therefore, the support vector regression method is suitable for the detection of water level anomalies in this application.

[0227] (2) Experimental results of abnormal water level points

[0228] This application proposes a time-series water level anomaly detection solution based on support vector regression. By utilizing the relationship between the upstream water level, the downstream water level at the previous moment, and the downstream water level at the current moment, and taking into account the time delay between water level data, this application first uses the longest common substring to find the water level sequence with the closest change trend, calculates the time delay based on the time difference between the corresponding upstream and downstream water levels, and establishes a water level relationship model by learning from historical water level data using the support vector regression algorithm. The established water level relationship model is then used to predict future water level data, and water level anomalies are found by analyzing the residuals between the predicted values ​​and the actual values.

[0229] 1. Data preprocessing experiment

[0230] Since the collection frequency of upstream and downstream water level data is different, it is necessary to normalize the water level data into water level time series data with a time interval of 30 minutes. The downstream part of the data before water level interval processing is Figure 4 .

[0231] Figure 5 Each point in the figure represents a collection point. The points in the figure are arranged from left to right in the order of collection time. The X-axis represents the collection time of the original downstream water level data, and the Y-axis represents the collected downstream water level value.

[0232] The downstream water level is processed into a standardized data graph. From the comparison graph of the water level data, it can be seen that the water level data processed at half-hour intervals maintains the changing characteristics of the water level at the original collection point, and the processed water level data is smoother, which is more valuable and meaningful in the subsequent analysis and processing of water level data.

[0233] Since the distribution range of the water level values ​​in the upstream and downstream is quite different, it is necessary to normalize the upstream and downstream water level data. The upstream and downstream water level data before normalization are Figure 6 .

[0234] The water level change rate upstream is different. Similar change trends upstream are completed in a shorter time, while downstream data takes a longer time. This application uses a section of water level data with a similar change trend to match the upstream and downstream water levels using a dynamic time difference bending algorithm. The water level data segment before matching is Figure 7 .

[0235] The dynamic time difference warping algorithm first uses the distance matrix of the upstream and downstream water levels to calculate the element values ​​of the first row and first column of the similarity matrix, and then uses the dynamic programming algorithm to calculate the values ​​of the elements in the other rows and columns of the similarity matrix. The element value of a grid point is the sum of the minimum value of the element value of the distance matrix corresponding to the grid point and the similarity of the grid point on its upper left, the grid point on the left, and the grid point on the upper side. After the upstream and downstream water level data are matched using the dynamic time difference warping algorithm, the water level data are aligned according to the same change trend. The upstream and downstream water level data matching diagram is shown in the figure below. Figure 8 .

[0236] Figure 8 Each point in the figure represents a collection point. The points are arranged from left to right in chronological order, with a half-hour interval. The X-axis represents the water level time of the corresponding point after matching, and the Y-axis represents the normalized upstream and downstream water level values ​​after matching. The green line represents the downstream water level, and the red line represents the upstream water level. The presence of a horizontal upstream water level line at the point to the left of the peak in the figure is due to the rapid change rate of the upstream data, while the downstream water level changes more slowly under the same trend. This results in one point corresponding to multiple points during matching. In the improved longest common substring, if the difference between the upstream and downstream water levels is less than ζ, the two pairs are considered equal. Based on the characteristic constant ζ = 0.005 of the upstream and downstream water level data, the corresponding upstream and downstream water level values ​​are obtained using the matching sequence generated above. Based on the distance between the upstream and downstream water level data, the water level points with the smallest water level distance are considered equal. The value of the longest common substring matrix is ​​the value of the element in its upper left corner plus 1. Otherwise, the value of the point is 0. The longest common substring finds the water level sequence with the most consecutive matching data. Find the largest data in the common substring matching matrix. This data is the length of the longest common substring of the upstream and downstream water levels. The horizontal and vertical coordinate positions corresponding to this data are the end positions of the matching string. The beginning of the upstream and downstream substrings is obtained based on the length and end position of the common substring. The specific position information is as follows: the longest matching length is 43, the starting position of the downstream water level substring is 85, and the starting position of the upstream water level substring is 85. The time of the upstream and downstream water levels and the corresponding water level values ​​are found through the position of the matching substring of the upstream water level in the water level sequence.

[0237] Based on the time attributes of the longest matching water level substring corresponding to the upstream and downstream water levels, the time difference between the upstream and downstream water levels is calculated. The average of all time differences in the matching substring is then calculated, and this value is the upstream and downstream water level delay. Based on actual data, the upstream and downstream water level delay is calculated to be 1 day, 15 hours, 23 minutes, and 23 seconds. The final water level delay is approximately 1 day, 15 hours, and 30 minutes.

[0238] 2. Water level data abnormal point experiment

[0239] This application uses the upstream water level and the water level at the previous moment as input and uses the support vector regression algorithm to establish a relationship model between the water level at the current moment. When using support vector regression to establish a relationship model, it is necessary to establish the form of the kernel function and the value of the penalty factor C, the kernel function parameter γ, and the loss tolerance distance ε. Then, the established water level relationship model is used to analyze the residual between the predicted value and the actual value. By analyzing the residual, water level data in the actual water level that does not conform to the change trend is identified. This water level data is the outlier.

[0240] Based on the derived upstream and downstream water level time delays, the upstream and downstream water level data are aligned according to the water level time delay. The water level model is then learned using the first 3,000 normal water level data points, and predictions are performed on the remaining 900 noisy water level data points. During the learning process, the prediction model is evaluated using metrics such as variance and standard deviation. If the learning effect is poor, the support vector regression algorithm parameters C, γ, and ε are adjusted to align with the changing characteristics of the current water level dataset.

[0241] The outlier detection solution proposed in this application requires the establishment of a water level relationship model. Since water level data is complex and changeable, the water level dataset does not belong to a linear relationship dataset. Therefore, a nonlinear kernel function is used. In support vector regression, the nonlinear kernel functions include Sigmoid kernel function, polynomial kernel function, and Gaussian kernel function.

[0242] When we used the Sigmoid kernel function to build a water level relationship model based on the initial data for the water level dataset, we found that the predicted values ​​for the water level data differed significantly from the actual values, resulting in poor results. Therefore, the Sigmoid kernel function was not an option. The predicted values ​​for the polynomial kernel function were not on the same order of magnitude as the actual values ​​for the water level data. The water level data did not conform to the functional characteristics of the polynomial kernel function, and the modeling efficiency of the polynomial kernel function decreased at higher degrees. Therefore, the polynomial kernel function was not suitable for the water level dataset.

[0243] Using the Gaussian kernel function in the support vector regression algorithm can effectively model water level relationships. The actual water level values ​​are basically consistent with the predicted values, and the prediction error can be controlled within a small range. The water level model established by the Gaussian kernel function conforms to the characteristics of the water level dataset. Therefore, this application uses the Gaussian kernel function to model water level relationships.

[0244] After determining that the kernel function used in this application was a Gaussian kernel function, the key parameters in the support vector regression were optimized. In the support vector regression algorithm, the parameter y = 1 / sample feature number was used in the experiment. This application used two factors: the upstream water level at the moment and the downstream water level at the previous moment. Therefore, the value of γ was set to 0.5. The penalty factor C and the loss distance ε were optimized. First, the modeling effects were compared using y = 0.5 and y = 1.

[0245] Support vector regression (SVR) shows that the prediction effect of γ = 0.5 is better than that of γ = 1. Moreover, by comparing the evaluation indicators of the water level relationship model, it can be seen that γ = 0.5 has a significantly better effect.

[0246] Finally, this application is compared with the hydrological time series anomaly detection scheme based on sliding window prediction. The water level anomaly detection of the sliding window is to predict the water level value using the idea of ​​sliding window, and then analyze the residual to obtain the anomaly point. The experiment compares the anomaly detection accuracy of the two schemes, as shown in the comparison figure. Figure 9 .

[0247] The number of samples in the water level data set is 900. It can be seen that when the outlier deviation value is increased to between (0.05-0.07), the detection accuracy of both schemes increases. At the same time, the increase in the number of outliers causes the detection accuracy of both schemes to decrease, but the accuracy of the scheme proposed in this application is still higher than that of the sliding window scheme. This scheme proposes a scheme to detect downstream water level anomalies based on upstream water level data and historical water levels. It has good detection effect when the downstream water level has only one upstream confluence point and the number of anomalies is small (less than 10%). When the number of anomalies is large, the detection effect is weakened. However, in reality, the number of water level anomalies in a reservoir generally does not exceed 10%, because in this case, water conservancy staff will deal with them. Therefore, the scheme proposed in this application is of great significance in the actual detection of water level anomalies.

Claims

1. A method for detecting abnormal points of water level delay in water conservancy big data time series, characterized by: Based on the characteristics of hydrological data, improvements are made in two processes: water level data preprocessing and downstream water level anomaly detection: A-Establish a water level data preprocessing method including upstream and downstream water level time delay: A1 constructs a preprocessing method for irregular data. Based on the characteristics of uneven temporal density and inconsistent variation range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals and normalized to convert the relationship between the upstream and downstream water level data from absolute values ​​to relative values. A2 Based on the characteristics of similar upstream and downstream water level variation trends and different variation rates, the dynamic time difference warping algorithm is used to compress and stretch the water level data on the time axis according to the similarity of the upstream and downstream water level variation trends, so that the upstream and downstream water level data correspond one to one under the same variation trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring, and the upstream and downstream water level time delay is calculated based on the time attribute of the longest common substring digital data. A3 uses the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and uses the upstream water level data to calculate the water level data value corresponding to that time, thereby obtaining the upstream and downstream water level data at the same time. B-Establish a water level anomaly detection method based on support vector regression: Based on the correlation between the water level value at a certain moment of the measuring station and the water level value at the previous moment, and the similar changing trends of the water level upstream and downstream of the station, assist in the detection of downstream water level anomalies, and establish a water level anomaly detection method on the basis of solving the upstream and downstream water level delay: B1 Analyze the characteristics of water level data, select the upstream water level data and the water level data at the previous moment that are related to the station water level data as auxiliary factors for real-time water level anomaly detection, and analyze the kernel function and its parameters in support vector regression; B2 Use support vector machine regression to establish a water level relationship model between the upstream water level and the station water level at the previous moment and the real-time water level of the station, and verify the evaluation indicators of the effect of the established water level model; B3 Predict the water level value according to the established water level relationship model, analyze the residual between the water level prediction value and the measured value, derive the normal interval range according to the residual distribution characteristics to find the abnormal residual value, and derive the water level anomaly based on the time attribute of the residual sequence; According to the characteristics of the water level dataset, the longest common substring algorithm is improved to meet the following requirements. Assume that the longest common substrings are: Upstream water level subsequence: S′ k = <s′1,s′2,s′3…,s′ k >where s′ i =(se i , sr i ); Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…,f′ k >where f′ i =(fo i ,fr i ); 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 The formula is established; 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant; If the water level sequence meets the above requirements, the longest common substring length of the upstream and downstream water level sequences is k. First, the common substring matrix DM of the upstream and downstream water level sequences is established. n First, calculate the first row and first column of the common substring matrix. The value of the first row element is whether the upstream water level data is approximately equal to the first downstream water level data. If equal, it is 1 otherwise 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If equal, it is 1 otherwise 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If equal, the value of the node is the value of the upper left element plus 1, otherwise it is 0. The calculation formula of the common substring matrix element dm(i,j) is as follows: As long as the absolute value of the difference between the upstream and downstream water levels is not greater than c, the two are considered equal, and the data corresponding to the longest common substring is the water level data with the closest water level change trend; Tips for selecting upstream water level data: The initial upstream and downstream water level time series are: Upstream water level time series: D n = <d1=(v1,t1),d2=(v2,t2),…,d n-1 =(v n-1 , t n-1 ), d n =(v n , t n )> Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ), g n =(h n ,u n )> Assume that the upstream time set is T = {t1, t2, L, t n-1 , t n }, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i -deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 ,t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT is i Calculate as follows: Upstream water level time series considering the upstream and downstream water level delay problem: UD n = <ud1,ud2,ud3…,ud n-1 ,ud n > where ud i =(z i ,nu i ), at this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend; Establish a water level correlation regression model and anomaly detection: When establishing a water level relationship model, the standard deviation, mean square error and r-square of the actual water level value and the predicted water level value in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the more consistent the water level relationship model is with the changing trend of the actual water level. The model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0,1] interval. When r_square is closer to 1, the function fits better, and the water level relationship model is suitable for the water level data set. If the evaluation index obtained by the water level relationship model is poor, the parameters in the support vector regression are readjusted until the evaluation index meets the actual requirements. The water level relationship established by the regression model is used to predict the water level value in the future. The anomaly of the water level detection data is obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation selects the unknown parameter based on the maximum probability of the sample occurring when it is used as an estimated value. Let the residual sequence be: Where: τ i is the absolute value of the difference between the actual value and the predicted value.

2. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Water level data preprocessing: Based on the similarity of water level changes, the section with the most similar upstream and downstream water level change trends is found. The upstream and downstream water levels are then calculated and normalized before similarity measurement. After the upstream and downstream water levels have been determined, the upstream water level data is sorted according to the downstream water level collection time to facilitate modeling of the water level relationship. First, the water level data is standardized, and the standardized upstream and downstream water level data are matched using the dynamic time difference bending algorithm. The upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data are found through the longest common substring method in the similarity measurement. The upper and lower water level delays are calculated according to the time attributes of the water level data. According to the delay, the corresponding upstream water level data is found according to the collection time of the downstream water level data for water level relationship modeling.

3. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Evenly spaced water level data: The upstream and downstream water level data are interpolated to have the same time interval. The upstream and downstream water level data are then interpolated and supplemented. The upstream and downstream water level data after even spacing are then similarly measured at the same time to obtain the time delay of the upstream and downstream water levels. Select appropriate water level data time interval: The water level time series data interval selected in this application is △t. The water level data does not change significantly within △t, which reduces the data redundancy caused by fine-grained time intervals. Assume that the upstream water level dataset is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i ,t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n }, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i ),u i-1 ≤u i , the upstream and downstream water levels are normalized into water level time series data starting at 0:00 every day with an interval of △t. The water level data will not change suddenly within △t. When the time point in the time series is not an integer multiple of the time interval △t, the two collection points closest to the required time point are selected, and the water level change pattern during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows: Select the start time startTime and the end time endTime for collecting data from upstream and downstream, and generate a time set R = {r1, r2, L, r n-1 , r n} with an interval of △t, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n are integer multiples of the time interval △t. Calculation of the water level data at the upstream r moment: First, search for a value greater than or equal to r in the upstream initial data. i The minimum time t i , if r i =t i The upstream water level is r i Water level value v at the moment i 'Equal to the initial upstream water level data v i , if r i ≠t i , that is, r i >t i , then there exists r i ∈[t i-1 ,t i ], in a relatively short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then r i The water level value v′ corresponding to the moment i Calculate as follows: The time series data of the upstream water level data after normalization is expressed as: D′ n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >where d′ i =(v′ i , r i ) The downstream water level data is processed in the same way. The processed water level data is: G′ n = <g′1,g′2,g′3,…,g′ n-1 ,g′ n >where g′ i =(h′ i ,r i ) where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data after the above processing are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one. The water level data with the same time arrangement are more reasonably compared for similarity.

4. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Water level data normalization: After the water level data is processed with uniform intervals, the upstream and downstream water level data are arranged correspondingly with the same time interval △t, and the absolute value of the data is converted into an overall relative value. The normalized data ignores the specific value of the data, which is convenient for depicting the change trend of the water level data. The linear normalization method is used to normalize the water level data within the range of (0,1); The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the interval (O, 1). First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 ,d′ n >, where d′ i =(v′ i ,r i ), the water level data is normalized as follows: After normalization, the upstream and downstream water level time series data are all in the interval (0,1), with the same water level measurement standard, and at the same time prepare basic data for the similarity comparison of upstream and downstream water levels.

5. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Determining upstream and downstream water level peaks: The upstream and downstream water level peaks are used to assist in calculating the delay. Specifically, the time it takes for the peak of the upstream water level to be transmitted downstream is used to roughly estimate the upstream and downstream water level delay. The similarity between the water level data adjacent to the peak is then measured to accurately determine the upstream and downstream water level delay. Before calculating the upstream and downstream water level delay, the peak values ​​of the upstream and downstream water levels are first determined. First, a moment is randomly selected in the downstream water level time series to be detected, and the moment with the largest peak value is searched within the time range of one day after the moment. If there are multiple peak values, the peak value with the largest water level value is selected. If there is no peak value within the time period, the trend of the water level change within the time period is judged. If it is a downward trend, the water level peak value is searched before the time period. If it is an upward trend, the downstream water level peak value is searched after the time period. After finding the downstream water level peak value, the peak value of the upstream water level is searched within three days before the downstream peak value. If there are multiple peak values, the moment with the larger peak value is selected to correspond to the peak value of the upstream water level. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, the similarity measurement of the water level data adjacent to the peak value is performed to obtain the upstream and downstream water level delay.

6. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Upstream and downstream water level matching based on dynamic time difference bending: Using the calculated upstream and downstream peak values, select the upstream and downstream water level data near the peak value, and use the dynamic time difference bending method to match the upstream and downstream water level data. Let the upstream water level time series be UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >, the downstream water level time series is DW n = <dw1,dw2,dw3,…,dw m-1 ,dw m >, construct a distance matrix, the element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment, and the difference between the upstream and downstream water levels is the distance; The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m*n Select a line from the upper left corner to the lower right corner. This line is the matching data pair of upstream and downstream water levels. The points along the line are the corresponding matching points of the upstream and downstream water level data. When matching upstream and downstream water level data, the selected matching path must meet the following requirements: Boundary conditions: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start at the lower right corner a mn End, you cannot arbitrarily start matching from the middle data; Continuity: If the upstream water point uwi and the downstream water point dw j If the match is successful, the next upstream matching point uw i+1 Only with downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points; Monotonicity: Points on the path must be aligned from left to right, and data cross-matching is not allowed; The path selection for each grid point can only be performed in three directions. If the grid point coordinates are (i, j), the coordinates of the next grid point can only be (i+1, j), (i, j+1), or (i+1, j+l). The dynamic time difference warping algorithm always selects the path with the smallest cumulative sum in the three directions when selecting the matching path. The matching distance matrix of the upstream and downstream water levels is defined as: Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the minimum neighboring element at that point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows: where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distance reaches the minimum value. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md is used. mn The element starts at the upper left corner of its neighbor md m-1n-1 、The above element md m-1n 、Left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when The upstream water level sequence generated by the dynamic time warping algorithm is: in The downstream water level sequence is: in 7. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Determination of water level delay based on the longest common substring: Use the longest common subsequence algorithm in the matched upstream and downstream water level data to find the upstream and downstream water level data substring with the highest similarity. Compare the time attributes corresponding to the upstream and downstream water level data of the longest common substring. The mean of the difference between the upstream and downstream water level sequence times is the time delay of the upstream and downstream water levels. The longest common substring is solved using the dynamic programming method.

8. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Upstream water level data selection: Use upstream and downstream water level data to detect anomalies. When detecting anomalies in downstream water level data, it is necessary to calculate the time corresponding to the upstream water level according to the upstream and downstream water level delay according to the downstream water level collection time, and then calculate the upstream water level value at that time.

9. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Establish a regression model and prediction for water level-related factors: Support vector regression builds a linear regression model to ensure that as many data set samples as possible fall within the regression model, i.e., the residual is as small as possible. When calculating loss in support vector regression, data deviations within a certain range are considered acceptable and are not counted in the loss. A loss distance constant ε>0 is defined. Support vector regression introduces an insensitive loss function to achieve a strategy of not calculating losses within a certain error range. Two relaxation factors are introduced. By introducing relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the introduction of the relaxation factors, the distance between the actual value and the regression model is transformed into a value less than or equal to the loss distance constant ε after adding the relaxation variables.

10. The method for detecting abnormal points of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Anomaly detection based on residuals: The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level data are detected by using the characteristics that the residuals conform to the normal distribution. The mean and variance of the residuals are calculated based on the maximum likelihood estimation. Conforms to the normal distribution The probability density function of the distribution is: Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is: Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative: According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is: Mean: variance: The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that is impossible to occur under normal circumstances, that is, the residual data is considered to be an outlier.

Images