Water conservancy big data time sequence water level time delay abnormal point detection method

Abstract

According to the water conservancy big data time sequence water level time delay abnormal point detection method, firstly, water level data are preprocessed, the water level data are made to be uniform in density in time and uniform in measurement standard in numerical value through standardization processing, and upstream and downstream water level data are matched through a dynamic time difference bending algorithm, so that the change rules are consistent; and then finding out a time sequence with the highest similarity in the matched water level data through a longest common substring method, and averaging to obtain the water level delay after differencing time attributes of upstream and downstream water level data in the common substring. A water level relation model is established through support vector regression by using an upstream water level value and a relation between a downstream water level value at a previous moment and a downstream current water level value, a water level predicted value is obtained by using the water level relation model, then a residual error between an actual value and the predicted value is calculated, and a water level abnormal point is obtained by analyzing a distribution model of the residual error. And the water level time delay abnormal point detection accuracy and precision are greatly improved.

Description

Technical Field

[0001] The present application relates to a method for detecting time-series water level delay anomalies, and in particular to a method for detecting time-series water level delay anomalies with water conservancy big data, belonging to the technical field of water conservancy big data detection. Background Art

[0002] Water conservancy projects are a project that benefits the country and the people by regulating natural surface water and groundwater to achieve flood control and drought relief. With the rapid development of computer technology and communication technology, society has entered the era of Internet+, and the informatization of water conservancy is also imminent. Hydrological monitoring stations use wireless communication modules to transmit monitoring data in real time, which can timely reflect the hydrological characteristics of different waters and greatly improve the work efficiency of hydrological and water conservancy departments. At the same time, water conservancy departments use the collected hydrological data to make water resource scheduling arrangements to prevent flood disasters. The digitization of hydrological information provides a theoretical basis for major decisions of water conservancy departments, provides intuitive data for water information, and realizes the scientific, forward-looking and correct management of water conservancy projects.

[0003] Although the rapid development of wireless sensing technology has realized the automatic collection and transmission of hydrological data, the hydrological monitoring equipment is placed in a natural state, and the collected data must also be transmitted through the network, which may cause inaccurate hydrological data. Due to factors such as network instability, signal interference, aging and damage of detection equipment, the data obtained during the collection and transmission of hydrological data deviates from the actual value. Changes in water conditions data at monitoring stations are caused by natural factors such as rainfall, equipment failures, and network instability. The accuracy of hydrological data provides data support for the water conservancy department to formulate effective water resources scheduling plans. Effective flood control and drought relief decisions are of great significance to the production and life of surrounding residents and the safety of life and property.

[0004] Anomaly detection can find out the observed values ​​that deviate from the actual values ​​by a higher or lower value, and can make preliminary corrections to the anomalies. At the same time, it can prompt the staff to check the water level values ​​in real time during special periods, and manually correct the water level values ​​to ensure the accuracy of the water level data and ensure that there are no flood accidents during special periods. For example, if the water level data is inaccurate during the collection and transmission process due to network instability or equipment aging, it will affect the water conservancy workers' judgment of the reservoir water conditions, and will also affect the analysis and research of the water conditions of the site in the later stage. Anomaly detection can prompt the staff to find the cause of the data error, so as to repair and replace the monitoring equipment or check whether there is a fault in the network. Anomaly detection can also make preliminary corrections to the abnormal data, so that the water conservancy staff can correctly judge the water conditions, and it also has good guiding significance and role for the later water conservancy work.

[0005] The problems that need to be solved by the existing water level abnormal point detection method and the key technical difficulties of this application include:

[0006] (1) The existing data anomaly detection scheme generally studies anomalies from two aspects. One aspect is to establish a relationship model between current data and historical data, and detect anomaly data based on probability calculation, distance, density or clustering methods. These methods only detect anomalies from a numerical point of view. The change of water level data is a gradual process, and the water level data at the same station at adjacent times are closely related. The other part of the method ignores the time attribute of water level data and the correlation of adjacent data, and only establishes a simple relationship between data, which is not suitable for complex and changeable water level data. In addition, due to the inconsistent change rate of upstream and downstream water levels, the slope difference is large and it is not suitable for direct similarity measurement. Therefore, the existing technology method is not suitable for water level delay anomaly detection.

[0007] (2) Since water level data has missing and redundant data in certain periods, and the water level data collection frequency and collection time of different stations are different, the water level data collected by the monitoring equipment cannot be directly used for data analysis and processing. The existing technology lacks standardization of water level data, lacks the removal of redundant data, and cannot complete the missing data. The data format is not unified and cannot meet the requirements of data mining. Due to the distance between the upstream and downstream measuring stations and the particularity of water resources, there is a certain delay in the change trend of the upstream and downstream water levels. The existing technology cannot find the section with the most similar change trend of the upstream and downstream water levels according to the similarity of water level changes, and cannot calculate the upstream and downstream water level delay. Before the similarity measurement, the upstream and downstream water levels are not standardized, and the similarity measurement cannot be performed. After finding the delay of the upstream and downstream water levels, the upstream water level data cannot be sorted according to the collection time of the downstream water level, and the water level relationship cannot be modeled. The time density of water level data is different, which is not convenient for data similarity measurement and subsequent data analysis and processing. The data mining cost is high, the data mining efficiency is low, and the practicality and effectiveness of the mined information are low, which is not conducive to the detection of water level delay anomalies.

[0008] (3) The existing technology cannot solve the problem of abnormal collection and transmission of water level data. It lacks analysis of the influencing factors related to the real-time water level of the downstream. It is impossible to use support vector regression to establish the relationship between water levels. There is a lack of explanation of the characteristics of the kernel function and important parameters in the support vector regression, and a lack of analysis of the impact of the parameters in the kernel function on the water level relationship model. There is a lack of using support vector regression to establish a relationship model between the real-time water level of the site and the water level of the upstream site at the corresponding moment and the water level of the site at the previous moment. There is a lack of setting of important parameters in the support vector regression based on the actual characteristics of the water level data set, and it is impossible to use the support vector regression method to establish a water level relationship model. It is impossible to predict the water level value based on the established water level relationship model, and it is impossible to calculate the residual between the predicted water level value and the actual value. There is a lack of detection of abnormal residuals through probability distribution methods using the residual conforming to the normal distribution characteristics, and it is impossible to derive abnormal water levels based on the time attributes of the residual sequence. Summary of the invention

[0009] This application is based on the characteristics of hydrological data and is developed and improved from two processes: water level data preprocessing and abnormal point detection of downstream water level. First, the water level data is preprocessed, and the water level data is standardized to make the density uniform in time and the numerical measurement standard unified. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their change rules consistent. Then, the time series with the highest similarity is found in the matched water level data by the longest common substring method. The time attribute of the upstream and downstream water level data in the common substring is used to find the difference and average to obtain the water level delay. The upstream and downstream water level data are aligned by time delay, and the aligned upstream and downstream data are used to detect abnormal points. Using the relationship between the upstream water level value and the previous moment downstream water level value and the current downstream water level value, a water level relationship model is established by support vector regression, and the water level prediction value is obtained by using the water level relationship model. Then, the residual between the actual value and the predicted value is calculated, and the water level abnormal point is obtained by analyzing the distribution model of the residual. The accuracy and precision of water level delay abnormal point detection are greatly improved.

[0010] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] The water level delay anomaly detection method for water conservancy big data is improved from two processes: water level data preprocessing and downstream water level anomaly detection based on hydrological data characteristics:

[0012] A- Establish a water level data preprocessing method including upstream and downstream water level delay:

[0013] A1 constructs a preprocessing method for irregular data. Based on the characteristics of uneven time density and inconsistent variation range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals to improve data availability and reduce data redundancy. Normalization is used to convert the absolute value of the upstream and downstream water level data into a relative value relationship.

[0014] A2 uses the dynamic time difference bending algorithm to compress and stretch the water level data on the time axis based on the similarity of the upstream and downstream water level change trends and the different change rates, so that the upstream and downstream water level data correspond one by one under the same change trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring, and the upstream and downstream water level delay is calculated according to the time attribute of the longest common substring digital data;

[0015] A3 uses the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and uses the upstream water level data to calculate the corresponding water level data value at that time to obtain the upstream and downstream water level data at the same time;

[0016] B- Establish a water level anomaly detection method based on support vector regression: Based on the correlation between the water level value at a certain moment of the measuring station and the water level value at the previous moment, and the similar change trend between the water level upstream and downstream of the station, these two water level factors are used to assist the downstream water level anomaly detection. A water level anomaly detection method is established on the basis of solving the upstream and downstream water level delay:

[0017] B1 analyzes the characteristics of water level data, selects upstream water level data related to the station water level data and water level data at the previous moment as auxiliary factors for real-time water level anomaly detection, and analyzes the kernel function and its parameters in support vector regression;

[0018] B2 uses support vector machine regression to establish a water level relationship model between the upstream water level and the site water level at the previous moment and the site real-time water level, and verifies the evaluation index of the effect of the established water level model;

[0019] B3 predicts the water level value based on the established water level relationship model. By analyzing the residuals between the predicted water level value and the measured value, the normal range is obtained based on the distribution characteristics of the residuals to find the abnormal residual values, and then the water level anomaly points are obtained based on the time attributes of the residual sequence.

[0020] Preferably, water level data is preprocessed: according to the similarity of water level changes, the section with the closest change trend of upstream and downstream water levels is found, and then the upstream and downstream water level delay is calculated. Before similarity measurement, the upstream and downstream water levels are standardized, and then the similarity measurement can be performed. After finding the time delay of the upstream and downstream water levels, the upstream water level data is sorted according to the collection time of the downstream water level, so as to facilitate the subsequent modeling of the water level relationship;

[0021] First, the water level data is standardized, and the standardized upstream and downstream water level data are matched using the dynamic time difference bending algorithm. The upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data are found through the longest common substring method in the similarity measurement. The upper and lower water level time delays are calculated according to the time attributes of the water level data. According to the time delay, the corresponding upstream water level data is found according to the collection time of the downstream water level data for water level relationship modeling.

[0022] Preferably, the water level data are evenly spaced: the upstream and downstream water level data have the same time interval, the upstream and downstream water level data are interpolated and supplemented, and the upstream and downstream water level data after even spacing are similarly measured at the same time to obtain the time delay of the upstream and downstream water levels;

[0023] Select appropriate water level data time interval: The water level time series data interval selected in this application is △t. The water level data does not change significantly within △t, which reduces the data redundancy caused by fine-grained time intervals.

[0024] Assume that the upstream water level dataset is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i ,t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n}, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i )、u i-1 ≤u i, the upstream and downstream water levels are standardized into water level time series data starting at 0:00 every day with an interval of △t. The water level data will not change suddenly within △t. When the time point in the time series is not an integer multiple of the time interval △t, the two collection points closest to the required time point are selected, and the water level change law during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows:

[0025] Select the start time startTime and end time endTime of the upstream and downstream data collection, and generate a time set with an interval of △t as R = {r1 ， r2,L,r n-1 , r n}, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n The calculation of upstream water level data at time r is an integer multiple of the time interval △t:

[0026] First, search for a value greater than or equal to r in the upstream initial data. i The minimum time t i , if r i =t i The upstream water level is r i The water level value v at the moment i , which is equal to the initial upstream water level data v i , if r i ≠ti, that is, r i >t i , then there exists r i ∈[t i-1 , t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then the water level value v′ corresponding to time ri is i Calculate as follows:

[0027]

[0028] The time series data of the upstream water level data after standardization is expressed as:

[0029] D′ n = <d′1,d′2,d′3,…,d′ n-1 ,d′ n >where d′ i =(v′ i , r i )

[0030] The downstream water level data is processed in the same way. The processed water level data is:

[0031] G′ n = <g′1,g′2,g′3,…,g′ n-1 ,g′ n >where g′ i =(h′ i ,r i )

[0032] where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data processed as above are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one, and the water level data with the same time arrangement are more reasonably compared for similarity.

[0033] Preferably, the water level data is normalized:

[0034] After the water level data is processed with uniform intervals, the upstream and downstream water level data are arranged correspondingly at the same time interval △t, and the absolute value of the data is converted into an overall relative value. The normalized data ignores the specific value of the data, which is convenient for describing the change trend of the water level data. The linear normalization method is used to normalize the water level data within the range of (0,1);

[0035] The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the (O, 1) interval. First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and the minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >, where d′ i =(v′ i , r i ), the water level data is normalized as follows:

[0036]

[0037] After normalization, the upstream and downstream water level time series data are all in the interval (0,1), with the same water level measurement standard, and at the same time, basic data are prepared for the similarity comparison of upstream and downstream water levels.

[0038] Preferably, the upstream and downstream water level peaks are determined: the upstream and downstream water level peaks are used to assist in calculating the delay, that is, how long does it take for the peak of the upstream water level to be transmitted to the downstream? This transmission time is a rough value of the upstream and downstream water level delay, and then the similarity measurement of the peak adjacent water level data is performed to accurately solve the upstream and downstream water level delay;

[0039] Before calculating the upstream and downstream water level delay, the peak values ​​of the upstream and downstream water levels are first determined. First, a moment is randomly selected in the downstream water level time series to be detected, and the moment with the largest peak value is found within the time range of one day after the moment. If there are multiple peak values, the peak with the largest water level value is selected. If there is no peak value within the time period, the trend of the water level change within the time period is judged. If it is a downward trend, the water level peak value is found before the time period. If it is an upward trend, the downstream water level peak value is found after the time period. After finding the downstream water level peak value, the peak value of the upstream water level is found within three days before the downstream peak value. If there are multiple peaks, the peak value of the upstream water level corresponding to the moment with the larger peak value is selected. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, the water level data adjacent to the peak values ​​are measured for similarity to obtain the upstream and downstream water level delays.

[0040] Preferably, the upstream and downstream water level matching based on dynamic time difference bending is as follows: using the calculated upstream and downstream peak values, selecting the upstream and downstream water level data near the peak values, and matching the upstream and downstream water level data using the dynamic time difference bending method, assuming that the upstream water level time series is UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >, the downstream water level time series is DW n = <dw1,dw2,dw3,…,dw m-1 , dw m >, construct a distance matrix, the element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment, and the difference between the upstream and downstream water levels is its distance;

[0041] The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m*n Select a line from the upper left corner to the lower right corner. This line is the matching data pair of the upstream and downstream water levels. The points along the line are the corresponding matching points of the upstream and downstream water level data.

[0042] In the process of matching upstream and downstream water level data, the selected matching path must meet the following requirements:

[0043] Boundary condition: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start to the lower right corner a mn End, you cannot arbitrarily start matching from the middle data;

[0044] Continuity: If the upstream water point uwi and the downstream water point dw j If the match is successful, the next upstream matching point uw i+1 Only downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points;

[0045] Monotonicity: The points on the path must be aligned from left to right, and data cross-matching cannot be performed;

[0046] The path selection of each grid point can only be carried out in three directions. Let the coordinates of the grid point be (i, j). The coordinates of the next grid point can only be (i+1, j), (i,.j+1) or (i+1, j+l). The dynamic time difference bending algorithm always selects the path with the smallest cumulative sum in three directions when selecting the matching path. The matching distance matrix of the upstream and downstream water levels is defined as:

[0047]

[0048] Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the smallest neighboring element to reach this point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows:

[0049]

[0050] where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distance reaches the minimum value. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md mn The element starts at the upper left corner of its neighbor md m-1n-1 、The above element md m-1n 、The left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when

[0051] The upstream water level sequence generated by the dynamic time difference bending algorithm is:

[0052]

[0053] The downstream water level sequence is:

[0054]

[0055] Preferably, the water level delay is determined based on the longest common substring: the longest common subsequence algorithm is used in the matched upstream and downstream water level data to find the upstream and downstream water level data substring with the highest similarity, and the time attributes corresponding to the upstream and downstream water level data of the longest common substring are compared. The mean value of the difference between the upstream and downstream water level sequence time is the time delay of the upstream and downstream water levels;

[0056] Assume that the longest common substrings are:

[0057] Upstream water level subsequence: S′ k = <s′1,s′2,s′3…,s′ k >where s′ i =(se i ,sr i );

[0058] Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…f′ k >where f i ′=(fo i ,fr i )

[0059] The longest common substring algorithm is improved according to the characteristics of the water level data set. The improved algorithm meets the following requirements:

[0060] 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 Formula established;

[0061] 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant;

[0062] If the water level sequence meets the above requirements, the length of the longest common substring of the upstream and downstream water level sequences is k;

[0063] The longest common substring is solved by dynamic programming. First, the common substring matrix DM of the upstream and downstream water level sequences is established. n, first calculate the first row and the first column of the common substring matrix. The value of the first row element is whether the data of the upstream water level is approximately equal to the data of the first downstream water level. If they are equal, it is 1, otherwise it is 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If they are equal, it is 1, otherwise it is 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If they are equal, the value of the node is the value of the upper left element plus 1, otherwise it is 0. The calculation formula of the common substring matrix element dm(i,j) is as follows:

[0064]

[0065] As long as the absolute value of the difference between the upstream water level and the downstream water level is not higher than c, the two are considered equal in theory, and the data corresponding to the longest common substring is the water level data with the closest water level change trend.

[0066] Preferably, the upstream water level data is selected: the upstream and downstream water level data are used to detect abnormal points. When detecting abnormal points in the downstream water level data, the time corresponding to the upstream water level needs to be calculated according to the upstream and downstream water level delays according to the downstream water level acquisition time, and then the upstream water level value at that time is calculated;

[0067] The initial upstream and downstream water level time series is:

[0068] Upstream water level time series: D n =<d1=(v1,t1), d2=(v2,t2),…,d n-1 =(v n-1 , t n-1 ), d n =(v n , t n )>

[0069] Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ), g n =(h n ,u n )>

[0070] The method of calculating the upstream water level based on the downstream water level is as follows: Assume that the upstream time set is T = {t1, t2, L, t n-1 , t n}, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i-deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 ,t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT i Calculate as follows:

[0071]

[0072] The upstream water level time series considering the upstream and downstream water level delay problem:

[0073] UD n = <ud1,ud2,ud3…,ud n-1 ,ud n > where ud i =(z i ,nu i )

[0074] At this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend. The upstream data and the downstream data at the same time are closely related.

[0075] Preferably, a regression model and prediction of water level related factors are established: support vector regression establishes a linear regression model so that as many data set samples as possible are located on the regression model, that is, the residual is as small as possible. When the support vector regression performs loss calculation, it is determined that data deviations within a certain range are acceptable and are not included in the loss. The loss distance constant ε>0 is defined. The support vector regression introduces an insensitive loss function to implement a strategy of not calculating losses within a certain error range.

[0076] Two relaxation factors are introduced. By introducing relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the introduction of relaxation factors, the distance between the actual value and the regression model is converted to be less than or equal to the loss distance constant ε after adding the relaxation variables.

[0077] When establishing the water level relationship model, the standard deviation, mean square error and r-square of the actual water level value and the predicted water level value in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the better the water level relationship model conforms to the changing trend of the actual water level, indicating that the model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0,1] interval. When r_square is approximately close to 1, it means that the function fits better, that is, the water level relationship model is suitable for the water level data set. If the evaluation index of the water level relationship model is poor, it is necessary to readjust the parameters in the support vector regression until the evaluation index meets the actual requirements.

[0078] Preferably, residual-based outlier detection:

[0079] The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level detection data are obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation selects the unknown parameters based on the maximum probability of the sample occurring when it is used as the estimated value. Suppose the residual sequence is: Where: τi is the absolute value of the difference between the actual value and the predicted value;

[0080] The abnormal points of water level data are detected by using the characteristics that the residual conforms to the normal distribution. The mean and variance of the residual are calculated based on the maximum likelihood estimation. The data set It conforms to the normal distribution The distribution probability density function is:

[0081]

[0082] Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is:

[0083]

[0084] Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative:

[0085]

[0086] According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is:

[0087]

[0088] The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that cannot occur under normal circumstances, that is, the residual data is considered to be an outlier.

[0089] Compared with the prior art, the innovations and advantages of this application are:

[0090] (1) This application is based on the characteristics of hydrological data and is developed and improved from two processes: water level data preprocessing and abnormal point detection of downstream water level. First, the water level data is preprocessed, and the water level data is standardized to make the density uniform in time and the numerical measurement standard unified. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their change rules consistent. Then, the time series with the highest similarity is found in the matched water level data by the longest common substring method. The time attribute of the upstream and downstream water level data in the common substring is used to find the difference and average to obtain the water level delay. Then, the upstream and downstream water level data are aligned by time delay, and the aligned upstream and downstream data are used to detect abnormal points. Using the relationship between the upstream water level value and the previous moment downstream water level value and the current downstream water level value, a water level relationship model is established by support vector regression. The water level prediction value is obtained by using the water level relationship model, and then the residual between the actual value and the predicted value is calculated. The water level abnormal point is obtained by analyzing the distribution model of the residual. The accuracy and precision of water level delay abnormal point detection are greatly improved.

[0091] (2) When utilizing the relationship between the upstream and downstream water levels, the present application noticed that there is a certain time delay in the changing trend of the upstream and downstream water levels, and established a water level data preprocessing method including the upstream and downstream water level time delay. Based on the similarity of the upstream and downstream water level changing trends and the different changing rates, the dynamic time difference bending algorithm is used to compress and stretch the water level data on the time axis according to the similarity of the upstream and downstream water level changing trends, so that the upstream and downstream water level data correspond one by one under the same changing trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring, and the upstream and downstream water level time delay is calculated according to the time attribute of the longest common substring digital data, and the upstream collection time corresponding to the downstream water level collection time is found. The upstream water level data is used to calculate the water level data value corresponding to the time, and the upstream and downstream water level data of the same time are obtained. The relationship between the upstream and downstream water levels is cleverly utilized to improve the quality of water level delay anomaly detection.

[0092] (3) When detecting water level anomalies, the present application considers the relationship between the upstream and downstream water levels and the relationship between the current water level value and the water level value at the adjacent historical moments, and uses the relationship between these variables and the current water level to assist in the detection of water level anomalies. Since water level data is a type of data with low precision but large sample size, a water level anomaly detection method based on support vector regression is established to process imprecise data, analyze the characteristics of water level data, and use support vector machine regression to establish a water level relationship model between the upstream water level and the site water level at the previous moment and the real-time water level of the site. The evaluation index of the established water level model effect is verified, and the water level value is predicted based on the established water level relationship model, which greatly improves the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is the symbols used in this application and their description diagram.

[0094] Figure 2 It is a flow chart for determining the peak water levels in the upstream and downstream.

[0095] Figure 3 It is a flow chart of establishing a real-time water level relationship model using the upstream water level and the water level at the previous moment.

[0096] Figure 4 This is a schematic diagram of the downstream data before water level interval processing.

[0097] Figure 5 It is a schematic diagram of the actual data points of the downstream water level in the data preprocessing experiment.

[0098] Figure 6 It is the upstream and downstream water level data graph before normalization in the data preprocessing experiment.

[0099] Figure 7 It is a schematic diagram of the water level data fragment before the dynamic time difference bending algorithm is matched.

[0100] Figure 8 This is a schematic diagram of the upstream and downstream water level data after matching with the dynamic time difference bending algorithm.

[0101] Fig. 9 This is a comparison chart of the accuracy of the method of this application and the outlier solution based on the sliding window. DETAILED DESCRIPTION

[0102] The following, in conjunction with the accompanying drawings, further describes the technical solution of the method for detecting time-series water level delay anomalies in water conservancy big data provided by the present application, so that those skilled in the art can better understand the present application and implement it.

[0103] Water level data is a special sequence with time attributes, and the temporal change pattern of water level values ​​is unpredictable. Based on the close relationship between the current water level value and the upstream water level and the adjacent historical water level values, the support vector regression algorithm is used to establish a relationship model between these two related factors and the current water level to assist in the detection of abnormal water level points downstream. In actual situations, although the upstream and downstream have similar change trends, there is a certain delay in the change of the upstream and downstream water levels. This application processes the upstream and downstream water level delays before detecting water level abnormalities.

[0104] Firstly, the water level data is preprocessed and standardized to make the water level data uniform in time density and unified in numerical measurement standards. The dynamic time difference bending algorithm is used to match the upstream and downstream water level data to make their changing patterns consistent. Then, the longest common substring method is used to find the time series with the highest similarity in the matched water level data. The time attributes of the upstream and downstream water level data in the common substring are used to find the difference and average to get the water level delay. Then, the upstream and downstream water level data are aligned through time delay, and the aligned upstream and downstream data are used to detect anomalies.

[0105] Using the relationship between the upstream water level value and the downstream water level value at the previous moment and the current downstream water level value, a water level relationship model is established through support vector regression. The water level prediction value is obtained using the water level relationship model, and then the residual between the actual value and the predicted value is calculated. The water level anomaly point is obtained by analyzing the distribution model of the residual.

[0106] A-Establish a water level data preprocessing method including upstream and downstream water level delay: A1 Construct a preprocessing method for irregular data. Based on the characteristics of uneven time density and inconsistent change range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals to improve data availability and reduce data redundancy. Normalization is used to change the relationship between the upstream and downstream water level data from absolute values ​​to relative values. A2 Based on the characteristics of similar upstream and downstream water level change trends and different change rates, the dynamic time difference bending algorithm is used to compress and stretch the water level data on the time axis according to the similarity of the upstream and downstream water level change trends, so that the upstream and downstream water level data correspond one by one under the same change trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level fragments with the highest similarity using the longest common substring, and the upstream and downstream water level delay is calculated according to the time attribute of the longest common substring digital data. A3 Use the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and use the upstream water level data to calculate the water level data value corresponding to the time, and obtain the upstream and downstream water level data at the same time;

[0107] B- Establishing a water level anomaly detection method based on support vector regression:

[0108] Based on the correlation between the water level value of the measuring station at a certain moment and the water level value at the previous moment, and the similar change trend between the water level upstream and downstream of the station, these two water level factors are used to assist the detection of abnormal water level points downstream. On the basis of solving the upstream and downstream water level delay, a water level abnormal point detection method based on support vector regression is established: B1 analyzes the characteristics of water level data, selects upstream water level data related to the station water level data and the water level data at the previous moment as auxiliary factors for real-time water level abnormal point detection, and analyzes the kernel function and its parameters in support vector regression; B2 uses support vector machine regression to establish a water level relationship model between the upstream water level and the station water level at the previous moment and the real-time water level of the station, and verifies the evaluation indicators of the effect of the established water level model;

[0109] B3 predicts the water level value based on the established water level relationship model. By analyzing the residuals between the predicted water level value and the measured value, the normal range is obtained based on the distribution characteristics of the residuals to find the abnormal residual values, and then the water level anomaly points are obtained based on the time attributes of the residual sequence.

[0110] 1. Water level data preprocessing

[0111] Since water level data has missing and redundant data in certain periods, and the collection frequency and time of water level data at different sites are different, it is necessary to standardize the water level data before detecting abnormal points in water level data. Data processing can remove redundant data, and missing data can be completed according to certain rules to unify the data format and meet the requirements of data mining. Water level data preprocessing makes water level data have the same time density, which is convenient for data similarity measurement and subsequent data analysis and processing. Data preprocessing makes data mining possible, and the processed data reduces the cost of data mining, improves the efficiency of data mining, and improves the practicality and effectiveness of mining information.

[0112] The purpose of this application is to detect abnormal points in water level data, but the water level data collected by the monitoring equipment cannot be directly used for data analysis and processing. Therefore, it is necessary to pre-process the water level data before detecting abnormal water level data. The upstream water level and the downstream measuring station water level have similar changing trends, but due to the distance between the upstream and downstream measuring stations and the particularity of water resources, there is a certain delay in the changing trend of the upstream and downstream water levels. This application finds the section with the closest changing trend of the upstream and downstream water levels based on the similarity of water level changes, and then calculates the upstream and downstream water level delays. Before measuring the similarity, the upstream and downstream water levels are standardized, and then the similarity can be measured. After finding the time delay of the upstream and downstream water levels, the upstream water level data is sorted according to the collection time of the downstream water level, so as to facilitate the subsequent modeling of the water level relationship.

[0113] This application first standardizes the water level data, matches the standardized upstream and downstream water level data using a dynamic time difference bending algorithm, finds the upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data through the longest common substring method in the similarity measurement, calculates the upper and lower water level time delays based on the time attributes of the water level data, and finds the corresponding upstream water level data based on the time delay and the collection time of the downstream water level data for water level relationship modeling.

[0114] 1. Water level preprocessing scenarios and objectives

[0115] 1. Scenario Description

[0116] Since the collection time and interval of upstream and downstream water level data are different, and the collected data are redundant and missing, it is necessary to standardize the water level data. Although the upstream and downstream water levels show similar change trends, since the water needs to pass through a long section of the river when flowing from the upstream station to the downstream station, the downstream water level will not rise immediately when the upstream water level rises, but will lag in time. Although the upstream and downstream water levels show similar change patterns, the change of the downstream water level will be delayed. When using the upstream water level to assist the downstream water level in abnormal detection, it is necessary to first solve the time delay problem of the upstream and downstream water levels.

[0117] 2. Preprocessing objectives

[0118] In response to the above problems, this application sets the following goals: (1) Water level data standardization: evenly space and normalize water level data to make it more consistent with the changing characteristics of water levels, and also make the upstream and downstream water level data within the same measurement range. (2) According to the changing trends of upstream and downstream water levels, the upstream and downstream water level data with the highest similarity are obtained by comparing the similarities of the upstream and downstream water levels, and the upstream and downstream water level delays are obtained using the time attributes of the water level data. (3) According to the time attributes of the water level value of the station to be tested, the water level value at the corresponding time upstream is selected using the collection time of the downstream station. The symbols used in this application and their descriptions are shown in Figure 1 .

[0119] 2. Water level data standardization

[0120] 1. Water level data is evenly spaced

[0121] Since the upstream and downstream water level data are collected by two different monitoring devices, and the sampling frequency and sampling time of the two monitoring devices are inconsistent, the time series water level data is unevenly distributed in time density, which is not conducive to the similarity comparison between the upstream and downstream water levels. Therefore, it is necessary to make the upstream and downstream water level data have the same time interval. Since the time intervals of the upstream and downstream water levels monitored are different, the uniform interval process needs to interpolate and supplement the upstream and downstream water level data according to certain rules. After uniform interval, the upstream and downstream water level data can be measured for similarity at the same time to obtain the time delay of the upstream and downstream water levels.

[0122] Select appropriate water level data time interval: The water level time series data interval selected in this application is △t. The water level data does not change significantly within △t, which reduces the data redundancy caused by fine-grained time intervals. Suppose the upstream water level data set is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i ,t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n}, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i )、u i-1 ≤u i , the upstream and downstream water levels are standardized into water level time series data starting at 0:00 every day with an interval of △t. The water level data will not change suddenly within △t. When the time point in the time series is not an integer multiple of the time interval △t, the two collection points closest to the required time point are selected, and the water level change law during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows: Select the start time startTime and end time endTime of the upstream and downstream collection data, and generate a time set with an interval of △t as R = {r1 ， r2,L,r n-1 , r n}, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n The calculation of the upstream water level data at time r is an integer multiple of the time interval △t: First, find the value greater than or equal to r in the upstream initial data. i The minimum time ti , if r i =t i The upstream water level is r i The water level value v at the moment i , which is equal to the initial upstream water level data v i , if r i ≠t i , that is, r i >t i , then there exists r i ∈[t i-1 , t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then r i The water level value v′ corresponding to the time i Calculate as follows:

[0123]

[0124] The time series data of the upstream water level data after standardization is expressed as:

[0125] D′ n = <d′1,d′2,d′3,…,d′ n-1 ,d′ n > where d' i =(v′ i ,r i )

[0126] The downstream water level data is processed in the same way. The processed water level data is:

[0127] G′ n = <g′1,g′2,g′3,…,g′ n-1 , g′ n >where g′ i =(h′ i ,r i )

[0128] where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data processed as above are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one, and the water level data with the same time arrangement are more reasonably compared for similarity.

[0129] 2. Water level data normalization

[0130] After the water level data is processed with uniform intervals, the upstream and downstream water level data are arranged in the same time interval △t, but the upstream and downstream water level values ​​are in different ranges, which is not conducive to the similarity measurement of water level values ​​of different scales. Normalization processing converts the absolute value of the data into an overall relative value. The normalized data ignores the specific value of the data and pays more attention to the change relationship of the data, which is convenient for describing the change trend of the water level data. The linear normalization method is used to normalize the water level data within the range of (0,1).

[0131] The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the (O, 1) interval. First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and the minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >, where d′ i =(v′ i , r i ), the water level data is normalized as follows:

[0132]

[0133] After normalization, the upstream and downstream water level time series data are all in the interval (0,1), with the same water level measurement standard, and at the same time, basic data are prepared for the similarity comparison of upstream and downstream water levels.

[0134] (III) Determination of water level delay based on similarity

[0135] 1. Determination of upstream and downstream water level peaks

[0136] The upstream and downstream water level peaks are used to assist in the calculation of delay, that is, how long does it take for the peak of the upstream water level to be transmitted to the downstream. This transmission time is a rough value of the upstream and downstream water level delay. The similarity of the peak adjacent water level data is then measured to accurately solve the upstream and downstream water level delay.

[0137] Before calculating the upstream and downstream water level delay, first determine the peak values ​​of the upstream and downstream water levels. First, randomly select a moment in the downstream water level time series to be detected, and look for the moment with the largest peak value within the time range of one day after the moment. If there are multiple peak values, select the peak value with the largest water level value. If there is no peak value within the time period, determine the trend of the water level change within the time period. If it is a downward trend, look for the water level peak value before the time period. If it is an upward trend, look for the downstream water level peak value after the time period. After finding the downstream water level peak value, look for the peak value of the upstream water level within three days before the downstream peak moment (i.e. 1.5 times the empirical value of the water level delay). If there are multiple peaks, select the peak value of the upstream water level corresponding to the moment with the larger peak value. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, perform similarity measurement on the water level data adjacent to the peak value to obtain the upstream and downstream water level delay. The flowchart for determining the peak values ​​of the upstream and downstream water levels is as follows: Figure 2 .

[0138] 2. Upstream and downstream water level matching based on dynamic time difference bending

[0139] Using the calculated upstream and downstream peak values, we select the upstream and downstream water level data near the peak values, and use the dynamic time difference bending method to match the upstream and downstream water level data. Let the upstream water level time series be UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >, the downstream water level time series is DW n = <dw1,dw2,dw3,…,dw m-1 , dw m >, construct a distance matrix, the element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment, and the difference between the upstream and downstream water levels is its distance. The upstream and downstream water level distance matrix is:

[0140] The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m*n Select a line connecting the upper left corner to the lower right corner. This line is the matching data pair of upstream and downstream water levels. The points passed by the line are the corresponding matching points of the upstream and downstream water level data.

[0141] In the process of matching upstream and downstream water level data, the selected matching path must meet the following requirements:

[0142] Boundary condition: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start to the lower right corner a mn End, you cannot arbitrarily start matching from the middle data;

[0143] Continuity: If the upstream water point uwi and the downstream water point dw jIf the match is successful, the next upstream matching point uw i+1 Only downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points;

[0144] Monotonicity: The points on the path must be aligned from left to right, and data cross-matching is not allowed, otherwise the order will be reversed;

[0145] Due to the monotonicity and continuity of the upstream and downstream water level matching path, the path selection of each grid point can only be carried out in three directions. If the grid point coordinates are (i, j), then the coordinates of the next grid point can only be (i+1, j), (i,.j+1) or (i+1, j+l). The dynamic time difference bending algorithm always selects the path with the smallest cumulative sum in three directions when selecting the matching path. The matching distance matrix of the upstream and downstream water levels is defined as:

[0146]

[0147] Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the smallest neighboring element to reach this point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows:

[0148]

[0149] where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distance reaches the minimum value. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md mn The element starts at the upper left corner of its neighbor md m-1n-1 、The above element md m-1n 、The left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when

[0150] The upstream water level sequence generated by the dynamic time difference bending algorithm is:

[0151]

[0152] The downstream water level sequence is:

[0153]

[0154] 3. Determine the water level delay based on the longest common substring

[0155] Due to the large difference in the change rate of the upstream and downstream data in certain time periods, there are still large differences between the matched data. The present application uses the longest common subsequence algorithm in the matched upstream and downstream water level data to find the upstream and downstream water level data substrings with the highest similarity, and compares the time attributes corresponding to the upstream and downstream water level data of the longest common substring. The mean of the difference in time between the upstream and downstream water level sequences is the time delay of the upstream and downstream water levels.

[0156] The present application searches for the longest common substring with the highest similarity in the matched upstream and downstream water level data, that is, searches for the upstream and downstream water level segments with the closest water level change trends, improves the longest common substring algorithm according to the characteristics of the water level data set, and determines that the difference between the upstream water level data and the downstream water level data is within a certain interval c when judging the common substring, that is, the two are determined to be equal, solves the problem of amplitude expansion and contraction of the upstream and downstream water level data when comparing similarities, selects the water level data near the upstream and downstream peak values, and uses the dynamic time difference bending algorithm to match the upstream and downstream water level data, and then finds the longest common substring with the closest change trend in the matched upstream and downstream water level data values, and then obtains the time delay of the upstream and downstream water levels through the time attribute of the water level data in the longest common substring;

[0157] Assume that the longest common substrings are:

[0158] Upstream water level subsequence: S′ k = <s′1,s′2,s′3…,s′ k >where s′ i =(se i ,sr i );

[0159] Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…,f′ k >where f′ i =(fo i ,fr i )

[0160] The longest common substring algorithm is improved according to the characteristics of the water level data set. The improved algorithm meets the following requirements:

[0161] 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 Formula established;

[0162] 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant;

[0163] If the water level sequence meets the above requirements, the length of the longest common substring of the upstream and downstream water level sequences is k;

[0164] The longest common substring is solved by dynamic programming. First, the common substring matrix DM of the upstream and downstream water level sequences is established. n , first calculate the first row and the first column of the common substring matrix. The value of the first row element is whether the data of the upstream water level is approximately equal to the data of the first downstream water level. If they are equal, it is 1, otherwise it is 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If they are equal, it is 1, otherwise it is 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If they are equal, the value of the node is the value of the upper left element plus 1, otherwise it is 0. The calculation formula of the common substring matrix element dm(i,j) is as follows:

[0165]

[0166] As long as the absolute value of the difference between the upstream water level and the downstream water level is not higher than c, the two are considered equal in theory, and the data corresponding to the longest common substring is the water level data with the closest water level change trend.

[0167] The longest common subsequence in the upstream and downstream water level sequences is a segment of data with the most similar change trends in the upstream and downstream water level data, that is, the upstream and downstream water level data sequences corresponding to this period of time have similar changes. The upstream and downstream water level time delay is calculated based on the time attributes of this segment of water level time series data.

[0168] Using the longest common substring of the upstream and downstream water level data, calculate the time difference of the corresponding moments according to the time attribute of the common substring water level data, and then average the obtained time differences:

[0169]

[0170] The longest common substring of the upstream and downstream water level change trends is a time series with the same length, and the upstream and downstream substring pairs are continuous in time. The upstream and downstream water level delay deltatime is the mean of the longest common substring time difference.

[0171] (IV) Upstream water level data selection

[0172] The upstream and downstream water level data are used to detect abnormal points. When detecting abnormal points in the downstream water level data, it is necessary to calculate the time corresponding to the upstream water level according to the upstream and downstream water level delay according to the downstream water level collection time, and then calculate the upstream water level value at that time;

[0173] The initial upstream and downstream water level time series is:

[0174] Upstream water level time series: D n =<d1=(v1,t1),d2=(v2,t2),…,d n-1 =(v n-1 ,t n-1 ),d n =(v n ,t n )>

[0175] Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ),g n =(h n ,u n )>

[0176] The method of calculating the upstream water level based on the downstream water level is as follows: Assume that the upstream time set is T = {t1, t2, L, t n-1 , t n}, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i -deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 ,t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT i Calculate as follows:

[0177]

[0178] The upstream water level time series considering the upstream and downstream water level delay problem:

[0179] UD n = <ud1,ud2,ud3…,ud n-1 ,ud n > where ud i =(z i ,nu i )

[0180] At this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend. The upstream data and the downstream data at the same time are closely related.

[0181] 2. Water Level Anomaly Detection by Support Vector Regression

[0182] In order to solve the abnormal problems in the collection and transmission of water level data, this application proposes a support vector regression water level anomaly point detection that takes into account the influence of time lag. After solving the upstream and downstream water level delay problem, the support vector regression algorithm is used to establish a relationship model between the downstream real-time water level and the upstream water level and the downstream water level at the previous moment. The future water level value is predicted by the established water level model, and the residual between the actual value and the predicted value is analyzed by the probability calculation method to obtain the abnormal residual value. Then, the water level value corresponding to the time at that moment is found through the time corresponding to the abnormal residual value. This water level value is the abnormal data.

[0183] (I) Support Vector Regression Kernel Function

[0184] The upstream water level and the water level at the previous moment that are closely related to the current moment are used to establish a relationship model with the current moment water level. The kernel function is introduced to reduce the dimension of the inner product of the water level and the complexity of the support vector regression algorithm. The relationship model between the upstream water level, the water level at the previous moment and the water level at the current moment is established. The Gaussian kernel function K(x, z) = exp(γ||xz|| 2 ) The parameter γ measures the influence of a single water level sample on the water level relationship model. The parameter γ is set according to the characteristics of the water level data set. The parameter γ defines the influence of a single sample on the regression function. When the parameter γ is small, the influence of a single sample on the regression function is large, and the support regression vector fits this sample by increasing the complexity of the regression function. Conversely, when γ is large, the influence of a single data on the regression function is small, and the complexity of the regression function is also high. In the Gaussian kernel function, the value of the parameter γ is 1 / the number of sample features.

[0185] (II) Establishing a regression model and prediction of water level related factors

[0186] The corresponding water level upstream and the water level at the previous moment of the point are used to establish a relationship model with the water level at the current moment of the point. The historical water level data is used for learning, and then the real-time water level data is predicted. The abnormal points are found by comparing the residuals of the water level data using the probability distribution method.

[0187] Support vector regression establishes a linear regression model so that as many data set samples as possible are located on the regression model, that is, the residual is as small as possible. When calculating the loss in support vector regression, it is determined that data deviations within a certain range are acceptable and are not included in the loss. The loss distance constant ε>0 is defined. Support vector regression introduces an insensitive loss function to implement a strategy of not calculating losses within a certain error range.

[0188] On the basis of the insensitive function, the objective function of the support vector regression problem is transformed into the objective function structural risk. The former is a measure of the complexity of the water level regression function, and the latter is the empirical risk used to describe the prediction effect between the prediction function and the actual value. The prediction error is defined by the combined results of the two items. According to the principle of structural risk minimization, the objective function expects that the complexity of the water level relationship model is low, and that the model is as small as possible for misclassification. The penalty factor C is the coordination of the relationship between the two. When C is large, the abnormal points are modeled as normal points. When C is small, the abnormal points will be excluded from the normal points and the slack variables will be used to penalize them.

[0189] Two relaxation factors are introduced. By introducing relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the introduction of relaxation factors, the distance between the actual value and the regression model is converted to be less than or equal to the loss distance constant ε after adding the relaxation variables.

[0190] When establishing the regression model, the loss is calculated for the values ​​whose errors are not within the range of the loss distance constant ε. When modeling the water level relationship, the loss needs to be as small as possible, but the small loss comes at the cost of increasing the complexity of the regression model. At this time, the penalty factor C is used to adjust the prediction effect of the water level relationship model and the complexity of the model, and the Lagrange multiplier is introduced. The Lagrange multiplier method can convert the objective function into a Lagrangian function.

[0191] Figure 3 A flow chart of the real-time water level relationship model established by the support vector regression algorithm using the upstream water level and the water level at the previous moment.

[0192] When establishing the water level relationship model, the standard deviation, mean square error and r-square of the actual water level value and the predicted water level value in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the better the water level relationship model conforms to the changing trend of the actual water level, indicating that the model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0,1] interval. When r_square is approximately close to 1, it means that the function fits better, that is, the water level relationship model is suitable for the water level data set. If the evaluation index of the water level relationship model is poor, it is necessary to readjust the parameters in the support vector regression until the evaluation index meets the actual requirements.

[0193] Support vector regression uses an insensitive loss function to improve the tolerance of the regression function to outliers, which is conducive to the regression function learning more about the trend of the data, rather than focusing more on the precise fitting of individual data. Support vector regression chooses to abandon the fitting and directly calculate the loss for data in the data set that deviates greatly from other sample points. The introduction of the insensitive loss function is conducive to the detection of water level anomalies. At the same time, the kernel function in support vector regression realizes the complex inner product calculation problem when linear inseparable data is mapped to high dimensions, avoiding the emergence of data dimensionality disaster and simplifying the complexity of calculation. The regression model established by support vector regression has good generalization ability, can learn complex water level relationship models, and better fit the changing trend of water level data.

[0194] 3. Outlier Detection Based on Residual

[0195] The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level detection data are obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation selects the unknown parameters based on the maximum probability of the sample occurring when it is used as the estimated value. Suppose the residual sequence is: Where: τ i is the absolute value of the difference between the actual value and the predicted value;

[0196] The abnormal points of water level data are detected by using the characteristics that the residual conforms to the normal distribution. The mean and variance of the residual are calculated based on the maximum likelihood estimation. The data set It conforms to the normal distribution The distribution probability density function is:

[0197]

[0198] Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is:

[0199]

[0200] Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative:

[0201]

[0202] According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is:

[0203]

[0204] The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that cannot occur under normal circumstances, that is, the residual data is considered to be an outlier.

[0205] Before solving the normal residual range, the normal range of the standard normal distribution is first calculated at the significance level, and then the normal interval of the water level residual is deduced based on the relationship between the general normal distribution and the standard normal distribution. The probability distribution density function of the standard (0,1) normal distribution is And the distribution function φ(x) is:

[0206]

[0207] Formula 13 is the probability distribution density function;

[0208]

[0209] Formula 13 is the distribution function;

[0210] According to the properties of standard normal distribution: φ(-x) = 1-φ(x);

[0211] Normal distribution is Its probability density function is:

[0212]

[0213] The data set η follows a (0,1) normal distribution, that is,

[0214]

[0215]

[0216] Under a given significance level α', find the maximum value of η;

[0217]

[0218] In the standard normal distribution, P(η>za )=a,0<a<1that is, P(n<z a )=1-a;

[0219] a and Z a The corresponding relationship is obtained by querying the formula of the normal distribution table, and the corresponding quantile z is obtained by querying the normal distribution table and using α' α' The values ​​are:

[0220]

[0221] The normal value of the residual in the water level relationship model is (0, τ max ), if the data falls outside the normal range, it is considered that the data is a low-probability event that will not occur under normal circumstances, and the residual water level data at that moment is considered abnormal data.

[0222] 3. Experimental plan and results analysis

[0223] The scheme proposed in this application is analyzed, experiments are conducted on real data sets and simulated data sets, and the experimental results are analyzed.

[0224] (I) Experimental plan analysis

[0225] This application fully analyzes the characteristics of water level data when detecting water level anomalies and finds that there is a close relationship between the upstream and downstream water levels. However, due to the certain distance between the upstream and downstream collection sites and the particularity of water resources, although the upstream and downstream water level data have similar change trends, there is a time delay problem. The anomaly point detection scheme proposed in this application first matches the upstream and downstream water level data using a dynamic time difference bending algorithm, and then uses the longest common substring method to find the water level segments with the closest change trends, and uses the time attributes of the water level data to calculate the delay.

[0226] The support vector regression method is suitable for detecting anomalies in water level data. Since the time intervals for collecting water level data by the liquid level meter are inconsistent and the time density of collected data is uneven, it is reasonable to allow a certain deviation between the model's predicted value and the actual value when performing data modeling under such irregular water level data. The established water level relationship model pays more attention to the changing trend of water level data, and will not cause blurring of the changing trend due to overfitting to ensure the accuracy of individual data prediction, thereby affecting the detection of water level anomalies. Therefore, the support vector regression method is suitable for the detection of water level anomalies in this application.

[0227] (II) Experimental results of abnormal water level points

[0228] The present application proposes a time-series water level anomaly detection scheme based on support vector regression. By utilizing the relationship between the upstream water level, the downstream water level at the previous moment and the downstream water level at the current moment, and taking into account the time delay between water level data, the present application first uses the longest common substring to find the water level sequence with the closest change trend, calculates the time delay based on the time difference between the corresponding upstream and downstream water levels, establishes a water level relationship model by learning from historical water level data using the support vector regression algorithm, and uses the established water level relationship model to predict future water level data, and finds water level anomalies by analyzing the residuals between the predicted values ​​and the actual values.

[0229] 1. Data preprocessing experiment

[0230] Since the collection frequency of upstream and downstream water level data is different, it is necessary to standardize the water level data into water level time series data with a time interval of 30 minutes. The downstream part of the data before water level interval processing is Figure 4 .

[0231] Figure 5 Each point in the figure represents a collection point. The points in the figure are arranged from left to right in the order of collection time. The X-axis represents the collection time of the original downstream water level data, and the Y-axis represents the collected downstream water level value.

[0232] The downstream water level is processed into a standardized data graph. From the comparison graph of the water level data, it can be seen that the water level data processed at half-hour intervals maintains the changing characteristics of the water level at the original collection point, and the processed water level data is smoother, which is more valuable and meaningful in the subsequent analysis and processing of water level data.

[0233] Since the distribution range of the water level values ​​in the upstream and downstream is quite different, it is necessary to normalize the upstream and downstream water level data. The upstream and downstream water level data before normalization are: Figure 6 .

[0234] The water level change rate in the upstream is different. Similar change trends are completed in a shorter time in the upstream, while the downstream data takes a longer time. This application uses a section of water level data with a similar change trend to match the upstream and downstream water levels using a dynamic time difference bending algorithm. The water level data segment before matching is Figure 7 .

[0235] The dynamic time difference warping algorithm first uses the distance matrix of the upstream and downstream water levels to calculate the element values ​​of the first row and the first column of the similarity matrix, and then uses the dynamic programming algorithm to calculate the values ​​of the other row and column elements in the similarity matrix. The element value of a grid point is the sum of the minimum value of the element value of the distance matrix corresponding to the grid point and the similarity of the upper left grid point, the left grid point, and the upper grid point. After the upstream and downstream water level data are matched using the dynamic time difference warping algorithm, the water level data are aligned according to the same change trend. The upstream and downstream water level data matching diagram is shown in the figure below. Figure 8 .

[0236] Figure 8 Each point in the figure represents a collection point. The points in the figure are arranged from left to right in chronological order, with a time interval of half an hour. The X-axis represents the water level time of the corresponding point after matching, and the Y-axis represents the normalized upstream and downstream water level values ​​after matching. The green line is the downstream water level, and the red line is the upstream water level. The point on the left side of the peak in the figure has a horizontal line of upstream water level because the upstream data changes faster, while the downstream water level changes slower under the same trend. When matching, one point corresponds to multiple points. In the improved longest common substring, if the difference between the upstream and downstream water levels is less than ζ, the two are considered to be equal value pairs. According to the characteristic constant ζ=0.005 of the upstream and downstream water level data, the corresponding upstream and downstream water level values ​​are obtained using the matching sequence generated above. According to the distance of the upstream and downstream water level data, for the water point with a smaller water level distance, the two are considered to be equal. The value of the longest common substring matrix is ​​the element value of its upper left corner plus 1, otherwise the value of the point is 0. The longest common substring finds the water level sequence with the most continuous matching data. Find the largest data in the common substring matching matrix. This data is the length of the longest common substring of the upstream and downstream water levels. The horizontal and vertical coordinates corresponding to this data are the positions at which the matching string ends. The beginning of the upstream and downstream substrings is obtained based on the length of the common substring and the end position. The specific position information is as follows: the longest matching length is 43, the starting position of the downstream water level substring is 85, and the starting position of the upstream water level substring is 85. The time of the upstream and downstream water levels and the corresponding water level values ​​are found through the position of the matching substring of the upstream water level in the water level sequence.

[0237] According to the time attribute of the longest matching water level substring corresponding to the upstream and downstream water levels, the time difference of the upstream and downstream water levels is calculated, and the average value of all time differences in the matching substring is calculated, which is the upstream and downstream water level delay. The delay of the upstream and downstream water levels calculated by actual data is: 1 day, 15 hours, 23 minutes and 23 seconds, and the final water level delay result is approximately 1 day, 15 hours and 30 minutes.

[0238] 2. Water level data abnormal point experiment

[0239] This application uses the upstream water level and the water level at the previous moment as input, and uses the support vector regression algorithm to establish a relationship model with the water level at the current moment. When using support vector regression to establish a relationship model, it is necessary to establish the form of the kernel function and the value of the penalty factor C, the kernel function parameter γ, and the loss tolerance distance ε, and then use the established water level relationship model to analyze the residual between the predicted value and the actual value. Through the analysis of the residual, the water level data that does not conform to the change trend in the actual water level is found, and this water level data is the abnormal point.

[0240] According to the obtained upstream and downstream water level time delay, the upstream and downstream water level data are aligned according to the water level time delay, and then the water level model is learned using the first 3000 normal water level data, and the remaining 900 water level data with noise are used for prediction. In the learning process, the learning model is evaluated by indicators such as variance and standard deviation in the prediction model. When the learning effect is not good, the parameters C, γ, and ε in the support vector regression algorithm are adjusted to make it consistent with the change characteristics of the current water level data set.

[0241] The outlier detection scheme proposed in this application requires the establishment of a water level relationship model. Since the water level data is complex and changeable, the water level data set does not belong to a linear relationship data set. Therefore, a nonlinear kernel function is used. In support vector regression, the nonlinear kernel functions include Sigmoid kernel function, polynomial kernel function, and Gaussian kernel function.

[0242] The water level relationship model established on the water level data set using the Sigmoid kernel function with the initial data found that the predicted value of the water level data was very different from the actual value and the effect was very poor, so the Sigmoid kernel function could not be used. The predicted value of the polynomial kernel function for the water level data was not at the same order of magnitude as the actual value. The water level data did not conform to the functional characteristics of the polynomial kernel function, and the modeling efficiency of the polynomial kernel function was low when the degree was high. Therefore, the polynomial kernel function was not suitable for the water level data set.

[0243] Using the Gaussian kernel function in the support vector regression algorithm can effectively model the water level relationship. The actual water level value is basically consistent with the predicted value and the prediction error can be controlled within a small range. The water level model established by the Gaussian kernel function conforms to the characteristics of the water level data set. Therefore, this application uses the Gaussian kernel function to model the water level relationship.

[0244] After determining that the kernel function used in this application is a Gaussian kernel function, the important parameters in the support vector regression are tuned. In the support vector regression algorithm, the parameter y=1 / sample feature number is used in the experiment. Two factors are selected in this application, namely the water level at the upstream moment and the water level at the previous moment of the downstream station. Therefore, the value of γ is 0.5, and the penalty factor C and the loss distance ε are optimized. First, the modeling effect is compared using y=0.5 and y=1.

[0245] Support vector regression From the water level data, it can be seen that when γ = 0.5, the prediction effect is better than that when γ = 1. Moreover, by comparing the evaluation indicators of the water level relationship model, it can be seen that γ = 0.5 has a significantly better effect.

[0246] Finally, this application is compared with the hydrological time series anomaly detection scheme based on sliding window prediction. The water level anomaly detection based on sliding window is to predict the water level value by using the sliding window idea, and then analyze the residual to get the anomaly point. The experiment compares the anomaly detection accuracy of the two schemes, as shown in the comparison chart. Fig. 9 .

[0247] The number of samples in the water level data set is 900. It can be seen that when the deviation value of the anomaly point is increased to between (0.05-0.07), the accuracy of the detection of the two schemes is improved. At the same time, the increase in the number of anomalies leads to a decrease in the accuracy of the detection of the two schemes, but the accuracy of the scheme proposed in this application is still higher than that of the sliding window scheme. This scheme proposes a scheme to detect the abnormal points of the downstream water level based on the upstream water level data and the historical water level. It has a good detection effect when the downstream water level has only one upstream confluence point and the number of abnormal points is small (less than 10%). When the number of abnormal points is large, the detection effect is weakened. However, in actual situations, the number of abnormal water level points in a reservoir generally does not exceed 10%, because in this case the water conservancy staff will deal with it. Therefore, the scheme proposed in this application is of great significance in the actual detection of abnormal water level points.

Claims

1. A method for detecting abnormal points of water level delay in water conservancy big data time series, characterized in that: Based on the characteristics of hydrological data, improvements are made in two processes: water level data preprocessing and abnormal point detection of downstream water levels: A- Establish a water level data preprocessing method including upstream and downstream water level delay: A1 constructs a preprocessing method for irregular data. Based on the characteristics of uneven time density and inconsistent variation range of upstream and downstream water level data, the upstream and downstream water level data are processed at uniform time intervals to improve data availability and reduce data redundancy. Normalization is used to convert the absolute value of the upstream and downstream water level data into a relative value relationship. A2 uses the dynamic time difference bending algorithm to compress and stretch the water level data on the time axis based on the similarity of the upstream and downstream water level change trends and the different change rates, so that the upstream and downstream water level data correspond one by one under the same change trend. The matched upstream and downstream water level data are used to find the upstream and downstream water level segments with the highest similarity using the longest common substring, and the upstream and downstream water level delay is calculated according to the time attribute of the longest common substring digital data; A3 uses the upstream and downstream water level delay to find the upstream collection time corresponding to the downstream water level collection time, and uses the upstream water level data to calculate the corresponding water level data value at that time to obtain the upstream and downstream water level data at the same time; B- Establish a water level anomaly detection method based on support vector regression: Based on the correlation between the water level value at a certain moment of the measuring station and the water level value at the previous moment, and the similar change trend between the water level upstream and downstream of the station, these two water level factors are used to assist the downstream water level anomaly detection. A water level anomaly detection method is established on the basis of solving the upstream and downstream water level delay: B1 analyzes the characteristics of water level data, selects upstream water level data related to the station water level data and water level data at the previous moment as auxiliary factors for real-time water level anomaly detection, and analyzes the kernel function and its parameters in support vector regression; B2 uses support vector machine regression to establish a water level relationship model between the upstream water level and the site water level at the previous moment and the site real-time water level, and verifies the evaluation index of the effect of the established water level model; B3 predicts the water level value based on the established water level relationship model. By analyzing the residuals between the predicted water level value and the measured value, the normal range is obtained based on the distribution characteristics of the residuals to find the abnormal residual values, and then the water level anomaly points are obtained based on the time attributes of the residual sequence.

2. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Water level data preprocessing: According to the similarity of water level changes, find the section with the most similar water level change trend between the upstream and downstream, and then calculate the upstream and downstream water level delay. Before measuring the similarity, the upstream and downstream water levels are standardized, and then the similarity can be measured. After finding the upstream and downstream water level delay, the upstream water level data is sorted according to the downstream water level collection time, so as to facilitate the subsequent modeling of the water level relationship; First, the water level data is standardized, and the standardized upstream and downstream water level data are matched using the dynamic time difference bending algorithm. The upstream and downstream substrings with the closest changes in the matched upstream and downstream water level data are found through the longest common substring method in the similarity measurement. The upper and lower water level time delays are calculated according to the time attributes of the water level data. According to the time delay, the corresponding upstream water level data is found according to the collection time of the downstream water level data for water level relationship modeling.

3. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Evenly spaced water level data: make the upstream and downstream water level data have the same time interval, interpolate and supplement the upstream and downstream water level data, measure the similarity of the evenly spaced upstream and downstream water level data at the same time, and then calculate the time delay of the upstream and downstream water levels; Select appropriate water level data time interval: The water level time series data interval selected in this application is Δt. The water level data does not change significantly within Δt, which reduces data redundancy caused by fine-grained time intervals. Assume that the upstream water level dataset is D n = <d1,d2,d3,…,d n-1 , d n >, where d i =(v i , t i ), t i-1 ≤t i , the initial time series set is T = {t1, t2, L, t n-1 , t n }, the downstream water level dataset is G n = <g1,g2,g3…,g n-1 , g n >, where g i =(h i ,u i )、u i-i ≤u i , the upstream and downstream water levels are standardized into water level time series data starting at 0:00 every day with an interval of Δt. The water level data will not change suddenly within Δt. When the time point in the time series is not an integer multiple of the time interval Δt, the two collection points closest to the required time point are selected, and the water level change law during this period is abstracted into a linear change. The regularization process of the upstream water level data is as follows: Select the start time startTime and end time endTime of the upstream and downstream data collection, and generate a time set with an interval of Δt as R = {r1, r2, L, r n-1 , r n }, where r i -r i-1 =Δt, r1>startTime, r n <endTime and r1, r n The calculation of the upstream water level data at time r is an integer multiple of the time interval Δt: First, search for a value greater than or equal to r in the upstream initial data. i The minimum time t i , if r i =t i The upstream water level is r i The water level value v at the moment i 'Equal to the initial water level collection data v upstream i , if r i ≠t i , that is, r i >t i , then there exists r i ∈[t i-1 , t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant growth rate, that is, the water level and time are a linear change law, and the growth rate is Then r i The water level value v corresponding to the time i Calculate as follows: The time series data of the upstream water level data after standardization is expressed as: D′ n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >where d′ i =(v′ i , r i ) The downstream water level data is processed in the same way. The processed water level data is: G′ n = <g1′,g′2,g′3,…,g′ n-1 , g′ n >where g′ i =(h′ i , r i ) where r i-1 <r i 、r i -r i-1 =Δt, the upstream and downstream water level data processed as above are evenly arranged according to the time interval Δt, and the upstream and downstream water level times correspond one to one, and the water level data with the same time arrangement are more reasonably compared for similarity.

4. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Water level data normalization: After the water level data is processed with uniform intervals, the upstream and downstream water level data are arranged correspondingly at the same time interval Δt, and the absolute value of the data is converted into an overall relative value. The normalized data ignores the specific value of the data, which is convenient for describing the change trend of the water level data. The linear normalization method is used to normalize the water level data within the range of (0, 1); The data is normalized based on the uniform interval processing of water level. The linear normalization of water level data is to map the standardized water level data to the interval (0, 1). First, the standardized water level time series is traversed to find the maximum value of the water level v max ' and the minimum value v min ', upstream water level series D' n = <d′1,d′2,d′3,…,d′ n-1 , d′ n >where d′ i =(v′ i , r i ), the water level data is normalized as follows: After normalization, the upstream and downstream water level time series data are all in the interval (0, 1), with the same water level measurement standard, and at the same time, basic data are prepared for the similarity comparison of the upstream and downstream water levels.

5. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Determine the peak values ​​of upstream and downstream water levels: The peak values ​​of upstream and downstream water levels are used to assist in calculating the delay, that is, how long does it take for the peak of the upstream water level to be transmitted to the downstream? This transmission time is a rough value of the upstream and downstream water level delay. Then, the similarity of the water level data adjacent to the peak value is measured to accurately solve the upstream and downstream water level delay. Before calculating the upstream and downstream water level delay, the peak values ​​of the upstream and downstream water levels are first determined. First, a moment is randomly selected in the downstream water level time series to be detected, and the moment with the largest peak value is found within the time range of one day after the moment. If there are multiple peak values, the peak with the largest water level value is selected. If there is no peak value within the time period, the trend of the water level change within the time period is judged. If it is a downward trend, the water level peak value is found before the time period. If it is an upward trend, the downstream water level peak value is found after the time period. After finding the downstream water level peak value, the peak value of the upstream water level is found within three days before the downstream peak value. If there are multiple peaks, the peak value of the upstream water level corresponding to the moment with the larger peak value is selected. After obtaining the peak values ​​corresponding to the upstream and downstream water levels, the water level data adjacent to the peak values ​​are measured for similarity to obtain the upstream and downstream water level delays.

6. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Upstream and downstream water level matching based on dynamic time difference bending: Using the calculated upstream and downstream peak values, select the upstream and downstream water level data near the peak value, and use the dynamic time difference bending method to match the upstream and downstream water level data. Suppose the upstream water level time series is UW m = <uw1,uw2,uw3,…,uw m-1 , uw m >Downstream water level time series is DW n = <dw1,dw2,dw3,…,dw n-1 , dw m >, construct a distance matrix, the element value of the matrix is ​​the distance between the upstream and downstream water levels at the corresponding moment, and the difference between the upstream and downstream water levels is its distance; The dynamic time warping algorithm uses dynamic programming to calculate the distance matrix A. m *n, select a line from the upper left corner to the lower right corner. This line is the matching data pair of the upstream and downstream water levels. The points along the line are the corresponding matching points of the upstream and downstream water level data. In the process of matching upstream and downstream water level data, the selected matching path must meet the following requirements: Boundary condition: When selecting a path, only the distance matrix A can be used. m*n The upper left corner of a 11 Start to the lower right corner a mn End, you cannot arbitrarily start matching from the middle data; Continuity: If the upstream water point uwi and the downstream water point dw j If the match is successful, the next upstream matching point uw i+1 Only downstream (dw j-1 .dw j .dw j+1 ) Water point matching is performed, but not across points; Monotonicity: The points on the path must be aligned from left to right, and data cross-matching cannot be performed; The path selection of each grid point can only be carried out in three directions. Let the coordinates of the grid point be (i, j). The coordinates of the next grid point can only be (i+1, j), (i,.j+1) or (i+1, j+l). The dynamic time difference bending algorithm always selects the path with the smallest cumulative sum in three directions when selecting the matching path. The matching distance matrix of the upstream and downstream water levels is defined as: Matching distance md ij is the current distance matrix element a ij The sum of the matching distances to the smallest neighboring element to reach this point. When the next element of the two sequences is directly matched, it is necessary to add twice the distance value of the corresponding element to increase the smoothness of the matching sequence. md ij It is expressed as follows: where a ij is the corresponding element in the distance matrix. When dynamic planning is completed, each element on the best matching path is the matching point corresponding to the upstream and downstream water levels. At the same time, this path is also the path where the upstream and downstream matching distance reaches the minimum value. When calculating the best matching pair of upstream and downstream water levels, the matching matrix md mn The element starts at the upper left corner of its neighbor md m-1n-1 、The above element md m-1n , the left element md mn-1 Select the previous matching node, the median of the three is equal to md ij -a ij The value of is the previous matching node. When backtracking to md 11 The search for matching pairs is completed when The upstream water level sequence generated by the dynamic time difference bending algorithm is: in The downstream water level sequence is: in 7. According to claim 1, the method for detecting abnormal points of water level delay in water conservancy big data time series is characterized in that: Determine the water level delay based on the longest common substring: Use the longest common subsequence algorithm in the matched upstream and downstream water level data to find the upstream and downstream water level data substring with the highest similarity, compare the time attributes corresponding to the upstream and downstream water level data of the longest common substring, and the mean value of the difference between the upstream and downstream water level sequence time is the time delay of the upstream and downstream water levels; Assume that the longest common substrings are: Upstream water level subsequence: S′ k = <s′ i , s′2,s′3…,s′ k >where s′ i =(se i ,sr i ); Downstream water level subsequence: F′ k = <f′1,f′2,f′3,…,f′ k >where f i ′=(fo i , fr i ) The longest common substring algorithm is improved according to the characteristics of the water level data set. The improved algorithm meets the following requirements: 1) Any 0≤a≤k-1 can make the formula sr a <sr a+1 and fr a <fr a+1 Formula established; 2) Any 0≤a≤k-1 satisfies |se a -so a |≤c where c is a constant; If the water level sequence meets the above requirements, the length of the longest common substring of the upstream and downstream water level sequences is k; The longest common substring is solved by dynamic programming. First, the common substring matrix DM of the upstream and downstream water level sequences is established. n , first calculate the first row and the first column of the common substring matrix. The value of the first row element is whether the data of the upstream water level is approximately equal to the data of the first downstream water level. If they are equal, it is 1, otherwise it is 0. Similarly, the first column is to compare whether the first data of the upstream water level is equal to the downstream water level sequence. If they are equal, it is 1, otherwise it is 0. Then use the dynamic programming method to calculate the values ​​of the remaining rows and columns in the common substring matrix, and compare whether the upstream water level value and the downstream water level value of the node are approximately equal. If they are equal, the value of the node is the value of the upper left element plus 1, if they are not equal, it is 0. The calculation formula of the common substring matrix element dm(i, j) is as follows: As long as the absolute value of the difference between the upstream water level and the downstream water level is not higher than c, the two are deemed to be equal in theory, and the data corresponding to the longest common substring is the water level data with the closest water level change trend.

8. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Upstream water level data selection: Use the upstream and downstream water level data to detect abnormal points. When detecting abnormal points in the downstream water level data, it is necessary to calculate the time corresponding to the upstream water level according to the upstream and downstream water level delay according to the downstream water level collection time, and then calculate the upstream water level value at that time; The initial upstream and downstream water level time series is: Upstream water level time series: D n = <d1=(v1,t1),d2=(v2,t2),…,d n-1 =(v n-1 , t n-1 ), d n =(v n , t n )>Downstream water level time series: G n = <g1=(h1,u1),g2=(h2,u2),…,g n-1 =(h n-1 ,u n-1 ), g n =(h n ,u n )> The method of calculating the upstream water level based on the downstream water level is as follows: Assume that the upstream time set is T = {t1, t2, L, t n-1 , t n }, downstream water level monitoring point g i =(h i ,u i ), calculate the time uT corresponding to the upstream water level = u i -deltaTime, first find the minimum time t greater than or equal to uT in the upstream initial data i , if uT=t i , then the upstream water level at time uT is z i Equal to the initial upstream water level data v i , if uT≠t i , that is, uT>t i , then there exists uT∈[t i-1 , t i ], in a short period of time, it is determined that the water level data is a process of uniform change at a constant speed, that is, the water level and time are a linear change law, then the water level value z corresponding to the time uT i Calculate as follows: The upstream water level time series considering the upstream and downstream water level delay problem: OUT n = <ud1,ud2,ud3…,td n-1 ,out n < out of it i =(z i ,now i ) At this time, the upstream and downstream water level data have the same collection time, and the upstream and downstream water level data collected at the same time have the same change trend. The upstream data and the downstream data at the same time are closely related.

9. According to the method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1, it is characterized in that: Establish a regression model and prediction of water level related factors: Support vector regression establishes a linear regression model so that as many data set samples as possible are located on the regression model, that is, the residual is as small as possible. When calculating the loss in support vector regression, it is determined that data deviations within a certain range are acceptable and are not included in the loss. The loss distance constant ε>0 is defined. Support vector regression introduces an insensitive loss function to implement a strategy of not calculating losses within a certain error range. Two relaxation factors are introduced. By introducing relaxation factors, the support vector regression makes the water level relationship model no longer strictly require the distance between the actual water level value and the function value. After the relaxation factors are introduced, the distance between the actual value and the regression model is converted to be less than or equal to the loss distance constant ε after adding the relaxation variables. When establishing the water level relationship model, the standard deviation, mean square error and r-square of the actual water level value and the predicted water level value in the water level relationship model are used to evaluate the water level relationship model. The closer the standard deviation and mean square error are to 0, the more the water level relationship model conforms to the changing trend of the actual water level, indicating that the model is well established. When calculating r_square, the mean of the actual water level is calculated first. r_square belongs to the data in the [0, 1] interval. When r_square is approximately close to 1, it means that the function fits better, that is, the water level relationship model is suitable for the water level data set. If the evaluation index of the water level relationship model is poor, it is necessary to readjust the parameters in the support vector regression until the evaluation index meets the actual requirements.

10. The method for detecting abnormal point of water level delay in water conservancy big data time series according to claim 1 is characterized in that: Residual-based outlier detection: The water level relationship established by the regression model is used to predict the water level value in the future. The abnormal points of the water level detection data are obtained by analyzing the residuals in the prediction results. The maximum likelihood estimation selects the unknown parameters based on the maximum probability of the sample occurring when it is used as the estimated value. Suppose the residual sequence is: Where: τ i is the absolute value of the difference between the actual value and the predicted value; The abnormal points of water level data are detected by using the characteristics that the residual conforms to the normal distribution. The mean and variance of the residual are calculated based on the maximum likelihood estimation. The data set It conforms to the normal distribution The distribution probability density function is: Use maximum likelihood estimation to solve the mean μ and variance σ 2 , the solution process is as follows: The maximum likelihood function is: Where M is the number of samples, and the mean μ, variance σ 2 Find the partial derivative: According to the above formula, the mean μ and variance σ of the water level residual are obtained. 2 The maximum likelihood estimate of is: Mean: variance: The water level residual data conforms to the normal distribution, and the acceptance domain and rejection domain of the residual data are obtained at a given significance level. If the residual data falls within the acceptance domain interval, it is considered normal data. If it falls within the rejection domain, it is considered that the data is a low-probability event that cannot occur under normal circumstances, that is, the residual data is considered to be an outlier.

Images