A method for identifying gross error data based on morphological characteristics of time series process line and algorithm automatic matching

Abstract

The application belongs to the field of water conservancy and hydropower engineering, and particularly relates to a coarse error data identification method based on time sequence process line shape feature and algorithm automatic matching, comprising: calculating and extracting the basic index parameters of a target safety monitoring data set, establishing a quantitative calculation method for the correlation of the environmental quantity and the effect quantity of dam monitoring data, and determining the monitoring data having strong correlation with the target monitoring data; establishing a set of existing safety monitoring data coarse error processing algorithms; determining the optimal algorithm order by comparing the error size between the characteristic parameters of the target data and the strongly correlated monitoring data, selecting the best algorithm as the coarse error calculation algorithm of the dam safety monitoring data; and finally, optimizing the parameters of the optimal algorithm by minimizing the error percentage. The method can comprehensively, accurately and effectively realize the coarse error identification of dam monitoring data, and achieve the goal of intelligent coarse error identification.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy and hydropower engineering, and specifically relates to a method for identifying gross error data based on automatic matching of time series process line morphological characteristics and algorithms. Background Technology

[0002] Prototype dam observation is an effective means of obtaining the true working state of a dam structure, and safety monitoring data is an important reference for determining the safety status of a dam. Due to factors such as the instantaneous working state of monitoring instruments and random measurement errors, dam safety monitoring data often contains a certain amount of gross errors. Improper processing of these errors will have a significant impact on the final structural safety status analysis results. Therefore, accurate identification of gross errors is the premise and foundation for safety monitoring data analysis and dam safety status evaluation, and is of great significance for setting dam safety monitoring indicators.

[0003] The purpose of gross error identification in monitoring data is to identify and eliminate erroneous data, while retaining data that truly reflects the safety of the dam structure, thus avoiding misjudgments and omissions of dam safety anomalies due to data-level issues. To date, gross error identification techniques based on various mathematical analysis methods have been widely applied, including process line methods (extreme value judgment, rate judgment, quartile judgment, etc.), statistical testing methods (Laida criterion, Chauville criterion, T-test, etc.), regression analysis, and spatial distribution methods. However, problems remain, such as insufficient algorithm universality, low identification efficiency, and low precision and recall, due to the large number of gross error categories and the complexity of their causes. This leads to significant human resource consumption for data verification and analysis. With the development of information technology, artificial intelligence-based gross error identification technology has also accelerated its research and application. It shows outstanding advantages in processing large amounts of multidimensional and nonlinear data sequences. However, it also has problems such as high requirements for the number and quality of samples needed for neural network training, a large amount of preliminary work, and a lack of mechanistic understanding for identifying gross errors from different causes. In summary, to achieve comprehensive, accurate, effective and reliable gross error identification of monitoring data, it is necessary to further consider the applicable scenarios of each algorithm, and autonomously match one or more optimal algorithms based on data characteristics to meet various indicators such as improving the computational efficiency, precision and recall of gross error identification. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a gross error data identification method based on the morphological characteristics of time series process lines and automatic matching of algorithms, which can comprehensively, accurately and effectively realize the gross error identification of dam monitoring data, and achieve the goal of high-efficiency and intelligent identification of gross errors.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for identifying gross error data based on the morphological characteristics of time series process lines and automatic matching with algorithms, comprising the following steps: S1. Obtain relevant monitoring data, calculate and extract basic index parameters from the target safety monitoring data set, and establish a quantitative calculation method for the correlation between environmental quantities and effect quantities of dam monitoring data to determine monitoring data that are strongly correlated with the target monitoring data. S2. Establish a set of existing algorithms for processing gross errors in safety monitoring data, and define the applicable scope and relevant parameters for each algorithm in a structured manner. S3. By comparing the error magnitude between the characteristic parameters of the target data and the strongly correlated monitoring data, the optimal algorithm ranking is determined, and the algorithm with the best ranking is selected as the gross error calculation algorithm for the dam safety monitoring data. S4. Based on the actual matching calculation results and the manual verification results, and based on the characteristic parameters of the strongly correlated monitoring data, the optimal algorithm selected in step S3 is optimized to determine the optimal parameters of the algorithm.

[0006] In the preferred embodiment, step S1 specifically includes the following steps: S1.1 Obtain relevant monitoring data, draw corresponding time series process lines based on the monitoring data, divide the monitoring intervals according to the monitoring duration, and calculate the curvature of the time series process line and the range of extreme value changes for each interval of each monitoring data. The specific calculation formula is as follows: (1) In the formula, This represents the curvature of the corresponding curve. y These are the actual monitoring values ​​from the dam monitoring data. (2) In the formula, rang The range of extreme value changes. To be the minimum value of the monitoring data, This represents the maximum value of the monitored data. S1.2. Based on association rules, establish the degree of correlation between various monitoring quantities of the dam, specifically defined as follows: (3) In the formula, For the event and events The confidence level, when If the value is greater than 0.5, then the event is considered... and events Related; Indicates an event The probability of occurrence; Based on the existing association rules, a formula for calculating the correlation degree of dam monitoring time series is defined. If dam monitoring quantity A and monitoring quantity B exist in the time series, then...n Time association rules that satisfy minimum support , i =1,2… n Then, based on the time series correlation between dam monitoring data A and B... and confidence level The formula is as follows: (4) (5) In the formula, the degree of correlation This reflects the degree of correlation between dam monitoring data A and B over time; the greater the correlation, the higher the confidence level. The confidence level reflects the reliability of the time correlation rule between monitoring quantities A and B of the dam. The higher the confidence level, the more reliable the correlation rule between monitoring quantities A and B in the time series. S1.3. Use the correlation degree determined in step S1.2 to determine whether there is a correlation between the various monitoring quantities of the dam.

[0007] In the preferred embodiment, the specific criteria for determining whether there is a correlation between the various monitoring quantities of the dam in step S1.3 are as follows: When the correlation between monitoring quantities A and B exceeds 0.5, and the monitoring data value of a certain point of monitoring quantity A is identified as an outlier, the water level monitoring values ​​of the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are correlated, and the outlier value identified by the horizontal displacement data is indeed an anomaly.

[0008] In the preferred embodiment, step S2 specifically includes the following steps: S2.1 Statistical analysis of existing dam monitoring data for identifying gross errors; S2.2. Use the existing gross error identification method for monitoring data of various dams as statistically analyzed in step S2.1 to identify gross errors, and mark the initially identified gross errors.

[0009] In a preferred embodiment, step S3 includes the following: S3.1. Remove the gross errors identified by each algorithm and define the range of gross errors as follows:

[0010] In the formula, Indicates the range of gross error values. The time corresponding to the gross error values ​​are respectively The monitoring data values ​​corresponding to the previous and next time nodes; the characteristic parameters of the monitoring data in each gross difference interval are calculated, including the curvature calculation of the time series process and the range of extreme value changes in the interval. At the same time, the characteristic parameters of the gross difference intervals corresponding to the monitoring data that are strongly correlated with the monitoring data are calculated. S3.2 Compare the feature parameters obtained from the gross error intervals of each algorithm with the feature parameters of the gross error intervals of the strongly correlated monitoring data, and compare them by percentage. Take the average value as the final error percentage of each algorithm, sort the algorithms according to their degree of similarity, and determine the algorithm with the highest ranking as the optimal algorithm.

[0011] In a preferred embodiment, in step S4, the parameters of the determined optimal algorithm are fine-tuned, and the error percentage between the feature parameters corresponding to the coarse difference interval and the feature parameters of the coarse difference interval of the strongly correlated monitoring data is compared. The optimal parameters of the algorithm are determined by minimizing the error percentage, and the coarse difference of the dam monitoring data is identified using these optimal parameters.

[0012] The present invention provides a method for identifying gross error data based on automatic matching of time series process line morphological features and algorithms, which has the following beneficial effects: 1. Step S1 of the present invention analyzes the curvature and extreme value fluctuation range of the time-series curve of the monitoring data, and at the same time analyzes the correlation between the monitoring data. Compared with other coarse difference identification methods, this step considers more comprehensive factors and takes into account the correlation between the data.

[0013] 2. In step S2 of the present invention, multiple gross difference identification methods are compared and the best result is adaptively selected, avoiding the shortcomings of traditional single identification algorithms that are only effective for a certain monitoring data, making the identification of gross differences in dam monitoring data more scientific.

[0014] 3. In step S3 of the present invention, regarding the method of comparing the accuracy of gross error values, this method can determine the error percentage of the characteristic parameter of the gross error value error interval with strongly correlated monitoring data. This not only considers the characteristic parameters of the time series curve of the monitoring data itself, but also refers to the characteristic parameters of other strongly correlated monitoring data, making the result more accurate.

[0015] 4. Step S4 of the present invention optimizes the optimal method by minimizing the percentage of characteristic parameter error in the gross difference error interval of strongly correlated monitoring data, making the gross difference algorithm more accurate and the results more reliable. Attached Figure Description

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1This is a flowchart illustrating the technical process of the present invention; Figure 2 This is a time-series diagram of the monitoring data in the embodiment; Figure 3 This is the result of linear fitting of water level and horizontal displacement data in the example; Figure 4 This is a coarse difference identification diagram from an example. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0019] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0020] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "and / or B" can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. Embodiments may be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Combination Figures 1-4 The specific embodiments of the present invention will be described in further detail below.

[0023] A method for identifying gross errors based on the morphological features of time-series processes and automatic matching with algorithms includes the following steps: S1. Obtain various monitoring data of the reservoir and calculate and extract basic index parameters from the target safety monitoring data set.

[0024] Specifically, it includes the following: S1.1 Obtain relevant monitoring data, draw corresponding time series process lines based on the monitoring data, and divide the monitoring intervals according to the monitoring duration. Monitoring data measured hourly is divided into daily intervals, monitoring data measured daily is divided into monthly intervals, and so on. Calculate the curvature of the time series process line and the range of extreme values ​​for each interval of each monitoring data. The specific calculation formula is as follows: (1) In the formula, y represents the curvature of the corresponding curve, and y represents the actual monitoring value of the dam.

[0025] (2) In the formula, rang The range of extreme value changes. To be the minimum value of the monitoring data, This represents the maximum value of the monitored data.

[0026] S1.2. Based on association rules, establish the degree of correlation between various monitoring quantities of the dam, specifically defined as follows: (3) In the formula, For the event and events The confidence level, when If the value is greater than 0.5, then the event is considered... and events Related; Indicates an event The probability of occurrence.

[0027] Based on the existing association rules, a formula for calculating the correlation degree of dam monitoring time series is defined. If dam monitoring quantity A and monitoring quantity B exist in the time series, then... n Time association rules that satisfy minimum support , i =1,2… n Then, based on the time series correlation between dam monitoring data A and B... and confidence level The formula is as follows: (4) (5) In the formula, the degree of correlation This reflects the degree of correlation between dam monitoring data A and B over time; the greater the correlation, the higher the confidence level. The confidence level reflects the reliability of the time correlation rule between monitoring quantities A and B of the dam. The higher the confidence level, the more reliable the correlation rule between monitoring quantities A and B in the time series. When the correlation and confidence levels of the two monitoring quantities of the dam meet the requirements, it is considered that the two monitoring quantities have a certain influence relationship.

[0028] S1.3. Using the correlation degree determined in step S1.2, determine whether there is a correlation between the various monitoring quantities of the dam. That is, the two major dam monitoring quantities are respectively the independent variable and the dependent variable, and there is a certain mathematical relationship between them. The gross error values ​​of the subsequent dam monitoring data can be further judged by the relationship between the two.

[0029] The specific judgment criteria are as follows: When the correlation between monitoring quantities A and B exceeds 0.5, and the monitoring data value of a certain point of monitoring quantity A is identified as an outlier, the water level monitoring values ​​of the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are correlated, and the outlier value identified by the horizontal displacement data is indeed an anomaly.

[0030] Taking horizontal displacement monitoring data as an example, the time series process line of this data is plotted as follows: Figure 2 As shown, the curvature and extreme value range of the time series process line are calculated according to the above formula. Simultaneously, the linear fitting results of the horizontal displacement monitoring data and the water level monitoring data are calculated as follows: Figure 3 As shown in Table 1, the correlation between water level monitoring data and horizontal displacement monitoring data was calculated using association rules. Table 1 shows the time-series correlation degree between the dam water level and horizontal displacement data as 0.8293 with a confidence level of 0.8846, which meets the minimum threshold requirement. Therefore, it is considered that there is a strong correlation between the dam water level and the horizontal displacement data EX4.

[0031]

[0032] Depend on Figure 2 Calculations show that the correlation between the reservoir's water level and horizontal displacement both exceed 0.5, indicating a significant influence relationship.

[0033] S2. Establish a set of existing algorithms for processing gross errors in safety monitoring data, and define the applicable scope and relevant parameters for each algorithm in a structured manner.

[0034] S2.1 Statistical analysis of existing dam monitoring data gross error identification methods, as shown in Table 2.

[0035]

[0036] S2.2 Select a monitoring dataset with normal monitoring data and randomly add noise. Use the existing gross difference identification method for monitoring data of various dams in step S2.1 to identify the gross difference.

[0037] S3. By comparing the error magnitudes between the characteristic parameters of the target data and the strongly correlated monitoring data, the optimal algorithm ranking is determined, and the algorithm with the best ranking is selected as the gross error calculation algorithm for this dam safety monitoring data, including the following: S3.1, Determine multiple corresponding gross error value intervals, and simultaneously determine the characteristic parameters of the error value intervals, through... Figure 2 It can be seen that the extreme value range of the reservoir fluctuates to a certain extent every month, and the corresponding monthly fluctuations are also unstable every year. Therefore, the characteristic parameter is mainly curvature. The error percentages corresponding to each algorithm are shown in Table 3 after calculation.

[0038]

[0039] S3.2 Rank the error percentages of each algorithm and select the density-based clustering algorithm as the best algorithm.

[0040] S4. The density-based clustering algorithm was optimized by minimizing the error percentage. Ultimately, a neighborhood radius of 1.05 was determined to be the optimal parameter, corresponding to the lowest error percentage. Using this optimal parameter as the parameter for the density-based clustering algorithm, coarse error identification was performed on the data. Figure 4 As shown.

[0041] Depend on Figure 4 It can be seen that the gross error values ​​of the horizontal displacement monitoring point correspond to July 17, 2020, July 18, 2020, and July 19, 2020. This matches the actual values, proving the feasibility of the present invention.

[0042] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying gross error data based on automatic matching of time-series process line morphological features and algorithms, characterized in that, Includes the following steps: S1. Obtain relevant monitoring data, calculate and extract basic index parameters from the target safety monitoring data set, and establish a quantitative calculation method for the correlation between environmental quantities and effect quantities of dam monitoring data to determine monitoring data that are strongly correlated with the target monitoring data. The specific steps include the following: S1.1 Obtain relevant monitoring data, draw corresponding time series process lines based on the monitoring data, divide the monitoring intervals according to the monitoring duration, and calculate the curvature of the time series process line and the range of extreme value changes for each interval of each monitoring data. The specific calculation formula is as follows: （1）； In the formula, This represents the curvature of the corresponding curve. y These are the actual monitoring values ​​from the dam monitoring data. （2）； In the formula, rang The range of extreme value changes. To be the minimum value of the monitoring data, This represents the maximum value of the monitored data. S1.

2. Based on association rules, establish the degree of correlation between various monitoring quantities of the dam, specifically defined as follows: （3）； In the formula, For the event and events The confidence level, when If the value is greater than 0.5, then the event is considered to be... and events Related; Indicates an event The probability of occurrence; Based on the existing association rules, a formula for calculating the correlation degree of dam monitoring time series is defined. If dam monitoring quantity A and monitoring quantity B exist in the time series, then... n Time association rules that satisfy minimum support , i =1,2… n Then, based on the time series correlation between dam monitoring data A and B... and confidence level The formula is as follows: （4）； （5） In the formula, the degree of correlation This reflects the degree of correlation between dam monitoring data A and B over time; the greater the correlation, the higher the confidence level. The confidence level reflects the reliability of the time correlation rule between monitoring quantities A and B of the dam. The higher the confidence level, the more reliable the correlation rule between monitoring quantities A and B in the time series. S1.

3. Use the correlation degree determined in step S1.2 to determine whether there is a correlation between the various monitoring quantities of the dam; S2. Establish a set of existing algorithms for processing gross errors in safety monitoring data, and define the applicable scope and relevant parameters for each algorithm in a structured manner. S3. By comparing the error magnitude between the characteristic parameters of the target data and the strongly correlated monitoring data, the optimal algorithm ranking is determined, and the algorithm with the best ranking is selected as the gross error calculation algorithm for the dam safety monitoring data. S4. Based on the actual matching calculation results and the manual verification results, and based on the characteristic parameters of the strongly correlated monitoring data, the optimal algorithm selected in step S3 is optimized to determine the optimal parameters of the algorithm.

2. The method for identifying gross error data based on automatic matching of time-series process line morphological features and algorithms as described in claim 1, characterized in that, In step S1.3, the specific criteria for determining whether there is a correlation between the various monitoring quantities of the dam are as follows: When the correlation between monitoring quantity A and B exceeds 0.5, when the monitoring data value of a certain point of monitoring quantity A is identified as an outlier, the monitoring value of monitoring quantity B in the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are related, and the outlier value identified by monitoring quantity A is indeed an anomaly.

3. The method for identifying gross error data based on automatic matching of time-series process line morphological features and algorithms as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S2.1 Statistical analysis of existing dam monitoring data for identifying gross errors; S2.

2. Use the existing gross error identification method for monitoring data of various dams as statistically analyzed in step S2.1 to identify gross errors, and mark the initially identified gross errors.

4. The method for identifying gross error data based on automatic matching of time-series process line morphological features and algorithms as described in claim 1, characterized in that, Step S3 includes the following: S3.

1. Remove the gross errors identified by each algorithm and define the range of gross errors as follows: In the formula, Indicates the range of gross error values. The time corresponding to the gross error values ​​are respectively The monitoring data values ​​corresponding to the previous and next time nodes; the characteristic parameters of the monitoring data in each gross difference interval are calculated, including the curvature calculation of the time series process and the range of extreme value changes in the interval. At the same time, the characteristic parameters of the gross difference intervals corresponding to the monitoring data that are strongly correlated with the monitoring data are calculated. S3.2 Compare the feature parameters obtained from the gross error intervals of each algorithm with the feature parameters of the gross error intervals of the strongly correlated monitoring data, and compare them by percentage. Take the average value as the final error percentage of each algorithm, sort the algorithms according to their similarity, and determine the algorithm with the highest ranking as the optimal algorithm.

5. The method for identifying gross error data based on automatic matching of time-series process line morphological features and algorithms as described in claim 1, characterized in that, In step S4, the parameters of the determined optimal algorithm are fine-tuned. At the same time, the error percentage between the feature parameters corresponding to the coarse difference interval and the feature parameters of the coarse difference interval of the strongly correlated monitoring data is compared. The optimal parameters of the algorithm are determined by minimizing the error percentage, and the coarse difference of the dam monitoring data is identified using these optimal parameters.

Images