Coarse error data identification method based on automatic matching of sequential process line morphological characteristics and algorithm

Abstract

The invention belongs to the field of water conservancy and hydropower engineering, and particularly relates to a gross error data identification method based on automatic matching of sequential process line morphological characteristics and algorithms, which comprises the following steps of: performing calculation and extraction of basic index parameters on a target safety monitoring data set, and establishing a quantitative calculation method of relevance between environment quantity and effect quantity of dam monitoring data; determining monitoring data having strong relevance with the target monitoring data; establishing an existing safety monitoring data gross error processing algorithm set; determining an optimal algorithm sequence by comparing errors between the characteristic parameters of the target data and the strong correlation monitoring data, and selecting an algorithm with the optimal sequence as a gross error calculation algorithm of the dam safety monitoring data; and finally, parameter optimization is carried out on the optimal algorithm by minimizing the error percentage. According to the method, gross error identification of dam monitoring data can be comprehensively, accurately and effectively realized, and the target of intelligent gross error identification is achieved.

Description

Technical Field

[0001] The invention belongs to the field of water conservancy and hydropower engineering, and in particular relates to a gross error data recognition method based on automatic matching of temporal process line morphological features and algorithms. Background Art

[0002] Dam prototype observation is an effective means to obtain the real working state of the dam structure, and safety monitoring data is an important reference for determining the safety status of the dam. Due to factors such as the instantaneous working state of the monitoring instrument and random measurement errors, the dam safety monitoring data often contains certain gross errors (referred to as gross errors). Improper processing will have a greater impact on the final structural safety status analysis results. Therefore, accurate identification of gross errors is the premise and basis for safety monitoring data analysis and dam safety status evaluation, and is of great significance to the setting of dam safety monitoring indicators.

[0003] The purpose of gross error identification of monitoring data is to identify and eliminate erroneous data, and retain data that truly reflects the safety of dam structures, so as to avoid misjudgment and omission of abnormal dam safety status due to data-level problems. So far, gross error identification technology based on a variety of mathematical analysis methods has been widely used, including process line method (extreme value judgment, rate judgment, quartile judgment, etc.), statistical test method (Laida criterion, Chauville criterion, T test, etc.), regression analysis method, spatial distribution method, etc., but there are still problems such as insufficient algorithm universality, low recognition efficiency, low precision and recall rate indicators due to factors such as many gross error categories and complex causes, which will consume more human resources for data review and analysis. With the development of information technology, the research and development and application of gross error identification technology based on artificial intelligence have also been accelerated. It has shown outstanding advantages in processing large, multi-dimensional, and nonlinear data sequences, but it also has problems such as high sample quantity and quality requirements for neural network training, huge preliminary workload, and lack of mechanism for dealing with gross error identification due to different causes. In summary, to achieve comprehensive, accurate, effective and reliable identification of gross errors in monitoring data, it is necessary to further comprehensively consider the applicable scenarios of each algorithm, and autonomously match the best one or several algorithms according to the data characteristics to meet the needs of improving multiple indicators such as gross error identification computational efficiency, precision and recall rate. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a gross error data identification method based on automatic matching of time series process line morphological characteristics and algorithms, which can comprehensively, accurately and effectively realize the gross error identification of dam monitoring data and achieve the goal of high-efficiency and intelligent gross error identification.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms, comprising the following steps:

[0006] S1. Obtain relevant monitoring data, calculate and extract basic indicator parameters for the target safety monitoring data set, and establish a quantitative calculation method for the correlation between the environmental quantity and effect quantity of the dam monitoring data to determine the monitoring data that has a strong correlation with the target monitoring data;

[0007] S2. Establish a set of existing safety monitoring data gross error processing algorithms, structurally mark the applicable scope of each algorithm and define applicable correlation parameters;

[0008] S3. Determine the optimal algorithm ranking by comparing the error sizes between the characteristic parameters of the target data and the strongly correlated monitoring data, and select the best ranked algorithm as the gross error calculation algorithm for this dam safety monitoring data;

[0009] S4. According to the actual matching calculation results and the manual verification results, and according to the characteristic parameters of the strongly correlated monitoring data, the optimal algorithm selected in step S3 is tuned to determine the optimal parameters of the algorithm.

[0010] In a preferred solution, in step S1, the specific steps include the following:

[0011] S1.1. Obtain relevant monitoring data, draw the corresponding time series process line based on the monitoring data, divide the corresponding monitoring intervals according to the monitoring duration, calculate the curvature of the time series process line of each interval of each monitoring data and the range of interval extreme value changes. The specific calculation formula is as follows:

[0012]

[0013] In the formula, ρ is the curvature of the corresponding curve, and y is the specific monitoring value of the actual dam monitoring data;

[0014] rang=[y min ,y max ]; (7)

[0015] In the formula, rang is the range of extreme value variation, y min is the minimum value of the monitoring data, y max is the maximum value of the monitoring data;

[0016] S1.2. Based on the association rules, the correlation degree between the monitoring quantities of the dam is established. The specific definitions are as follows:

[0017]

[0018] Where P c (α→β) is the confidence of event α and event β. When P cWhen (α→β) is greater than 0.5, it is considered that event α is associated with event β; P α represents the probability of event α;

[0019] On the basis of the original association rules, the relevant calculation formula of the dam monitoring time series association is defined. If the dam monitoring quantity A and the monitoring quantity B are in the time series, there are n time association rules α that meet the minimum support i →β i , i = 1, 2…n, then the correlation degree P of dam monitoring quantities A and B based on time series ts and confidence level P tc The formula is as follows:

[0020]

[0021] In the formula, the correlation degree P ts (A→B) reflects the degree of correlation between the dam monitoring data A and B in the time series. The greater the correlation, the higher the confidence level P. tc (A→B) reflects the credibility of the temporal association rule of the dam monitoring quantities A and B. The higher the confidence, the more credible the association rule of the monitoring quantities A and B in the time series.

[0022] S1.3. Use the correlation determined in step S1.2 to determine whether there is a correlation between the various monitoring quantities of the dam.

[0023] In the preferred solution, in step S1.3, the specific criteria for determining whether there is a correlation between the various monitoring quantities of the dam are:

[0024] When the correlation between monitoring quantities A and B exceeds 0.5, when the monitoring data value of a certain point of monitoring quantity A is identified as a gross error value, the water level monitoring value of the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are associated, and the gross error value identified in the horizontal displacement data is considered to be an abnormal point.

[0025] In a preferred solution, in step S2, the specific steps include the following:

[0026] S2.1. Method for identifying gross errors in existing dam monitoring data;

[0027] S2.2, using the existing gross error value identification method of each dam monitoring data counted in step S2.1 to identify the gross error value, and mark the gross error value initially identified.

[0028] In a preferred solution, step S3 includes the following:

[0029] S3.1. Eliminate the gross error values ​​identified by each algorithm and define the gross error value interval as follows:

[0030] rang 粗 =[d t-1 ,d t+1 ]

[0031] In the formula, rang 粗差值 Indicates the gross error interval, d t-1 ,d t+1 are the time d corresponding to the gross error value t The monitoring data values ​​corresponding to the previous time node and the next time node are calculated; characteristic parameters are calculated for the monitoring data of each gross error value interval, including the curvature calculation of the time series process and the range of interval extreme value changes, and characteristic parameters are calculated for the gross error value interval corresponding to the monitoring data that has a strong correlation with the monitoring data;

[0032] S3.2. Compare the characteristic parameters obtained from the gross error value intervals corresponding to each algorithm with the characteristic parameters from the gross error value intervals corresponding to the monitoring data with strong correlation, and compare them by percentage. Take the average value as the final error percentage of each algorithm, sort the algorithms according to the degree of proximity, and determine the algorithm with the highest ranking as the optimal algorithm.

[0033] In the preferred scheme, in step S4, the parameters of the determined optimal algorithm are fine-tuned, and the error percentage of the characteristic parameters corresponding to the gross error value interval and the gross error value interval characteristic parameters of the monitoring data with strong correlation are compared. The optimal parameters of the algorithm are determined by minimizing the error percentage, and the gross error values ​​of the dam monitoring data are identified using the optimal parameters.

[0034] The present invention provides a method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms, which has the following beneficial effects:

[0035] 1. Step S1 of the present invention analyzes the curvature and extreme value fluctuation range of the monitoring data time series curve, and analyzes the correlation between the monitoring data. Compared with other gross error identification methods, this step considers more comprehensive factors and takes into account the correlation between the data.

[0036] 2. In step S2 of the present invention, multiple gross error value identification methods are compared, and the best result is adaptively selected, thereby avoiding the disadvantage that the traditional single identification algorithm is only effective for a certain monitoring data, making the identification of gross error values ​​of dam monitoring data more scientific.

[0037] 3. In the method of comparing the accuracy of gross error values ​​in step S3 of the present invention, the method can be determined by the characteristic parameter error percentage of the gross error interval of the monitoring data with strong correlation. In this way, not only the characteristic parameters of the timing curve of the monitoring data itself are considered, but also the characteristic parameters of other strongly correlated monitoring data are referred to, so that the result is more accurate.

[0038] 4. Step S4 of the present invention optimizes the parameters of the optimal method by minimizing the characteristic parameter error percentage of the gross error value error interval of the strongly correlated monitoring data, so that the algorithm of the gross error value is more accurate and the result is more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0040] Figure 1 It is a diagram illustrating the technical process of the present invention;

[0041] Figure 2 It is a timing diagram of monitoring data in the embodiment;

[0042] Figure 3 The linear fitting result of the water level and horizontal displacement data in the embodiment;

[0043] Figure 4 This is a diagram for identifying gross error values ​​in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution will be clearly and completely described below in combination with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0046] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0047] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0048] The technical solution of the present invention is described in detail with specific embodiments below. The embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0049] Combination Figure 1 to Figure 4 The specific embodiments of the present invention are described in further detail.

[0050] A method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms comprises the following steps:

[0051] S1. Obtain various monitoring data of the reservoir, and calculate and extract basic indicator parameters of the target safety monitoring data set.

[0052] The details include:

[0053] S1.1. Obtain relevant monitoring data, draw the corresponding time series process line based on the monitoring data, and divide the corresponding monitoring intervals according to the monitoring duration. The monitoring data measured every hour is divided into one day as one interval, and the monitoring data measured every day is divided into one month as one interval. And so on. Calculate the curvature of the time series process line and the range of the extreme value of each interval of each monitoring data. The specific calculation formula is as follows:

[0054]

[0055] Where ρ is the curvature of the corresponding curve, and y is the specific monitoring value of the actual dam monitoring data.

[0056] rang=[y min ,y max ] (12)

[0057] In the formula, rang is the range of extreme value variation, y min is the minimum value of the monitoring data, y max It is the maximum value of the monitoring data.

[0058] S1.2. Based on the association rules, the correlation degree between the monitoring quantities of the dam is established. The specific definitions are as follows:

[0059]

[0060] Where P c (α→β) is the confidence of event α and event β. When P c When (α→β) is greater than 0.5, it is considered that event α is associated with event β; P α represents the probability of event α occurring.

[0061] On the basis of the original association rules, the relevant calculation formula of the dam monitoring time series association is defined. If the dam monitoring quantity A and the monitoring quantity B are in the time series, there are n time association rules α that meet the minimum support i →β i , i = 1, 2…n, then the correlation degree P of dam monitoring quantities A and B based on time series ts and confidence level P tc The formula is as follows:

[0062]

[0063] In the formula, the correlation degree P ts (A→B) reflects the degree of correlation between the dam monitoring data A and B in the time series. The greater the correlation, the higher the confidence level P. tc (A→B) reflects the credibility of the temporal association rules of the dam monitoring quantities A and B. The higher the confidence, the more credible the association rules of the monitoring quantities A and B in the time series. When the correlation and confidence of the two dam monitoring quantities meet the requirements, it is considered that the two monitoring quantities have a certain influence relationship.

[0064] S1.3, use the correlation determined in step S1.2 to determine whether there is a correlation between the various monitoring quantities of the dam, that is, the two dam monitoring quantities are used as independent variables and dependent variables respectively, and there is a certain mathematical relationship. The gross error value of the subsequent dam monitoring data can be strengthened by the relationship between the two.

[0065] The specific judgment criteria are:

[0066] When the correlation between monitoring quantities A and B exceeds 0.5, when the monitoring data value of a certain point of monitoring quantity A is identified as a gross error value, the water level monitoring value of the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are associated, and the gross error value identified in the horizontal displacement data is considered to be an abnormal point.

[0067] Taking the horizontal displacement monitoring data as an example, the time series process line of the data is drawn as follows: Figure 2As shown, the curvature and extreme value range of the time series process line are calculated according to the above formula. At the same time, the linear fitting results of the horizontal displacement monitoring data and the water level monitoring data are calculated as follows: Figure 3 As shown, the association rules are introduced to calculate the correlation between the water level monitoring and the horizontal displacement monitoring. The results are shown in Table 1. From Table 2, it can be seen that the time series correlation between the dam water level and the horizontal displacement is 0.8293, and the confidence is 0.8846, which meets the minimum threshold requirement. Therefore, it is considered that there is a strong correlation between the dam water level and the horizontal displacement of the horizontal position data EX4.

[0068] Table 1 Support and confidence of subsequences that meet the threshold

[0069]

[0070] Depend on Figure 2 It is calculated that the correlation degree between the reservoir water level and horizontal displacement exceeds 0.5, indicating an obvious influence relationship.

[0071] S2. Establish a set of existing safety monitoring data gross error processing algorithms, structurally mark the scope of application of each algorithm, and define applicable correlation parameters.

[0072] S2.1. The existing methods for identifying gross errors in dam monitoring data are statistically analyzed, as shown in Table 2.

[0073]

[0075] S2.2. Select a monitoring data set with normal monitoring quantity, randomly add noise points, and use the existing gross error value identification method of each dam monitoring data in step S2.1 to identify the gross error value.

[0076] S3. By comparing the error size between the characteristic parameters of the target data and the strongly correlated monitoring data, the optimal algorithm ranking is determined, and the best algorithm is selected as the gross error calculation algorithm for the dam safety monitoring data, including the following:

[0077] S3.1, determine the corresponding multiple gross error value intervals, and at the same time determine the characteristic parameters of the error value intervals, through Figure 2 It can be seen that the extreme value range of the reservoir fluctuates to a certain extent every month, and the fluctuations in the corresponding months of each year are also unstable. Therefore, the characteristic parameter is mainly based on curvature. After calculation, the error percentages corresponding to each algorithm are shown in Table 3.

[0078] Table 3 Error percentage of each method

[0079]

[0080] S3.2. Sort the error percentage of each algorithm and take the density-based clustering algorithm as the most effective algorithm.

[0081] S4. Optimize the parameters of the density-based clustering algorithm by minimizing the error percentage. Finally, it is determined that the domain radius of the density-based clustering algorithm is 1.05, which is the optimal parameter and the corresponding error percentage is the smallest. This optimal parameter is used as the parameter of the density-based clustering algorithm to identify the gross error value of the data. Figure 4 shown.

[0082] Depend on Figure 4 It can be obtained that the dates corresponding to the gross error values ​​of the horizontal displacement monitoring points are July 17, 2020, July 18, 2020, and July 19, 2020. This is consistent with the actual situation, proving the feasibility of the present invention.

[0083] The above descriptions are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms, characterized in that: The following steps are involved: S1. Obtain relevant monitoring data, calculate and extract basic indicator parameters for the target safety monitoring data set, and establish a quantitative calculation method for the correlation between the environmental quantity and effect quantity of the dam monitoring data to determine the monitoring data that has a strong correlation with the target monitoring data; S2. Establish a set of existing safety monitoring data gross error processing algorithms, structurally mark the applicable scope of each algorithm and define applicable correlation parameters; S3. Determine the optimal algorithm ranking by comparing the error sizes between the characteristic parameters of the target data and the strongly correlated monitoring data, and select the best ranked algorithm as the gross error calculation algorithm for this dam safety monitoring data; S4. According to the actual matching calculation results and the manual verification results, and according to the characteristic parameters of the strongly correlated monitoring data, the optimal algorithm selected in step S3 is tuned to determine the optimal parameters of the algorithm.

2. According to claim 1, a method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms is characterized in that: In step S1, the specific steps include the following: S1.

1. Obtain relevant monitoring data, draw the corresponding time series process line based on the monitoring data, divide the corresponding monitoring intervals according to the monitoring duration, calculate the curvature of the time series process line of each interval of each monitoring data and the range of interval extreme value changes. The specific calculation formula is as follows: In the formula, ρ is the curvature of the corresponding curve, and y is the specific monitoring value of the actual dam monitoring data; rang=[and min ,and max ]; (2) In the formula, rang is the range of extreme value variation, y min is the minimum value of the monitoring data, y max is the maximum value of the monitoring data; S1.

2. Based on the association rules, the correlation degree between the monitoring quantities of the dam is established. The specific definitions are as follows: Where P c (α→β) is the confidence of event α and event β. When P c When (α→β) is greater than 0.5, it is considered that event α is associated with event β; P α represents the probability of event α; On the basis of the original association rules, the relevant calculation formula of the dam monitoring time series association is defined. If the dam monitoring quantity A and the monitoring quantity B are in the time series, there are n time association rules α that meet the minimum support i →β i , i = 1, 2…n, then the correlation degree P of dam monitoring quantities A and B based on time series ts and confidence level P tc The formula is as follows: In the formula, the correlation degree P ts (A→B) reflects the degree of correlation between the dam monitoring data A and B in the time series. The greater the correlation, the higher the confidence level P. tc (A→B) reflects the credibility of the temporal association rule of the dam monitoring quantities A and B. The higher the confidence, the more credible the association rule of the monitoring quantities A and B in the time series. S1.

3. Use the correlation determined in step S1.2 to determine whether there is a correlation between the various monitoring quantities of the dam.

3. The method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms according to claim 3 is characterized in that: In step S1.3, the specific criteria for determining whether there is a correlation between the various monitoring quantities of the dam are: When the correlation between monitoring quantities A and B exceeds 0.5, when the monitoring data value of a certain point of monitoring quantity A is identified as a gross error value, the water level monitoring value of the same period can be compared. If the correlation between the two sets of data in this period does not exceed 0.5, it is considered that the data at this time does not conform to the fact that the two are associated, and the gross error value identified in the horizontal displacement data is considered to be an abnormal point.

4. The method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms according to claim 1 is characterized in that: In step S2, the specific steps include the following: S2.

1. Method for identifying gross errors in existing dam monitoring data; S2.2, using the existing gross error identification method of each dam monitoring data counted in step S2.1 to identify gross errors, and mark the initially identified gross errors.

5. The method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms according to claim 1 is characterized in that: The step S3 includes the following: S3.

1. Eliminate the gross error values ​​identified by each algorithm and define the gross error value interval as follows: rank 粗 =[d t-1 ,d t+1 ] In the formula, rang 粗差值 Indicates the gross error interval, d t-1 ,d t+1 are the time d corresponding to the gross error value t The monitoring data values ​​corresponding to the previous time node and the next time node are calculated; characteristic parameters are calculated for the monitoring data of each gross error value interval, including the curvature calculation of the time series process and the range of interval extreme value changes, and characteristic parameters are calculated for the gross error value interval corresponding to the monitoring data that has a strong correlation with the monitoring data; S3.

2. Compare the characteristic parameters obtained from the gross error value intervals corresponding to each algorithm with the characteristic parameters from the gross error value intervals corresponding to the monitoring data with strong correlation, and compare them by percentage. Take the average value as the final error percentage of each algorithm, sort the algorithms according to the degree of proximity, and determine the algorithm with the highest ranking as the optimal algorithm.

6. The method for identifying gross error data based on automatic matching of temporal process line morphological features and algorithms according to claim 1 is characterized in that: In step S4, the parameters of the determined optimal algorithm are fine-tuned, and the error percentages of the characteristic parameters corresponding to the gross error value interval and the gross error value interval characteristic parameters of the monitoring data with strong correlation are compared. The optimal parameters of the algorithm are determined by minimizing the error percentage, and the gross error values ​​of the dam monitoring data are identified using these optimal parameters.

Images