Automatic data gross error elimination method for dam safety monitoring

Abstract

The invention discloses a dam safety monitoring automation data gross error elimination method, and belongs to the technical field of dam safety monitoring automation. The method comprises the following steps: collecting dam safety monitoring automation data, obtaining an observation value, and storing the observation value in a database; eliminating gross errors of dam safety monitoring automation data by adopting a three-time mean error method to obtain X data; performing curve fitting of the X data by adopting a Fourier four-order number to obtain a fitting observation value set; taking the fitting observation value set as a truth value of automatic detection data, replacing an observation value average number in a three-time median error method, obtaining a difference value between the observation value and the fitting observation value, calculating a median error, judging the observation value of which the difference value between the observation value and the fitting observation value exceeds a threshold value as a gross error, and removing the gross error again, therefore, the data gross error elimination efficiency and accuracy are improved, and the situation that gross error elimination is not in place is avoided.

Description

Technical Field

[0001] This invention belongs to the field of dam safety monitoring automation technology, and specifically relates to a method for eliminating gross errors in dam safety monitoring automation data. Background Technology

[0002] Safety monitoring is a crucial means of understanding the operational status of a dam during its operational phase, and dam safety monitoring data is the direct source of data for assessing the dam's health status. Therefore, the validity of dam safety monitoring data must be guaranteed. With the continuous development of automated dam safety monitoring, automated monitoring systems are gradually replacing traditional manual observation systems due to their advantages such as high observation frequency, real-time performance, high precision and accuracy, cost-effectiveness, and stability and reliability. However, automated monitoring system data is also affected by factors such as human error, external environment, and the instrument itself, leading to some monitoring data that significantly deviate from the overall trend. These data that deviate significantly from reasonable measurements are called gross errors.

[0003] Commonly used methods for identifying gross errors in monitoring data include three-times mean square error (MSE), manual interpretation, envelope method, and statistical discrimination. However, these methods often require manually setting thresholds and building complex models, resulting in low efficiency or accuracy when dealing with large volumes of monitoring data. Three-times mean square error (MSE) is based on the probability theory of normal data distribution to determine whether data contains gross errors. However, for dam safety monitoring data, which exhibits interannual cyclical variations due to external factors such as temperature and water level, the MSE method cannot effectively remove jumps within the normal fluctuation range, thus limiting its practical application. It can only eliminate gross errors that significantly deviate from the normal data. Manual interpretation is inefficient and relies heavily on the operator's expertise, making it difficult to meet the needs of detecting gross errors in massive automated monitoring data. Envelope method and statistical discrimination require building complex models, necessitating different parameter settings and model adjustments for different data sequences, also limiting their practical application. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned problems by providing a method for eliminating gross errors in automated dam safety monitoring data. This method aims to improve the efficiency and accuracy limitations of existing methods in monitoring dam safety data, thereby enhancing the precision and efficiency of eliminating gross errors in automated dam safety monitoring data.

[0005] The technical solution adopted in this invention is as follows: A method for eliminating gross errors in automated dam safety monitoring data, the method comprising the following steps: Step S100: Collect automated data for dam safety monitoring, obtain real-time observation values, and store them in the database; Step S200: Use the 3x error method to remove gross errors from the automated dam safety monitoring data to obtain X data; Step S300: Use the fourth-order Fourier transform to perform curve fitting on the X data to obtain a set of fitted observations; Step S400: Use the fitted set of observations as the true value of the automated detection data, replacing the average of the observations in the 3x standard error method, to obtain the difference between the observed value and the fitted observation value, and calculate the standard error. Set a threshold, and determine the observations whose difference between the observed value and the fitted observation value exceeds the threshold as gross errors and remove them again to obtain automated dam safety monitoring data without gross errors.

[0006] Furthermore, step S200 includes the following steps: Step S201: Calculate the mean of the observed values :

[0007] in, For the observed values, The observation time for monitoring; Step S202: Calculate the difference between the observed value and its mean. :

[0008] Step S203: Calculate the mean error :

[0009] Step S204: Difference With mean error By comparing and eliminating outliers, we obtain X data that does not contain outliers.

[0010] Furthermore, step S300 includes the following steps: Step S301: Fourier Series The formula is as follows:

[0011]

[0012]

[0013]

[0014]

[0015] in, For a period of time, ; Step S302: Calculate the period of the Fourier transform data The X data sequence is subjected to a Discrete Fast Fourier Transform (DFT). Then, the first half of the transformed data is taken, and the square of the modulus of its complex number is calculated. The extreme values ​​of the square of the modulus are then used as the fitting period for the corresponding data set. :

[0016] Step S303: Perform curve fitting using the fourth-order Fourier transform:

[0017] Substitute the formula parameters from step S301 into step S303; Step S304: Substitute the X data sequence into the formula of step S303 to obtain the set of fitted observations. , .

[0018] Furthermore, step S400 includes the following steps: Step S401: Fit the set of observations As the true value of automated monitoring data, it replaces the average of the observed values ​​in step S200. The difference between the observed values ​​and the fitted observed values ​​is obtained. :

[0019] in, These are the observed values; Step S402: Calculate the mean error :

[0020] Step S403: Difference With mean error By comparing and eliminating gross errors again, we obtain automated dam safety monitoring data free of gross errors.

[0021] Furthermore, the parameters in step S303 The calculation is done by... Substitute the formula from step S301 into the calculation.

[0022] Furthermore, the integral in step S300 is in the form of discrete integral.

[0023] Furthermore, the dam safety monitoring automation data is first processed by removing outliers using the 3x standard error method to obtain X data; then, the standard error obtained by the 3x standard error method is used to further remove outliers, resulting in dam safety monitoring automation data free of outliers.

[0024] Furthermore, the dam safety monitoring automation data obtained in step S403 without gross errors is tested to determine whether it contains gross errors. If it does not contain gross errors, the data is used as the final data; if it does contain gross errors, steps S200-S400 are repeated to remove them.

[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention adopts a method for removing gross errors in automated dam safety monitoring data based on Fourier series fitting. By combining the Fourier series curve fitting capability with the 3 times mean square error theory, gross errors in automated dam safety monitoring data are removed twice, thereby improving the efficiency and accuracy of gross error removal and avoiding incomplete removal of gross errors. 2. This invention utilizes Fourier series to fit the overall trend of monitoring data sequences, enabling more accurate capture of the inherent changing patterns and periodic characteristics of the data. This results in fitted observations that more closely approximate the true trend of the data. It effectively avoids interference from individual outliers on the mean, allowing subsequent calculations of differences to more accurately reflect the deviation of individual observations from the overall trend. Based on this, it calculates the standard error and identifies gross errors. Compared to the traditional 3x standard error gross error removal method that relies solely on the mean of observations as a reference, this invention significantly improves the accuracy and reliability of gross error identification, reducing misjudgments and omissions. Particularly in fields with extremely high data quality requirements, such as dam safety monitoring, it can quickly and accurately remove gross errors, ensuring the validity of monitoring data and providing more robust data support for dam safety assessment and early warning decisions, thereby improving the overall efficiency and level of dam safety management. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a distribution map of the original observation values ​​of this invention; Figure 3 This is a distribution diagram of the observations after removing outliers in this invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings.

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] Commonly used methods for identifying gross errors in monitoring data require manual setting of thresholds and the establishment of complex models, and they suffer from low efficiency or low accuracy when dealing with large amounts of monitoring data.

[0030] This invention utilizes Fourier series to fit the overall trend of monitoring data sequences, enabling more accurate capture of the inherent patterns and periodic characteristics of the data. This results in fitted observations that more closely approximate the true trend of the data. It effectively avoids interference from individual outliers on the mean, ensuring that subsequent difference calculations more accurately reflect the deviation of individual observations from the overall trend. Based on this, the standard error is then recalculated. This method performs gross error identification, significantly improving the accuracy and reliability of gross error identification compared to traditional three-fold error gross error removal methods that rely solely on the average of observed values ​​as a reference benchmark. It also reduces false positives and false negatives. Particularly important for fields with extremely high data quality requirements, such as dam safety monitoring, this method can quickly and accurately remove gross errors, ensuring the validity of monitoring data and providing stronger data support for dam safety assessments and early warning decisions. Ultimately, this enhances the overall efficiency and level of dam safety management.

[0031] like Figures 1-3 As shown, this invention calculates the theoretical values ​​of automated dam safety monitoring data by using Fourier fourth-order number fitting curves, and eliminates gross errors by combining Fourier fourth-order number fitting of automated dam safety monitoring data with the 3x mean error theory, ensuring the accuracy and efficiency of gross error elimination.

[0032] A method for eliminating gross errors in automated dam safety monitoring data includes the following steps: Step S100: Collect automated data for dam safety monitoring, obtain real-time observation values, and store them in the database; The existing data acquisition equipment is used to collect automated data for dam safety monitoring, and the real-time observation values ​​are recorded and temporarily stored in the database for easy storage and retrieval at any time. Step S200: Use the 3x error method to remove gross errors from the automated dam safety monitoring data to obtain X data; Step S201: Extract real-time observations from the database and calculate the average of the observations. :

[0033] in, For the observed values, The observation time for monitoring; Step S202: Calculate the difference between the observed value and the mean of the observed values. :

[0034] Step S203: Calculate the mean error :

[0035] Step S204: Difference With mean error Compare, set if If the observation is identified as a gross error, it will be discarded to obtain X data without gross errors.

[0036] Step S300: Use the fourth-order Fourier transform to perform curve fitting on the X data to obtain a set of fitted observations; Curve fitting was performed using the fourth-order Fourier series.

[0037] Fourier series The formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] in, For a period of time, ;parameter The calculation is done by... Substitute the values ​​into the formula above to perform the calculation.

[0043] Calculate the period of Fourier transform data The X data sequence is subjected to a Discrete Fast Fourier Transform (DFT). Then, the first half of the transformed data is taken, and the square of the modulus of its complex number is calculated. The extreme values ​​of the square of the modulus are then used as the fitting period for the corresponding data set. :

[0044] Substitute the X data sequence obtained in step S200 into the Fourier fourth-order curve fitting formula in step S300 to obtain the set of fitted observations. , .

[0045] Step S400: Use the fitted set of observations as the true value of the automated detection data, replacing the average of the observations in the 3x standard error method, to obtain the difference between the observed value and the fitted observation value, and calculate the standard error. Set a threshold, and determine the observations whose difference between the observed value and the fitted observation value exceeds the threshold as gross errors and remove them again to obtain automated dam safety monitoring data without gross errors.

[0046] Step S401: Fit the set of observations As the true value of automated monitoring data, it replaces the average of the observed values ​​in step S200. The difference between the observed values ​​and the fitted observed values ​​is obtained. :

[0047] in, These are the observed values; Step S402: Calculate the mean error :

[0048] Step S403: Difference With mean error Compare, if | |>3 If the observed value is identified as a gross error, it will be removed to obtain dam safety monitoring automation data free of gross errors; thus, the gross error removal of dam safety monitoring automation data is completed.

[0049] like Figures 2-3 As shown, Figure 2 This is a distribution map of the original observations. Figure 3 The graph shows the distribution of observations after curve fitting and outlier removal using the 3x error method and Fourier fourth-order calculus. The graph shows that the distribution of observations after outlier removal is more uniform, resulting in higher accuracy of the observed values.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for eliminating gross errors in automated dam safety monitoring data, characterized in that, The method includes the following steps: Step S100: Collect automated data for dam safety monitoring, obtain real-time observation values, and store them in the database; Step S200: Use the 3x error method to remove gross errors from the automated dam safety monitoring data to obtain X data; Step S300: Use the fourth-order Fourier transform to perform curve fitting on the X data to obtain a set of fitted observations; Step S400: Use the fitted set of observations as the true value of the automated detection data, replacing the average of the observations in the 3x standard error method, to obtain the difference between the observed value and the fitted observation value, and calculate the standard error. Set a threshold, and determine the observations whose difference between the observed value and the fitted observation value exceeds the threshold as gross errors and remove them again to obtain automated dam safety monitoring data without gross errors.

2. The method for eliminating gross errors in automated dam safety monitoring data according to claim 1, characterized in that, Step S200 includes the following steps: Step S201: Calculate the mean of the observed values : in, For the observed values, The observation time for monitoring; Step S202: Calculate the difference between the observed value and its mean. : Step S203: Calculate the mean error : Step S204: Difference With mean error By comparing and eliminating outliers, we obtain X data that does not contain outliers.

3. The method for eliminating gross errors in automated dam safety monitoring data according to claim 2, characterized in that, Step S300 includes the following steps: Step S301: Fourier Series The formula is as follows: in, For a period of time, ; Step S302: Calculate the period of the Fourier transform data The X data sequence is subjected to a Discrete Fast Fourier Transform (DFT). Then, the first half of the transformed data is taken, and the square of the modulus of its complex number is calculated. The extreme values ​​of the square of the modulus are then used as the fitting period for the corresponding data set. : Step S303: Perform curve fitting using the fourth-order Fourier transform: Substitute the formula parameters from step S301 into step S303; Step S304: Substitute the X data sequence into the formula of step S303 to obtain the set of fitted observations. , .

4. The method for eliminating gross errors in automated dam safety monitoring data according to claim 3, characterized in that, Step S400 includes the following steps: Step S401: Fit the set of observations As the true value of automated monitoring data, it replaces the average of the observed values ​​in step S200. The difference between the observed values ​​and the fitted observed values ​​is obtained. : in, These are the observed values; Step S402: Calculate the mean error : Step S403: Difference With mean error By comparing and eliminating gross errors again, we obtain automated dam safety monitoring data free of gross errors.

5. The method for eliminating gross errors in automated dam safety monitoring data according to claim 3, characterized in that, Parameters in step S303 The calculation is done by... Substitute the formula from step S301 into the calculation.

6. The method for eliminating gross errors in automated dam safety monitoring data according to claim 3, characterized in that, The integral in step S300 is in discrete integral form.

7. The method for eliminating gross errors in automated dam safety monitoring data according to claim 1, characterized in that, The dam safety monitoring automation data is first processed by removing outliers using the 3x standard error method to obtain X data; then, the standard error obtained by fitting Fourier fourth-order order data with the 3x standard error method is used to remove outliers a second time, resulting in dam safety monitoring automation data free of outliers.

8. The method for eliminating gross errors in automated dam safety monitoring data according to claim 4, characterized in that, The final automated dam safety monitoring data obtained in step S403 without gross errors is tested to determine whether it contains gross errors. If it does not contain gross errors, the data is used as the final data; if it does contain gross errors, steps S200-S400 are repeated to remove them.

Images