Arch dam deformation monitoring data preprocessing method and system

Abstract

The invention relates to the technical field of arch dam deformation monitoring data processing, in particular to an arch dam deformation monitoring data preprocessing method and system, and the method comprises the steps: collecting arch dam deformation monitoring data; constructing a track matrix for the original deformation monitoring data sequence, and performing singular value decomposition on the track matrix to obtain a feature value and a feature vector; selecting a principal component according to the contribution rate of the characteristic value, and reconstructing a signal; the reconstructed signals are analyzed, and gross errors are identified and eliminated; dividing the deformation monitoring data without the gross error into a training set and a test set; constructing a neural network model, and training the neural network model by adopting the training set; and filling missing data in the test set by using the trained neural network model. According to the invention, monitoring data gross error identification and monitoring data missing processing are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of arch dam deformation monitoring data processing, and in particular relates to a method and system for preprocessing arch dam deformation monitoring data. Background Art

[0002] Due to the unique structural form and complex operating environment, arch dams exhibit complex, variable, uncertain, and time-varying operational characteristics. Deformation is the most intuitive indicator of changes in arch dam structural properties, and effective deformation monitoring data is crucial for timely understanding of the dam's operational status. However, during arch dam deformation monitoring, instrument problems or human factors can cause gross errors and missing data in the monitoring data. These errors and missing data can interfere with the analysis of the arch dam's deformation state, further affecting the objective evaluation and early warning of the arch dam's deformation state. Therefore, it is necessary to explore and establish methods for handling gross errors and missing data to provide objective monitoring data for subsequent early warning analysis of the arch dam's deformation state.

[0003] Traditional methods for identifying gross errors in dam deformation monitoring data primarily include process line methods and statistical tests. While the process line method is simple and intuitive, it relies heavily on the analyst's experience, making it difficult to identify subtle gross errors and only suitable for rough estimations of deformation data. Statistical test methods often assume that deformation data adhere to a certain distribution pattern when identifying gross errors, making them inapplicable for small sample sizes. Regarding missing data, traditional single-value interpolation methods, such as linear interpolation and cubic spline interpolation, often require constructing an idealized objective function for the overall deformation trend. However, in practical engineering, it is difficult to accurately represent arch dam deformation changes using such functions. Furthermore, for arch dam deformation monitoring data with multiple missing values, the accuracy of traditional time series-based interpolation methods relies on the availability of environmental variables and the accuracy of the constructed model. Furthermore, spatially proximal point-based interpolation methods estimate missing values ​​using only the neighboring measurement points, failing to fully capture the characteristics of the missing segments. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to address the deficiencies of the existing technology and provide a method and system for preprocessing arch dam deformation monitoring data. The improved singular spectrum analysis is used to identify gross errors in arch dam deformation monitoring data, and to address the problem of missing deformation data sequences caused by missing measurements and gross errors during arch dam deformation monitoring. The LSTM neural network model is comprehensively used to process missing monitoring data.

[0005] Technical solution: The arch dam deformation monitoring data preprocessing method of the present invention comprises the following steps: S1. Collect arch dam deformation monitoring data; S2. Using the improved singular spectrum analysis method to identify gross errors in deformation monitoring data, including the following sub-steps: S201, select the window length L, add the length of L to the front and back ends of the original deformation monitoring data sequence. Pseudo data, construct trajectory matrix; S202, performing singular value decomposition on the trajectory matrix to obtain eigenvalues ​​and eigenvectors; S203, selecting the principal component according to the contribution rate of the eigenvalue and reconstructing the signal; S204, analyzing the reconstructed signal, identifying gross errors and removing them; S3. Using the neural network model, missing data is filled in the deformation monitoring data after eliminating gross errors, including the following sub-steps: S301, dividing the deformation monitoring data after eliminating gross errors into a training set and a test set; S302, constructing a neural network model, and training the neural network model using a training set; S303. Use the trained neural network model to fill in the missing data in the test set.

[0006] To further improve the above technical solution, the S201 implementation process includes: in the original deformation monitoring data sequence The length added at both ends is Pseudo data to form a new deformation sequence ; The implementation process of S202 includes: constructing a new deformation sequence The trajectory matrix , , Trajectory Matrix Perform SSA-PD analysis to obtain the eigenvalues ​​and corresponding eigenvectors. , ,

[0007] Where, is an elementary matrix; The trajectory matrix The left and right eigenvectors of is a singular value; d is the number of eigenvalues ​​greater than 0 in the covariance matrix S; The step S203 includes: selecting the corresponding principal component according to the contribution rate of the obtained eigenvalue to reconstruct the deformation sequence, and obtaining the residual component ; The S20 includes: The criterion is used to identify the gross errors of the modulus extreme points of the residual component sequence, and this step is repeated until all gross errors are screened out.

[0008] Furthermore, the S302 includes: constructing an LSTM neural network model, including determining a network structure and hyperparameter settings; the network structure of the LSTM neural network model includes at least one hidden layer, and each hidden layer includes multiple LSTM units.

[0009] Furthermore, the training set and test set are standardized. The standardized training set is used as the input value of the LSTM neural network model. The LSTM neural network model adds a forget gate inside each neuron , input gate , output gate ; Obtain the state of the hidden layer at the previous moment and the deformation monitoring data at the current moment and input them into the forget gate. Calculate and decide whether the obtained value is adopted, and then the input gate Generate the next neuron state vector and initial output, and finally the output gate Get the final output value.

[0010] Furthermore, the forget gate Read the state of the hidden layer at the previous moment And the deformation monitoring data input at the current moment , calculated using the sigmoid function and assigned to , the calculation formula is as follows:

[0011] Where: represents the forget gate weight, Represents the bias term corresponding to the forget gate, Represents the sigmoid function of the forget gate, The calculated value is between [0,1], where "0" means "completely discarded" and "1" means "completely retained"; The input gate consists of two modules: the sigmoid layer and the tanh layer. The sigmoid layer determines the value to be input, and its calculation formula is:

[0012] Where: represents the input gate weight, Represents the bias term corresponding to the input gate, Represents the sigmoid function of the input gate; The tanh layer determines a new candidate value vector whose new contribution to the neuron state is:

[0013] Where: tanh represents the hyperbolic tangent activation function; The input gate updates the neuron state. Updated to , the calculation formula is:

[0014] Based on the current state of the neuron, the output gate The tanh function gets the final output value ; , ,

[0015] Where: represents the output gate weight, Represents the bias term corresponding to the output gate, Represents the sigmoid function of the output gate.

[0016] Furthermore, the prediction effect is calculated based on the output values ​​of the arch dam deformation monitoring data. By comparing the prediction effects of the LSTM neural network model under different parameters, the optimal LSTM neural network model parameters are selected, and the LSTM neural network model used for missing value processing of the arch dam deformation monitoring data is determined. The test set data is input and the determined LSTM neural network model is used to process the missing values ​​of the arch dam deformation monitoring data.

[0017] The arch dam deformation monitoring data processing system for implementing the above method includes: Data collection module, a module for collecting arch dam deformation monitoring data; Improved singular spectrum analysis module, used to identify and eliminate gross errors in deformation monitoring data; The neural network model is used to fill in missing data of deformation monitoring data after eliminating gross errors.

[0018] Beneficial effect: Compared with the existing technology, the advantages of the present invention are: based on the analysis of the shortcomings of the traditional deformation data gross error identification method, the present invention proposes a gross error identification method for arch dam deformation monitoring data based on improved singular spectrum analysis, uses singular spectrum analysis technology to extract the time series signal and residual high-frequency signal of the arch dam deformation monitoring data, and combines the 3σ criterion to distinguish the high-frequency components, thereby realizing the identification of gross errors.

[0019] Aiming at the problem of missing deformation data caused by missing measurements and elimination of gross errors in the process of arch dam deformation monitoring, a method for processing missing deformation monitoring data of arch dam based on deep learning was proposed using LSTM neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a processing flow chart of gross error identification of deformation monitoring data using the improved singular spectrum analysis method of the present invention; Figure 2 This is a processing flow chart of the deep learning algorithm in the present invention for filling missing data in deformation monitoring data that has eliminated gross errors. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0022] Example 1: Figure 1 The process of identifying gross errors in the arch dam deformation monitoring data using the arch dam deformation monitoring data preprocessing method shown includes the following steps: S1. Collect arch dam deformation monitoring data; S2. Use the improved singular spectrum analysis method to identify gross errors in deformation monitoring data; S3. Use deep learning algorithms to fill in missing data in deformation monitoring data after eliminating gross errors.

[0023] The specific implementation steps of S2 are: S201: In the original deformation monitoring data sequence The length added at both ends is Pseudo data to form a new deformation sequence .

[0024] S202: Assume that the new arch dam deformation monitoring data is a one-dimensional time series, and construct its trajectory matrix using the following formula ,

[0025] Where: is the trajectory matrix, L is the window length, L≤N / 2, N is the number of samples plus 2 , K=N-L+1; matrix The elements in the sub-diagonal direction are equal. SSA depends on the only parameter, the window length L. The choice of L is very important. When the window length is approximately equal to one-third or one-quarter of the length of the monitoring sequence, it is sufficient to capture all the information of the original sequence.

[0026] By Trajectory Matrix Perform SSA-PD analysis to obtain singular values ​​and corresponding eigenvectors; Trajectory Matrix δ The covariance matrix of S forL L Calculate the eigenvalue of the square matrix And the eigenvector U, according to After sorting in descending order, the trajectory matrix δ The singular value decomposition of is: ; ; ;

[0027] Where, is an elementary matrix; The trajectory matrix The left and right eigenvectors of is a singular value; d is the number of eigenvalues ​​greater than 0 in the covariance matrix S.

[0028] S203: Select the corresponding principal component according to the contribution rate of the obtained eigenvalue to reconstruct the deformation sequence and obtain the residual component; in order to obtain the components reconstructed by the singular values ​​of the deformation sequence from the 1st order to the dth order, all singular values ​​except the dth order are set to zero, thereby constructing a new singular value matrix :

[0029] Will get Convert to a new time series of length N, assuming The basic elements are L K-order matrix , , , No. The principal components are converted into a new sequence by diagonal averaging , Right now:

[0030] Then the original deformation monitoring time series is reconstructed into a new time series: ; Where: r is the number of principal components used to reconstruct the time series.

[0031] S204: Combine The criterion is used to identify the gross errors of the modulus extreme points of the residual component sequence, and this step is repeated until all gross errors are screened out.

[0032] During arch dam deformation monitoring, missing deformation monitoring data often occurs due to issues such as poor instrumentation. In some cases, missing key deformation monitoring data reduces the information contained in the original data, easily leading to misjudgment of the arch dam's deformation behavior. Therefore, complete and valid monitoring data is crucial for arch dam deformation status analysis and early warning, so missing data needs to be processed.

[0033] To address the problem of missing deformation data sequences caused by missing measurements during arch dam deformation monitoring and the elimination of monitoring values ​​with gross errors, a deep learning-based method for processing missing values ​​in arch dam deformation monitoring data was adopted, providing a new technical means for processing missing values ​​in arch dam deformation monitoring data. The LSTM neural network adds three gates to each neuron, namely the forget gate. , Input Gate , Output Gate .

[0034] The above S3 is based on the LSTM neural network to process missing values ​​of arch dam deformation monitoring data, as shown in the following example: Figure 2 As shown, the specific implementation steps are as follows: S301: Divide the arch dam deformation monitoring data into a training set and a test set, and perform standardization processing; S302: Use the training set obtained after S301 as the model input value to obtain the deformation monitoring data sequence , the last output and the neuron state vector ; S303: The forget gate determines the information that should be forgotten by the neuron. The forget gate reads the state of the hidden layer at the previous moment. , and the deformation monitoring data input at the current moment , and finally assign the output value to ,

[0035] Where: Indicates the arch dam deformation monitoring data input at the current moment; Indicates the state of the hidden layer at the previous moment; represents weight; represents the corresponding bias term; Represents the sigmoid function of the forget gate.

[0036] Passing the Gate of Oblivion The calculated output value is between [0,1], "0" means "completely discarded" and "1" means "completely retained". After the forget gate determines the value that should be forgotten, the hyperbolic tangent activation function tanh function determines whether its information is adopted by the input gate to generate the next neuron state vector , input gate Get the initial output,

[0037] Where: represents the input gate weight, Represents the bias term corresponding to the input gate, Represents the sigmoid function of the input gate; The new contributions to neuronal states are:

[0038] The two parts of the input gate will update the neuron state. Updated to . Indicates discarding the information that needs to be discarded, and then Add together, the calculation formula is:

[0039] Based on the current state of the neuron, the output gate The tanh function gets the output value , and its calculation formula is: , ,

[0040] Where: represents the output gate weight, Represents the bias term corresponding to the output gate, Represents the sigmoid function of the output gate.

[0041] The LSTM network uses the Adam algorithm for stochastic gradient descent. Stochastic gradient descent maintains a single learning rate for all weight updates, and the learning rate does not change during training. Adam, on the other hand, calculates first- and second-order moment estimates of the gradient to design independent adaptive learning rates for different parameters. While maintaining a single learning rate for each parameter to improve performance on sparse gradients, the Adam algorithm adaptively maintains a learning rate for each parameter based on the mean of the nearest magnitude of the weight gradient, expressed as:

[0042] Where: and are the first-order momentum term and the second-order momentum term respectively; and are the exponential decay rates of the first-order moment estimate and the second-order moment estimate, which are generally taken as 0.9 and 0.999 respectively; and are their respective correction values; Indicates the The weight matrix of the time step; Indicates the The gradient of the time step cost function with respect to the weight matrix; represents the learning rate; In order to avoid the denominator being 0, the value .

[0043] S304: Obtain the predicted values ​​of each arch dam deformation monitoring data through the above steps, calculate the evaluation index of the prediction effect, and select the optimal LSTM model parameters by comparing the prediction accuracy evaluation index under different parameters. Finally, determine the LSTM model used for missing value processing of the arch dam deformation monitoring data. S305: Input the test set data and use the LSTM model determined in Step 4 to process the missing values ​​of the arch dam deformation monitoring data.

[0044] Based on the analysis of the shortcomings of traditional deformation data gross error identification methods, this paper proposes a gross error identification method for arch dam deformation monitoring data based on improved singular spectrum analysis. The singular spectrum analysis technology is used to extract the time series signal and residual high-frequency signal of the arch dam deformation monitoring data, and the high-frequency component is discriminated in combination with the 3σ criterion, thereby realizing the identification of gross errors.

[0045] Aiming at the problem of missing deformation data caused by missing measurements and elimination of gross errors in the process of arch dam deformation monitoring, a method for processing missing deformation monitoring data of arch dam based on deep learning was proposed using LSTM neural network.

[0046] Example 2: This example provides an arch dam deformation monitoring data processing system for implementing the method in Example 1, including: Data collection module, a module for collecting arch dam deformation monitoring data; Improved singular spectrum analysis module, used to identify and eliminate gross errors in deformation monitoring data; Neural network model to fill missing data in deformation monitoring data after eliminating gross errors As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for preprocessing arch dam deformation monitoring data, characterized in that: The following steps are involved: S1. Collect arch dam deformation monitoring data; S2. Using the improved singular spectrum analysis method to identify gross errors in deformation monitoring data, including the following sub-steps: S201, select a window length L, add a window length of L to the front and back ends of the original deformation monitoring data sequence. Pseudo data, construct trajectory matrix; S202, performing singular value decomposition on the trajectory matrix to obtain eigenvalues ​​and eigenvectors; S203, selecting the principal component according to the contribution rate of the eigenvalue and reconstructing the signal; S204, analyzing the reconstructed signal, identifying gross errors and removing them; S3, using the neural network model to fill in the missing data of the deformation monitoring data after eliminating the gross errors, including the following sub-steps: S301, dividing the deformation monitoring data after eliminating gross errors into a training set and a test set; S302, constructing a neural network model, and training the neural network model using a training set; S303: Use the trained neural network model to fill in the missing data in the test set.

2. The arch dam deformation monitoring data preprocessing method according to claim 1 is characterized in that: The implementation process of S201 includes: The length added at both ends is Pseudo data of ; The implementation process of S202 includes: constructing a new deformation sequence The trajectory matrix , , Trajectory Matrix Perform SSA-PD analysis to obtain the eigenvalues ​​and corresponding eigenvectors. In the formula, is an elementary matrix; The trajectory matrix The left and right eigenvectors of is a singular value; d is the number of eigenvalues ​​greater than 0 in the covariance matrix S; The step S203 includes: selecting the corresponding principal component according to the contribution rate of the obtained eigenvalue to reconstruct the deformation sequence, and obtaining the residual component ; The S20 includes: The criterion is used to identify the gross errors of the modulus extreme points of the residual component sequence, and this step is repeated until all gross errors are screened out.

3. The arch dam deformation monitoring data preprocessing method according to claim 1 or 2, characterized in that: The S302 includes: constructing an LSTM neural network model, including determining a network structure and hyperparameter settings; the network structure of the LSTM neural network model includes at least one hidden layer, and each hidden layer includes multiple LSTM units.

4. The arch dam deformation monitoring data preprocessing method according to claim 3 is characterized in that: The training set and the test set are standardized. The standardized training set is used as the input value of the LSTM neural network model. The LSTM neural network model adds a forget gate inside each neuron. , Input Gate , output gate ; Obtain the state of the hidden layer at the previous moment and the deformation monitoring data at the current moment and input them into the forget gate. Calculate and decide whether the obtained value is adopted, and then the input gate Generate the next neuron state vector and initial output, and finally the output gate Get the final output value.

5. The arch dam deformation monitoring data preprocessing method according to claim 4 is characterized in that: Forget Gate Read the state of the hidden layer at the previous moment And the deformation monitoring data input at the current moment , calculated using the sigmoid function and assigned to , the calculation formula is as follows: Where: represents the forget gate weight, represents the bias term corresponding to the forget gate, represents the sigmoid function of the forget gate, The calculated value is between [0,1], "0" means "completely discarded", and "1" means "completely retained"; The input gate includes two modules: the sigmoid layer and the tanh layer. The sigmoid layer determines the value to be input, and its calculation formula is: Where: represents the input gate weight, represents the bias term corresponding to the input gate, Represents the sigmoid function of the input gate; The tanh layer determines a new candidate value vector whose new contribution to the neuron state is: Where: tanh represents the hyperbolic tangent activation function; The input gate updates the neuron state. Updated to , the calculation formula is: Based on the current neuron state, the output gate The tanh function gets the final output value ; Where: represents the output gate weight, represents the bias term corresponding to the output gate, Represents the sigmoid function of the output gate.

6. The arch dam deformation monitoring data preprocessing method according to claim 4 is characterized in that: The prediction effect is calculated based on the output value of the arch dam deformation monitoring data. By comparing the prediction effects of the LSTM neural network model under different parameters, the optimal LSTM neural network model parameters are selected to determine the final LSTM neural network model for processing missing values ​​of the arch dam deformation monitoring data. The test set data is input and the determined LSTM neural network model is used to process the missing values ​​of the arch dam deformation monitoring data.

7. An arch dam deformation monitoring data processing system, used to implement any of the methods described in claims 1 to 6, characterized in that: include: Data collection module, a module for collecting deformation monitoring data of arch dam; Improved singular spectrum analysis module, used to identify and eliminate gross errors in deformation monitoring data; The neural network model is used to fill in the missing data of the deformation monitoring data after eliminating gross errors.

Images