A method for complex collinearity resolution and overfitting suppression modeling in dam safety monitoring

Abstract

The application discloses a complex collinearity elimination and overfitting inhibition modeling method in dam safety monitoring, and belongs to the technical field of dam safety monitoring. The method firstly constructs a basic robust regression model, and diagnoses whether the prediction curve of the model is violently shaken to judge the influence of complex collinearity; if the complex collinearity exists, the method optimizes the model by combining a Tukey double weight function, an iterative weighted least square method and a robust ridge regression of generalized cross-validation. Then, the performance of the model on a test set is diagnosed to judge overfitting; if the overfitting exists, the method carries out feature purification and dimension reduction by using a robust PCA based on a FAST-MCD robust covariance estimation, and rebuilds the model. The application solves the problems of complex collinearity caused by multi-order lag variables and overfitting caused by high-dimensional features in dam monitoring modeling by using a progressive adaptive optimization architecture, significantly improves the stability and generalization ability of the model, and provides a more reliable model basis for dam safety early warning.

Description

Technical Field

[0001] This invention belongs to the field of smart water conservancy emergency monitoring and early warning technology, specifically involving a multicollinearity elimination and overfitting suppression modeling method in dam safety monitoring. Background Technology

[0002] Dam safety monitoring is a core technical aspect of ensuring the long-term stable operation of reservoir dams and preventing engineering safety risks. By continuously monitoring dam displacement, seepage pressure, uplift pressure, stress, and strain over a long period, combined with scientific data analysis and modeling methods, structural anomalies of the dam can be detected in a timely manner. This provides a basis for risk warning and engineering decision-making, and is of great significance for ensuring the safety of reservoir dam projects and protecting the lives and property of people downstream.

[0003] In the field of dam safety monitoring data analysis, the regression model-based method for identifying anomalies in monitoring data is the most widely used technical approach in current engineering practice. The core logic of this method is as follows: A statistical regression model is constructed between dam effects (displacement, seepage pressure, uplift pressure, etc.) and environmental quantities (water level, rainfall, temperature, time-dependent factors, etc.). The model calculates predicted values ​​for the monitored data. Then, an early warning threshold is set based on the residual statistics between the measured and predicted values. When the residual exceeds the threshold, it is determined to be an anomaly, thus achieving real-time early warning of the dam's structural behavior.

[0004] Existing regression modeling techniques for dam monitoring mainly include classical least squares regression, stepwise regression, and various robust regression methods. Among them, robust regression methods, by introducing robust estimation theories such as M-estimation, L-estimation, and R-estimation, and employing mechanisms such as Tukey's double-weight function and Huber loss function, weaken the impact of outliers on model parameter estimation, thereby improving the model's resistance to interference from abnormal monitoring data to a certain extent. This has become the mainstream technique for dam monitoring modeling.

[0005] However, existing dam monitoring regression modeling techniques still face the following pressing technical challenges in practical engineering applications:

[0006] First, existing technologies do not effectively address the common problem of multicollinearity among independent variables in the modeling process. The independent variables in dam monitoring modeling typically include multi-level water levels, multi-level rainfall, time-dependent factors, and autoregressive factors. To capture the lagged effects of environmental factors on dam effects, a large number of multi-level lagged derived variables need to be constructed. These independent variables naturally exhibit explicit or implicit multicollinearity relationships. Even robust regression methods resistant to outlier interference cannot fundamentally eliminate the negative impact of multicollinearity on the model. Multicollinearity leads to unstable regression coefficient estimates and variance inflation, causing unnecessary and drastic fluctuations in model predictions. This directly affects the accuracy of subsequent residual statistics and early warning threshold calculations, and in severe cases, can even lead to false alarms and missed alarms.

[0007] Secondly, existing models are prone to overfitting in high-dimensional feature scenarios. With the advancement of monitoring technology and the improvement of data acquisition accuracy, dam monitoring modeling needs to introduce more order environmental factors and autoregressive factors to fully capture the temporal variation characteristics of the data, resulting in a significant increase in the dimensionality of the modeling features. Existing regression modeling methods generally do not introduce effective dimensionality reduction mechanisms or regularization constraints, making the models prone to overfitting to noise and redundant information in the training data. This leads to good performance on the training set, but severely insufficient generalization ability on new monitoring data, resulting in large prediction bias and reducing the universality and reliability of anomaly identification.

[0008] In addition, existing modeling methods mostly adopt a one-time modeling mode with a fixed process, lacking an adaptive optimization mechanism for different data scenarios, and cannot flexibly adjust the modeling strategy based on the model diagnostic results. The engineering practicality and adaptability need to be further improved. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a modeling method for resolving multicollinearity and suppressing overfitting in dam safety monitoring. Through model diagnosis and on-demand triggered optimization processes, it solves the multicollinearity and overfitting problems existing in the existing modeling methods, improves the stability and generalization ability of the regression model, and can flexibly stop at the corresponding step according to the model diagnosis results to output an optimized regression model that meets the requirements and adapts to the modeling needs of different data scenarios.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] This invention provides a method for resolving multicollinearity and suppressing overfitting in dam safety monitoring, comprising the following steps:

[0012] S1. Divide the dam monitoring data into training set and test set according to time sequence. Construct a basic robust regression model based on the historical effect size and environmental quantity in the training set. Solve the basic robust regression model to obtain the estimated value of the regression coefficient vector.

[0013] S2. Plot a curve comparing the changes in the actual observed values ​​of the training set with the predicted values ​​of the basic robust regression model. If the predicted results show violent oscillations, irregular large fluctuations, or severely unstable fitting patterns, it is determined that multicollinearity has had a substantial negative impact. Robust ridge regression optimization is then performed. This optimization is based on the robust ridge regression model, using iterative weighted least squares combined with the Tukey double-weight function to update sample weights. The optimal L2 regularization parameter is adaptively selected through the generalized cross-validation criterion, and the robust regression coefficients with regularization constraints are solved. If the predicted curve is smooth and the overall trend closely matches the actual data, the overfitting diagnosis stage is directly entered. The predicted values ​​of the basic robust regression model are obtained based on the estimated values ​​of the historical environmental variable matrix and the regression coefficient vector.

[0014] S3. Plot the curves comparing the changes in the actual observed values ​​and the model predicted values ​​on the test set. If the deviation between the model predicted curve and the actual observed curve on the test set is too large and the trend fit is poor, it is determined that overfitting has had a substantial negative impact. Perform robust PCA feature purification optimization. The robust PCA feature purification optimization uses the FAST-MCD algorithm based on the minimum covariance determinant to estimate the robust covariance matrix of the independent variables. After feature decomposition, principal components are selected according to the cumulative variance contribution rate to complete feature dimensionality reduction. Based on the dimensionality reduction features, a robust regression model is reconstructed. If the predicted curve and the actual curve on the test set fit well overall and have the same trend, the current regression model is directly output.

[0015] As a preferred technical solution, in step S1, the basic robust regression model is as follows:

[0016] ;

[0017] in, This is a vector of effect sizes composed of historical observations; The matrix represents historical environmental variables containing multiple lag terms. This is a vector of regression coefficients; This is the random error term;

[0018] The basic robust regression model is solved using the M-estimation method, and the estimated values ​​of the regression coefficient vector are shown in the following equation:

[0019] ;

[0020] in, This represents the estimated value of the regression coefficient vector obtained by using the M-estimation method; For equivalent weight matrices, This indicates transpose.

[0021] As a preferred technical solution, step S2, the robust ridge regression optimization specifically includes:

[0022] S21. Set the initial weight vector to an all-1 vector and preset the ridge regression regularization parameters. The range of values ​​for is defined, and the iteration termination condition is set.

[0023] S22. Substitute the initial weight matrix into the robust ridge regression model formula to calculate the initial regression coefficients;

[0024] S23. Calculate the training set residuals based on the solved initial regression coefficients, use the Tukey double weight function to calculate the weight of each sample, and form all the sample weights into a diagonal matrix to obtain the updated equivalent weight matrix.

[0025] S24. Substitute the updated weight matrix back into the robust ridge regression model formula, recalculate the regression coefficients, and repeat the weight update and coefficient solution process until the iteration termination condition is met.

[0026] S25, Traverse the presets The range of values, for each Calculate the corresponding GCV value and select the one with the smallest GCV value. As the optimal regularization parameter;

[0027] S26. Substitute the optimal regularization parameter into the iterative process to complete the full iterative solution and obtain the final robust ridge regression model.

[0028] As a preferred technical solution, in step S22, the robust ridge regression model is shown in the following equation:

[0029] ;

[0030] in, These are the initial regression coefficients. These are the initial test parameters that are preset within the value range. The matrix of environmental variables contains multiple lag terms. For effect size vectors, It is the identity matrix. This is the initial weight matrix.

[0031] As a preferred technical solution, the formula for calculating the GCV value in step S25 is as follows:

[0032] ; in, For the sample size, For the sum of squared residuals, For the hat matrix, For the trace of the hat matrix; compare all of them one by one. The value corresponding to the GCV is selected, with the smallest GCV value. As the optimal regularization parameter .

[0033] As a preferred technical solution, the robust PCA feature cleanup optimization specifically includes:

[0034] S31. The FAST-MCD algorithm based on the minimum covariance determinant is used to perform robust covariance matrix estimation on the independent variable matrix.

[0035] S32. Perform eigenvalue decomposition on the estimated robust covariance matrix, calculate the variance contribution rate and cumulative variance contribution rate corresponding to each eigenvalue, and select the eigenvectors corresponding to the first k eigenvalues ​​with a cumulative variance contribution rate ≥ 95% to form the projection matrix.

[0036] S33. Project the original independent variable matrix onto the selected principal component space to complete feature dimensionality reduction. Use the dimensionality-reduced feature matrix as the new independent variable to reconstruct the robust regression model.

[0037] S34. Repeat the overfitting judgment operation on the reconstructed regression model. If the model still has an overfitting problem, adjust the cumulative variance contribution rate threshold of the principal components to reduce the number of features, and re-perform feature dimensionality reduction and modeling. If the model has no overfitting problem, output the model as the final regression model.

[0038] As a preferred technical solution, in step S31, robust covariance matrix estimation is performed on the independent variable matrix, specifically as follows:

[0039] S311, Preset subset capacity threshold ,in This is a proportionality constant, with a value ranging from 0.7 to 0.8. The total number of samples, This indicates rounding up; outlier detection is performed on each variable dimension of the independent variable matrix, and the interquartile range method is used to screen suspicious outlier samples to construct a candidate sample pool;

[0040] S312. Extract an initial subset from the candidate sample pool, iteratively perform subset mean and covariance matrix calculations, robust Mahalanobis distance calculations for all samples, and find the sample with the minimum Mahalanobis distance. The process of reselecting samples continues until the determinant of the covariance matrix converges.

[0041] S313. Take the covariance matrix corresponding to the subset after iterative convergence as the robust covariance estimate of the independent variable matrix.

[0042] As a preferred technical solution, in step S311, the interquartile range method is used to screen suspicious abnormal samples and construct a candidate sample pool, specifically as follows:

[0043] For the independent variable matrix Outlier detection is performed on each variable dimension, and the first quartile of each variable is calculated. Third and quartiles and interquartile range The threshold for outlier detection is If a sample exceeds the outlier threshold in more than 10% of the variable dimensions, it is marked as a suspected outlier and placed in the observation area; the remaining samples form a candidate sample pool. If the number of samples in the candidate sample pool is insufficient... Then, the tolerance ratio is relaxed step by step until the number of samples in the candidate sample pool is ≥ Randomly select from the candidate sample pool The samples constitute the initial subset. .

[0044] As a preferred technical solution, step S33 specifically includes:

[0045] S331, Based on projection matrix The original independent variable matrix Perform a dimension reduction projection to obtain a new feature matrix after dimension reduction;

[0046] S332, using the reduced-dimensionality feature matrix For the new independent variable, the effect size vector Still using the dependent variable, repeat the optimization of the basic robust regression model or robust ridge regression in step S1 or step S2 to obtain a new basic robust regression model or robust ridge regression model.

[0047] As a preferred technical solution, in step S1, the effect quantities include dam displacement, seepage pressure, and uplift pressure, and the environmental quantities include water level, rainfall, temperature, time factor, and their multi-order lag derivative variables.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] (1) The present invention adopts a progressive adaptive optimization architecture. Through the diagnosis of the substantial impact of complex collinearity and overfitting, the corresponding optimization steps are triggered as needed. It can flexibly adapt to different data scenarios, avoid unnecessary computational overhead, and has strong engineering practicality.

[0050] (2) By combining the Tukey double weight function, iterative weighted least squares method and GCV adaptive parameter selection robust ridge regression method, this invention effectively eliminates the complex collinearity problem caused by multi-order lag environmental variables, avoids the variance expansion of regression coefficients and fluctuation of predicted values, and significantly improves the stability of the model.

[0051] (3) The present invention effectively reduces high-dimensional data redundancy, suppresses model overfitting, improves the model’s generalization ability to new monitoring data, and improves the reliability of anomaly identification by using a robust PCA feature purification method based on FAST-MCD robust covariance estimation.

[0052] (4) The robust PCA method of the present invention effectively screens out outliers in the independent variables through IQR relaxation screening and C-step iterative optimization, so that covariance estimation and feature dimensionality reduction are not affected by abnormal samples, the model coefficient estimation is more stable, and the anti-interference ability is stronger. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the multicollinearity resolution and overfit suppression modeling method in dam safety monitoring according to an embodiment of the present invention;

[0055] Figure 2 This is a comparison chart of the fitting effects of robust regression, robust ridge regression, and robust PCA modeling at a flow measurement point of a secondary dam in an embodiment of the present invention.

[0056] Figure 3 This is a comparison chart of the fitting effects of robust regression, robust ridge regression, and robust PCA modeling at a water level measuring point of a secondary dam in an embodiment of the present invention.

[0057] Figure 4 This is a comparison chart of the fitting effects of robust regression, robust ridge regression, and robust PCA modeling at a water level measuring point of a secondary dam in an embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0059] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0060] Terminology definition:

[0061] PCA: Principal Component Analysis, is an unsupervised linear dimensionality reduction method. In this invention, it is used to reduce the dimensionality and clean up the features of multi-hysteresis environmental quantities of a dam, eliminating multicollinearity among independent variables, reducing redundancy in high-dimensional data, and weakening the interference of outlier noise on the principal component direction.

[0062] MinCovDet: Minimum Covariance Determinant, a multivariate statistically robust covariance estimation method. Its core logic is to select the subset of the dataset with the smallest covariance determinant without outliers, and construct mean and covariance estimates that are not contaminated by outliers.

[0063] FAST-MCD: Fast Minimum Covariance Determinant, an efficient iterative implementation of MinCovDet. This invention replaces the traditional full quantum set traversal with C-step iterative optimization, significantly reducing computational complexity while maintaining robustness, thus providing an efficient and reliable foundation for robust PCA covariance estimation.

[0064] GCV: Generalized Cross-Validation, is a method for optimizing model parameters without requiring a separate validation set. In this invention, it is used to automatically determine the optimal L2 regularization parameter for robust ridge regression, avoiding the sample size loss of traditional cross-validation and adapting to parameter optimization scenarios for dam monitoring time series data.

[0065] IRLS: Iteratively Reweighted Least Squares, is the core algorithm for robust regression. In this invention, it is implemented using the Tukey double-weight loss function. By iteratively updating sample weights, the influence of outliers on the estimation of regression coefficients is reduced, while also adapting to the regularization constraints of ridge regression.

[0066] IQR: Interquartile Range, a statistical indicator that measures the dispersion of data, defined as the difference between the third quartile and the first quartile. In this invention, it is used to perform initial screening of one-dimensional outliers for independent variables, construct a clean initial sample pool, and accelerate the convergence of the FAST-MCD algorithm.

[0067] Robust PCA: Principal component analysis method combined with robust covariance estimation using FAST-MCD. In this invention, feature purification for multi-lag environmental quantities in dam monitoring can effectively eliminate multicollinearity among features and reduce redundancy in high-dimensional data in scenarios with outliers in independent variables, providing reliable dimensionality reduction features for subsequent modeling.

[0068] Robust Ridge Regression: A regression modeling method combining Iterative Weighted Least Squares (IRLS), Tukey's double-weighted loss function, and L2 regularization. This invention can simultaneously suppress outlier interference in the dependent variable, eliminate multicollinearity of independent variables, reduce the risk of model overfitting, and adapt to the non-ideal characteristics of dam monitoring data.

[0069] Multicollinearity: High correlations between independent variables in modeling can lead to unstable regression coefficient estimates and variance inflation, thus affecting the model's prediction accuracy. In this invention, it specifically refers to the strong correlations between multiple lag environmental quantities related to the dam, which is one of the core problems causing model parameter distortion.

[0070] Overfitting: A phenomenon where a model overfits noise, outliers, or a specific distribution in the training set data, resulting in good performance on the training set but poor generalization ability on new data. In this invention, it specifically refers to the problem of a model overlearning non-common fluctuations in the dam monitoring training set, leading to ineffective predictions of future monitoring data.

[0071] Effect quantities: In dam safety monitoring, physical quantities that reflect changes in structural behavior, such as displacement, seepage pressure, and uplift pressure, are the dependent variables in the regression modeling of this invention.

[0072] Environmental variables: External factors that affect the changes in dam effect quantities, such as water level, rainfall, temperature, time factor, and the set of multi-order lagged variables derived therefrom, are the core independent variables in this invention's modeling.

[0073] Multi-stage lagged environmental variables: In order to capture the lagged effects of environmental factors on dam effects, a set of lagged derived variables is constructed from the original environmental variables (such as water level and temperature). This is the core independent variable in the modeling of this invention and also the main source of multicollinearity.

[0074] Tukey's dual-weight function: a robust loss function that assigns differentiated weights to samples with different residuals, reduces the weight of outlier samples with excessively large residuals, thereby weakening their impact on the estimation of regression model parameters. In this invention, it is used for weight updates in iterative weighted least squares.

[0075] like Figure 1 As shown in this embodiment, a multicollinearity resolution and overfitting suppression modeling method for dam safety monitoring is provided. Through model diagnosis and on-demand triggered optimization processes, it solves the multicollinearity and overfitting problems existing in current modeling methods, improving the stability and generalization ability of the regression model. The method can flexibly stop at the corresponding step based on the model diagnosis results, outputting an optimized regression model that meets the requirements and adapting to the modeling needs of different data scenarios. Specifically, it includes the following steps:

[0076] S1. Construct a basic robust regression model; specifically including:

[0077] S11. Divide the dam monitoring data into training set and test set according to time sequence. For example, if the sample size is small, take the first 80% as training set and the last 20% as test set; if the sample size is large, take the first 70% as training set and the last 30% as test set.

[0078] It is understandable that dam monitoring data is divided according to time sequence because dam monitoring data is a typical time series data with clear time-series dependence and autocorrelation. The test set must lag behind the training set in time in order to truly simulate the actual scenario of "predicting the effect size of future moments based on existing historical data" in engineering applications.

[0079] S12. Construct a basic robust regression model using historical effect sizes and environmental factors from the training set. The basic robust regression model is shown in Equation (1):

[0080]

[0081] In the formula, This is a vector of effect sizes composed of historical observations; The matrix represents historical environmental variables containing multiple lag terms. This is a vector of regression coefficients; This is the random error term.

[0082] S13. Solve the basic robust regression model using the M-estimation method to obtain the regression coefficient vector. The estimated value The estimated value is shown in equation (2):

[0083]

[0084] In the formula, The equivalent weight matrix is ​​calculated using the Tukey double weight function to reduce the influence of outliers in the dependent variable. This indicates transpose.

[0085] S2. Diagnosis of the impact of complex collinearity on the basic robust regression model and optimization of robust ridge regression;

[0086] Since the modeling process introduces multi-stage lag environmental factors, multicollinearity among independent variables is an inherent objective phenomenon. Therefore, it is not necessary to determine whether multicollinearity exists or not, but only whether multicollinearity has a substantial negative impact on the model.

[0087] At this point, it is necessary to plot a curve comparing the changes in the actual observed values ​​of the training set with the predicted values ​​of the basic robust regression model. If the prediction results show violent oscillations, irregular large fluctuations, or severely unstable fitting patterns, it is determined that the multicollinearity caused by lagged variables has led to the divergence and coefficient explosion of regression coefficients, and the robust ridge regression optimization step needs to be entered. If the prediction curve is smooth, the overall trend of change closely matches the actual data, and the model runs stably, then the next step of overfitting diagnosis can be directly entered. Figure 2 , Figure 3 As shown in the first model fitting plot, the prediction results all exhibit significant oscillations and fluctuations; for example... Figure 4 As shown in the first model fitting graph, the prediction results show a severely unstable fitting pattern, and all of them need to enter the robust ridge regression optimization step.

[0088] Here, the formula for calculating the predicted value of the basic robust regression model is shown in the following equation:

[0089]

[0090] In the formula, These are the model's predicted values; Regression coefficient vector The estimated value; It is a matrix of historical environmental variables containing multiple lag terms.

[0091] Furthermore, it is necessary to plot a curve comparing the changes in the actual observed values ​​of the training set with the predicted values ​​of the basic robust regression model. This is done by plotting the sample data order on the horizontal axis and the actual effect size and the predicted values ​​of the basic robust regression model on the vertical axis, thus comparing the two on the same coordinate system.

[0092] Furthermore, the robust ridge regression optimization specifically involves:

[0093] S21. Initialize parameters and weight matrix:

[0094] First, initialize the sample equivalent weight matrix. Set the initial weight vector to an all-one vector, i.e., the initial weight matrix. ( (The identity matrix); and several ridge regression regularization parameters are preset. (usually in) Select from, such as preset , , , And set the iteration termination condition (such as setting the upper limit of the number of iterations to 100, or the absolute difference between two adjacent regression coefficients being less than 100). Stop iteration when the time is right.

[0095] S22. Solving for the initial robust ridge regression coefficients:

[0096] The initial weight matrix Substitute the values ​​into the robust ridge regression model formula to calculate the initial regression coefficients. The robust ridge regression model is shown in equation (3):

[0097]

[0098] In the formula, The initial test parameters are preset within the value range, and the ridge regression regularization parameters are defined in step S21. One of the specific values, The matrix of historical environmental variables contains multiple lag terms. This is the effect size vector composed of historical observations. It is an identity matrix.

[0099] S23. Update the weight matrix based on Tukey's double-weight function:

[0100] Based on the initial regression coefficients obtained Calculate the training set residuals The weight of each sample is calculated using the Tukey double-weight function, and the equivalent weight matrix is ​​updated accordingly. The expression for the Tukey double weight function is shown in equation (4):

[0101]

[0102] In the formula, Let be the weight of the i-th sample; To standardize the residuals, ,in Let be the original residual of the i-th sample. When i is 0, it is the residual of the training set. ; The robust adjustment parameter for the Tukey double-weight function is generally set to c = 4.685. The robust scaling estimator of the residuals can be calculated using the median absolute deviation (MAD) of the residuals, as shown in equation (5):

[0103]

[0104] By assembling all sample weights into a diagonal matrix, we obtain the updated equivalent weight matrix. .

[0105] S24. Iteratively solve for the robust ridge regression coefficients:

[0106] The updated equivalent weight matrix Substitute the values ​​back into the Steady Ridge regression model formula and recalculate the regression coefficients. As shown in equation (6):

[0107]

[0108] Repeat steps S23 and S24 to update the weight matrix sequentially. and regression coefficients ( (This is the number of iterations), until the iteration termination condition is met, and the test parameter is obtained. Corresponding stable regression coefficient and weight matrix .

[0109] S25. Selecting the optimal regularization parameter based on the GCV criterion :

[0110] Traverse the preset The range of values, for each Repeat steps S22-S24 for each value to obtain the corresponding stable regression coefficients and model predicted values. For each Substitute the value into the GCV calculation formula to calculate the corresponding GCV value. The GCV calculation formula is shown in formula (7):

[0111] In the formula, For the sample size, For the sum of squared residuals, For the hat matrix, The trace of the hat matrix (i.e., the sum of the elements on the main diagonal of the matrix); compare all... The value corresponding to the GCV is selected, with the smallest GCV value. As the optimal regularization parameter .

[0112] S26. Determine the final robust ridge regression model: Optimal regularization parameters... Substitute the steps from S22 to S24 into the iterative process, complete a full iteration again, and obtain the final robust ridge regression coefficients. The corresponding final robust ridge regression model is:

[0113] S3. Overfitting diagnosis and robust PCA feature cleanup;

[0114] Plot a curve comparing the changes in the observed values ​​and model predictions on the test set (the model is either the basic robust regression model from step S1 or the robust ridge regression optimization model from step S2). If the model's predicted curve deviates significantly from the observed curve on the test set, the trend fit is poor, or there is a significant fitting bias or high dispersion between the predicted and observed values, then overfitting has negatively impacted the model's generalization ability, and robust PCA feature cleanup is required. If the predicted curve and observed curve on the test set generally fit well, the trend is consistent, the dispersion is small, and the model's generalization ability is good, then the current regression model is directly output. Figure 2 The second model fitting plot shows that the prediction results are stable with small fluctuations, correctly fitting non-abnormal cases, and the current regression model can be directly output. For example... Figure 3 As shown in the second model fitting plot, the prediction results still exhibit some oscillations; for example... Figure 4 As shown in the second model fitting diagram, the prediction results still show unstable fitting patterns, all of which need to go through the robust PCA feature purification step.

[0115] Furthermore, the actual measured values ​​of the test set are the real monitoring data of the effect quantities (displacement, seepage pressure, uplift pressure, etc.) in the test set samples divided in step S11, which are obtained by monitoring equipment.

[0116] Furthermore, the model predictions here depend on the diagnosis and execution results in step S2: if robust ridge regression optimization is not performed, the basic robust regression model in step S1 is used; if robust ridge regression optimization is performed, then the model is the robust ridge regression optimized model in step S2.

[0117] Furthermore, the robust PCA feature cleanup is as follows:

[0118] S31, MinCovDet Robust Covariance Matrix Estimation: The FAST-MCD algorithm, based on the minimum covariance determinant (MinCovDet), is used to estimate the independent variable matrix. The following steps are performed to estimate the robust covariance matrix:

[0119] S311. Setting Subset Size and IQR Relaxation Screening: Preset Subset Size Threshold ,in Take 0.75 (can be adjusted between 0.7 and 0.8 depending on the proportion of outliers). The total number of samples, Indicates rounding up; for the independent variable matrix Outlier detection is performed on each variable dimension, and the first quartile of each variable is calculated. Third and quartiles and interquartile range The threshold for outlier detection is If a sample exceeds the corresponding threshold in more than 10% of the variable dimensions, it is marked as a suspected anomalous sample and placed in the observation area; the remaining samples form a candidate sample pool. If the number of samples in the candidate sample pool is insufficient... Then, the tolerance ratio is relaxed step by step (e.g., 20%, 30%, etc.) until the number of samples in the candidate sample pool is ≥ Randomly select from the candidate sample pool The samples constitute the initial subset. .

[0120] S312, C-step iterative optimization of FAST-MCD: Let the current subset be... Iteratively execute the following steps until convergence:

[0121] (a) Computation subset mean With covariance matrix The calculation formula is as follows:

[0122]

[0123]

[0124] (b) Calculate all One sample to Robust Mahal distance:

[0125]

[0126] (c) Select Mahalanobis distance smallest Each sample constitutes a new subset. ;

[0127] (d) Calculate the new subset The corresponding covariance matrix ,like (Preset tolerance), or If the number of iterations reaches 50 (the upper limit), then stop the iteration; otherwise, let... Repeat steps (a)-(d).

[0128] S313, Robust Covariance Matrix Output: Take the subset after iterative convergence. The corresponding covariance matrix is ​​used as the independent variable matrix. Robust covariance estimation ,Right now:

[0129]

[0130] In the formula, Represents the determinant of a matrix. Represents a convergent subset The covariance matrix can effectively avoid interference from outlier samples, providing a reliable foundation for subsequent feature dimensionality reduction.

[0131] S32. Eigenvalue Decomposition and Principal Component Selection: The estimated robust covariance matrix... Perform eigenvalue decomposition and select effective principal components to complete dimensionality reduction. The specific steps are as follows:

[0132] S321, Regarding robust covariance matrices Perform eigenvalue decomposition to obtain eigenvalues. ( (number of independent variables) and the corresponding eigenvectors The eigenvalue decomposition satisfies ;

[0133] S322. Calculate the variance contribution rate and cumulative variance contribution rate corresponding to each eigenvalue:

[0134] Variance contribution rate:

[0135] Cumulative variance contribution rate:

[0136] S323. Select the cumulative variance contribution rate. The former The eigenvectors corresponding to the eigenvalues ​​constitute the projection matrix. .

[0137] S33. Feature Reduction and Remodeling: Reducing the Original Independent Variable Matrix Projecting onto the selected principal component space, feature dimensionality reduction is performed, and a robust (ridge) regression model is reconstructed. The specific operations are as follows:

[0138] S331, Based on projection matrix The original independent variable matrix Perform dimensionality reduction projection to obtain the new feature matrix after dimensionality reduction. , for 3D matrix ( For the sample size, (Number of principal components)

[0139] S332, using the reduced-dimensionality feature matrix For the new independent variable, the effect size vector Still using the dependent variable, repeat the robust (ridge) regression modeling in step S1 or S2 to obtain a new robust (ridge) regression model.

[0140] S34. Iterative Diagnosis and Model Output: For the reconstructed regression model, repeat the overfitting assessment step S3 (plotting the comparison curve between the measured and predicted values ​​on the test set). If the reconstructed regression model still has an overfitting problem, adjust the cumulative variance contribution rate threshold of the principal components to reduce the number of features (it can be appropriately reduced to between 85% and 95%), and repeat robust PCA steps S32-S33 to re-perform feature dimensionality reduction and modeling. If the model has no overfitting problem, output this model as the final regression model after overfitting optimization, thus completing the solution to the overfitting problem. Figure 3 The third model fitting plot shows that the model has no overfitting problem and can be output as the final regression model. Figure 4 The third model fitting plot shows that the model is significantly improved compared to the previous model, but there are still fluctuation problems. The cumulative variance contribution rate threshold of the principal component selection can be appropriately adjusted, and robust PCA feature purification can be performed again.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for resolving multicollinearity and suppressing overfitting in dam safety monitoring, characterized in that, Includes the following steps: S1. Divide the dam monitoring data into training set and test set according to time sequence. Construct a basic robust regression model based on the historical effect size and environmental quantity in the training set. Solve the basic robust regression model to obtain the estimated value of the regression coefficient vector. S2. Plot a curve comparing the changes in the actual observed values ​​of the training set with the predicted values ​​of the basic robust regression model. If the predicted results show violent oscillations, irregular large fluctuations, or severely unstable fitting patterns, it is determined that multicollinearity has had a substantial negative impact. Robust ridge regression optimization is then performed. This optimization is based on the robust ridge regression model, using iterative weighted least squares combined with the Tukey double-weight function to update sample weights. The optimal L2 regularization parameter is adaptively selected through the generalized cross-validation criterion, and the robust regression coefficients with regularization constraints are solved. If the predicted curve is smooth and the overall trend closely matches the actual data, the overfitting diagnosis stage is directly entered. The predicted values ​​of the basic robust regression model are obtained based on the estimated values ​​of the historical environmental variable matrix and the regression coefficient vector. S3. Plot the curves comparing the changes in the actual observed values ​​and the model predicted values ​​on the test set. If the deviation between the model predicted curve and the actual observed curve on the test set is too large and the trend fit is poor, it is determined that overfitting has had a substantial negative impact. Perform robust PCA feature purification optimization. The robust PCA feature purification optimization uses the FAST-MCD algorithm based on the minimum covariance determinant to estimate the robust covariance matrix of the independent variables. After feature decomposition, principal components are selected according to the cumulative variance contribution rate to complete feature dimensionality reduction. Based on the dimensionality reduction features, a robust regression model is reconstructed. If the predicted curve and the actual curve on the test set fit well overall and have the same trend, the current regression model is directly output.

2. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S1, the basic robust regression model is shown in the following equation: ； in, This is a vector of effect sizes composed of historical observations; The matrix represents historical environmental variables containing multiple lag terms. This is a vector of regression coefficients; This is the random error term; The basic robust regression model is solved using the M-estimation method, and the estimated values ​​of the regression coefficient vector are shown in the following equation: ； in, This represents the estimated value of the regression coefficient vector obtained by using the M-estimation method; For equivalent weight matrices, This indicates transpose.

3. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S2, the robust ridge regression optimization specifically includes: S21. Set the initial weight vector to an all-1 vector and preset the ridge regression regularization parameters. The range of values ​​for is defined, and the iteration termination condition is set. S22. Substitute the initial weight matrix into the robust ridge regression model formula to calculate the initial regression coefficients; S23. Calculate the training set residuals based on the solved initial regression coefficients, use the Tukey double weight function to calculate the weight of each sample, and form all the sample weights into a diagonal matrix to obtain the updated equivalent weight matrix. S24. Substitute the updated weight matrix back into the robust ridge regression model formula, recalculate the regression coefficients, and repeat the weight update and coefficient solution process until the iteration termination condition is met. S25, Traverse the presets The range of values, for each Calculate the corresponding GCV value and select the one with the smallest GCV value. As the optimal regularization parameter; S26. Substitute the optimal regularization parameter into the iterative process to complete the full iterative solution and obtain the final robust ridge regression model.

4. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S22, the robust ridge regression model is shown in the following equation: ； in, These are the initial regression coefficients. These are the initial test parameters that are preset within the value range. The matrix of environmental variables contains multiple lag terms. For effect size vectors, It is the identity matrix. This is the initial weight matrix.

5. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S25, the formula for calculating the GCV value is as follows: ； in, For the sample size, For the sum of squared residuals, For the hat matrix, For the trace of the hat matrix; compare all of them one by one. The value corresponding to the GCV is selected, with the smallest GCV value. As the optimal regularization parameter .

6. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, The robust PCA feature cleanup optimization specifically includes: S31. The FAST-MCD algorithm based on the minimum covariance determinant is used to perform robust covariance matrix estimation on the independent variable matrix. S32. Perform eigenvalue decomposition on the estimated robust covariance matrix, calculate the variance contribution rate and cumulative variance contribution rate corresponding to each eigenvalue, and select the eigenvectors corresponding to the first k eigenvalues ​​with a cumulative variance contribution rate ≥ 95% to form the projection matrix. S33. Project the original independent variable matrix onto the selected principal component space to complete feature dimensionality reduction. Use the dimensionality-reduced feature matrix as the new independent variable to reconstruct the robust regression model. S34. Repeat the overfitting judgment operation on the reconstructed regression model. If the model still has an overfitting problem, adjust the cumulative variance contribution rate threshold of the principal component selection to reduce the number of features, and re-perform feature dimensionality reduction and modeling. If the model does not have an overfitting problem, output the model as the final regression model.

7. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S31, robust covariance matrix estimation is performed on the independent variable matrix, specifically as follows: S311, Preset subset capacity threshold ,in This is a proportionality constant, with a value ranging from 0.7 to 0.

8. The total number of samples, This indicates rounding up; outlier detection is performed on each variable dimension of the independent variable matrix, and the interquartile range method is used to screen suspicious outlier samples to construct a candidate sample pool; S312. Extract an initial subset from the candidate sample pool, iteratively perform subset mean and covariance matrix calculation, robust Mahalanobis distance calculation for all samples, and find the sample with the minimum Mahalanobis distance. The process of reselecting samples continues until the determinant of the covariance matrix converges. S313. Take the covariance matrix corresponding to the subset after iterative convergence as the robust covariance estimate of the independent variable matrix.

8. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S311, the interquartile range method is used to screen suspicious abnormal samples and construct a candidate sample pool, specifically as follows: For the independent variable matrix Outlier detection is performed on each variable dimension, and the first quartile of each variable is calculated. Third and quartiles and interquartile range The threshold for outlier detection is ; If a sample exceeds the outlier threshold in more than 10% of the variable dimensions, it is marked as a suspected outlier and placed in the observation area; the remaining samples form a candidate sample pool. If the number of samples in the candidate sample pool is insufficient... Then, the tolerance ratio is relaxed step by step until the number of samples in the candidate sample pool is ≥ Randomly select from the candidate sample pool The samples constitute the initial subset. .

9. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, Step S33 is as follows: S331, Based on projection matrix The original independent variable matrix Perform a dimension reduction projection to obtain a new feature matrix after dimension reduction; S332, using the reduced-dimensionality feature matrix For the new independent variable, the effect size vector Still using the dependent variable, repeat the optimization of the basic robust regression model or robust ridge regression in step S1 or step S2 to obtain a new basic robust regression model or robust ridge regression model.

10. The method for resolving multicollinearity and suppressing overfitting in dam safety monitoring according to claim 1, characterized in that, In step S1, the effect quantities include dam displacement, seepage pressure, and uplift pressure, and the environmental quantities include water level, rainfall, temperature, time factor, and their multi-order lag derivative variables.

Images