Method for enhancing earth-rock dam breaching database based on CTGAN

Abstract

The application discloses a CTGAN-based earth-rock dam breaching database enhancement method, which comprises the following steps: constructing and preprocessing an original earth-rock dam breaching database, and screening key condition variables; constructing and training a conditional generative adversarial network (CTGAN) with the condition variables as constraints, and weighting in the training according to the category frequency; inputting target conditions and noise into the trained generator to generate enhanced samples; and comparing a heat map and statistical deviation of the enhanced samples and the original data to achieve enhancement of the earth-rock dam breaching database. The CTGAN-based earth-rock dam breaching database enhancement method disclosed by the application solves the problem that, in the prior art, due to the scarcity of measured samples of dam breaching and the incapability of traditional data enhancement methods to capture the conditional dependence relationship between dam breaching variables, the fidelity of generated samples is low, and it is difficult to effectively expand the dam breaching database.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy and hydropower engineering technology, specifically relating to a database enhancement method for earth-rock dam breakage based on CTGAN. Background Technology

[0002] As earth-rock dams age, they generally face prominent problems such as structural aging, low flood control standards, and inadequate flood discharge facilities. Simultaneously, the frequency and intensity of natural disasters such as extreme rainfall, strong earthquakes, and reservoir bank landslides have increased significantly, exacerbating the damage to earth-rock dams and leading to a sharp increase in their failure risk. Due to the scarcity of actual dam failure cases and the lack of monitoring data, existing models generally have low accuracy in predicting key characteristic parameters such as dam failure peak flow and dam failure duration. Therefore, it is urgent to introduce new technologies or methods to expand the dam failure database and improve the generalization ability and robustness of prediction models.

[0003] In recent years, the introduction of artificial intelligence algorithms has improved the accuracy of dam break parameter prediction and risk assessment to some extent, but its application effect is still limited by the insufficient amount of available data. Data bottlenecks restrict the model's representational ability and generalization performance, becoming a key factor restricting the development of dam break prediction technology. Data augmentation techniques, by transforming and expanding the original data, can effectively increase sample diversity, thereby improving the robustness and generalization ability of the model. However, research on data augmentation for earth-rock dam break databases remains very weak. Existing methods are mostly based on simple copying, interpolation, or linear transformations, lacking the ability to model the complex conditional dependencies between variables during the dam break process. They cannot generate high-quality augmented samples with physical rationality and controllable conditions while maintaining the statistical consistency of the original data, making it difficult to meet the dual requirements of dam break prediction models for training data diversity and fidelity. Summary of the Invention

[0004] The purpose of this invention is to provide a CTGAN-based method for augmenting the dam failure database of earth-rock dams, which solves the problems in the prior art where the scarcity of measured dam failure samples and the inability of traditional data augmentation methods to capture the conditional dependencies between dam failure variables lead to low sample fidelity and difficulty in effectively expanding the dam failure database.

[0005] The technical solution adopted in this invention is a CTGAN-based method for enhancing the earth-rock dam failure database, comprising: constructing and preprocessing the original earth-rock dam failure database, and screening key condition variables; constructing and training a conditional generative adversarial network with the condition variables as constraints, and weighting the network according to class frequency during training; inputting the target conditions and noise into the trained generator to generate enhanced samples; and comparing the enhanced samples with the original data using heatmaps and statistical biases to enhance the earth-rock dam failure database.

[0006] The technical solution of this invention is also characterized by, Includes the following steps: S1. Construct an original database of earth-rock dam failures. The samples in the original database contain basic information parameters and failure characteristic parameters. Both the basic information parameters and failure characteristic parameters include discrete and continuous variables. The original database is preprocessed to obtain a standardized database. S2, based on a standardized database, uses correlation analysis to calculate the correlation coefficient between each basic information parameter and each dam-break characteristic parameter, and selects the basic information parameters that meet the preset correlation threshold with each dam-break characteristic parameter as the key condition variable set; the key condition variable set includes discrete variables and continuous variables; S3: Encode and concatenate the discrete variables in the key condition variable set to form the target condition vector; establish a conditional generative adversarial network model containing a generator G and a discriminator D; the generator G takes the result of concatenating the target condition vector with random noise as input and outputs data samples; the discriminator D takes the result of concatenating the sample data with the corresponding target condition vector as input and outputs the authenticity judgment result. S4. The generator G and discriminator D are trained iteratively using an adversarial loss function. During the training process, weighting coefficients are set according to the class frequency to obtain the trained generator G and discriminator D. S5, input the set target condition vector and random noise into the trained generator G to generate augmented sample data that satisfies the target condition constraints; S6 performs statistical consistency analysis between the enhanced sample data and the normalized database. By comparing correlation heatmaps and evaluating preset statistical indicators, the fidelity of the enhanced sample data is verified, thereby enhancing the database of earth-rock dam failures.

[0007] Basic information parameters include dam type and dam height. h d dam crest width W and storage capacity V The characteristic parameters of dam failure include dam type. DT Defeat mode FM Water level above the breach h w Reservoir capacity above the breach V w , depth of the ulcer h b Average width of the ulcer B ave Peak flow rate during dam break Q p and the duration of the dam breach T f The discrete variable in the basic information parameters is the dam type, and the continuous variable is the dam height. h d dam crest width W and storage capacityV The discrete variable in the dam failure characteristic parameters is the dam type. DT and the collapse mode FM The continuous variable is the water level above the breach. h w Reservoir capacity above the breach V w , depth of the ulcer h b Average width of the ulcer B ave Peak flow rate during dam break Q p and the duration of the dam breach T f Dam type and dam type DT All of these include homogeneous dams, core wall dams, and panel dams.

[0008] The preprocessing steps in S1 are as follows: Missing data in the original database of earth-rock dam failure is filled using the mean imputation method; outliers are identified using a box plot method with a threshold of 1.5 times the interquartile range, and corrected using cubic spline interpolation; discrete variables are one-hot encoded; and continuous variables are normalized using the following formula: (1) In the formula, x′ These are the normalized variable values; x The original variable value; x min and x max These are the minimum and maximum values ​​of the variable in the original database of earth-rock dam failures, respectively.

[0009] The correlation analysis method in S2 is the Pearson correlation coefficient method, and its calculation method is as follows: (2) Where r is the Pearson correlation coefficient, x i The value of the i-th sample of the basic information parameter, y i The value of the i-th sample of the dam failure characteristic parameter; The basic parameter is the mean of all samples. The mean of all samples for dam failure characteristics is used. Based on the standardized database, the correlation coefficient between each basic information parameter and each dam failure characteristic parameter is calculated. The absolute value threshold of the correlation coefficient |r|≥0.3 is set, and the basic information parameters that meet this threshold are selected as the set of key condition variables.

[0010] In S3, the generator G uses a fully connected neural network. The random noise is sampled from a standard normal distribution and has a dimension of 128. The target conditional vector is concatenated with the random noise and used as the input to the generator G. The concatenation is calculated as follows: (3) In the formula, z in The input vector to the generator; z It is a random noise vector; d z For random noise vectors z The dimension; dim( z in ) is a generator G The total dimension of the input vector; The total length after one-hot encoding of all discrete attribute variables. m The number of discrete variables, K j For the first j The number of categories for each discrete variable; The hidden layers of generator G use the ReLU activation function. In the output layer, neurons corresponding to continuous variables use the linear activation function, and neurons corresponding to discrete variables use the Softmax activation function. The output data sample is represented as follows: (4) In the formula, This is the complete augmented synthetic dam-break sample vector output by the generator. θ G For generator network parameters, This is a continuous variable vector in the generated dam-break characteristic parameters. This is the vector of discrete variables in the generated dam-break characteristic parameters.

[0011] The alternating iterative training in S4 includes: fixing the parameters of the generator G, updating the parameters of the discriminator D using samples from the normalized database and the augmented samples output by the generator G; fixing the parameters of the discriminator D, updating the parameters of the generator G through backpropagation of the discrimination results, repeating this process until the preset number of iterations or the loss function converges, so that the augmented sample data generated by the generator G approximates the true sample distribution of the normalized database. The adversarial loss function is expressed as: (5) In the formula, min G、 max D To illustrate the min-max optimization process, the generator G The discriminant improves the quality of generated samples by minimizing the objective function. D This enhances the ability to distinguish between real and generated samples by maximizing the objective function; V ( D , G ) represents the adversarial loss function; Ex~Pdata Represents the distribution of real data P data The mathematical expectation, where x These are real samples taken from the original database; E z Pz Represents the prior noise distribution P z The mathematical expectation; D ( G ( z , c ), c ) indicates that the discriminator evaluates the generated samples. G ( z , c ) and its corresponding condition vector c The output probability.

[0012] The weighting coefficients in S4 are set as follows: the weighting coefficients for each category are inversely proportional to the frequency of occurrence of samples of that category in the normalized database. The weighting coefficients are expressed as follows: (6) In the formula: ω k For the first k Class weighted coefficients; f k For the first k The proportion of class samples in the total sample.

[0013] S6 pre-sets statistical indicators including mean absolute deviation and root mean square error. The formulas for calculating mean absolute deviation and root mean square error are as follows: (7) (8) In the formula, MAD The mean absolute deviation, RMSE This is the root mean square error; m This represents the total number of variables involved in the correlation analysis. For all i and j Perform a double summation to include the correlation coefficients between all different pairs of variables; Variables in the original database for earth-rock dam failure i With variables j The Pearson correlation coefficient between them; The Pearson correlation coefficient between variables i and j in the augmented sample data generated by the conditional generative adversarial network model.

[0014] The beneficial effects of this invention are: (1) This invention addresses the problems of scarce measured samples, insufficient coverage of key feature parameters, and difficulty in capturing complex conditional dependencies between variables in the prediction of earth-rock dam failure risk. Based on the Conditional Table Generative Adversarial Network (CTGAN) algorithm, it constructs conditional vectors to achieve controllable data generation under variable constraints, embeds a weighted learning mechanism based on class frequency to improve the generation quality of low-frequency class samples, and combines heat map analysis to verify the fidelity of the augmented data, thereby achieving high-quality augmentation and reliability assessment of the earth-rock dam failure database under missing data conditions.

[0015] (2) The CTGAN-based database enhancement method for earth-rock dam breakage provided by this invention significantly increases the scale of the dam breakage database and effectively alleviates the problem of sample scarcity. While maintaining the original data feature distribution, this invention uses the CTGAN model to enhance the limited dam breakage samples, with the increase of each feature parameter reaching more than 3 times, providing more sufficient data support for the training of data-driven dam breakage prediction models and risk analysis.

[0016] (3) The CTGAN-based database enhancement method for earth-rock dam breakage provided by this invention achieves high-quality sample generation under conditional constraints, enhancing the relevance and representativeness of the data. This invention introduces a target conditional vector to achieve directional sample generation under specific dam breakage patterns, enhancing the mean absolute deviation (MAD) between the data and the original data. MAD ) and root mean square error ( RMSE The values ​​were only 0.036 and 0.052 respectively, both far below the consistency evaluation threshold of 0.1, realizing the transformation from traditional random expansion to conditionally controllable generation and improving the fidelity of the generated data.

[0017] (4) The CTGAN-based database enhancement method for earth-rock dam breakage provided by this invention effectively improves the class imbalance problem and enhances the learning ability of minority class samples. This invention introduces a weighted loss mechanism based on class frequency, increasing the weight of minority class samples to 4 times that of majority class samples (5.0 / 1.25), which significantly enhances the model's learning ability for minority class features and reduces the bias of the model towards majority class samples during training. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of the condition table generation adversarial network model of the present invention; Figure 2 This is a flowchart illustrating the CTGAN-based database enhancement method for earth-rock dam breakage in this invention. Figure 3 This is a statistical chart showing the classification of dam failure modes in the original database of earth-rock dam failures of this invention. Figure 4This is a classification and statistical diagram of dam types in the original database of earth-rock dam failures of this invention. Figure 5 This is a schematic diagram of the characteristic variables in the original database of earth-rock dam failure according to the present invention; Figure 6 This is a three-dimensional scatter plot of the peak dam-break flow rate of the present invention; Figure 7 This is a three-dimensional scatter plot of the ulcer depth of the present invention; Figure 8 This is a three-dimensional scatter plot of the average width of the ulcer in this invention; Figure 9 This is a three-dimensional scatter plot of the dam's duration according to the present invention; Figure 10 A schematic diagram comparing the raw data of the breach flood peak flow and the CTGAN-enhanced data in this invention; Figure 11 A schematic diagram comparing the original data of the ulcer depth of this invention with the thermal data enhanced by CTGAN; Figure 12 A schematic diagram comparing the original data of the average ulcer width of this invention with the thermal data enhanced by CTGAN; Figure 13 A schematic diagram comparing the original data of the dam breach duration with the CTGAN-enhanced data of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 This embodiment provides a method for enhancing the earth-rock dam failure database based on CTGAN, including: constructing and preprocessing the original earth-rock dam failure database, and screening key condition variables; constructing and training a conditional generative adversarial network with the condition variables as constraints, and weighting the network according to class frequency during training; inputting the target conditions and noise into the trained generator to generate enhanced samples; and comparing the enhanced samples with the original data using heatmaps and statistical bias to enhance the earth-rock dam failure database.

[0021] Example 2 Based on Example 1, such as Figure 1 and Figure 2 As shown, this invention provides a method for enhancing the database of earth-rock dam breakage based on CTGAN, specifically including the following steps: S1. Construct an original database of earth-rock dam failures. The samples in the original database contain basic information parameters and failure characteristic parameters. Both the basic information parameters and failure characteristic parameters include discrete and continuous variables. The original database is preprocessed to obtain a standardized database. S2, based on a standardized database, uses correlation analysis to calculate the correlation coefficient between each basic information parameter and each dam-break characteristic parameter, and selects the basic information parameters that meet the preset correlation threshold with each dam-break characteristic parameter as the key condition variable set; the key condition variable set includes discrete variables and continuous variables; S3: Encode and concatenate the discrete variables in the key condition variable set to form the target condition vector; establish a conditional generative adversarial network model containing a generator G and a discriminator D; the generator G takes the result of concatenating the target condition vector with random noise as input and outputs data samples; the discriminator D takes the result of concatenating the sample data with the corresponding target condition vector as input and outputs the authenticity judgment result. S4. The generator G and discriminator D are trained iteratively using an adversarial loss function. During the training process, weighting coefficients are set according to the class frequency to obtain the trained generator G and discriminator D. S5, input the set target condition vector and random noise into the trained generator G to generate augmented sample data that satisfies the target condition constraints; S6 performs statistical consistency analysis between the enhanced sample data and the normalized database. By comparing correlation heatmaps and evaluating preset statistical indicators, the fidelity of the enhanced sample data is verified, thereby enhancing the database of earth-rock dam failures.

[0022] Example 3 Based on the above embodiments, S1 specifically involves: constructing a raw database of earth-rock dam failure parameters. The database samples include two categories: basic information parameters and failure characteristic parameters. Basic information parameters include dam type and dam height. h d dam crest width W and storage capacity V The characteristic parameters of dam failure include dam type. DT Defeat mode FM Water level above the breach h w Reservoir capacity above the breach V w , depth of the ulcer h b Average width of the ulcer B ave Peak flow rate during dam break Q p and the duration of the dam breach T fThe variables are divided into two categories: continuous variables and discrete variables. The database is preprocessed, specifically as follows: missing data in the original database of earth-rock dam failure is imputed using the mean imputation method; outliers are identified using box plots with a threshold of 1.5 times the interquartile range, and corrected using cubic spline interpolation; then, discrete variables are one-hot encoded: a set of discrete attribute variables is established as follows: D ={ d 1, d 2,…, d m}, where each variable d j The values ​​can take on a finite number of categories. One-hot encoding is performed on each discrete variable: if the variable... d j have K j If there are 3 possible categories, then encode them into a string of length 1. K j The binary vector is generated where only the position corresponding to the category is 1, and the rest are 0. Then, the one-hot encoding results of all discrete variables are concatenated along the dimension to obtain the condition vector. c This characterizes the category constraints that the generated samples should satisfy. Finally, continuous variables are normalized. To eliminate the influence of different units and magnitudes on model training, normalization is used to linearly map each continuous variable to the [0, 1] interval, and its expression is: (1) In the formula: x′ These are the normalized variable values; x The original variable value; x min and x max These are the minimum and maximum values ​​of the variable, respectively.

[0023] Example 4 Based on the above embodiments, S2 specifically involves key feature extraction. To reduce the interference of redundant variables on the model training process and mitigate the adverse effects of high-dimensional sparsity on the training stability and generated data quality of generative adversarial networks, Pearson correlation coefficients are used to extract key features from the preprocessed database. Pearson correlation coefficients r The correlation coefficient ranges from -1 to 1. A positive correlation coefficient indicates a positive correlation between the two variables, while a negative correlation indicates a negative correlation. A larger absolute value indicates a stronger linear correlation between the variables. Correlation coefficients of 1, 0, and -1 correspond to three scenarios: perfect positive correlation, no linear correlation, and perfect negative correlation. The calculation formula is as follows: (2) In the formula: For variables x The average value of the data; For variables y The average value of the data.

[0024] Set the threshold to Calculate the Pearson correlation coefficient between each candidate variable and the target dam failure parameter, and then use the correlation coefficient to... r The absolute values ​​are sorted in descending order to select variables that are highly correlated with the target dam failure parameters, which are then used as the input feature set for the CTGAN model.

[0025] Example 5 Based on the above embodiments, S3 specifically involves: constructing a conditional table generative adversarial network model. The core architecture includes a generator. G With discriminator D Two network modules that engage in a game-like interaction. Generator. G The goal is to learn a mapping from the latent space to the original data space, subject to conditional vectors. c The constraint is to first sample a random noise vector from a prior distribution. z N ( 0 , I ), dimension set to d z .Will z With condition vector c Concatenate along the feature dimension, generator G The splicing method is as follows: (3) In the formula, z in The input vector to the generator; z It is a random noise vector; d z For random noise vectors z The dimension; dim( z in ) is a generator G The total dimension of the input vector; The total length after one-hot encoding of all discrete attribute variables. m The number of discrete variables, K j For the first j The number of categories for each discrete variable; The output neuron for the continuous variable uses a linear activation function to output the normalized predicted value of the continuous variable. The output neuron corresponding to the discrete variable uses the Softmax activation function to output the probability distribution of the discrete variable belonging to each category, and obtains the final discrete category pseudo-label through sampling. Generator G The complete synthetic sample output is: (4) In the formula, This is the complete augmented synthetic dam-break sample vector output by the generator. θ G For generator network parameters, This is a continuous variable vector in the generated dam-break characteristic parameters. This is the vector of discrete variables in the generated dam-break characteristic parameters.

[0026] Among them, the discriminator D The task is to distinguish between real samples and generated samples, and simultaneously check whether the sample matches a given conditional vector. This embodiment is designed as follows: the discriminator simultaneously receives one sample (real sample) and one generated sample (generated sample). s Or generate samples and the condition vector corresponding to the sample. c Input vector d in It is composed of three parts: the normalized continuous variable part of the sample, the discrete variable part in one-hot encoded form, and the known condition vector corresponding to the input sample. c .

[0027] Example 6 Based on the above embodiments, S4 specifically involves: applying the minimum-maximum adversarial loss function to the generator. G With discriminator D Alternating iterative training is performed. A minimum-maximum adversarial loss function is used for alternating iterative training, and weighting coefficients are set according to the frequency of occurrence of samples from each class to mitigate the impact of class imbalance on model training. The objective function is: (5) Where: min G max D This represents a min-max optimization process, i.e., a generator. G The discriminant improves the quality of generated samples by minimizing the objective function. D This enhances the ability to distinguish between real and generated samples by maximizing the objective function; V ( D , G The generator and discriminator use an adversarial loss function as the basis for game-theoretic optimization around this function. E x~PdataRepresents the distribution of real data P data The mathematical expectation, where x These are real samples taken from the original database; E z Pz Represents the prior noise distribution P z The mathematical expectation; D ( G ( z , c ), c ) indicates that the discriminator evaluates the generated samples. G ( z , c ) and its corresponding condition vector c The output probability. Weighting coefficients are set based on category frequency: (6) In the formula: ω k For the first k Class weighted coefficients; f k For the first k The proportion of class samples in the total sample.

[0028] Example 7 Based on the above embodiments, S5 specifically involves: after the model training is completed, the target condition vector... c With random noise vector z Input the trained generator G This generates augmented sample data that satisfies the target constraints. After sufficient adversarial training, the generator... G The mapping from the latent space to the original data space has been learned, and it can be used based on a given conditional vector. c Generate new samples that meet the corresponding conditions. At this point, specify the required condition vector and randomly sample from the noise distribution, then input it into the generator to generate augmented data in batches. Augmented data includes breach flood peak flow (…). Q p ), ulcer depth ( h b ), average width of the ulcer ( B ave ), duration of the dam breach ( T f Variables such as , the above parameters are the core indicators for evaluating the severity and scope of the consequences of a dam failure.

[0029] Example 8 Based on the above embodiments, S7 specifically involves performing a statistical consistency analysis between the augmented sample data and the original database. To more accurately determine whether the augmented data retains the original relevant structure, this embodiment also introduces the following two statistical indicators: Mean absolute deviation: (7) Root mean square error: (8) In the formula: RMSE The root mean square error (RMSE) is the value of the data. A smaller value indicates that the augmented data and the original data have a more consistent correlation structure. RMSE It exhibits high fidelity when the value is less than 0.15; m The number of variables involved in the analysis (such as peak flood discharge, breach depth, average breach width, dam break duration, etc.). For all i and j Perform a double summation to include the correlation coefficients between all different pairs of variables; variables in the original database i With variables j The correlation coefficient between them; To enhance the variables in the data i With variables j The correlation coefficient between them.

[0030] in, MAD The closer it is to 0, the more consistent the related structure of the augmented data and the original data. RMSE It is more sensitive to larger deviations and can effectively detect local correlation distortions. If MAD <0.1 and RMSE If the value is less than 0.15 and there is no significant difference in color distribution in the heatmap, then the augmented data is considered to have successfully preserved the dependencies between the original variables and has high fidelity.

[0031] Example 9 This embodiment takes a pre-constructed earth-rock dam failure case database as an example to provide a method for enhancing the earth-rock dam failure database based on conditional table generative adversarial networks, specifically: (1) The original database is established using a generation structure suitable for categorical variables.

[0032] Taking a pre-existing database of earth-rock dam failure cases as an example, the database contains data on 294 earth-rock dam failure instances, including information such as failure time, failure mode, dam type, dam height, dam axis length, dam crest width, average dam width, reservoir capacity, water level above the breach, reservoir capacity above the breach, breach depth, average breach width, peak flow rate, and failure duration. In the database statistics, failure modes are categorized only into overtopping failures and failures caused by seepage, such as... Figure 3 As shown. In the statistics of dam types, data on homogeneous dams (HD), core-wall dams (CD), and face-panel dams (FD) are classified, as follows: Figure 4 As shown, regarding dam failure modes, 72.45% of dam failures were due to overtopping caused by rainfall or landslides, 23.81% were due to seepage failure, and the failure mode of the remaining 3.74% of dam failure cases could not be recorded. Regarding dam type analysis, 79.25% of the dams were homogeneous dams (HD), 14.97% were core-wall dams (CD), and concrete-faced rockfill dams (FD) accounted for 2.38%, with 3.40% of the dam data lacking relevant dam type information.

[0033] In this embodiment, the data that primarily influences the enhanced processing method includes the breach flood peak flow. Q p 133 data points, such as Figure 5 As shown, it also includes the depth of the ulcer. h b Data points: 188, average ulcer width ( B ave ) 98 data points and the duration of the dam breach T f The data consists of 64 continuous variables, with variables having a missing rate exceeding 30% being removed. Outliers were identified using box plots and corrected or deleted based on practical physical meanings, such as the average breach width not exceeding the dam crest length. One-hot encoding was used for discrete variables. The dam breach mode was selected as overtopping dam breach, with two types (overtopping and seepage failure), encoded as [1,0] or [0,1]. The dam types were categorized as homogeneous dams, core-wall dams, and face dams, encoded as [1,0,0], [0,1,0], and [0,0,1]. The one-hot codes of all discrete variables were concatenated to obtain the conditional vector. c For continuous variables, normalization is applied, mapping all continuous variables to the interval [0,1].

[0034] (2) Key feature extraction based on Pearson correlation coefficient.

[0035] Peak flow rate of the breach Q p Taking the example as the target variable, calculate its Pearson correlation coefficient with each candidate variable: 1) Peak flow of the breach and reservoir capacity above the breach. Vw correlation coefficient r The correlation coefficient is 0.85, indicating a strong positive correlation; 2) it is related to dam height. h d correlation coefficient r The correlation coefficient was 0.62, indicating a moderate positive correlation; 3) it was correlated with the average width of the ulcer. B ave correlation coefficient r The correlation coefficient with the dam-break mode (overtopping code is 1, seepage failure is 0) is 0.58; 4) The correlation coefficient with the dam-break mode (overtopping code is 1, seepage failure is 0). r The value is 0.47. Set the threshold | r |≥0.3, filter out those with Q p The relevant feature set. Similarly, for h b , B ave , T f Calculate them separately, and take the union to get the final input feature set. The dimension is reduced from the original 15 to 9, which effectively reduces redundancy.

[0036] (3) Construction of the adversarial network model for generating conditional tables.

[0037] This implementation uses the CTGAN framework, generator G and discriminator D All are fully connected neural networks. The specific structure and hyperparameter settings are as follows: Selecting condition vectors c The one-hot encoding for the overtopping dam failure mode is [1, 0]. Random noise. z Sampling from a standard normal distribution, dimension d z Set to 128. Generator input. z in From noise vector z and condition vector c The network is constructed by concatenation, resulting in an input dimension of 128 plus a conditional vector dimension of 2, totaling 130. The generator network consists of an embedding layer of 50 dimensions, followed by two hidden layers, each with 256 neurons, using ReLU activation, and finally connected to the output layer. In the output layer, consecutive output nodes correspond to the peak flow rate of the breach. Q p , depth of the ulcer h b Average width of the ulcer B ave and the duration of the dam breach T fThe normalized value; the probability of the two classes corresponding to the dam-break mode at the discrete output node, using the Softmax activation function. The input of the discriminator consists of three parts: continuous variables from real samples (or generated samples), one-hot codes of discrete variables, and a condition vector. c There are four continuous variables, a discrete one-hot encoding length of 2, and a conditional vector length of 2. Therefore, the total input dimension of the discriminator is 4 + 2 + 2, which equals 8. The discriminator network contains two hidden layers, each with 256 neurons and the LeakyReLU activation function. The output layer is a single node that uses the Sigmoid activation function to output the probability.

[0038] (4) Calculate and iteratively optimize based on the minimum-maximum adversarial loss function.

[0039] Fixed generator parameters θ G A batch of samples was sampled from the real dataset. x i , c i}, sample a batch of random noise from the noise distribution { z j}, and obtain the generated sample through the generator { j} Calculate the discriminator loss and update θ D Fixed discriminator parameters θ D Resample the noise and conditional vector, calculate the generator loss (i.e., the discriminator's judgment error on the generated samples), and update... θ G The above steps are repeated until the preset number of iterations is reached or the discriminator loss converges.

[0040] (5) Quantify the output and analyze the enhanced data results.

[0041] Peak flow rate at the breach ( Q p The number of data points increased from 133 to 524, and the depth of the breach ( h b The number of data points increased from 188 to 752, and the average width of the breach ( B ave The number of data points increased from 98 to 392, and the duration of the dam breach ( T f The number of data items increased from 64 to 252 (see Table 1).

[0042] Table 1 Comparison of raw data and augmented data

[0043] Based on a comparison of the raw and enhanced data, i) the peak flow rate at the breach. Q p The average value of the original data is 4614.97 m³ / s, and the average value of the augmented data is 4608.65 m³ / s. The relative deviation is less than 0.14%, and the maximum, minimum, and median values ​​are all within a reasonable range, indicating that the generator accurately captured the values. Q p Distribution characteristics; ii) Ulcer depth h b The original mean was 7.74m, and the augmented mean was 9.86m, a slight increase but with a consistent overall distribution. This may be due to the generator's interpolation expansion in rare high-value regions, consistent with the expectation of augmenting small sample data; iii) Original data B ave The original mean was 41.08m, and the augmented mean was 53.53m. The generated data maintained the unimodal skewed shape of the original distribution, and the central trend remained stable; iv) Original data T f The original mean was 1.31 h, and the augmented mean was 1.17 h, almost overlapping. Furthermore, the minimum (0.01 h) and maximum (9.44 h) values ​​in the augmented data covered the range of the original data, providing richer sample diversity. In addition, auxiliary variables such as water level above the breach and reservoir capacity in the augmented data also exhibited similar statistical patterns to the original data, validating the effectiveness of the condition generation strategy.

[0044] (6) Perform statistical consistency analysis on the enhanced sample data and the original database.

[0045] Figure 6 Demonstrated the peak flow of the breach Q p Correlation comparison. In the original data, with Q p The variable with the strongest correlation is the reservoir capacity above the breach ( V w The correlation coefficient is approximately 0.85; the augmentation data generated by CTGAN also exhibits... Q p and V w The correlation between the height and the value of the data is positive (approximately 0.82), and the direction and intensity of the correlation with other variables (such as dam height and breach width) are consistent with the original data. Figure 7 The depth of the ulcer h b A comparison of heatmaps. In the original data, h b With storage capacity ( VThe correlation between the reservoir capacity and dam height is moderate (approximately 0.6–0.7); the reservoir capacity in the enhanced data is mainly concentrated around 10. 6 ~10 8 Within the m³ range, its correlation structure matches the original data well, with no spurious strong correlations or sign reversals. Figure 8 Average width of the ulcer B ave Heatmap comparison. In the original data... B ave It shows a positive correlation with the dam break mode (overtopping); the proportions of overtopping and seepage failure in the enhanced data are 50.77% and 49.23%, respectively, which is more balanced than the overtopping-dominated distribution in the original data. However, the correlation heatmap between variables still maintains the original trend, indicating that CTGAN can learn the real dependencies under imbalanced data. Figure 9 Duration of the dam breach T f Heatmap comparison. In the original data... T f It is positively correlated with reservoir capacity and breach width, and also has some correlation with dam failure patterns. Figure 10 The two heatmaps are a thermal comparison of variables related to the peak flow rate of the dam break. The color blocks and the distribution of positive and negative correlations in the two heatmaps are completely matched, and the variable correlation structure is not distorted. Figure 11 The graph shows a thermal comparison of variables related to the depth of the breach. The original data shows a moderate positive correlation with the dam height and total reservoir capacity. The magnitude and trend of the correlation coefficients in the enhanced data are consistent with the original data, and no spurious strong correlations or reversals of correlation signs are observed. Figure 12 The heatmap shows the correlation between variables related to the average width of the breach. The original data showed a high proportion of overtopping dam breach samples and an imbalance in categories. After enhancement, the proportion of overtopping and seepage failure samples tended to be balanced. However, the correlation strength and trend between Bave and each variable in the heatmap were completely consistent with the original database, proving that the weighted CTGAN does not distort the inherent correlation of variables while balancing the category distribution. Figure 13 This is a heat map comparing variables related to the duration of the dam breach; the original data shows a positive correlation between the duration of the dam breach and the reservoir capacity and the breach width, while the enhanced data shows very small deviations in the color block distribution and correlation coefficient values ​​of the heat map.

[0046] Peak flow rate of the breach Q p Above the breach capacity V w Taking consistency analysis as an example, in the original data The value is 0.85, in the augmented data. The absolute deviation between the two values ​​is only 0.03, which is far less than the threshold of 0.1. (This is achieved through...) Figure 10-13 A comparison of the heatmaps reveals a high degree of similarity in the distribution of color blocks between the two heatmaps, and calculations show that... MAD =0.036、 RMSE =0.052, all meeting the consistency requirements. By comparing the distribution of the original data and the augmented data, the following conclusions can be drawn: (1) The original small sample database (such as Q p Expanded from 133 to 524, h b (1) The data was expanded from 188 to 752, providing sufficient training data for the downstream prediction model; (2) The generated data were highly consistent with the original data in terms of univariate statistical characteristics (mean, variance, quantiles) and multivariate correlation structure, and the heat map comparison intuitively proved the fidelity; (3) The introduction of class frequency weighted loss alleviated the sample imbalance problem of the dam failure mode; (4) The consistency analysis obtained MAD and RMSE All of them meet the consistency requirements, do not introduce false correlations, and do not distort the original correlation structure, and can be fully used for subsequent dam failure risk modeling.

Claims

1. A database augmentation method for earth-rock dam breakage based on CTGAN, characterized in that, include: Construct and preprocess the original database of earth-rock dam failure, and screen key condition variables; A conditional generative adversarial network is constructed and trained using conditional variables as constraints, with class-frequency weighting during training; target conditions and noise are input into the trained generator to generate enhanced samples. Heatmaps and statistical bias comparisons were performed between the enhanced samples and the original data to enhance the database of earth-rock dam failures.

2. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 1, characterized in that, Includes the following steps: S1. Construct a raw database of earth-rock dam failures. The samples in the raw database include basic information parameters and failure characteristic parameters. Both the basic information parameters and failure characteristic parameters include discrete and continuous variables. Preprocess the raw database to obtain a standardized database. S2, Based on the standardized database, a correlation analysis method is used to calculate the correlation coefficient between each basic information parameter and each dam-break characteristic parameter, and basic information parameters that meet the preset correlation threshold with each dam-break characteristic parameter are selected as the key condition variable set; the key condition variable set includes discrete variables and continuous variables; S3, Encode and concatenate the discrete variables in the key condition variable set to form a target condition vector; Establish a conditional generative adversarial network model containing a generator G and a discriminator D; The generator G takes the result of concatenating the target condition vector with random noise as input and outputs data samples. The discriminator D takes the result of concatenating the sample data with the corresponding target condition vector as input and outputs the authenticity judgment result. S4, the generator G and the discriminator D are trained alternately and iteratively using an adversarial loss function; During training, weighting coefficients are set according to class frequency to obtain the trained generator G and discriminator D; S5, input the set target condition vector and random noise into the trained generator G to generate augmented sample data that satisfies the target condition constraints; S6. Perform statistical consistency analysis on the enhanced sample data and the normalized database. Verify the fidelity of the enhanced sample data by comparing the correlation heatmap and evaluating the data with preset statistical indicators, thereby enhancing the database of earth-rock dam failures.

3. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The basic information parameters include dam type and dam height. h d dam crest width W and storage capacity V The dam failure characteristic parameters include dam type. DT Defeat mode FM Water level above the breach h w Reservoir capacity above the breach V w , depth of the ulcer h b Average width of the ulcer B ave Peak flow rate during dam break Q p and the duration of the dam breach T f The discrete variable in the basic information parameters is the dam type, and the continuous variable is the dam height. h d dam crest width W and storage capacity V The discrete variable in the dam failure characteristic parameters is the dam type. DT and the collapse mode FM The continuous variable is the water level above the breach. h w Reservoir capacity above the breach V w , depth of the ulcer h b Average width of the ulcer B ave Peak flow rate during dam break Q p and the duration of the dam breach T f The dam type and dam type mentioned DT All of these include homogeneous dams, core wall dams, and panel dams.

4. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The preprocessing steps described in S1 are as follows: missing data in the original database of earth-rock dam failure are filled using the mean imputation method; abnormal data are identified using the box plot method with a threshold of 1.5 times the interquartile range; and corrected using the cubic spline interpolation method. Discrete variables are encoded using one-hot encoding, and continuous variables are normalized using the following formula: （1） In the formula, x′ These are the normalized variable values; x The original variable value; x min and x max These are the minimum and maximum values ​​of the variable in the original database of earth-rock dam failures, respectively.

5. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The correlation analysis method described in S2 is the Pearson correlation coefficient method, which is calculated as follows: （2） Where r is the Pearson correlation coefficient, x i The value of the i-th sample of the basic information parameter, y i The value of the i-th sample of the dam failure characteristic parameter; The basic parameter is the mean of all samples. The mean of all samples for dam failure characteristics is used; based on the standardized database, the correlation coefficient between each basic information parameter and each dam failure characteristic parameter is calculated, and the absolute value threshold of the correlation coefficient |r|≥0.3 is set. Basic information parameters that meet this threshold are selected as the set of key condition variables.

6. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, In S3, the generator G employs a fully connected neural network. The random noise is sampled from a standard normal distribution and has a dimension of 128. The target conditional vector is concatenated with the random noise and used as the input to the generator G. The concatenation is calculated as follows: （3） In the formula: z in The input vector to the generator; z It is a random noise vector; d z For random noise vectors z The dimension; dim( z in ) is a generator G The total dimension of the input vector; The total length after one-hot encoding of all discrete attribute variables. m The number of discrete variables, K j For the first j The number of categories for each discrete variable; The generator G uses the ReLU activation function in its hidden layer, and the neurons in the output layer corresponding to continuous variables use the linear activation function, while the neurons corresponding to discrete variables use the Softmax activation function. The output data sample is represented as follows: （4） In the formula, This is the complete augmented synthetic dam-break sample vector output by the generator. θ G For generator network parameters, This is a continuous variable vector in the generated dam-break characteristic parameters. This is the vector of discrete variables in the generated dam-break characteristic parameters.

7. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The alternating iterative training described in S4 includes: fixing the parameters of the generator G, updating the parameters of the discriminator D using samples from the normalized database and the enhanced samples output by the generator G; fixing the parameters of the discriminator D, updating the parameters of the generator G through backpropagation of the discrimination results, and repeating this process until a preset number of iterations or the loss function converges, so that the enhanced sample data generated by the generator G approximates the true sample distribution of the normalized database; The adversarial loss function is expressed as: （5） In the formula, min G、 max D To illustrate the min-max optimization process, the generator G The discriminant improves the quality of generated samples by minimizing the objective function. D This enhances the ability to distinguish between real and generated samples by maximizing the objective function; V ( D , G ) represents the adversarial loss function; E x~Pdata Represents the distribution of real data P data The mathematical expectation, where x These are real samples taken from the original database; E z Pz Represents the prior noise distribution P z The mathematical expectation; D ( G ( z , c ), c ) indicates that the discriminator evaluates the generated samples. G ( z , c ) and its corresponding condition vector c The output probability.

8. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The weighting coefficients described in S4 are set as follows: the weighting coefficients for each category are inversely proportional to the frequency of occurrence of samples of that category in the normalized database. The weighting coefficients are expressed as follows: （6） In the formula: ω k For the first k Class weighted coefficients; f k For the first k The proportion of class samples in the total sample.

9. The method for enhancing the database of earth-rock dam breakage based on CTGAN according to claim 2, characterized in that, The preset statistical indicators mentioned in S6 include mean absolute deviation and root mean square error, and the calculation formulas for the mean absolute deviation and the root mean square error are as follows: （7） （8） In the formula, MAD The mean absolute deviation, RMSE This is the root mean square error; m This represents the total number of variables involved in the correlation analysis. For all i and j Perform a double summation to include the correlation coefficients between all different pairs of variables; Variables in the original database for earth-rock dam failure i With variables j The Pearson correlation coefficient between them; The Pearson correlation coefficient between variables i and j in the augmented sample data generated by the conditional generative adversarial network model.

Images