A Method for Evaluating Historical Rain Gauge Data Based on Generative Adversarial Networks and Transformer Models
By generating virtual rainfall data using generative adversarial networks and Transformer models, the problem of data distortion from rain gauges is solved, improving the accuracy and reliability of rainfall data and providing quantitative assessment of data errors and anomaly detection capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGTZE RIVER WATER CONSERVANCY COMMISSION HYDROLOGY BUREAU UPPER YANGTZE RIVER HYDROLOGY & WATER RESOURCES SURVEY BUREAU
- Filing Date
- 2025-05-20
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, rainfall data from rain gauges suffers from data distortion due to sensor errors, observation environment errors, and human errors, and there is a lack of systematic evaluation methods.
A generative adversarial network and a Transformer model are used to generate virtual rainfall data through generator and discriminator training. The Transformer model is then used to extract rainfall features, and evaluation indicators are calculated by combining Pearson correlation coefficient and cosine distance to construct an evaluation method for historical data of rain gauge stations.
It improves the accuracy and reliability of rainfall data, provides a macro-level quantitative assessment of data errors, and can effectively detect and supplement data anomalies.
Smart Images
Figure CN120542970B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of rain gauge evaluation technology, and in particular to a method for evaluating historical rain gauge data based on generative adversarial networks and the Transformer model. Background Technology
[0002] Rain gauge data quality is the core foundation of hydrological monitoring. Accurate rainfall data can improve the accuracy of hydrological forecasts and disaster warnings, enhance the practicality of hydrological models, and improve the accuracy of water resource assessments.
[0003] Currently, rainfall data is mostly collected by various sensors installed on rain gauges. The reasons for inaccurate rainfall data may include: 1. Sensor errors, such as measurement errors of tipping bucket rain gauges; 2. Observation environment errors, such as errors caused by obstruction; 3. Human errors, such as errors caused by manual recording or transmission errors; 4. Data loss, such as data loss caused by equipment failure.
[0004] To address the above issues, current research focuses primarily on missing data. Determining whether missing data exists is relatively straightforward, employing methods such as data sequence continuity testing, best-case checks, and data integrity statistics. However, even with complete data, there is no systematic method for assessing data distortion caused by sensor errors, observational environment errors, and human error. Summary of the Invention
[0005] This invention discloses a method for evaluating historical rain gauge data based on generative adversarial networks and the Transformer model. The specific method is as follows:
[0006] Obtain historical rainfall data from calibrated rain gauge stations. The calibrated historical rainfall data is the historical data with the highest accuracy in rainfall measurement.
[0007] The generator generates the first set of calibrated virtual rainfall data based on the calibrated historical rainfall data.
[0008] The discriminator judges the authenticity of the calibrated historical rainfall data and the first calibrated virtual rainfall data;
[0009] A Transformer model is trained using calibrated historical rainfall data, and the trained Transformer model outputs a second calibrated virtual rainfall data.
[0010] Based on the calibrated historical rainfall data, the first calibrated virtual rainfall data, and the second calibrated virtual rainfall data, the calibration evaluation index is calculated.
[0011] Obtain the first to N other historical rainfall data, input them into the trained generator and Transformer model, and calculate the other evaluation indicators of the first to N other historical rainfall data based on the output results of the generator and Transformer model.
[0012] Based on the calibrated evaluation indicators and several other evaluation indicators, historical data from rain gauge stations are evaluated.
[0013] Furthermore, historical rainfall data from calibrated rain gauges are obtained, using the following specific methods:
[0014] Obtain all historical rainfall data from rain gauge stations;
[0015] Historical rainfall data is read in segments using a fixed-length sliding window to generate several rainfall arrays.
[0016] The Pearson correlation coefficient between each rainfall array and the remaining rainfall arrays was calculated and summed to serve as the selection criterion for rainfall arrays.
[0017] Change the sliding window length, repeatedly read historical rainfall data, and calculate the selection criteria for the rainfall array;
[0018] If the highest selected index remains unchanged after a preset number of repetitions, the rainfall array corresponding to the highest selected index will be used as the calibration historical rainfall data.
[0019] Further, the first calibration virtual rainfall data is generated, specifically as follows:
[0020] Based on the calibrated historical rainfall data, data with similarity above a preset value are selected from all historical rainfall data as training samples.
[0021] The generator is trained using training samples, and the generator continuously outputs virtual rainfall data.
[0022] The generated virtual rainfall data is mixed with the training samples and then input into the discriminator, which then distinguishes between real and virtual rainfall.
[0023] Iteratively update the generator and discriminator parameters until the value function converges;
[0024] The first calibration virtual rainfall data is generated using the converged generator.
[0025] Furthermore, the Transformer model is trained using calibrated historical rainfall data, as follows:
[0026] Convert each word in the training samples into a vector;
[0027] Embed positional information and preserve word order to complete positional encoding;
[0028] In the encoder, the encoded data is projected into three spaces: Q, K, and V, and Q, K, and V are divided into h heads; the self-attention score is calculated, and the outputs of all heads are concatenated; the outputs of all heads are processed by a feedforward network to extract data features.
[0029] In the decoder, the multi-head self-attention score is masked; the decoder's Q is used to query the encoder's K and V; the decoder output is mapped to the vocabulary size dimension through the same feedforward network as the encoder, generating the word probability distribution at each position; and the final sequence is output through a greedy search.
[0030] Furthermore, the calibration evaluation indicators are calculated using the following method:
[0031] The cosine distance between the calibrated historical rainfall data and the first calibrated virtual rainfall data, plus the cosine distance between the calibrated historical rainfall data and the second calibrated virtual rainfall data, is used as the calibration evaluation index.
[0032] Furthermore, the remaining evaluation indicators for the first to Nth historical rainfall data are calculated using the following method:
[0033] Randomly remove some historical rainfall data;
[0034] The remaining historical rainfall data are input into the trained generator and Transformer model respectively, and the generator and Transformer model output the corresponding first remaining virtual rainfall data and second basic virtual rainfall data respectively.
[0035] The cosine distance between the complete remaining historical rainfall data and the first remaining virtual rainfall data, plus the cosine distance between the complete remaining historical rainfall data and the second remaining virtual rainfall data, is used as the evaluation index for the remaining historical rainfall data.
[0036] Furthermore, based on the calibrated evaluation indicators and several other evaluation indicators, historical data from rain gauge stations are obtained for evaluation. The specific methods are as follows:
[0037] The absolute value of the difference between the cumulative calibrated evaluation index and all other evaluation indicators is divided by the number of other evaluation indicators. The result is then compared with the evaluation grading threshold to obtain the evaluation grading of historical rain gauge data.
[0038] Due to the adoption of the above technical solutions, the present invention has the following beneficial effects:
[0039] 1. This invention trains a deep learning model by selecting data with high accuracy. The parameter architecture of the deep learning model records the rainfall characteristics of rain gauges. Then, the virtual data generated by the deep learning model is used as the criterion for judging whether the historical rainfall data is distorted. This method solves the technical problem of difficulty in judging rainfall distortion.
[0040] 2. This invention employs generative adversarial networks and the Transformer model. These two models use different principles when fitting data features, and by combining the advantages of both models, the reliability of the final judgment is increased.
[0041] 3. This invention constructs evaluation indicators, which macroscopically quantifies the error in rainfall data, providing a data foundation for evaluation.
[0042] 4. In selecting historical rainfall data for calibration, this invention uses time windows of different lengths to read the data, which can find calibration data with smaller errors and improve the model training effect.
[0043] 5. In generating training data, this invention identifies data samples similar to the calibration data, which can improve the quality of the training data.
[0044] 6. When using generative adversarial networks and Transformer models to predict and complete rainfall data, the data can be selected to complete the data if there are abrupt changes, which effectively achieves data anomaly detection.
[0045] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0046] The accompanying drawings of this invention are described below.
[0047] Figure 1 This is a schematic diagram of the overall process of the present invention.
[0048] Figure 2 A schematic diagram illustrating the principle of generating adversarial networks.
[0049] Figure 3 This is a schematic diagram illustrating the principle of the Transformer model. Detailed Implementation
[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0051] A method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models, the specific steps of which are as follows:
[0052] S1. Obtain historical rainfall data from rain gauge calibration stations. , To calibrate historical rainfall data centrally Rainfall amount.
[0053] In step S1, historical rainfall data from rain gauge calibration is obtained. The specific steps are as follows:
[0054] S11. Obtain all historical rainfall data from rain gauge stations. ,in The first in all historical rainfall data Rainfall amount, .
[0055] S12. Read historical rainfall data in segments using a first fixed-length sliding window to generate several rainfall arrays. , ;in, For the first In the rainfall array, the first The rainfall amount, i.e., the first fixed length of the sliding window is .
[0056] S13. Calculate the Pearson correlation coefficient between each rainfall data set and the remaining rainfall data sets, using the following formula:
[0057]
[0058] In the formula, For the first The rainfall array and the first Pearson correlation coefficients for rainfall data sets. It is covariance. It is the standard deviation. The mean;
[0059] For each rainfall data set, the Pearson correlation coefficients associated with it are summed, calculated using the following formula:
[0060]
[0061] For the first The selection criteria for each rainfall array are as follows: Of the selected indicators, the one with the highest value is retained as the candidate. The first fixed length of the sliding window is The optimal alternative target at that time.
[0062] S14. Change the length of the sliding window to the second fixed length. Generate several rainfall arrays , ;in, For the first In the rainfall array, the first Rainfall amount; Record As the second fixed length of the sliding window The optimal alternative target at that time;
[0063] S15. Repeatedly change the length of the sliding window until... , ...the maximum value among the selected indicators did not change within the preset number of length changes, so the rainfall array corresponding to the highest selected indicator was used as the calibration historical rainfall data. .
[0064] S2. Based on the calibrated historical rainfall data, a first calibrated virtual rainfall data is generated using a generative adversarial network.
[0065] In step S2, the generative adversarial network is trained, and the specific steps are as follows:
[0066] S21. Based on the calibrated historical rainfall data, select data with a similarity value above the preset value from all historical rainfall data as training samples.
[0067] In step S21, the similarity calculation is the same as in step S13.
[0068] S22. Train the generator with training samples, and the generator continuously outputs virtual rainfall data.
[0069] S23. After mixing the generated virtual rainfall data with the training samples, input the mixture into the discriminator, which then performs real-virtual discrimination.
[0070] S24. Iteratively update the generator and discriminator parameters until the value function converges.
[0071] S25. Generate the first calibration virtual rainfall data using the converged generator. .
[0072] The value function formula for generative adversarial networks is as follows:
[0073] Fixed generator Optimize the discriminator :
[0074]
[0075] Fixed discriminator Optimize generator :
[0076]
[0077] In the formula, For the discriminator to analyze real data The discrimination probability; For the generator from random noise The generated data; The distribution of real data; For random noise Distribution; This is the expected value.
[0078] During training, the generator is fixed. Optimize the discriminator and fixed discriminator Optimize generator The two actions are performed alternately until the training reaches Nash equilibrium, i.e., the generator. The generated data distribution Approximates the true data distribution And the discriminator Unable to distinguish between real and fake data .
[0079] S3. Based on the calibrated historical rainfall data, generate the second calibrated virtual rainfall data using the Transformer model.
[0080] In step S3, the Transformer model is trained, and the specific method is as follows:
[0081] S31. Convert each word in the training samples into a vector.
[0082] S32. Embed position information and preserve word order to complete position encoding.
[0083] S33. In the encoder, the encoded data is projected into three spaces: Q, K, and V. Q, K, and V are divided into h heads. The self-attention score is calculated and the outputs of all heads are concatenated. The outputs of all heads are used to extract data features through a feedforward network.
[0084] S34. In the decoder, mask the multi-head self-attention scores; use the decoder's Q to query the encoder's K and V; through the same feedforward network as the encoder, map the decoder output to the vocabulary size dimension to generate the word probability distribution at each position; output the final sequence through a greedy search.
[0085] S35. Generate second-calibration virtual rainfall data using the Transformer model. .
[0086] In step S3, the formulas for calculating the self-attention mechanism query Q, key K, and value V are as follows:
[0087]
[0088]
[0089]
[0090] In the formula, For the input matrix, , For sequence length, For feature dimensions; , , This is a learnable weight matrix.
[0091] The formula for calculating attention score is as follows:
[0092]
[0093] In the formula, Calculate the correlation between tokens. Normalized weights.
[0094] The formula for calculating multi-head attention is as follows:
[0095]
[0096] Among them, attention head The calculation formula is as follows:
[0097]
[0098] This is an independent weight matrix for each head.
[0099] The specific formula for location coding is as follows:
[0100]
[0101]
[0102] In the formula, The position of the token. For dimensional indexing.
[0103] S4. The cosine distance between the calibrated historical rainfall data and the first calibrated virtual rainfall data, plus the cosine distance between the calibrated historical rainfall data and the second calibrated virtual rainfall data, is used as the calibration evaluation index. .
[0104] S5. Obtain the first to N remaining historical rainfall data, remove some data, and then complete the data using a generative adversarial network and a Transformer model respectively.
[0105] S6. Based on the completion results, calculate the remaining evaluation indicators for the first to Nth historical rainfall data.
[0106] In step S6, the remaining evaluation indicators for the first to Nth historical rainfall data are: .
[0107] S7. Calculate the evaluation of historical rain gauge data. The specific formula is as follows:
[0108]
[0109] In the above evaluation, the lower the grade, the higher the evaluation of the historical rainfall data, that is, the higher the accuracy.
[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models, characterized in that, The specific method is as follows: Obtain historical rainfall data from calibrated rain gauge stations; The generator generates the first set of calibrated virtual rainfall data based on the calibrated historical rainfall data. The discriminator judges the authenticity of the calibrated historical rainfall data and the first calibrated virtual rainfall data; A Transformer model is trained using calibrated historical rainfall data, and the trained Transformer model outputs a second calibrated virtual rainfall data. Based on the calibrated historical rainfall data, the first calibrated virtual rainfall data, and the second calibrated virtual rainfall data, the calibration evaluation index is calculated. Obtain the first to N other historical rainfall data, input them into the trained generator and Transformer model, and calculate the other evaluation indicators of the first to N other historical rainfall data based on the output results of the generator and Transformer model. Based on the calibrated evaluation indicators and several other evaluation indicators, historical data of rain gauges are evaluated. The specific method for obtaining historical rainfall data from calibrated rain gauges is as follows: Obtain all historical rainfall data from rain gauge stations; Historical rainfall data is read in segments using a fixed-length sliding window to generate several rainfall arrays. The Pearson correlation coefficient between each rainfall array and the remaining rainfall arrays was calculated and summed to serve as the selection criterion for rainfall arrays. Change the sliding window length, repeatedly read historical rainfall data, and calculate the selection criteria for the rainfall array; If the maximum value of the selected indicator remains unchanged after a preset number of repetitions, the rainfall array corresponding to the maximum value of the selected indicator will be used as the calibration historical rainfall data.
2. The method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models as described in claim 1, characterized in that, The specific method for generating the first calibration virtual rainfall data is as follows: Based on the calibrated historical rainfall data, data with similarity above a preset value are selected from all historical rainfall data as training samples. The generator is trained using training samples, and the generator continuously outputs virtual rainfall data. The generated virtual rainfall data is mixed with the training samples and then input into the discriminator, which then distinguishes between real and virtual rainfall. Iteratively update the generator and discriminator parameters until the value function converges; The first calibration virtual rainfall data is generated using the converged generator.
3. The method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models as described in claim 2, characterized in that, The Transformer model is trained using calibrated historical rainfall data, as follows: Convert each word in the training samples into a vector; Embed positional information and preserve word order to complete positional encoding; In the encoder, the encoded data is projected into three spaces: Q, K, and V, and Q, K, and V are divided into h heads; the self-attention score is calculated, and the outputs of all heads are concatenated; the outputs of all heads are processed by a feedforward network to extract data features. In the decoder, the mask multi-head self-attention score; The decoder's Q query queries the encoder's K and V; the decoder output is mapped to the vocabulary size dimension through the same feedforward network as the encoder, generating the word probability distribution at each position; the final sequence is output through a greedy search.
4. The method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models as described in claim 1, characterized in that, The specific method for calculating the calibration evaluation index is as follows: The cosine distance between the calibrated historical rainfall data and the first calibrated virtual rainfall data, plus the cosine distance between the calibrated historical rainfall data and the second calibrated virtual rainfall data, is used as the calibration evaluation index.
5. The method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models as described in claim 4, characterized in that, The remaining evaluation indicators for the first to Nth historical rainfall data are calculated using the following method: Randomly eliminate some of the remaining historical rainfall data; The remaining historical rainfall data are input into the trained generator and Transformer model respectively, and the generator and Transformer model output the corresponding first remaining virtual rainfall data and the second remaining virtual rainfall data respectively. The cosine distance between the remaining historical rainfall data and the first remaining virtual rainfall data, plus the cosine distance between the remaining historical rainfall data and the second remaining virtual rainfall data, is used as the remaining evaluation index for the remaining historical rainfall data.
6. The method for evaluating historical rain gauge data based on generative adversarial networks and Transformer models as described in claim 5, characterized in that, Based on the calibrated evaluation indicators and several other evaluation indicators, historical data from rain gauge stations are evaluated. The specific method is as follows: The absolute value of the difference between the cumulative calibrated evaluation index and all other evaluation indicators is divided by the number of other evaluation indicators. The result is then compared with the evaluation grading threshold to obtain the evaluation grading of historical rain gauge data.