Real-time correction method for CMA measured rainfall data based on radar echo data

Abstract

The application discloses a kind of based on radar echo data to CMA real-time correction method of measured rainfall data, including steps 1, obtain weather radar echo data and CMA measured rainfall data;Step 2, with ground measured rainfall station position as anchor point, select corresponding CMA data and radar echo data;Step 3, integrate radar echo data and CMA rainfall data, construct training dataset and verification dataset;Step 4, with Light GBM algorithm as core, model is constructed, and training dataset is used to train model;Step 5, through the model trained in real time, radar echo data is analyzed, and CMA data is compared to identify problem area, then the rainfall value calculated using radar echo is replaced, and the rainfall data after fusion is re-interpolated to obtain corrected rainfall data;The technical problems, such as precision error, of the existing technology by corresponding rainfall station measured data generated CMA grid layer are solved.

Description

Technical Field

[0001] This invention belongs to the field of rainfall monitoring technology, and in particular relates to a method for real-time correction of CMA measured rainfall data based on radar echo data. Background Technology

[0002] CMA data is raster layer data generated from measured rainfall data from rain gauge stations using the inverse distance interpolation method. Its accuracy depends on the accuracy and stability of the rain gauge station (point) measurement data. Rain gauge stations (points) are widely used at the county, township, and village levels by meteorological, flood control, hydrological, and land resources departments due to their characteristics of requiring no human supervision, simple operation, powerful functions, fast transmission, small footprint, high reliability, long lifespan, and stable continuous long-term operation. This has formed a network of rain gauge stations (points) for measuring precipitation, enabling relevant units to comprehensively, accurately, and promptly grasp local rainfall information during the flood season, thus improving flood prevention and disaster relief capabilities. However, in actual use, many grassroots rain gauge stations (points) have not played their due role. This is mainly reflected in deficiencies in the layout of the rain gauge network, prominent problems in station (point) site selection, improper installation, inadequate maintenance, and poor data transmission, resulting in inaccurate rainfall records, missing measurements, and data distortion, failing to accurately reflect the actual local and current rainfall situation. This also leads to accuracy errors in the CMA raster layer generated from the corresponding rain gauge data. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method for real-time correction of CMA measured rainfall data based on radar echo data, so as to solve the technical problems such as the accuracy error of the CMA raster layer generated by the corresponding rain gauge measured data in the prior art.

[0004] The technical solution of this invention is:

[0005] A method for real-time correction of CMA measured rainfall data based on radar echo data, the method comprising:

[0006] Step 1: Acquire weather radar echo data and CMA measured rainfall data;

[0007] Step 2: Using the location of the ground-based measured rain gauge station as the anchor point, select the corresponding CMA data and radar echo data;

[0008] Step 3: Integrate radar echo data and CMA rainfall data to construct training and validation datasets;

[0009] Step 4: Build a model based on the Light GBM algorithm and train the model using the training dataset;

[0010] Step 5: Perform real-time rainfall analysis on radar echo data using the trained model, compare with CMA data, identify problem areas, replace with rainfall values ​​calculated from radar echoes, and re-interpolate the fused rainfall data to obtain corrected rainfall data.

[0011] In step 2, when selecting the corresponding CMA data and radar echo data using the location of the ground-based rain gauge station as the anchor point, the rain gauge station is in normal working condition, and data measurement, transmission, and reporting are all normal. The CMA data format is 5 times that of the radar echo data. When taking the radar echo data, the data of 25 grids, 5 grids in each direction (5 grids in each direction) are taken from the center of the measured station as feature data.

[0012] Before constructing the training dataset, the data needs to be preprocessed. The preprocessing method is as follows: Since the CMA data grid represents a 5km*5km range, while the radar echo grid represents a 1km*1km range, when establishing the data mapping, one CMA grid corresponds to one radar echo grid matrix, and each matrix contains 25 grids. To obtain such a training dataset, the radar echo data and CMA historical rainfall data need to be associated and then the datasets merged. The association and merging method is as follows: data with the same timestamp are found from the radar echo data and CMA measured rainfall data as the merged dataset.

[0013] Step 4 describes the method of building a model based on the Light GBM algorithm and training the model using a training dataset. This includes: inputting the training dataset into the built Light GBM model in batches; dividing the training dataset and validation dataset into a certain proportion using pre-set parameters; continuously adjusting the parameter values ​​during model training; calculating and outputting the accuracy and loss function results of the model on the current data in this iteration of the training set while training; and continuing until the loss function reaches its minimum or the iteration count is completed, thus finally obtaining the trained model.

[0014] During the training of the LightGBM model, the weight values ​​are continuously adjusted according to the loss function. Each adjustment is determined by the learning rate, and the calculation formula is as follows:

[0015] .

[0016] The formula for the loss function is:

[0017]

[0018] The mean squared error loss function (MSE) measures the mean of the squared differences between predicted and observed values. The formula is:

[0019] .

[0020] When analyzing rainfall data from radar echoes, a range of error is defined to determine the accuracy of the precipitation values ​​derived from the radar echoes. Values ​​within this range are considered correct.

[0021] Beneficial effects of this invention:

[0022] This invention first preprocesses the raw radar echo data to obtain hourly radar echo layer data. Ground-based measurement stations (in normal working condition) are manually selected as sampling points, and radar echo data and measured rainfall data for historical periods are collected at these points. These are then integrated to form a radar echo-rainfall dataset for model training. The ensemble learning algorithm model is trained based on this training dataset. During training, the experimental results are organized and compared, and the relevant parameters of the model are continuously adjusted until the optimal model and parameters are obtained. The experimental data and results are compared and analyzed to arrive at the final conclusion. After the model stabilizes, rainfall analysis can be performed on the radar echo data in real time. By comparing with CMA data, problem areas are identified, and rainfall values ​​calculated from the radar echoes are used to replace the problematic areas. The fused rainfall data is then re-interpolated to obtain the corrected rainfall data.

[0023] Features of this invention:

[0024] This invention integrates precipitation data recorded by ground rain gauges (CMA data) with radar echo data for precipitation analysis, which helps to correct CMA errors and improve the accuracy of CMA data.

[0025] This invention uses machine learning methods to analyze the rainfall intensity level of radar echo data, inverts the radar echo data into rainfall data, and cross-compares it with CMA data to correct the CMA data.

[0026] This invention combines radar echo data with precipitation data collected at corresponding times, and uses an ensemble learning algorithm for learning and training to construct a strong model that ensures accuracy while also having good practicality to meet real-world needs.

[0027] This invention solves the technical problems of accuracy errors in CMA raster layers generated from actual rain gauge data in existing technologies. Detailed Implementation

[0028] The premise of this invention is that radar echo data can reflect the intensity of actual precipitation to a certain extent. By comparing the CMA measured precipitation raster layer and the radar echo raster layer, a positive correlation is found between the actual precipitation values ​​and the radar echo reflectivity data. In areas with strong rainfall intensity, the radar echo reflectivity values ​​are also high, and the two show similarity in terms of region and shape.

[0029] The implementation method of this invention includes:

[0030] The first step is to acquire weather radar echo data (1km*1km grid) and CMA measured rainfall data (5km*5km grid). The former is mainly used to obtain radar echo characteristics, while the latter is used as tag data for the corresponding time and point.

[0031] The second step is data sampling. Using the location of the ground-based rain gauge station as the anchor point, corresponding CMA data and radar echo data are selected. The ground-based rain gauge station serving as the anchor point needs to be manually verified to be in normal working order, with data measurement, transmission, and reporting all functioning correctly. Considering that the CMA data format is five times that of the radar echo data (one-sided), when acquiring radar echo data, data from 25 grids (5 grids horizontally and 5 grids vertically) centered on the ground-based rain gauge station are taken as feature data.

[0032] The third step, after data collection is complete, is to integrate radar echo data and CMA rainfall data to construct a training dataset.

[0033] The fourth step is to build an ensemble learning model, using the Light GBM algorithm as the core, and train the model using a training dataset. After the model is trained, its accuracy needs to be validated. Only after successful validation can it be used for precipitation estimation.

[0034] CMA data originates from ground-based rain gauge stations. However, in actual use, these stations suffer from inaccuracies due to issues such as poor layout, problematic site selection, inadequate installation, insufficient maintenance, and poor data transmission. This leads to errors in the accuracy of the CMA raster layer data. After the aforementioned model is built and trained, it can be used to invert precipitation values ​​from radar echo data. The inverted precipitation values ​​are then used to replace the problematic CMA data.

[0035] Data source:

[0036] The radar echo data file used in this invention is a grid data file that has already merged radar echoes of various types and bands. The radar scans the elevation angle above it every 5 minutes, rotating vertically 360 degrees at each elevation angle to collect echo data from the upper region. Depending on the different atmospheric conditions above the radar scanning range, different echo values ​​will be generated, thus forming a radar echo reflectivity data file. The radar echoes from different regions and time periods are then summarized and merged to obtain the data file used in this invention.

[0037] CMA measured rainfall data is reported by ground rain gauges at 15-minute intervals; the raw data is point data, and the CMA raster layer is obtained through inverse distance interpolation, which is the raster data used in this invention.

[0038] Data preprocessing:

[0039] In actual precipitation measurements, uncontrollable human error, instrument malfunctions, and data transmission and storage issues can lead to data containing noise, outliers, and missing values. Therefore, we must clean the source data using relevant background knowledge of precipitation measurement and select normally functioning stations as sampling points to ensure data reliability. Only in this way can we obtain an accurate and efficient model.

[0040] To train the model, a radar echo training dataset is needed. This dataset is divided into two parts: attributes and their corresponding results. The results part contains rainfall data collected at different times from meteorological stations at different coordinates in the CMA historical rainfall data. The attributes part contains the radar echo reflectivity values ​​of multiple grids above the corresponding meteorological station coordinates. Since CMA data grids represent a 5km x 5km area, while radar echo grids represent a 1km x 1km area, the data mapping is established so that one CMA grid corresponds to one radar echo grid matrix, with each matrix containing 25 grids. To obtain such a training dataset, the radar echo data needs to be associated with the CMA historical rainfall data and then the datasets merged.

[0041] Both radar echo data and CMA historical rainfall data are collected at specific time intervals. Therefore, we can consider using time as an attribute to associate and merge the two types of data. The two types of data are then merged to generate a training dataset.

[0042] Traditional estimation methods lack consideration of the spatiotemporal distribution of radar echoes. In this invention, the CMA data grid format is 5km*5km, while the radar echo grid format is 1km*1km. This means that the 25 radar echo grid data actually reflect only one measured rainfall value; both are essentially area data, so theoretically, the spatiotemporal distribution characteristics on a plane should be considered. Furthermore, radar data is susceptible to interference; analyzing only the reflectivity value of a single point inevitably leads to distortion and fails to accurately reflect the rainfall situation. Therefore, this invention utilizes machine learning methods to fully consider the spatiotemporal distribution of radar echoes, establishing a mapping from CMA point rainfall to the radar echo matrix, and constructing a machine learning model.

[0043] Current GBDT tools are mostly decision tree algorithms based on pre-sorted methods, such as XgBoost. These algorithms require pre-sorting the data, which means storing not only the feature values ​​but also the sorted features, consuming significant memory. Secondly, they incur substantial time overhead, as gain calculations are required during traversal, which is very costly. Furthermore, existing methods are not optimized for caching and cannot be improved by optimizing for cache.

[0044] The aforementioned problems are precisely the areas that the LightGBM algorithm aims to optimize and address. LightGBM primarily proposes the following improvements:

[0045] (1) The Histogram algorithm replaces the pre-sorting method for constructing data structures, which can significantly improve speed and reduce time. After adopting the histogram algorithm, only discrete values ​​need to be saved, and the original feature values ​​do not need to be used.

[0046] Furthermore, using histogram algorithms can effectively reduce memory usage when training with radar echo data. This is because the histogram operator only needs to store the discretized feature values, which can be stored using an 8-bit integer (unit8_t), eliminating the need for 32-bit int and float types. Therefore, memory consumption when training with multi-layer radar echo data can be reduced to one-eighth of the original amount.

[0047] (2) LightGBM further optimizes the histogram algorithm by employing a depth-constrained leaf-wise growth strategy. This reduces errors and contributes to improving the accuracy of rainfall intensity level analysis results.

[0048] (3) LightGBM uses the Gradient-based One-Side Sampling (GOSS) algorithm to train the sample based on the single gradient of a single feature on a certain feature. The implementation method is shown in the formula. In the study of radar echo inversion of rainfall in this invention, LightGBM will use this algorithm to train and sample radar echo data.

[0049]

[0050] in, For radar echo training dataset { … The vector in} (derived from the radar echo raster matrix of the CMA data raster pair), The set of gradient results of the loss function { ... The gradient results generated in each iteration of}. Let be the gain value generated by feature j at node d. A and B are subsets of the radar echo dataset obtained after sorting and calculation, respectively.

[0051] (4) Use the Exclusive Feature Bundling algorithm to merge features to solve the sparsity problem in multi-layer radar echo data, making the relatively sparse radar echo data matrix into a dense matrix, thereby improving the training speed.

[0052] LightGBM algorithm model training

[0053] This invention uses the radar echo-CMA combined dataset as the training dataset for the LightGBM algorithm model, continuously adjusting it during training to obtain satisfactory results. The process of using LightGBM to perform rainfall analysis on radar echoes is a regression task.

[0054] The standard training process involves dividing the pre-processed radar echo training dataset into multiple batches and inputting them into the built LightGBM model. Using pre-set parameters, the training and validation datasets are divided into a specific ratio. During model training, the parameter values ​​are continuously adjusted. Simultaneously, the model's accuracy on the training set for each iteration and the loss function result are calculated and output. This continues until the loss function reaches its minimum and remains relatively stable, or the iteration count is completed, ultimately resulting in a model with relatively good accuracy on the training data. Therefore, model training can be summarized in three steps: first, defining the model; second, evaluating the model during training; and third, obtaining a satisfactory model.

[0055] In training LightGBM, commonly used parameters include: learning rate α, maximum tree depth (max-depth), minimum number of data-in-leaf entries (min-data-in-leaf), feature fraction (proportion of randomly acquired features), number of iterations, regularization parameters, etc. Some research has been conducted on some of these parameters:

[0056] (1) Loss function

[0057] Learning through a loss function is a method to evaluate how well an algorithm models given data. The model training process is essentially a continuous, dynamic adjustment to reduce bias. The loss function produces a non-negative real value. If the model's training performance is not ideal, the loss function will produce a larger value; conversely, it will produce a smaller value. Therefore, the smaller the loss function value, the better the model's performance. The final selected model is the one with the smallest loss function value. For example, for a training set with N samples, the final loss function result is shown in the following formula:

[0058]

[0059] There is no fixed pattern for choosing a loss function; the choice should be based on the specific task and the actual situation. The Mean Squared Error (MSE) loss function is frequently used in regression problems. It measures the mean of the squared differences between predicted and observed values, as shown in the formula below:

[0060]

[0061] (2) Learning rate α

[0062] The learning rate, also known as the learning speed or progress of a model during training, is a crucial parameter in the training process. It determines whether and when the objective function converges to a local minimum. A suitable learning rate enables the objective function to converge to a local minimum within a reasonable timeframe.

[0063] When training the LightGBM model using a multi-layer radar echo reflectivity dataset, the weight values ​​need to be continuously adjusted according to the loss function, and the magnitude of each adjustment is determined by the learning rate. The calculation formula is as follows:

[0064] ;

[0065] The learning rate has a decisive impact on the training speed of a model. When the learning rate is set too small, the model converges very slowly; while when the learning rate is set too large, the model will oscillate continuously during the convergence process, or even fail to converge. Therefore, the learning rate needs to be continuously adjusted in real time during training.

[0066] (3) Tree depth max-depth

[0067] Most current decision tree learning algorithms grow trees using a level (depth)-wise strategy, while LightGBM uses a leaf-wise (best-first) decision tree growth strategy. It selects the leaf node with the largest delta loss for growth. When growing the same number of leaves, the leaf-wise algorithm can reduce loss more than the level-wise algorithm. When the dataset is small, leaf-wise methods can lead to overfitting. Therefore, LightGBM can use an additional parameter, max-depth, to limit the tree depth and avoid overfitting (tree growth still uses a leaf-wise strategy).

[0068] (4) The proportion of randomly obtained feature values

[0069] It is also called column sampling, which is the proportion of features randomly selected in each iteration. For example, when the selected value is 0.8, 80% of the multi-layer radar echo data will be selected to participate in the construction of the tree. Its value can be set between 0 and 1. When this value is reduced, the training speed can be improved and overfitting can be avoided.

[0070] (5) The minimum amount of data in each leaf (min-data-in-leaf)

[0071] It is also a relatively important parameter. When this value is increased, it can prevent the model from overfitting. Its value is usually set relatively large during training.

[0072] (6) Number of iterations

[0073] Each iteration uses all samples in the training set for training once, and all training samples undergo one forward pass and one backward pass. The model goes through multiple iterations, the number of which can be determined based on the specific circumstances during the training process.

[0074] (7) Regularization parameter

[0075] Regularization parameters are seen in the training of many machine learning algorithms. Commonly used additional terms are generally two types: L-regularization and L-regularization. They can be regarded as penalty terms of the loss function. The purpose of using them is to reduce the complexity of the model to improve its generalization ability. In real-world environments, they can better analyze rainfall intensity levels, and achieve good performance not only on the data used in experiments.

[0076] All data points in the processed radar echo reflectance-CMA measured rainfall dataset are randomly separated in an appropriate proportion to form training and validation sets. The larger dataset is used to train the LightGBM model. In each iteration, the model is adjusted using the residual between the predicted and actual rainfall intensity levels from the previous iteration. This continuous adjustment and refinement of the residual value leads to the optimal algorithm model for rainfall analysis. The validation set, consisting of untrained data, is used for hyperparameter tuning and preliminary evaluation of the model's predictive capabilities.

[0077] This invention introduces and constructs the LightGBM algorithm model for analyzing rainfall intensity. Before training, the model parameters are first set. Since the final result is the rainfall value corresponding to the radar echo reflectance value, a regression model is constructed, using the MSE function as the model's loss function, also known as the evaluation function. Because LightGBM is inherently a gradient boosting algorithm, the classic and commonly used GBDT is selected as the basic algorithm type in the experiments, with 2000 iterations. Other parameter values ​​of the model are determined through multiple experiments and result comparisons.

[0078] The max-depth value is assigned to 6.

[0079] Because radar echoes are easily affected by interference, even traditional filtering methods cannot eliminate errors. Therefore, when determining the accuracy of precipitation values ​​derived from radar echoes, a certain error range can be defined; values ​​within this range can be considered correct. After multiple experiments, the relationship between error range and accuracy was statistically analyzed as follows:

[0080]

[0081] As shown in the table above, the accuracy decreases as the error range decreases. Considering that this invention does not use the precipitation values ​​derived from radar echoes on a large scale, but only uses these values ​​locally to correct errors in CMA-measured rainfall data, an error range of 5 mm is acceptable. The experimental accuracy of 68.3% is also acceptable. Of course, with the accumulation of experimental data and continuous optimization and training of the model, the accuracy can be further improved.

[0082] After the model passes testing, the next step is to verify its practical application. First, the radar echo is cross-compared with the CMA rainfall raster layer to identify the problem areas. Then, the model is used to analyze the rainfall from the radar echo, retrieve the rainfall values, and make numerical corrections to the problem areas in the CMA raster.

Claims

1. A method for real-time correction of CMA measured rainfall data based on radar echo data, characterized in that: The method includes: Step 1: Acquire weather radar echo data and CMA measured rainfall data; Step 2: Using the location of the ground-based measured rain gauge station as the anchor point, select the corresponding CMA data and radar echo data; Step 3: Integrate radar echo data and CMA rainfall data to construct training and validation datasets; Step 4: Build a model based on the Light GBM algorithm and train the model using the training dataset. This includes: inputting the training dataset into the built Light GBM model in batches, dividing the training dataset and validation dataset into a certain proportion using the pre-set parameters, continuously adjusting the parameter values ​​during model training, calculating and outputting the accuracy and loss function results of the model on the current data in this iteration of the training set while training, until the result of the loss function reaches the minimum or the number of iterations is completed, thus finally obtaining the trained model. During the training of the LightGBM model, the weight values ​​are continuously adjusted according to the loss function. Each adjustment is determined by the learning rate, and the calculation formula is as follows: ； The formula for the loss function is: ； The mean squared error loss function (MSE) measures the mean of the squared differences between predicted and observed values. The formula is: ； Step 5: Perform real-time rainfall analysis on radar echo data using the trained model, compare with CMA data, identify problem areas, replace with rainfall values ​​calculated from radar echoes, and re-interpolate the fused rainfall data to obtain corrected rainfall data.

2. The method for real-time correction of CMA measured rainfall data based on radar echo data according to claim 1, characterized in that: In step 2, when selecting the corresponding CMA data and radar echo data using the location of the ground-based rain gauge station as the anchor point, the rain gauge station is in normal working condition, and data measurement, transmission, and reporting are all normal. The CMA data format is 5 times that of the radar echo data. When taking the radar echo data, the data of 25 grids, 5 grids in each direction (5 grids in each direction) are taken from the center of the measured station as feature data.

3. The method for real-time correction of CMA measured rainfall data based on radar echo data according to claim 1, characterized in that: Before constructing the training dataset, the data needs to be preprocessed. The preprocessing method is as follows: Since the grid of CMA data represents a range of 5km*5km, while the radar echo grid is a range of 1km*1km, when establishing the data mapping, it is represented that one CMA grid corresponds to one radar echo grid matrix, and each matrix contains 25 grids. To obtain such a training dataset, the radar echo data and CMA historical rainfall data need to be correlated and then merged. The correlation and merging method is to find data with the same timestamp from the radar echo data and CMA measured rainfall data as the merged dataset.

4. The method for real-time correction of CMA measured rainfall data based on radar echo data according to claim 1, characterized in that: When analyzing rainfall data from radar echoes, a range of error is defined to determine the accuracy of the precipitation values ​​derived from the radar echoes. Values ​​within this range are considered correct.

Images