A method for filling atmospheric precipitable water by integrating sounding data and reanalysis data

Abstract

The present invention discloses a method for filling atmospheric precipitable water by integrating sounding data and reanalysis data, comprising: obtaining sounding data through the weather data network of the University of Wyoming and the National Center for Environmental Information of the United States, and obtaining reanalysis data near the corresponding sounding station through the European Center for Medium-Term Weather Forecasts. With the location of the sounding station as the center, the meteorological data on four grid points in the reanalysis data are converted to the location of the corresponding sounding station by using a bilinear interpolation method. Then, the atmospheric precipitable water of the corresponding station is calculated by an integral formula under the sounding data, and an atmospheric precipitable water prediction model is constructed based on a multi-layer perceptron and a random forest algorithm with the sounding data as a training set. Finally, the missing atmospheric precipitable water time series in the sounding data is filled by integrating the prediction results of the multi-layer perceptron and the random forest model.

Description

Technical Field

[0001] The present invention relates to the fields of meteorology and artificial intelligence algorithms, and in particular to a method for filling atmospheric precipitable water by integrating sounding data and reanalysis data. Background Art

[0002] Precipitable Water Vapor (PWV) refers to the total amount of liquid water vapor from the ground to the top of the troposphere in a vertical column of air per unit cross-sectional area. PWV is not only the most active part of the atmosphere, but also has a significant impact on the vertical stability of the atmosphere. Therefore, obtaining accurate and complete PWV data is of vital importance for meteorological research. At present, the acquisition of PWV mainly relies on the following methods: radiosonde, global navigation satellite system and satellite remote sensing. Among them, radiosonde can provide detailed atmospheric structure and meteorological parameters, but data may be missing due to technical failures, severe weather conditions or maintenance problems, making it impossible to obtain complete trend information of PWV. In addition, PWV data can also be inverted by using reanalysis data obtained by assimilating observational data through the atmospheric circulation model. The advantage of this method is that it can provide spatially complete, temporally continuous and globally covered PWV data. Therefore, when radiosonde data is insufficient, reanalysis data can be used to supplement PWV information to ensure data integrity and continuity. Summary of the invention

[0003] The present invention proposes a method for filling atmospheric precipitable water by fusing sounding data and reanalysis data, aiming to improve the integrity and reliability of radiosonde data sets. Taking into account the technical failures, extreme meteorological conditions and maintenance obstacles that may be encountered in the data acquisition process of the prior art, the present invention introduces a reanalysis data set to effectively supplement the missing data of the radiosonde in the PWV time series. The introduction of the reanalysis data set not only significantly enhances the continuity and integrity of the PWV time series, but also provides a solid data foundation for the research of meteorology and related fields, thereby improving the accuracy of data analysis and model construction. The method for filling atmospheric precipitable water by fusing sounding data and reanalysis data described in the present invention comprises the following steps:

[0004] Step 1. Obtain multi-source meteorological element data: The multi-source meteorological element data include sounding data and reanalysis data, wherein the sounding data come from the University of Wyoming Weather Data Network and the National Center for Environmental Information, and the reanalysis data comes from the European Centre for Medium-Range Weather Forecasts.

[0005] Step 2. Calculation of initial PWV value based on sounding data: obtain sounding data through the University of Wyoming Weather Data Network and the National Center for Environmental Information of the United States, with a sampling interval of 12 hours, including temperature, air pressure, geopotential height, relative humidity, dew point temperature and station location information. According to the formula Calculate the water vapor pressure at different pressure layers, where is the dew point temperature, when hour, , ,when hour, , Since the sounding data provided by the National Center for Environmental Information of the United States lacks dew point temperature, it is necessary to use the formula Calculate the dew point temperature where is the temperature, is the relative humidity, According to the formula Calculate the specific humidity of different pressure layers, where is the water vapor pressure, is the air pressure. According to the formula Calculate PWV, where is the specific humidity, is the acceleration due to gravity, is the air pressure, is the surface air pressure.

[0006] Step 3. Reanalysis data reduction processing: ERA5 grid point data is obtained from the European Center for Medium-Range Weather Forecasts, with a sampling interval of 12 hours, including temperature, air pressure, geopotential height, relative humidity, and station location information. The four ERA5 data near the sounding station are reduced to the corresponding station location using the bilinear interpolation method. The core idea of ​​the bilinear interpolation method is to apply linear interpolation in two directions to estimate the value of a two-dimensional function at any point. Assume a two-dimensional grid with four known grid points. , , , , then the bilinear interpolation method can be used to determine any point in the grid The specific mathematical expression is as follows:

[0007] .

[0008] Step 4. PWV time series filling: Construct a PWV time series filling model based on the correlation analysis results of Ps, T, Rh and PWV in the study area. Due to the complex nonlinear relationship and certain collinearity problems between Ps, T, RH and PWV, the traditional multivariate linear regression fitting model cannot fit the data well, which ultimately affects the accuracy of the filling results. In view of the fact that artificial intelligence algorithms have powerful nonlinear fitting capabilities, they can effectively avoid the accumulation of errors that may occur during the PWV calculation process. Among them, deep learning algorithms (such as multi-layer perceptrons (Multiple Linear Regression, MLR), etc.) show excellent performance on large-scale data and complex pattern recognition tasks, while traditional machine learning algorithms (such as random forests (RandomForest, RF), etc.) are more suitable for scenarios with interpretable requirements and small data sets. Therefore, the present invention uses the MLP-RF integrated algorithm to construct a PWV filling model.

[0009] Step 5. MLP-PWV model construction:

[0010] (1) According to the formula Normalize the data set to eliminate the effects of different dimensions;

[0011] (2) Constructing feature matrix and the target variable vector , where the feature matrix Usually contains a bias term (a column of all 1s) and the actual feature column;

[0012] ;

[0013] (3) Construct the network architecture, including the input layer (with 4 nodes, corresponding to the feature matrix Columns), hidden layers (one or more hidden layers, each with neurons) and the output layer (target variable );

[0014] (4) Initialization parameters, randomly initialize the network weight matrix and the bias vector ;

[0015] (5) For each input sample, according to the formula and Perform the forward propagation calculation, where is the activation output of the hidden layer, is the activation function; for the output layer, according to the formula and Perform the forward propagation calculation, where is the PWV value predicted by the model, is a linear activation function, is the weight matrix connecting the input layer to the hidden layer, is the weight matrix connecting the hidden layer to the output layer, is the bias vector corresponding to the hidden layer, is the bias vector corresponding to the output layer, and the hidden layer activation output and the weight matrix The dot product of Result;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] (6) Use Mean Square Error (MSE) as the loss function , which measures the difference between the predicted value and the actual observed value;

[0021] (7) Calculate the gradient of the loss with respect to each weight and bias and backpropagate using the chain rule: , , , ;

[0022] (8) Update weights and biases using gradient descent or other optimization algorithms: , ,in is the learning rate, is the hierarchical index, For the The value of the layer weight matrix after an optimization update, For the The value of the layer weight matrix before the update, For the The value of the layer's bias vector after an optimization update, For the The value of the layer bias vector before updating;

[0023] (9) Repeat the iterative training. When the model performance is optimal, save the optimal model and apply it to the prediction of new data sets.

[0024] Step 6. RF-PWV model construction:

[0025] (1) According to the formula Normalize the data set to eliminate the effects of different dimensions;

[0026] (2) Constructing feature matrix and the target variable vector ;

[0027] ;

[0028] ;

[0029] (3) Determine the number of decision trees in the random forest And determine the number of features to use for each tree during training ;

[0030] (4) Training the random forest model: For each decision tree, a subset needs to be randomly selected from the training set as the training data for the tree (bootstrap sampling and other methods can be used). For each node of the decision tree, features and find the optimal split point among these features, that is, the feature and threshold that minimizes the node impurity (such as using Gini impurity or information gain), and then split the node according to the optimal split point, and repeat this step until all nodes become leaf nodes, and finally construct The final prediction value is the average of all decision tree predictions.

[0031] (5) Using Mean Square Error (MSE) as the loss function , which measures the difference between the predicted value and the actual observed value;

[0032] (6) Continuous Adjustment , and other hyperparameters to determine the optimal model and apply it to predictions on new data sets.

[0033] Step 7. Based on the trained MLP-PWV and RF-PWV models, the ERA5 meteorological parameter data (Ps, T, and RH) obtained by bilinear interpolation are used as characteristic factors to predict PWV, and the root mean square error (RMSE) and mean absolute error (MAE) are used as evaluation indicators. The calculation formulas are: and The prediction results of the MLP-PWV and RF-PWV models were then fused using the weighted average method to improve the accuracy and robustness of PWV prediction.

[0034] Step 8. Fusion of model prediction results:

[0035] (1) Use the same validation set to evaluate the performance of different models;

[0036] (2) Use the gradient descent method to minimize the loss function of the fusion result , to find the optimal weight ratio of different models, The calculation formula is: ,in are the weights of the MLP and RF models, is the weight of the regularization term whose weights sum to 1, It is The true target value of samples, and The MLP and RF models are Prediction of samples;

[0037] (3) According to the determined weights, the prediction results of each model are weighted averaged to obtain the fused prediction results. samples, the prediction after fusion The calculation is as follows: ,in and The MLP and RF models are The prediction of samples, and is the optimal weight found by gradient descent;

[0038] (4) Use appropriate performance evaluation indicators (such as RMSE, MAE, etc.) to evaluate the performance of the fusion model on the validation set to complete the PWV filling process of the present invention.

[0039] The present invention has the following beneficial technical effects:

[0040] During the data collection process of radiosondes, factors such as technical failures, extreme meteorological conditions and maintenance obstacles may lead to the loss of key meteorological element data, which poses a challenge to the continuity and integrity of the PWV time series. To address this problem, the present invention proposes an innovative PWV time series filling method, which uses advanced artificial intelligence algorithms. First, the ERA5 reanalysis data is accurately matched with the geographical location using bilinear interpolation technology to ensure the correspondence with each sounding site. Subsequently, the MLP and RF algorithms are introduced, and the PWV prediction model is constructed in combination with the sounding data. Finally, the ERA5 reanalysis data is introduced to effectively fill in the missing parts in the PWV time series based on the model. This method significantly improves the continuity and integrity of the PWV time series, provides more solid data support for meteorological research and related fields, and thus enhances the depth of data analysis and the accuracy of model construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0042] Figure 1 This is a flow chart of a PWV time series filling method proposed by the present invention.

[0043] Figure 2 It is a schematic diagram of bilinear interpolation. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 As shown, a method for filling atmospheric precipitable water by integrating sounding data and reanalysis data of the present invention comprises the following steps:

[0045] Step 1. Obtain multi-source meteorological element data for China during 2009-2019: The multi-source meteorological element data include sounding data and reanalysis data, including air pressure, temperature, relative humidity and geopotential height. The sounding data come from the University of Wyoming Weather Data Network and the National Center for Environmental Information of the United States, with a temporal resolution of 12h. A total of 52 experimental sites were selected; the reanalysis data come from the European Center for Medium-Range Weather Forecasts, with a spatial resolution of 0.25°. 0.25°, time resolution is 1h.

[0046] In this example, the method of converting the reanalysis data in step 1 to the corresponding sounding station by bilinear interpolation is as follows: Figure 2 As shown: According to the following formula:

[0047] ;

[0048] Through four known grid points , , , Determine any point in the grid The value of .

[0049] Step 2. Calculate the initial PWV values ​​from 2009 to 2019 using the sounding data provided by IGRA and the University of Wyoming, according to the formula Calculate the water vapor pressure at different pressure layers, where is the dew point temperature, when hour, , ,when hour, , Since the sounding data provided by the National Center for Environmental Information of the United States lacks dew point temperature, it is necessary to use the formula Calculate the dew point temperature where is the temperature, is the relative humidity, According to the formula Calculate the specific humidity of different pressure layers, where is the water vapor pressure, is the air pressure. According to the formula Calculate PWV, where is the specific humidity, is the acceleration due to gravity, is the air pressure, is the surface air pressure.

[0050] Step 3. Verify the correlation between air pressure, relative humidity, temperature, etc. and PWV, and build a PWV prediction model based on random forest and multi-layer perceptron algorithms. The RF-PWV model uses the sounding data from 2009 to 2017 as training data and the reanalysis data from 2018 to 2019 as verification data. The model is built through the following specific steps:

[0051] (1) According to the formula Normalize the data set to eliminate the effects of different dimensions;

[0052] (2) Constructing feature matrix and the target variable vector ;

[0053] ;

[0054] ;

[0055] (3) Determine the number of decision trees in the random forest And determine the number of features to use for each tree during training ;

[0056] (4) Training the random forest model: For each decision tree, a subset needs to be randomly selected from the training set as the training data for the tree (bootstrap sampling and other methods can be used). For each node of the decision tree, features and find the optimal split point among these features, that is, the feature and threshold that minimizes the node impurity (such as using Gini impurity or information gain), and then split the node according to the optimal split point, and repeat this step until all nodes become leaf nodes, and finally construct The final prediction value is the average of all decision tree predictions.

[0057] (5) Using Mean Square Error (MSE) as the loss function , which measures the difference between the predicted value and the actual observed value;

[0058] (6) Continuous Adjustment , And other hyperparameters to determine the optimal model and apply it to the prediction of new data sets. The MLP-PWV model also uses the sounding data from 2009 to 2017 as training data and the reanalysis data from 2018 to 2019 as validation data. The model is constructed through the following specific steps:

[0059] (1) According to the formula Normalize the data set to eliminate the effects of different dimensions;

[0060] (2) Constructing feature matrix and the target variable vector , where the feature matrix Usually contains a bias term (a column of all 1s) and the actual feature column;

[0061] ;

[0062] (3) Construct the network architecture, including the input layer (with 4 nodes, corresponding to the feature matrix Columns), hidden layers (one or more hidden layers, each with neurons) and the output layer (target variable );

[0063] (4) Initialization parameters, randomly initialize the network weight matrix and the bias vector ;

[0064] (5) For each input sample, according to the formula and Perform the forward propagation calculation, where is the activation output of the hidden layer, is the activation function; for the output layer, according to the formula and Perform the forward propagation calculation, where is the PWV value predicted by the model, is a linear activation function; is the weight matrix connecting the input layer to the hidden layer, is the weight matrix connecting the hidden layer to the output layer, is the bias vector corresponding to the hidden layer, is the bias vector corresponding to the output layer, and the hidden layer activation output and the weight matrix The dot product of Result;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] (6) Use Mean Square Error (MSE) as the loss function , which measures the difference between the predicted value and the actual observed value;

[0070] (7) Calculate the gradient of the loss with respect to each weight and bias and backpropagate using the chain rule: , , , ;

[0071] (8) Update weights and biases using gradient descent or other optimization algorithms: , ,in is the learning rate; is the hierarchical index, For the The value of the layer weight matrix after an optimization update, For the The value of the layer weight matrix before the update, For the The value of the layer's bias vector after an optimization update, For the The value of the layer bias vector before updating;

[0072] (9) Repeat the iterative training. When the model performance is optimal, save the optimal model and apply it to the prediction of new data sets.

[0073] Step 4. Given that artificial intelligence algorithms have powerful nonlinear fitting capabilities and can effectively avoid the accumulation of errors that may occur during PWV calculation, deep learning algorithms (such as multi-layer perceptrons, etc.) show excellent performance on large-scale data and complex pattern recognition tasks, while traditional machine learning algorithms (such as random forests, etc.) are more suitable for scenarios with interpretability requirements and small data sets. Therefore, the present invention uses the weighted average method to fuse the model results of MLP-PWV and RF-PWV, and the specific steps of the fusion are as follows:

[0074] (1) Use the same validation set to evaluate the performance of different models;

[0075] (2) Use the gradient descent method to minimize the loss function of the fusion result , to find the optimal weight ratio of different models, The calculation formula is: ,in are the weights of the MLP and RF models, is the weight of the regularization term whose weights sum to 1, It is The true target value of samples, and The MLP and RF models are Prediction of samples;

[0076] (3) According to the determined weights, the prediction results of each model are weighted averaged to obtain the fused prediction results. samples, the prediction after fusion The calculation is as follows: ,in and The MLP and RF models are The prediction of samples, and is the optimal weight found by gradient descent;

[0077] (4) Use appropriate performance evaluation indicators (such as RMSE, MAE, etc.) to evaluate the performance of the fusion model on the validation set to complete the PWV filling process of the present invention.

[0078] When the present invention is used, the sounding data is first used to calculate the initial PWV value. When a missing PWV time series appears, the MLP-RF-PWV model constructed based on the reanalysis data is used to fill in the missing initial PWV value. This method not only significantly enhances the continuity and integrity of the PWV time series, but also provides a solid data foundation for the research of meteorology and related fields, thereby improving the accuracy of data analysis and model construction.

[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for filling atmospheric precipitable water by integrating sounding data and reanalysis data, characterized in that: The method comprises the following steps: Step 1. Acquire multi-source meteorological element data; the multi-source meteorological element data includes sounding data and reanalysis data; Step 2. Calculate the initial value of atmospheric precipitable water (PWV) based on the sounding data; including calculating the water vapor pressure, dew point temperature, specific humidity of different pressure layers, and calculating PWV; Step 3. Reanalysis data reduction processing: Obtain ERA5 grid point data, and use bilinear interpolation to reduce the four ERA5 data near the sounding station to the corresponding station position. The bilinear interpolation method is to apply linear interpolation in two directions to estimate the value of a two-dimensional function at any point; Step 4. PWV time series filling: construct a PWV time series filling model based on the correlation analysis results of the surface air pressure Ps, surface temperature T, relative humidity Rh and PWV in the study area; Step 5. MLP-PWV model construction; Step 6. RF-PWV model construction; Step 7. Based on the trained MLP-PWV and RF-PWV models, the ERA5 meteorological parameter data including Ps, T and RH obtained by bilinear interpolation method are used as characteristic factors to predict PWV; Step 8. Fusion of model prediction results; In step 5, the MLP-PWV model construction includes the following steps: Step (1) According to the formula Normalize the data set to eliminate the effects of different dimensions; Step (2) Construct feature matrix and the target variable vector , where the feature matrix Usually contains a bias term, which is a column of all 1s, and the actual feature column; ； Step (3) constructs a network structure, including an input layer, a hidden layer, and an output layer; the input layer has 4 nodes, corresponding to the feature matrix Columns, set one or more hidden layers, each hidden layer has neurons, the output layer outputs the target variable ; Step (4) Initialize parameters and randomly initialize the network weight matrix and the bias vector ; Step (5) For each input sample, according to the formula and Perform the forward propagation calculation, where is the activation output of the hidden layer, is the activation function; for the output layer, according to the formula and Perform the forward propagation calculation, where is the PWV value predicted by the model, is a linear activation function, is the weight matrix connecting the input layer to the hidden layer, is the weight matrix connecting the hidden layer to the output layer, is the bias vector corresponding to the hidden layer, is the bias vector corresponding to the output layer, and the hidden layer activation output and the weight matrix The dot product of Result; ； ； ； ； Step (6) uses mean square error MSE as the loss function , which measures the difference between the predicted value and the actual observed value; Step (7) calculates the gradient of the loss with respect to each weight and bias and backpropagates using the chain rule: , , , ; Step (8) updates the weights and biases using gradient descent or other optimization algorithms: , ,in is the learning rate, is the hierarchical index, For the The value of the layer weight matrix after an optimization update, For the The value of the layer weight matrix before the update, For the The value of the layer's bias vector after an optimization update, For the The value of the layer bias vector before updating; Step (9) repeats the iterative training, and when the model performance is optimal, saves the optimal model and applies it to the prediction of the new data set; in step 6, the RF-PWV model construction includes the following steps: Step (1) According to the formula Normalize the data set to eliminate the effects of different dimensions; Step (2) Construct feature matrix and the target variable vector ; ； ； Step (3) Determine the number of decision trees in the random forest And determine the number of features to use for each tree during training ; Step (4) Train the random forest model. For each decision tree, a subset is randomly selected from the training set as the training data for the tree. For each node of the decision tree, a random forest model is randomly selected. features and find the optimal split point among these features, that is, the feature and threshold that minimizes the node impurity, then split the node according to the optimal split point, and repeat this step until all nodes become leaf nodes, and finally construct The final prediction value is the average of all decision tree predictions. Step (5) uses mean square error MSE as the loss function , which measures the difference between the predicted value and the actual observed value; Step (6) Continuous Adjustment , and other hyperparameters to determine the optimal model and apply it to predictions on new data sets.

2. The method according to claim 1, characterized in that In step 1, the sounding data come from the University of Wyoming Weather Data Network and the National Centers for Environmental Information, and the reanalysis data come from the European Centre for Medium-Range Weather Forecasts.

3. The method according to claim 1, characterized in that In step 2, According to the formula Calculate the water vapor pressure at different pressure layers, where is the dew point temperature, when hour, , ,when hour, , ; According to the formula Calculate the dew point temperature where is the temperature, is the relative humidity, ; According to the formula Calculate the specific humidity of different pressure layers, where is the water vapor pressure, For air pressure According to the formula Calculate PWV, where is the specific humidity, is the acceleration due to gravity, is the air pressure, is the surface air pressure.

4. The method according to claim 3, characterized in that In step 3, assume a two-dimensional grid with four known grid points: , , , , then the bilinear interpolation method can be used to determine any point in the grid The specific mathematical expression is as follows: 。 5. The method according to claim 1, characterized in that: In step 7, based on the trained MLP-PWV and RF-PWV models, the ERA5 meteorological parameter data obtained by bilinear interpolation is used as the characteristic factor to predict PWV. The ERA5 meteorological parameter data includes Ps, T and RH, and the root mean square error RMSE and mean absolute error MAE are used as evaluation indicators. The calculation formulas are: and ; Then the weighted average method is used to fuse the prediction results of MLP-PWV and RF-PWV models.

6. The method according to claim 1, characterized in that In step 8, the fusion of model prediction results includes the following steps: Step (1) evaluates the performance of different models using the same validation set; Step (2) uses the gradient descent method to minimize the loss function of the fusion result , to find the optimal weight ratio of different models, The calculation formula is: ,in are the weights of the MLP and RF models, is the weight of the regularization term whose weights sum to 1, It is The true target value of samples, and The MLP and RF models are Prediction of samples; Step (3) According to the determined weights, the prediction results of each model are weighted averaged to obtain the fused prediction results. samples, the prediction after fusion The calculation is as follows: ,in and The MLP and RF models are The prediction of samples, and is the optimal weight found by gradient descent; Step (4) uses appropriate performance evaluation metrics to evaluate the performance of the fusion model on the validation set and completes the PWV filling process.

Images