A lightning nowcasting method based on GNSS data

Abstract

The present invention discloses a lightning nowcasting method based on GNSS data, comprising the following steps: S1, data collection; S2, preprocessing the collected data; S3, constructing a lightning nowcasting model, training the lightning nowcasting model, and then verifying and evaluating the lightning nowcasting model, and finally determining the prediction ability and prediction accuracy of the lightning nowcasting model; S4, using the collected data to perform lightning prediction through the lightning nowcasting model. The present invention adopts the above-mentioned lightning nowcasting method based on GNSS data, establishes a lightning prediction model through a machine learning method, and combines the atmospheric precipitable water and other meteorological parameters derived from GNSS to achieve good prediction performance.

Description

Technical Field

[0001] The present invention relates to the technical field of lightning nowcasting, and in particular to a lightning nowcasting method based on GNSS data. Background Art

[0002] Lightning is a pulsed discharge phenomenon in the atmosphere and is one of the common weather phenomena in the earth's atmosphere. There is no doubt that lightning disasters are one of the most serious natural disasters that cannot be ignored. Because of the huge physical effects such as strong current, scorching high temperature, strong electromagnetic radiation, etc., it can produce great damage and cause serious economic losses and casualties to humans.

[0003] Atmospheric water vapor is the main driving force of weather changes and has a good indicative effect on the weather. In particular, in short-term severe convective weather (such as thunderstorms) of small and medium scales, there are drastic temporal and spatial changes in water vapor. The latent heat during the phase change of water vapor will cause uneven heating of the various layers of the atmosphere, thus affecting the strength of the vertical movement of the near-surface atmosphere and the evolution and development of convective weather. Atmospheric water vapor is not only of great significance to weather and climate, but also has an important impact on the country's agriculture and animal husbandry, changes in water resource distribution, ecosystems, and living environment.

[0004] Traditional atmospheric water vapor monitoring methods, such as radiosondes, ground-based microwave radiometers, near-infrared sensors, sun photometers, and meteorological radars, have all developed relatively maturely. However, with the urgent demand and development of atmospheric water vapor research, the current traditional technologies can no longer meet the requirements of near-real-time, high-precision, and high-temporal and spatial resolution atmospheric water vapor monitoring. The specific problems are as follows:

[0005] 1) Low temporal and spatial resolution; 2) Unable to work around the clock; 3) Limited observation area; 4) High cost.

[0006] Since lightning weather is sudden and short-lived, it is very necessary to predict lightning events in a timely manner. Therefore, the present invention performs lightning nowcasting based on GNSS water vapor. With the rapid development of global navigation satellite system technology, it has the advantages of high temporal resolution, high accuracy, and strong data continuity.

[0007] In today's meteorological field, with the development and update of detection methods and equipment, a large amount of raw data has been accumulated with the support of comprehensive radars, satellites, rich detection equipment and various numerical models. Traditional calculation methods cannot deeply mine it. It is difficult to easily sort out the correlation and change trends between various actual meteorological elements only through relevant data analysis. The use of machine learning can often find out the laws of meteorological development from abstract data information more easily and accurately. Summary of the invention

[0008] The purpose of the present invention is to provide a lightning nowcasting method based on GNSS data. Through a machine learning method, a lightning forecasting model is established. The model includes two parts: lightning prediction and outlier detection. The model combines the atmospheric precipitable water content and other meteorological parameters derived from GNSS to achieve good forecasting skills.

[0009] To achieve the above object, the present invention provides a lightning nowcasting method based on GNSS data, comprising the following steps:

[0010] S1, data collection;

[0011] S2, preprocessing the collected data;

[0012] S3. Construct a lightning nowcasting model, train the lightning nowcasting model, and then verify and evaluate the lightning nowcasting model to ultimately determine the prediction capability and prediction accuracy of the lightning nowcasting model;

[0013] S4. Use the collected data to make lightning predictions through a lightning nowcasting model.

[0014] Preferably, in step S1, tropospheric water vapor data, meteorological data and lightning data are collected by using a GNSS receiver and a meteorological sensor, wherein the meteorological data includes temperature, humidity, pressure and rainfall.

[0015] Preferably, in step S2, preprocessing the collected data specifically includes:

[0016] The collected data is processed for outliers and missing values. The data is fitted into a function using linear interpolation, and then the outliers and missing values ​​are interpolated.

[0017] According to the presence or absence of lightning activity within a radius of R kilometers, the lightning data are labeled as “0” or “1” as the ground truth for the prediction model.

[0018] Preferably, in step S3, a lightning nowcasting model is established based on a machine learning algorithm using tropospheric water vapor data and meteorological data, and the lightning nowcasting model is trained using historical tropospheric water vapor data and meteorological data, and the lightning nowcasting model is verified and evaluated to determine the prediction capability and prediction accuracy of the lightning nowcasting model.

[0019] Preferably, step S3 specifically includes the following steps:

[0020] S31. Data division:

[0021] The entire data set is divided into two parts, data set 1 and data set 2, according to the year ratio;

[0022] S32. Determine the best hyperparameters and models:

[0023] According to the proportion of lightning in the data in a year, the K-fold cross-validation method is used to reduce the dependence of the optimal hyperparameters and model on the data. On the basis of the K-fold cross-validation, the three-layer perceptron ensemble model Easy Ensemble is used to automatically search for the optimal hyperparameters and save the model. The data set 1 is input into the model for retraining to obtain the lightning prediction results on the data set 1.

[0024] S33, outlier detection:

[0025] Compare the prediction result obtained in step S32 with the true value, add an outlier label to data set one, and use this data set one to train an outlier detection model, the outlier detection model consists of a fully connected layer and a softmax module, extract features through the fully connected layer to output the predicted probability, and the softmax module maps the predicted probability to a classification value of 0 or 1, where 0 indicates no lightning and 1 indicates lightning; finally, put data set two into the trained outlier detection model for outlier detection, generate an outlier label, and compare the distribution of "abnormal" samples detected in data set one and data set two; repeat steps S32 to S33, select the data with the highest similarity score, and execute the corresponding data;

[0026] S34, Lightning Forecast:

[0027] On the dataset 2 with outlier labels generated in the above step S33, the optimal parameters determined in step S32 are used to train and test the final lightning approach prediction model, and the model is trained and evaluated in combination with k-fold cross validation. Samples with abnormal outlier labels in the dataset 2 are not included in the calculation of the evaluation index.

[0028] Preferably, in step S33, the outlier label is whether the lightning prediction result and the true value comparison result are the same or different, and the outlier label is added to the data set to mark the sample. If the lightning prediction result and the true value comparison result are the same, the sample is marked as "normal"; if the lightning prediction result and the true value comparison result are different, the sample is marked as "abnormal".

[0029] Preferably, in step S33, the distributions of the two "abnormal" samples are compared by KL divergence to calculate the similarity score. For two probability distributions P and Q, the KL divergence is defined as follows:

[0030]

[0031] Where i represents a discrete event of the distribution, and P(i) and Q(i) are the probabilities of two distributions on event i.

[0032] Preferably, in step S34, the evaluation indicators include detection rate, false alarm rate and threat score.

[0033] Therefore, the present invention adopts the above-mentioned lightning nowcasting method based on GNSS data, and establishes a lightning forecasting model through a machine learning method. The model includes two parts: lightning prediction and outlier detection, and combines the atmospheric precipitable water and other meteorological parameters derived from GNSS to achieve good prediction performance.

[0034] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of an embodiment of a lightning nowcasting method based on GNSS data of the present invention;

[0036] Figure 2 1 is a flow chart of step S3 of an embodiment of a lightning nowcasting method based on GNSS data of the present invention;

[0037] Figure 3 It is a structural schematic diagram of a machine learning model of an embodiment of a lightning nowcasting method based on GNSS data of the present invention;

[0038] Figure 4 It is a comparison diagram of lightning prediction results evaluation of a 0-10min lightning nowcasting model and a convective effective potential energy (CAPE) model (threshold method) in an embodiment of a lightning nowcasting method based on GNSS data of the present invention;

[0039] Figure 5 It is a lightning prediction result evaluation comparison diagram of a 10-20min lightning nowcasting model and a convective effective potential energy (CAPE) model (threshold method) of an embodiment of a lightning nowcasting method based on GNSS data of the present invention;

[0040] Figure 6 It is a comparison diagram of lightning prediction result evaluation of a 20-30min lightning nowcasting model and a convective effective potential energy (CAPE) model (threshold method) in an embodiment of a lightning nowcasting method based on GNSS data of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0042] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0043] like Figure 1As shown, a lightning nowcasting method based on GNSS data includes the following steps:

[0044] S1, data collection;

[0045] Tropospheric water vapor data, meteorological data and lightning data are collected through GNSS receivers and meteorological sensors. The meteorological data includes temperature, humidity, pressure and rainfall.

[0046] S2. Preprocessing the collected tropospheric water vapor data, meteorological data and lightning data, including:

[0047] The collected data is processed for outliers and missing values. The data is fitted into a function using linear interpolation, and then the outliers and missing values ​​are interpolated to ensure the quality and integrity of the data.

[0048] The lightning data are marked as “0” or “1” according to the presence or absence of lightning activity within a radius of R kilometers, which serves as the ground truth for the prediction model. Here, R = 30 kilometers.

[0049] S3. Establish a lightning nowcasting model based on machine learning algorithms using tropospheric water vapor data and meteorological data, train the lightning nowcasting model using historical tropospheric water vapor data and meteorological data, verify and evaluate the lightning nowcasting model, and ultimately determine the prediction capability and accuracy of the lightning nowcasting model.

[0050] like Figure 2 As shown, the specific steps include:

[0051] S31. Data division:

[0052] The entire data set is divided into two parts, data set 1 and data set 2, according to the year ratio;

[0053] The time range used in this embodiment is 5 years of data from 2018 to 2022, and the years are divided into 3:2, 2018-2020 is data set one, and 2021-2022 is data set two.

[0054] S32. Determine the best hyperparameters and models:

[0055] This process is performed on Dataset 1. According to the lightning proportion in the data in a year, lightning events account for about 4% and non-lightning events account for about 96%. In order to solve the problem of extremely unbalanced data sets, the K-fold cross-validation method is used to divide the training set data into K parts, train the K-1 parts of random combinations, and validate the remaining K parts, so as to achieve K groups of cross-cyclic training to reduce the dependence of the optimal hyperparameters and models on the data;

[0056] Based on k-fold cross validation, the Easy Ensemble model of the three-layer perceptron is used to automatically search for the optimal hyperparameters and save the model, such as Figure 3 As shown in the figure, first, some samples are randomly selected from the minority class samples to create multiple balanced subsets, and the classifier is trained for each subset, and the classification results are obtained by voting; then the results are input into the three-layer perceptron, and the input layer, hidden layer, and output layer are composed of 9 neurons, 15 neurons, and 1 neuron respectively. Data set 1 is input into the model for retraining, and the lightning prediction results on data set 1 are obtained.

[0057] S33, outlier detection:

[0058] Compare the prediction result obtained in step S32 with the true value. After adding an outlier label to dataset one, use this dataset one to train an outlier detection model. The anomaly detection model consists of a 128*16*2 fully connected layer and a softmax module. The fully connected layer extracts features to output predicted probabilities, and the softmax module maps the predicted probabilities to classification values ​​of 0 or 1, where 0 indicates no lightning and 1 indicates lightning. Finally, dataset two is placed in the trained outlier detection model for outlier detection, outlier labels are generated, and the distribution of "abnormal" samples detected in dataset one and dataset two is compared. Repeat steps S32 to S33, select the data with the highest similarity score, and execute the corresponding data.

[0059] In the present invention, the outlier label is whether the lightning prediction result is the same or different from the true value comparison result. The outlier label is added to the data set to mark the sample. If the lightning prediction result is the same as the true value comparison result, the sample is marked as "normal"; if the lightning prediction result is different from the true value comparison result, the sample is marked as "abnormal".

[0060] The present invention compares the distribution of two "abnormal" samples through KL divergence and calculates the similarity score. For two probability distributions P and Q, KL divergence is defined as follows:

[0061]

[0062] Where i represents a discrete event of the distribution, and P(i) and Q(i) are the probabilities of two distributions on event i.

[0063] S34, Lightning Forecast:

[0064] On the data set 2 with outlier labels generated in the above step S33, the optimal parameters determined in step S32 are used to train and test the final lightning approach prediction model, and the model is trained and evaluated in combination with k-fold cross validation, and samples with outlier labels as abnormal in the data set 2 are not included in the calculation of the evaluation index. The evaluation indexes of the present invention include detection rate (Probability of Detection, POD), false alarm rate (False Alarm Rate, FAR) and threat score (Threat Score, TS).

[0065] S4. Lightning prediction is carried out through the lightning nowcasting model.

[0066] The tropospheric water vapor data and meteorological data collected in real time are used to predict lightning through the lightning nowcasting model (MLmodel). This process predicts lightning for the next 0-10 minutes, 10-20 minutes, and 20-30 minutes respectively.

[0067] The traditional lightning prediction method CAPE model (threshold method) mainly determines whether the convective effective potential energy exceeds the set threshold. If it exceeds, it is considered that lightning has occurred, otherwise it is considered that there is no lightning.

[0068] like Figure 4-6 As shown in FIG. 1 , the lightning prediction results of the lightning nowcasting model provided by the present invention and the convective effective potential energy (CAPE) model (threshold method) are compared. From the lightning prediction result evaluation diagram, it can be seen that the machine learning model proposed by the present invention is Figure 4 As shown in the figure, the evaluation indexes of the lightning nowcasting model provided by the present invention can reach POD=0.95, FAR=0.19, and TS=0.77. The evaluation indexes of the Convective Potential Energy (CAPE) model (threshold method) are: POD=0.45, FAR=0.86, TS=0.10

[0069] Compared with the traditional convective effective potential energy (CAPE) model, it always shows good performance, with the highest performance in detection rate POD, false alarm rate FAR, and threat score TS. Moreover, in the lightning prediction of 0-10min, 10-20min, and 20-30min, the model performance does not show a significant decrease as the prediction time becomes longer.

[0070] Therefore, the present invention adopts the above-mentioned lightning nowcasting method based on GNSS data, and establishes a lightning forecasting model through a machine learning method. The model includes two parts: lightning prediction and outlier detection. At the same time, it combines the atmospheric precipitable water content and other meteorological parameters derived from GNSS to achieve good prediction accuracy. As the prediction time becomes longer, the model performance remains excellent.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A lightning nowcasting method based on GNSS data, characterized in that: The following steps are involved: S1, data collection; S2, preprocessing the collected data; S3. Construct a lightning nowcasting model, train the lightning nowcasting model, and then verify and evaluate the lightning nowcasting model to ultimately determine the prediction capability and prediction accuracy of the lightning nowcasting model; S4, using the collected data to perform lightning prediction through a lightning nowcasting model; Step S3 specifically includes the following steps: S31. Data division: The entire data set is divided into two parts, data set 1 and data set 2, according to the year ratio; S32. Determine the best hyperparameters and models: According to the proportion of lightning in the data in a year, the K-fold cross-validation method is used to reduce the dependence of the optimal hyperparameters and model on the data. On the basis of the K-fold cross-validation, the three-layer perceptron ensemble model Easy Ensemble is used to automatically search for the optimal hyperparameters and save the model. The data set 1 is input into the model for retraining to obtain the lightning prediction results on the data set 1. S33, outlier detection: Compare the prediction result obtained in step S32 with the true value, add an outlier label to data set one, and use this data set one to train an outlier detection model, the outlier detection model consists of a fully connected layer and a softmax module, extract features through the fully connected layer to output the predicted probability, and the softmax module maps the predicted probability to a classification value of 0 or 1, where 0 indicates no lightning and 1 indicates lightning; finally, put data set two into the trained outlier detection model for outlier detection, generate an outlier label, and compare the distribution of "abnormal" samples detected in data set one and data set two; repeat steps S32 to S33, select the data with the highest similarity score, and execute the corresponding data; S34, Lightning Forecast: On the dataset 2 with outlier labels generated in the above step S33, the optimal parameters determined in step S32 are used to train and test the final lightning approach prediction model, and the model is trained and evaluated in combination with k-fold cross validation. Samples with abnormal outlier labels in the dataset 2 are not included in the calculation of the evaluation index.

2. The lightning nowcasting method based on GNSS data according to claim 1, characterized in that: In step S1, tropospheric water vapor data, meteorological data and lightning data are collected by using a GNSS receiver and a meteorological sensor, wherein the meteorological data includes temperature, humidity, pressure and rainfall.

3. The lightning nowcasting method based on GNSS data according to claim 2, characterized in that: In step S2, preprocessing the collected data specifically includes: The collected data is processed for outliers and missing values. The data is fitted into a function using linear interpolation, and then the outliers and missing values ​​are interpolated. Depending on the presence or absence of lightning activity within a radius of R kilometers, the lightning data are labeled as "0" or "1" and serve as the ground truth for the prediction model.

4. The lightning nowcasting method based on GNSS data according to claim 3, characterized in that: In step S3, a lightning nowcasting model is established based on a machine learning algorithm using tropospheric water vapor data and meteorological data, and the lightning nowcasting model is trained using historical tropospheric water vapor data and meteorological data. The lightning nowcasting model is verified and evaluated to determine the prediction capability and accuracy of the lightning nowcasting model.

5. The lightning nowcasting method based on GNSS data according to claim 4, characterized in that: In step S33, the outlier label is whether the lightning prediction result is the same as or different from the true value comparison result. The outlier label is added to the data set to mark the sample. If the lightning prediction result is the same as the true value comparison result, the sample is marked as "normal"; if the lightning prediction result is different from the true value comparison result, the sample is marked as "abnormal".

6. The lightning nowcasting method based on GNSS data according to claim 5, characterized in that: In step S33, the distributions of the two "abnormal" samples are compared by KL divergence to calculate the similarity score. For two probability distributions P and Q, the KL divergence is defined as follows: Where x represents a discrete event of the distribution, and P(x) and Q(x) are the probabilities of two distributions on event x.

7. The lightning nowcasting method based on GNSS data according to claim 6, characterized in that: In step S34, the evaluation indicators include detection rate, false alarm rate and threat score.

Images