A method for identifying severe convective clouds using multi-source data based on machine learning

Abstract

The present invention discloses a method for identifying strong convective clouds based on multi-source data of machine learning, including collecting weather data of national ground meteorological observation stations in historical strong convective weather, as well as geostationary satellite data and radar data, forming original data samples with strong convective marks of geostationary satellite channel data and station matching through time and space matching, and pre-processing to form original strong convective cloud data set; extracting features from the original strong convective cloud data set using random forest algorithm, combining with original features to form strong convective cloud data set with new features, and then inputting into a logistic regression model, training to obtain a strong convective cloud recognition model; obtaining geostationary satellite data and using the strong convective cloud recognition model to identify strong convective clouds. The present invention uses multi-source data to form a strong convective cloud data set, improves the accuracy of data samples, and uses RF algorithm + LR model to achieve the effect of identifying strong convective clouds in all weather and full coverage, overcoming the limitations of traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of meteorological identification technology, and in particular to a method for identifying severe convective clouds using multi-source data based on machine learning. Background Art

[0002] Severe convective clouds refer to clouds generated by convection, especially when the atmosphere is unstable, cold on top and warm on the bottom, and the warm air in the lower layer rises. When it reaches a certain height, water vapor condenses to form clouds. This type of cloud may exist alone or in clusters. When convective clouds gradually gather and increase, it may indicate the arrival of severe convective weather, such as heavy precipitation, hail, thunderstorms and gale. The existing indicators for the classification of severe convective weather (China Meteorological Administration standards): short-term heavy precipitation with hourly precipitation greater than or equal to 20 mm, convective gale of level 8 to 11 (greater than 17.2), hail greater than or equal to 20 mm, etc., occurring alone or simultaneously.

[0003] The significance of severe convective cloud monitoring is mainly reflected in the following aspects:

[0004] (1) Early warning of disastrous weather: Severe convective weather often brings disastrous consequences, such as heavy rain, hail, tornadoes, etc., which threaten people’s lives and property. Monitoring of severe convective clouds can timely detect and warn of these weather phenomena, thereby reducing disaster losses.

[0005] (2) Improving the accuracy of weather forecasts: By monitoring severe convective clouds, we can more accurately understand the evolution and development of weather systems and improve the accuracy of weather forecasts.

[0006] (3) Scientific research: The study of severe convective clouds is of great significance for understanding the basic principles of atmospheric dynamics, cloud physics and meteorology. At the same time, these studies are also helpful for improving weather forecasts and climate prediction models.

[0007] Large-scale real-time monitoring of severe convective weather mainly uses weather radars and geostationary satellites with high temporal and spatial resolution, which can capture small and medium-scale weather processes such as the development of new cumulus clouds and effectively evaluate convection triggering conditions. Among them, radar monitoring uses radar to monitor the echo intensity and speed of cloud clusters. Common features include reflectivity intensity, vertical speed, lightning frequency, etc., which can identify strong convective clouds. Satellite remote sensing is a method of identifying strong convective clouds through the threshold method. This method is based on the characteristics that the temperature at the top of the strong convective cloud layer is lower than the ground and the reflectivity is higher. Convective clouds are identified by setting appropriate thresholds. Due to the development of artificial intelligence, the method of identifying strong convective clouds based on deep learning has become a hot topic. It mainly uses geostationary meteorological satellite images and deep learning models for automatic identification.

[0008] Conventional radar observations have certain defects in monitoring severe convective weather. The radar's observation range and resolution are limited, and it cannot cover all areas, especially in special terrain areas such as mountainous areas and deserts. The radar's observation capabilities will be greatly affected, especially in the northwest and western regions, where it is impossible to scan the entire area.

[0009] Limitations of traditional geostationary satellite identification methods. The traditional objective method requires forecasters to identify and monitor severe convection by setting corresponding thresholds for observation data and different regions based on statistical results and physical mechanisms. Changes in thresholds may lead to algorithm complexity and identification uncertainty, further causing the problem of high false alarm rate in severe convection identification (Schultz et al, 2009; Harris et al, 2010). At present, geostationary satellites identify intensity flows based on spectral characteristics and texture characteristics, using single-band threshold methods or multi-band threshold methods to identify severe convective clouds. This method is also limited by the setting of thresholds and is mainly used for regional severe convective cloud identification. It has not yet taken advantage of the wide range of satellite monitoring.

[0010] Existing deep learning sample labeling defects. Many deep learning method samples use CloudSat satellite cloud classification products (2B-CLDCLASS) or other satellite cloud classification products, which mainly identify deep convective clouds (Zheng Yiqin, 2020), which cannot characterize all strong convective clouds. In addition, these secondary cloud classification products are made based on traditional geostationary satellite identification methods, and their own identification accuracy is problematic. Summary of the invention

[0011] In view of the above problems in the above-mentioned prior art, the present invention provides a method for identifying strong convective clouds based on multi-source data of machine learning, which combines FY-4B geostationary satellite, radar data, national ground meteorological observation station data and artificial intelligence technology to solve the problems of radar coverage, uncertain threshold and high false alarm rate of identifying strong convective clouds by geostationary satellite method, and deep learning sample defects, so as to achieve the effect of identifying strong convective clouds with full coverage around the clock.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0013] A method for identifying severe convective clouds using multi-source data based on machine learning, comprising the following steps:

[0014] S10. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, as well as geostationary satellite data and radar data, obtain radar combined reflectivity data and geostationary satellite brightness temperature channel data through time-space matching, comprehensively judge and mark whether severe convective weather occurs at national ground meteorological observation stations, and form raw data samples with severe convection marks that match geostationary satellite channel data and stations;

[0015] S20, preprocessing the original data samples with strong convection marks to form an original strong convective cloud cluster data set;

[0016] S30, using the random forest RF algorithm to extract the features of the strong convective cloud data from the original strong convective cloud data set, and combining them with the original features of the original strong convective cloud data set to form a strong convective cloud data set with new features, and then inputting the strong convective cloud data set into the logistic regression LR model to train and obtain a strong convective cloud recognition model;

[0017] S40, acquiring geostationary satellite data, and using the severe convective cloud identification model to identify severe convective clouds.

[0018] Specifically, the specific process of step S10 includes:

[0019] S11. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, select stations that meet the requirements of severe convective weather, and determine the time when severe convective weather occurs;

[0020] S12. According to the time when severe convective weather occurs at the site, the geostationary satellite data of the same period is queried to obtain the most recent geostationary satellite data and radar combined reflectivity data to complete time matching;

[0021] S13, under the condition of time matching, according to the latitude and longitude of the national ground meteorological observation station, extract the latitude and longitude information of the geostationary satellite and the latitude and longitude information of the radar data to complete the spatial information matching;

[0022] S14, integrating the time matching and spatial information matching data to form multi-source data of the site;

[0023] S15. In the multi-source data of the station, the stations whose weather data meet the severe convective weather index data, whose radar combined reflectivity data is greater than 35dbz, and whose geostationary satellite brightness temperature channel value is greater than 30 are marked as severe convective stations, and the other stations are marked as non-severe convective stations;

[0024] S16. Extract all channel data of the geostationary satellite corresponding to the nearest grid point of the matched marked station as features, and form raw data samples for identifying severe convective clouds together with the marking labels of the corresponding stations.

[0025] Specifically, the original severe convective cloud cluster dataset in step S20 is based on the geostationary satellite channel information and the corresponding severe convective labels of the national ground meteorological observation station. The original severe convective cloud cluster dataset is represented by {(x1, y1), (x2, y2), ..., (x m ,y m )}, where y m Indicates the strong convection label corresponding to the mth station. In the strong convection label, 1 indicates strong convection and 0 indicates non-strong convection. m represents the geostationary satellite channel information corresponding to the mth station, and the geostationary satellite channel information of the ith station is represented by x i :(x i1 ,x i2 ,x i3 ,…,x in ), n is the number of geostationary satellite channels.

[0026] Specifically, in step S30, the process of extracting the features of the strong convective cloud data from the original strong convective cloud data set using the random forest RF algorithm and combining the features with the original features of the strong convective cloud data set to form a strong convective cloud data set with new features is as follows:

[0027] Assume that the random forest has T trees, and each sample x i The OOB predicted class at each tree is t(x i ), where t is the index of the tree, t(x i ) is the tree t for sample x i The prediction category of

[0028] Calculate the OOB prediction frequency of each sample on each prediction category, expressed as:

[0029]

[0030] Among them, f k (x i ) represents the sample x i The frequency of being predicted as category k, I is the indicator function, which returns 1 when the condition is true, otherwise it returns 0;

[0031] The OOB prediction frequency is used as the new feature after extraction. For a K classification problem, each sample x i Get K new features, denoted as z i ={f1(x i ),f2(x i ),…,f n (x i )};

[0032] The new features obtained are combined with the original features of the original severe convective cloud dataset to form a severe convective cloud dataset with new features, which is expressed as {(x1∪z1,y1), (x2∪z2,y2), …, (x m ∪z m ,y m )}, which can be simplified as {(d1,y1), (d2,y2), …, (d m ,y m )}.

[0033] Specifically, in step S30, the severe convective cloud dataset is input into the logistic regression LR model, and the process of training and obtaining the severe convective cloud recognition model is as follows:

[0034] The logistic regression model is used to estimate the parameter d in the severe convective cloud dataset. i and i Fitting is performed to estimate the parameters by maximizing the likelihood function and using the gradient descent optimization algorithm. The logistic regression model is expressed as:

[0035]

[0036] Among them, d i is the feature vector, W is the model parameter, T represents the transpose operation, and e is the base of the natural logarithm;

[0037] The loss function of logistic regression uses the cross entropy damage function, which is expressed as:

[0038]

[0039] Among them, m is the number of samples, y i is the true label of the i-th sample, and hθ is the model’s predicted probability for the i-th sample;

[0040] After training the model using the severe convective cloud dataset, a severe convective cloud recognition model was obtained.

[0041] Specifically, after the strong convective cloud recognition model is obtained in step S30, the model is tested using the test set data divided from the strong convective cloud data set, and the hit rate and missed rate of the strong convective cloud are calculated using the confusion matrix.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention uses multi-source data to mark national ground meteorological observation sites, takes the FY-4B geostationary satellite brightness temperature channel as a feature, forms a severe convective cloud dataset, and uses the random forest RF algorithm to mine data features of the severe convective cloud dataset. The new features are input into the logistic regression LR model for training to obtain a severe convective cloud recognition model. In the monitoring of severe convective weather, severe convective clouds can be identified with full coverage around the clock, overcoming the limitations of traditional methods.

[0044] (2) The present invention uses the "out-of-bag" (OOB) samples of the random forest RF algorithm to generate new features, which has the advantages of no data leakage, reduced variance, better prediction model, efficient feature selection and saving computing resources. It is then combined with the original data set and classified using the logistic regression LR model. The prediction accuracy and accuracy of the severe convective cloud recognition model trained in this way are better. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention.

[0046] Figure 2 It is a schematic diagram of the processing process in the embodiment of the present invention.

[0047] Figure 3 Schematic diagram of the confusion matrix of the experimental results of the embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0049] Example

[0050] like Figure 1 to Figure 2 As shown, the method for identifying severe convective clouds using multi-source data based on machine learning includes the following steps:

[0051] S10. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, as well as geostationary satellite data and radar data. Obtain radar combined reflectivity data and geostationary satellite brightness temperature channel data through time-space matching. Comprehensively judge and mark whether severe convective weather occurs at national ground meteorological observation stations, and form raw data samples with severe convection marks that match geostationary satellite channel data and stations. The specific process includes:

[0052] S11. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, select stations that meet the requirements of severe convective weather, and determine the time when severe convective weather occurs;

[0053] S12. According to the time when severe convective weather occurs at the site, the geostationary satellite data of the same period is queried to obtain the most recent geostationary satellite data and radar combined reflectivity data to complete time matching;

[0054] S13, under the condition of time matching, according to the latitude and longitude of the national ground meteorological observation station, extract the latitude and longitude information of the geostationary satellite and the latitude and longitude information of the radar data to complete the spatial information matching;

[0055] S14, integrating the time matching and spatial information matching data to form multi-source data of the site;

[0056] S15. In the multi-source data of the station, the station whose weather data meets the severe convective weather index data, whose radar combined reflectivity data is greater than 35dbz, and whose geostationary satellite brightness temperature channel value is greater than 30 is marked as a severe convective station (the mark label is 1), and the other stations are marked as non-severe convective stations (the mark label is 0);

[0057] S16. Extract all channel data of the geostationary satellite corresponding to the nearest grid point of the matched marked station as features, and form raw data samples for identifying severe convective cloud clusters with the marking labels of the corresponding stations. Table 1 shows an example of raw data samples of severe convective cloud clusters.

[0058]

[0059] Table 1 Element data samples of severe convective clouds

[0060] S20. Preprocess the original data samples with strong convection labels to form an original strong convective cloud cluster dataset. The original strong convective cloud cluster dataset is based on the geostationary satellite channel information and the corresponding strong convection labels of the national ground meteorological observation station. The original strong convective cloud cluster dataset is represented by {(x1, y1), (x2, y2), …, (x m ,y m )}, where y m Indicates the strong convection label corresponding to the mth station. In the strong convection label, 1 indicates strong convection and 0 indicates non-strong convection. m represents the geostationary satellite channel information corresponding to the mth station, and the geostationary satellite channel information of the ith station is represented by x i :(x i1 ,x i2 ,x i3 ,…,x in), n is the number of geostationary satellite channels. In the process of machine learning, the data set is usually divided into a training set and a test set according to a set ratio. For example, in this embodiment, the training set and the test set are divided into a ratio of 8:2, which provides a guarantee for subsequent machine learning to improve the accuracy of prediction, reduce the risk of overfitting, speed up the training speed of the model, and increase the interpretability of the model.

[0061] S30. Use the random forest RF algorithm to extract the features of the strong convective cloud data from the original strong convective cloud data set, and combine them with the original features of the original strong convective cloud data set to form a strong convective cloud data set with new features, specifically:

[0062] Assume that the random forest has T trees, and each sample x i The OOB predicted class at each tree is t(x i ), where t is the index of the tree, t(x i ) is the tree t for sample x i The prediction category of

[0063] Calculate the OOB prediction frequency of each sample on each prediction category, expressed as:

[0064]

[0065] Among them, f k (x i ) represents the sample x i The frequency of being predicted as category k, I is the indicator function, which returns 1 when the condition is true, otherwise it returns 0;

[0066] The OOB prediction frequency is used as the new feature after extraction. For a K classification problem, each sample x i Get K new features, denoted as z i ={f1(x i ),f2(x i ),…,f n (x i )};

[0067] The new features obtained are combined with the original features of the original severe convective cloud dataset to form a severe convective cloud dataset with new features, which is expressed as {(x1∪z1,y1), (x2∪z2,y2), …, (x m ∪z m ,y m )}, which can be simplified as {(d1,y1), (d2,y2), …, (d m ,y m )}.

[0068] Then the severe convective cloud cluster dataset is input into the logistic regression LR model to train the severe convective cloud cluster recognition model, specifically:

[0069] The logistic regression model is used to estimate the parameter d in the severe convective cloud dataset. i and i Fitting is performed to estimate the parameters by maximizing the likelihood function and using the gradient descent optimization algorithm. The logistic regression model is expressed as:

[0070]

[0071] Among them, d i is the feature vector, W is the model parameter (weight and intercept), T represents the transpose operation, and e is the base of the natural logarithm;

[0072] The loss function of logistic regression uses the cross entropy damage function, which is expressed as:

[0073]

[0074] Among them, m is the number of samples, y i is the true label of the i-th sample, and hθ is the model’s predicted probability for the i-th sample;

[0075] After training the model using the severe convective cloud dataset, a severe convective cloud recognition model was obtained.

[0076] Finally, the model was tested using the test set data divided from the severe convective cloud dataset, and the hit rate and missed rate of severe convective clouds were calculated using the confusion matrix. Figure 3 As shown, according to the test experimental results of the model using the test set data, the records in the data set are summarized in the form of a confusion matrix according to the two criteria of the real category and the classification judgment made by the model. The test data set has 426,196 samples, 70,844 marked strong convection samples, and 355,352 non-strong convection samples. By inputting the test data set into the model and making predictions, the marked labels are compared with the prediction results, among which the strong convection hit rate is 96.65%, and the strong convection miss rate is 3.35%. The experimental results show that the present invention can well identify strong convective clouds.

[0077] S40, acquiring geostationary satellite data, and using the severe convective cloud identification model to identify severe convective clouds.

[0078] In summary, compared with traditional physical models, the machine learning model used in the present invention can better capture the dynamic change process of severe convective clouds, improve prediction accuracy, and use multi-source data to identify severe convection, thereby improving the accuracy of samples.

[0079] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and performing non-creative work on this basis should fall within the protection scope of the present invention.

Claims

1. A method for identifying severe convective clouds using multi-source data based on machine learning, characterized in that: The following steps are involved: S10. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, as well as geostationary satellite data and radar data, obtain radar combined reflectivity data and geostationary satellite brightness temperature channel data through time-space matching, comprehensively judge and mark whether severe convective weather occurs at national ground meteorological observation stations, and form raw data samples with severe convection marks that match geostationary satellite channel data and stations; S20, preprocessing the original data samples with strong convection marks to form an original strong convective cloud cluster data set; S30, using the random forest RF algorithm to extract the features of the strong convective cloud data from the original strong convective cloud data set, wherein the OOB prediction frequency of each sample in the original strong convective cloud data set in each prediction category is calculated, and the OOB prediction frequency is used as the extracted new feature; The obtained new features are combined with the original features of the original severe convective cloud dataset to form a severe convective cloud dataset with new features. The severe convective cloud dataset with new features is then input into the logistic regression LR model to train and obtain a severe convective cloud recognition model. S40, acquiring geostationary satellite data, and using the severe convective cloud identification model to identify severe convective clouds.

2. The method for identifying severe convective clouds using multi-source data based on machine learning according to claim 1 is characterized in that: The specific process of step S10 includes: S11. Collect weather data including precipitation and maximum wind speed from national ground meteorological observation stations during historical severe convective weather, select stations that meet the requirements of severe convective weather, and determine the time when severe convective weather occurs; S12. According to the time when severe convective weather occurs at the site, the geostationary satellite data of the same period is queried to obtain the most recent geostationary satellite data and radar combined reflectivity data to complete time matching; S13, under the condition of time matching, according to the latitude and longitude of the national ground meteorological observation station, extract the latitude and longitude information of the geostationary satellite and the latitude and longitude information of the radar data to complete the spatial information matching; S14, integrating the time matching and spatial information matching data to form multi-source data of the site; S15. In the multi-source data of the station, the stations whose weather data meet the severe convective weather index data, whose radar combined reflectivity data is greater than 35dbz, and whose geostationary satellite brightness temperature channel value is greater than 30 are marked as severe convective stations, and the other stations are marked as non-severe convective stations; S16. Extract all channel data of the geostationary satellite corresponding to the nearest grid point of the matched marked station as features, and form raw data samples for identifying severe convective clouds together with the marking labels of the corresponding stations.

3. The method for identifying severe convective clouds using multi-source data based on machine learning according to claim 2 is characterized in that: The original severe convective cloud cluster dataset in step S20 is based on the geostationary satellite channel information and the corresponding severe convective labels of the national ground meteorological observation station. The original severe convective cloud cluster dataset is represented by {(x1, y1), (x2, y2), ..., (x m ,y m )}, where y m Indicates the strong convection label corresponding to the mth station. In the strong convection label, 1 indicates strong convection and 0 indicates non-strong convection. m represents the geostationary satellite channel information corresponding to the mth station, and the geostationary satellite channel information of the ith station is represented by x i :(x i1 ,x i2 ,x i3 ,…,x in ), n is the number of geostationary satellite channels.

4. The method for identifying severe convective clouds using multi-source data based on machine learning according to claim 3 is characterized in that: The process of extracting the features of the strong convective cloud data from the original strong convective cloud data set using the random forest RF algorithm in step S30, and combining the features with the original features of the strong convective cloud data set to form a strong convective cloud data set with new features is as follows: Assume that the random forest has T trees, and each sample x i The OOB predicted class at each tree is t(x i ), where t is the index of the tree, t(x i ) is the tree t for sample x i The prediction category of Calculate the OOB prediction frequency of each sample on each prediction category, expressed as: Among them, f k (x i ) represents the sample x i The frequency of being predicted as category k, I is the indicator function, which returns 1 when the condition is true, otherwise it returns 0; The OOB prediction frequency is used as the new feature after extraction. For a K classification problem, each sample x i Get K new features, denoted as z i ={f1(x i ),f2(x i ),…,f n (x i )}; The new features obtained are combined with the original features of the original severe convective cloud dataset to form a severe convective cloud dataset with new features, which is expressed as {(x1∪z1,y1), (x2∪z2,y2), …, (x m ∪z m ,y m )}, which can be simplified as {(d1,y1), (d2,y2), …, (d m ,y m )}.

5. The method for identifying severe convective clouds using multi-source data based on machine learning according to claim 4 is characterized in that: In step S30, the severe convective cloud dataset is input into the logistic regression LR model, and the process of training and obtaining the severe convective cloud recognition model is as follows: The logistic regression model is used to estimate the parameter d in the severe convective cloud dataset. i and i Fitting is performed to estimate the parameters by maximizing the likelihood function and using the gradient descent optimization algorithm. The logistic regression model is expressed as: Among them, d i is the eigenvector, W is the model parameter, T represents the transpose operation, and e is the base of the natural logarithm; The loss function of logistic regression uses the cross entropy damage function, which is expressed as: Among them, m is the number of samples, y i is the true label of the i-th sample, and hθ is the model’s predicted probability for the i-th sample; After training the model using the severe convective cloud dataset, a severe convective cloud recognition model was obtained.

6. The method for identifying severe convective clouds using multi-source data based on machine learning according to claim 5, characterized in that: After the strong convective cloud recognition model is obtained in step S30, the model is tested using the test set data divided from the strong convective cloud data set, and the hit rate and missed rate of the strong convective cloud are calculated using the confusion matrix.

Images