A lightning early warning artificial intelligence data set labeling method

Abstract

The present application relates to the technical field of thunder and lightning early warning, in particular to a lightning early warning artificial intelligence data set labeling method, scientific and accurate data set is needed for artificial intelligence training, how to accurately standardize the position of thunderstorm cloud cluster and the intensity of lightning activity is crucial to the model training effect, the technology used in the present application is a lightning activity intensity labeling method. Through the lightning positioning system of networking observation, the probability and distribution range of lightning occurrence are quantified, and the problem of easy data representativeness of lightning original data in a short observation period is solved.

Description

Technical Field

[0001] This invention relates to the field of lightning warning technology, specifically to a method for labeling an artificial intelligence dataset for lightning warning. Background Technology

[0002] Lightning is a highly destructive natural disaster that poses a serious threat to aerospace, power transmission, field operations, and public safety. With the development of artificial intelligence, deep learning-based lightning warning models have shown great potential due to their powerful nonlinear fitting capabilities. However, the performance of these models is highly dependent on training with a large amount of high-quality, highly consistent labeled data. Currently, the main methods for obtaining labeled lightning warning data have the following limitations:

[0003] Manual labeling based on expert experience: Traditional methods heavily rely on the experience of meteorological experts, manually determining and classifying thunderstorm activity by comprehensively analyzing radar, satellite, and lightning location data. This method is inefficient, costly, and easily influenced by subjective factors, making it difficult to meet the needs of training large datasets. Simple statistics based on single observation data: Some existing automated methods typically only perform simple spatial gridding counting of lightning location data, directly using the number of lightning strikes as an indicator of thunderstorm activity intensity. However, lightning occurrence is highly random and discontinuous. Simple counting methods cannot effectively characterize the overall structure, development intensity, and movement trend of thunderstorm clouds, resulting in noisy and poorly representative datasets, thus affecting the accuracy and reliability of subsequent AI model warnings.

[0004] Unlike weather elements such as temperature, humidity, pressure, wind, precipitation, and visibility, lightning occurrence is more random and discontinuous. Therefore, data representativeness issues easily arise within a short observation period, and the location of lightning strikes may not effectively represent the probability of lightning occurrence at that moment. This study used VLF three-dimensional lightning locator data to create a dataset with a spatial resolution of 0.01*0.01 degrees and a temporal resolution of 5 minutes. This resulted in a small number of lightning sample points for each time period, leading to spatial and temporal discontinuities and making it difficult to characterize the lightning probability distribution at that specific moment.

[0005] As can be seen from the above reasons, raw lightning data cannot be used as a dataset for efficient model training. A method is needed to convert the raw scattered data into a temporally and spatially continuous lightning probability distribution.

[0006] Therefore, there is an urgent need in this field for a lightning warning dataset labeling method that can be automated, objective, and accurately reflect the physical characteristics of thunderstorm clouds. Summary of the Invention

[0007] The purpose of this invention is to address the problem of how to automatically and efficiently generate high-quality, graded, labeled data from raw lightning observation data without human intervention. This data accurately reflects the activity characteristics of thunderstorm clouds and is suitable for training artificial intelligence models.

[0008] 1. How to automatically and intelligently clean raw lightning data to remove noise and invalid isolated lightning points, while retaining meaningful lightning clusters.

[0009] 2. How to overcome the randomness and discontinuity of lightning occurrence so that the generated dataset can not only reflect the location of lightning occurrence, but also effectively characterize the probabilistic existence, spatial scale and dynamic movement of the thunderstorm cloud behind it.

[0010] 3. How to integrate prior physical knowledge into the data labeling process so that the generated labels not only have statistical significance but also conform to the physical laws of thunderstorm activity.

[0011] 4. How to implement flexible hierarchical labeling to meet the training needs of different levels of lightning warning models (such as binary classification of "present / absent" or multi-level warning of "blue, yellow, orange, red").

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] Step S1: Data cleaning and initial screening. Obtain the raw data from lightning network observations. This data must include at least the time, latitude and longitude location, and intensity information of the lightning occurrence. First, obviously unreliable abnormal lightning records are removed based on a preset physical threshold (e.g., intensity less than 10kA). Then, a density-based spatiotemporal clustering algorithm (e.g., DBSCAN) is used, setting a spatial neighborhood radius (e.g., 5km) and a time window (e.g., 1 minute) to cluster the lightning data. Clusters with fewer than a preset threshold (e.g., 3 clusters) are considered noise or isolated lightning points and are removed. This step aims to remove invalid data and retain large-scale, reliable lightning activity information.

[0014] Step S2: Data Gridding and Frequency Statistics. The lightning data cleaned in Step S1 is interpolated into a regular geographic grid (e.g., a 0.01° × 0.01° latitude and longitude grid). For each grid, the number of lightning strikes occurring within a short time interval (e.g., a warning time window) is counted. To prevent an excessive number of lightning strikes in a single grid from having an undue impact on the overall weight, the statistically obtained lightning frequencies are non-linearly stratified. For example, fewer than 5 strikes are classified as Level 1, 5 to 10 strikes as Level 2, and more than 10 strikes as Level 3. This step transforms discrete point data into regular grid data and preliminarily quantifies the intensity of local lightning activity.

[0015] Step S3: Probabilistic Thunderstorm Activity. To address the data representativeness issue caused by the randomness and discontinuity of lightning occurrences, this step introduces a probability diffusion model. Specifically, a convolution kernel based on a two-dimensional or three-dimensional (time dimension) normal distribution is designed. The lightning level weights of each grid obtained in Step S2 are assigned probabilistic values ​​to their surrounding temporal and spatial neighborhoods. Through convolution operations, a continuous thunderstorm activity probability field is generated. This operation smoothly diffuses the influence of individual lightning points, and the generated thunderstorm activity probability field better reflects the overall spatial range, core location, and movement trend of thunderstorm clouds, weakening the randomness of individual lightning strikes.

[0016] Step S4: Data standardization. The thunderstorm activity probability field (which can be considered as a thunderstorm index) calculated in Step S3 is normalized by using the Min-Max scaling method to uniformly scale the index values ​​of all grid points in the entire dataset to the [0,1] interval. The formula is:

[0017]

[0018] Where I is the original exponent, and I_min and I_max are the minimum and maximum values ​​in the entire dataset, respectively. This step eliminates the dimensional differences between data from different times and regions, making the data distribution more stable and facilitating AI model learning and convergence.

[0019] Step S5: Hierarchical Labeling Output. Based on the final application requirements, the standardized thunderstorm probability field is hierarchically labeled. For example, for a model requiring binary classification (thunderstorm present / absent), a probability threshold (e.g., 0.3) can be set; values ​​above this threshold are labeled as 1 (thunderstorm present), otherwise as 0 (no thunderstorm). For models requiring multi-level warnings (e.g., blue, yellow, orange, red), multiple increasing probability thresholds (e.g., 10%, 30%, 60%) can be set to divide the data into different warning levels. Finally, a gridded dataset with automatically generated labels is output, which can be directly used to train the thunderstorm warning AI model.

[0020] Preferably, the contribution weighting function is:

[0021]

[0022] Where d is the distance between the lightning point and the target point, and σ is the spatial weighting parameter (which can be adjusted according to the characteristics of the region).

[0023] This process is viewed as a spatial two-dimensional convolution operation, using a convolution kernel to smooth the lightning distribution, making neighboring lightning contribute more to the target point, thereby enhancing the continuity and representativeness of the data.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] Through a series of unsupervised data processing steps, the raw, discrete lightning location observation data is transformed into continuous, gridded data that reflects the probability and intensity of thunderstorm clouds, and labels that can be used for supervised learning or reinforcement learning are automatically generated.

[0026] Fully automated and highly efficient: This invention is based entirely on algorithmic processes to transform raw data into labeled data without any manual intervention, which greatly improves data production efficiency and can quickly generate massive amounts of labeled data to meet the needs of AI model training.

[0027] High-quality annotation with clear physical meaning: Through spatiotemporal clustering and probabilistic convolution, the annotation data generated by this invention is no longer a simple lightning count, but a probability field that can reflect the overall characteristics of thunderstorm clouds. This effectively reduces the noise caused by the randomness of lightning, and the annotation results are more in line with the physical laws of thunderstorm activity, thus enabling the training of more accurate AI models.

[0028] Highly flexible and scalable: Key parameters in the algorithm (such as cluster radius, hierarchical threshold, convolution kernel size, and hierarchical threshold) can be adjusted according to actual application scenarios and data, demonstrating good adaptability. This method can easily integrate other data sources (such as radar reflectivity) to further optimize annotation results.

[0029] Standardized output with strong compatibility: The final output data has been standardized and can be flexibly generated into binary or multi-class labels according to the needs of different AI models. It is highly compatible with mainstream artificial intelligence frameworks and can be directly applied to model training and business systems. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of a two-dimensional convolution kernel for a lightning event.

[0031] Figure 2 A schematic diagram of a 3D convolution kernel for lightning event weights;

[0032] Figure 3 This diagram illustrates the original lightning data, two-dimensional convolutional labels, and spatiotemporal three-dimensional convolutional labels. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Please see Figure 1-3This invention provides an embodiment: Step S1: Data cleaning and initial screening, obtaining raw data from lightning network observations, wherein the data includes at least the time, latitude and longitude location, and intensity information of the lightning occurrence. First, obviously unreliable abnormal lightning records are removed based on a preset physical threshold (e.g., intensity less than 10kA). Subsequently, a density-based spatiotemporal clustering algorithm (e.g., DBSCAN) is used to cluster the lightning data, setting a spatial neighborhood radius (e.g., 5km) and a time window (e.g., 1 minute). Clusters with fewer than a preset threshold (e.g., 3 clusters) are considered noise or isolated lightning points and are removed. This step aims to remove invalid data and retain large-scale, reliable lightning activity information.

[0035] Step S2: Data Gridding and Frequency Statistics. The lightning data cleaned in Step S1 is interpolated into a regular geographic grid (e.g., a 0.01° × 0.01° latitude and longitude grid). For each grid, the number of lightning strikes occurring within a short time interval (e.g., a warning time window) is counted. To prevent an excessive number of lightning strikes in a single grid from having an undue impact on the overall weight, the statistically obtained lightning frequencies are non-linearly stratified. For example, fewer than 5 strikes are classified as Level 1, 5 to 10 strikes as Level 2, and more than 10 strikes as Level 3. This step transforms discrete point data into regular grid data and preliminarily quantifies the intensity of local lightning activity.

[0036] Step S3: To address the data representativeness issue caused by the randomness and discontinuity of lightning occurrences, this step introduces a probability diffusion model. Specifically, a two-dimensional ( Figure 1 ) or three-dimensional (time dimension, Figure 2 A normally distributed convolution kernel is used to assign probability values ​​to the lightning level weights of each grid obtained in step S2, and then to its surrounding temporal and spatial neighborhoods. Through convolution operations, a continuous thunderstorm activity probability field is generated. Figure 3 This operation allows the impact of a single lightning strike to be smoothly diffused, and the resulting thunderstorm activity probability field better reflects the overall spatial range, core location, and movement trend of the thunderstorm cloud, thus reducing the randomness of individual lightning strikes.

[0037] Step S4: Data standardization. The thunderstorm activity probability field (which can be considered as a thunderstorm index) calculated in Step S3 is normalized by using the Min-Max scaling method to uniformly scale the index values ​​of all grid points in the entire dataset to the [0,1] interval. The formula is:

[0038]

[0039] Where I is the original exponent, and I_min and I_max are the minimum and maximum values ​​in the entire dataset, respectively. This step eliminates the dimensional differences between data from different times and regions, making the data distribution more stable and facilitating AI model learning and convergence.

[0040] Step S5: Hierarchical Labeling Output. Based on the final application requirements, the standardized thunderstorm probability field is hierarchically labeled. For example, for a model requiring binary classification (thunderstorm present / absent), a probability threshold (e.g., 0.3) can be set; values ​​above this threshold are labeled as 1 (thunderstorm present), otherwise as 0 (no thunderstorm). For models requiring multi-level warnings (e.g., blue, yellow, orange, red), multiple increasing probability thresholds (e.g., 10%, 30%, 60%) can be set to divide the data into different warning levels. Finally, a gridded dataset with automatically generated labels is output, which can be directly used to train the thunderstorm warning AI model.

[0041] Preferably, the contribution weighting function is:

[0042]

[0043] Where d is the distance between the lightning point and the target point, and σ is the spatial weighting parameter (which can be adjusted according to the characteristics of the region).

[0044] This process is viewed as a spatial two-dimensional convolution operation, using a convolution kernel to smooth the lightning distribution, making neighboring lightning contribute more to the target point, thereby enhancing the continuity and representativeness of the data.

[0045] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for labeling an artificial intelligence dataset for lightning early warning, characterized in that: Step S1: Data cleaning and initial screening. Obtain the raw data of lightning network observation. The raw data includes the time, latitude and longitude location and intensity information of lightning occurrence. According to the preset physical threshold, obviously unreliable abnormal lightning records are removed. A density-based spatiotemporal clustering algorithm is used to cluster the lightning data. Clusters with fewer than the preset threshold are regarded as noise or isolated lightning points and are removed. This step aims to remove invalid data and retain large-scale and reliable lightning activity information. Step S2: Data gridding and frequency statistics. The lightning data cleaned in step S1 is interpolated into a regular geographic grid. For each grid, the number of lightning events occurring within a short time interval is counted. To prevent the number of lightning events in a single grid from having too much impact on the overall weight, the lightning frequency obtained is processed by non-linear hierarchical processing. Step S3: Introduce a probability diffusion model. Design a convolution kernel based on a two-dimensional or three-dimensional (time dimension) normal distribution. Assign probability values ​​to the lightning level weights of each grid obtained in Step S2, assigning them to their surrounding temporal and spatial neighborhoods. Through convolution operations, a continuous thunderstorm activity probability field is generated. This operation smoothly diffuses the influence of individual lightning strikes, and the generated thunderstorm activity probability field better reflects the overall spatial range, core location, and movement trend of thunderstorm clouds, weakening the randomness of individual lightning strikes. Step S4: Data standardization. The thunderstorm activity probability field calculated in step S3 is normalized by using the Min-Max scaling method to uniformly scale the exponent values ​​of all grid points in the entire dataset to the interval [0, 1]. The formula is as follows: Where I is the original exponent, and I_min and I_max are the minimum and maximum values ​​in the entire dataset, respectively. This step eliminates the dimensional differences between data from different times and regions, making the data distribution more stable and facilitating AI model learning and convergence; Step S5: Hierarchical labeling output. Based on the final application requirements, the standardized thunderstorm probability field is hierarchically labeled.

2. The annotation method for an artificial intelligence dataset for lightning early warning according to claim 1, characterized in that: The probability diffusion model is designed with a convolution kernel based on a two-dimensional or three-dimensional (time dimension) normal distribution. The lightning level weight of each grid obtained in step S2 is assigned a probability value to its surrounding time and space neighborhood. Through convolution operation, a continuous thunderstorm activity probability field is generated, so that the influence of a single lightning point is smoothly diffused. The generated thunderstorm activity probability field can better reflect the overall spatial range, core location and movement trend of thunderstorm clouds, and weaken the randomness of a single lightning.

3. The annotation method for an artificial intelligence dataset for lightning early warning according to claim 1, characterized in that: The nonlinear hierarchical processing is classified as follows: less than 5 times is classified as level 1, 5 to 10 times is classified as level 2, and more than 10 times is classified as level 3. Step S2 converts discrete point data into regular grid data and initially quantifies the intensity of local lightning activity.

Images