A lightning near prediction method based on multi-source heterogeneous image sequence fusion

Abstract

The application discloses a lightning near-forecasting method based on multi-source heterogeneous image sequence fusion, and belongs to the technical field of space-time forecasting. The method comprises the following steps: acquiring radar product images, multi-channel satellite infrared images and lightning location images, and constructing input tensors after targeted pre-processing of various data; constructing a lightning forecasting model composed of a geographical consistency division module, a space-time homogenization module, a cross-modal fusion module and a non-autoregressive decoder module; dividing the input tensors into non-overlapping geographical subgraphs based on the geographical consistency principle, and generating isomorphic features through the space-time homogenization module under the premise of preserving the original space-time resolution of each data source; fusing the isomorphic features through the cross-modal fusion module to generate fused space-time features; and predicting multiple frames of future lightning probability fields in parallel through the non-autoregressive decoder module to realize lightning near-forecasting every 6 minutes in the next 1 hour. The application can improve the precision, stability and generalization ability of lightning forecasting.

Description

Technical Field

[0001] This invention relates to the fields of image processing and spatiotemporal prediction technology, and in particular to a lightning proximity prediction method based on the fusion of multi-source heterogeneous image sequences. Background Technology

[0002] Currently, lightning proximity prediction mainly relies on empirical judgments based on thresholds set by single observation indicators such as radar echo intensity and cloud top brightness temperature, or uses deep learning models (such as ConvLSTM, 3D-UNet, SimVP, etc.) to fuse and predict multi-source meteorological data. A typical approach is to first resample data from different sources such as satellites, radar, and lightning location networks, which have different spatiotemporal resolutions, and then forcibly align them to a unified spatiotemporal grid before stitching them together into a multi-channel tensor input model. However, this type of method has the following significant drawbacks:

[0003] 1) Information distortion: The resampling process introduces interpolation errors, smoothing high-frequency details and distorting low-frequency structures, thus compromising the physical consistency of the original observations;

[0004] 2) Ignoring heterogeneous characteristics: Treating multi-source data simply as image channels without modeling their physical semantic differences and spatial dependencies results in limited fusion performance;

[0005] 3) Poor prediction stability: Most mainstream models use an autoregressive architecture for frame-by-frame prediction, which is prone to a sharp drop in long-term prediction performance due to error accumulation. Furthermore, the commonly used transposed convolutional upsampling method is prone to producing a "chessboard effect," generating isolated false lightning regions.

[0006] 4) Weak generalization ability: It is difficult to adapt to multi-source data inputs with different regions and different resolutions, and lacks the ability to flexibly adapt to the length and spatial scale of time series.

[0007] Therefore, there is an urgent need for a lightning nowcasting method that does not require resampling, can directly fuse original multi-source heterogeneous image sequences, and supports long-term prediction. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to provide a lightning proximity prediction method based on the fusion of multi-source heterogeneous image sequences. By introducing a geographic consistency embedding mechanism, the method effectively fuses and collaboratively models radar, satellite, and lightning observation image sequences while preserving the original resolution of each data source, thereby generating a more accurate, continuous, and physically consistent future lightning probability field and achieving lightning proximity prediction.

[0009] Technical solution: A lightning proximity prediction method based on the fusion of multi-source heterogeneous image sequences, comprising the following steps:

[0010] S1: Acquire radar image sequences, satellite infrared cloud image sequences, and lightning location image sequences. After preprocessing various types of data, construct the input tensor.

[0011] S2, constructing a lightning prediction model consisting of a geographic consistency partitioning module, a spatiotemporal homogenization module, a cross-modal fusion module, and a non-autoregressive decoder module;

[0012] The input tensor is divided into non-overlapping geographical subgraphs by the geographic consistency partitioning module, and then processed by the spatiotemporal homogenization module to obtain isomorphic features;

[0013] Channel-spatial attention fusion is performed on isomorphic features through a cross-modal fusion module to generate fused spatiotemporal features;

[0014] By using a non-autoregressive decoder prediction module to process the fused spatiotemporal features, a high-resolution lightning probability field is output, enabling near-term lightning prediction for the next hour.

[0015] S3 trains and optimizes the lightning prediction model, saving the best lightning prediction model for future lightning predictions.

[0016] Furthermore, the radar image sequence is subjected to max-min normalization, the satellite infrared cloud image sequence is subjected to mean-standard deviation normalization, and the lightning location image sequence is subjected to exponential normalization.

[0017] Furthermore, the steps to obtain isomorphic features include:

[0018] SA1 embeds spatial features into the geographic submap at all time steps to obtain the spatial feature set of each geographic submap. T represents the length of the time series. For the spatial features of data source s at time step t;

[0019] SA2, for each geographic submap, sets spatial features. As the input sequence, the time features are embedded through a lightweight time attention module to obtain the fused features of the time steps;

[0020] SA3 performs global average pooling on the fused features of the time step to obtain the final isomorphic features of the current geographic subgraph.

[0021] Furthermore, the implementation steps of the spatial feature embedding are as follows:

[0022] S11, for each data source Within each geographic submap, the corresponding original observation feature vector is extracted for each time step. ,in, , and These represent the number of channels, height, and width corresponding to the geographic submap of data source s, respectively. Represents the set of real numbers;

[0023] S12, through an embedding network consisting of two linear layers, extracts all the original observed feature vectors. After being flattened, it is projected onto a unified high-dimensional embedded space:

[0024] ,

[0025] in, and These are the trainable weight matrices. and These are the corresponding bias vectors. For activation function, These are the features after embedding.

[0026] Furthermore, the implementation steps for embedding temporal features using a lightweight temporal attention module are as follows:

[0027] S21, input sequence The embedding vector at each time step is added to the corresponding time position encoding vector to obtain the embedding sequence with time position information. :

[0028] ,

[0029] in, The position encoding vector at time step t is calculated as follows:

[0030] ,

[0031] Where t is the time step index, i is the embedding dimension index, and d is the embedding dimension;

[0032] S22, embedding the sequence with time location information Input multi-head self-attention mechanism, Linear projection for query, key, and value:

[0033] , , ;

[0034] in, , , These are the weight matrices, Q and Q, respectively. V represents the query, key, and value, respectively; the output of single-head attention is:

[0035] ,

[0036] in, This represents the weighted aggregation result at time step t under the attention head j. It is a scaling factor; express transpose;

[0037] S23 will output single-head attention. By concatenating along the feature dimensions and performing a linear transformation, the fused features for that time step are obtained. Represented as:

[0038] ,

[0039] Where j = 1, 2, 3, ..., h; h is the total number of attention heads. This is the weight matrix.

[0040] Furthermore, the implementation steps for fusing spatiotemporal features are as follows:

[0041] S201: Concatenate the isomorphic features of the geographic submaps from all data sources along the channel dimension to obtain the feature matrix. The channel attention submodule is used to calculate global average pooling and max pooling along the spatial dimension of the feature matrix E. Channel attention weights are generated through a shared linear layer, as shown in the following formula:

[0042] ,

[0043] in, , These are the trainable weights. It is the Sigmoid activation function. This is the channel attention weight matrix; For activation function, Indicates a tie pooling, Indicates max pooling;

[0044] S202, channel attention weight matrix Broadcast to the spatial dimension of the joint feature matrix and multiply element-wise to obtain the channel-enhanced features. ;

[0045] S203, utilizing the spatial attention submodule to enhance channel features Global average pooling and max pooling are calculated along the channel dimension, and spatial attention weights are generated through convolutional layers, as shown in the following formula:

[0046] ,

[0047] in, This represents the convolution operation. Indicates channel splicing. This is the spatial attention weight matrix;

[0048] S204, spatial attention weight matrix With channel enhancement features Element-wise multiplication yields the final fused spatiotemporal features, calculated using the following formula:

[0049] ,

[0050] in, Indicates the fusion of spatiotemporal features, This indicates element-wise multiplication.

[0051] Furthermore, when processing the fused spatiotemporal features, the non-autoregressive decoder module combines the high-resolution lightning probability field output by PixelShuffle upsampling to achieve near-term lightning prediction for the next hour; the implementation steps are as follows:

[0052] SB1 defines N learnable query vectors. If each query vector corresponds to the latent space representation of a future time step, then the number of predicted frames is N;

[0053] SB2 will integrate spatiotemporal features The token sequence is flattened, copied multiple times in the time dimension, and then input into the Transformer decoder. Cross-attention calculation is performed with the learnable query vector to output the latent feature representations at future time steps. The spatial resolution of the latent feature representations is restored through the PixelShuffle upsampling layer, mapping the latent feature representations to a grid size consistent with the lightning tag.

[0054] SB3 maps the upsampled features to a single channel through a 1×1 convolutional layer to obtain a lightning probability prediction map.

[0055] Compared with the prior art, the significant advantages of this invention are as follows:

[0056] 1. To address the issues of loss of original physical information and distortion of spatiotemporal structure caused by forced resampling, this invention introduces a geographic consistency embedding mechanism, sets a baseline geographic range, divides all data sources into the same geographic unit, and achieves effective fusion and collaborative modeling of radar, satellite, and lightning observation image sequences while preserving the original resolution of each data source. This results in a more accurate, continuous, and physically consistent future lightning probability field; it also enables lightning nowcasting without resampling, can directly fuse original multi-source heterogeneous image sequences, and supports long-term prediction.

[0057] 2. This invention employs a geographic consistency partitioning module, which preserves the original spatiotemporal resolution and physical information of multi-source data and combines an attention mechanism to accurately capture the evolution patterns of convective systems, significantly improving prediction accuracy. Compared to mainstream baseline models such as ConvLSTM and 3D-UNet, the probability of detection (POD) is improved by 3.6%, the critical success index (CSI) is improved by 3.1%, and the accuracy is improved by 0.06%.

[0058] 3. The present invention uses a non-autoregressive decoder to avoid the error accumulation problem of autoregressive models, and uses PixelShuffle upsampling to ensure spatial continuity and reduce the false lightning probability zone, which can meet the requirement of accurate prediction for a long time of 1 hour and has strong long-term prediction stability.

[0059] 4. The geographic consistency partitioning module of this invention adopts the geographic consistency principle and maps multi-source data of arbitrary spatial resolution to a unified local feature space through geographic spatial partitioning; it uses a lightweight temporal attention embedding module to aggregate multi-source data of arbitrary temporal resolution in the time dimension, which can be flexibly applied to the fusion of multi-source data with different spatial granularities and different time intervals, and finally outputs a lightning probability map for the next 1 hour every 6 minutes, so as to clarify the time, location and probability of lightning occurrence, and provide reliable data support for severe convective weather warning, infrastructure protection and public safety management. Attached Figure Description

[0060] Figure 1 This is the overall flowchart of the present invention;

[0061] Figure 2 This is a schematic diagram illustrating the data flow within the lightning prediction model;

[0062] Figure 3 This is a structural diagram of a non-autoregressive decoder module;

[0063] Figure 4 This is a visual comparison chart of the prediction results of the present invention and various comparative models. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 The diagram shows a flowchart of the lightning proximity prediction method based on the fusion of multi-source heterogeneous image sequences according to the present invention. This invention uses geospatial data as a unified benchmark to perform lossless segmentation and feature embedding on multi-source heterogeneous image sequences. It then fuses spatiotemporal features through cross-modal attention and utilizes a non-autoregressive decoder to generate a high-resolution probability map of future lightning in parallel. The specific steps are as follows:

[0066] Step 1: Obtain multi-source heterogeneous image sequences;

[0067] Obtain remote sensing image sequences containing at least the following three types as input:

[0068] Radar image sequences: such as combined reflectivity, vertical integrated liquid water content, echo top height, etc. (spatial-temporal resolution: 1 km / 6 min).

[0069] Satellite infrared cloud image sequence: such as multiple infrared channels of the Himawari satellite (spatial-temporal resolution: 5 km / 10 min);

[0070] Lightning location image sequence: Gridded lightning event map (1km / 6 min) generated by lightning detection network.

[0071] Note: Each data source maintains its original spatiotemporal resolution and is not resampled or interpolated for alignment.

[0072] After performing targeted preprocessing on various types of data, an input tensor is constructed. The preprocessing is as follows: mean-standard deviation normalization is performed on satellite infrared cloud image sequences, maximum-minimum normalization is performed on radar image sequences, and exponential normalization is performed on lightning location image sequences.

[0073] Step 2, construct a lightning prediction model;

[0074] like Figure 2 As shown, the lightning prediction model includes a geographic consistency partitioning module, a spatiotemporal homogenization module, a cross-modal fusion module, and a non-autoregressive decoder module;

[0075] The geographic consistency partitioning module addresses the issue of inconsistent spatial resolution among multi-source images (radar, satellite, lightning). While preserving the original spatial resolution of each data source, it divides the entire study area (e.g., the Pearl River Delta) into several local geographic sub-regions (geographic sub-maps). This allows multi-source images of different resolutions to be embedded and fused based on the same geographic semantic units, avoiding information distortion caused by traditional resampling.

[0076] Specifically, a baseline range for local geographic submaps is established within the study area, dividing it into several non-overlapping local geographic submaps. For different data sources, each geographic submap has the same geographic scale (e.g., a rectangular area of ​​latitude and longitude with fixed sides), serving as a common spatial unit for subsequent multi-source data alignment. This partitioning process relies solely on geospatial information and does not involve resampling of the original images, ensuring that each data source maintains its original spatial resolution and physical structure.

[0077] The spatiotemporal homogenization module is used to spatiotemporally embed and align the original observations of various data sources within the geographic submap to address the heterogeneity of multi-source data in terms of spatial resolution and temporal sampling frequency.

[0078] Specifically, for each geographic submap, its original pixel blocks are extracted from various data sources, and these original pixel blocks are mapped to spatial features of a unified dimension through multiple linear layers. Subsequently, a lightweight temporal attention mechanism is introduced to adaptively aggregate sequences at different time steps and temporal resolutions, generating isomorphic spatiotemporal feature representations for each geographic submap. This process effectively decouples spatial partitioning from feature modeling, enabling the lightning prediction model to flexibly adapt to inputs with arbitrary combinations of spatiotemporal resolutions, while avoiding information distortion caused by forced alignment.

[0079] The cross-modal fusion module is used to deeply fuse isomorphic spatiotemporal features from different physical modalities such as radar, satellite, and lightning positioning, and explicitly model their semantic complementarity and spatial dependencies.

[0080] Specifically, firstly, features from each modality within the same geographic submap are concatenated along the channel dimension to form a joint feature tensor. Then, a convolutional block attention module is employed: firstly, a channel attention submodule dynamically evaluates the importance of each modal channel based on global information and assigns weights; secondly, a spatial attention submodule focuses on key regions of convective system evolution, enhancing spatial feature representation. The final output fused features preserve the physical semantic differences between multi-source data while strengthening cross-modal cooperative signals that are indicative of lightning occurrence.

[0081] The non-autoregressive decoder module is used to efficiently and stably generate high-resolution lightning probability fields for multiple future frames, supporting long-term nowcasting.

[0082] Specifically, a set of learnable query vectors is predefined, with each query corresponding to a future prediction time. The fused spatiotemporal features are used as contextual memory, and the hidden state representations of all future time moments are computed in parallel through the Transformer decoder. Subsequently, a sub-pixel convolution (PixelShuffle) upsampling strategy is adopted to restore the low-dimensional feature map to the spatial resolution of the original lightning label. Finally, a pixel-wise lightning occurrence probability is generated through a lightweight output layer.

[0083] This non-autoregressive decoder module avoids the error accumulation problem caused by frame-by-frame recursion in traditional autoregressive models, while ensuring the spatial continuity and physical rationality of the prediction results, making it suitable for accurate lightning proximity prediction.

[0084] Step 3: Subgraph partitioning and spatiotemporal embedding based on geographical consistency;

[0085] To address the spatial heterogeneity of multi-source image sequences, the entire study area is divided into non-overlapping geographic submaps according to a fixed geographic range. For each data source pixel block within a geographic submap, the data is projected to a unified high-dimensional space through two fully connected layers to obtain spatial embedding features. Then, a lightweight temporal attention embedding module (LATE) is introduced to map image sequences of different lengths to a unified temporal dimension.

[0086] Step 31, Geographic submap division;

[0087] For the range is kilometer The research area is kilometers long, and the benchmark range is set as follows: kilometer kilometers, dividing the entire study area into A geographical submap;

[0088] Step 32, spatial feature embedding;

[0089] For each data source Within each geographic subgraph G, the corresponding original observation feature vector is extracted for each time step. ,in, , and These represent the number of channels, height, and width corresponding to the geographic submap of data source s, respectively. It represents the set of real numbers.

[0090] Then, through an embedding network consisting of two linear layers, all the original observed feature vectors are processed. After being flattened, it is projected onto a unified high-dimensional embedded space:

[0091] (1)

[0092] in, and These are the trainable weight matrices. and These are the corresponding bias vectors. For activation function, These are the features after embedding.

[0093] By embedding spatial features into the geographic submap at all time steps, the spatial feature set of each geographic submap is obtained. , Let be the spatial features of data source s at time step t, and T represent the length of the time series.

[0094] Step 33, Temporal feature embedding;

[0095] For each geographic submap, the spatial feature set As the input sequence, the time dimension is aggregated using the lightweight temporal attention module (LATE). The specific steps are as follows:

[0096] First, the input sequence The embedding vector at each time step is added to the corresponding time position encoding vector to obtain the embedding sequence with time position information. :

[0097] (2)

[0098] The position encoding vector at time step t is calculated as follows:

[0099] (3)

[0100] Where t is the time step index, i is the embedding dimension index, and d is the embedding dimension.

[0101] Then, the embedded sequence with time location information A multi-head self-attention mechanism is used to model the dependencies between different time steps and to achieve a unified representation of heterogeneous temporal resolution data.

[0102] Through a multi-head self-attention mechanism, firstly... Linear projection is defined as query, key, and value:

[0103] (4)

[0104] in, , , These are the weight matrices, Q and Q, respectively. V represents the query, key, and value, respectively.

[0105] The output of single-head attention is:

[0106] (5)

[0107] in, This represents the weighted aggregation result at time step t under the attention head j. It is a scaling factor to prevent the dot product from becoming too large. express The transpose of .

[0108] Output of single-head attention By concatenating along the feature dimensions and performing a linear transformation, the fused features for that time step are obtained. Represented as:

[0109] (6)

[0110] Where h is the total number of attention heads, j = 1, 2, 3, ..., h, This is the weight matrix.

[0111] Finally, the fusion features of time steps Perform global average pooling to obtain the final isomorphic features of the current geographic subgraph G.

[0112] Step 4: Channel-spatial attention fusion is performed on the isomorphic features through the cross-modal fusion module to generate fused spatiotemporal features;

[0113] Concatenate the isomorphic features of the geographic submaps from all data sources along the channel dimension to obtain the feature matrix. The channel attention submodule is used to process the feature matrix. Global average pooling and max pooling are calculated along the spatial dimension, and channel attention weights are generated through a shared linear layer, as shown in the following formula:

[0114] (7)

[0115] in, , These are the trainable weights. For the Sigmoid activation function, This is the channel attention weight matrix; For activation function, Indicates a tie pooling, This indicates max pooling.

[0116] Channel attention weight matrix Broadcast to the spatial dimension of the joint feature matrix and multiply element-wise to obtain the channel-enhanced features. Then, the spatial attention submodule is used to enhance the channel features. Global average pooling and max pooling are calculated along the channel dimension, and spatial attention weights are generated through convolutional layers, as shown in the following formula:

[0117] (8)

[0118] in, This represents the convolution operation. Indicates channel splicing. This is the spatial attention weight matrix.

[0119] Finally, the spatial attention weight matrix is... With channel enhancement features Element-wise multiplication yields the final fused spatiotemporal features, calculated using the following formula:

[0120] (9)

[0121] in, Indicates the fusion of spatiotemporal features, This indicates element-wise multiplication.

[0122] Step 5: Process the fused spatiotemporal features using the non-autoregressive decoder module, and combine the high-resolution lightning probability field output by PixelShuffle upsampling to achieve near-term lightning prediction for the next hour.

[0123] like Figure 3 As shown, the non-autoregressive decoder module includes a learnable query vector set, a Transformer decoder, and a PixelShuffle upsampling layer.

[0124] First, define N learnable query vectors. Each query vector corresponds to a latent spatial representation at a future time step, meaning the number of predicted frames is N. Then, the spatiotemporal features are fused. The token sequence is flattened, copied 10 times in the time dimension, and then input into the Transformer decoder. Cross-attention is calculated with the learnable query vector to output the latent feature representations for future time steps. A PixelShuffle upsampling layer then restores the spatial resolution of the latent feature representations, mapping them to a grid size consistent with the lightning label. Finally, a 1×1 convolutional layer maps the upsampled features to a single channel, corresponding to the predicted lightning probability map.

[0125] Step 6: Lightning prediction model training and optimization;

[0126] Dice Loss is used as the loss function. To alleviate the class imbalance problem caused by the sparseness of lightning events:

[0127] (10)

[0128] in, and These are the indices along the height and width of the lightning image, respectively. This is a lightning probability image predicted by a lightning prediction model. The first in Line number Column pixel values, It is the first of the real-image lightning tags in G. Line number The pixel values ​​of the column; This is a true lightning label image (1 indicates lightning occurred, 0 indicates no lightning). This is a lightning probability map predicted by the model, where H and W are the grid sizes. This is a smoothing factor.

[0129] The Adam optimizer and cosine annealing learning rate scheduling are used in conjunction with an early stopping strategy to prevent overfitting, and the best lightning prediction model is saved for lightning prediction.

[0130] In this embodiment, the selected study area is the Pearl River Delta region (latitude and longitude range: 22.17°N–24.97°N, 110.46°E–113.26°E). This region experiences high temperatures and humidity in summer, with abundant water vapor. Simultaneously, influenced by land-sea thermal differences and topography, severe convective weather occurs frequently, resulting in a relatively high frequency and density of lightning occurrences in southern my country. Abundant lightning observation samples can provide sufficient data for the training and validation of the multi-source data fusion model, avoiding the problem of insufficient model generalization ability due to sample scarcity.

[0131] Data sources: Radar product data (6-minute temporal resolution, 1km spatial resolution, 3 channels), satellite infrared data (band08, ​​band10, band11, band16, 10-minute temporal resolution, 5km spatial resolution, 4 channels), and lightning location data (6-minute temporal resolution, 1km spatial resolution, 1 channel); the dataset was constructed using data from 2023 and 2024.

[0132] Model parameters: The baseline size of the geographic submap is set to 10km, divided into 28×28 geographic units; the number of attention heads in the LATE module is 16; the number of prediction frames is N=10 (corresponding to 1 hour, 1 frame every 6 minutes); the number of training rounds is 500, and the initial learning rate is set to 0.0001.

[0133] Step F1, data preprocessing;

[0134] Different preprocessing methods are used for different types of remote sensing data to facilitate model training. Satellite data is normalized by subtracting their mean and dividing by the standard deviation; radar data is normalized using a max-min normalization method, where the maximum and minimum values ​​are obtained by traversing the entire training set; due to the severe skewness of lightning data and the large deviation in frequency range between different grid points, exponential normalization is used, as shown in the following formula:

[0135] (11)

[0136] Where x represents the original data and y represents the normalized data.

[0137] Step F2, model training;

[0138] Lightning prediction model construction: Based on the PyTorch framework, four modules were built: a geographic consistency partitioning module, a spatiotemporal homogenization module, a cross-modal fusion module, and a non-autoregressive decoder module. Figure 2 The data flow shown completes the module connection.

[0139] Training process: The preprocessed tensor data is input into the lightning prediction model. The Adam optimizer and DiceLoss loss function are used, combined with a cosine annealing learning rate scheduler and an early stopping strategy for training. Training is stopped when the loss on the validation set does not decrease for 20 consecutive rounds, and the optimal lightning prediction model parameters are saved.

[0140] Step F3, Prediction and Evaluation;

[0141] Prediction: Input the test set data into the trained lightning prediction model and output a lightning probability map for the next 10 frames.

[0142] Performance evaluation: Probability of detection (POD), accuracy, and critical success index (CSI) were used as evaluation metrics and compared with three baseline models: ConvLSTM, 3D-UNet, and SimVP.

[0143] Step F4, Result of the Example;

[0144] By employing a geographic consistency partitioning module and a non-resampling strategy, the original physical information of multi-source data is fully preserved, avoiding information decay and artifacts common in traditional methods. In the Lightning mission prediction, compared to the best baseline model, POD improved by 3.6%, CSI by 3.1%, and accuracy by 0.06%.

[0145] To intuitively analyze the prediction results, Figure 4 The image shows prediction examples of the present invention and various comparative models, with T=1 representing the first frame. The first row contains a sequence of lightning images from the past hour, the second row contains a sequence of lightning tags for the next hour, and rows 3-6 contain the prediction results of the comparative models and the lightning prediction model of the present invention. It can be seen that the lightning prediction model of the present invention has the highest degree of agreement in predicting the spatiotemporal evolution trend and impact area of ​​lightning.

Claims

1. A lightning proximity prediction method based on the fusion of multi-source heterogeneous image sequences, characterized in that, Includes the following steps: S1: Acquire radar image sequences, satellite infrared cloud image sequences, and lightning location image sequences. After preprocessing various types of data, construct the input tensor. S2, constructing a lightning prediction model consisting of a geographic consistency partitioning module, a spatiotemporal homogenization module, a cross-modal fusion module, and a non-autoregressive decoder module; The input tensor is divided into non-overlapping geographical subgraphs by the geographic consistency partitioning module, and then processed by the spatiotemporal homogenization module to obtain isomorphic features; Channel-spatial attention fusion is performed on isomorphic features through a cross-modal fusion module to generate fused spatiotemporal features; By using a non-autoregressive decoder module to process the fused spatiotemporal features, a high-resolution lightning probability field is output, enabling near-term lightning prediction for the next hour. S3 trains and optimizes the lightning prediction model, and saves the best lightning prediction model for lightning prediction. The steps for obtaining isomorphic features include: SA1 embeds spatial features into the geographic submap at all time steps to obtain the spatial feature set of each geographic submap. T represents the length of the time series. For the spatial features of data source s at time step t; SA2, for each geographic submap, sets spatial features. As the input sequence, the time features are embedded through a lightweight time attention module to obtain the fused features of the time steps; SA3 performs global average pooling on the fused features of the time step to obtain the final isomorphic features of the current geographic subgraph. When processing the fused spatiotemporal features, the non-autoregressive decoder module combines the high-resolution lightning probability field output by PixelShuffle upsampling to achieve near-term lightning prediction for the next hour; the implementation steps are as follows: SB1 defines N learnable query vectors. If each query vector corresponds to the latent space representation of a future time step, then the number of predicted frames is N; SB2 will integrate spatiotemporal features The token sequence is flattened, copied multiple times in the time dimension, and then input into the Transformer decoder. Cross-attention calculation is performed with the learnable query vector to output the latent feature representations at future time steps. The spatial resolution of the latent feature representations is restored through the PixelShuffle upsampling layer, mapping the latent feature representations to a grid size consistent with the lightning tag. SB3 maps the upsampled features to a single channel through a 1×1 convolutional layer to obtain a lightning probability prediction map.

2. The lightning proximity prediction method based on multi-source heterogeneous image sequence fusion according to claim 1, characterized in that, The radar image sequence was subjected to maximum-min normalization, the satellite infrared cloud image sequence was subjected to mean-standard deviation normalization, and the lightning location image sequence was subjected to exponential normalization.

3. The lightning proximity prediction method based on multi-source heterogeneous image sequence fusion according to claim 1, characterized in that, The steps for implementing the spatial feature embedding are as follows: S11, for each data source Within each geographic submap, the corresponding original observation feature vector is extracted for each time step. ,in, , and These represent the number of channels, height, and width corresponding to the geographic submap of data source s, respectively. Represents the set of real numbers; S12, through an embedding network consisting of two linear layers, extracts all the original observed feature vectors. After being flattened, it is projected onto a unified high-dimensional embedded space: ， in, and These are the trainable weight matrices. and These are the corresponding bias vectors. For activation function, These are the features after embedding.

4. The lightning proximity prediction method based on multi-source heterogeneous image sequence fusion according to claim 1, characterized in that, The implementation steps for embedding temporal features using a lightweight temporal attention module are as follows: S21, input sequence The embedding vector at each time step is added to the corresponding time position encoding vector to obtain the embedding sequence with time position information. : ， in, The position encoding vector at time step t is calculated as follows: ， Where t is the time step index, i is the embedding dimension index, and d is the embedding dimension; S22, embedding the sequence with time location information Input multi-head self-attention mechanism, Linear projection for query, key, and value: ， ， ； in, , , These are the weight matrices, Q and Q, respectively. V represents the query, key, and value, respectively; the output of single-head attention is: ， in, This represents the weighted aggregation result at time step t under the attention head j. It is a scaling factor; express transpose; S23 will output single-head attention. By concatenating along the feature dimensions and performing a linear transformation, the fused features for that time step are obtained. Represented as: ， Where j = 1, 2, 3, ..., h; h is the total number of attention heads. This is the weight matrix.

5. The lightning proximity prediction method based on multi-source heterogeneous image sequence fusion according to claim 1, characterized in that, The steps to obtain the fused spatiotemporal features are as follows: S201: Concatenate the isomorphic features of the geographic submaps from all data sources along the channel dimension to obtain the feature matrix. The channel attention submodule is used to calculate global average pooling and max pooling along the spatial dimension of the feature matrix E. Channel attention weights are generated through a shared linear layer, as shown in the following formula: ， in, , These are the trainable weights. It is the Sigmoid activation function. This is the channel attention weight matrix; For activation function, Indicates a tie pooling, Indicates max pooling; S202, channel attention weight matrix Broadcast to the spatial dimension of the joint feature matrix and multiply element-wise to obtain the channel-enhanced features. ; S203, utilizing the spatial attention submodule to enhance channel features Global average pooling and max pooling are calculated along the channel dimension, and spatial attention weights are generated through convolutional layers, as shown in the following formula: ， in, This represents the convolution operation. Indicates channel splicing. This is the spatial attention weight matrix; S204, spatial attention weight matrix With channel enhancement features Element-wise multiplication yields the final fused spatiotemporal features, calculated using the following formula: ， in, Indicates the fusion of spatiotemporal features, This indicates element-wise multiplication.

Images