A thunderstorm nowcasting method based on physical gating and cross-modal attention

Abstract

The present application relates to the technical field of weather forecast and deep learning cross, and particularly relates to a thunder and lightning near-forecast method based on physical gating and cross-modal attention. The thunder and lightning prediction model provided by the present application predicts the thunder and lightning count grid sequence in a future time period based on the thunder and lightning count grid sequence, the radar VIL image sequence and the satellite IR image sequence in a recent time period. The thunder and lightning prediction model comprises a physical gating encoder, a sparse thunder and lightning encoder, a cross-modal feature fusion module and a time series prediction module. The present application encodes the data of different modes in view of the physical differences between modes, ensures that the physical meaning of the fused features is clear, and ensures the interpretability of the model. The present application solves the defects of the existing thunder and lightning near-forecast method based on multi-modal fusion, such as lack of interpretability, insufficient interaction between modes, complex structure and the need to introduce additional physical constraints.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of weather forecasting and deep learning, and in particular to a lightning nowcasting method based on physical gating and cross-modal attention. Background Technology

[0002] Lightning is a severe weather process triggered by strong convective motion, accompanied by flashes, heavy precipitation, and strong winds. Traditional numerical weather prediction methods rely on atmospheric state variables and the microphysical process equations of lightning to construct lightning evolution models to predict lightning occurrence areas. However, due to the diverse data types, low value density, and strong coupling of lightning microphysical processes, traditional numerical weather prediction methods are insufficient for small- to medium-scale lightning forecasting needs and suffer from high computational complexity and time consumption. In recent years, deep learning methods have been widely used in precipitation and lightning nowcasting due to their strong spatiotemporal feature extraction and nonlinear modeling capabilities. However, forecasting methods based on a single lightning mode still have many shortcomings. On the one hand, extrapolating from historical lightning distributions does not consider the atmospheric environment supporting lightning evolution, making it difficult to characterize the evolution of lightning at different life cycles; on the other hand, extrapolating solely from historical lightning distributions leads to error accumulation when predicting multiple future steps, resulting in unstable prediction performance and requiring further improvement in interpretability. Therefore, current research has gradually shifted towards multimodal fusion forecasting methods. These methods unify multi-source observation data such as radar, satellite, and meteorological station data into the same feature space, taking into account both the spatiotemporal characteristics of lightning evolution and the background of convection development, which can effectively improve the accuracy and reliability of forecasts.

[0003] Existing lightning nowcasting methods based on multimodal fusion can be categorized by fusion method into splicing-based fusion, multi-task fusion, and generative fusion. Splicing-based fusion can be further divided into early, mid-term, and late-term fusion based on the timing of fusion. Early fusion merges channels of different modalities at the input layer, but may disrupt the structure of each modality. Mid-term fusion encodes different modalities and then merges them, modeling the semantic feature relationships between different modalities, but requires a reasonable modality alignment mechanism. Late-term fusion first predicts each modality separately and then weights the prediction results, making it easy to analyze the contribution of each modality, but the prediction process lacks intermodal interaction. Multi-task fusion divides lightning nowcasting into a main lightning prediction task and several related auxiliary tasks such as precipitation prediction and wind field prediction, with each task sharing some encoders; however, the division of auxiliary tasks and the design of the interaction network are relatively complex. Generative fusion unifies multimodal data into the same feature space, encodes the prediction results through a generator, and then evaluates and calculates the loss through a discriminator to continuously optimize the generation results, thereby maintaining good clarity in long-term time-series prediction tasks. However, it requires the introduction of additional physical constraints and the computational complexity is usually high. Summary of the Invention

[0004] To overcome the shortcomings of existing multimodal fusion-based lightning nowcasting methods, such as lack of interpretability, insufficient intermodal interaction, and complex structure, which require the introduction of additional physical constraints, this invention proposes a lightning nowcasting method based on physical gating and cross-modal attention, which has extremely strong interpretability and reliability.

[0005] By adding a physical gated encoder, a sparse lightning encoder, and a cross-modal feature fusion network to the ConvLSTM time series prediction model, radar VIL images, satellite IR images, and historical lightning count grids are encoded respectively. The fusion weights of environmental features and lightning features are dynamically adjusted through an environmental consistency gating mechanism, thereby constructing lightning fusion features supported by physical environmental evidence and improving the accuracy and reliability of lightning nowcasting.

[0006] This invention proposes a lightning nowcasting method based on physical gating and cross-modal attention. It constructs a lightning prediction model based on a lightning count grid sequence and a radar VIL image sequence from the most recent time period. and satellite IR image sequences Predict lightning count grid sequences over future time periods; The lightning prediction model includes: a physical gated encoder, a sparse lightning encoder, a cross-modal feature fusion module, and a time-series prediction module; The physical gated encoder includes: a first physical gated branch, a second physical gated branch, a first pointwise convolutional unit, and a first channel attention unit; the first physical gated branch processes radar VIL image sequences. Perform high-value gating processing to obtain the gating output. The second physical gated branch controls the satellite IR image sequence. Perform low-value gating processing to obtain the gating output. Gated output and Historical environment features are obtained by dimensionality blending through the first pointwise convolutional unit and then channel attention being extracted by the first channel attention unit. ; A sparse lightning encoder performs sparse encoding on a lightning counting grid sequence to obtain historical lightning features. ; The cross-modal feature fusion module employs a cross-modal attention mechanism to fuse historical environmental features. and historical lightning characteristics , to obtain fused temporal features The lightning count grid for multiple future moments is obtained by integrating the temporal features and processing them through the temporal prediction module.

[0007] Preferably, the cross-modal feature fusion module includes a cross-modal attention unit, a fourth-channel attention unit, and an environment consistency gating unit; Cross-modal attention units with historical lightning features Generate query vector Q based on historical environmental features Generate key vector K and value vector V, and construct environment retrieval feature A through an attention mechanism; then process the environment fusion feature through the fourth channel attention unit. ; Environmental consistency gating unit for gating features and gating features The environment consistency map is obtained by multiplying the max pooling results pixel by pixel. Then to , and The concatenated results are sequentially subjected to convolution, group normalization, activation processing, convolution, and reactivation to obtain the environment gating map. Then, using an environmental gating graph As a feature of mask fusion environment fusion and historical lightning characteristics The fused temporal features were obtained.

[0008] Preferably, the first physical gating branch and the second physical gating branch have the same framework structure and are collectively referred to as physical gating branches. They extract channel attention from the multidimensional convolutional features of the input data, perform gating processing on the input data, and then multiply the gating features and channel attention features pixel by pixel to obtain the gating output.

[0009] Preferably, the physical gating branch includes: a gating unit, a two-dimensional convolutional branch, a lateral convolutional branch, a longitudinal convolutional branch, a second pointwise convolutional unit, a second channel attention unit, and a feature fusion unit; the gating unit first processes the input data sequence Normalization is performed, and then the normalized values ​​are gated to obtain the gated features. Two-dimensional convolutional branches, horizontal convolutional branches, and vertical convolutional branches are set in parallel to process the input data. Two-dimensional convolution, horizontal convolution, and vertical convolution are performed. The three convolutions are then concatenated in dimensions by the second pointwise convolutional unit before being processed again to obtain a hybrid convolution. Hybrid Convolution The second channel attention unit is input, and group normalization, activation processing, and channel attention extraction are performed sequentially to obtain attention features. The feature fusion unit controls the gating features. and attention features Pixel-by-pixel multiplication yields the gated output. .

[0010] Preferably, the sparse lightning encoder includes a lightning logarithmic intensity unit, a neighborhood risk unit, a deformable convolution unit, and a third-channel attention unit; The lightning logarithmic intensity cell is used to perform logarithmic processing on the lightning counting grid sequence Y to obtain the logarithmic intensity map. The neighborhood risk unit uses a binary function to extract neighborhood risk zone features based on the lightning occurrence location, resulting in a neighborhood risk map. ; Log-intensity plot and neighboring risk map After splicing and channel blending, historical lightning features are extracted and filtered by deformable convolutional units and third-channel attention units. .

[0011] Preferably, during model training, the model parameters are updated by backpropagating gradients through the total loss function, which includes location intensity regression loss and neighborhood hit loss. Location intensity regression loss is used to constrain the intensity difference between the predicted lightning count and the actual lightning count at the corresponding spatial location and the corresponding prediction time. Neighborhood hit loss is used to constrain the location of lightning strikes.

[0012] Preferably, the neighborhood hit loss L nh for: ; Where sum represents summing all pixel values ​​across all grid cells; , and These are the weights for the missed detection loss, false alarm loss, and precise location loss, respectively. P represents the continuous predicted response map. B is a binary lightning occurrence diagram. ; The threshold for determining the occurrence of lightning. is the slope coefficient of the soft thresholding function; sigmoid is the activation function; bin is a binary function; To predict the neighborhood response map, , For true neighborhood allowable graphs, , where r is the neighborhood radius.

[0013] Preferably, the location intensity regression loss adopts the MAE loss function.

[0014] Preferably, the time-series prediction module uses a convolutional long short-term memory network (ConvLSTM).

[0015] The present invention proposes a lightning nowcasting system based on physical gating and cross-modal attention, comprising a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to implement the lightning nowcasting method based on physical gating and cross-modal attention.

[0016] The advantages of this invention are: (1) This invention addresses the different physical meanings of radar VIL images, satellite IR images, and lightning counting grids by designing physical gating encoders and sparse lightning encoders respectively. Specifically, radar VIL images are gated with high-value physical gating to highlight areas with strong echoes and high liquid water content, while satellite IR images are gated with low-value physical gating to highlight areas with low brightness and cold cloud tops, making it easier for the model to capture key convective environmental features of lightning occurrence. This invention addresses the physical differences between modes by specifically encoding data from different modes, ensuring the clear physical meaning of the fused features and guaranteeing the interpretability of the model.

[0017] (2) The present invention introduces hybrid convolution in the physical gate encoder, combining conventional two-dimensional convolution, lateral convolution and longitudinal convolution, which is beneficial for extracting local strong convection structure, horizontal structure and vertical structure features.

[0018] (3) In view of the characteristics of strong sparsity, local bursts and temporal locality of lightning counting grid, the present invention constructs a sparse lightning encoder, encodes the lightning counting grid into a logarithmic intensity map and a binary neighborhood risk map, and extracts the spatial offset features of lightning through deformable convolution, thereby improving the ability of the constructed features to express the sparse lightning region and spatiotemporal local shape features.

[0019] (4) The present invention uses cross-modal attention unit and environment consistency gating unit to perform fusion feature fusion, retrieves environmental features through lightning features, and dynamically adjusts the fusion weights according to the consistency between VIL, IR and recent lightning activity, which can reduce the interference of unreliable environmental information on the prediction results and improve the credibility and robustness of multimodal fusion.

[0020] (5) In the training process, the present invention uses location intensity loss and neighborhood hit loss together, which not only constrains the lightning count intensity, but also allows reasonable small-scale spatial offset in lightning nowcasting, which is conducive to improving the hit capability of lightning targets, reducing isolated false alarm points and over-smoothing phenomena, thereby improving the overall effect of future multi-step lightning nowcasting. Attached Figure Description

[0021] Figure 1 This is a structural diagram of the lightning prediction model proposed in this invention; Figure 2 This is a structural diagram of the physical gate encoder proposed in this invention; Figure 3 This is a structural diagram of the sparse lightning encoder proposed in this invention; Figure 4 This is a structural diagram of the cross-modal feature fusion module proposed in this invention; Figure 5 The flowchart is a lightning nowcasting method based on physical gating and cross-modal attention proposed in this invention. Figure 6 A visual comparison of the actual lightning distribution, ConvLSTM, SimVPv2, and the lightning nowcasting method proposed in this invention; Figure 7 The graph shows a performance comparison between ConvLSTM, SimVPv2, and the lightning nowcasting method proposed in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] This implementation proposes a lightning prediction model based on recent lightning observation sequences and radar VIL image sequences. and satellite IR image sequences Predict lightning count grid sequences over future time periods.

[0024] ; ; in, , , and These represent the radar VIL images of the Nth extracted frame, the (N-1)th extracted frame, the second extracted frame, and the most recent extracted frame, respectively, within the historical time period. , , and These represent the Nth, (N-1)th, second, and most recently extracted satellite IR images from the historical time period.

[0025] like Figure 1 As shown, the lightning prediction model includes: a physical gated encoder, a sparse lightning encoder, a cross-modal feature fusion module, and a time-series prediction module.

[0026] like Figure 2As shown, the physical gate encoder includes: a first physical gate branch, a second physical gate branch, a first pointwise convolutional unit, and a first channel attention unit.

[0027] The first and second physical gated branches have the same framework structure and are collectively referred to as physical gated branches. They extract channel attention from the multidimensional convolutional features of the input data, perform gated processing on the input data, and then multiply the gated features pixel-by-pixel with the channel attention features to obtain the gated output. The first and second physical gated branches are used to process radar VIL image sequences, respectively. and satellite IR image sequences .

[0028] The physical gating branch includes: gating unit, two-dimensional convolutional branch, lateral convolutional branch, longitudinal convolutional branch, second pointwise convolutional unit, second channel attention unit, and feature fusion unit.

[0029] The gating unit first processes the input data sequence Normalization is performed, and then the normalized values ​​are gated to obtain the gated features. .

[0030] Let the input data sequence be denoted as N is the sequence length. , and The input data sequences are respectively The 1st, 2nd, and Nth data points in the input data sequence. The normalization formula is as follows: ; in, The input sequence after normalization, , and They are respectively , and The normalized value.

[0031] The two-dimensional convolutional branch, the horizontal convolutional branch, and the vertical convolutional branch are set in parallel, respectively processing the input data. Perform two-dimensional convolution, horizontal convolution, and vertical convolution to obtain the convolution result. , and k is the kernel size parameter (height or width); the three are concatenated dimensionally through the second pointwise convolutional unit and then convolved to obtain a hybrid convolution. Hybrid Convolution The second channel attention unit is input, and group normalization, activation processing, and channel attention extraction are performed sequentially to obtain attention features. Concat stands for concatenation, Conv stands for convolution, GN stands for group normalization, SiLU is the activation function, and CA (Channel Attention Mechanism) is the channel attention mechanism.

[0032] Feature fusion unit for gating features and attention features Pixel-by-pixel multiplication yields the gated output. ; For learnable gating strength coefficients, This indicates pixel-by-pixel multiplication.

[0033] Regions with higher VIL values ​​typically correspond to stronger convection development environments. The gating units in the first physical gating branch employ high-value gating, and their processing formula is expressed as follows: ; ; in, For radar VIL image sequences Gating features, for The normalized sequence; The learnable scaling factor is preferably obtained through a non-negative constraint function to ensure the monotonicity of the gating function; This is the gate threshold; To prevent division by zero of constants.

[0034] Low brightness temperature regions in satellite IR images typically correspond to cold cloud top regions. The gating unit in the second physical gating branch uses low-value gating to highlight these low brightness temperature regions. The processing formula is as follows: ; ; in, Satellite IR image sequence Gating features, for The normalized sequence, The learnable scaling factor. This is the gate threshold; To prevent division by zero of constants.

[0035] In practice, let k=3. The first physical gated branch corresponds to the radar VIL image sequence. The processing procedure is expressed by the formula: ; ; ; ; ; in, , and These are radar VIL image sequences. Two-dimensional convolution, horizontal convolution, and vertical convolution. For radar VIL image sequences attentional characteristics For learnable gating strength coefficients, Indicates pixel-by-pixel multiplication. For radar VIL image sequences The gated output, i.e. the output of the first physical gated branch.

[0036] The second physical gating branch for satellite IR image sequences The processing procedure is expressed by the formula: ; ; ; ; ; in, , and These are satellite IR image sequences. Two-dimensional convolution, horizontal convolution, and vertical convolution. Satellite IR image sequence attentional characteristics For learnable gating strength coefficients, Indicates pixel-by-pixel multiplication; Satellite IR image sequence The gated output, i.e. the output of the second physical gated branch.

[0037] The structure of the first pointwise convolutional unit is the same as that of the second pointwise convolutional unit, and the structure of the first channel attention unit is the same as that of the second channel attention unit. Gated output. and After dimension concatenation by the first pointwise convolutional unit, the convolutional process is performed again. The convolution result is input into the first channel attention unit, where group normalization, activation processing, and channel attention extraction are performed sequentially to obtain historical environment features. The processing steps of the first pointwise convolutional unit and the first channel attention unit are expressed by the following formula: .

[0038] like Figure 3 As shown, the sparse lightning encoder includes a lightning logarithmic intensity unit, a neighborhood risk unit, a deformable convolution unit, and a third-channel attention unit.

[0039] The lightning logarithmic intensity cell is used to perform logarithmic processing on the lightning counting grid sequence Y to mitigate the effects of extreme values. The logarithmic intensity plot output by the lightning logarithmic intensity cell is denoted as... , where log represents the logarithm of the natural logarithm e. The lightning logarithmic intensity unit is used to preserve lightning count intensity information and reduce the impact of extreme count values ​​on model training.

[0040] Neighborhood risk units are used to construct risk areas near lightning occurrence points. Features of these neighborhood risk areas are extracted based on the lightning occurrence location, enabling the model to focus on small-scale spatial offsets around the lightning occurrence point. Specifically, the neighborhood risk map is obtained by binarizing the lightning counting grid sequence Y using the following formula. ; ; ; Where Y is the lightning counting grid sequence; This is a binarized image; bin is a binarization function used to convert cells in the grid with counts greater than a threshold. Grid points are marked as 1, and the rest are marked as 0; For neighborhood risk maps, For a fixed two-dimensional Gaussian kernel, · represents a two-dimensional convolution with a kernel size of 5 and a standard deviation of 1.

[0041] Log-intensity plot and neighboring risk map After splicing and channel blending, historical lightning features are extracted and filtered by deformable convolutional units and third-channel attention units. As the output of a sparse lightning encoder, this process is expressed by the following formula: ; Here, DConv represents deformable convolution, which can adaptively adjust the convolution sampling position according to the input features, thereby extracting the spatial offset features of lightning activity. CA represents channel attention.

[0042] like Figure 4 As shown, the cross-modal feature fusion module includes a cross-modal attention unit, a fourth-channel attention unit, and an environment consistency gating unit.

[0043] Cross-modal attention units with historical lightning features Generate query vector Q based on historical environmental features Generate key vector K and value vector V, construct environment retrieval feature A through an attention mechanism and perform 1×1 convolution. The convolution result is then processed by the fourth channel attention unit for group normalization, non-linear activation and channel attention to obtain the environment fusion feature. .

[0044] ; ; in, Let K be the dimension of the key vector.

[0045] The environmental consistency gating unit generates an environmental gating map based on the VIL gating map, IR gating map, and environmental consistency map, and performs weighted fusion of environmental fusion features and lightning features based on the environmental gating map.

[0046] Environmental consistency gating unit fusion VIL gating map (fusion radar VIL image sequence) (Gating characteristics) IR-gated image (satellite IR image sequence) (Gating characteristics) Obtain the environmental consistency map This is used to represent the spatial consistency between strong echo regions and cold cloud top regions; then the VIL-gated map is... IR gated diagram Consistency diagram with environment The environmental gating map is obtained by fusion processing. Then through the environmental gating diagram Processing environmental integration characteristics Obtain fusion time series features As input to the time series prediction module.

[0047] The processing procedure of the environmental consistency gating unit is expressed by the following formula: ; ; ; in, Indicates pixel-by-pixel multiplication. This is the IR gating graph after max pooling. To integrate temporal features, As an environmental integration feature, This is a characteristic of historical lightning.

[0048] The temporal prediction module is implemented using a Convolutional Long Short-Term Memory (ConvLSTM) network, and uses the fused temporal features output by the cross-modal feature fusion module. As input, the hidden state and memory state are recursively updated by multi-layer ConvLSTM units while maintaining the two-dimensional spatial structure. The future time-series hidden features are output time-by-time. The future hidden features are mapped to the future lightning prediction map sequence through convolutional layers and 1×1 prediction convolutional layers, and the prediction results are constrained to be non-negative values ​​by a non-negative activation function.

[0049] In practice, ConvLSTM comprises a temporal encoder and a temporal decoder, each consisting of four ConvLSTM units. Each ConvLSTM unit receives input features and the hidden state from the previous time step at the current historical moment, and recursively updates the hidden and memory states. When a hidden or memory state is nonexistent, it is initialized to a zero tensor. In subsequent embodiments, the ConvLSTM units of the temporal encoder recursively update the fused features from the past 60 minutes to the current time step by step over 13 frames. After the recursion is completed, each layer of ConvLSTM units outputs the hidden and memory states from the last historical moment as the initial state of the corresponding ConvLSTM unit in the decoder. Using these states as initial conditions, the temporal decoder progressively predicts the lightning count grid for 12 frames over the next 5 to 60 minutes.

[0050] Training a lightning prediction model involves the following steps: S1. Construct the learning dataset, and denote the training samples as {(Y, , ); }, where Y is the lightning count grid sequence over a historical time period, For future lightning counting grid sequences; ; ; in, , , and These represent the lightning count grids for historical time steps t-(N-1), t-(N-2), t-1, and t, respectively. , and These represent the lightning count grid prediction values ​​at future time steps t+1, t+2, and t+T, respectively, where T is the prediction time step length.

[0051] This is a sequence of radar VIL images. For satellite IR image sequences: ; ; in, , , and These represent the radar VIL images of the Nth extracted frame, the (N-1)th extracted frame, the second extracted frame, and the most recent extracted frame in the historical time period, respectively, and correspond to historical time steps t-(N-1), t-(N-2), t-1, and t, respectively; , , and These represent the Nth, N-1th, second, and most recently extracted satellite IR images within the historical time period, respectively, and correspond to historical time steps t-(N-1), t-(N-2), t-1, and t.

[0052] In practice, the historical data is first organized, and the lightning observation data is organized into a lightning counting grid. Then, the lightning counting grid, radar VIL images, and satellite IR images are time-aligned and spatially aligned. Finally, frames are extracted to construct training samples {(Y, , ); }

[0053] In specific implementation, 13 frames (N=13) of lightning grid sequences, radar VIL image sequences, and satellite IR image sequences can be extracted at equal intervals from the past 60 minutes to the current time within the same area as historical sequences; 12 frames of equally spaced lightning grid sequences can be extracted from the next 5 minutes to the next 60 minutes as future lightning prediction map sequences. The above time length is only a preferred implementation method, and other lengths can be set according to forecast service requirements.

[0054] In practice, lightning event samples can be organized into data items including sample number, lightning sequence to be predicted (lightning grid sequence in the future time period), historical lightning sequence (lightning grid sequence in the historical time period), event identifier, and historical environmental sequence (including radar VIL image sequence and satellite IR image sequence in the historical time period). Training set, validation set and test set are constructed according to the preset division method for the training and evaluation of the lightning nowcasting network.

[0055] S2. Train the lightning prediction model on the learning dataset. During the training process, backpropagate the gradient through the total loss function to update the model parameters until the model converges.

[0056] The total loss function includes location intensity regression loss and neighborhood hit loss.

[0057] Location intensity regression loss MAE loss is used to constrain the intensity difference between the predicted lightning count and the actual lightning count at the corresponding spatial location and the corresponding prediction time: ; in, For future lightning counting grid sequences, This represents the predicted future lightning count grid sequence output by the lightning prediction model.

[0058] Considering the spatial locality of lightning occurrence, small-scale location shifts should not be entirely considered prediction failures. This embodiment introduces a neighborhood hit loss to constrain the lightning occurrence location while allowing for small-scale spatial shifts. Its construction method is as follows: S31. Convert the predicted values ​​of the future lightning count grid sequence into a continuous predicted response map P using a soft thresholding function, and then convert the actual future lightning count grid sequence... Convert to binary lightning occurrence diagram B: ; ; in, The threshold for determining the occurrence of lightning. θ is the slope coefficient of the soft thresholding function; sigmoid is the activation function; bin is a binary function that sets the value to 1 at the corresponding grid position when the actual lightning count is greater than or equal to the threshold θ, and 0 otherwise. S32. For a neighborhood radius r, construct a predicted neighborhood response map. and the true neighborhood allowable graph : ; ; Here, AvgPool represents layer normalization, used to measure whether there is a continuous predicted response near the actual lightning point; MaxPool represents max pooling, used to expand the neighborhood of the actual lightning occurrence area to form a false alarm tolerance region, so that small-scale prediction offsets near the actual lightning point are not directly counted as false alarms. Compared with max pooling, average pooling can avoid a single isolated high-response point covering the entire neighborhood, thereby reducing point-like or grid-like false predictions.

[0059] S33. Define the loss due to missed reporting. False alarm losses and optional precise location loss The neighborhood hit loss is obtained by weighted summation of the three factors. ; ; ; ; Where sum represents summing all pixel values ​​across all grid cells; Used to penalize situations where the predicted response is insufficient within the neighborhood of the actual lightning location; This is used to punish predicted responses outside the neighborhood of actual lightning strikes, thereby suppressing false alarms; This is an optional precise location constraint. When its weight is set to 0, the model accepts a reasonable spatial offset within the neighborhood radius.

[0060] ; in, , and These are the weights for underreporting loss, false alarm loss, and precise location loss, respectively.

[0061] The total loss function is: ; in, and , respectively, are the weights of the location intensity regression loss and the neighborhood hit loss, and s is the constraint coefficient.

[0062] To avoid the neighborhood hit loss accounting for too high a proportion of the total loss when positive lightning samples are sparse, this embodiment introduces a constraint coefficient s to proportionally constrain the contribution of the neighborhood hit loss: ; in, The percentage of the largest contribution of neighborhood hit loss to the total loss. To prevent division by zero, the constant `stopgrad` indicates that the scaling factor is not included in gradient backpropagation during model training. This means that at regular intervals during training, the constraint coefficient `s` is updated using the loss function, and in each interval, the model parameters are not updated; the model proceeds directly to the next iteration.

[0063] By using this ratio constraint, the neighborhood hit loss can improve the lightning location hit capability, while avoiding excessive dominance of the optimization direction in sparse positive sample batches.

[0064] In this embodiment, the predicted future lightning counting grid sequence is: The actual future lightning counting grid sequence is ,in and All of these can be represented as a spatiotemporal grid tensor of [B0,T,1,H,W], where B0 is the batch size, T is the prediction time step, and H and W are the spatial grid sizes.

[0065] like Figure 5As shown, this embodiment proposes a lightning nowcasting method based on physical gating and cross-modal attention, which includes the following steps: St1, Obtain lightning observation sequences and radar VIL image sequences of the target area over a historical time period. and satellite IR image sequences The lightning observation sequence is converted into a lightning counting grid sequence; St2, time-aligned lightning count grid sequence, radar VIL image sequence and satellite IR image sequences Input the lightning prediction model to obtain the predicted values ​​of the lightning count grid sequence over the future time period.

[0066] The following specific embodiments are used to verify the above-mentioned lightning prediction model and the lightning nowcasting method based on physical gating and cross-modal attention.

[0067] In this embodiment, the input data includes radar VIL image sequences, satellite IR image sequences, and historical lightning count grid sequences within the same time period. The radar VIL image sequences and satellite IR image sequences, after spatial scale unification, maintain the same spatial resolution as the lightning count grid sequences. The modal data are arranged at equal time intervals and divided into historical sequences and sequences to be predicted. Specifically, the historical sequence can be set as a 13-frame equally spaced sequence from the past 60 minutes to the current moment, and the sequence to be predicted can be set as a 12-frame equally spaced lightning grid sequence from the next 5 minutes to the next 60 minutes.

[0068] In this embodiment, the parameters are set as follows: location intensity regression loss weight. =1.0, neighborhood hit loss weight =0.1, threshold for determining the occurrence of lightning. =1.0, the slope coefficient of the soft threshold function =5.0, neighborhood radius r=1, underreporting loss weight =1.5, False Alarm Loss Weight =2.0, precise location loss weight =0.0, the maximum contribution ratio of neighborhood hit loss =0.5.

[0069] This embodiment provides comparative experimental results of ConvLSTM, SimVPv2, and the proposed method of this invention. Figure 6 For visual comparison charts, Figure 7 This is a performance comparison chart.

[0070] Experiment 1: ConvLSTM, using only historical lightning count grid sequences as input, extracts features from the historical lightning count grid sequences through 3×3 convolution, GN and SiLU, and then inputs them into the ConvLSTM temporal prediction network for future multi-step prediction; Experiment 2: SimVPv2, using only historical lightning count grid sequences as input, extracts features from the historical lightning count grid sequences through 3×3 convolution, GN and SiLU, and then inputs them into the SimVPv2 temporal prediction network for future multi-step prediction; Experiment 3: Proposed, using the lightning nowcasting method described in this invention, the input is a radar VIL image sequence, a satellite IR image sequence, and a historical lightning count grid sequence. After constructing fused temporal features through a physical gate encoder, a sparse lightning encoder, and a cross-modal feature fusion module, the fused temporal features are input into a ConvLSTM temporal prediction network for future multi-step prediction.

[0071] In this embodiment, the learning dataset is the 2018-2019 SEVIR lightning forecast dataset. After quality control, the total number of events in the dataset is 2259. The learning dataset is randomly divided into training, validation, and test sets in a 7:2:1 ratio. The random seed (the initial state value of the pseudo-random number generator) is 42. The resulting training set has 1582 events, the validation set has 451 events, and the test set has 226 events. In all three experiments, the spatial scale of the learning dataset is uniformly 48. 48; The batch size of the model is set to 4, the initial learning rate is 0.0001, and the number of training epochs is 100.

[0072] The evaluation metrics selected in this embodiment are as follows: 1. CSI (Critical Success Index): The proportion of hit samples to the sum of hit, false alarm, and false alarm samples; 2. POD (Probability of Detection): The proportion of hit samples out of the total number of real events; 3. FAR (False Alarm Rate): The proportion of false alarms to the total number of predicted events; 4. BIAS (Prediction Bias): The ratio of the total number of predicted events to the total number of actual events.

[0073] Visual comparison Figure 6The results show that ConvLSTM predictions have a high false alarm rate and blurred boundaries in high-value lightning regions during long-term prediction tasks. While SimVPv2 can model the spatiotemporal evolution of historical lightning sequences through spatiotemporal mapping and the gSTA module, its predictions still suffer from local location shifts and unstable intensity distributions. In contrast, the method of this invention integrates historical lightning information, radar VIL information, and satellite IR information, which better preserves the location, morphology, and intensity distribution of the main lightning activity, resulting in higher consistency between the predicted area and the actual lightning observation area. Performance Comparison Figure 7 The results show that, compared to the ConvLSTM method, the average CSI of the method presented in this invention increased from 0.301 to 0.323, the average POD increased from 0.516 to 0.531, the average FAR decreased from 0.580 to 0.549, and the average BIAS decreased from 1.23 to 1.18. Compared to the SimVPv2 method, the average CSI of the method presented in this invention increased from 0.288 to 0.323, the average POD increased from 0.504 to 0.531, the average FAR decreased from 0.598 to 0.549, and the average BIAS decreased from 1.26 to 1.18. It is evident that the method presented in this invention simultaneously improves the hit rate and overall forecast score in lightning nowcasting, and effectively reduces the false alarm rate and over-report bias, demonstrating the superiority of this invention in multi-source environmental constraints, sparse lightning feature encoding, and environmental consistency fusion.

[0074] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0075] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0076] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A lightning nowcasting method based on physical gating and cross-modal attention, characterized in that, A lightning prediction model is constructed based on a lightning count grid sequence and a radar VIL image sequence over the most recent time period. and satellite IR image sequences Predict lightning count grid sequences over future time periods; The lightning prediction model includes: a physical gated encoder, a sparse lightning encoder, a cross-modal feature fusion module, and a time-series prediction module; The physical gated encoder includes: a first physical gated branch, a second physical gated branch, a first pointwise convolutional unit, and a first channel attention unit; the first physical gated branch processes radar VIL image sequences. Perform high-value gating processing to obtain the gating output. The second physical gated branch controls the satellite IR image sequence. Perform low-value gating processing to obtain the gating output. Gated output and Historical environment features are obtained by dimensionality blending through the first pointwise convolutional unit and then channel attention being extracted by the first channel attention unit. ; A sparse lightning encoder performs sparse encoding on a lightning counting grid sequence to obtain historical lightning features. ; The cross-modal feature fusion module employs a cross-modal attention mechanism to fuse historical environmental features. and historical lightning characteristics , to obtain fused temporal features The lightning count grid for multiple future moments is obtained by integrating the temporal features and processing them through the temporal prediction module.

2. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 1, characterized in that, The cross-modal feature fusion module includes a cross-modal attention unit, a fourth-channel attention unit, and an environment consistency gating unit; Cross-modal attention units with historical lightning features Generate query vector Q based on historical environmental features Generate key vector K and value vector V, and construct environment retrieval feature A through an attention mechanism; Then, the environment fusion features are obtained through processing by the fourth-channel attention unit. ; Environmental consistency gating unit for gating features and gating features The environment consistency map is obtained by multiplying the max pooling results pixel by pixel. Then to , and The concatenated results are sequentially subjected to convolution, group normalization, activation processing, convolution, and reactivation to obtain the environment gating map. Then, using an environmental gating graph As a feature of mask fusion environment fusion and historical lightning characteristics The fused temporal features were obtained.

3. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 1, characterized in that, The first and second physical gated branches have the same framework structure and are collectively referred to as physical gated branches. They extract channel attention from the multidimensional convolutional features of the input data, perform gated processing on the input data, and then multiply the gated features and channel attention features pixel by pixel to obtain the gated output.

4. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 3, characterized in that, The physical gating branch includes: a gating unit, a two-dimensional convolutional branch, a lateral convolutional branch, a longitudinal convolutional branch, a second pointwise convolutional unit, a second channel attention unit, and a feature fusion unit; the gating unit first processes the input data sequence. Normalization is performed, and then the normalized values ​​are gated to obtain the gated features. Two-dimensional convolutional branches, horizontal convolutional branches, and vertical convolutional branches are set in parallel to process the input data. Two-dimensional convolution, horizontal convolution, and vertical convolution are performed. The three convolutions are then concatenated in dimensions by the second pointwise convolutional unit before being processed again to obtain a hybrid convolution. Hybrid Convolution The second channel attention unit is input, and group normalization, activation processing, and channel attention extraction are performed sequentially to obtain attention features. The feature fusion unit controls the gating features. and attention characteristics Pixel-by-pixel multiplication yields the gated output. .

5. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 1, characterized in that, The sparse lightning encoder includes a lightning logarithmic intensity unit, a neighborhood risk unit, a deformable convolution unit, and a third-channel attention unit; The lightning logarithmic intensity cell is used to perform logarithmic processing on the lightning counting grid sequence Y to obtain the logarithmic intensity map. The neighborhood risk unit uses a binary function to extract neighborhood risk zone features based on the lightning occurrence location, resulting in a neighborhood risk map. ; Log-intensity plot and neighboring risk map After splicing and channel blending, historical lightning features are extracted and filtered by deformable convolutional units and third-channel attention units. .

6. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 1, characterized in that, During model training, the model parameters are updated by backpropagating gradients through the total loss function, which includes location intensity regression loss and neighborhood hit loss. Location intensity regression loss is used to constrain the intensity difference between the predicted lightning count and the actual lightning count at the corresponding spatial location and the corresponding prediction time. Neighborhood hit loss is used to constrain the location of lightning strikes.

7. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 6, characterized in that, Neighborhood hit loss L nh for: Where sum represents summing all pixel values ​​across all grid cells; , and These are the weights for the missed detection loss, false alarm loss, and precise location loss, respectively. P represents the continuous predicted response map. , B is the predicted future lightning count grid sequence; B is the binary lightning occurrence map. , For a real future lightning count grid sequence; The threshold for determining the occurrence of lightning. is the slope coefficient of the soft thresholding function; sigmoid is the activation function; bin is a binary function; To predict the neighborhood response map, , For true neighborhood allowable graphs, , where r is the neighborhood radius.

8. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 6, characterized in that, The location intensity regression loss uses the MAE loss function.

9. The lightning nowcasting method based on physical gating and cross-modal attention as described in claim 1, characterized in that, The timing prediction module uses a convolutional long short-term memory network, ConvLSTM.

10. A lightning nowcasting system based on physical gating and cross-modal attention, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to implement the lightning nowcasting method based on physical gating and cross-modal attention as described in any one of claims 1-9.

Images