A multi-modal short-time precipitation prediction method and system based on a DGAFNet network

Abstract

The application provides a multi-modal short-time precipitation prediction method and system based on a DGAFNet network, which comprises the following steps: obtaining a multi-source data set composed of radar echoes and satellite cloud image, and preprocessing the multi-source data set; constructing a network model based on the DGAFNet, embedding a space-time feature extraction and fusion mechanism into the network model in the framework of the DGAFNet to generate an initial short-time rainfall prediction model; training the initial short-time rainfall prediction model by using the preprocessed multi-source data set, and optimizing the network parameters through a combined loss function to obtain a final short-time rainfall prediction model; and inputting a to-be-tested radar echo and satellite cloud image into the final short-time rainfall prediction model to obtain a short-time precipitation prediction result. The application significantly improves the space-time accuracy, reliability and generalization ability of short-time precipitation prediction, and reduces the false positive and false negative rates of short-time precipitation.

Description

Technical Field

[0001] This application relates to the field of weather forecasting technology, specifically to a multimodal short-term precipitation forecasting method and system based on the DGAFNet network. Background Technology

[0002] Short-term precipitation forecasting typically refers to predicting the intensity and distribution of precipitation in a local area within the next 0-2 hours. It is the most time-sensitive and challenging aspect of meteorological forecasting. It plays a crucial role in urban flood control and drainage, traffic control (such as flight takeoffs and landings, and subway operations), agricultural production, and early warning of geological disasters such as mudslides. Accurate and timely short-term forecasts can buy decision-makers valuable response time, thereby greatly reducing loss of life and property. However, the atmospheric system is a highly nonlinear, chaotic, and complex dynamic system. Precipitation processes are often accompanied by multi-scale spatiotemporal variations. How to extract effective information from massive amounts of observational data and achieve high-resolution, high-accuracy real-time forecasts has always been a pressing problem in the meteorological field. Traditional short-term forecasting methods are mainly divided into numerical weather prediction (NWP) based on physical equations and optical flow methods based on radar echo extrapolation. The NWP method, based on atmospheric dynamic equations, has clear physical meaning, but its calculations are extremely complex, sensitive to initial conditions, and often suffer from a "spin-up" effect. This results in poor performance within short-term forecast lead times of 0-2 hours and makes it difficult to meet the real-time requirements for high spatiotemporal resolution. The optical flow method performs linear extrapolation by tracking the moving vector of radar echoes. While computationally fast, it assumes that echo intensity and shape remain constant during motion, failing to simulate the formation, dissipation, deformation, and nonlinear evolution of precipitation clouds. Therefore, its accuracy rapidly decreases as the forecast time increases.

[0003] In recent years, with the rapid development of deep learning technology and the accumulation of meteorological big data (such as Doppler radar and geostationary satellites), data-driven methods have shown great potential in short-term precipitation forecasting. Early mainstream methods included Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). CNNs (such as U-Net) excel at extracting spatial features, but are limited by the local receptive field of the convolutional kernel, making it difficult to capture long-distance spatial dependencies. RNNs (such as ConvLSTM and PredRNNs), while effectively modeling time-series information through gating mechanisms, are prone to gradient vanishing or exploding problems when processing long sequences, and are difficult to parallelize, resulting in low training efficiency. Subsequently, the Transformer architecture based on self-attention mechanisms, due to its powerful global modeling capabilities, was introduced into the meteorological field, such as Rainformer and Earthformer, significantly improving forecast performance. However, the computational complexity of the standard Transformer increases quadratically with sequence length, and it is less efficient than convolutional networks in extracting local high-frequency details (such as strong echo edges).

[0004] Most current deep learning-based precipitation forecasting models employ an autoregressive generation approach, using historical frames to predict the next frame and then recursively using the prediction results as input to predict the future. While this method can learn temporal evolution, it is prone to error accumulation: if a prediction in a certain frame is ambiguous or biased, these errors will be amplified in subsequent time steps, causing long-term predictions to become smooth and losing important high-frequency texture details in the precipitation field. Furthermore, existing research largely focuses on data mining from single modalities (primarily radar echoes), neglecting the complementary information provided by other modalities such as satellite cloud images, including cloud top structure and moisture transport, thus limiting the model's ability to understand complex weather processes.

[0005] A search revealed the following related patent applications in the prior art: Chinese invention patent application CN117574074A, entitled "A Short-Term Forecasting Method for Severe Convective Precipitation Based on Transformer," utilizes a Transformer model to process dual-polarization radar data. It designs a network by analyzing the spatial distribution characteristics of the radar data and introduces a loss function based on frequency domain energy separation to alleviate prediction ambiguity. Although this method attempts to preserve high-frequency information through frequency domain weighting, it still essentially relies on single radar observation data. In areas where radar beams are blocked or long-range detection capabilities are reduced, the lack of single-mode information leads to a decrease in forecast accuracy. Furthermore, this method does not fully consider the differences in receptive field requirements and evolution patterns of precipitation systems at different scales (such as large-scale stratiform clouds and small-scale convective cells). Chinese invention patent application CN117908164A, entitled "A Precipitation Prediction Method Based on a Deep Dual-Branch Structure," proposes a dual-branch network structure containing a physical branch (PhyCell) and a residual branch (SimVP), attempting to combine physical laws with residual features extracted by deep learning. While this method improves the interpretability of predictions to some extent by introducing physical constraints, its fusion of bi-branch features is relatively simple (usually summation), making it difficult to capture the nonlinear interactions between different features at a deeper level. More importantly, this model is also mainly based on radar echo sequences, lacking the utilization of satellite infrared cloud imagery data that reflects the early development characteristics of precipitation, thus limiting its forecasting ability in the early stages of strong convection. Chinese invention patent application CN118298222A, entitled "A Short-Term Precipitation Forecasting Method Based on Multi-Scale Attention and Convolutional Fusion," combines meteorological data and geographic information, employs a Transformer-based encoding and decoding structure, and introduces a convolutional network in the decoding layer to fuse local features. Although this method attempts to combine global and local information, its multimodal fusion typically involves simple channel concatenation (Early Fusion) of data from different sources at the input end. Because radar echoes (reflecting precipitation) and satellite cloud images (reflecting cloud top temperature) differ significantly in physical meaning and numerical distribution, simple input-end splicing makes it difficult for the model to effectively learn the complex nonlinear mapping relationship between the two, and it cannot achieve multi-scale interaction and correction in the deep feature space. Chinese invention patent application CN119128448A, entitled "High-Resolution Rainfall Analysis Generation Method Based on Multi-Source Modal Fusion Deep Learning," constructs the MF-ST-Unet model and fuses multi-source observation data from radar, satellite, and other sources to generate a high-resolution rainfall analysis field. Although this method captures rainfall characteristics through a non-equal-weighted multidimensional loss function, its core still focuses on the fusion of historical precipitation modes and real-time observations. When dealing with the rapid evolution of precipitation systems within a very short time, its modeling ability for dynamic mechanisms is limited, and its generalization performance in removing noise interference under complex weather conditions still needs improvement.Chinese invention patent application CN120823519A, entitled "A Deep Learning-Based SAR Sea Surface Rainfall Detection Method," utilizes a deep learning model (such as YOLOv11) to process synthetic aperture radar (SAR) images and introduces an attention mechanism to automatically extract sea surface rainfall areas. While this method improves detection accuracy, it remains essentially limited to visual detection of single-sensor SAR images. Due to the lack of comprehensive utilization of atmospheric parameters such as multi-channel satellite radiation, its ability to eliminate interference from strong sea surface winds is limited, and it has not yet achieved deep inversion of precipitation intensity and evolution trends.

[0006] Therefore, there is an urgent need for a short-term precipitation forecasting method and system based on modal data to improve the accuracy of short-term precipitation forecasting. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide a multimodal short-term precipitation prediction method and system based on the DGAFNet network.

[0008] According to the first aspect of this application, a multimodal short-term precipitation prediction method based on the DGAFNet network is provided, comprising: A multi-source dataset consisting of radar echoes and satellite cloud images is acquired, and the multi-source dataset is preprocessed. A network model is constructed based on DGAFNet, and a spatiotemporal feature extraction and fusion mechanism is embedded in the DGAFNet framework to generate the initial short-term rainfall prediction model. The initial short-term rainfall prediction model is trained using the preprocessed multi-source dataset, and the network parameters are optimized by back-tuning through a combined loss function to obtain the final short-term rainfall prediction model. The radar echo and satellite cloud image to be measured are input into the final short-term rainfall prediction model to obtain the short-term precipitation prediction result.

[0009] Optionally, a multi-source dataset consisting of radar echoes and satellite cloud images is acquired, and the multi-source dataset is preprocessed, including: Obtain a multi-source dataset consisting of radar echo sequences and satellite cloud image sequences; Align each frame of radar echo in the radar echo sequence with each frame of satellite cloud image in the satellite cloud image sequence in terms of time and space. The aligned radar echoes are divided into several image blocks as radar image blocks, and the aligned satellite cloud images are divided into several image blocks as satellite image blocks. The location of each radar image block and satellite image block is encoded.

[0010] Optionally, the step of constructing a network model based on DGAFNet, and embedding a spatiotemporal feature extraction and fusion mechanism into the network model within the framework of DGAFNet to generate the initial short-term rainfall prediction model, includes: Based on the DGAFNet framework, two parallel branches, radar and satellite, are constructed. The radar branch includes a radar encoding branch and a radar decoding branch, and the satellite branch includes a satellite encoding branch and a satellite decoding branch. Both the radar encoding branch and the satellite encoding branch include at least two stacked LASL modules. The radar encoding branch is used to encode radar image patches carrying location encoding information and extract radar features, while the satellite encoding branch is used to encode satellite image patches carrying location encoding information and extract satellite features. The LASL modules are used to extract temporal-spatial scale features. A void space pyramid pooling module is introduced at the input of the last LASL module in the radar coding branch; A progressive cross-attention fusion module is introduced into the radar decoding branch; The initial short-term rainfall prediction model is generated.

[0011] Optionally, the LASL module is an improvement on the existing SwinLSTM module, including: For the existing SwinLSTM module, the multilayer perceptron is replaced with a convolutional feedforward neural network; the convolutional feedforward neural network includes a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer and a 1×1 convolutional layer connected in sequence.

[0012] Optionally, the void space pyramid pooling module includes five parallel branches, one of which uses a global average pooling layer, and the other four branches use four convolutional layers with different dilation rates; the outputs of the five parallel branches are spliced ​​and fused and then input into the last LASL module of the radar coding branch.

[0013] Optionally, the radar decoding branch includes several radar decoding layers, which are used to decode radar features; the satellite decoding branch includes several satellite decoding layers, which are used to decode satellite features; and the progressive cross-attention fusion module is embedded in the input of the preset radar decoding layer and is used to integrate satellite features into radar features, specifically including: Obtain the nonlinear dependency between radar features and satellite features at the same level; By utilizing macroscopic cloud top evolution information in satellite features, radar features are characterized and enhanced based on the nonlinear dependency relationship to obtain enhanced radar features. Through the residual connection structure in the progressive cross-attention fusion module, the enhanced radar features are residually fused with the original radar features and satellite features to obtain the fused target features and output them to the next level of radar decoding layer.

[0014] Optionally, the step of training the initial short-term rainfall prediction model using the preprocessed multi-source dataset and optimizing the network parameters through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model includes: In the preprocessed multi-source dataset, radar image patches corresponding to the radar echoes of the first n frames and satellite image patches corresponding to the satellite cloud images of the first n frames are selected as input image data for the initial short-term rainfall prediction model. Radar image patches corresponding to the radar echoes of the last m frames and satellite image patches corresponding to the satellite cloud images of the last m frames are selected as ground truth labels for training the initial short-term rainfall prediction model. Set a maximum number of iterations, use the input image data as the input to the initial short-term rainfall prediction model and perform forward propagation to generate the corresponding prediction frame; Construct a combined loss function, and use the combined loss function to obtain the difference between the predicted frame and the ground truth label as the loss value; With minimizing the loss value as the training objective, the parameters are back-tuned within the maximum number of iterations. By monitoring the loss of the source dataset and implementing an early stopping strategy, the optimal state of the model is locked, and the final short-term rainfall prediction model is obtained.

[0015] Optionally, the loss function of the combination The expression is as follows: in, Indicates satellite branch loss; Indicates radar branch loss; A real satellite cloud image representing time t; This represents the predicted satellite cloud image at time t. Represents the actual radar echo at time t; The predicted radar echo at time t represents the total length of the radar echo sequence.

[0016] According to a second aspect of this application, a multimodal short-term precipitation prediction system based on a DGAFNet network is provided, comprising: The acquisition module is used to acquire a multi-source dataset consisting of radar echoes and satellite cloud images, and to preprocess the multi-source dataset. The prediction model building module is used to build a network model based on DGAFNet. Within the framework of DGAFNet, a spatiotemporal feature extraction and fusion mechanism is embedded into the network model to generate the initial short-term rainfall prediction model. The training module is used to train the initial short-term rainfall prediction model using the preprocessed multi-source dataset, and to optimize the network parameters through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model. The output module inputs the radar echo and satellite cloud image to be measured into the final short-term rainfall prediction model to obtain the short-term precipitation prediction result.

[0017] According to a third aspect of this application, an electronic device is provided, comprising: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the multimodal short-term precipitation prediction method based on the DGAFNet network proposed in the first aspect of this application according to the obtained program instructions.

[0018] The multimodal short-term precipitation prediction method based on the DGAFNet network provided in this application addresses the limitations of single radar data in short-term precipitation prediction, such as uneven spatial coverage and inability to comprehensively represent the multi-source driving factors of precipitation evolution. It introduces satellite cloud images and radar echoes as multimodal data outputs. Through complementary fusion of multi-source data, it effectively compensates for the one-sidedness and limitations of single radar data in feature representation, enriching the spatiotemporal and physical characteristics related to precipitation. Simultaneously, it introduces a spatiotemporal feature extraction and fusion mechanism. This mechanism can adaptively mine and learn the complex nonlinear relationship between radar and satellite features based on the feature differences of different input data and the laws of precipitation evolution, thereby significantly improving the spatiotemporal accuracy, reliability, and generalization ability of short-term precipitation prediction, and reducing the false alarm and missed alarm rates of short-term precipitation.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a multimodal short-term precipitation prediction method based on DGAFNet network in one embodiment of this application; Figure 2 This is a schematic diagram of the construction of a DGAFNet-based network model in one embodiment of this application; Figure 3This is a schematic diagram of an improved method for the local sensing SwinLSTM module in one embodiment of this application, wherein (a) is a schematic diagram of the local sensing SwinLSTM module; (b) is a schematic diagram of the Swin convolutional block; and (c) is a schematic diagram of the convolutional feedforward network. Figure 4 This is a schematic diagram of the initial short-term rainfall prediction model in one embodiment of this application; Figure 5 This is a schematic diagram of the GCAFM module in one embodiment of this application; Figure 6 This is a comparison chart of visualization results on various network models in one embodiment of this application; Figure 7 This is a schematic diagram of a multimodal short-term precipitation prediction system in one embodiment of this application. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0023] In recent years, with the rapid development of deep learning technology and the accumulation of meteorological big data, data-driven methods have shown great potential in short-term precipitation forecasting. Currently, most deep learning-based precipitation forecasting models employ an autoregressive generation method. While this method can learn temporal evolution, it is prone to error accumulation: once a prediction in a certain frame is ambiguous or biased, these errors are amplified in subsequent time steps, leading to a smoothing of long-term forecasts and the loss of important high-frequency texture details in the precipitation field. Furthermore, existing research largely focuses on data mining of single modes (mainly radar echoes), neglecting the complementary information provided by other modal data such as satellite cloud images, including cloud top structure and water vapor transport, thus limiting the model's ability to understand complex weather processes. Based on these problems, this application provides a multimodal short-term precipitation forecasting method based on the DGAFNet network to address these issues.

[0024] Reference Figure 1As shown in the figure, this application provides a multimodal short-term precipitation prediction method based on the DGAFNet network, including: S1. Obtain a multi-source dataset consisting of radar echoes and satellite cloud images, and preprocess the multi-source dataset; S2. Construct a network model based on DGAFNet, and embed a spatiotemporal feature extraction and fusion mechanism into the network model within the framework of DGAFNet to generate the initial short-term rainfall prediction model. S3. Using the preprocessed multi-source dataset, the initial short-term rainfall prediction model is trained, and the network parameters are optimized by back-tuning through the combined loss function to obtain the final short-term rainfall prediction model. S4. Input the radar echo and satellite cloud image to be measured into the final short-term precipitation prediction model to obtain the short-term precipitation prediction results.

[0025] The embodiments described above address the limitations of single radar data in short-term precipitation forecasting, which suffers from uneven spatial coverage and an inability to comprehensively characterize the multi-source driving factors of precipitation evolution. To address these limitations, a dual-branch progressive cross-attention fusion network (DGAFNet, Dynamic Graph Attention Fusion Network) is proposed. This network incorporates satellite cloud images and radar echoes as multi-modal data outputs. Through complementary fusion of multi-source data, it effectively compensates for the one-sidedness and limitations of single radar data in feature representation, enriching the spatiotemporal and physical characteristics related to precipitation. Simultaneously, a spatiotemporal feature extraction and fusion mechanism is introduced. This mechanism adaptively mines and learns the complex nonlinear relationship between radar and satellite features based on the feature differences of different input data and the evolution patterns of precipitation. This significantly improves the spatiotemporal accuracy, reliability, and generalization ability of short-term precipitation forecasting, while reducing the false alarm and missed alarm rates.

[0026] Among them, DGAFNet is a dual-branch deep learning model that integrates Long Short-Term Memory (LSTM) network and attention mechanism. It is often used for multimodal data processing, feature fusion and target detection / prediction tasks in complex scenarios, and has potential application value in fields such as meteorological radar data processing and severe convective weather analysis.

[0027] In some specific embodiments of this application, acquiring a multi-source dataset consisting of radar echoes and satellite cloud images, and preprocessing the multi-source dataset, may further include: S11. Obtain a multi-source dataset consisting of radar echo sequences and satellite cloud image sequences; S12. Align each frame of radar echo in the radar echo sequence and each frame of satellite cloud image in the satellite cloud image sequence in terms of time and space. S13. Divide the aligned radar echo into several image blocks as radar image blocks, and divide the aligned satellite cloud image into several image blocks as satellite image blocks. Perform position coding on each radar image block and satellite image block.

[0028] For example, the above steps acquire a multi-source dataset (i.e., a multimodal meteorological dataset) composed of radar echo images and satellite cloud images. When acquiring the multi-source dataset, continuous sampling of the radar echo images and satellite cloud images at fixed frame rates is performed to form a multi-source dataset from the continuously sampled multimodal images, thus creating training samples. During the formation of training samples, the images in the dataset are filtered and preprocessed. Since the original data may contain missing values ​​or non-precipitation events, filtering is necessary to ensure data quality. For example, this embodiment uses the publicly available dataset SEVIR (Storm Event Imagery), collected by MIT, which contains meteorological events from 2017 to 2019. The sampling interval is 5 minutes (which can also be adjusted to 10 minutes). This embodiment uses continuous sampling, i.e., radar echo sequence data and satellite cloud image sequence data from the past hour, and radar echo sequence data from the next hour. The radar echo sequence and satellite cloud image sequence of the past hour (6 frames, 10-minute intervals) are used as input, and the radar echo image of the next hour (6 frames) is used as the ground truth to compare with the predicted radar echo image to obtain the prediction performance.

[0029] The multi-source dataset refers to a collection containing radar echo and satellite infrared (IR069, IR107) images. Radar echo sequence data refers to a set of radar vertical integrated liquid (VIL) image data arranged in a time series. Satellite cloud image sequence data refers to a set of satellite infrared brightness temperature (IRBT) image data arranged in a time series. In this embodiment, it specifically includes observation images from the infrared water vapor channel (e.g., IR069) and the infrared window channel (e.g., IR107). The VIL image pixel range is 0-254; this embodiment selects VIL data from 2017-2019 and IR069 and IR107 satellite data from 2018-2019. Each training iteration uses sequential sampling, dividing the data into training and test sets (e.g., a 6:1 ratio). The first 6 frames of multimodal data are used as input to the DGAFNet model, and the last 6 frames of real radar echoes are used for supervised training.

[0030] In some possible embodiments, after acquiring the multi-source dataset, the method further includes: preprocessing the radar echo and satellite cloud images. Preprocessing includes: uniformly interpolating and resizing the images in the dataset (e.g., to 96x96) and performing normalization. By preprocessing the multi-source dataset, the high-dimensional continuous images are split into non-overlapping image blocks and mapped to low-dimensional vectors, reducing computational complexity and preserving the local spatial location information of the image blocks. The split coded block images are then fed in parallel into the radar branch encoder and the satellite branch encoder to obtain the spatiotemporal evolution features in different modal sequences.

[0031] Reference Figure 2 As shown, the input multi-source dataset (radar and satellite) is first sorted by time to form time-aligned sequence data, allowing the network model to better capture the dynamic evolution of the meteorological system over time and better understand the temporal context. Then, each frame of the sequence data is normalized and resampled, and image patching is performed. The pixel matrix is ​​converted into an embedding vector through a linear projection layer to form coded block features. The coded block features are then fed in parallel into the radar branch and the satellite branch for multi-scale feature extraction. The radar branch uses ASPP to capture precipitation-scale features, while the satellite branch extracts cloud image background features. To fully utilize the guiding role of satellite data on radar echoes, a progressive cross-attention fusion module (GCAFM) is used for deep interaction and fusion. Finally, the prediction result is obtained through a reconstruction layer.

[0032] For example, before feeding the images into the network model, the images in the input multi-source dataset are first sized and divided into blocks to adapt to the Transformer architecture. For example, the original high-resolution radar echo images and satellite cloud images are uniformly interpolated to a size of 96x96. Then, the image block size is set to 4x4 (or according to the specific setting), and each frame image is divided into 24x25 image blocks (a total of 576 patches). Then, these image blocks are projected to a specified dimension (e.g., C=96) through a linear embedding layer and fed into the radar encoder and satellite encoder in parallel. The strong convection features inside the radar echo and the macroscopic cloud system features of the satellite cloud image are learned through a dual-branch encoding method.

[0033] Preprocessing divides the radar echoes in the radar echo sequence and the satellite cloud images in the satellite cloud image sequence into non-overlapping image patches. Each image patch is mapped into a low-dimensional embedding vector through a linear projection layer. The radar echoes and satellite cloud images (satellite infrared cloud image data) are data of different modes. Normalization processing is performed on the data of different modes. The radar echo data adopts the maximum-minimum normalization, and the satellite infrared data (IR069, IR107) adopts the Z-score normalization.

[0034] The embodiments described above in this application, by introducing satellite cloud image data as a supplementary mode, effectively compensate for the limitations of single radar data by utilizing the cloud top structure and early convection information provided by the satellite cloud image data, and significantly improve the prediction accuracy of short-term precipitation.

[0035] In some specific embodiments of this application, a network model is constructed based on DGAFNet. Within the framework of DGAFNet, a spatiotemporal feature extraction and fusion mechanism is embedded into this network model to generate the initial short-term rainfall prediction model. This may further include: S21. Based on the DGAFNet framework, construct two parallel radar branches and a satellite branch. The radar branch includes a radar coding branch and a radar decoding branch, and the satellite branch includes a satellite coding branch and a satellite decoding branch. Both the radar coding branch and the satellite coding branch include at least two stacked LASL modules. The radar coding branch is used to encode radar image patches carrying location coding information and extract radar features, while the satellite coding branch is used to encode satellite image patches carrying location coding information and extract satellite features. The LASL module is used to extract temporal-spatial scale features. S22. Introduce a void space pyramid pooling module at the input of the last LASL module in the radar coding branch; S23. Introduce a progressive cross-attention fusion module in the radar decoding branch; S24. Generate the initial short-term rainfall prediction model.

[0036] For example, the above steps build a network model based on DGAFNet, design a dual-branch encoder-decoder structure, and embed a Locally Aware SwinLSTM (LASL) module, a Hollow Spatial Pyramid Pooling (ASPP) module, and a Progressive Cross-Attention Fusion module into the DGAFNet framework to finally generate the initial short-term rainfall prediction model, specifically including: A dual-branch encoder is constructed to extract multi-scale spatiotemporal features (i.e., time-space scale features) from radar echoes and satellite images, respectively. The satellite coding branch in the dual-branch encoder is used to obtain global-local spatial features of satellite cloud images to assist radar prediction.

[0037] A void space pyramid pooling module is introduced into the radar branch to obtain spatial features of different scales in radar echoes, thus solving the problem of uneven scale distribution of precipitation systems. The local perception SwinLSTM module serves as the core spatiotemporal processing unit, utilizing a window attention mechanism to capture long-range dependencies. A dual-branch decoder is constructed, and a progressive cross-attention fusion module is introduced during the decoding process to dynamically integrate satellite features into radar features; Generate the initial short-term rainfall prediction model.

[0038] The above embodiments of this application refer to... Figure 3 and Figure 4 As shown, the core component is an improved LASL unit based on the Swin Transformer. The first part is the radar branch, which, in addition to LASL, also embeds a dilated spatial pyramid pooling module to obtain multi-scale contextual information through dilated convolutions with different expansion rates. The second part is the satellite branch, which also uses LASL to extract spatiotemporal evolution features of cloud images. Finally, in the decoding branch, satellite features are injected into the radar branch through a GCAFM module to generate the initial short-term precipitation prediction model. By adaptively focusing on key areas (such as strong convection cores), effective integration of cross-modal information is achieved. This application, by constructing a DGAFNet network model and embedding a dual-branch structure and GCAFM mechanism, can generate a short-term precipitation prediction model. By constructing an ASPP module to obtain spatial features of radar echoes at different scales, it solves the problem of uneven scale in precipitation systems. Simultaneously, a satellite branch is introduced in parallel to utilize cloud top temperature information, and then a cross-attention mechanism is used to effectively fuse multi-modal features, improving the accuracy and reliability of short-term precipitation prediction. Figure 3 In the middle, C t H represents the cell state at time t; t X represents the hidden state at time t; t z represents the input at time t; l express l The feature tensor of the hierarchy.

[0039] In some specific embodiments of this application, the LASL module is an improvement on the existing SwinLSTM module, and may further include: For the existing SwinLSTM module, the multilayer perceptron is replaced with a convolutional feedforward neural network; the convolutional feedforward neural network includes a 1×1 convolutional layer, a 3×3 depthwise separable convolutional layer and a 1×1 convolutional layer connected in sequence.

[0040] For example, this application first constructs a location-aware SwinTransformer module, namely the Swin-Conv module. The Swin-Conv module (i.e., the Swin convolutional block) is used to obtain spatial scale features of radar echoes and satellite cloud images at different scales, i.e., spatial multi-scale features. The Long Short-Term Memory (LSTM) network is used to obtain temporal scale features of radar echoes and satellite cloud images at different scales, i.e., temporal multi-scale features. The long and short-term time evolution patterns are learned through the temporal multi-scale features. The Swin-Conv module is introduced into the LSTM network to obtain the SwinLSTM module. The Swin-Conv module can replace the convolutional layers used for calculating gating and state updates in the traditional LSTM network. By adopting the self-attention mechanism of the Transformer framework (i.e., the SwinTransformer module) and the LSTM network, global and local features of radar and satellite patterns can be extracted respectively. The LASL module improves upon the traditional SwinLSTM module by utilizing a window attention mechanism to model global long-range dependencies and replacing the multilayer perceptron (MLP) with a convolutional feedforward neural network (convFNN) to introduce local spatial inductive bias. This module effectively combines macroscopic evolution with microscopic details (such as strong echo edges), enhancing its ability to model complex precipitation systems. The convFNN consists of sequentially connected 1×1 convolutions (dimensionality upscaling), 3×3 depthwise separable convolutions (spatial interaction), and 1×1 convolutions (dimensionality reduction), thereby explicitly capturing local spatial details and texture information during feature transformation and enhancing the ability to model strong echo edges. For the SwinLSTM module, its moving window-based self-attention mechanism is retained to model global spatial dependencies.

[0041] Using the local perception SwinLSTM as the basic unit, the local perception SwinLSTM improves upon the standard SwinLSTM by replacing the original multilayer perceptron with a convolutional feedforward neural network (i.e., convolutional feedforward network). This process can be expressed as follows: while using the moving window attention mechanism of the Swin Transformer to capture long-range dependencies, the convFNN supplements the local inductive bias by introducing convolutional operations.

[0042] The structure of convFNN includes: Where Conv is a 1×1 convolution, Bnorm is batch normalization, and GELU is a non-linear activation function; DWConv is a 3×3 depthwise separable convolution. This is an estimate of the convFNN value; This is the output value of convFNN. This approach can simultaneously utilize the Attention mechanism to focus on global evolution (such as typhoon paths) and the convolution mechanism to focus on local details (such as sharp precipitation boundaries), thereby improving the model's representational ability.

[0043] The radar coding branch and the satellite coding branch are two parallel symmetrical branches: Satellite coding branch: Models global and local spatial dependencies through a three-layer stacked LASL module, and uses macroscopic cloud top structure and vertical evolution information provided by satellite observations to help improve the prediction accuracy of radar echoes; Radar coding branch: Integrates a void space pyramid pooling module through a three-layer stacked LASL module to capture multi-scale spatiotemporal features in radar echoes and solve the problem of uneven scale distribution of precipitation systems.

[0044] During the encoding process, radar and satellite image patches are encoded using three stacked encoding layers, each containing an LASL module. The LASL module, as the core spatiotemporal processing unit, utilizes a window attention mechanism to capture long-range dependencies and introduces local inductive bias using a convolutional feedforward neural network. Patch Merging modules for downsampling and dimensionality fusion are also connected to the outputs of the LASL modules in both the radar and satellite encoding branches.

[0045] The embodiments described above in this application design a local perception SwinLSTM, which innovatively uses convFNN instead of MLP. This enables the model to capture long-distance dependencies (such as typhoon paths) using a self-attention mechanism, while also enhancing the preservation of local textures and edge details (such as the center of a rainstorm) through convolution, thus alleviating the problem of blurry predicted images.

[0046] In some specific embodiments of this application, the void space pyramid pooling module includes five parallel branches, one of which uses a global average pooling layer, and the other four branches use four convolutional layers with different dilation rates; the outputs of the five parallel branches are spliced ​​and fused and then input into the last LASL module of the radar coding branch.

[0047] For example, a hollow spatial pyramid pooling module is introduced before the third coding layer of the radar coding branch to obtain multi-scale context enhancement features of the radar. By learning the pattern of different receptive fields required for different scale targets (such as compact convection kernels and broad rainbands) in the precipitation system through hollow convolution with different dilation rates, multi-scale spatial details and global context information are encoded. Specifically, four sets of convolutional layers with different dilation rates (1, 6, 12, 18) and a global average pooling layer are introduced. Through parallel sampling and fusion, rich features that can adapt to the uneven scale distribution of radar echo height are extracted. The four sets of convolutional layers with different dilation rates include: a 1×1 convolutional branch that maintains the original receptive field; three 3×3 hollow convolutional branches with dilation rates of 6, 12, and 18, which significantly expand the receptive field without increasing the number of parameters and capture large-scale precipitation band features; and a global average pooling branch for obtaining image-level global context information. Finally, the outputs of these five branches are concatenated and fused. This multi-scale approach helps the network capture both small, intensely convective cells and broad, layered cloud features simultaneously.

[0048] In the embodiments described above, an ASPP module is introduced into the radar branch. By utilizing convolution with holes of different expansion rates, it can adaptively capture precipitation features at multiple scales, effectively solving the problem that existing models cannot simultaneously take into account both large-scale rainbands and small, strong convective cores.

[0049] In some specific embodiments of this application, the radar decoding branch includes several radar decoding layers, which are used to decode radar features; the satellite decoding branch includes several satellite decoding layers, which are used to decode satellite features; and the progressive cross-attention fusion module is embedded in the input of the preset radar decoding layer to integrate satellite features into radar features, and may further include: S241. Obtain the nonlinear dependency relationship between radar features and satellite features at the same level; S242. Using macroscopic cloud top evolution information in satellite features, radar features are characterized and enhanced based on nonlinear dependencies to obtain enhanced radar features. S243. Through the residual connection structure in the progressive cross-attention fusion module, the enhanced radar features are residually fused with the original radar features and satellite features to finally obtain the fused target features and output them to the next level of radar decoding layer.

[0050] For example, the decoding branch adopts a dual-branch structure symmetrical to the encoding branch, and gradually restores the spatial resolution by using Patch Expanding. Both the radar decoding layer and the satellite decoding layer in the decoding branch are equipped with LASL modules, and a progressive cross-attention fusion module is introduced at specific levels (such as the input of the second and third layers) to integrate satellite information to assist radar prediction. GCAFM calculates the nonlinear dependence between radar and satellite features through the cross-attention mechanism, enhances the radar feature representation by using the macroscopic cloud top evolution information provided by the satellite, and ensures the effective transfer and fusion of features through residual connections.

[0051] Reference Figure 5 As shown, the progressive cross-attention fusion module is applied as follows: The radar feature R and satellite feature S of the same level are concatenated and normalized in the channel dimension to obtain the intermediate feature M.

[0052] The query vector Q, key vector K, and value vector V are generated using convolutional layers in the progressive cross-attention fusion module. Q is derived from intermediate features, and K and V are also transformed from intermediate features (or, depending on the specific variant, Q is from radar, and K / V are from satellite). The similarity between the query vector and the key vector is calculated to obtain attention weights, which are then applied to the value vector, thereby capturing the nonlinear influence of satellite cloud imagery information on radar echo evolution. The formula can be expressed as: Where Softmax represents the output layer activation function; T' represents the transpose; d represents the dimension of the query vector Q; A represents the attention output; Linear(A) represents a linear transformation of the attention output A; R represents positional encoding; S represents the residual input; SmoothConv represents smooth convolution; and F represents smooth convolution output. In this way, radar features and satellite features interact effectively. A cross-attention mechanism adaptively adjusts the contribution weight of satellite cloud images to radar echo prediction, thereby achieving more accurate predictions. Further smoothing convolution further improves the fused features.

[0053] The fused features are processed by linear layers and smooth convolutions, and then residually connected to the original input before being fed into the next radar decoding layer.

[0054] The embodiments described above employ a progressive cross-attention fusion module. Compared to simple addition or splicing fusion, this module can adaptively learn the complex nonlinear relationship between radar and satellite features, achieving effective interaction of deep semantic features.

[0055] In some specific embodiments of this application, the initial short-term rainfall prediction model is trained using a preprocessed multi-source dataset, and the network parameters are optimized through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model, which may further include: S31. In the preprocessed multi-source dataset, select the radar image blocks corresponding to the radar echoes of the first n frames and the satellite image blocks corresponding to the satellite cloud images of the first n frames as the input image data of the initial short-term rainfall prediction model, and select the radar image blocks corresponding to the radar echoes of the last m frames and the satellite image blocks corresponding to the satellite cloud images of the last m frames as the ground truth labels for training the initial short-term rainfall prediction model. S32. Set the maximum number of iterations, use the input image data as the initial input to the short-term rainfall prediction model and perform forward propagation to generate the corresponding prediction frame; S33. Construct a combined loss function and use the combined loss function to obtain the difference between the predicted frame and the ground truth label as the loss value. S34. Using the minimization of the loss value as the training objective, perform reverse parameter tuning within the maximum number of iterations. By monitoring the loss of the multi-source dataset and implementing an early stopping strategy, lock the optimal state of the initial short-term rainfall prediction model and obtain the final short-term rainfall prediction model.

[0056] For example, the above steps utilize a portion of the data from the multi-source dataset as the training set to train the initial short-term rainfall prediction model, and then perform back-tuning to optimize the network parameters using a combined loss function (i.e., the total loss function) to obtain the final short-term rainfall prediction model, specifically including: The first n frames of images from the multi-source dataset are used as input, and the last m frames are used to train the initial short-term rainfall prediction model. The training data includes input image data and ground truth labels. The first n frames are from radar echo sequences and satellite cloud image sequences from historical moments, and the last m frames are from radar echo sequences and satellite cloud image sequences from future moments. Determine the initial number of training rounds, enhance the model's generalization ability by rotating the training data, set the initial learning rate, and adjust it using a cosine annealing adjustment strategy; The input image data in the training data is fed into the DGAFNet network for forward propagation to obtain prediction frames, which include predicted radar echo images and predicted satellite cloud images. The difference between the predicted frame and the real image is calculated by combining the loss function (i.e., the total loss function that includes radar prediction loss and satellite prediction loss) to obtain the loss value. The combined loss function is used to constrain both the radar branch and the satellite branch simultaneously. The model is trained to approach optimal performance by inversely tuning the loss value and minimizing the difference between the real and predicted frames, thus obtaining the final short-term rainfall prediction model.

[0057] In the embodiments described above, the first n=6 frames of images from the multi-source dataset are used as input, and the subsequent m=6 frames of radar images are used to train the initial short-term rainfall prediction model (simultaneously, the satellite branch can also perform self-supervised prediction). An initial learning rate is set (e.g., The model is then optimized using the Adam optimizer. Training data is fed into the DGAFNet network for forward propagation to obtain predicted frames. The difference between the predicted and ground truth images is calculated by combining loss functions to obtain the loss value. Backward tuning is performed based on the loss value, minimizing the difference between the ground truth and predicted frames to train the model towards optimal performance.

[0058] In some specific embodiments of this application, the total loss function The expression is as follows: in, Indicates satellite branch loss; Indicates radar branch loss; A real satellite cloud image representing time t; This represents the predicted satellite cloud image at time t. Represents the actual radar echo at time t; The predicted radar echo at time t represents the total length of the radar echo sequence.

[0059] For example, the total length of the radar echo sequence is the same as the length of the satellite cloud image sequence. This loss function combines the advantages of L1 loss (MAE) and L2 loss (MSE). L1 loss helps maintain image sharpness and edge details, while L2 loss is more sensitive to large errors and helps fit the overall distribution. In this embodiment, the batch size is set to 4, training is performed for 100 epochs, and an early stopping strategy is implemented to prevent overfitting.

[0060] This application, through the embedding of spatiotemporal feature extraction and fusion mechanisms, enables the DGAFNet model to more comprehensively capture spatiotemporal variation information in multimodal data, thereby improving the accuracy of short-term rainfall prediction. Utilizing ASPP multi-scale feature extraction and GCAFM cross-attention fusion mechanisms, the model can better handle data changes at different time and spatial scales, especially the auxiliary role of satellite cloud images on radar echoes, enhancing its generalization ability for high-intensity precipitation weather. Using a combined loss function, the DGAFNet model training process is more efficient, balancing pixel-level errors and overall distribution differences, ultimately improving the accuracy of short-term rainfall prediction. This application, through the DGAFNet model, can more comprehensively capture spatiotemporal variation information in multi-source radar and satellite data. By utilizing multi-scale feature extraction and progressive cross-attention fusion mechanisms to handle data changes at different time and spatial scales, and combining a combined loss function, the model training process is more efficient, converging to the optimal solution faster and improving the accuracy of short-term rainfall prediction.

[0061] The present application will be further described below with reference to specific embodiments in order to better understand the above technical solutions of the present application. It should be understood that the following are only some examples and are not intended to limit the present application.

[0062] Example 1: As a specific implementation, SEVIR data from 2017 to 2019 was first acquired. Then, a model was built based on DGAFNet, using a dual-branch encoder-decoder architecture. The radar branch used ASPP to extract multi-scale features, while the satellite branch extracted cloud image features. Both branches used LASL units to balance global and local information. In the decoding stage, satellite information was fused using the GCAFM module. Next, the model was trained and optimized using an L1+L2 combined loss function to improve the short-term rainfall prediction model, generating the final short-term rainfall prediction model and improving the accuracy of the prediction results.

[0063] The following experiments utilize radar echo images and satellite images from the publicly available SEVIR dataset to evaluate and verify the effectiveness of the short-term rainfall prediction method based on DGAFNet proposed in this application.

[0064] The environment for this experiment is Ubuntu 20.04.5 operating system, the deep learning framework is PyTorch 2.0.0 platform, and the server configuration is NVIDIA RTX 3090 GPU for training.

[0065] This application compares itself with the current state-of-the-art (SOTA) models, such as ConvLSTM, PredRNN, PredRNN++, MIMO, and MotionRNN, using structural similarity coefficient (SSIM), mean squared error (MSE), mean absolute error (MAE), critical success index (CSI), hit rate (POD), and Heidrick skill score (HSS) as evaluation metrics.

[0066] Experimental results show that the proposed DGAFNet outperforms the comparative models in all metrics, especially in the CSI score for high-intensity precipitation r (e.g., VIL > 32 mm, corresponding to severe convective weather, unit: mm / h), demonstrating the effectiveness of introducing satellite branches, ASPP modules, and the convFNN structure. As shown in the table below, the model in this application achieves optimal performance in all four evaluation metrics.

[0067] Table 1. Comparison of CSI results between this application and existing algorithms. Table 2 shows the POD comparison results between this application and existing algorithms. Table 3 Comparison results of this application and existing algorithms MSE & SSIM In summary, this application was compared with RNN-based and Transformer-based models on four experimental metrics. The experimental results are shown in Tables 1-3. It can be seen that this application achieved optimal performance at all thresholds, especially at a high-intensity precipitation of 10 mm / h. Compared with the suboptimal method (i.e., MIMO), this application improved CSI and POD by 19.3% and 38.3%, respectively. Furthermore, this application significantly improved the MSE metric compared to existing models. Further, as... Figure 6 As shown, the predictive performance presented in this application is significantly better than other comparative models. Figure 6 In this paper, T represents the evolution time. During the one-hour evolution process, the precipitation echo map generated by this application maintained the highest consistency with the actual predicted value (GT), accurately capturing the morphological outline and movement trend of the precipitation system. Compared with the severe "smoothing effect" and echo intensity decay that occur with the increase of prediction time in models such as ConvLSTM, PredRNN and MIMO, this application effectively preserves the detailed features of the core area of ​​heavy precipitation (red and orange areas in the figure), successfully alleviating the boundary ambiguity problem commonly encountered by deep learning in short-term forecasts, and demonstrating excellent spatiotemporal feature preservation capabilities.

[0068] This application improves the accuracy of predicting rainfall in the next hour by using a two-branch multi-source multi-scale coding system to fully learn the evolution information of the precipitation system in time and space, and can reliably predict complex radar echo distribution and movement trends.

[0069] Based on the same inventive concept, another embodiment of this application provides a multimodal short-term precipitation prediction system 100 based on a DGAFNet network, referring to... Figure 7 As shown, it includes: The acquisition module 110 is used to acquire a multi-source dataset consisting of radar echoes and satellite cloud images, and to preprocess the multi-source dataset. The prediction model building module 120 is used to build a network model based on DGAFNet. In the framework of DGAFNet, a spatiotemporal feature extraction and fusion mechanism is embedded into the network model to generate the initial short-term rainfall prediction model. Training module 130 is used to train the initial short-term rainfall prediction model using the preprocessed multi-source dataset, and to optimize the network parameters through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model. The output module 140 inputs the radar echo and satellite cloud image to be tested into the final short-term precipitation prediction model to obtain the short-term precipitation prediction result.

[0070] This application utilizes the DGAFNet network model and combines it with a spatiotemporal feature extraction and fusion mechanism through a data acquisition module, a prediction model construction module, a training module, and an output module. This effectively improves the accuracy of short-term precipitation prediction. By training the prediction model with multi-source datasets and performing reverse optimization using a combined loss function, accurate prediction of short-term rainfall is achieved.

[0071] Specifically, the prediction model building module includes a radar coding branch, a satellite coding branch, and a progressive cross-attention fusion module. The prediction model in this module is trained using a multi-source dataset. Based on DGAFNet, it performs collaborative extrapolation of radar and satellite multimodal data, effectively improving the accuracy of short-term rainfall prediction, especially its ability to capture high-intensity precipitation. The radar branch incorporates the ASPP module, which extracts and fuses multi-scale features through multi-dilation rate convolution, effectively addressing the problem of uneven scale in precipitation systems and enhancing the modeling ability for echoes of different sizes. Both branches of the encoder use improved LASL units and replace MLP with convFNN, significantly enhancing the preservation of local texture and edge details while capturing long-range dependencies. The decoding stage introduces the GCAFM module, which dynamically fuses macroscopic information from satellite cloud images through a progressive cross-attention mechanism, assisting the radar branch in more accurately predicting echo evolution. The introduction of a combined L1 and L2 loss function balances constraints on pixel-level errors and overall distribution, effectively improving the structural similarity of the predicted image and reducing the mean square error, thus mitigating the ambiguity problem in long-term predictions.

[0072] The embodiments described above in this application provide a network architecture based on DGAFNet. By deeply fusing radar echo and satellite cloud image data, it not only significantly improves the accuracy of short-term precipitation forecasting, especially under extreme weather conditions, but also effectively solves the problems of feature extraction and spatiotemporal evolution modeling of multi-scale precipitation systems.

[0073] It should be noted that the modules in the multimodal short-term precipitation prediction system based on the DGAFNet network provided in the above embodiments of this application correspond to the steps of the multimodal short-term precipitation prediction system method based on the DGAFNet network in any of the above embodiments. Those skilled in the art can refer to the step features of the multimodal short-term precipitation prediction system method based on the DGAFNet network to implement the corresponding modules in the multimodal short-term precipitation prediction system based on the DGAFNet network, which will not be repeated here.

[0074] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described multimodal short-term precipitation prediction method based on the DGAFNet network according to the obtained program instructions.

[0075] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0076] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0077] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0078] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0079] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0084] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A multimodal short-term precipitation prediction method based on DGAFNet network, characterized in that, include: A multi-source dataset consisting of radar echoes and satellite cloud images is acquired, and the multi-source dataset is preprocessed. A network model is constructed based on DGAFNet, and a spatiotemporal feature extraction and fusion mechanism is embedded in the DGAFNet framework to generate the initial short-term rainfall prediction model. The initial short-term rainfall prediction model is trained using the preprocessed multi-source dataset, and the network parameters are optimized by back-tuning through a combined loss function to obtain the final short-term rainfall prediction model. The radar echo and satellite cloud image to be measured are input into the final short-term precipitation prediction model to obtain the short-term precipitation prediction result.

2. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 1, characterized in that, Acquire a multi-source dataset consisting of radar echoes and satellite cloud images, and preprocess the multi-source dataset, including: Acquire a multi-source dataset consisting of radar echo sequences and satellite cloud image sequences; Align each frame of radar echo in the radar echo sequence with each frame of satellite cloud image in the satellite cloud image sequence in terms of time and space. The aligned radar echoes are divided into several image blocks as radar image blocks, and the aligned satellite cloud images are divided into several image blocks as satellite image blocks. The location of each radar image block and satellite image block is encoded.

3. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 2, characterized in that, The network model built based on DGAFNet, and the spatiotemporal feature extraction and fusion mechanism embedded in the DGAFNet framework to generate the initial short-term rainfall prediction model, includes: Based on the DGAFNet framework, two parallel branches, radar and satellite, are constructed. The radar branch includes a radar encoding branch and a radar decoding branch, and the satellite branch includes a satellite encoding branch and a satellite decoding branch. Both the radar encoding branch and the satellite encoding branch include at least two stacked LASL modules. The radar encoding branch is used to encode radar image patches carrying location encoding information and extract radar features, while the satellite encoding branch is used to encode satellite image patches carrying location encoding information and extract satellite features. The LASL modules are used to extract temporal-spatial scale features. A void space pyramid pooling module is introduced at the input of the last LASL module in the radar coding branch; A progressive cross-attention fusion module is introduced into the radar decoding branch; The initial short-term rainfall prediction model is generated.

4. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 3, characterized in that, The LASL module is an improvement on the existing SwinLSTM module, including: For the existing SwinLSTM module, the multilayer perceptron is replaced with a convolutional feedforward neural network; the convolutional feedforward neural network includes a 1×1 convolutional layer, a 3×3 depth-separable convolutional layer and a 1×1 convolutional layer connected in sequence.

5. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 3, characterized in that, The void space pyramid pooling module includes five parallel branches, one of which uses a global average pooling layer, and the other four branches use four convolutional layers with different dilation rates. The outputs of the five parallel branches are spliced ​​and fused and then input into the last LASL module of the radar coding branch.

6. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 3, characterized in that, The radar decoding branch includes several radar decoding layers, which are used to decode radar features. The satellite decoding branch includes several satellite decoding layers, which are used to decode satellite features. The progressive cross-attention fusion module is embedded in the input of the preset radar decoding layer and is used to integrate satellite features into radar features, specifically including: Obtain the nonlinear dependency relationship between radar features and satellite features at the same level; By utilizing macroscopic cloud top evolution information in satellite features, radar features are characterized and enhanced based on the nonlinear dependency relationship to obtain enhanced radar features. Through the residual connection structure in the progressive cross-attention fusion module, the enhanced radar features are residually fused with the original radar features and satellite features to obtain the fused target features and output them to the next level of radar decoding layer.

7. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 2, characterized in that, The process involves training the initial short-term rainfall prediction model using the preprocessed multi-source dataset, and then optimizing the network parameters through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model, including: In the preprocessed multi-source dataset, radar image patches corresponding to the first n frames of radar echoes and satellite image patches corresponding to the first n frames of satellite cloud images are selected as input image data for the initial short-term rainfall prediction model. Radar image patches corresponding to the last m frames of radar echoes and satellite image patches corresponding to the last m frames of satellite cloud images are selected as ground truth labels for training the initial short-term rainfall prediction model. Set a maximum number of iterations, use the input image data as the input to the initial short-term rainfall prediction model and perform forward propagation to generate the corresponding prediction frame; Construct a combined loss function, and use the combined loss function to obtain the difference between the predicted frame and the ground truth label as the loss value; With minimizing the loss value as the training objective, the parameters are back-tuned within the maximum number of iterations. By monitoring the loss of the source dataset and implementing an early stopping strategy, the optimal state of the model is locked, and the final short-term rainfall prediction model is obtained.

8. The multimodal short-term precipitation prediction method based on DGAFNet network according to claim 7, characterized in that, The loss function of the combination The expression is as follows: ； ； ； in, Indicates satellite branch loss; Indicates radar branch loss; A real satellite cloud image representing time t; This represents the predicted satellite cloud image at time t. Represents the actual radar echo at time t; The predicted radar echo at time t represents the total length of the radar echo sequence.

9. A multimodal short-term precipitation prediction system based on DGAFNet network, characterized in that, include: The acquisition module is used to acquire a multi-source dataset consisting of radar echoes and satellite cloud images, and to preprocess the multi-source dataset. The prediction model building module is used to build a network model based on DGAFNet. Within the framework of DGAFNet, a spatiotemporal feature extraction and fusion mechanism is embedded into the network model to generate the initial short-term rainfall prediction model. The training module is used to train the initial short-term rainfall prediction model using the preprocessed multi-source dataset, and to optimize the network parameters through back-tuning using a combined loss function to obtain the final short-term rainfall prediction model. The output module inputs the radar echo and satellite cloud image to be measured into the final short-term rainfall prediction model to obtain the short-term precipitation prediction result.

10. An electronic device, characterized in that, include: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of claims 1-8 according to the obtained program instructions.

Images