A multi-modal fusion short-impending precipitation prediction method and device

Abstract

The application discloses a short-term precipitation prediction method and device of multi-modal fusion in the technical field of short-term precipitation prediction, which comprises the following steps: extracting time sequence features of meteorological station data and radar echo data by using a meteorological station feature extractor and a radar echo feature extractor respectively to obtain meteorological station time sequence features and radar echo time sequence features; encoding the meteorological station time sequence features and the radar echo time sequence features by using a meteorological station encoder and a radar echo encoder respectively to obtain meteorological station encoding features and radar echo encoding features; querying and aligning the meteorological station encoding features and the radar echo encoding features by using a cross-modal feature encoder to obtain multi-modal fusion features; and performing short-term precipitation prediction and error correction based on the multi-modal fusion features by using a predictor to obtain short-term precipitation prediction results. The application significantly improves the accuracy and stability of short-term precipitation prediction.

Description

Technical Field

[0001] This invention relates to a multimodal fusion method and apparatus for short-term precipitation prediction, belonging to the field of short-term precipitation prediction technology. Background Technology

[0002] To provide early warning guidance and reference for production activities in areas such as disaster prevention and mitigation, infrastructure construction, traffic management, and urban activities, short-term quantitative precipitation forecasting has become a major research focus in recent years. However, meteorological analysis is a complex engineering process, and although artificial intelligence has shown potential in weather forecasting, its application in short-term precipitation prediction is still in a rapid development stage. In recent years, the use of artificial intelligence to generate accurate and quantitative short-term precipitation forecasts has attracted increasing attention from researchers.

[0003] Short-term precipitation forecasting utilizes historical multi-source radar data and ground meteorological station observations to predict rainfall amounts for the next few hours. After years of development, numerical weather prediction methods have played a crucial role in both short- and long-term forecasting of meteorological systems. However, this method struggles to accurately forecast small-scale weather systems with short formation and dissipation cycles. This has spurred the development and implementation of data-driven methods based on artificial intelligence for short-term precipitation forecasting.

[0004] Data-driven models can be broadly categorized into two types: spatiotemporal sequence prediction models based on radar echoes, and sequence regression and graph spatiotemporal models based on precipitation time series from ground stations. The former can learn the underlying spatiotemporal patterns of echo motion and evolution, but there is a nonlinear inversion error between echo reflectivity and precipitation amount; the latter can utilize station network time series, but it is insufficient in characterizing the spatial structure and scale characteristics of heavy precipitation. Therefore, how to fully utilize the spatiotemporal attributes of radar echo sequences to characterize the movement and dissipation of precipitation clouds, and effectively integrate them with multi-source observations from ground stations, has become one of the core issues in short-term precipitation prediction. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion method and apparatus for short-term precipitation prediction. By fusing the characteristics of radar data and ground meteorological station observation data, short-term precipitation prediction is performed based on multi-source data, which significantly improves the accuracy and stability of short-term precipitation prediction.

[0006] To achieve the above objectives / to solve the above technical problems, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a multimodal fusion method for short-term precipitation prediction, comprising:

[0008] Acquire meteorological station data and radar echo data;

[0009] The time-series features of meteorological station data and radar echo data are extracted using meteorological station feature extractor and radar echo feature extractor, respectively, to obtain the time-series features of meteorological stations and radar echoes.

[0010] The meteorological station time series features and radar echo time series features are encoded using meteorological station encoders and radar echo encoders, respectively, to obtain meteorological station encoded features and radar echo encoded features;

[0011] A multimodal fusion feature is obtained by querying and aligning the coded features of meteorological stations and radar echoes using a cross-modal feature encoder.

[0012] Based on multimodal fusion features, a predictor is used to predict and correct short-term precipitation, resulting in short-term precipitation prediction results.

[0013] Optionally, a meteorological station feature extractor is used to extract the time-series features of the meteorological station data, resulting in the following meteorological station time-series features:

[0014] Feature extraction of meteorological station data using an LSTM network was performed to obtain hidden features of the stations.

[0015] Graph neural networks are used to perform spatial topology aggregation on the hidden features of sites to obtain features with enhanced spatial topology.

[0016] By using a fully connected network to linearly map the spatial topology-enhanced features, the temporal features of meteorological stations can be obtained.

[0017] Optionally, the step of using a graph neural network to perform spatial topological aggregation on the hidden features of sites to obtain spatially topologically enhanced features includes:

[0018] by Each meteorological station is a node in the graph, forming a node set. Using the relationships between various meteorological stations as edges, form an edge set E to construct a station graph of meteorological station data. ;

[0019] Topological weights are assigned to each node in the station map based on the distance and historical correlation between various meteorological stations. The calculation formula is as follows:

[0020] ;

[0021] ;

[0022] in, For the first The weather station and the first The spherical distance between weather stations and They represent the first The longitude and latitude of each weather station and They represent the first The longitude and latitude of each weather station This function represents the calculation of spherical distance based on the latitude and longitude of a station. Indicates the first The weather station and the first The correlation coefficient between historical precipitation sequences of individual meteorological stations and They represent the first The weather station and the first Each weather station during historical periods Precipitation observation sequence within, This represents the correlation coefficient calculation function, where T is the data sampling duration;

[0023] Using the weighted site map, neighborhood aggregation is performed on the hidden features of each site at each time step to obtain features with spatial topology enhancement.

[0024] Optionally, a radar echo feature extractor is used to extract the temporal features of the radar echo data, resulting in radar echo temporal features, including:

[0025] The radar echo data is downsampled using the Inception downsampling network to obtain downsampled radar echo data;

[0026] The spatiotemporal features of downsampled radar echo data are extracted using a recurrent convolutional neural network with stacked 3 layers of ConvLSTM, thus obtaining two-dimensional radar latent features.

[0027] Wavelet transform is applied to the two-dimensional radar latent features output by the recurrent convolutional neural network to obtain multi-scale radar features;

[0028] By pooling and linear mapping the multi-scale radar features, the timing features of the radar echo are obtained.

[0029] Optionally, both the meteorological station encoder and the radar echo encoder adopt a Transformer-based single-mode encoder. The single-mode encoder includes a multi-layer structure, with each layer containing a multi-head attention layer and a multi-layer perceptron layer. A residual connection and a normalization layer are provided after each layer.

[0030] Optionally, the meteorological station encoder and the radar echo encoder are used to encode the time-series features of the meteorological station and the radar echo, respectively, to obtain the coded features of the meteorological station and the coded features of the radar echo, including:

[0031] Define the dynamic score that changes with weather , The formula is as follows:

[0032] ;

[0033] in, Indicates the rate of change of radar echo. This indicates the spatial translational velocity of strong echo regions or convective cloud clusters in radar echoes. This indicates the percentage of heavy rainfall. They are respectively Weighting coefficients;

[0034] Based on dynamic scoring The formula for calculating the window length of an adaptive sliding window is as follows:

[0035] ;

[0036] in, Indicates scoring based on dynamics. The selected adaptive sliding window length, This represents the maximum window length of the adaptive sliding window. This represents the minimum window length of the adaptive sliding window;

[0037] Based on dynamic scoring Definition follows Adaptive time decay scale :

[0038] ;

[0039] in, This represents the maximum value of the time decay scale. This represents the minimum value of the time decay scale;

[0040] An adaptive sliding window is used to select the effective feature sequence lengths of the input meteorological station encoder and radar echo encoder from the meteorological station time series features and radar echo time series features, so as to obtain the filtered meteorological station time series features and radar echo time series features.

[0041] Based on time decay scale Calculate the exponentially decaying weights at each time step within the adaptive sliding window:

[0042] ;

[0043] ;

[0044] in, This represents the unnormalized exponentially decaying weight at time t. This represents the normalized exponentially decaying weight at time t. and , This represents the unnormalized exponentially decaying weight at time k. To adapt the window length of the sliding window;

[0045] The feature values ​​at each moment in the selected meteorological station time series features and radar echo time series features are weighted using exponential decay weights to obtain the weighted meteorological station time series features and radar echo time series features.

[0046] The weighted meteorological station time series features and radar echo time series features are input into the meteorological station encoder and radar echo encoder, respectively, to obtain the meteorological station coding features and radar echo coding features.

[0047] Optionally, the cross-modal feature encoder includes multiple cross-modal layers, each of which includes two cross-attention sub-layers in two directions, two intramodal self-attention sub-layers, and two feedforward networks, with gated residual units provided between the cross-attention sub-layers and the self-attention sub-layers.

[0048] Optionally, a cross-modal feature encoder is used to query and align the coded features of meteorological stations and radar echoes to obtain multimodal fusion features, including:

[0049] In each cross-modal layer, the feature information of meteorological station coding features and radar echo coding features is exchanged and aligned using the cross-attention sublayer, resulting in:

[0050] ;

[0051] ;

[0052] in, Indicates the first Pay attention to the characteristics of the radar echo data at time t after meteorological station data enhancement from the sub-layer output. Indicates the first Pay attention to the characteristics of meteorological station data at time t after radar echo data enhancement in the sub-layer output. This represents the cross-attention calculation using radar echo coding features as the query vector and meteorological station coding features as the key and value vectors. This represents the cross-attention calculation using meteorological station coding features as the query vector and radar echo coding features as the key and value vectors. Indicates the first Characteristics of radar echo data output from the cross-mode layer. Indicates the first Characteristics of meteorological station data output from the cross-modal layer. To adapt the window length of the sliding window;

[0053] Based on the output characteristics of the cross-attention sublayer, the cross-modal increment U for each mode is obtained, where the modes include radar echoes and meteorological stations;

[0054] Based on the cross-modal increment U, the output features of the cross-attention sublayer are optimized using gated residual unit control, as shown in the following formula:

[0055] ;

[0056] ;

[0057] ;

[0058] in, Indicates the cross-modal increment The context vector obtained by pooling. Indicates pooling operation; The gating coefficient, This represents the Sigmoid activation function. This indicates that feature concatenation is performed on the current modal features Z and c. and These represent the learnable gated weight matrix and bias parameters, respectively. This represents the output characteristics after gated residual fusion. For element-wise multiplication, Presentation layer normalization operation;

[0059] Attention is calculated on the output features of the gated residual unit using a self-attention sublayer:

[0060] ;

[0061] ;

[0062] in, , The first Characteristics of radar echo data and meteorological station data at time t from the output of the self-attention sublayer. This represents the self-attention calculation within the radar echo mode. This represents the self-attention calculation within the weather station mode. , These are the radar echo data characteristics and meteorological station data characteristics at time t, respectively, output by the gated residual unit;

[0063] The radar echo data characteristics and meteorological station data characteristics output by the l-th cross-mode layer are obtained through a feedforward network.

[0064] By stacking multiple cross-modal layers, radar-side fusion features that incorporate meteorological station information are output. and site-side fusion features that incorporate radar echo information ;

[0065] The radar-side fusion features and the site-side fusion features are concatenated and linearly mapped to obtain the multimodal fusion features, as shown in the following formula:

[0066] ;

[0067] in, Indicates multimodal fusion features, This represents a fusion mapping network.

[0068] Optionally, the step of using a predictor to perform short-term precipitation prediction and error correction based on multimodal fusion features to obtain short-term precipitation prediction results includes:

[0069] Based on multimodal fusion features, the initial prediction result is output using the main prediction branch of the predictor. ;

[0070] Calculate the residual correction using the residual correction branch of the predictor:

[0071] ;

[0072] in, This represents the residual correction amount. This represents the multilayer perceptron in the residual correction branch. Represents the hidden features of the predictor. Indicates optional environmental statistical characteristics, Indicates the prediction time step length. Total number of weather stations;

[0073] Using residual correction For the initial prediction results After making corrections, the final short-term precipitation forecast is obtained:

[0074] ;

[0075] in, This indicates the results of short-term precipitation forecasts.

[0076] Secondly, the present invention provides a multimodal fusion short-term precipitation prediction device, comprising:

[0077] The data acquisition module is used to acquire meteorological station data and radar echo data;

[0078] The feature extraction module is used to extract the temporal features of meteorological station data and radar echo data using meteorological station feature extractor and radar echo feature extractor respectively, so as to obtain meteorological station temporal features and radar echo temporal features.

[0079] The feature encoding module is used to encode the time-series features of meteorological stations and the time-series features of radar echoes using meteorological station encoders and radar echo encoders, respectively, to obtain the coded features of meteorological stations and the coded features of radar echoes.

[0080] The feature fusion module is used to query and align the coded features of meteorological stations and radar echoes using a cross-modal feature encoder to obtain multimodal fused features.

[0081] The prediction module is used to perform short-term precipitation prediction and error correction based on multimodal fusion features, and obtain short-term precipitation prediction results.

[0082] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0083] This invention proposes a multimodal fusion method and apparatus for short-term precipitation forecasting. It effectively utilizes the spatiotemporal attributes of radar echo data and multi-source observation data from ground meteorological stations. By extracting radar echo features and meteorological station features separately, and then encoding and fusing these features, multimodal fusion short-term precipitation forecasting is achieved, significantly improving the accuracy and stability of short-term precipitation forecasting, particularly in the forecasting of moderate to heavy rain and storm threshold ranges. Furthermore, this invention can predict more accurate precipitation areas, effectively improving forecasting performance. This invention is applicable to providing forecasting guidance in fields such as disaster prevention and relief, urban flooding early warning, and road traffic management, and has significant practical value. Attached Figure Description

[0084] Figure 1 A schematic diagram illustrating the steps of a multimodal fusion method for short-term precipitation prediction provided by the present invention;

[0085] Figure 2 This is a schematic diagram of the algorithm flow of the short-term precipitation prediction method in an embodiment of the present invention;

[0086] Figure 3 This is a schematic diagram of the structure of the meteorological station feature extractor in an embodiment of the present invention;

[0087] Figure 4 This is a schematic diagram of the radar echo feature extractor in an embodiment of the present invention;

[0088] Figure 5 This is a comparison chart of TS scores for short-term precipitation prediction results using different methods in this embodiment of the invention.

[0089] Figure 6This is a comparison chart of the MSE and MAE indices of the short-term precipitation prediction results after different methods were used to predict short-term precipitation in the embodiments of the present invention.

[0090] Figure 7 This is a schematic diagram of the structure of a multimodal fusion short-term precipitation prediction device provided by the present invention. Detailed Implementation

[0091] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0092] Example 1

[0093] This embodiment introduces a multimodal fusion method for short-term precipitation prediction, such as... Figure 1 , Figure 2 As shown, it includes the following steps:

[0094] Step A: Obtain meteorological station data and radar echo data.

[0095] This invention uses precipitation data collected from meteorological stations within a target area over a past period as meteorological station data, and radar images of the target area as radar echo data.

[0096] Meteorological station data is prone to missing data and data anomalies. Therefore, it is necessary to preprocess the collected meteorological station data, including filling in missing or missed data periods, and reducing the impact of non-stationary noise on subsequent modeling.

[0097] This invention employs an inverse distance-weighted interpolation method to perform interpolation within a temporal or spatial neighborhood, processing the data before and after missing values ​​to complete meteorological station data preprocessing. This method determines the precipitation value of the station to be interpolated by calculating the square of the inverse of the distance between the known station and the station to be interpolated. Let the station to be interpolated be... The set of valid sites in its surrounding area (preset area) is as follows: The calculation formula for interpolation is:

[0098] ;

[0099] ;

[0100] in, Let i be the weight of the i-th weather station. For the meteorological stations to be interpolated and the first The distances between the weather stations are calculated from the latitude and longitude coordinates of each station, where n is the number of surrounding stations of the weather station to be interpolated. For the precipitation at the meteorological station to be interpolated, Let be the precipitation at the i-th meteorological station.

[0101] After completing the meteorological station data, Z-score is used to standardize the data sequence for each meteorological station. The standardization formula is as follows: ,in, The mean, The variance is denoted as , and the standardized site time series tensor is . T is the data sampling duration. This represents the total number of weather stations.

[0102] Step B: Use the meteorological station feature extractor and the radar echo feature extractor to extract the time series features of the meteorological station data and the radar echo data respectively, and obtain the time series features of the meteorological station and the radar echo.

[0103] This invention sets up two parallel single-modal feature extractors. The meteorological station feature extractor uses a fully connected layer and a temporal network (LSTM) to extract the temporal dependence of station precipitation data. The radar echo feature extractor uses the Inception structure to extract features of radar echo data. Specifically, it uses the Inception downsampling network and a spatiotemporal network (ConvLSTM) to extract pixel-level spatiotemporal features of radar.

[0104] In this embodiment of the invention, the meteorological station data only uses precipitation observation data; therefore, the station data at each moment is a 1×N vector. Figure 3 As shown, the meteorological station feature extractor consists of an LSTM and a fully connected layer. The LSTM models the temporal dependencies of time-series data through memory cells and gating mechanisms. For each new input, if the input gate... When activated, its information is stored in the memory unit; conversely, if the forget gate... Activation then activates previously stored information. It will be forgotten; the latest unit output will go through the output gate. Propagation to the final state The station hidden features output from the LSTM are input into a graph neural network for spatial topology aggregation, outputting spatially topologically enhanced features. Finally, these enhanced features are passed through a two-layer fully connected network to output the meteorological station time feature vector for that moment. The temporal characteristics of meteorological stations are constructed by utilizing the characteristics of meteorological stations at all times within time period T.

[0105] This invention utilizes graph neural networks for spatiotemporal modeling, enhancing the model's ability to model the spatial and temporal dependencies of complex weather phenomena. The specific operation is as follows:

[0106] First, construct a site map, to Each meteorological station is a node in the graph, forming a node set. Edges are formed based on the relationships between different meteorological stations, forming an edge set E, and a weighted undirected graph is constructed. Short-term precipitation has obvious spatial continuity and propagation. Precipitation changes between neighboring stations are often correlated. Constructing a station map can explicitly introduce the spatial relationship between meteorological stations into the model.

[0107] The topological weights between nodes in the site graph are determined by a combination of two priors: geographical distance and historical relevance.

[0108] ;

[0109] ;

[0110] in, For the first The weather station and the first The spherical distance between weather stations and They represent the first The longitude and latitude of each weather station and They represent the first The longitude and latitude of each weather station This function represents the calculation of spherical distance based on the latitude and longitude of a station. Indicates the first The weather station and the first The correlation coefficient between historical precipitation sequences of individual meteorological stations and They represent the first The weather station and the first Each weather station during historical periods Precipitation observation sequence within, This represents the correlation coefficient calculation function, and its range of values ​​is... .

[0111] Graph neural networks utilize a station map to aggregate the hidden features of stations at each historical moment, ensuring that each station's features not only include its own historical precipitation variation information but also spatial influence information from surrounding related stations. This enhances the model's ability to model precipitation area movement, localized heavy precipitation expansion, and spatial dependencies between stations. The aggregated hidden features are then stacked according to historical moments. Finally, a linear mapping is performed along the time dimension on the stacked hidden features to obtain spatially topologically enhanced temporal features of meteorological stations. .

[0112] like Figure 4 As shown, the radar echo feature extractor first downsamples the input radar echo data to obtain downsampled radar echo data. Inception uses three different sizes of convolutional kernels (1×1, 3×3, and 5×5) and pooling layers to improve network sparsity, reduce image size, and decrease computational load. Then, the radar echo feature extractor uses a stacked 3-layer ConvLSTM to extract the spatiotemporal features of the downsampled radar echo data.

[0113] The feature extraction process of the radar echo feature extractor can be represented as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] in, The radar echo data at time t, This refers to the Inception downsampling network. The radar echo data at time t after downsampling. Indicates 3 layers The constructed recurrent convolutional neural network, It has 3 floors Output characteristics This represents a multilayer perceptron network. This indicates the average pooling operation. express Radar echo features extracted at any time.

[0118] To separate large-scale cloud morphology (low frequency) from localized severe convection details (high frequency) and improve the model's sensitivity to details and subtle changes in precipitation processes, this invention employs a 3-layer... A wavelet transform is applied after the constructed recurrent convolutional neural network and before average pooling. This wavelet scale transform extracts multi-scale features from the radar echo data. The input to the wavelet transform is the two-dimensional radar latent feature map corresponding to each historical moment, and the output is a multi-scale radar feature map fused from low-frequency structure and high-frequency details. This is then further processed by pooling and linear mapping to form the radar echo features. First, wavelet decomposition is performed on the radar latent feature map to obtain… The radar features are divided into subbands: LL is the low-frequency approximation subband obtained from two-dimensional wavelet decomposition; LH is the low-frequency subband in the horizontal direction and the high-frequency subband in the vertical direction; HL is the high-frequency subband in the horizontal direction and the low-frequency subband in the vertical direction; and HH is the diagonal high-frequency subband. LL is mainly used to preserve the large-scale continuous structure of precipitation clouds, while LH, HL, and HH are mainly used to capture details of local strong convection. After soft-threshold denoising of the high-frequency subband, it is then fused with the low-frequency components using channel weighting to obtain multi-scale radar features.

[0119] The radar echo time sequence characteristics are composed of radar echo characteristics at all times within time period T.

[0120] The ConvLSTM network is a neural network model that combines Convolutional Neural Networks (CNNs) and LSTMs. It adds convolutional operations to the internal states and gate operations of the LSTM to leverage the spatial correlations of the input data, similar to CNNs. At each time step, the ConvLSTM receives the input sequence and generates one hidden state and one unit state. The formula for a single-layer ConvLSTM network is as follows:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] in, This represents the input gate of the ConvLSTM network, which determines the new input. Which information needs to be written into memory cells? middle; Indicates the convolution operation; This represents the forget gate in a ConvLSTM network, which determines the memory cells from the previous time step. Which information should be retained, and which should be forgotten? This represents the output gate of the ConvLSTM network, used to update the hidden state. ; represents the hidden state at the previous time step; W represents the convolution kernel parameters of each gate or hidden layer; and b represents the bias parameters of each gate or hidden layer.

[0127] Step C: Encode the time-series features of the meteorological station and the radar echo using the meteorological station encoder and the radar echo encoder respectively, to obtain the coded features of the meteorological station and the coded features of the radar echo.

[0128] In this embodiment of the invention, both the meteorological station encoder and the radar echo encoder adopt a Transformer-based single-modal encoder. Each layer of the encoder contains a multi-head attention layer and a multi-layer perceptron (MLP) layer. In order to alleviate gradient vanishing and achieve faster training, residual connections and normalization layers are added after each layer.

[0129] In the feature encoding process, to avoid noise introduced by redundant data, this invention adopts an adaptive sliding window mechanism to select the effective feature sequence length of the input encoder. The length of the input sequence is dynamically adjusted according to the convection intensity and echo evolution rate. When the convection is intense, a shorter historical window is used to avoid interference from outdated information; when the weather is relatively stable, a longer historical window is used to retain more historical evolution information.

[0130] Define comprehensive dynamic score , The higher the value, the more drastic the weather changes.

[0131] ;

[0132] in, Indicates the rate of change of radar echo. This indicates the spatial translational velocity of strong echo regions or convective cloud clusters in radar echoes. This represents the percentage of heavy rainfall, obtained after normalization. ; They are respectively The weighting coefficients. In this embodiment of the invention, Take values ​​of 0.5, 0.3, and 0.2 respectively.

[0133] When the dynamics are large Take a shorter window when the situation is stable. A longer window is selected. In this embodiment of the invention, the minimum window length of the adaptive sliding window is chosen. Maximum window length of adaptive sliding window Assuming the data sampling frequency is 1 hour, the shortest input corresponds to 6 historical time steps and the longest input corresponds to 12 historical time steps.

[0134] Based on dynamic scoring The window length of an adaptive sliding window is defined as:

[0135] ;

[0136] in, Indicates scoring based on dynamics. The selected adaptive sliding window length.

[0137] Assign a time decay scale to different historical moments within the adaptive sliding window, and define the following... Adaptive time decay scale :

[0138] ;

[0139] in, The time constant representing exponential decay determines the rate at which historical weights decrease. The larger the size, the slower the decay. The smaller the value, the faster the decay. This represents the maximum value of the time decay scale. Minimum value representing the time decay scale .

[0140] Based on time decay scale Give each historical step size within the adaptive sliding window Assign exponential weights and normalize:

[0141] ;

[0142] ;

[0143] in, This represents the unnormalized exponentially decaying weight at time t, with the weight decreasing the further away from the current time. This indicates the highest weight given in the most recent hour; This represents the normalized exponentially decaying weights at time t, obtained by normalization. and This facilitates stable training and aligns with different scales; This represents the unnormalized exponentially decaying weight at time k.

[0144] After selecting the effective feature sequence length of the input encoder using an adaptive sliding window mechanism, then... The selected feature sequence is weighted, and then the weighted features are input into a single-modal feature encoder.

[0145] Feature weighting allows historical features closer to the current time to receive higher weights, while the weights of historical features farther away from the current time are gradually reduced, thereby reducing the interference of outdated historical information on short-term predictions.

[0146] The input to the single-mode feature encoder is a matrix composed of radar echo time-series features. A matrix composed of the time-series characteristics of meteorological stations The output is the radar echo encoded feature. and meteorological station coding characteristics ,in, The effective input sequence length selected for the adaptive sliding window mechanism.

[0147] The adaptive window and temporal weights do not change the form of the objective function. Temporal smoothing regularization is added during model training to constrain the attention distribution, as shown in the following formula:

[0148] ;

[0149] in, For time smoothing regularization, For a single-mode feature encoder in the first... The average attention vector at each historical moment is used to suppress non-physical oscillations. This represents the squared L2 norm. Introducing this regularization term can suppress drastic jumps in attention distribution between adjacent historical moments, improve model training stability, and reduce non-physical jitter.

[0150] Step D: Utilize a cross-modal feature encoder to query and align meteorological station coding features and radar echo coding features through hierarchical multidimensional cross-attention, thereby obtaining multimodal fusion features.

[0151] The cross-modal feature encoder comprises multiple cross-modal layers. Each cross-modal layer consists of two-directional cross-attention sublayers, two intra-modal self-attention sublayers, and two feedforward networks. The two-directional cross-attention sublayers are used to query radar echo encoded features for meteorological station encoded features and vice versa. Similar to the single-modal feature encoder, normalization layers and residual connections are added between the attention sublayers. Different numbers of cross-modal layers can be stacked for different tasks, with the output of each layer serving as the input to the next.

[0152] Step D01: In each cross-modal layer, a cross-attention sub-layer is first applied to exchange and align the feature information of radar data and meteorological station data, and to learn the joint cross-modal representation, such as the correlation between heavy precipitation data from meteorological stations and radar echo data.

[0153] In the In the layer, calculation Characteristics of radar echo data output from the layer Characteristics of meteorological station data The similarity and output of the first The cross-modal characteristics of the layer are expressed in the following formula:

[0154] ;

[0155] ;

[0156] in, Indicates the first Pay attention to the characteristics of the radar echo data at time t after meteorological station data enhancement from the sub-layer output. Indicates the first Pay attention to the characteristics of meteorological station data at time t after radar echo data enhancement in the sub-layer output. This represents the cross-attention calculation using radar echo coding features as the query vector and meteorological station coding features as the key and value vectors. This represents the cross-attention calculation using meteorological station coding features as the query vector and radar echo coding features as the key and value vectors.

[0157] Step D02: To suppress noise generated during multimodal alignment, this invention introduces a gated residual unit for soft selection of cross-modal fusion. After the cross-modal feature encoder completes bidirectional alignment between the radar and meteorological stations through bidirectional cross-attention, it obtains the increment of each mode based on the supplementary information obtained by one mode from the other mode. For the station branch, U represents supplementary information provided to the station features by radar echo features; for the radar branch, U represents supplementary information provided to the radar features by meteorological station features. To suppress the amplification of noise alignment and enhance numerical stability, this invention sets a gated residual unit after the cross-attention sublayer and before the self-attention sublayer, and uses the gated residual unit to control the intensity of the incremental U injected into the current modal feature.

[0158] Let the current modal input feature be Z( or The cross-modal increment obtained by cross-attention is ,definition:

[0159] ;

[0160] ;

[0161] ;

[0162] in, Indicates the cross-modal increment The context vector obtained by pooling, Indicates pooling operation, This represents the feature dimension of vector c; The gating coefficient, This represents the Sigmoid activation function; This indicates that features of Z and c are concatenated. and These represent the learnable gated weight matrix and bias parameters, respectively. The gated weights can suppress the impact of anomalous alignment on the backbone representation. This represents the output characteristics after gated residual fusion. For element-wise multiplication, Presentation layer normalization operation.

[0163] Step D03: To further establish internal connections between cross-modal features, a self-attention sub-layer is connected after the cross-attention sub-layer. The attention calculation formula is as follows:

[0164] ;

[0165] ;

[0166] in, , The first Characteristics of radar echo data and meteorological station data at time t output by the self-attention sub-layer; This represents the self-attention calculation within the radar echo mode, used to establish the correlation between different radar echo feature markers; This represents the self-attention calculation within a meteorological station mode, used to establish the association between feature labels of different meteorological stations; , These are the radar echo data characteristics and meteorological station data characteristics at time t, respectively, output by the gated residual unit.

[0167] Step D04, Sequence Features and The output features of the l-th cross-modal layer are obtained through a feedforward network. and .

[0168] Step D05 involves stacking multiple cross-modal layers, ultimately outputting radar-side fusion features that incorporate meteorological station information. and site-side fusion features that incorporate radar echo information .

[0169] The radar-side fusion features and the site-side fusion features are concatenated and linearly mapped to obtain a unified multimodal fusion feature, which is then input into the predictor. Since the prediction target of this invention is precipitation, the predictor can focus on using the site-side fusion features.

[0170] Multimodal fusion features are represented as follows:

[0171] ;

[0172] in, Indicates multimodal fusion features, This represents a fusion mapping network.

[0173] Step E: Based on multimodal fusion features, a predictor is used to predict short-term precipitation. The predictor includes a main prediction branch and a residual correction branch. The main prediction branch is used to output the initial prediction results. The residual correction branch is used to output the residual correction amount. After residual correction, the final prediction result is obtained.

[0174] ;

[0175] ;

[0176] in, This represents the residual correction amount. This represents the multilayer perceptron in the residual correction branch. Represents the hidden features of the predictor. This indicates optional environmental statistical characteristics, including the actual precipitation statistics for the most recent 1 to 2 hours, and the regional average radar echo intensity, etc. Indicates the prediction time step length. Total number of weather stations; This indicates the final prediction result. This indicates the initial prediction result output by the main prediction branch.

[0177] By using an error correction branch to perform residual learning, systematic bias is reduced while generating the final prediction result, thus improving the robustness of the prediction result to outliers in the data.

[0178] In this embodiment of the invention, the parameters of each model are optimized through model training and validation to ensure effectiveness and generalization ability in short-term precipitation prediction tasks. First, the data is divided into a training set and a validation set. The training set is used for model training, and the validation set is used to evaluate model performance. 80% of the data is used as the training set, and the remaining 20% ​​is used as the test set. Using meteorological station precipitation data and radar echo data from the past 12 hours as input, the cumulative precipitation for the next 3 or 6 hours is predicted.

[0179] In each training cycle, training data is input into the model to obtain the model's prediction results. Then, the model's output prediction results are aligned with the actual precipitation labels, and the model parameters are optimized by minimizing the loss function.

[0180] ;

[0181] in, Indicates the first The first prediction time step, the first The actual precipitation at each weather station Indicates the first The first prediction time step, the first Precipitation forecasts for each weather station The loss function is represented in this embodiment. Loss function.

[0182] To evaluate the performance of the method of the present invention, embodiments of the present invention compare the method of the present invention with some advanced and widely used models and model data used in meteorological forecasting operations. The comparison methods include GRAPES_MESO, DCRNN, GWNN, ConvLSTM, and PredRNN. Among them, DCRNN and GWNN perform short-term precipitation forecasts based on meteorological station data, while ConvLSTM and PredRNN perform short-term precipitation forecasts based on radar echo data. All four models are implemented using the PyTorch framework, and the GRAPES_MESO model data is downloaded from the National Meteorological Science Data Center of China.

[0183] Hourly precipitation data from 85 automatic weather stations in Guangdong Province and 90 automatic weather stations in Guangxi Zhuang Autonomous Region from June to September 2016 to 2019 were selected as the original meteorological station dataset, with a data interval of 1 hour. Radar mosaic data of South China was selected as the original radar echo data, with a radar echo data interval of 6 minutes. 3-hour short-term precipitation forecasts were performed using the method of this invention and comparative methods. The TS score, MSE, and MAE indices were used as evaluation indicators, and the prediction results of different methods were compared. Figure 5 , Figure 6 As shown.

[0184] As shown in the figure, the radar data-based comparison method outperforms the site data-based method, especially in forecasting heavy rainfall. This is attributed to the fact that larger echo values ​​in radar echo data are more readily correlated with heavy precipitation. While the site data-based comparison method, due to the use of the MSE loss function, tends to favor intermediate values, it offers advantages such as smaller models, faster training speeds, and wider site distribution. Our proposed method (Ours) integrates the advantages of both types of data, resulting in greater accuracy, particularly in heavy rainfall prediction, and can help us better prevent severe weather events during the summer. Our proposed method outperforms other comparison methods in TS score, MSE, and MAE indices, fully demonstrating its effectiveness.

[0185] The radar echo feature extractor of this invention employs a ConvLSTM structure, and the final prediction results show better performance than using ConvLSTM alone, demonstrating the effectiveness of the multi-modal fusion strategy in short-term precipitation prediction. Specifically, this invention has a significant advantage in predicting heavy precipitation. The input data includes meteorological station data, which more directly reflects heavy precipitation; larger values ​​represent stronger precipitation, a phenomenon not particularly evident in weak precipitation. The inclusion of meteorological station data helps the model learn relevant patterns.

[0186] Since both radar echo data and meteorological station data have very obvious characteristics of precipitation location, the method of this invention is also very effective in predicting precipitation location.

[0187] The above experiments demonstrate that the method of this invention can predict more accurate precipitation locations and intensities, effectively improving prediction performance. Furthermore, the experiments prove the effectiveness of the multimodal fusion strategy in short-term precipitation prediction tasks.

[0188] Example 2

[0189] Based on the same inventive concept as Embodiment 1, this embodiment introduces a multimodal fusion short-term precipitation prediction device, such as... Figure 7 As shown, it includes:

[0190] The data acquisition module is used to acquire meteorological station data and radar echo data.

[0191] The feature extraction module is used to extract the temporal features of meteorological station data and radar echo data using meteorological station feature extractor and radar echo feature extractor respectively, so as to obtain the temporal features of meteorological stations and radar echoes.

[0192] The feature encoding module is used to encode the time-series features of meteorological stations and the time-series features of radar echoes using meteorological station encoders and radar echo encoders, respectively, to obtain the coded features of meteorological stations and the coded features of radar echoes.

[0193] The feature fusion module is used to query and align the coded features of meteorological stations and radar echoes using a cross-modal feature encoder to obtain multimodal fused features.

[0194] The prediction module is used to perform short-term precipitation prediction and error correction based on multimodal fusion features, and obtain short-term precipitation prediction results.

[0195] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0196] In summary, this invention extracts temporal features from meteorological station data and radar echo data using two single-modal feature extractors, respectively. It enhances the model's ability to model the spatial and temporal dependencies of complex weather phenomena through graph neural networks, and utilizes wavelet transform to extract multi-scale features from radar echo images, enhancing the model's sensitivity to details and subtle changes during precipitation. A Transformer single-modal encoder encodes the temporal features, strengthening dependencies. A cross-modal feature encoder is used to cross-learn the feature information of the two modalities, and multi-modal fusion is used to enhance the model's prediction accuracy for short-term precipitation. Finally, a predictor is used for precipitation prediction, and a residual correction branch is incorporated for residual learning, reducing systematic bias while generating the final prediction result and improving the robustness of the prediction result to outliers in the data.

[0197] This invention can predict more accurate precipitation locations and intensity, effectively improving prediction performance and demonstrating the effectiveness of multimodal fusion strategies in short-term precipitation prediction tasks.

[0198] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0200] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0201] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0202] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multimodal fusion method for short-term precipitation prediction, characterized in that, include: Acquire meteorological station data and radar echo data; The time-series features of meteorological station data and radar echo data are extracted using meteorological station feature extractor and radar echo feature extractor, respectively, to obtain the time-series features of meteorological stations and radar echoes. The meteorological station time series features and radar echo time series features are encoded using meteorological station encoders and radar echo encoders, respectively, to obtain meteorological station encoded features and radar echo encoded features; A multimodal fusion feature is obtained by querying and aligning the coded features of meteorological stations and radar echoes using a cross-modal feature encoder. Based on multimodal fusion features, a predictor is used to predict and correct short-term precipitation, resulting in short-term precipitation prediction results.

2. The short-term precipitation prediction method according to claim 1, characterized in that, The time-series features of meteorological station data are extracted using a meteorological station feature extractor, resulting in the following: Feature extraction of meteorological station data using an LSTM network was performed to obtain hidden features of the stations. Graph neural networks are used to perform spatial topology aggregation on the hidden features of sites to obtain features with enhanced spatial topology. By using a fully connected network to linearly map the spatial topology-enhanced features, the temporal features of meteorological stations can be obtained.

3. The short-term precipitation prediction method according to claim 2, characterized in that, The method of using graph neural networks to perform spatial topological aggregation on hidden site features to obtain spatially topologically enhanced features includes: by Each meteorological station is a node in the graph, forming a node set. Using the relationships between various meteorological stations as edges, form an edge set E to construct a station graph of meteorological station data. ; Topological weights are assigned to each node in the station map based on the distance and historical correlation between various meteorological stations. The calculation formula is as follows: ； ； in, For the first The weather station and the first The spherical distance between weather stations and They represent the first The longitude and latitude of each weather station and They represent the first The longitude and latitude of each weather station This function represents the calculation of spherical distance based on the latitude and longitude of a station. Indicates the first The weather station and the first The correlation coefficient between historical precipitation sequences of individual meteorological stations and They represent the first The weather station and the first Each weather station during historical periods Precipitation observation sequence within, This represents the correlation coefficient calculation function, where T is the data sampling duration; Using the weighted site map, neighborhood aggregation is performed on the hidden features of each site at each time step to obtain features with spatial topology enhancement.

4. The short-term precipitation prediction method according to claim 1, characterized in that, The radar echo feature extractor is used to extract the temporal features of the radar echo data, resulting in radar echo temporal features, including: The radar echo data is downsampled using the Inception downsampling network to obtain downsampled radar echo data; The spatiotemporal features of downsampled radar echo data are extracted using a recurrent convolutional neural network with stacked 3 layers of ConvLSTM, thus obtaining two-dimensional radar latent features. Wavelet transform is applied to the two-dimensional radar latent features output by the recurrent convolutional neural network to obtain multi-scale radar features; By pooling and linear mapping the multi-scale radar features, the timing features of the radar echo are obtained.

5. The short-term precipitation prediction method according to claim 1, characterized in that, Both the meteorological station encoder and the radar echo encoder adopt a Transformer-based single-mode encoder. The single-mode encoder includes a multi-layer structure, with each layer containing a multi-head attention layer and a multi-layer perceptron layer. Residual connections and normalization layers are provided after each layer.

6. The short-term precipitation prediction method according to claim 5, characterized in that, The time-series features of meteorological stations and radar echoes are encoded using meteorological station encoders and radar echo encoders, respectively, to obtain coded features of meteorological stations and coded features of radar echoes, including: Define the dynamic score that changes with weather , The formula is as follows: ； in, Indicates the rate of change of radar echo. This indicates the spatial translational velocity of strong echo regions or convective cloud clusters in radar echoes. This indicates the percentage of heavy rainfall. They are respectively Weighting coefficients; Based on dynamic scoring The formula for calculating the window length of an adaptive sliding window is as follows: ； in, Indicates scoring based on dynamics. The selected window length of the adaptive sliding window. This represents the maximum window length of the adaptive sliding window. This represents the minimum window length of the adaptive sliding window; Based on dynamic scoring Definition follows Adaptive time decay scale : ； in, This represents the maximum value of the time decay scale. This represents the minimum value of the time decay scale; An adaptive sliding window is used to select the effective feature sequence lengths of the input meteorological station encoder and radar echo encoder from the meteorological station time series features and radar echo time series features, so as to obtain the filtered meteorological station time series features and radar echo time series features. Based on time decay scale Calculate the exponentially decaying weights at each time step within the adaptive sliding window: ； ； in, This represents the unnormalized exponentially decaying weight at time t. This represents the normalized exponentially decaying weight at time t. and , This represents the unnormalized exponentially decaying weight at time k. To adapt the window length of the sliding window; The feature values ​​at each moment in the selected meteorological station time series features and radar echo time series features are weighted using exponential decay weights to obtain the weighted meteorological station time series features and radar echo time series features. The weighted meteorological station time series features and radar echo time series features are input into the meteorological station encoder and radar echo encoder, respectively, to obtain the meteorological station coding features and radar echo coding features.

7. The short-term precipitation prediction method according to claim 1, characterized in that, The cross-modal feature encoder includes multiple cross-modal layers. Each cross-modal layer includes two cross-attention sub-layers in two directions, two intramodal self-attention sub-layers, and two feedforward networks. A gated residual unit is provided between the cross-attention sub-layers and the self-attention sub-layers.

8. The short-term precipitation prediction method according to claim 7, characterized in that, A cross-modal feature encoder is used to query and align the coded features of meteorological stations and radar echoes to obtain multimodal fusion features, including: In each cross-modal layer, the feature information of meteorological station coding features and radar echo coding features is exchanged and aligned using the cross-attention sublayer, resulting in: ； ； in, Indicates the first Pay attention to the characteristics of the radar echo data at time t after meteorological station data enhancement from the sub-layer output. Indicates the first Pay attention to the characteristics of meteorological station data at time t after radar echo data enhancement in the sub-layer output. This represents the cross-attention calculation using radar echo coding features as the query vector and meteorological station coding features as the key and value vectors. This represents the cross-attention calculation using meteorological station coding features as the query vector and radar echo coding features as the key and value vectors. Indicates the first Characteristics of radar echo data output from the cross-mode layer. Indicates the first Characteristics of meteorological station data output from the cross-modal layer. To adapt the window length of the sliding window; Based on the output characteristics of the cross-attention sublayer, the cross-modal increment U for each mode is obtained, where the modes include radar echoes and meteorological stations; Based on the cross-modal increment U, the output features of the cross-attention sublayer are optimized using gated residual unit control, as shown in the following formula: ； ； ； in, Indicates the cross-modal increment The context vector obtained by pooling. Indicates pooling operation; The gating coefficient, This represents the Sigmoid activation function. This indicates that feature concatenation is performed on the current modal features Z and c. and These represent the learnable gated weight matrix and bias parameters, respectively. This represents the output characteristics after gated residual fusion. For element-wise multiplication, Presentation layer normalization operation; Attention is calculated on the output features of the gated residual unit using a self-attention sublayer: ； ； in, , The first Characteristics of radar echo data and meteorological station data at time t from the output of the self-attention sublayer. This represents the self-attention calculation within the radar echo mode. This represents the self-attention calculation within the weather station mode. , These are the radar echo data characteristics and meteorological station data characteristics at time t, respectively, output by the gated residual unit; The radar echo data characteristics and meteorological station data characteristics output by the l-th cross-mode layer are obtained through a feedforward network. By stacking multiple cross-modal layers, radar-side fusion features that incorporate meteorological station information are output. and site-side fusion features that incorporate radar echo information ; The radar-side fusion features and the site-side fusion features are concatenated and linearly mapped to obtain the multimodal fusion features, as shown in the following formula: ； in, Indicates multimodal fusion features, This represents a fusion mapping network.

9. The short-term precipitation prediction method according to claim 1, characterized in that, The method based on multimodal fusion features utilizes a predictor to perform short-term precipitation prediction and error correction, resulting in short-term precipitation prediction results, including: Based on multimodal fusion features, the initial prediction result is output using the main prediction branch of the predictor. ; Calculate the residual correction using the residual correction branch of the predictor: ； in, This represents the residual correction amount. This represents the multilayer perceptron in the residual correction branch. Represents the hidden features of the predictor. Indicates optional environmental statistical characteristics, Indicates the prediction time step length. This represents the total number of weather stations. Using residual correction For the initial prediction results After making corrections, the final short-term precipitation forecast is obtained: ； in, This indicates the short-term precipitation forecast results.

10. A multimodal fusion short-term precipitation forecasting device, characterized in that, include: The data acquisition module is used to acquire meteorological station data and radar echo data; The feature extraction module is used to extract the temporal features of meteorological station data and radar echo data using meteorological station feature extractor and radar echo feature extractor respectively, so as to obtain meteorological station temporal features and radar echo temporal features. The feature encoding module is used to encode the time-series features of meteorological stations and the time-series features of radar echoes using meteorological station encoders and radar echo encoders, respectively, to obtain the coded features of meteorological stations and the coded features of radar echoes. The feature fusion module is used to query and align the coded features of meteorological stations and radar echoes using a cross-modal feature encoder to obtain multimodal fused features. The prediction module is used to perform short-term precipitation prediction and error correction based on multimodal fusion features, and obtain short-term precipitation prediction results.

Images