An improved short-term precipitation forecast method using random mask and transformer

Abstract

The application discloses an improved short-term and nowcasting precipitation prediction method using random masks and a Transformer, and belongs to the field of precipitation prediction. The improved short-term and nowcasting precipitation prediction method using random masks and a Transformer comprises the following steps: S1, constructing a random mask spatiotemporal sequence image; S2, constructing a network model, and inputting the spatiotemporal sequence image marked with the mask into the network for model training; the network model comprises an encoder-decoder structure with a UNet as a core model, a SwinTransformer module is embedded in the encoder, and an SE-Net attention mechanism is introduced; S3, in the model training process, a prediction value is obtained through a forward propagation process of the input image, then the model is continuously fine-tuned according to a loss function, the loss function is minimized, and the accurate prediction capability of the model is realized; and S4, L1+L2 regularization is used in the training process to prevent overfitting. The high-order non-stationarity in the modeling spatiotemporal sequence is improved, and the short-term and long-term dependence information in the spatiotemporal sequence is learned at the same time, so that the prediction accuracy of the model is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of precipitation prediction, and particularly relates to a short-term precipitation prediction method improved by using random masks and a Transformer. BACKGROUND

[0002] With the continuous development of current technology, short-term precipitation prediction has always been an important problem in the field of weather prediction, and its goal is to accurately and timely predict the rainfall intensity of a local area within a relatively short period of time (0-6 hours), which plays a crucial role in many fields such as economy, agriculture, commerce, transportation industry and electric power public utilities. Short-term precipitation prediction can be defined as a spatio-temporal sequence prediction problem, and the image extrapolation technology based on deep learning can effectively solve this problem, i.e. predicting the future M frames of image sequence according to the previous N frames of image sequence. This technology has been widely used in weather prediction, video prediction, traffic flow prediction and other fields, but it still has great limitations in prediction accuracy, which cannot meet the needs of actual business. First, natural spatio-temporal processes have high-order non-stationarity in many aspects, such as the generation, dissipation, accumulation or deformation of radar echoes in short-term precipitation prediction. The result is that the predicted image tends to be blurred. Secondly, on the one hand, when the target changes rapidly, the future image should be generated based on the nearby frames rather than the distant frames, which requires the prediction model to learn the short-term information in the spatio-temporal sequence. On the other hand, when the moving objects in the scene are frequently entangled together, it is difficult to separate them to generate future frames, which requires the prediction model to extract the context information in the image and the long-term information between the sequence images. Therefore, modeling the high-order non-stationarity in radar echo images and simultaneously learning the short-term and long-term dependency information in the image sequence is crucial for accurate prediction of future precipitation intensity.

[0003] The spatio-temporal sequence prediction model is mainly divided into three categories: the model based on recurrent neural network (RNN), the model based on convolutional neural network (CNN) and the model based on Transformer. The model based on RNN has a relatively obvious advantage in capturing long-term dependence information in spatio-temporal sequence by injecting gate mechanism to learn important information in sequence and forgetting secondary information. In 2015, Xingjian Shi et al. pioneered the combination of convolution structure and LSTM, and proposed a new network ConvLSTM that can learn spatial and temporal dimension features simultaneously. In 2016, Xingjian Shi et al. continued to propose TrajGRU for the local invariance problem of the convolution structure of ConvLSTM network. In 2017, Yunbo Wang et al. proposed a "Z" shaped network PredRNN for the defect that ConvLSTM is independent of each other at each time step. In 2018, Yunbo Wang et al. proposed PredRNN++, which uses a deep network of a new recursive structure Causal LSTM, and proposes Gradient Highway Unit to prevent gradient vanishing caused by long time. In 2019, Yunbo Wang et al. proposed MIM network for the high-order non-stationary problem of radar image, referring to the difference idea of classical time series prediction. In 2021, Haixu Wu et al. decomposed physical motion into transient change and motion trend, and proposed a new spatio-temporal prediction model MotionRNN. However, the model based on recurrent neural network has the characteristics of continuity, which is very time-consuming in the process of back propagation, and it is difficult to meet the timeliness demand of short-term precipitation prediction.

[0004] The encoder-decoder structure based on convolution has also achieved relatively superior performance in short-term precipitation prediction tasks. In 2019, Shreya Agrawal et al. introduced UNet network into precipitation prediction task, using convolution operation to capture spatial correlation, and stacking multiple radar frames to form time dimension to extract time correlation. In 2021, Kevin Trebing et al. proposed SmaAt-UNet, which uses only one quarter of the trainable parameters to get the performance comparable to UNet model. However, convolution operation is based on local connection to extract image features, which has great limitations in learning long-term dependence information of spatio-temporal sequence.

[0005] Transformer was first proposed in natural language processing (NLP), but due to its ability to extract long-term dependency information in sequences and good parallelism, it has been successfully introduced into many other fields. In 2021, Alexey Dosovitskiy et al. introduced the Transformer architecture into the field of computer vision and proposed the ViT model; in 2021, Ze Liu et al. proposed the Swin Transformer model, which limits self-attention calculation in non-overlapping local windows by shifting windows, while allowing cross-window connections, improving efficiency and reducing computational complexity to some extent. However, the Transformer-based model has high performance requirements for computers and has certain limitations in learning short-term dependencies of spatio-temporal sequences, and there are still great difficulties in directly applying it to short-term precipitation prediction tasks. SUMMARY

[0006] In view of the deficiencies of the prior art, the purpose of the present application is to provide an improved short-term precipitation prediction method using random masking and Transformer.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] An improved short-term precipitation prediction method using random masking and Transformer, comprising the following steps:

[0009] S1, randomly mask the spatio-temporal sequence image;

[0010] S2, construct a network model and input the masked spatio-temporal sequence image into the network for model training; the network model comprises an encoder-decoder structure with UNet as the core model, the SwinTransformer module is embedded in the encoder, and the SENet attention mechanism is introduced;

[0011] S3, during the model training process, the input image obtains a predicted value through the forward propagation process, and then the loss function is used for backward optimization to continuously fine-tune the model, minimize the loss function, and realize the accurate prediction ability of the model;

[0012] S4, L1+L2 regularization is used during the training process to prevent overfitting.

[0013] Further, in S1, the patches of the image sequence are randomly masked, and then the masked areas are marked, and the marked image sequence is input into the network;

[0014] And in S1, the input image with a masking rate of 75% is used for training, and the batch normalization operation is applied to the input image after random masking, so that it conforms to the Gaussian distribution to stabilize the training process.

[0015] Further, in S2, the encoder comprises a double convolution operation, a max pooling operation, a SwinTransformer module and an SE-Net attention mechanism; the double convolution operation is used to double the feature channel number of the image, the max pooling is used to halve the size of the feature map, and four double convolution operations and max pooling operations are arranged alternately to learn short-term dependency information in the space-time sequence; the SwinTransformer module is embedded in the last part of the encoder to learn long-term dependency information in the space-time sequence; the SE-Net attention mechanism is introduced between the double convolution and max pooling operations at each layer to focus on important information in the channel dimension and suppress secondary information that is not important to the current task.

[0016] Further, in S2, the Swin Transformer module comprises a Patch Partition, a Linear Embedding and a Swin Transformer Block; first, the picture sequence is processed by the PatchPartition layer for block processing, the feature map is divided into multiple disjoint regions, then the channel data of each pixel is linearly transformed by the Linear Embedding layer, and finally the feature extraction is performed by the Swin Transformer Block layer.

[0017] Further, the W-MSA module in the Swin Transformer Block is used to limit the multi-head self-attention calculation within each local window, the SW-MSA module enables information to be transmitted between adjacent windows, and the calculation process of the multi-head self-attention is as follows:

[0018]

[0019]

[0020] MultiHead(Q,K,V)=Concat(head1,...,head h ) (3)

[0021] Wherein, the physical meanings of Q, K and V are query vector, key vector and value vector respectively, W Q , W K , W V represent the convolution kernel, represents the dimension size of the query vector, and B represents the relative position bias.

[0022] Further, the SENet attention mechanism includes Squeeze and Excitation operations, first performing Squeeze operation on the feature map obtained by convolution to obtain global features in the channel dimension, then performing Excitation operation on the obtained global features to learn the relationship between each channel and the weight size of different channels, and then multiplying the obtained weight by the initial feature map to obtain the final feature.

[0023] Further, in S3, during the model training process, forward propagation and backward optimization are performed, assuming that there are training samples N(x i ,y i ), where i∈[1,N] and the input is The standard output is The predicted output is The loss function is defined as MSE, i.e. the Euclidean distance between the predicted value and the true value, as follows:

[0024]

[0025] Further, in S4, the expressions of L1 and L2 regularizations are shown in formulas (5) and (6) respectively:

[0026] L1(w)=α∑ i |w i | (5)

[0027]

[0028] where α is a constant for controlling the degree of regularization, w i represents the reciprocal of the weight, and L1 regularization prevents overfitting by making the weight vector sparse during optimization; L1 regularization is added as a penalty term in the loss function, and the final loss function is shown in formula (7), and L2 regularization is deployed by setting the weight_decay parameter of the Adam optimizer;

[0029]

[0030] Further, during the model training process, a learning rate decay strategy is introduced, and the decay process is shown in formula (8):

[0031]

[0032] where decay_rate represents the initial coefficient, epoch i represents the i-th training, and α0 represents the initial learning rate.

[0033] An improved short-term precipitation prediction system using random mask and Transformer, comprising:

[0034] An image processing module: for random mask spatio-temporal sequence image;

[0035] A model construction module: for constructing a network model, and inputting the spatio-temporal sequence image marked by the mask into the network for model training; the network model comprises an encoder-decoder structure with UNet as the core model, a Swin Transformer module is embedded in the encoder, and an SENet attention mechanism is introduced;

[0036] A prediction module: in the model training process, the input image obtains a prediction value through a forward propagation process, and then continuously fine-tunes the model according to the loss function, minimizes the loss function, and realizes the accurate prediction ability of the model;

[0037] An optimization training module: L1+L2 regularization is used in the training process to prevent overfitting.

[0038] The beneficial effects of the present application: the Swin Transformer basic module is embedded in the UNet model, and the SENet attention module is equipped, the image sequence after random mask is used as input, the high-order non-stationary in the spatio-temporal sequence is modeled, and the short-term and long-term dependence information in the spatio-temporal sequence is learned at the same time, and the prediction accuracy of the model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below, and obviously, other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0040] Figure 1 is a method flowchart of the present application;

[0041] Figure 2 is a network model structure diagram of the present application;

[0042] Figure 3 is a Swin Transformer module structure diagram of the present application;

[0043] Figure 4 is a Swin Transformer calculation detail diagram of the present application;

[0044] Figure 5 is an SE Net attention mechanism structure diagram of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, a short-term precipitation forecasting method improved by using random masks and Transformers includes the following steps:

[0047] S1, random masked spatiotemporal sequence image;

[0048] Before inputting the image, the patches in the image sequence are first randomly masked, and then the masked regions are marked. The marked image sequence is then input into the network, and missing pixels are reconstructed using the unmasked patches to train the network's ability to model higher-order nonstationarities. To accelerate the model's training speed and improve its prediction accuracy, this invention employs a high-ratio masking scheme, using input images with a mask ratio of 75% for training, achieving optimal prediction results. Furthermore, batch normalization is applied to the randomly masked input images to make them conform to a Gaussian distribution, thus stabilizing the training process.

[0049] S2, construct a network model and input the masked spatiotemporal sequence images into the network for training and feature extraction;

[0050] like Figure 2 As shown, the network model includes an encoder-decoder structure with UNet as the core model, embedding a SwingTransformer module in the encoder and introducing the SENet attention mechanism.

[0051] The encoder comprises a double convolution operation, a maximum pooling operation, a SwinTransformer module and an SE-Net attention mechanism; the double convolution operation is used for doubling the feature channel number of the image, and the maximum pooling is used for halving the size of the feature map, four double convolution operations and maximum pooling operations are staggered in the application to learn short-term dependence information in the space-time sequence by using the inherent locality feature; the Swin Transformer module is embedded in the last part of the encoder for learning long-term dependence information in the space-time sequence; the obtained encoder part has the advantages of UNet and Swin Transformer, and can capture short-term and long-term dependencies in the space-time sequence at the same time. In order to further improve the feature extraction capability of the encoder, the SE-Net attention mechanism is introduced between the double convolution and maximum pooling operations of each layer to focus on important information in the channel dimension and suppress unimportant secondary information for the current task.

[0052] As shown in Figure 3 , the Swin Transformer module comprises image segmentation (Patch Partition), linear mapping (Linear Embedding) and Swin Transformer Block. First, the picture sequence is processed by the Patch Partition layer for block processing, and the feature map is divided into multiple disjoint regions, then the channel data of each pixel is linearly transformed by the Linear Embedding layer, and finally the feature extraction is performed by the Swin Transformer Block layer.

[0053] The W-MSA module in the Swin Transformer Block is used to limit the multi-head self-attention calculation within each local window, which can effectively reduce the calculation amount of self-attention calculation, and the SW-MSA module can allow information to be transmitted in adjacent windows, thereby realizing global modeling, wherein the learning process of W-MSA and SW-MSA is as shown in Figure 4 , and the calculation process of multi-head self-attention is as shown in formulas (1)-(3):

[0054]

[0055]

[0056] MultiHead(Q,K,V)=Concat(head1,...,head h ) (3)

[0057] Wherein, the physical meanings of Q, K and V are query vector, key vector and value vector respectively, W Q , W K, W V represents a convolution kernel, represents the dimension size of the query vector, and B represents relative position bias.

[0058] The SENet attention mechanism includes Squeeze and Excitation operations. First, the Squeeze operation is performed on the feature map obtained by convolution to obtain global features in the channel dimension. Then, the Excitation operation is performed on the obtained global features to learn the relationship between channels and the weight size of different channels. Finally, the obtained weight is multiplied by the initial feature map to obtain the final feature. The SENet attention mechanism can make the model focus on the channel features with more information and suppress the unimportant channel features, thereby improving the performance of the model. The overall structure diagram is shown in Figure 5 .

[0059] S3, in the model training process, reverse reconstruction is performed; the input image obtains a prediction value through the forward propagation process, and then the loss function is used for reverse optimization to continuously fine-tune the model, minimize the loss function, and realize the accurate prediction ability of the model.

[0060] The decoder part includes double convolution operation, upsampling and jump connection, which are used to restore the original resolution of the feature map. The upsampling operation is realized by the method of bilinear interpolation, and the jump connection part realizes the fusion of the bottom position information and the deep semantic information by splicing with the feature map of the current layer in the encoder.

[0061] In the model training process, forward propagation and reverse optimization are performed. Assuming that there are training samples N(x i ,y i ), wherein i∈[1,N], the input is The standard output is The predicted output is The loss function is defined as MSE, that is, the Euclidean distance between the predicted value and the true value, as shown in equation (4):

[0062]

[0063] The input image obtains a prediction value through the forward propagation process, and then the loss function is used for reverse optimization to continuously fine-tune the model, minimize the loss function, so that o i and y i are infinitely close, thereby realizing the accurate prediction ability of the model.

[0064] S4, in order to prevent overfitting, enhance the generalization ability of the model, the concept of regularization is introduced, L1+L2 regularization is used in the training process, the expressions of L1, L2 regularization are shown in formulas (5) and (6) respectively:

[0065] L1(w)=α∑ i |w i | (5)

[0066]

[0067] Wherein, alpha is a constant, used to control the degree of regularization, w i Indicates the reciprocal of the weight, L1 regularization prevents overfitting by making the weight vector sparse during optimization, compared with L1 regularization, L2 regularization tends to punish the weight vector with large value. The present application adds L1 regularization as a penalty term in the loss function, and the final loss function is shown in formula (7). At the same time, the present application deploys L2 regularization by setting the weight_decay parameter of Adam optimizer, wherein the penalty coefficients alpha of L1 and L2 regularization are both set to 0.0001.

[0068]

[0069] In the model training process, in order to control the update speed of learning rate, make it oscillate around the optimal value, so as to speed up the training speed, the present application also introduces the learning rate decay strategy, and the decay process is shown in formula (8):

[0070]

[0071] Wherein, decay_rate represents the initial coefficient, epoch i Indicates the i-th training, alpha0 represents the initial learning rate; the larger the learning rate, the faster the convergence speed of the model, so a larger learning rate will be set in the early stage of training to ensure accelerated convergence, when the training reaches a certain degree, the too large learning rate may make the model fall into local optimal solution, and reducing the learning rate can make the convergence step smaller, so that the model learns more optimized.

[0072] In the description of the present specification, the description of the terms "one embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0073] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. An improved short-term precipitation forecast method using random masking and a Transformer, characterized in that, The method comprises the following steps: S1, randomly mask the spatio-temporal sequence image; S2, construct a network model, and input the masked spatio-temporal sequence image into the network for model training; the network model comprises an encoder-decoder structure with UNet as the core model, a SwinTransformer module embedded in the encoder, and an SENet attention mechanism; S3, during the model training process, the input image obtains a predicted value through a forward propagation process, and then the model is continuously fine-tuned according to the loss function, the loss function is minimized, and the accurate prediction ability of the model is realized; S4, L1+L2 regularization is used in the training process to prevent overfitting; In S2, the encoder comprises double convolution operation, maximum pooling operation, SwinTransformer module and SENet attention mechanism; the double convolution operation is used to double the feature channel number of the image, the maximum pooling operation is used to halve the size of the feature map, and the four double convolution operations and maximum pooling operations are staggered to learn the short-term dependence information in the spatio-temporal sequence; the SwinTransformer module is embedded in the last part of the encoder to learn the long-term dependence information in the spatio-temporal sequence; The SENet attention mechanism is introduced between the double convolution and maximum pooling operations at each layer to focus on important information in the channel dimension and suppress secondary information that is not important to the current task; In S2, the Swin Transformer module comprises Patch Partition, Linear Embedding and SwinTransformer Block; first, the picture sequence is processed by the Patch Partition layer for block processing, the feature map is divided into multiple disjoint regions, then the channel data of each pixel is linearly transformed by the Linear Embedding layer, and finally the feature is extracted by the SwinTransformer Block layer; The SENet attention mechanism comprises Squeeze and Excitation operations, first, the Squeeze operation is performed on the feature map obtained by convolution to obtain global features in the channel dimension, then the Excitation operation is performed on the global features to learn the relationship between channels and the weight size of different channels, and finally the obtained weight is multiplied by the original feature map to obtain the final feature.

2. The short-term precipitation forecast method improved by random masking and the Transformer according to claim 1, characterized in that, In S1, the patches of the image sequence are randomly masked, and then the masked regions are labeled, and the labeled image sequence is input into the network; And in S1, the input image with a mask rate of 75% is used for training, and the batch normalization operation is applied to the input image after random masking, so that it conforms to the Gaussian distribution to stabilize the training process.

3. The short-term precipitation forecast method improved by random masking and the Transformer according to claim 1, characterized in that, The W-MSA module in the Swin Transformer Block is used to limit the multi-head self-attention calculation within each local window, the SW-MSA module can enable information to be transmitted between adjacent windows, and the calculation process of the multi-head self-attention is: MultiHead(Q, K, V) = Concat(head1,..., head h ) (3) Wherein, the physical meaning of Q, K, V is query vector, key vector and value vector respectively, W Q , W K , W V represents the convolution kernel, represents the dimension size of the query vector, B represents the relative position bias.

4. The short-term precipitation forecast method improved by random masking and the Transformer of claim 1, wherein, In S3, during the model training process, two steps of forward propagation and backward tuning are performed, assuming there are training samples N(x i ,y i ), where i∈[1,N] and the input is The standard output is The predicted output is The loss function is defined as MSE, i.e. the Euclidean distance between the predicted value and the true value, as follows:

5. The short-term precipitation forecast method improved by random masking and the Transformer according to claim 1, characterized in that, In S4, the L1 and L2 regularization expressions are shown in formulas (5) and (6) respectively: L1(w) = a∑ i |w i | (5) where a is a constant to control the degree of regularization, w i represents the inverse of the weight, and L1 regularization prevents overfitting by making the weight vector sparse during the optimization process; L1 regularization is added as a penalty term in the loss function, and the final loss function is shown in equation (7). At the same time, L2 regularization is deployed by setting the weight_decay parameter of the Adam optimizer; 6. The short-term precipitation forecast method improved by random masking and the Transformer according to claim 5, characterized in that, During the model training process, a learning rate decay strategy is introduced, and the decay process is shown in formula (8): wherein decay_rate represents an initial coefficient, epoch i represents the i-th training, and a0represents an initial learning rate.

7. An improved nowcasting precipitation forecasting system using random masking and Transformer, performing the forecasting method of any one of claims 1-6, characterized in that, Including: Image processing module: for random mask spatio-temporal sequence image; Model construction module: for constructing network model, and inputting the spatio-temporal sequence image marked with mask into network for model training; the network model includes an encoder-decoder structure with UNet as the core model, and the SwinTransformer module is embedded in the encoder, and the SENet attention mechanism is introduced; Prediction module: during the model training process, the input image obtains the prediction value through the forward propagation process, and then the loss function is used for backward optimization to continuously fine-tune the model, minimize the loss function, and realize the accurate prediction ability of the model; Optimization training module: L1+L2 regularization is used in the training process to prevent overfitting.

Images