A radar echo extrapolation method based on self-attention spatiotemporal neural network model

Abstract

The present invention provides a radar echo extrapolation method based on a self-attention spatiotemporal neural network model, comprising: data preprocessing: performing denoising, single frame classification, event classification, normalization and historical and future segment division processing on original PPI combined reflectivity radar echo data that has been quality controlled by correlation coefficients in sequence; training the neural network model: transferring the training set into a SAB‑STN for training, and saving the neural network model parameters after the training is completed; testing the neural network model: inputting the unlabeled grid data in the test set into the neural network model, and finally outputting the future echo sequence prediction result.

Description

Technical Field

[0001] The invention belongs to the field of automated analysis of nowcasting, and in particular relates to a radar echo extrapolation method based on a self-attention spatiotemporal neural network model. Background Art

[0002] Nowcasting refers to the weather forecast for the current to 0-2 hours in the future. It is one of the most important weather forecasting services. The main forecast objects of nowcasting include thunderstorms, precipitation and severe convection, and are often carried out with the help of Doppler weather radar products and satellite cloud image data. Because in most cases, the upper limit of the warning time of these forecast objects is no more than 2 hours, and the application of numerical models to forecast these weather in the next 0-2 hours usually has a large error. Since image data is used, nowcasting is still based on identifying, tracking and predicting the image features of radar echoes and cloud images. However, with the development and maturity of related services, nowcasting has also been continuously optimized and improved by combining other technologies. As a form of nowcasting, radar echo extrapolation technology has been widely used in business, and the echo products it relies on are diverse in response to different business needs. Among them, single-station forecasts often comprehensively consider the basic reflectivity echoes of the head-up display (PPI, Plan Position Indicator) and the volume scan (RHI, Range Height Indicator), while the nowcast of a wider area often uses a combined reflectivity image generated by the PPI basic reflectivity products of multiple stations.

[0003] At present, radar echo extrapolation methods are divided into traditional methods and machine learning methods, and traditional methods include artificial experience extrapolation, centroid method, cross-correlation method and optical flow method. Specifically, artificial experience extrapolation is still the most widely used method, which mainly relies on the forecaster's knowledge and experience accumulation. In some forecasting tasks, for some weather targets, combined with some non-radar data comprehensive analysis, it can obtain a more general and reliable forecast result. The limitation of artificial experience extrapolation is that it not only requires sufficient professional knowledge, but also cannot obtain more intuitive and accurate extrapolation results. The centroid method focuses on echo monomers and tracks their motion. It usually analyzes the monomer characteristics from multiple factors such as echo intensity, shape, size and distribution, and then calculates the position of the monomer centroid at each time, and linearly fits the monomer motion vector and path, and then extrapolates the future position of the centroid. The cross-correlation method first divides the entire image into multiple sub-images, then calculates the correlation coefficient between the sub-images of adjacent echo frames, and then uses the maximum correlation coefficient to obtain the correlation between sub-images at different times to determine the echo motion vector used as the basis for extrapolation. The optical flow method analyzes echo motion based on the optical flow model and uses the optical flow to obtain a more accurate motion vector in the entire observation domain. The limitations of the above traditional methods are: (1) they are limited by various artificial rules, empirical cognition and actual conditions, and cannot grasp the chaotic system more accurately; (2) based on the linear extrapolation of motion vectors, there is a lack of more prediction details in the evolution of echo generation and disappearance, and even for weather systems with rapid changes such as severe convection, it still lacks satisfactory prediction capabilities. Compared with traditional extrapolation methods, extrapolation methods based on machine learning are good at mining implicit echo motion features from given data distributions and have better nonlinear fitting capabilities, including convolutional neural networks CNN, recurrent neural networks RNN, conditional generative adversarial networks cGAN, etc. However, the most important factor affecting the effectiveness of machine learning methods is still sufficient and high-quality data. Secondly, the performance of machine learning algorithms also varies depending on the application scenarios. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is that the existing traditional method of radar echo extrapolation relies on experience and linear extrapolation of motion vectors, resulting in insufficient prediction accuracy, and the existing machine learning methods are insufficient in the use of echo data and extraction of echo motion features. A data-driven method is proposed to design a self-attention spatiotemporal network to realize radar echo extrapolation, and a matching training data set cleaning method is proposed. A neural network model is trained using a large number of historical echo sequences obtained by the data cleaning method to predict future echo states, which can reduce the errors and instabilities caused by linear extrapolation of motion vectors, enhance the perception and prediction capabilities of chaotic systems, and improve the intelligence level of radar echo extrapolation. The following steps are included:

[0005] Step 1, data preprocessing: perform data preprocessing on the PPI combined reflectivity radar echo products that have been quality controlled by correlation coefficients, divide each fixed-length echo sequence into historical segments and future segments, and obtain training sets and test sets;

[0006] Step 2, training the neural network model: input the echo sequence data in the training set into the Self-Attention-Based Spatiotemporal Net (SAB-STN) neural network model for training, and save the neural network model parameters after the training is completed;

[0007] Step 3, test the neural network model: input the historical echo sequence data in the test set into the neural network model trained in step 2 to test the neural network model, and output the prediction results extrapolated from the historical echoes to the future.

[0008] Step 1 includes the following steps:

[0009] Step 1-1: Echo image preprocessing: preprocess all echo images in the data set, where the data set comes from the National Meteorological Data Network and contains radar combined reflectivity for the past five years;

[0010] Step 1-2: echo sequence preprocessing: According to the preprocessing results of the echo images, all echo sequences are preprocessed, including sequence category labeling, sequence classification, sequence category balanced sampling and sequence history and future segment division;

[0011] Step 1-3: Divide the data set: Divide the preprocessed echo sequence into the test set according to the year, and the rest into the training set.

[0012] Step 1-1 includes the following steps:

[0013] Step 1-1-1: Image denoising. Removing suspicious non-precipitation echoes from the echo image includes the following steps:

[0014] Step 1-1-1-1: Convert the original single channel (channel number channel = 1) gray value echo image x raw The pixel value pix is ​​converted into the echo intensity value I, and the calculation formula is as follows:

[0015]

[0016] Step 1-1-1-2, set the minimum threshold I of precipitation echo intensity according to the data source th , according to I th The echo image is subjected to a first noise reduction with the echo intensity value I obtained in step 1-1-1-1. The first noise reduction is defined as follows: for an echo intensity less than the minimum threshold value Ith The pixels that are detected and the pixels that are missing are NaN, and their values ​​are reset to zero. In other cases, the pixel values ​​remain unchanged. The formula is as follows:

[0017]

[0018] After the first noise reduction is completed, a single-channel echo image x is obtained by removing the lower echo intensity. raw ;

[0019] Step 1-1-1-3: The echo image x obtained by denoising once raw Perform connected domain analysis, obtain the area of ​​each connected domain in each image ccA, the area of ​​the connected domain ccA, and set the minimum threshold of the connected domain area ccA th =13, according to ccA th The echo image X is subjected to secondary denoising with ccA. The secondary denoising is specified as follows: For an area smaller than ccA th The connected domain, the pixel value in the connected domain is reset to zero, otherwise the pixel value remains unchanged. The formula is as follows:

[0020]

[0021] After the secondary denoising is completed, a single-channel echo image x without small-area clutter is obtained;

[0022] Step 1-1-2: Image classification, including the following steps:

[0023] Step 1-1-2-1: Count the number of pixels num whose pixel values ​​are not 0 in each denoised echo image x pix > 0, calculate the number of pixels num in the entire observation domain ob The percentage of the echo area to the observable area is P 1 , the formula is as follows:

[0024]

[0025] according to Is it 0? First, classify x into two categories, where: represents the echo image without echo after noise reduction, Represents the echo image with echo after noise reduction, and then sets the area ratio interval set (The values ​​in the set are percentages):

[0026]

[0027] And according to of The area ratio interval will Divided into 4 categories;

[0028] Step 1-1-2-2: Set the echo intensity level set θ I (The value unit in the set is dBZ):

[0029] Θ I ={(0, 20], (20, 30], (30, 40], (40, 70]},

[0030] Statistics for each The number of pixels with the same pixel value at each echo level Calculation and num pix >0, that is, the area ratio of a certain echo level area to the echo area P i , the formula is as follows:

[0031]

[0032] According to the last P that is not 0 after sorting in order i The corresponding i will Divide into 4 categories, and then combine the Divided into 4 categories, The final classification is 16 categories, plus Then x is divided into 17 categories;

[0033] in addition The corresponding label is (0, 0);

[0034] Step 1-1-3: Perform image standardization, the formula is as follows:

[0035]

[0036] Among them, pix is ​​the original pixel value, pix norm is the normalized pixel value.

[0037] Step 1-2 includes the following steps:

[0038] Step 1-2-1: Label the sequence category labels and create an echo category label sequence corresponding to the echo sequence:

[0039] According to the echo image category labels obtained in step 1-1-2, the category labels of all images in an echo sequence X are arranged in the order of the echo sequence to obtain the echo category label sequence X cl ;

[0040] Step 1-2-2: Sequence classification: Use the K-Means clustering algorithm to classify the echo category label sequence X clThe sequence is clustered in two category dimensions, and the respective echo sequence clustering results are obtained in the two category dimensions. Then, the two types of clustering results are combined to obtain a complete clustering result. Among them, the sequence sample distance used in the K-Means clustering algorithm is the Euclidean distance;

[0041] Step 1-2-3: Perform category balanced sampling on the sequence clustering results. The echo sequence sampling formula is as follows:

[0042]

[0043] Where N is the number of categories in the echo sequence data set, n j is the total number of samples of category j, p i is the probability of sampling an echo sequence from category i;

[0044] Step 1-2-4: Divide the sequence into historical and future segments, and set the historical frame number and future frame number as the input length T of the SAB-STN neural network model respectively input With the extrapolated target length T target , then T input +T target Indicates the length of the entire sequence, and uses it to divide the echo sequence X into historical input segments X input with the future extrapolation segment X target Two parts: front and back:

[0045]

[0046] where x i represents the i-th frame image in the sequence (i=1, 2, ..., T input +T target );

[0047] Reshape the echo image size: (height, width) is reshaped to (448, 448), where height represents the height and width represents the width. At this time, the data shape of an echo sequence (length, channel, height, width) is (T input , 1, 448, 448), represented as an echo sequence It represents a high-dimensional real number space consisting of four dimensions: length L, number of channels C, height H, and width W. Length represents the length, and channel represents the number of channels.

[0048] Step 1-2-2 includes the following steps:

[0049] Step 1-2-2-1: Select the K-Means cluster center, set the number of categories k, and select the initialized k echo category label sequence samples X clAs the initial cluster center cs:

[0050] cs=cs 1 ,cs 2 , ..., cs k ;

[0051] where cs k Refers to the kth cluster center;

[0052] Step 1-2-2-2: Classify each sample. For each sample X cl Calculate its Euclidean distance dist to the k cluster centers and divide it into the class corresponding to the cluster center with the smallest distance:

[0053]

[0054] The arg min function returns the variable value that makes the function reach the minimum value. (i) represents the i-th sample X cl,i The cluster center cs that is closest to the k clusters j , where j = 1, 2, ..k, the calculation formula of the function dist is as follows:

[0055]

[0056] Among them, a and b represent different echo category label sequences with the same length. i , b i Respectively represent the echo label vectors of a and b at sequence position i;

[0057] Step 1-2-2-3: Update the centroid of the class: For each category, recalculate the centroid of all samples belonging to that category, i.e., the cluster center:

[0058]

[0059] where c i Indicates cs i The corresponding class, i = 1, 2, ..k;

[0060] Step 1-2-2-4: Get the clustering results. Repeat step 1-2-2-3 until convergence. At this time, determine the samples contained in each of the k clusters.

[0061] Step 2 includes the following steps:

[0062] Step 2-1: Establish a neural network model: build each module of the model and initialize model parameters;

[0063] Step 2-2: Initialize training hyperparameters: Use Adam optimizer to train the neural network model, set the learning rate lr, the number of samples contained in a group of samples input each time batchsize and the number of epochs for training set traversal;

[0064] Step 2-3: Neural network model parameter training and update: Use mean square error (MSE) as the loss function, take a batch size of data each time to input the neural network model to train and optimize the neural network model parameters, update the parameters through the back propagation algorithm, and initialize the model parameters using a combination of model pre-training and random initialization. The loss function MSE calculation formula is as follows:

[0065]

[0066] Where pix i represents the i-th pixel in a real echo image, It indicates that the real echo image corresponds to the ith pixel in the generated echo image, and height×width indicates the number of pixels in the image.

[0067] Step 2-1 includes the following steps:

[0068] Step 2-1-1: Construct encoder En: The encoder is a convolutional neural network, which is used to convert the original echo image into a higher-level image representation;

[0069] Step 2-1-2: Construct an echo series processor ESP (Echo Series Processor): The sequence processor consists of two parts: an ESP encoder ESP-En and an ESP decoder ESP-De. The self-attention module SAB is used as the basic unit to learn the temporal correlation between the image representations at each moment generated by the encoder in an echo sequence, as well as the spatial correlation of the image representations at each moment;

[0070] Step 2-1-3: Construct a decoder De: The decoder is a convolutional neural network used to convert the features generated by the sequence processor into extrapolated results.

[0071] Step 2-1-4: Initialize model parameters: For the convolution kernel in the SAB-STN neural network model, its weight is initialized using the Kaiming normal distribution method, and its bias is initialized using the 0 value; for the standardization layer in the model, its weight is initialized using the 1 value, and its bias is initialized using the 0 value.

[0072] Step 2-1-1 includes the following steps:

[0073] Step 2-1-1-1: Encoder En includes three convolution modules Conv2d En，1, Conv2d En,2 , Conv2d En,3 , two layers of maximum pooling layer MaxPool2d En,1 , MaxPool2d En,2 .

[0074] The modules are arranged in series, the order is Conv2d En,1 , MaxPool2d En,1 , Conv2d En,2 , MaxPool2d En,2 , Conv2d En,3 , where all convolution modules contain a convolution layer, a nonlinear activation function layer LeakyReLU, and Conv2d En,2 With Conv2d En,3 Each contains an additional normalization layer GroupNorm, and is placed before the convolutional layer. En,1 The input data size is (448,448), the convolution kernel size of the convolution layer is (3,3), the number of input channels is 1, the number of output channels is 4, the stride is 1, the zero padding is 1, the α parameter of LeakyReLU is 0.02, and the number of groups of GroupNorm is 1; MaxPool2d En，1 Conv2d is the maximum pooling layer with a kernel size of 2, a stride of 2, and zero padding of 0, which can reduce the size of the feature map by 1 / 2; En，2 The input data size is (224, 224), the number of convolutional layer input channels is 4, the number of output channels is 8, and other parameter settings are the same as Conv2d En，1 Consistent; MaxPool2d En,2 With MaxPool2d En，1 Consistent; Conv2d En,3 The input data size is (112,112), the number of convolutional layer input channels is 8, the number of output channels is 16, and other parameter settings are the same as Conv2d En，1 In addition, the input data enters Conv2d En，1 The 5-dimensional shape (batchsize, length, channel, height, width) is reshaped to 4-dimensional (batchsize*length, channel, height, width), which is expressed as

[0075] At this time, the shape is (batchsize,length,channel,height,width)=(batchsize,T input,1,448,448) input data is processed by the encoder En to obtain a shape of (batchsize*T input , 16, 112, 112) echo image characterization.

[0076] Step 2-1-2 includes the following steps:

[0077] Step 2-1-2-1: The self-attention module SAB includes a self-attention layer SA, a nonlinear activation function layer GeLU, and a normalization layer GroupNorm. The core of SAB lies in the self-attention layer SA.

[0078] In the self-attention layer SA, the input end simultaneously passes in the query item source and the key-value item source, and the query item source passes through the query convolution Conv2d Q Generate query item Q, key-value item source passes through key-value convolution Conv2d KV Generate key-value items KV, Conv2d Q , Conv2d KV Both are channel-wise two-dimensional convolutional layers, where Conv2d Q The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is equal to the number of output channels and is 16, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d KV The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 32, the stride is 1, the zero padding is 1, the number of groups is 16, the tile side length is regionsize, Q and KV are divided into several non-overlapping tiles with a side length of regionsize in the image height and width dimensions, and KV is divided into two parts along the channel dimension to obtain the key item K and the value item V. At this time, the shape of Q, K, and V is (batchsize,length,channel,height*width,patch), patch represents the number of tiles, that is,

[0079]

[0080] According to the self-attention mechanism, Q and K are used to calculate the attention weight score:

[0081] score=softmax(similarity(Q,K)),

[0082] The function similarity is used to calculate the matching degree mr between Q and K. Here, Q and K are reshaped separately, and then the corresponding values ​​of each dimension are multiplied. The scale factor scale is used to control the scale of the result. At this time, the shape of the matching degree mr is (batchsize, length q ,length kv , channel, height*width, patch), the calculation formula is:

[0083]

[0084] The function softmax is used to convert the matching degree mr into length kv The dimension is converted into a weight value score with a sum of 1, expressed as:

[0085]

[0086] After obtaining the attention weight score, reshape V so that V and score are multiplied by the corresponding values ​​of each dimension and along the length kv The dimension summation realizes the weighted summation of attention and obtains the self-attention calculation result. The self-attention calculation Self-Attention(Q, K, V) with Q, K, V as input is expressed as:

[0087]

[0088] The self-attention calculation results need to restore and reshape the image blocks, and then process them through a layer of depth-separable convolution module SepConv. The network in SepConv is arranged in series, with a channel-by-channel convolution layer Conv2d in turn. SC1 And a point-by-point convolution layer Conv2d SC2 , where Conv2d SC1 The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 64, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d SC2 The input data size is (112, 112), the convolution kernel size is (1, 1), the number of input channels is 64, the number of output channels is 16, the stride is 1, and the zero padding is 0.

[0089] The modules in the entire self-attention module SAB are arranged in series, namely GroupNorm, SA and GeLU. In addition, the output of SAB is the sum of the output of GroupNorm and the output of GeLU, that is, there is a residual connection between the two.

[0090] Step 2-1-2-2: The modules inside the ESP encoder ESP-En are serially spliced, including a time embedding layer TE (Time Embedding) and a multi-layer self-attention network ESP-En-M;

[0091] The time embedding layer TE does not contain trainable parameters. The time embedding te is generated based on the sine and cosine functions. The formula is as follows:

[0092]

[0093] Where t represents the tth time and i represents the i-th channel. The time embedding te generated by this formula has a shape of (batchsize, length, channel), that is, It is also necessary to copy, expand and reshape the time embedding te in the height and width dimensions so that its shape becomes (batchsize*length,channel,height,width), that is, The input of the temporal embedding layer TE is the output of the encoder En in step 2-1-1, and the output of the temporal embedding layer TE is the sum of the input of the temporal embedding layer TE and te, and serves as the input of the multi-layer self-attention network ESP-En-M, whose shape is (batchsize*T input , 16, 112, 112);

[0094] Multi-layer self-attention network ESP-En-M consists of num en_layer The input data is only the output data from the time embedding layer TE and it also serves as the query item source and key-value item source of the network. The shape of its output data is (batchsize*T input , 16, 112, 112);

[0095] Step 2-1-2-3: The modules inside the ESP decoder ESP-De are serially spliced, including a time embedding layer TE and a multi-layer self-attention network ESP-De-M.

[0096] The time embedding layer TE structure of ESP decoder ESP-De is consistent with TE of ESP-En. The input data of ESP decoder ESP-De is in the shape of (batchsize*T target ,16,112,112) is a tensor of all 0s.

[0097] Multi-layer self-attention network ESP-De-M consists of num de_layer SA modules are cascaded together, but the difference from ESP-En-M is that: first, num de_layer must be an even number; second, if the first condition is met, num de_layer The SA modules are paired in pairs. In a module pair, the query item source and key-value item source of the lower-level SA are both from the module one layer lower than the layer. The data shape is (batchsize*T target,16,112,112), while the query item source and key-value item source of the higher-level SA are the output of the lower-level SA and the output of ESP-En, and the corresponding data shapes are (batchsize*T target ,16,112,112) and (batchsize*T input , 16, 112, 112);

[0098] Step 2-1-3 includes the following steps:

[0099] Step 2-1-3-1: The modules in the decoder De are connected in series. There are three convolution modules and two anti-pooling layers, namely Conv2d De，1 ,MaxUnpool2d De,1 , Conv2d De，2 ,MaxUnpool2d De，2 , Conv2d De，3 Among them, all convolution modules contain a normalization layer GroupNorm, a convolution layer, and Conv2d De,1 With Conv2d De，2 Each contains an additional nonlinear activation function layer LeakyReLU, which is placed after the convolution layer. De，1 The input data size is (112, 112), the number of GroupNorm groups is 1, the convolution kernel size of the convolution layer is (3, 3), the number of input channels is 16, the number of output channels is 8, the stride is 1, the zero padding is 1, and the α parameter of LeakyReLU is 0.02; MaxUnpool2d De,1 The maximum unpooling layer with a kernel size of 2, a step size of 2, and zero padding of 0 can enlarge the feature map size to 2 times; Conv2d De,2 The input data size is (224, 224), the number of convolutional layer input channels is 8, the number of output channels is 4, and other parameter settings are the same as Conv2d De，1 Consistent;MaxUnpool2d De,2 with MaxUnpool2d De，1 Consistent; Conv2d De，3 The input data size is (448,448), the convolution kernel size is (1,1), the number of input channels is 4, the number of output channels is 1, the stride is 1, and the zero padding is 0. At this time, the shape of the output data is 4-dimensional (batchsize*T target ,1,448,448), the generation result of the entire self-attention spatiotemporal network SAB-STN needs to be reshaped to (batchsize,T target, 448,448,1), and then a truncation operation is performed to limit the generated result to the interval [-1, 1].

[0100] Step 3 includes the following steps:

[0101] Step 3-1: Read test samples: Input the test set generated in step 1 into the neural network model generated in step 2;

[0102] Step 3-2: Forward propagation of the neural network model. After the test set is input into the neural network model, the extrapolated results are generated.

[0103] The present invention has the following beneficial effects:

[0104] (1) The data preprocessing process distinguishes the echo sequences representing different precipitation processes in the data, and further performs category-balanced sampling on the echo sequences, so that the neural network model can effectively identify different precipitation weather conditions and learn the echo motion characteristics under different precipitation systems, thereby refining the echo extrapolation results;

[0105] (2) The ESP module of the neural network model has a self-attention mechanism. Through the self-attention mechanism, ESP can learn the temporal correlation of echo motion from the entire time scale of an echo sequence. At the same time, ESP relies on two-dimensional convolution to obtain the local spatial correlation within each frame of the echo image. The combined effect of the attention mechanism and the two-dimensional convolution enables the neural network model to have the ability to learn the echo motion characteristics in the local two-dimensional space in the full time domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0107] Figure 1 It is a flow chart of the present invention.

[0108] Figure 2 This is a network architecture diagram.

[0109] Figure 3 The encoder structure diagram.

[0110] Figure 4 This is the structure diagram of the self-attention module.

[0111] Figure 5 This is the encoder structure diagram of the echo sequence processor.

[0112] Figure 6 Diagram of the decoder structure of the echo sequence processor.

[0113] Figure 7 Construct a graph for the decoder.

[0114] Figure 8 This is a visualization of the denoising effect of an echo image sample.

[0115] Fig. 9 It is a forecast map for echo events with mixed cloud precipitation. DETAILED DESCRIPTION

[0116] like Figure 1 As shown, the present invention discloses that the present invention provides a radar echo extrapolation method based on a self-attention spatiotemporal network, and takes the extrapolation of radar echoes in Shanghai in 2021 as an example to demonstrate the method, including the following steps:

[0117] Step 1, data preprocessing: collect the PPI combined reflectivity of Shanghai from 2017 to 2021 from the National Meteorological Data Network as a data set, perform data preprocessing on it, divide each fixed-length echo sequence into historical segments and future segments, and obtain a training set containing data from 2017 to 2020 and a test set containing data from 2021;

[0118] Step 2, training the network: input the echo sequence data in the training set into the SAB-STN network for training, and save the network parameters after the training is completed;

[0119] Step 3, test the network: input the historical echo sequence data in the test set into the neural network model trained in step 2 for network testing, and output the prediction results extrapolated from the historical echoes to the future.

[0120] Step 1 includes the following steps:

[0121] Step 1-1: echo image preprocessing: preprocess all echo images in the data set;

[0122] Step 1-2: echo sequence preprocessing: According to the preprocessing results of the echo images, all echo sequences are preprocessed, including sequence category labeling, sequence classification, sequence category balanced sampling and sequence history and future segment division;

[0123] Step 1-3: Divide the data set: Divide the preprocessed echo sequences into 2021 data into a test set by year, and the rest into a training set. The test set includes 1,405 sequences and the training set includes 40,000 sequences.

[0124] Step 1-1 includes the following steps:

[0125] Step 1-1-1: Image denoising. Removing suspicious non-precipitation echoes from the echo image includes the following steps:

[0126] Step 1-1-1-1: Convert the original single channel (channel number channel = 1) gray value echo image x rawThe pixel value pix is ​​converted into the echo intensity value I, and the calculation formula is as follows:

[0127]

[0128] Step 1-1-1-2, set the minimum threshold I of precipitation echo intensity according to the data source th , according to I th The echo image is subjected to a first noise reduction with the echo intensity value I obtained in step 1-1-1-1. The first noise reduction is defined as follows: for an echo intensity less than the minimum threshold value I th The values ​​of pixels with missing values ​​(NaN) are reset to zero. In other cases, the pixel values ​​remain unchanged. The formula is as follows:

[0129]

[0130] After the first noise reduction is completed, a single-channel echo image x is obtained by removing the lower echo intensity. raw ;

[0131] Step 1-1-1-3: The echo image x obtained by denoising once raw Perform connected domain analysis, obtain the area of ​​each connected domain in each image ccA, the area of ​​each connected domain ccA, and take the minimum threshold of the connected domain area ccA th =13, according to ccA th The echo image X is subjected to secondary denoising with ccA. The secondary denoising is specified as follows: For an area smaller than ccA th The connected domain, the pixel value in the connected domain is reset to zero, otherwise the pixel value remains unchanged. The formula is as follows:

[0132]

[0133] After the secondary denoising is completed, a single-channel echo image x without small clutter is obtained. The denoising visualization effect of an echo image sample is as follows: Figure 8 As shown;

[0134] Step 1-1-2: Image classification, including the following steps:

[0135] Step 1-1-2-1: Count the number of pixels num whose pixel values ​​are not 0 in each denoised echo image x pix > 0, calculate the number of pixels num in the entire observation domain ob The percentage of the echo area to the observable area is P 1 , the formula is as follows:

[0136]

[0137] according to Is it 0? First, classify x into two categories, where: represents the echo image without echo after noise reduction, Represents the echo image with echo after noise reduction, and then sets the area ratio interval set (The values ​​in the set are percentages):

[0138]

[0139] And according to of The area ratio interval will Divided into 4 categories;

[0140] Step 1-1-2-2: Set the echo intensity level set θ I (The value unit in the set is dBZ):

[0141] Θ I ={(0, 20], (20, 30], (30, 40], (40, 70]},

[0142] Statistics for each The number of pixels with the same pixel value at each echo level Calculate its value with num pix >0, that is, the area ratio of a certain echo level area to the echo area P i , the formula is as follows:

[0143]

[0144] According to the last P that is not 0 after sorting in order i The corresponding i will Divide into 4 categories, and then combine the Divided into 4 categories, The final classification is 16 categories, plus Then x is divided into 17 categories, and the corresponding labels of each category are set as shown in Table 1:

[0145] Table 1

[0146]

[0147]

[0148] in addition The corresponding label is (0, 0);

[0149] Step 1-1-3: Image standardization, the standardization formula is as follows:

[0150]

[0151] Among them, pix is ​​the original pixel value, pixnorm is the normalized pixel value.

[0152] Step 1-2 includes the following steps:

[0153] Step 1-2-1: Sequence category labeling, making the echo category label sequence corresponding to the echo sequence:

[0154] According to the echo image category labels obtained in step 1-1-2, the category labels of all images in an echo sequence X are arranged in the order of the echo sequence to obtain the echo category label sequence X cl ;

[0155] Step 1-2-2: Sequence classification: Use K-Means clustering algorithm to classify the echo category label sequence X cl The sequences are clustered in two category dimensions, and the clustering results of the respective echo sequences are obtained in the two category dimensions. Then, a complete clustering result is obtained by combining the two clustering results. Among them, the sequence sample distance used in the K-Means clustering algorithm is the Euclidean distance. Among the 40,000 training data used, the unsupervised clustering results are as follows: there are 7 categories with intensity below 30dBZ, including 12,117 samples; there are 6 categories with intensity between 30 and 45dBZ, including 17,250 samples; there are 6 categories with intensity above 45dBZ, including 10,663 samples;

[0156] Step 1-2-3: Sequence category balanced sampling, perform category balanced sampling on the sequence clustering results, and the echo sequence sampling formula is as follows:

[0157]

[0158] Where N is the number of categories in the echo sequence data set, n j is the total number of samples of category j, p i is the probability of sampling an echo sequence from category i;

[0159] Step 1-2-4: Divide the sequence into historical and future segments, and set the number of historical frames and the number of future frames as the input length T of the SAB-STN neural network model respectively input = 10 and the extrapolated target length T target =20, then T input +T target Indicates that the length of the entire sequence is 30, and the echo sequence X is divided into historical input segments X input with the future extrapolation segment X target Two parts: front and back:

[0160]

[0161] where x irepresents the i-th frame image in the sequence (i=1, 2, ..., T input +T target );

[0162] Then the echo image size is reshaped (height height, width width) to (448, 448). At this time, the data shape (length, channel, height, width) of an echo sequence is (T input , 1, 448, 448), expressed as That is, the echo sequence X belongs to a high-dimensional real number space consisting of four dimensions: length L, number of channels C, height H, and width W.

[0163] Step 1-2-2 includes the following steps:

[0164] Step 1-2-2-1: K-Means cluster center selection, set the number of categories k = 5, select the initialized k echo category label sequence samples X cl As the initial cluster center cs:

[0165] cs=cs 1 ,cs 2 , ..., cs k ;

[0166] where cs 1 Refers to the first cluster center, and so on.

[0167] Step 1-2-2-2: Classify each sample. For each sample X cl Calculate its Euclidean distance dist to the k cluster centers and divide it into the class corresponding to the cluster center with the smallest distance:

[0168]

[0169] The arg min function returns the variable value that makes the function reach the minimum value. (i) represents the i-th sample X cl,i The class center cs that is closest to the k classes j , where j = 1, 2, ..k, the dist calculation formula is as follows:

[0170]

[0171] Among them, a and b represent different echo category label sequences with the same length. i , b i Respectively represent the echo label vectors of a and b at sequence position i;

[0172] Step 1-2-2-3: Update the centroid of the class. For each class, recalculate the centroid of all samples belonging to that class, i.e., the cluster center:

[0173]

[0174] where c i Indicates cs i The corresponding class (i=1, 2, ..k);

[0175] Step 1-2-2-4: Get the clustering result. Repeat steps 1-2-2-3 and 1-2-2-4 until convergence. At this time, determine the samples contained in each of the k classes.

[0176] Step 2 includes the following steps:

[0177] Step 2-1: Neural network model establishment: Figure 2 As shown, build each module of the SAB-STN model and initialize the model parameters;

[0178] Step 2-2: Initialize training hyperparameters: Use Adam optimizer to train the network, set learning rate lr = 0.001, batchsize = 4, and training set epoch = 20;

[0179] Step 2-3: Network parameter training and update: Use mean square error (MSE) as the loss function, take a batch size of data each time to input the network to train and optimize the network parameters, update the parameters through the back propagation algorithm, and initialize the model parameters using a combination of model pre-training and random initialization. The MSE loss function formula is as follows:

[0180]

[0181] Where pix i represents the i-th pixel in a real echo image, It indicates that the real echo image corresponds to the i-th pixel in the generated echo image, and 448×448 indicates the number of pixels in the image.

[0182] Step 2-1 includes the following steps:

[0183] Step 2-1-1: Encoder En construction: The encoder is a convolutional network used to convert the original echo image into a higher-level image representation;

[0184] Step 2-1-2: Echo Series Processor (ESP) construction: The sequence processor consists of two parts: ESP encoder ESP-En and ESP decoder ESP-De. The self-attention module SAB is the basic unit, which is used to learn the temporal correlation between the image representations at each moment generated by the encoder in an echo sequence, as well as the spatial correlation of the image representations at each moment;

[0185] Step 2-1-3: Construction of the decoder: The decoder is a convolutional network used to convert the features generated by the sequence processor into extrapolated results.

[0186] Step 2-1-4: Initialize model parameters: For the convolution kernel in the SAB-STN neural network model, its weight is initialized using the Kaiming normal distribution method, and its bias is initialized using the 0 value; for the standardization layer in the model, its weight is initialized using the 1 value, and its bias is initialized using the 0 value.

[0187] Step 2-1-1 includes the following steps:

[0188] Step 2-1-1-1: Encoder En includes three convolution modules Conv2d En，1 , Conv2d En,2 , Conv2d En,3 , two layers of maximum pooling layer MaxPool2d En，1 , MaxPool2d En,2 .

[0189] like Figure 3 As shown, the modules of the encoder En are arranged in series, and the order is Conv2d En,1 , MaxPool2d En,1 , Conv2d En，2 , MaxPool2d En,2 , Conv2d En，3 , where all convolution modules contain a convolution layer, a nonlinear activation function layer LeakyReLU, and Conv2d En,2 With Conv2d En，3 Each contains an additional normalization layer GroupNorm, and is placed before the convolutional layer. En，1 The input data size is (448,448), the convolution kernel size of the convolution layer is (3,3), the number of input channels is 1, the number of output channels is 4, the stride is 1, the zero padding is 1, the α parameter of LeakyReLU is 0.02, and the number of groups of GroupNorm is 1; MaxPool2d En,1The maximum pooling layer with a kernel size of 2, a stride of 2, and zero padding of 0 can reduce the size of the feature map by 1 / 2; Conv2d En,2 The input data size is (224, 224), the number of convolutional layer input channels is 4, the number of output channels is 8, and other parameter settings are the same as Conv2d En,1 Consistent; MaxPool2d En,2 With MaxPool2d En,1 Consistent; Conv2d En,3 The input data size is (112,112), the number of convolutional layer input channels is 8, the number of output channels is 16, and other parameter settings are the same as Conv2d En,1 In addition, the input data enters Conv2d En , 1 The 5-dimensional shape (batchsize = 4, length = 10, channel = 16, height = 112, width = 112) is reshaped to 4 dimensions (batchsize*length = 40, channel = 16, height = 112, width = 112), which is represented as

[0190] At this time, the input data with a shape of (batchsize, length, channel, height, width) = (4, 10, 1,448, 448) is processed by En to obtain an echo image representation with a shape of (40, 16, 112, 112).

[0191] Step 2-1-2 includes the following steps:

[0192] Step 2-1-2-1: Figure 4 As shown, the self-attention module SAB includes a self-attention layer SA, a nonlinear activation function layer GeLU, and a normalization layer GroupNorm, where the core of SAB lies in the self-attention layer SA;

[0193] In the SA layer, the input end simultaneously passes in the query item source and the key-value item source, and the query item source passes through the query convolution Conv2d Q Generate query item Q, key-value item source passes through key-value convolution Conv2d KV Generate key-value items KV, Conv2d Q , Conv2d KV Both are channel-wise two-dimensional convolutional layers, where Conv2d QThe input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is equal to the number of output channels and is 16, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d KV The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 32, the stride is 1, the zero padding is 1, the number of groups is 16, the tile side length value is regionsize=14, Q and KV are divided into several non-overlapping tiles with a side length of regionsize in the image height and width dimensions, and KV is divided into two parts along the channel dimension to obtain the key item K and the value item V. At this time, the shape of Q, K, and V is (batchsize,length,channel,height*width,patch) and patch represents the number of tiles, which is expressed as According to the self-attention mechanism, Q and K are used to calculate the attention weight score, that is:

[0194] score=softmax(similarity(Q,K)),

[0195] The function similarity is used to calculate the matching degree mr between Q and K. Here, Q and K are reshaped separately, and then the corresponding values ​​of each dimension are multiplied. The scale factor scale is used to control the scale of the result. At this time, the shape of the matching degree mr is (batchsize, length q ,length kv , channel, height*width, patch), the calculation formula is:

[0196]

[0197] The function softmax is used to convert the matching degree mr into length kv The dimension is converted into a weight value score with a sum of 1, expressed as:

[0198]

[0199] After obtaining the attention weight score, reshape V so that it is multiplied by the corresponding values ​​of each dimension of Score and kv The dimension summation realizes the weighted summation of attention and obtains the self-attention calculation result. The self-attention calculation Self-Attention(Q, K, V) with Q, K, V as input is expressed as:

[0200]

[0201] The self-attention calculation results need to restore and reshape the image blocks, and then process them through a layer of depth-separable convolution module SepConv. The network in SepConv is arranged in series, with a channel-by-channel convolution layer Conv2d in turn. SC1 And a point-by-point convolution layer Conv2d SC2 , where Conv2d SC1 The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 64, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d SC2 The input data size is (112, 112), the convolution kernel size is (1, 1), the number of input channels is 64, the number of output channels is 16, the stride is 1, and the zero padding is 0.

[0202] The modules in the entire SAB are arranged in series, namely GroupNorm, SA and GeLU. In addition, the output of SAB is the sum of the output of GroupNorm and the output of GeLU, that is, there is a residual connection between the two.

[0203] Step 2-1-2-2: Figure 5 As shown, the modules inside the ESP encoder ESP-En are serially spliced, including a time embedding layer TE (Time Embedding) and a multi-layer self-attention network ESP-En-M;

[0204] The time embedding layer TE does not contain trainable parameters. The time embedding te is generated based on the sine and cosine functions. The formula is as follows:

[0205]

[0206] Where t represents the tth time and i represents the i-th channel. The time embedding te generated by this formula has a shape of (batchsize = 4, length = 10, channel = 16), that is, It also needs to be replicated, expanded and reshaped in the height and width dimensions to make its shape become (batchsize*length=40,channel=16,height=112,width=112), that is The input of TE is the output of En in step 2-1-1, and the output of TE is the sum of the input of TE and te, and serves as the input of ESP-En-M, with a shape of (40, 16, 112, 112);

[0207] Multi-layer self-attention network ESP-En-M consists of num en_layer=4 SA modules are cascaded and spliced ​​together. Its input data only comes from the output data of TE and is also used as the query item source and key value item source of the network. The shape of its output data is (40, 16, 112, 112);

[0208] Step 2-1-2-3: Figure 6 As shown, the modules inside the ESP decoder ESP-De are serially spliced, including a time embedding layer TE and a multi-layer self-attention network ESP-De-M.

[0209] The TE structure of ESP-De is consistent with that of ESP-En, but in terms of input data, the data of the former is a full-zero tensor with a shape of (80, 16, 112, 112).

[0210] Multi-layer self-attention network ESP-De-M consists of num de_layer = 4 SA modules are cascaded together, but the difference from ESP-En-M is that: first, num de_layer must be an even number; second, if the first condition is met, num de_layer The SA modules are paired up in pairs. In a module pair, the query item source and key-value item source of the lower-level SA are both from the module one level lower than the current level, and the data shape is (80, 16, 112, 112). The query item source and key-value item source of the higher-level SA are the output of the lower-level SA and the output of ESP-En, and the corresponding data shapes are (80, 16, 112, 112) and (80, 16, 112, 112) respectively.

[0211] Step 2-1-3 includes the following steps:

[0212] Step 2-1-3-1: Figure 7 As shown in the figure, the modules in the decoder De are connected in series, with a total of three convolutional modules and two anti-pooling layers, namely Conv2d De，1 ,MaxUnpool2d De,1 , Conv2d De,2 ,MaxUnpool2d De,2 , Conv2d De,3 Among them, all convolution modules contain a normalization layer GroupNorm, a convolution layer, and Conv2d De,1 With Conv2d De，2 Each contains an additional nonlinear activation function layer LeakyReLU, which is placed after the convolution layer. De，1The input data size is (112, 112), the number of GroupNorm groups is 1, the convolution kernel size of the convolution layer is (3, 3), the number of input channels is 16, the number of output channels is 8, the stride is 1, the zero padding is 1, and the α parameter of LeakyReLU is 0.02; MaxUnpool2d De, 1 is the maximum unpooling layer with a kernel size of 2, a step size of 2, and zero padding of 0, which can enlarge the feature map size to 2 times; Conv2d De，2 The input data size is (224, 224), the number of convolutional layer input channels is 8, the number of output channels is 4, and other parameter settings are the same as Conv2d De,1 Consistent;MaxUnpool2d De，2 with MaxUnpool2d De,1 Consistent; Conv2d De,3 The input data size is (448, 448), the convolution kernel size of the convolution layer is (1, 1), the number of input channels is 4, the number of output channels is 1, the stride is 1, and the zero padding is 0. At this time, the shape of the output data is 4-dimensional (80, 1, 448, 448). The generation result of the entire self-attention spatiotemporal network SAB-STN needs to be reshaped to (4, 20, 448, 448, 1), and then the generation result is limited to the interval [-1, 1] by truncation.

[0213] Step 3 includes the following steps:

[0214] Step 3-1: Read test samples: Input the test set generated in step 1 into the neural network model generated in step 2;

[0215] Step 3-2: Forward propagation of the neural network model. After the test set is input into the neural network model, the extrapolated result is generated. Taking the prediction of the echo event of mixed cloud precipitation as an example, the result is as follows: Fig. 9 As shown, the first 10 frames are input data, and the last 20 frames are extrapolated data;

[0216] The most commonly used data in radar echo extrapolation is the single-channel PPI combined reflectivity within 2 hours. The pixel grid on the map represents the projection of the real geographic horizontal space at a certain resolution, and the pixel value represents the mapping of the echo intensity. Through an echo sequence, the movement change process and possible movement trend of the weather phenomenon represented by the local echo can be observed more accurately. If a sufficient number of echo sequences of the entire process of a certain type of weather phenomenon are collected, the early stage of the weather phenomenon can be identified by statistical methods, and then this echo sequence can be used as the basis for extrapolation to analyze the potential movement changes, and then obtain the possible echo state in the future time; when the specified time interval passes, a new echo sequence needs to be obtained to determine the current weather change stage, and then extrapolate. Studies have shown that deep neural networks have excellent data fitting capabilities, making them a statistical method for extrapolation. Specifically, deep neural networks can not only fit the basic horizontal movement of echoes and echo intensity changes in the potential space, but also fit the echo movement unique to different weather phenomena. In other words, deep neural networks can identify the most likely change stage of what weather phenomenon the process belongs to based on the known echo movement process, and then make extrapolated predictions.

[0217] At present, there are echo extrapolation methods based on neural networks such as CNN, RNN and cGAN, but these neural networks lack the ability to learn the spatiotemporal correlation of echo sequences from the entire time domain in principle. Compared with other networks, the self-attention network can learn the echo motion characteristics of any horizontal space on the echo image from the entire echo sequence time period. Therefore, the present invention focuses on the self-attention mechanism and combines the characteristics of echo sequence data to propose a SAB-STN network suitable for echo extrapolation.

[0218] The present invention provides a radar echo extrapolation method based on a self-attention spatiotemporal neural network model. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A radar echo extrapolation method based on a self-attention spatiotemporal neural network model, characterized in that: The following steps are involved: Step 1, data preprocessing: perform data preprocessing on the PPI combined reflectivity radar echo products that have been quality controlled by correlation coefficients, divide each fixed-length echo sequence into historical segments and future segments, and obtain training sets and test sets; Step 2, training the neural network model: input the echo sequence data in the training set into the SAB-STN neural network model for training, and save the neural network model parameters after the training is completed; Step 3, testing the neural network model: inputting the historical echo sequence data in the test set into the neural network model trained in step 2 to test the neural network model, and outputting the prediction results extrapolated from the historical echoes to the future; Step 2 includes the following steps: Step 2-1: Establish a neural network model: build each module of the neural network model and initialize model parameters; Step 2-2: Initialize training hyperparameters: Use Adam optimizer to train the neural network model, set the learning rate lr, the number of samples contained in a group of samples input each time batchsize and the number of epochs for training set traversal; Step 2-3: Neural network model parameter training and update: Use mean square error (MSE) as the loss function, take a batch size of data each time to input the neural network model to train and optimize the neural network model parameters, update the parameters through the back propagation algorithm, and initialize the model parameters using a combination of model pre-training and random initialization. The loss function MSE calculation formula is as follows: Where pix i represents the i-th pixel in a real echo image, Indicates that the real echo image corresponds to the i-th pixel in the generated echo image, and height×width indicates the number of pixels in the image; Step 2-1 includes the following steps: Step 2-1-1: Construct encoder En: The encoder is a convolutional neural network model, which is used to convert the original echo image into an abstract image representation; Step 2-1-2: Construct an echo sequence processor ESP: The sequence processor includes two parts: an ESP encoder ESP-En and an ESP decoder ESP-De. The self-attention module SAB is used as a basic unit to learn the temporal correlation between the image representations at each moment generated by the encoder in an echo sequence, as well as the spatial correlation of the image representations at each moment; Step 2-1-3: Construct a decoder De: The decoder is a convolutional neural network model, which is used to convert the features generated by the sequence processor into extrapolated results; Step 2-1-4: Initialize model parameters: For the convolution kernel in the SAB-STN neural network model, the weights are initialized using the Kaiming normal distribution method, and the bias is initialized using the 0 value; for the standardization layer in the model, the weights are initialized using the 1 value, and the bias is initialized using the 0 value.

2. The method according to claim 1, characterized in that Step 1 includes the following steps: Step 1-1: echo image preprocessing: preprocess all echo images in the data set; Step 1-2: echo sequence preprocessing; Step 1-3: Divide the data set: Divide the preprocessed echo sequence into a test set and a training set.

3. The method according to claim 2, characterized in that Step 1-1 includes the following steps: Step 1-1-1: Image denoising: removing suspicious non-precipitation echoes from the echo image, including the following steps: Step 1-1-1-1: Convert the original single channel grayscale value echo image x raw The pixel value pix is ​​converted into the echo intensity value, and the calculation formula is as follows: Step 1-1-1-2, set the minimum threshold I of precipitation echo intensity according to the data source th , according to the minimum threshold I th The echo image is subjected to a first noise reduction with the echo intensity value I obtained in step 1-1-1-1. The first noise reduction is defined as follows: for an echo intensity less than the minimum threshold value I th The pixels that are detected and the pixels that are missing are NaN, and their values ​​are reset to zero. In other cases, the pixel values ​​remain unchanged. The formula is as follows: After the first noise reduction is completed, a single-channel echo image x is obtained without the lower echo intensity. raw ; Step 1-1-1-3: The echo image x obtained by denoising once raw Perform connected domain analysis, obtain the area of ​​each connected domain in each image, and set the minimum threshold of the connected domain area ccA th , according to ccA th The echo image X is subjected to secondary denoising with ccA. The secondary denoising is specified as follows: For an area smaller than ccA th The connected domain, the pixel value in the connected domain is reset to zero, otherwise the pixel value remains unchanged. The formula is as follows: After the secondary denoising is completed, a single-channel echo image x without small-area clutter is obtained; Step 1-1-2: Image classification, including the following steps: Step 1-1-2-1: Count the number of pixels num whose pixel values ​​are not 0 in each denoised echo image x pix > 0, calculate the number of pixels num in the entire observation domain ob The percentage of the echo area to the observable area The formula is as follows: according to Is it 0? First, classify X into two categories, where: represents the echo image without echo after noise reduction, Indicates the echo image with echo after noise reduction, set the area ratio interval set And according to of The area ratio interval will Divided into 4 categories; Step 1-1-2-2: Set the echo intensity level set θ I : I I ={(0,20],(20,30],(30,40],(40,70]}, Statistics for each The number of pixels with the same pixel value at each echo level Calculation and num pix >0, that is, the area ratio of the echo grade area to the echo area P i , the formula is as follows: According to the last P that is not 0 after sorting in order i The corresponding i will Divide into 4 categories, combined with the Divided into 4 categories, The final classification is 16 categories, plus Then x is divided into 17 categories; The corresponding label is (0,0); Step 1-1-3: Perform image standardization, the formula is as follows: Among them, pix is ​​the original pixel value, pix norm is the normalized pixel value.

4. The method according to claim 3, characterized in that Step 1-2 includes the following steps: Step 1-2-1: Label the sequence category labels and create an echo category label sequence corresponding to the echo sequence: According to the echo image category labels obtained in step 1-1-2, the category labels of all images in an echo sequence X are arranged in the order of the echo sequence to obtain the echo category label sequence X cl ; Step 1-2-2: Use the K-Means clustering algorithm to cluster the echo category label sequence X cl The sequence is clustered in two category dimensions, and the respective echo sequence clustering results are obtained in the two category dimensions. Then, the two types of clustering results are combined to obtain a complete clustering result. Among them, the sequence sample distance used in the K-Means clustering algorithm is the Euclidean distance; Step 1-2-3: Perform category balanced sampling on the sequence clustering results. The echo sequence sampling formula is as follows: Where N is the number of categories in the echo sequence data set, n j is the total number of samples of category j, p i is the probability of sampling an echo sequence from category i; Step 1-2-4: Divide the sequence into historical and future segments, and set the historical frame number and future frame number as the input length T of the SAB-STN neural network model respectively input With the extrapolated target length T target , then T input +T target Indicates the length of the entire sequence, dividing the echo sequence X into historical input segments X input with the future extrapolation segment X target Two parts: front and back: where x i Represents the i-th frame image in the sequence; Reshape the echo image size: (height, width) is reshaped to (448,448), where height represents the height and width represents the width. At this time, the data shape of an echo sequence (length, channel, height, width) is (T input , 1,448,448), represented as an echo sequence It represents a high-dimensional real number space consisting of four dimensions: length L, number of channels C, height H, and width W. Length represents the length, and channel represents the number of channels.

5. The method according to claim 4, characterized in that Step 1-2-2 includes the following steps: Step 1-2-2-1: Select the K-Means cluster center, set the number of categories k, and select the initialized k echo category label sequence samples X cl As the initial cluster center cs: cs=cs1, cs2,..., cs k ; where cs k Refers to the kth cluster center; Step 1-2-2-2: Classify each sample: For each sample X cl Calculate its Euclidean distance dist to the k cluster centers and divide it into the class corresponding to the cluster center with the smallest distance: The arg min function returns the variable value that makes the function reach the minimum value. (i) represents the i-th sample X cl，i The cluster center cs that is closest to the k clusters j , where j = 1, 2, ... k, the calculation formula of the function dist is as follows: Among them, a and b represent different echo category label sequences with the same length. i , b i Respectively represent the echo label vectors of a and b at sequence position i; Step 1-2-2-3: Update the centroid of the class: For each category, recalculate the centroid of all samples belonging to that category, i.e., the cluster center: where c i Indicates cs i The corresponding class, i=1,2,..k; Step 1-2-2-4: Obtain clustering results: Repeat steps 1-2-2-3 until convergence, at which time determine the samples contained in each of the k classes.

6. The method according to claim 5, characterized in that Step 2-1-1 includes the following steps: Step 2-1-1-1: Encoder En includes three convolution modules Conv2d En，1 , Conv2d En，2 , Conv2d En，3 , two layers of maximum pooling layer MaxPool2d En，1 , MaxPool2d En，2 ; The modules are arranged in series, the order is Conv2d En，1 , MaxPool2d En，1 , Conv2d En，2 , MaxPool2d En，2 , Conv2d En，3 , where all convolution modules contain a convolution layer, a nonlinear activation function layer LeakyReLU, and Conv2d En，2 With Conv2d En，3 Each contains an additional normalization layer GroupNorm, which is placed before the convolutional layer; Conv2d En，1 The input data size is (448,448), the convolution kernel size of the convolution layer is (3,3), the number of input channels is 1, the number of output channels is 4, the stride is 1, the zero padding is 1, the α parameter of LeakyReLU is 0.02, and the number of groups of GroupNorm is 1; MaxPool2d En，1 Conv2d is the maximum pooling layer with a kernel size of 2, a stride of 2, and zero padding of 0, which can reduce the size of the feature map by 1 / 2; En，2 The input data size is (224, 224), the number of convolutional layer input channels is 4, and the number of output channels is 8; MaxPool2d En，2 With MaxPool2d En，1 Consistent; Conv2d En，3 The input data size is (112,112), the number of convolutional layer input channels is 8, and the number of output channels is 16; the input data enters Conv2d En，1 The 5-dimensional shape (batchsize, length, channel, height, width) is reshaped to 4-dimensional (batchsize*length, channel, height, width), which is expressed as At this time, the shape is (batchsize,length,channel,height,width)=(batchsize,T input ,1,448,448) input data is processed by the encoder En to obtain a shape of (batchsize*T input ,16,112,112)’s echo image representation.

7. The method according to claim 6, characterized in that Step 2-1-2 includes the following steps: Step 2-1-2-1: The self-attention module SAB includes a self-attention layer SA, a nonlinear activation function layer GeLU, and a normalization layer GroupNorm; In the self-attention layer SA, the input end simultaneously passes in the query item source and the key-value item source, and the query item source passes through the query convolution Conv2d Q Generate query item Q, key-value item source passes through key-value convolution Conv2d KV Generate key-value items KV, Conv2d Q , Conv2d KV Both are channel-wise two-dimensional convolutional layers, where Conv2d Q The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is equal to the number of output channels and is 16, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d KV The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 32, the stride is 1, the zero padding is 1, the number of groups is 16, the tile side length is regionsize, Q and KV are divided into non-overlapping tiles with a side length of regionsize in the image height and width dimensions, and KV is divided into two parts along the channel dimension to obtain the key item K and the value item V. At this time, the shapes of Q, K, and V are (batchsize,length,channel,height*width,patch), where patch represents the number of tiles, which is expressed as According to the self-attention mechanism, Q and K are used to calculate the attention weight score: score=softmax(similarity(Q,K)), The function similarity is used to calculate the matching degree mr between Q and K. Here, Q and K are reshaped separately, and then the corresponding values ​​of each dimension are multiplied, and the scale factor scale is used to control the scale of the result. At this time, the shape of the matching degree mr is (batchsize, length q ,length kv ,channel,height*width,patch), the calculation formula is: The function softmax is used to convert the matching degree mr into length kv The dimension is converted into a weight value score with a sum of 1, expressed as: After obtaining the attention weight score, reshape V so that V and score are multiplied by the corresponding values ​​of each dimension and along the length kv The dimension summation realizes the weighted summation of attention and obtains the self-attention calculation result. The self-attention calculation Self-Attention(Q, K, V) with Q, K, V as input is expressed as: The self-attention calculation results need to restore and reshape the blocks, and then process them through a layer of depth-separable convolution module SepConv; the network in SepConv is arranged in series, and each layer is a channel-by-channel convolution layer Conv2d. SC1 And a point-by-point convolution layer Conv2d SC2 , where Conv2d SC1 The input data size is (112,112), the convolution kernel size is (3,3), the number of input channels is 16, the number of output channels is 64, the step size is 1, the zero padding is 1, the number of groups is 16, and Conv2d SC2 The input data size is (112,112), the convolution kernel size is (1,1), the number of input channels is 64, the number of output channels is 16, the stride is 1, and the zero padding is 0; The modules in the entire self-attention module SAB are arranged in series, namely GroupNorm, SA and GeLU. In addition, the output of SAB is the sum of the output of GroupNorm and the output of GeLU; Step 2-1-2-2: The modules inside the ESP encoder ESP-En are serially spliced, including a time embedding layer TE and a multi-layer self-attention network ESP-En-M; The time embedding layer TE does not contain trainable parameters. The time embedding te is generated based on the sine and cosine functions. The formula is as follows: Among them, t represents the tth time, i represents the i-th channel; the time embedding te shape is (batchsize, length, channel), that is, It is also necessary to copy, expand and reshape the time embedding te in the height and width dimensions so that its shape becomes (batchsize*length,channel,height,width), that is, The input of the temporal embedding layer TE is the output of the encoder En in step 2-1-1, and the output of the temporal embedding layer TE is the sum of the input of the temporal embedding layer TE and te, and serves as the input of the multi-layer self-attention network ESP-En-M, with a shape of (batchsize*T input ,16,112,112); Multi-layer self-attention network ESP-En-M consists of num en_layer The input data is only the output data from the time embedding layer TE and it also serves as the query item source and key-value item source of the network. The shape of the output data is (batchsize*T input ,16,112,112); Step 2-1-2-3: The modules inside the ESP decoder ESP-De are serially spliced, including a time embedding layer TE and a multi-layer self-attention network ESP-De-M; The time embedding layer TE structure of ESP decoder ESP-De is consistent with TE of ESP-En. The input data of ESP decoder ESP-De is in the shape of (batchsize*T target ,16,112,112) tensor of all 0s; Multi-layer self-attention network ESP-De-M consists of num de_layer SA modules are cascaded and spliced ​​together, num de_layer is an even number, and num de_layer The SA modules are paired in pairs. In a module pair, the query item source and key-value item source of the lower-level SA are both from the module one layer lower than the layer. The data shape is (batchsize*T target ,16,112,112), while the query item source and key-value item source of the higher-level SA are the output of the lower-level SA and the output of ESP-En, and the corresponding data shapes are (batchsize*T target ,16,112,112) and (batchsize*T input ,16,112,112).

8. The method according to claim 7, characterized in that Step 2-1-3 includes the following steps: Step 2-1-3-1: The modules in the decoder De are connected in series. There are three convolution modules and two anti-pooling layers, namely Conv2d De，1 ,MaxUnpool2d De，1 , Conv2d De，2 ,MaxUnpool2d De，2 , Conv2d De，3 ; Among them, all convolution modules contain a normalization layer GroupNorm, a convolution layer, and Conv2d De，1 With Conv2d De，2 Each contains an additional nonlinear activation function layer LeakyReLU, which is placed after the convolution layer; Conv2d De，1 The input data size is (112,112), the number of GroupNorm groups is 1, the convolution kernel size of the convolution layer is (3,3), the number of input channels is 16, the number of output channels is 8, the stride is 1, the zero padding is 1, and the α parameter of LeakyReLU is 0.02; MaxUnpool2d De，1 The maximum unpooling layer with a kernel size of 2, a step size of 2, and zero padding of 0 can enlarge the feature map size to 2 times; Conv2d De，2 The input data size is (224, 224), the number of convolutional layer input channels is 8, and the number of output channels is 4; MaxUnpoo12d De，2 with MaxUnpool2d De，1 Consistent; Conv2d De，3 The input data size is (448,448), the convolution kernel size of the convolution layer is (1,1), the number of input channels is 4, the number of output channels is 1, the step size is 1, and the zero padding is 0; the shape of the output data is 4-dimensional (batchsize*T target ,1,448,448), the generation result of the entire self-attention spatiotemporal network SAB-STN needs to be reshaped to (batchsize,T target ,448,448,1), and then the generated result is limited to the interval [-1,1] by truncation operation; Step 3 includes the following steps: Step 3-1: Read test samples: Input the test set generated in step 1 into the neural network model generated in step 2; Step 3-2: Forward propagation of the neural network model. After the test set is input into the neural network model, the extrapolated results are generated.

Images