Radar echo extrapolation method, device and system
By adopting ST-LSTM recurrent neural network based on 3D convolution and time channel attention in radar echo extrapolation, the problem of fuzzy and insufficient accuracy in the existing technology is solved, and higher accuracy in short-term precipitation prediction is achieved sex and image clarity.
Patent Information
- Application Number
- CN202210818471.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-07-13
AI Technical Summary
The existing radar echo extrapolation method is not effective in long-term tasks, and has extrapolation fuzzy problems, making it difficult to meet the accuracy and real-time requirements of short-term precipitation prediction.
Using ST-LSTM recurrent neural network improved based on 3D convolution and time channel attention mechanism, three-dimensional data is generated by processing two-dimensional radar echo data, extracting spatiotemporal and spatial feature information, and weighting feature information on the time dimension to inverse it Reconstruct to improve external replication accuracy.
It significantly improves the accuracy of short-term precipitation prediction, alleviates the extrapolation blur problem, can effectively capture the short-term dynamic feature information of radar echoes, and improves extrapolation accuracy and image clarity.
Smart Images

Figure CN115113165B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of radar echo extrapolation, and relates to a radar echo extrapolation method, device and system, and more specifically to a radar echo extrapolation method, device and system based on an improved ST-LSTM recurrent neural network with 3D convolution operation and time channel attention mechanism. Background Art
[0002] Short-term forecasts usually refer to short-term weather forecasts within 12 hours and near-term weather forecasts within 2 hours. Short-term forecasts focus on small and medium-scale weather systems with strong suddenness, short life cycles, and rapid evolution, especially severe convective weather systems. Compared with long-term precipitation forecasts, it has higher requirements in terms of accuracy and real-time performance. Accurately predicting short-term precipitation can assist people in assessing road waterlogging, guiding traffic, and improving the accuracy of urban heavy precipitation warnings, which is of great significance for disaster prevention and mitigation.
[0003] Radar echo extrapolation is a common method for short-term precipitation forecasting. Compared with traditional numerical weather forecasting methods, it can achieve faster and more accurate forecasts. Its principle is to use the echo data detected by the radar to determine the intensity distribution of the echo and the moving speed and direction of the echo body, and to predict the radar echo state after a certain period of time by linear or nonlinear extrapolation of the echo body. Traditional radar echo extrapolation techniques based on morphology mainly include the cross-correlation method and the monomer centroid method. The cross-correlation method selects the spatial optimization correlation coefficients of two consecutive time periods to establish a fitting relationship, which has low data utilization. For echo images with large velocity gradients and fast evolution, the inversion error is large, and it is difficult to deduce the actual weather conditions. The centroid tracking method can achieve good results in stable precipitation forecasting, but in local convective weather, the echo evolves rapidly and cannot meet the conservation conditions, and the forecast effect will decline rapidly with time. The optical flow method was proposed by Gibson. The optical flow method calculates the optical flow field from a continuous image sequence, and uses the changes in pixels in the image sequence in the time domain and the correlation between images of adjacent frames to establish the corresponding relationship between the previous frames, and then calculates the motion information of the radar echo between images of adjacent frames. The optical flow method can capture the motion and change information of the radar echo, but does not make full use of the echo image information over a longer period of time and the forecast has a lag, which cannot meet the real-time forecast needs.
[0004] In the context of meteorological big data, deep learning has been widely used in the field of meteorology. Deep learning methods can be used to find potential complex relationships in a large amount of data and learn relatively concentrated feature representation patterns, which is more conducive to making accurate forecasts of meteorological elements in a specified area in the future time period. Shi et al. proposed a convolutional long short-term memory (ConvLSTM) neural network structure based on LSTM, which achieved higher prediction accuracy than the optical flow method. Since then, many improved variant structures have been developed based on Conv-LSTM. For example, Shi Xingjian introduced the idea of optical flow trajectory and proposed TrajectoryGRU (TrajGRU). This model can actively learn the position change structure for repeated connections, further improving the accuracy of forecasts. Villegas et al. combined the Encoder-Decoder structure with ConvLSTM to establish a motion and content decomposition model. The model simplifies the prediction task, effectively handles the complex evolution of pixels in the video, and can independently capture the spatial structure of the image and the corresponding temporal dynamics. Wang introduced the spatial memory unit (ST-LSTM) in ConvLSTM and proposed PredRNN and PredRNN++, which enhance the ability to capture short-term dynamic changes by using a larger convolutional neural receptive field. Wang Yunbo et al. proposed the MIM model to learn high-level nonlinear spatiotemporal dynamic information. Lin et al. proposed a self-attention mechanism ConvLSTM, which effectively captures long-term spatial dependencies. Xu et al. combined the generation ability of GAN with the prediction ability of LSTM network and proposed a GAN-LSTM model for spatiotemporal sequence prediction. Compared with traditional methods, the radar echo extrapolation method based on deep learning has the advantages of high data utilization efficiency and high prediction accuracy.
[0005] Compared with traditional methods, the use of deep learning for echo extrapolation has the advantages of high data utilization and accurate prediction accuracy, but there is still an obvious extrapolation ambiguity problem, and the effect is not good in long-term radar echo extrapolation tasks. Summary of the invention
[0006] In view of the above problems, the present invention proposes a radar echo extrapolation method, device and system, which have stronger characterization capabilities and more accurate extrapolation results.
[0007] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:
[0008] In a first aspect, the present invention provides a radar echo extrapolation method, comprising:
[0009] Processing the acquired two-dimensional radar echo data to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time;
[0010] Using a feature extractor constructed based on 3D convolution, the spatiotemporal feature information in the three-dimensional radar echo data is extracted to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information;
[0011] Inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times;
[0012] Inputting the high-level abstract feature information output at different times into a channel attention module based on a time channel to obtain feature information weighted on the time channel;
[0013] Based on the weighted feature information on the time channel, reverse reconstruction is performed to restore the temporal and spatial information contained in the feature information and obtain the final radar echo extrapolation result.
[0014] Optionally, the method for generating three-dimensional radar echo data includes:
[0015] Use a sliding window with a length of M and a step size of S to preprocess the acquired two-dimensional radar echo data;
[0016] The continuous M frames of images in each sliding window are spliced in turn in the time dimension as the input data at each moment to obtain the three-dimensional radar echo data.
[0017] Optionally, the feature extractor includes multiple layers of 2D convolutional networks arranged in sequence, and a layer of 3D convolutional network.
[0018] Optionally, the method for acquiring the three-dimensional feature map includes:
[0019] Using the multi-layer 2D convolutional network to perform a convolution operation on each time channel of the three-dimensional radar echo data to obtain an initial three-dimensional feature map;
[0020] The one-layer 3D convolutional network is used to extract spatiotemporal feature information from the initial three-dimensional feature map to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information.
[0021] Optionally, the recurrent neural network is constructed by the following method:
[0022] Get the standard ST-LSTM recurrent unit;
[0023] The ordinary convolution in the standard ST-LSTM recurrent unit is replaced by 3D convolution.
[0024] Optionally, the expression of the recurrent neural network includes:
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035] Among them, tanh is the tanh function, σ is the sigmoid function; * is the 3D convolution operation; ⊙ is the Hadamard product; + represents matrix addition, X t is the three-dimensional feature map output by the feature extractor at time step t; is the input information, which is the hidden state of layer l from time step t-1; is the input information, which is the memory state from the t-1 time step; is the input information, which is the spatial memory state from the l-1 layer at time step t; g t and g′ t The fused input information is obtained by performing 3D convolution operation on each input information by the time memory unit and the space memory unit at time step t; i t and i t ′ are the input gates obtained after the time memory unit and the spatial memory unit at time step t perform 3D convolution operations on each input information. The input gate is used to control how much information at the current moment is written into the memory unit; f t and f′ t The forget gates are obtained by performing 3D convolution operations on each input information by the time memory unit and the space memory unit at time step t. The forget gate is used to control how much information of the previous moment is retained; and They are respectively the temporal memory state and spatial memory state of layer l at time step t; t is the output gate, responsible for updating the hidden state of time step t W ijrepresents the learnable weight parameters in the convolution kernel that performs the convolution operation on the input information i in the gated unit. The subscripts x, h, c, and m are used to identify the input information X, hidden state information H, temporal memory state information C, and spatial state memory information M, respectively. The subscripts g, i, f, and o are used to identify the input modulation gate, input gate, forget gate, and output gate, respectively. 1×1 represents the learnable parameters corresponding to the 1×1 convolution kernel; b j is the bias term in gating unit j.
[0036] Optionally, the expression of the channel attention module of the time channel is:
[0037] ω=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0038]
[0039] Among them, σ is the sigmoid function, MLP is the multi-layer perceptron, is the product of the corresponding elements of the matrix, AvgPool(F) is the spatial global average pooling of F, and MaxPool(F) is the spatial global maximum pooling of F; F is the three-dimensional feature information obtained by stacking the high-level abstract feature information at different times output by the recurrent neural network on the time channel. The three dimensions are the width of the feature information, the height of the feature information, and the number of channels of the feature information. ω is the weight matrix, and each channel of ω represents a special detector, which is used to determine which moments of information have an important influence on the radar echo extrapolation result and give it a higher weight; F′ is the weighted feature information obtained by multiplying the weight matrix ω with the three-dimensional feature information F on the time channel.
[0040] Optionally, the method for obtaining the final radar echo extrapolation result includes:
[0041] The feature information weighted on the time channel is input into the reverse reconstruction module, which is a single-layer 3D convolutional network, the number of channels of its convolution kernel is the number of channels of the weighted feature information, and the number of convolution kernels is the number of extrapolated time steps;
[0042] The reverse reconstruction module performs convolution operation based on the received data, restores the time and space information contained in the weighted feature information, and obtains the final radar echo extrapolation result.
[0043] In a second aspect, the present invention provides a radar echo extrapolation device, comprising: a data processing module, a feature extractor, a recurrent neural network module, a channel attention module and a reverse reconstruction module arranged in sequence;
[0044] The acquired two-dimensional radar echo data is processed by the data processing module to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time;
[0045] Using a feature extractor constructed based on 3D convolution, the spatiotemporal feature information in the three-dimensional radar echo data is extracted to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information;
[0046] Inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times;
[0047] Inputting the high-level abstract feature information output at different times into a channel attention module based on a time channel to obtain feature information weighted on the time channel;
[0048] The weighted feature information on the time channel is input into the reverse reconstruction module for reverse reconstruction to restore the temporal and spatial information contained in the feature information and obtain the final radar echo extrapolation result.
[0049] In a third aspect, the present invention provides a radar echo extrapolation system, including a storage medium and a processor;
[0050] The storage medium is used to store instructions;
[0051] The processor is configured to operate according to the instructions to execute the steps of the method according to any one of the first aspects.
[0052] Compared with the prior art, the present invention has the following beneficial effects:
[0053] The radar echo extrapolation method, device and system proposed in the present invention can significantly improve the accuracy of short-term precipitation prediction, while effectively solving the extrapolation ambiguity problem, and have a certain extrapolation capability for highly nonlinear and random sequence images.
[0054] The present invention first uses 2D convolution combined with 3D convolution layer to extract short-term dynamic feature information of radar echo data. Then, the short-term dynamic feature information obtained in the previous step is input into the recurrent neural network, the spatiotemporal information is modeled, and spliced in the time dimension to obtain high-level feature information. The attention mechanism based on the time channel is used to extract important feature information, and different weights are given to feature information at different times to suppress secondary information, which alleviates the extrapolation ambiguity problem to a certain extent and improves the extrapolation accuracy.
[0055] The biggest obstacle to the spatiotemporal sequence prediction task is how to accurately model the image evolution in an unsupervised manner to learn complex spatiotemporal relationships, which requires the model to not only have strong spatial information extraction capabilities, but also to have temporal information fusion capabilities, and to alleviate the problem of old information loss during information flow transmission to a certain extent. To this end, the present invention uses an attention mechanism based on a time channel and an ST-LSTM recurrent neural network integrated with 3D convolution to solve the above problems. The ST-LSTM recurrent neural network integrated with 3D convolution effectively extracts semantics from images in an unsupervised manner, makes full use of all input image features, and has the ability to learn temporal dependencies and spatial information in long-term information flows. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments and in conjunction with the accompanying drawings, wherein:
[0057] Figure 1 The present invention is an overall flow chart of a radar echo extrapolation method according to an embodiment of the present invention;
[0058] Figure 2 A schematic diagram of using a sliding window to process data;
[0059] Figure 3(a) is a schematic diagram of the structure of a standard ST-LSTM recurrent unit;
[0060] Figure 3(b) is a schematic diagram of the structure of the ST-LSTM recurrent unit with integrated 3D convolution;
[0061] Figure 4 The figure is a schematic diagram of the overall structure of a radar echo extrapolation device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.
[0063] The application principle of the present invention is described in detail below in conjunction with the accompanying drawings.
[0064] Example 1
[0065] The present invention provides a radar echo extrapolation method, comprising the following steps:
[0066] Step (1) processes the acquired two-dimensional radar echo data to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time;
[0067] Step (2) using a feature extractor constructed based on 3D convolution to extract spatiotemporal feature information from the three-dimensional radar echo data to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information;
[0068] Step (3) inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times;
[0069] Step (4) inputting the high-level abstract feature information output at different times into a channel attention module based on the time channel to obtain feature information weighted on the time channel;
[0070] Step (5) performs reverse reconstruction based on the weighted feature information on the time channel to restore the spatiotemporal information contained in the feature information. The MSE index is used to determine whether the extrapolated image quality is qualified. If qualified, the final radar echo extrapolation result is obtained. Otherwise, the model training continues.
[0071] The method in the embodiment of the present invention can improve the ability to capture short-term dynamic feature information in the radar echo extrapolation task, extract important feature information, and alleviate the extrapolation ambiguity problem as the prediction time increases.
[0072] Radar echo extrapolation is essentially a spatiotemporal sequence prediction problem. Radar echo data contains rich spatiotemporal information. In order to fully extract spatiotemporal information in subsequent steps and enhance the representation ability of the model, the original data needs to be processed first. In a specific implementation of an embodiment of the present invention, the method for generating three-dimensional radar echo data includes:
[0073] Use a sliding window with a length of M and a step size of S to preprocess the acquired two-dimensional radar echo data;
[0074] The continuous M frames of images in each sliding window are sequentially spliced in the time dimension as input data at each moment to obtain three-dimensional radar echo data. In the specific implementation process, the following specific steps can be used to perform:
[0075] First, a sliding window with a length of M and a step size of S is used to preprocess the N consecutive frames of radar image data. Then, the consecutive M frames of images in each sliding window are sequentially spliced in the time dimension as input data at a moment. The input data has three dimensions, each of which represents width, height, and time. Figure 2 The process of processing a training sample (N=5 frames) using a sliding window with a length of M=3 and a step size of S=1 is demonstrated. Each time, three consecutive frames of images in the sliding window are spliced in the time dimension to obtain a three-dimensional input data. The original 5 frames of two-dimensional data are converted into 3 three-dimensional data after sliding window processing.
[0076] Since the three-dimensional radar echo data obtained in step (1) is specifically three-dimensional radar echo image sequence data, it not only has time dependence, but also has spatial dynamic information. Ordinary convolution operations can only extract local static feature information of a single radar echo image, and it is difficult to effectively capture the effective evolution trend of the radar echo contained in the continuous radar echo image. In addition, the receptive field of a single-layer convolutional network is limited, and it is easy to ignore the spatial information of the entire image. It is usually necessary to use a stacked multi-layer convolutional network to expand the receptive field. To this end, in a specific implementation of an embodiment of the present invention, considering that the amount of training parameters of 3D convolution is much larger than that of 2D convolution, simply stacking 3D convolutions will make the model difficult to fit. Therefore, the embodiment of the present invention proposes to construct a feature extractor based on 2D convolution and 3D convolution, and the constructed feature extractor includes multiple layers of 2D convolutional networks arranged in sequence, and a layer of 3D convolutional network; the method for obtaining the three-dimensional feature map includes:
[0077] Using the multi-layer 2D convolutional network to perform a convolution operation on each time channel of the three-dimensional radar echo data to obtain an initial three-dimensional feature map;
[0078] The one-layer 3D convolutional network is used to extract spatiotemporal feature information from the initial three-dimensional feature map to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information.
[0079] After the three-dimensional radar echo data is processed by the feature extractor in step (2), a three-dimensional feature map is obtained. Compared with ordinary input data, the three-dimensional feature map has an additional time dimension, so the internal structure of the ST-LSTM cyclic unit also needs to be modified to be compatible with the size of the input data. According to the characteristic that the input data at a moment has three dimensions, in the embodiment of the present invention, the ordinary convolution (conv) inside the standard ST-LSTM cyclic unit is replaced with a 3D convolution (3D conv) to extract the short-term dynamic information between the data at adjacent moments. The result of the standard ST-LSTM cyclic unit is the prior art, and no further explanation is given in the embodiment of the present invention. The original standard ST-LSTM cyclic unit structure is shown in Figure 3 (a), and the structure of the ST-LSTM cyclic unit integrated with 3D convolution proposed in the embodiment of the present invention is shown in Figure 3 (b). The main structures of the two are the same, and the difference is that Figure 3 (b) uses 3D convolution to replace the ordinary convolution inside the standard ST-LSTM cyclic unit.
[0080] The ST-LSTM recurrent unit shown in Figure 3(b) contains two memory units, namely the time memory unit and spatial memory units The part marked by the left box inside the ST-LSTM recurrent unit is the time memory unit used to control the horizontal transmission and update of time information; the part marked by the right box is the spatial memory unit used to control the vertical transmission and update of spatial information. The transmission control of time information and spatial information is respectively controlled by the input gates i t and i′ t , forget gate f t and f′ t Implementation. The ST-LSTM recurrent unit with integrated 3D convolution can be expressed by formulas (1)-(10).
[0081]
[0082]
[0083]
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
[0090]
[0091] Among them, tanh is the tanh function, σ is the sigmoid function; * is the 3D convolution operation; ⊙ is the Hadamard product; + represents matrix addition, X t is the three-dimensional feature map output by the feature extractor at time step t; is the input information, which is the hidden state of layer l from time step t-1; is the input information, which is the memory state from the t-1 time step; is the input information, which is the spatial memory state from the l-1 layer at time step t; g t and g′ t The fused input information is obtained by performing 3D convolution operation on each input information by the time memory unit and the space memory unit at time step t; i t and i′ t The input gates are obtained by performing 3D convolution operations on each input information by the time memory unit and the spatial memory unit at time step t. The input gate is used to control how much information at the current moment is written into the memory unit; ft and f t ′ are the forget gates obtained by performing 3D convolution operations on each input information by the time memory unit and the space memory unit at time step t. The forget gate is used to control how much information of the previous moment is retained; and They are respectively the temporal memory state and spatial memory state of layer l at time step t; t is the output gate, responsible for updating the hidden state of time step t W ij represents the learnable weight parameters in the convolution kernel that performs the convolution operation on the input information i in the gated unit. The subscripts x, h, c, and m are used to identify the input information X, hidden state information H, temporal memory state information C, and spatial state memory information M, respectively. The subscripts g, i, f, and o are used to identify the input modulation gate, input gate, forget gate, and output gate, respectively. 1×1 represents the learnable parameters corresponding to the 1×1 convolution kernel; b j is the bias term in the gate unit j. In addition, the vertical spatial information propagates in a zigzag pattern in the recurrent neural network. The spatial information at the bottom layer at the current moment Spatial information output from the highest layer at the previous moment This information transmission process is as follows Figure 4 Such a structure not only strengthens the connection in the time dimension, but also prevents the bottom-level recurrent unit from ignoring the high-level spatial information of the previous moment, thus improving the model's ability to represent spatiotemporal information.
[0092] After the processing of the recurrent neural network in step (3), high-level abstract feature information output at different times will be obtained. These high-level abstract feature information contain the spatiotemporal feature information learned from the original data. How to further extract information that has a greater impact on the radar echo extrapolation from these long-term feature information is the key to improving the extrapolation accuracy and solving the extrapolation ambiguity problem. To this end, it is proposed in an embodiment of the present invention that a channel attention module based on the time channel is added after the recurrent neural network to improve the model's ability to extract important features. The expression of the channel attention module of the time channel is:
[0093] ω=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0094]
[0095] Among them, is the sigmoid function, MLP is the multi-layer perceptron, is the product of the corresponding elements of the matrix, F is the three-dimensional feature information obtained by stacking the high-level abstract feature information output by the recurrent neural network at different times on the time channel. The three dimensions are the width of the feature information, the height of the feature information, and the number of channels of the feature information. AvgPool(F) is the spatial global average pooling of F, and MaxPool(F) is the spatial global maximum pooling of F.
[0096] The temporal channel-based attention module has been Figure 4 The dotted box is used to mark the attention module. The attention module first stacks the high-level abstract feature information at different moments output by the recurrent neural network on the time channel to obtain a three-dimensional feature information F; then performs spatial global average pooling and maximum pooling on it respectively to obtain two 1×1×n weight matrices, where n is the number of channels, and inputs them into a shared multi-layer perceptron (MLP) network to obtain two weight matrices, and adds them together; then a sigmoid activation function is used to obtain the final weight matrix ω, where each channel of ω represents a special detector, which is used to determine which moments of information have an important influence on the radar echo extrapolation result and assign it a higher weight; finally, the weight matrix ω is multiplied by the three-dimensional feature information F to obtain the weighted feature information F′ on the time channel, and the weighted feature information on the time channel is used as the input of the reverse reconstruction module.
[0097] The weighted feature information obtained in step (4) needs to be reversely reconstructed to obtain the final extrapolation result. The focus of the radar echo extrapolation task is to extract the spatiotemporal information in the radar data. The reverse reconstruction module structure is relatively simple. In the embodiment of the present invention, a single-layer 3D convolutional network is used to construct a reverse reconstruction module. The number of channels of the convolution kernel used in the 3D convolution is the number of channels of the weighted feature information, and the number of convolution kernels is the number of time steps for extrapolation. The specific process is to input the weighted feature information into the 3D convolutional network for convolution operation, restore the spatiotemporal information contained in the feature information, and obtain the final radar echo extrapolation result.
[0098] It can be seen that in the embodiment of the present invention, the short-term dynamic feature information of the radar echo data is first extracted by combining 2D convolution with 3D convolution layer. Then, the short-term dynamic feature information obtained in the previous step is input into the recurrent neural network to model the spatiotemporal information; then, it is spliced in the time dimension to obtain high-level abstract feature information, and the important feature information is extracted by using the attention mechanism based on the time channel, and different weights are given to the feature information at different times to suppress the secondary information, which alleviates the extrapolation ambiguity problem to a certain extent and improves the extrapolation accuracy.
[0099] The biggest obstacle to the spatiotemporal sequence prediction task is how to accurately model the image evolution in an unsupervised manner to learn complex spatiotemporal relationships, which requires the model to not only have strong spatial information extraction capabilities, but also to have temporal information fusion capabilities, and to alleviate the problem of old information loss during information flow transmission to a certain extent. In the embodiment of the present invention, an attention mechanism based on a time channel and an ST-LSTM recurrent neural network integrated with 3D volumes are used to solve the above problems, effectively extract semantics from images in an unsupervised manner, make full use of all input image features, and have the ability to learn temporal dependencies and spatial information in long-term information flows.
[0100] The present invention inputs the radar echo image of the past moment and extrapolates the radar echo image of the future moment. First, the sliding window is used to expand the two-dimensional radar echo image in the time dimension, and the 2D convolution layer and the 3D convolution layer are combined to extract the features of the radar echo image data. The ST-LSTM recurrent neural network with integrated 3D convolution is designed to model the spatiotemporal information, which enhances the characterization ability of the spatiotemporal information and can fully capture the evolution trend of the radar echo generation and disappearance. In addition, the present invention also designs an attention module based on the time channel to extract feature information that has an important influence on the radar echo extrapolation result. The present invention also uses the universal evaluation index MSE (calculated using the complete and ground-truth images in the pixel space) to quantify the model performance. The quantification results show that the radar echo extrapolation image obtained by the present invention has higher accuracy and clarity.
[0101] Example 2
[0102] The embodiment of the present invention provides a radar echo extrapolation device, comprising: a data processing module, a feature extractor, a recurrent neural network module, a channel attention module and a reverse reconstruction module arranged in sequence;
[0103] The acquired two-dimensional radar echo data is processed by the data processing module to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time;
[0104] Using a feature extractor constructed based on 3D convolution, the spatiotemporal feature information in the three-dimensional radar echo data is extracted to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information;
[0105] Inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times;
[0106] Inputting the high-level abstract feature information output at different times into a channel attention module based on a time channel to obtain feature information weighted on the time channel;
[0107] The weighted feature information on the time channel is input into the reverse reconstruction module for reverse reconstruction to restore the temporal and spatial information contained in the feature information and obtain the final radar echo extrapolation result.
[0108] The device in the embodiment of the present invention can improve the ability to capture short-term dynamic feature information in the radar echo extrapolation task, extract important feature information, and alleviate the extrapolation ambiguity problem as the prediction time increases.
[0109] Radar echo extrapolation is essentially a spatiotemporal sequence prediction problem. Radar echo data contains rich spatiotemporal information. In order to fully extract spatiotemporal information in subsequent steps and enhance the representation ability of the model, the original data needs to be processed first. In a specific implementation of an embodiment of the present invention, the method for generating three-dimensional radar echo data includes:
[0110] Use a sliding window with a length of M and a step size of S to preprocess the acquired two-dimensional radar echo data;
[0111] The continuous M frames of images in each sliding window are sequentially spliced in the time dimension as input data at each moment to obtain three-dimensional radar echo data. In the specific implementation process, the following specific steps can be used to perform:
[0112] First, a sliding window with a length of M and a step size of S is used to preprocess the N consecutive frames of radar image data. Then, the consecutive M frames of images in each sliding window are sequentially spliced in the time dimension as input data at a moment. The input data has three dimensions, each of which represents width, height, and time. Figure 2 The process of processing a training sample (N=5 frames) using a sliding window with a length of M=3 and a step size of S=1 is demonstrated. Each time, three consecutive frames of images in the sliding window are spliced in the time dimension to obtain a three-dimensional input data. The original 5 frames of two-dimensional data are converted into 3 three-dimensional data after sliding window processing.
[0113] Since the three-dimensional radar echo data obtained in step (1) is specifically three-dimensional radar echo image sequence data, it not only has time dependence, but also has spatial dynamic information. Ordinary convolution operations can only extract local static feature information of a single radar echo image, and it is difficult to effectively capture the effective evolution trend of the radar echo contained in the continuous radar echo image. In addition, the receptive field of a single-layer convolutional network is limited, and it is easy to ignore the spatial information of the entire image. It is usually necessary to use a stacked multi-layer convolutional network to expand the receptive field. To this end, in a specific implementation of an embodiment of the present invention, considering that the amount of training parameters of 3D convolution is much larger than that of 2D convolution, simply stacking 3D convolutions will make the model difficult to fit. Therefore, the embodiment of the present invention proposes to construct a feature extractor based on 2D convolution and 3D convolution, and the constructed feature extractor includes multiple layers of 2D convolutional networks arranged in sequence, and a layer of 3D convolutional network; the method for obtaining the three-dimensional feature map includes:
[0114] Using the multi-layer 2D convolutional network to perform a convolution operation on each time channel of the three-dimensional radar echo data to obtain an initial three-dimensional feature map;
[0115] The one-layer 3D convolutional network is used to extract spatiotemporal feature information from the initial three-dimensional feature map to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information.
[0116] After the three-dimensional radar echo data is processed by the feature extractor in step (2), a three-dimensional feature map is obtained. Compared with ordinary input data, the three-dimensional feature map has an additional time dimension, so the internal structure of the ST-LSTM cyclic unit also needs to be modified to be compatible with the size of the input data. According to the characteristic that the input data at a moment has three dimensions, in the embodiment of the present invention, the ordinary convolution (conv) inside the standard ST-LSTM cyclic unit is replaced with a 3D convolution (3D conv) to extract the short-term dynamic information between the data at adjacent moments. The result of the standard ST-LSTM cyclic unit is the prior art, and no further explanation is given in the embodiment of the present invention. The original standard ST-LSTM cyclic unit structure is shown in Figure 3 (a), and the structure of the ST-LSTM cyclic unit integrated with 3D convolution proposed in the embodiment of the present invention is shown in Figure 3 (b). The main structures of the two are the same, and the difference is that Figure 3 (b) uses 3D convolution to replace the ordinary convolution inside the standard ST-LSTM cyclic unit.
[0117] The ST-LSTM recurrent unit shown in Figure 3(b) contains two memory units, namely the time memory unit and spatial memory units The part marked by the left box inside the ST-LSTM recurrent unit is the time memory unit used to control the horizontal transmission and update of time information; the part marked by the right box is the spatial memory unit used to control the vertical transmission and update of spatial information. The transmission control of time information and spatial information is respectively controlled by the input gates i t and i′ t , forget gate f t and f′ t Implementation. The ST-LSTM recurrent unit with integrated 3D convolution can be expressed by formulas (1)-(10).
[0118]
[0119]
[0120]
[0121]
[0122]
[0123]
[0124]
[0125]
[0126]
[0127]
[0128] Among them, tanh is the tanh function, σ is the sigmoid function; * is the 3D convolution operation; ⊙ is the Hadamard product; + represents matrix addition, X t is the three-dimensional feature map output by the feature extractor at time step t; is the input information, which is the hidden state of layer l from time step t-1; is the input information, which is the memory state from the t-1 time step; is the input information, which is the spatial memory state from the l-1 layer at time step t; g t and g′ t The fused input information is obtained by performing 3D convolution operation on each input information by the time memory unit and the space memory unit at time step t; i t and i′ t The input gates are obtained by performing 3D convolution operations on each input information by the time memory unit and the spatial memory unit at time step t. The input gate is used to control how much information at the current moment is written into the memory unit; ft and f′ t The forget gates are obtained by performing 3D convolution operations on each input information by the time memory unit and the space memory unit at time step t. The forget gate is used to control how much information of the previous moment is retained; and They are respectively the temporal memory state and spatial memory state of layer l at time step t; t is the output gate, responsible for updating the hidden state of time step t W ij represents the learnable weight parameters in the convolution kernel that performs the convolution operation on the input information i in the gated unit. The subscripts x, h, c, and m are used to identify the input information X, hidden state information H, temporal memory state information C, and spatial state memory information M, respectively. The subscripts g, i, f, and o are used to identify the input modulation gate, input gate, forget gate, and output gate, respectively. 1×1 represents the learnable parameters corresponding to the 1×1 convolution kernel; b j is the bias term in the gate unit j. In addition, the vertical spatial information propagates in a zigzag pattern in the recurrent neural network. The spatial information at the bottom layer at the current moment Spatial information output from the highest layer at the previous moment This information transmission process is as follows Figure 4 Such a structure not only strengthens the connection in the time dimension, but also prevents the bottom-level recurrent unit from ignoring the high-level spatial information of the previous moment, thus improving the model's ability to represent spatiotemporal information.
[0129] After the processing of the recurrent neural network in step (3), high-level abstract feature information output at different times will be obtained. These high-level abstract feature information contain the spatiotemporal feature information learned from the original data. How to further extract information that has a greater impact on the radar echo extrapolation from these long-term feature information is the key to improving the extrapolation accuracy and solving the extrapolation ambiguity problem. To this end, it is proposed in an embodiment of the present invention that a channel attention module based on the time channel is added after the recurrent neural network to improve the model's ability to extract important features. The expression of the channel attention module of the time channel is:
[0130] ω=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0131]
[0132] Among them, is the sigmoid function, MLP is the multi-layer perceptron, is the product of the corresponding elements of the matrix, F is the three-dimensional feature information obtained by stacking the high-level abstract feature information output by the recurrent neural network at different times on the time channel. The three dimensions are the width of the feature information, the height of the feature information, and the number of channels of the feature information. AvgPool(F) is the spatial global average pooling of F, and MaxPool(F) is the spatial global maximum pooling of F.
[0133] The temporal channel-based attention module has been Figure 4 The dotted box is used to mark the attention module. The attention module first stacks the high-level abstract feature information at different times output by the recurrent neural network on the time channel to obtain a three-dimensional feature information F; then performs spatial global average pooling and maximum pooling on it respectively to obtain two 1×1×n weight matrices, where n is the number of channels, and inputs them into a shared multi-layer perceptron (MLP) network to obtain two weight matrices, and adds them together; then a sigmoid activation function is used to obtain the final weight matrix ω, each channel of ω represents a special detector, which is used to determine which moments of information have an important influence on the radar echo extrapolation result and assign it a higher weight; finally, the weight matrix ω is multiplied by the feature information F to obtain the weighted feature information F′ on the time channel, and the weighted feature information on the time channel is used as the input of the reverse reconstruction module.
[0134] The weighted feature information obtained in step (4) needs to be reversely reconstructed to obtain the final extrapolation result. The focus of the radar echo extrapolation task is to extract the spatiotemporal information in the radar data. The reverse reconstruction module structure is relatively simple. In the embodiment of the present invention, a single-layer 3D convolutional network is used to construct a reverse reconstruction module. The number of channels of the convolution kernel used in the 3D convolution is the number of channels of the weighted feature information, and the number of convolution kernels is the number of time steps for extrapolation. The specific process is to input the weighted feature information into the 3D convolutional network for convolution operation, restore the spatiotemporal information contained in the feature information, and obtain the final radar echo extrapolation result.
[0135] It can be seen that in the embodiment of the present invention, the short-term dynamic feature information of the radar echo data is first extracted by combining 2D convolution with 3D convolution layer. Then, the short-term dynamic feature information obtained in the previous step is input into the recurrent neural network to model the spatiotemporal information; then, it is spliced in the time dimension to obtain high-level abstract feature information, and the important feature information is extracted by using the attention mechanism based on the time channel, and different weights are given to the feature information at different times to suppress the secondary information, which alleviates the extrapolation ambiguity problem to a certain extent and improves the extrapolation accuracy.
[0136] The biggest obstacle to the spatiotemporal sequence prediction task is how to accurately model the image evolution in an unsupervised manner to learn complex spatiotemporal relationships, which requires the model to not only have strong spatial information extraction capabilities, but also to have temporal information fusion capabilities, and to alleviate the problem of old information loss during information flow transmission to a certain extent. In the embodiment of the present invention, an attention mechanism based on a time channel and an ST-LSTM recurrent neural network integrated with 3D volumes are used to solve the above problems, effectively extract semantics from images in an unsupervised manner, make full use of all input image features, and have the ability to learn temporal dependencies and spatial information in long-term information flows.
[0137] The present invention inputs the radar echo image of the past moment and extrapolates the radar echo image of the future moment. First, the sliding window is used to expand the two-dimensional radar echo image in the time dimension, and the 2D convolution layer and the 3D convolution layer are combined to extract the features of the radar echo image data. The ST-LSTM recurrent neural network with integrated 3D convolution is designed to model the spatiotemporal information, which enhances the characterization ability of spatiotemporal information and can fully capture the evolution trend of radar echo generation and disappearance. In addition, the present invention also designs an attention module based on the time channel to extract feature information that has an important influence on the radar echo extrapolation result. The present invention uses the general evaluation index MSE (calculated using the complete and ground-truth images in the pixel space) to quantify the model performance. The quantification results show that the radar echo extrapolation image obtained by the present invention has higher accuracy and clarity.
[0138] Example 3
[0139] Based on the same inventive concept as that of Embodiment 1, a radar echo extrapolation system is provided in an embodiment of the present invention, including a storage medium and a processor;
[0140] The storage medium is used to store instructions;
[0141] The processor is used to operate according to the instructions to execute the steps of the method according to any one of Embodiment 1.
[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0144] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0146] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.
[0147] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A radar echo extrapolation method, characterized in that: include: Processing the acquired two-dimensional radar echo data to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time; Using a feature extractor constructed based on 3D convolution, the spatiotemporal feature information in the three-dimensional radar echo data is extracted to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information; Inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times; Inputting the high-level abstract feature information output at different times into a channel attention module based on a time channel to obtain feature information weighted on the time channel; Based on the weighted characteristic information on the time channel, reverse reconstruction is performed to restore the temporal and spatial information contained in the characteristic information to obtain the final radar echo extrapolation result; The feature extractor includes multiple layers of 2D convolutional networks and a layer of 3D convolutional network arranged in sequence; The method for acquiring the three-dimensional feature map includes: Using the multi-layer 2D convolutional network to perform a convolution operation on each time channel of the three-dimensional radar echo data to obtain an initial three-dimensional feature map; Extracting spatiotemporal feature information from the initial three-dimensional feature map using the one-layer 3D convolutional network to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information; The recurrent neural network is constructed by the following method: Get the standard ST-LSTM recurrent unit; Replace the ordinary convolution in the standard ST-LSTM recurrent unit with a 3D convolution; The expression of the recurrent neural network includes: Among them, tanh is the tanh function, σ is the sigmoid function; * is the 3D convolution operation; ⊙ is the Hadamard product; + represents matrix addition, X t is the three-dimensional feature map output by the feature extractor at time step t; is the input information, which is the hidden state of layer l from time step t-1; is the input information, which is the memory state from the t-1 time step; is the input information, which is the spatial memory state from the l-1 layer at time step t; g t and g t ′ is the fused input information obtained by performing 3D convolution operation on each input information by the time memory unit and the spatial memory unit at time step t; i t and i t ′ are the input gates obtained after the time memory unit and the spatial memory unit at time step t perform 3D convolution operations on each input information. The input gate is used to control how much information at the current moment is written into the memory unit; f t and f t ′ are the forget gates obtained by performing 3D convolution operations on each input information by the time memory unit and the space memory unit at time step t. The forget gate is used to control how much information of the previous moment is retained; and They are respectively the temporal memory state and spatial memory state of layer l at time step t; t is the output gate, responsible for updating the hidden state of time step t W ij represents the learnable weight parameters in the convolution kernel that performs the convolution operation on the input information i in the gate control unit j. The subscripts x, h, c, and m are used to identify the input information X, hidden state information H, time memory state information C, and spatial state memory information M, respectively; the subscripts g, i, f, and o are used to identify the input modulation gate, input gate, forget gate, and output gate, respectively. 1×1 represents the learnable parameters corresponding to the 1×1 convolution kernel; b j is the bias term in gating unit j.
2. A radar echo extrapolation method according to claim 1, characterized in that: The method for generating three-dimensional radar echo data comprises: Use a sliding window with a length of M and a step size of S to preprocess the acquired two-dimensional radar echo data; The continuous M frames of images in each sliding window are spliced in turn in the time dimension as the input data at each moment to obtain the three-dimensional radar echo data.
3. A radar echo extrapolation method according to claim 1, characterized in that: The expression of the channel attention module of the time channel is: ω=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Among them, σ is the sigmoid function, MLP is the multi-layer perceptron, is the product of the corresponding elements of the matrix, AvgPool(F) is the spatial global average pooling of F, and MaxPool(F) is the spatial global maximum pooling of F; F is the three-dimensional feature information obtained by stacking the high-level abstract feature information at different times output by the recurrent neural network on the time channel. The three dimensions are the width of the feature information, the height of the feature information, and the number of channels of the feature information. ω is the weight matrix, and each channel of ω represents a special detector, which is used to determine which moments of information have an important influence on the radar echo extrapolation result and give it a higher weight; F′ is the weighted feature information on the time channel obtained by multiplying the weight matrix ω with the three-dimensional feature information F.
4. A radar echo extrapolation method according to claim 1, characterized in that: The method for obtaining the final radar echo extrapolation result includes: The feature information weighted on the time channel is input into the reverse reconstruction module, which is a single-layer 3D convolutional network, the number of channels of its convolution kernel is the number of channels of the weighted feature information, and the number of convolution kernels is the number of extrapolated time steps; The reverse reconstruction module performs convolution operation based on the received data, restores the time and space information contained in the weighted feature information, and obtains the final radar echo extrapolation result.
5. A radar echo extrapolation device, characterized in that: include: A data processing module, a feature extractor, a recurrent neural network module, a channel attention module, and a reverse reconstruction module are arranged in sequence; The acquired two-dimensional radar echo data is processed by the data processing module to generate three-dimensional radar echo data, wherein the three-dimensional radar echo data includes three dimensions: width, height and time; Using a feature extractor constructed based on 3D convolution, the spatiotemporal feature information in the three-dimensional radar echo data is extracted to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information; Inputting the three-dimensional feature map into a recurrent neural network module to obtain high-level abstract feature information output at different times; Inputting the high-level abstract feature information output at different times into a channel attention module based on a time channel to obtain feature information weighted on the time channel; The weighted feature information on the time channel is input into the reverse reconstruction module for reverse reconstruction, restoring the temporal and spatial information contained in the feature information to obtain the final radar echo extrapolation result; The feature extractor includes multiple layers of 2D convolutional networks and a layer of 3D convolutional network arranged in sequence; The method for acquiring the three-dimensional feature map includes: Using the multi-layer 2D convolutional network to perform a convolution operation on each time channel of the three-dimensional radar echo data to obtain an initial three-dimensional feature map; Extracting spatiotemporal feature information from the initial three-dimensional feature map using the one-layer 3D convolutional network to obtain a three-dimensional feature map containing overall spatial information and short-term dynamic information; The recurrent neural network is constructed by the following method: Get the standard ST-LSTM recurrent unit; Replace the ordinary convolution in the standard ST-LSTM recurrent unit with a 3D convolution; The expression of the recurrent neural network includes: Among them, tanh is the tanh function, σ is the sigmoid function; * is the 3D convolution operation; ⊙ is the Hadamard product; + represents matrix addition, X t is the three-dimensional feature map output by the feature extractor at time step t; is the input information, which is the hidden state of layer l from time step t-1; is the input information, which is the memory state from the t-1 time step; is the input information, which is the spatial memory state from the l-1 layer at time step t; g t and g t ′ is the fused input information obtained by performing 3D convolution operation on each input information by the time memory unit and the spatial memory unit at time step t; i t and i t ′ are the input gates obtained after the time memory unit and the spatial memory unit at time step t perform 3D convolution operations on each input information. The input gate is used to control how much information at the current moment is written into the memory unit; f t and f t ′ are the forget gates obtained by performing 3D convolution operations on each input information by the time memory unit and the space memory unit at time step t. The forget gate is used to control how much information of the previous moment is retained; and They are respectively the temporal memory state and spatial memory state of layer l at time step t; t is the output gate, responsible for updating the hidden state of time step t W ij represents the learnable weight parameters in the convolution kernel that performs the convolution operation on the input information i in the gate control unit j. The subscripts x, h, c, and m are used to identify the input information X, hidden state information H, time memory state information C, and spatial state memory information M, respectively; the subscripts g, i, f, and o are used to identify the input modulation gate, input gate, forget gate, and output gate, respectively. 1×1 represents the learnable parameters corresponding to the 1×1 convolution kernel; b j is the bias term in gating unit j.
6. A radar echo extrapolation system, characterized in that: including storage media and processors; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-4.
