Spatio-temporal downscaling method, device, equipment and medium based on deep learning

Abstract

The present invention provides a spatio-temporal downscaling method, device, equipment and medium based on deep learning, which relates to the technical field of electric digital data, and includes: obtaining to-be-processed forecast data, forecast time limit information and geographical information; preprocessing the to-be-processed forecast data, and inputting the preprocessed to-be-processed forecast data, forecast time limit information and geographical information into a pre-trained spatio-temporal downscaling model; through the spatio-temporal downscaling model, extracting a feature map of the to-be-processed forecast data, and using the forecast time limit information to improve the time limit adaptability of the feature map to obtain a target feature map, after performing non-integer magnification upsampling processing on the target feature map, generating target forecast data after spatio-temporal downscaling in combination with geographical information. The present invention can better improve the spatio-temporal resolution of forecast data, and at the same time effectively reduce the forecast deviation, thereby significantly improving the forecast effect.

Description

Technical Field

[0001] The present invention relates to the technical field of electric digital data, and in particular to a spatio-temporal downscaling method, device, equipment and medium based on deep learning. Background Art

[0002] With the rapid development of technology, people's requirements for weather forecasting are getting higher and higher, and there is an urgent need for high-precision forecasts with high temporal and spatial resolutions to meet the needs of production and life. The ECMWF model in Europe is a globally leading numerical model, and most meteorological stations in various provinces and cities mostly refer to its forecast results. In order to provide more accurate forecasts with higher temporal and spatial resolutions and reduce the proportion of subjective forecasts, many researchers have tried various methods to improve the spatio-temporal resolution of forecasts. Although the dynamic downscaling method performs excellently in improving the forecast accuracy, its high computational cost and relatively slow speed have become a major limitation in its application. In contrast, the artificial intelligence (AI) downscaling technology can not only achieve good results with low resource consumption, but also attract attention due to its flexibility. However, most of the existing AI downscalings are integer-magnification downscaling models, which have a gap with the non-integer magnification downscaling requirements common in actual needs. Therefore, developing a new method applicable to downscaling at any ratio has become one of the current research focuses. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a spatio-temporal downscaling method, device, equipment and medium based on deep learning, which can better improve the spatio-temporal resolution of forecast data, effectively reduce the forecast deviation, and thus significantly improve the spatio-temporal downscaling effect.

[0004] In the first aspect, an embodiment of the present invention provides a spatio-temporal downscaling method based on deep learning, including:

[0005] Obtaining the forecast data to be processed, forecast time limit information and geographical information;

[0006] Preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, forecast time limit information and geographical information into a pre-trained spatio-temporal downscaling model;

[0007] Through the spatio-temporal downscaling model, extracting the feature map of the forecast data to be processed, using the forecast time limit information to improve the time limit adaptability of the feature map to obtain the target feature map, and after performing non-integer magnification upsampling processing on the target feature map, generating the target forecast data after spatio-temporal downscaling in combination with geographical information.

[0008] In one embodiment, the spatio-temporal downscaling model includes a multi-scale feature fusion network, a forecast time limit adaptation network, a non-integer magnification upsampling layer and a terrain fusion network;

[0009] Through a spatio-temporal downscaling model, extract the feature map of the forecast data to be processed, and use the forecast lead time information to improve the timeliness adaptability of the feature map to obtain the target feature map. After performing non-integer magnification upsampling on the target feature map, combine the geographical information to generate the target forecast data after spatio-temporal downscaling, including:

[0010] Through a multi-scale feature fusion network, perform multi-scale spatio-temporal feature extraction and fusion on the forecast data to be processed to obtain the feature map of the forecast data to be processed;

[0011] Through multiple levels of forecast lead time adaptation sub-networks within the forecast lead time adaptation network, process the feature map to improve the timeliness adaptability of the feature map to obtain the target feature map;

[0012] Through a non-integer magnification upsampling network, based on the forecast data to be processed and the forecast lead time information, adjust the network parameters of the non-integer magnification upsampling layer, and use the adjusted network parameters to perform non-integer magnification upsampling on the target feature map to obtain the upsampled feature map;

[0013] Through a terrain fusion network, based on the fusion result of the upsampled feature map and the geographical information, generate the target forecast data after spatio-temporal downscaling.

[0014] In one implementation, the forecast lead time adaptation network includes multiple levels of forecast lead time adaptation sub-networks, and each level of forecast lead time adaptation sub-network includes a spatial attention convolution unit and an activation function unit;

[0015] Through multiple levels of forecast lead time adaptation sub-networks within the forecast lead time adaptation network, process the feature map to improve the timeliness adaptability of the feature map to obtain the target feature map, including:

[0016] In each level of forecast lead time adaptation sub-network, through the spatial attention convolution unit, based on the forecast lead time information, adjust the network parameters of the spatial attention convolution unit, and use the adjusted network parameters to perform a convolution operation on the feature map. The feature map after the convolution operation is output after being processed by the activation function unit;

[0017] Use the feature map output by the last-level forecast lead time adaptation sub-network as the target feature map.

[0018] In one implementation, through the spatial attention convolution unit, based on the forecast lead time information, adjust the network parameters of the spatial attention convolution unit, including:

[0019] Process the forecast lead time information through multiple levels of fully connected layers within the spatial attention convolution unit to generate a tensor;

[0020] The tensor is respectively fused with the convolution kernels and biases of the convolution layers in the spatial attention convolution unit to obtain new convolution kernels and new biases, thereby realizing the adjustment of the network parameters of the spatial attention convolution unit.

[0021] In one implementation, through a non-integer magnification upsampling network, the network parameters of the non-integer magnification upsampling layer are adjusted based on the to-be-processed forecast data and the forecast time limit information, including:

[0022] Perform spatial projection on the to-be-processed forecast data to project the pixels in the to-be-processed forecast data from the current space to the spatio-temporal downscaling space, and obtain the coordinate information of the pixels in the spatio-temporal downscaling space;

[0023] Based on the coordinate information of the pixels in the current space and the coordinate information of the pixels in the spatio-temporal downscaling space, determine the relative distance corresponding to the pixels;

[0024] The multi-level fully connected layers in the non-integer magnification upsampling network process the relative distance corresponding to the pixels and the forecast time limit information to respectively obtain the offset of the grid sampling layer in the non-integer magnification upsampling network and the convolution kernels and biases of the convolution layers, thereby realizing the adjustment of the network parameters of the non-integer magnification upsampling layer.

[0025] In one implementation, using the adjusted network parameters to perform non-integer magnification upsampling processing on the target feature map to obtain the upsampled feature map, including:

[0026] Through the adjusted grid sampling layer, perform upsampling processing on the target feature map and the coordinate information of the pixels in the spatio-temporal downscaling space to obtain the output result of the grid sampling layer;

[0027] Through the adjusted convolution layer, perform a convolution operation on the output result of the grid sampling layer to obtain the output result of the convolution layer;

[0028] Perform an addition process on the output result of the grid sampling layer and the output result of the convolution layer to obtain the upsampled feature map.

[0029] In one implementation, the method further includes:

[0030] Obtain historical forecast data, historical forecast time limit information, geographical information, and historical high-resolution land surface data;

[0031] Preprocess the historical forecast data and the historical high-resolution land surface data respectively;

[0032] Use the preprocessed historical forecast data, historical forecast time limit information, and geographical information as model inputs, and use the preprocessed historical high-resolution land surface data as training labels to construct a training dataset;

[0033] The spatio-temporal downscaling model is trained using a training data set.

[0034] In a second aspect, an embodiment of the present invention further provides a spatio-temporal downscaling device based on deep learning, including:

[0035] A data acquisition module, configured to acquire forecast data to be processed, forecast time limit information, and geographical information;

[0036] A data preprocessing module, configured to preprocess the forecast data to be processed, and input the preprocessed forecast data to be processed, forecast time limit information, and geographical information into a pre-trained spatio-temporal downscaling model;

[0037] A spatio-temporal downscaling module, configured to extract a feature map of the forecast data to be processed through the spatio-temporal downscaling model, improve the timeliness adaptability of the feature map using the forecast time limit information to obtain a target feature map, and after performing non-integer magnification upsampling processing on the target feature map, generate target forecast data after spatio-temporal downscaling in combination with geographical information.

[0038] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, where the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to implement the method according to any one of the first aspect.

[0040] A spatio-temporal downscaling method, device, device, and medium based on deep learning provided by an embodiment of the present invention first preprocesses forecast data to be processed, and inputs the preprocessed forecast data to be processed, forecast time limit information, and geographical information into a pre-trained spatio-temporal downscaling model; then, through the spatio-temporal downscaling model, extracts a feature map of the forecast data to be processed, and uses the forecast time limit information to improve the timeliness adaptability of the feature map to obtain a target feature map, and after performing non-integer magnification upsampling processing on the target feature map, generates target forecast data after spatio-temporal downscaling in combination with geographical information. The above method can simultaneously achieve efficient downscaling in both the time dimension and the space dimension by using a spatio-temporal downscaling model under a unified framework, and introduces forecast time limit information to improve the timeliness adaptability of the feature map to optimize the data processing process within different prediction periods, thereby effectively reducing the deviation in the target forecast data and significantly improving the spatio-temporal downscaling effect.

[0041] Other features and advantages of the present invention will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present invention. The objectives and other advantages of the present invention are realized and attained by the structure particularly pointed out in the specification, claims and drawings.

[0042] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, provides a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic flowchart of a spatio-temporal downscaling method based on deep learning provided by an embodiment of the present invention;

[0045] Figure 2 It is a schematic flowchart of a spatio-temporal downscaling model provided by an embodiment of the present invention;

[0046] Figure 3 It is a schematic structural diagram of a spatio-temporal downscaling model provided by an embodiment of the present invention;

[0047] Figure 4 It is a schematic structural diagram of a spatial attention convolution unit in a prediction time adaptation sub-network provided by an embodiment of the present invention;

[0048] Figure 5 It is a schematic structural diagram of a non-integer multiple upsampling network provided by an embodiment of the present invention;

[0049] Figure 6 It is a schematic structural diagram of a spatio-temporal downscaling device based on deep learning provided by an embodiment of the present invention;

[0050] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Currently, related technical means can only improve the spatio-temporal resolution of forecasts, but cannot effectively reduce forecast biases, resulting in poor spatio-temporal downscaling effects. Based on this, the embodiments of the present invention provide a spatio-temporal downscaling method, device, equipment, and medium based on deep learning, which can effectively improve the spatio-temporal resolution of forecast data and reduce forecast biases, thereby significantly enhancing the spatio-temporal downscaling effect.

[0053] To facilitate the understanding of this embodiment, a spatio-temporal downscaling method based on deep learning disclosed in the embodiments of the present invention will be introduced in detail first. Refer to Figure 1 the schematic flowchart of a spatio-temporal downscaling method based on deep learning shown in the figure. The method mainly includes the following steps S102 to S106:

[0054] Step S102: Obtain the forecast data to be processed, forecast validity information, and geographical information.

[0055] Among them, the forecast data to be processed can be multiple forecast variables of C1D provided by the European Centre for Medium-Range Weather Forecasts (ECMWF). The forecast validity information is the valid period of the forecast data to be processed, and the geographical information includes digital elevation model (DEM), soil sand content ratio, land type, etc.

[0056] Step S104: Preprocess the forecast data to be processed, and input the preprocessed forecast data to be processed, forecast validity information, and geographical information into a pre-trained spatio-temporal downscaling model.

[0057] In one example, the preprocessing includes normalization processing, preferably the maximum-minimum normalization method, and the normalized forecast data to be processed, forecast validity information, and geographical information are used as the input of the spatio-temporal downscaling model.

[0058] Step S106: Through the spatio-temporal downscaling model, extract the feature map of the forecast data to be processed, and use the forecast validity information to improve the timeliness adaptability of the feature map to obtain the target feature map. After performing non-integer magnification upsampling processing on the target feature map, combine the geographical information to generate the target forecast data after spatio-temporal downscaling.

[0059] Among them, the spatio-temporal downscaling model includes a multi-scale feature fusion network, a forecast lead time adaptation network, a non-integer magnification upsampling layer, and a terrain fusion network. The input of the multi-scale feature fusion network is the preprocessed forecast data to be processed, and the output is a feature map. This network is used to extract and fuse multi-scale spatio-temporal features of the forecast data to be processed; the input of the forecast lead time adaptation network is the forecast lead time information and the feature map output by the multi-scale feature fusion network, and the output is a target feature map. This network is used to update its own network parameters using the forecast lead time information, and process the feature map using the updated network parameters to generate a target feature map with better lead time adaptability; the input of the non-integer magnification upsampling layer is the forecast lead time information and the target feature map output by the forecast lead time adaptation network. This network is used to update its own network parameters using the forecast lead time information, and perform non-integer magnification upsampling processing on the target feature map to obtain an upsampled feature map; the input of the terrain fusion network is the terrain information and the upsampled feature map, and the output is the target forecast data after spatio-temporal downscaling. This network is used to fuse the terrain information and the upsampled feature map to generate the target forecast data after spatio-temporal downscaling.

[0060] The spatio-temporal downscaling method based on deep learning provided by the embodiments of the present invention can simultaneously achieve efficient downscaling in both the time dimension and the space dimension by adopting a spatio-temporal downscaling model under a unified framework, and introduce forecast lead time information to improve the lead time adaptability of the feature map, so as to optimize the data processing process within different prediction periods, thereby effectively reducing the bias in the target forecast data, and further significantly improving the spatio-temporal downscaling effect.

[0061] Before using the spatio-temporal downscaling model to perform spatio-temporal downscaling processing on the forecast data to be processed, it is necessary to pre-train the spatio-temporal downscaling model. Refer to Figure 2 The flow schematic diagram of a spatio-temporal downscaling model shown, including processes such as dataset production, quality control and normalization, model construction, model training, model saving and evaluation.

[0062] For ease of understanding, the embodiments of the present invention provide a specific implementation manner for training a spatio-temporal downscaling model, including the following steps S202 to step S208:

[0063] Step S202, obtain historical forecast data, historical forecast lead time information, geographical information, and historical high-resolution land surface data. Among them, the historical forecast data can be multiple forecast variables of C1D in the historical period provided by the European Centre for Medium-Range Weather Forecasts (ECMWF). The historical forecast lead time information is the valid period of the historical forecast data. The geographical information includes digital elevation model (DEM), soil sand content ratio, land type, etc. The historical high-resolution road surface data is also the variable provided by the High-Resolution Land Data Assimilation System (HRCLDAS), and this variable is used as a label.

[0064] Step S204: Preprocess the historical forecast data and the historical high-resolution land surface data respectively. Among them, the preprocessing of the historical forecast data includes normalization processing, and the preprocessing of the historical high-resolution land surface data includes quality control processing and normalization processing.

[0065] Optionally, the quality control processing adopts a combination of threshold value test and row-column standard deviation analysis. Specifically, the threshold value test aims to ensure that the values of all grid points are within a reasonable range in terms of climatology; for data points outside the range, the average value is used as a substitute. The row-column standard deviation analysis identifies potential anomalies by calculating the standard deviation of each row and each column in the image. If it is found that the standard deviation of consecutive multiple rows or columns is zero, this sample will be marked as suspicious and further confirmed whether there is an actual anomaly through visualization means. Once confirmed as an anomaly, the relevant sample will be removed from the dataset.

[0066] The normalization uses the maximum-minimum normalization method. The normalization formula is:

[0067]

[0068] where X * is the normalized historical forecast data or historical high-resolution land surface data, X is the historical forecast data or historical high-resolution land surface data, max(X) is the maximum value of the historical forecast data or historical high-resolution land surface data, and min(X) is the minimum value of the historical forecast data or historical high-resolution land surface data.

[0069] Step S206: Use the preprocessed historical forecast data, historical forecast time limit information, and geographical information as model inputs, and use the preprocessed historical high-resolution land surface data as training labels to construct a training dataset.

[0070] Step S208: Train the spatio-temporal downscaling model using the training dataset. In specific implementation, after inputting the preprocessed historical forecast data, historical forecast time limit information, and geographical information in the training dataset into the spatio-temporal downscaling model, through the spatio-temporal downscaling model, extract the feature map of the historical forecast data, and use the historical forecast time limit information to improve the time limit adaptability of the feature map to obtain the target feature map. After performing non-integer magnification upsampling processing on the target feature map, combine the geographical information to generate the target historical forecast data after spatio-temporal downscaling. Finally, train the spatio-temporal downscaling model based on the target historical forecast data and the historical high-resolution land surface data.

[0071] Preferably, the Adam optimizer is used during model training, with an initial learning rate of 2e-4 and a SmoothL1 loss function. After the model training is completed, it is necessary to save the model parameters so that the spatio-temporal downscaling model with these parameters can be used subsequently to generate high spatio-temporal resolution forecast data (i.e., target forecast data).

[0072] To facilitate the understanding of the process of spatio-temporal downscaling of the output data by the spatio-temporal downscaling model during the training stage and the prediction stage (i.e., the aforementioned step S208 and the aforementioned step S106), an embodiment of the present invention provides a specific implementation.

[0073] First, the structure of the spatio-temporal downscaling model is explained. This spatio-temporal downscaling model can adapt to forecast data at any magnification. An embodiment of the present invention provides a Figure 3 specific structural schematic diagram of a spatio-temporal downscaling model as shown. The spatio-temporal downscaling model includes a multi-scale feature fusion network (Unet3d), a forecast time adaption network, a non-integer magnification upsampling layer (Upsample), and a terrain fusion network (Conv2d). The forecast time adaption network includes multiple levels (such as 2 levels) of forecast time adaption sub-networks, and each level of the forecast time adaption sub-network includes a spatial attention convolution unit (SA_Conv) and an activation function unit (ReLU). Additionally, Figure 3 it is also shown that the input of the spatio-temporal downscaling model includes preprocessed forecast data to be processed (LR), forecast time information (t), and geographical information (Static_data), and the output is target forecast data (HR).

[0074] Taking the prediction stage as an example, an embodiment of the present invention provides a specific implementation of the spatio-temporal downscaling model for the output data. See the following steps 1 to 4:

[0075] Step 1, through the multi-scale feature fusion network, perform multi-scale spatio-temporal feature extraction and fusion on the forecast data to be processed to obtain a feature map of the forecast data to be processed.

[0076] In a specific implementation, the multi-scale feature fusion network is based on the 3D U-Net architecture and uses 3D convolution (Conv3d) technology to extract spatio-temporal features (i.e., feature maps) from the forecast data to be processed. To enhance the non-linear fitting ability of the model, ReLU is used as the activation function, and a channel attention mechanism is introduced after each ReLU activation to strengthen the expression of important features.

[0077] Step 2, through the multiple levels of forecast time adaption sub-networks in the forecast time adaption network, process the feature map to improve the time adaptability of the feature map to obtain a target feature map.

[0078] In one implementation, in each lead-time adaptation sub-network, through the spatial attention convolution unit, the network parameters of the spatial attention convolution unit are adjusted based on the lead-time information, and the feature map is convolved using the adjusted network parameters. The feature map after the convolution operation is processed by the activation function unit and then output. The feature map output by the last lead-time adaptation sub-network is used as the target feature map. Among them, the adjusted network parameters include the bias and convolution kernel of the convolution unit.

[0079] The design purpose of the lead-time adaptation network is to improve the adaptability of the model to different lead times and reduce the prediction error between different prediction periods. It combines the input feature map with the lead-time information and uses the spatial attention convolution (SA_Conv) operation - that is, first processes the lead-time information with a fully connected layer to generate a tensor, and then adjusts the convolution kernel parameters with this tensor to form a new convolution kernel. After each SA_Conv, the ReLU activation function is applied to further improve the model performance. Taking Figure 3 as an example, the feature map is processed by the spatial attention convolution unit, the activation function unit, another spatial attention convolution unit, and another activation function unit in sequence, and then the target feature map is output.

[0080] The embodiment of the present invention provides a Figure 4 structural schematic diagram of the spatial attention convolution unit in a lead-time adaptation sub-network as shown in Figure 4 which shows that the spatial attention convolution unit includes multiple levels (such as 2 levels) of fully connected layers FC and a convolution unit Conv. On the basis of this structure, the embodiment of the present invention provides a specific implementation manner of the SA_Conv operation, including:

[0081] (1), Process the lead-time information through multiple levels of fully connected layers in the spatial attention convolution unit to generate a tensor. Please continue to refer to Figure 4 , the lead-time information t generates a tensor after passing through two fully connected layers.

[0082] (2), Fuse the tensor with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit respectively to obtain a new convolution kernel and a new bias, so as to realize the adjustment of the network parameters of the spatial attention convolution unit. Please continue to refer to Figure 4 , there are two preset tensors Tensor, one of which represents the current bias of the convolution layer, and the other represents the current convolution kernel of the convolution layer. Fuse the tensor generated based on the lead-time information t with the two tensors Tensor respectively, and then the new convolution kernel Kernel and the new bias Bias can be obtained.

[0083] On this basis, a new convolution kernel Kernel and a new bias Bias are configured into the convolutional layer Conv to process the feature map feature output by the multi-scale feature fusion network using the convolutional layer Conv, and a target feature map feature is obtained.

[0084] Step 3: Through a non-integer magnification upsampling network, adjust the network parameters of the non-integer magnification upsampling layer based on the forecast data to be processed and the forecast time limit information, and perform non-integer magnification upsampling processing on the target feature map using the adjusted network parameters to obtain an upsampled feature map.

[0085] Considering that the super-resolution requirements of meteorological data often involve non-integer magnification expansion, this solution does not use the traditional PixelShuffle method, but uses an upsampling algorithm specifically designed for non-integer magnification. See Figure 5 The structural schematic diagram of a non-integer magnification upsampling network shown. The non-integer magnification upsampling network includes multiple fully connected layers FC, a grid sampling layer grid_sample, and a convolutional layer Conv. Specifically, it includes 3 fully connected layers FC, and the 3rd level includes 3 fully connected layers FC with different parameters.

[0086] Based on this structure, an embodiment of the present invention provides a specific implementation manner of non-integer magnification upsampling processing. It includes:

[0087] (1) Perform spatial projection on the forecast data to be processed to project the pixels in the forecast data to be processed from the current space to the spatio-temporal downscaling space, and obtain the coordinate information of the pixels in the spatio-temporal downscaling space. Specifically, each pixel on the forecast data HR to be processed is projected onto the LR space (i.e., the spatio-temporal downscaling space) to calculate the coordinate information C(x), C(y) on the LR space.

[0088] (2) Based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatio-temporal downscaling space, determine the relative distance corresponding to the pixel. Specifically, calculate the coordinate information of the pixel in the current space and the coordinate information C(x), C(y) of the pixel in the spatio-temporal downscaling space, and determine the relative distance R(x), R(y) corresponding to the pixel.

[0089] (3) Process the relative distance corresponding to the pixel and the forecast time limit information through multiple fully connected layers in the non-integer magnification upsampling network to respectively obtain the offset of the grid sampling layer and the convolution kernel and bias of the convolutional layer in the non-integer magnification upsampling network, and realize the adjustment of the network parameters of the non-integer magnification upsampling layer. Please continue to refer to Figure 5, after the relative distances R(x), R(y) and the forecast lead time information t are processed by the 3-layer fully connected layer FC, the offset of the grid sampling layer grid_sample and the convolution kernel Kernel and bias Bias of the convolutional layer Conv are obtained.

[0090] (4) Through the grid sampling layer with adjusted parameters, upsample the target feature map and the coordinate information of the pixels in the spatio-temporal downscaling space to obtain the output result of the grid sampling layer. Please continue to refer to Figure 5 , configure the offset to the grid sampling layer grid_sample to utilize the grid sampling layer grid_sample to upsample the target feature map feature output by the forecast lead time adaptation network.

[0091] (5) Through the convolutional layer with adjusted parameters, perform a convolution operation on the output result of the grid sampling layer to obtain the output result of the convolutional layer. Please continue to refer to Figure 5 , configure the convolution kernel Kernel and bias Bias to the convolutional layer Conv to utilize the convolutional layer Conv to perform a convolution operation on the output result of the grid sampling layer.

[0092] (6) Add the output result of the grid sampling layer and the output result of the convolutional layer to obtain the upsampled feature map. Please continue to refer to Figure 5 , the grid sampling layer grid_sample and the convolutional layer Conv are skip-connected, that is, add the output result of the grid sampling layer grid_sample and the output result of the convolutional layer Conv to obtain the upsampled feature map feature.

[0093] Step 4, through the terrain fusion network, based on the fusion result of the upsampled feature map and the geographical information, generate the target forecast data after spatio-temporal downscaling. In a specific implementation, after completing the upsampling step, the terrain fusion network receives geographical information data such as digital elevation model (DEM), soil sediment content ratio, land type, etc. and the upsampled feature map. All this information is concatenated along the channel dimension and then fed into a series of convolutional layers for in-depth processing, and finally output the target forecast data with higher resolution.

[0094] Furthermore, the spatio-temporal downscaling model can be evaluated using the target forecast data, and the evaluation metrics are MAE and RMSE. Specifically, MAE and RMSE can be calculated according to the following formulas:

[0095]

[0096] where n is the amount of forecast data to be processed, Y iis the target forecast data corresponding to the i-th forecast data to be processed, is the true value corresponding to the i-th forecast data to be processed (such as high-resolution land surface data).

[0097] In summary, first, a dataset is constructed and the data is subjected to quality control and normalization. Then, a spatio-temporal downscaling model is built and the dataset is used to train the model. Finally, the trained model is used for real-time downscaling. The present invention uses deep learning to perform spatio-temporal downscaling at any magnification. By adopting a spatio-temporal downscaling model under a unified architecture, the present invention can simultaneously achieve efficient downscaling in both the time and space dimensions, and introduce a forecast time adaption layer to optimize data processing within different prediction periods, thereby effectively reducing the bias in the forecast results. Through experimental verification, the ECMWF model spatio-temporal downscaling technology disclosed in the embodiments of the present invention can reduce the spatial resolution of the ECMWF model from 0.125° to any resolution between 0.125° and 0.01°, and the time resolution from 3h to 1h. While improving the spatio-temporal resolution, it can also significantly correct the bias in the forecast. The embodiments of the present invention can reduce the spatio-temporal resolution and reduce the forecast error through this objective method, providing a reference for forecasters and better serving people's production and life, meeting the current society's need for forecast accuracy.

[0098] Based on the foregoing embodiments, an embodiment of the present invention provides a spatio-temporal downscaling device based on deep learning. Refer to Figure 6 the structural schematic diagram of a spatio-temporal downscaling device based on deep learning shown. The device mainly includes the following parts:

[0099] A data acquisition module 602, configured to acquire forecast data to be processed, forecast time information, and geographical information;

[0100] A data preprocessing module 604, configured to preprocess the forecast data to be processed, and input the preprocessed forecast data to be processed, forecast time information, and geographical information into a pre-trained spatio-temporal downscaling model;

[0101] A spatio-temporal downscaling module 606, configured to extract a feature map of the forecast data to be processed through the spatio-temporal downscaling model, improve the time adaptability of the feature map by using the forecast time information to obtain a target feature map, and after performing non-integer magnification upsampling processing on the target feature map, generate spatio-temporally downscaled target forecast data in combination with geographical information.

[0102] The spatio-temporal downscaling device provided by the embodiment of the present invention can efficiently perform downscaling in both the time dimension and the space dimension by adopting a spatio-temporal downscaling model under a unified framework, introduce the forecast lead time information to improve the timeliness adaptability of the feature map, optimize the data processing process within different prediction periods, thereby effectively reducing the bias in the target forecast data, and further significantly improving the spatio-temporal downscaling effect.

[0103] In one implementation, the spatio-temporal downscaling model includes a multi-scale feature fusion network, a forecast lead time adaptation network, a non-integer magnification upsampling layer, and a terrain fusion network; the spatio-temporal downscaling module 606 is specifically configured to:

[0104] Through the multi-scale feature fusion network, perform multi-scale spatio-temporal feature extraction and fusion on the forecast data to be processed to obtain the feature map of the forecast data to be processed;

[0105] Through the multi-level forecast lead time adaptation sub-networks in the forecast lead time adaptation network, process the feature map to improve the timeliness adaptability of the feature map to obtain the target feature map;

[0106] Through the non-integer magnification upsampling network, adjust the network parameters of the non-integer magnification upsampling layer based on the forecast data to be processed and the forecast lead time information, and perform non-integer magnification upsampling processing on the target feature map using the adjusted network parameters to obtain the upsampled feature map;

[0107] Through the terrain fusion network, generate the target forecast data after spatio-temporal downscaling based on the fusion result of the upsampled feature map and the geographical information.

[0108] In one implementation, the forecast lead time adaptation network includes multi-level forecast lead time adaptation sub-networks, and each level of forecast lead time adaptation sub-network includes a spatial attention convolution unit and an activation function unit; the spatio-temporal downscaling module 606 is specifically configured to:

[0109] In each level of forecast lead time adaptation sub-network, through the spatial attention convolution unit, adjust the network parameters of the spatial attention convolution unit based on the forecast lead time information, and perform a convolution operation on the feature map using the adjusted network parameters. The feature map after the convolution operation is processed by the activation function unit and then output;

[0110] Use the feature map output by the last forecast lead time adaptation sub-network as the target feature map.

[0111] In one implementation, the spatio-temporal downscaling module 606 is specifically configured to:

[0112] Process the forecast lead time information through the multi-level fully connected layers in the spatial attention convolution unit to generate a tensor;

[0113] The tensor is respectively fused with the convolution kernels and biases of the convolution layers in the spatial attention convolution unit to obtain new convolution kernels and new biases, thereby realizing the adjustment of the network parameters of the spatial attention convolution unit.

[0114] In one implementation, the spatio-temporal downscaling module 606 is specifically configured to:

[0115] Perform a spatial projection on the to-be-processed forecast data to project the pixels in the to-be-processed forecast data from the current space to the spatio-temporal downscaling space, and obtain the coordinate information of the pixels in the spatio-temporal downscaling space;

[0116] Based on the coordinate information of the pixels in the current space and the coordinate information of the pixels in the spatio-temporal downscaling space, determine the relative distance corresponding to the pixels;

[0117] Process the relative distance corresponding to the pixels and the forecast lead time information through the multi-level fully connected layers in the non-integer magnification upsampling network, and respectively obtain the offsets of the grid sampling layer and the convolution kernels and biases of the convolution layers in the non-integer magnification upsampling network, thereby realizing the adjustment of the network parameters of the non-integer magnification upsampling layer.

[0118] In one implementation, the spatio-temporal downscaling module 606 is specifically configured to:

[0119] Perform an upsampling process on the target feature map and the coordinate information of the pixels in the spatio-temporal downscaling space through the grid sampling layer with adjusted parameters, and obtain the output result of the grid sampling layer;

[0120] Perform a convolution operation on the output result of the grid sampling layer through the convolution layer with adjusted parameters, and obtain the output result of the convolution layer;

[0121] Perform an addition process on the output result of the grid sampling layer and the output result of the convolution layer to obtain the upsampled feature map.

[0122] In one implementation, it further includes a model training module, which is used to:

[0123] Obtain historical forecast data, historical forecast lead time information, geographical information, and historical high-resolution land surface data;

[0124] Preprocess the historical forecast data and the historical high-resolution land surface data respectively;

[0125] Use the preprocessed historical forecast data, historical forecast lead time information, and geographical information as model inputs, and use the preprocessed historical high-resolution land surface data as training labels to construct a training dataset;

[0126] Use the training dataset to train the spatio-temporal downscaling model.

[0127] The device provided by the embodiments of the present invention has the same implementation principle and technical effects as those of the foregoing method embodiments. For the sake of brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding contents in the foregoing method embodiments.

[0128] The embodiments of the present invention provide an electronic device. Specifically, the electronic device includes a processor and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processor, it executes the method according to any one of the foregoing embodiments.

[0129] Figure 7 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes: a processor 70, a memory 71, a bus 72, and a communication interface 73. The processor 70, the communication interface 73, and the memory 71 are connected through the bus 72; the processor 70 is used to execute an executable module stored in the memory 71, such as a computer program.

[0130] Among them, the memory 71 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 73 (which may be wired or wireless), a communication connection is realized between the system network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0131] The bus 72 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 7 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0132] Among them, the memory 71 is used to store a program. After receiving an execution instruction, the processor 70 executes the program. The method executed by the device defined by the flow process disclosed in any one of the foregoing embodiments of the present invention can be applied to the processor 70 or implemented by the processor 70.

[0133] The processor 70 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 70 or the instructions in the form of software. The above-mentioned processor 70 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 71, and the processor 70 reads the information in the memory 71 and combines its hardware to complete the steps of the above method.

[0134] The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation, reference can be made to the foregoing method embodiments, and details are not described herein again.

[0135] If the above-mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0136] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A spatiotemporal downscaling method based on deep learning, characterized in that: include: Obtain forecast data to be processed, forecast timeliness information and geographic information; Preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, the forecast timeliness information and the geographic information into a pre-trained spatiotemporal downscaling model; The feature map of the forecast data to be processed is extracted by the spatiotemporal downscaling model, and the time adaptability of the feature map is improved by using the forecast time information to obtain a target feature map. After performing non-integer multiple upsampling processing on the target feature map, the target forecast data after spatiotemporal downscaling is generated in combination with the geographic information; The spatiotemporal downscaling model includes a non-integer rate upsampling layer and a terrain fusion network; After performing non-integer multiple upsampling processing on the target feature map, generating the target forecast data after spatiotemporal downscaling in combination with the geographic information, including: Performing spatial projection on the forecast data to be processed, so as to project pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtaining coordinate information of the pixels in the spatiotemporal downscaling space; Determine the relative distance corresponding to the pixel based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space; The relative distance corresponding to the pixel and the forecast timeliness information are processed by the multi-level fully connected layer in the non-integer rate upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer rate upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer rate upsampling layer; By adjusting the parameters of the grid sampling layer, upsampling the target feature map and the coordinate information of the pixel in the spatiotemporal downscaling space is performed to obtain an output result of the grid sampling layer; By adjusting the parameters of the convolution layer, a convolution operation is performed on the output result of the grid sampling layer to obtain the output result of the convolution layer; Adding the output result of the grid sampling layer to the output result of the convolution layer to obtain an upsampled feature map; Through the terrain fusion network, based on the fusion result of the upsampled feature map and the geographic information, the target forecast data after spatiotemporal downscaling is generated.

2. The deep learning-based spatiotemporal downscaling method according to claim 1, characterized in that: The spatiotemporal downscaling model also includes a multi-scale feature fusion network and a forecast timeliness adaptation network; Extracting the characteristic graph of the forecast data to be processed, and using the forecast timeliness information to improve the timeliness adaptability of the characteristic graph to obtain a target characteristic graph, including: Through the multi-scale feature fusion network, multi-scale spatiotemporal feature extraction and fusion are performed on the forecast data to be processed to obtain a feature map of the forecast data to be processed; The feature graph is processed through a multi-level forecast time-effectiveness adaptation sub-network within the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature graph and obtain a target feature graph.

3. The deep learning-based spatiotemporal downscaling method according to claim 2, characterized in that: The forecast timeliness adaptation network includes multiple levels of forecast timeliness adaptation sub-networks, and each level of the forecast timeliness adaptation sub-network includes a spatial attention convolution unit and an activation function unit; The feature graph is processed by a multi-level forecast time-effectiveness adaptation sub-network in the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature graph to obtain a target feature graph, including: In each level of the forecast time adaptation subnetwork, the network parameters of the spatial attention convolution unit are adjusted based on the forecast time information through the spatial attention convolution unit, and the feature map is convolved using the adjusted network parameters, and the feature map after the convolution operation is processed by the activation function unit and then output; The feature map output by the last forecast time adaptation sub-network is used as the target feature map.

4. The deep learning-based spatiotemporal downscaling method according to claim 3, characterized in that: By means of the spatial attention convolution unit, adjusting the network parameters of the spatial attention convolution unit based on the forecast timeliness information includes: Processing the forecast timeliness information through a multi-level fully connected layer in the spatial attention convolution unit to generate a tensor; The tensor is fused with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit respectively to obtain a new convolution kernel and a new bias, thereby adjusting the network parameters of the spatial attention convolution unit.

5. The deep learning-based spatiotemporal downscaling method according to claim 1, characterized in that: The method further comprises: Obtain historical forecast data, historical forecast time information, geographic information and historical high-resolution land surface data; Preprocessing the historical forecast data and the historical high-resolution land surface data respectively; The pre-processed historical forecast data, the historical forecast timeliness information, and the geographic information are used as model inputs, and the pre-processed historical high-resolution land surface data are used as training labels to construct a training data set; The spatiotemporal downscaling model is trained using the training data set.

6. A spatiotemporal downscaling device based on deep learning, characterized in that: include: A data acquisition module is used to obtain forecast data to be processed, forecast timeliness information and geographic information; A data preprocessing module, used for preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, the forecast timeliness information and the geographic information into a pre-trained spatiotemporal downscaling model; A spatiotemporal downscaling module is used to extract the feature map of the forecast data to be processed through the spatiotemporal downscaling model, and use the forecast timeliness information to improve the timeliness adaptability of the feature map to obtain a target feature map, perform non-integer multiple upsampling processing on the target feature map, and generate the target forecast data after spatiotemporal downscaling in combination with the geographic information; The spatiotemporal downscaling model includes a non-integer upsampling layer and a terrain fusion network; the spatiotemporal downscaling module is specifically used for: Performing spatial projection on the forecast data to be processed, so as to project pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtaining coordinate information of the pixels in the spatiotemporal downscaling space; Determine the relative distance corresponding to the pixel based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space; The relative distance corresponding to the pixel and the forecast timeliness information are processed by the multi-level fully connected layer in the non-integer rate upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer rate upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer rate upsampling layer; By adjusting the parameters of the grid sampling layer, upsampling the target feature map and the coordinate information of the pixel in the spatiotemporal downscaling space is performed to obtain an output result of the grid sampling layer; By adjusting the parameters of the convolution layer, a convolution operation is performed on the output result of the grid sampling layer to obtain the output result of the convolution layer; Adding the output result of the grid sampling layer to the output result of the convolution layer to obtain an upsampled feature map; Through the terrain fusion network, based on the fusion result of the upsampled feature map and the geographic information, the target forecast data after spatiotemporal downscaling is generated.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images