Time-space downscaling method and device based on deep learning, equipment and medium

Abstract

The invention provides a time-space downscaling method and device based on deep learning, equipment and a medium, and relates to the technical field of electric digital data, and the method comprises the steps: obtaining to-be-processed forecast data, forecast aging information and geographic information; pre-processing the to-be-processed forecast data, and inputting the pre-processed to-be-processed forecast data, the forecast aging information and the geographic information into a time-airborne downscaling model obtained by pre-training; the method comprises the following steps: extracting a feature map of to-be-processed forecast data through a time-airborne downscaling model, improving the timeliness adaptability of the feature map by utilizing forecast timeliness information to obtain a target feature map, performing non-integer multiplying power up-sampling processing on the target feature map, and generating target forecast data after time-airborne downscaling in combination with geographic information. According to the method, the temporal-spatial resolution of the forecast data can be well improved, and meanwhile, the forecast deviation is effectively reduced, so that the forecast effect is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of electric digital data technology, and in particular to a deep learning-based spatiotemporal downscaling method, device, equipment and medium. Background Art

[0002] With the rapid development of science and technology, people have higher and higher requirements for weather forecasts, and they are in urgent need of high-precision and high-temporal and spatial resolution forecasts to meet the needs of production and life. The European ECMWF model is the world's leading numerical model, and most provincial and municipal meteorological stations refer to its forecast results. In order to provide more accurate forecast results with higher temporal and spatial resolution and reduce the proportion of subjective forecasts, many researchers have tried various methods to improve the temporal and spatial resolution of forecasts. Although the dynamic downscaling method has excellent performance in improving forecast accuracy, its high computational cost and relatively slow speed have become a major limitation of its application. In contrast, artificial intelligence (AI) downscaling technology can not only achieve good results with low resource consumption, but also attract attention for its flexibility. However, most of the existing AI downscaling models are integer multiples, which is different from the non-integer multiples downscaling requirements commonly seen in actual needs. Therefore, the development of new methods suitable for arbitrary ratio downscaling has become one of the focuses of current research. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a deep learning-based spatiotemporal downscaling method, device, equipment and medium, which can better improve the spatiotemporal resolution of forecast data, while effectively reducing forecast deviations, thereby significantly improving the spatiotemporal downscaling effect.

[0004] In a first aspect, an embodiment of the present invention provides a spatiotemporal downscaling method based on deep learning, comprising:

[0005] Obtain forecast data to be processed, forecast timeliness information and geographic information;

[0006] Preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, forecast timeliness information and geographic information into the pre-trained spatiotemporal downscaling model;

[0007] Through the spatiotemporal downscaling model, the characteristic map of the forecast data to be processed is extracted, and the forecast timeliness information is used to improve the timeliness adaptability of the characteristic map to obtain the target characteristic map. After the target characteristic map is upsampled at a non-integer rate, the target forecast data after spatiotemporal downscaling is generated in combination with geographic information.

[0008] In one embodiment, the spatiotemporal downscaling model includes a multi-scale feature fusion network, a forecast time adaptation network, a non-integer rate upsampling layer, and a terrain fusion network;

[0009] Through the spatiotemporal downscaling model, the feature map of the forecast data to be processed is extracted, and the time adaptability of the feature map is improved by using the forecast time information to obtain the target feature map. After the target feature map is upsampled at a non-integer multiple, the target forecast data after spatiotemporal downscaling is generated in combination with the geographic information, including:

[0010] Through the multi-scale feature fusion network, multi-scale spatiotemporal features are extracted and fused for the forecast data to be processed, so as to obtain the feature map of the forecast data to be processed;

[0011] The feature map is processed through a multi-level forecast time-effectiveness adaptation sub-network in the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature map and obtain a target feature map;

[0012] Through a non-integer rate upsampling network, the network parameters of the non-integer rate upsampling layer are adjusted based on the forecast data to be processed and the forecast timeliness information, and the target feature map is subjected to non-integer rate upsampling processing using the adjusted network parameters to obtain an upsampled feature map;

[0013] Through the terrain fusion network, the target forecast data after spatiotemporal downscaling is generated based on the fusion results of the upsampled feature map and geographic information.

[0014] In one embodiment, the forecast timeliness adaptation network includes multiple levels of forecast timeliness adaptation sub-networks, and each level of forecast timeliness adaptation sub-network includes a spatial attention convolution unit and an activation function unit;

[0015] The feature map is processed through the multi-level forecast time adaptation sub-network in the forecast time adaptation network to improve the time adaptability of the feature map to obtain the target feature map, including:

[0016] In each level of forecast time adaptation sub-network, the network parameters of the spatial attention convolution unit are adjusted based on the forecast time information through the spatial attention convolution unit, and the feature map is convolved using the adjusted network parameters. The feature map after the convolution operation is processed by the activation function unit and then output;

[0017] The feature map output by the last prediction time adaptation sub-network is used as the target feature map.

[0018] In one embodiment, adjusting the network parameters of the spatial attention convolution unit based on the forecast timeliness information through the spatial attention convolution unit includes:

[0019] The forecast timeliness information is processed through the multi-level fully connected layers in the spatial attention convolution unit to generate tensors;

[0020] The tensor is fused with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit respectively to obtain a new convolution kernel and a new bias to adjust the network parameters of the spatial attention convolution unit.

[0021] In one embodiment, the network parameters of the non-integer rate upsampling layer are adjusted based on the forecast data to be processed and the forecast timeliness information through the non-integer rate upsampling network, including:

[0022] Performing spatial projection on the forecast data to be processed, so as to project pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtaining the coordinate information of the pixels in the spatiotemporal downscaling space;

[0023] Based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space, the relative distance corresponding to the pixel is determined;

[0024] The relative distance and prediction time information corresponding to the pixels are processed by the multi-level fully connected layers in the non-integer rate upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer rate upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer rate upsampling layer.

[0025] In one embodiment, performing non-integer upsampling processing on the target feature map using the adjusted network parameters to obtain an upsampled feature map includes:

[0026] Through the grid sampling layer with adjusted parameters, the coordinate information of the target feature map and pixels in the spatiotemporal downscaling space is upsampled to obtain the output result of the grid sampling layer;

[0027] Through the convolution layer with adjusted parameters, the output result of the grid sampling layer is convolved to obtain the output result of the convolution layer;

[0028] The output of the grid sampling layer is added to the output of the convolutional layer to obtain the upsampled feature map.

[0029] In one embodiment, the method further comprises:

[0030] Obtain historical forecast data, historical forecast time information, geographic information and historical high-resolution land surface data;

[0031] Preprocess the historical forecast data and historical high-resolution land surface data respectively;

[0032] The preprocessed historical forecast data, historical forecast timeliness information, and geographic information are used as model inputs, and the preprocessed historical high-resolution land surface data are used as training labels to construct a training data set.

[0033] The spatiotemporal downscaling model is trained using the training dataset.

[0034] In a second aspect, an embodiment of the present invention further provides a spatiotemporal downscaling device based on deep learning, comprising:

[0035] A data acquisition module is used to obtain forecast data to be processed, forecast timeliness information and geographic information;

[0036] A data preprocessing module is used to preprocess the forecast data to be processed, and input the preprocessed forecast data to be processed, forecast timeliness information and geographic information into the pre-trained spatiotemporal downscaling model;

[0037] The spatiotemporal downscaling module is used to extract the feature map of the forecast data to be processed through the spatiotemporal downscaling model, and use the forecast timeliness information to improve the timeliness adaptability of the feature map to obtain the target feature map. After performing non-integer multiple upsampling on the target feature map, it is combined with geographic information to generate the target forecast data after spatiotemporal downscaling.

[0038] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0040] The embodiment of the present invention provides a spatiotemporal downscaling method, device, equipment and medium based on deep learning. First, the forecast data to be processed is preprocessed, and the preprocessed forecast data to be processed, forecast timeliness information and geographic information are input into the pre-trained spatiotemporal downscaling model; then, the feature map of the forecast data to be processed is extracted through the spatiotemporal downscaling model, and the forecast timeliness information is used to improve the timeliness adaptability of the feature map to obtain the target feature map, and the target feature map is subjected to non-integer multiple upsampling processing, and then the target forecast data after spatiotemporal downscaling is generated in combination with the geographic information. The above method can achieve efficient downscaling in both the time dimension and the space dimension by adopting a spatiotemporal downscaling model under a unified framework, and introduces forecast timeliness information to improve the timeliness adaptability of the feature map, so as to optimize the data processing process in different prediction periods, thereby effectively reducing the deviation in the target forecast data, and thus significantly improving the spatiotemporal downscaling effect.

[0041] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a process flow of a spatiotemporal downscaling method based on deep learning provided in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a flow chart of a spatiotemporal downscaling model provided in an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the specific structure of a spatiotemporal downscaling model provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a spatial attention convolution unit in a forecast time adaptation subnetwork provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of a non-integer rate upsampling network provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the structure of a deep learning-based spatiotemporal downscaling device provided in an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] At present, relevant technical means can only improve the spatiotemporal resolution of forecasts, but cannot effectively reduce forecast deviations, resulting in poor spatiotemporal downscaling effects. Based on this, the present invention provides a spatiotemporal downscaling method, device, equipment and medium based on deep learning, which can effectively improve the spatiotemporal resolution of forecast data and effectively reduce forecast deviations, thereby significantly improving the spatiotemporal downscaling effect.

[0053] To facilitate understanding of this embodiment, a spatiotemporal downscaling method based on deep learning disclosed in an embodiment of the present invention is first introduced in detail. Figure 1 The flowchart of a spatiotemporal downscaling method based on deep learning is shown, and the method mainly includes the following steps S102 to S106:

[0054] Step S102, obtaining forecast data to be processed, forecast timeliness information and geographic information.

[0055] The forecast data to be processed may be multiple C1D forecast variables provided by the European Centre for Medium-Range Weather Forecasts (ECMWF), the forecast validity information is the validity period of the forecast data to be processed, and the geographic information includes digital elevation model (DEM), soil sand content, land type, etc.

[0056] Step S104, preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, forecast timeliness information and geographic information into the pre-trained spatiotemporal downscaling model.

[0057] In one example, the preprocessing includes normalization processing, preferably a maximum and minimum value normalization method, and the normalized forecast data to be processed, forecast timeliness information and geographic information are used as inputs of the spatiotemporal downscaling model.

[0058] Step S106, extracting the feature map of the forecast data to be processed through the spatiotemporal downscaling model, and using the forecast timeliness information to improve the timeliness adaptability of the feature map to obtain the target feature map, performing non-integer multiple upsampling on the target feature map, and generating the target forecast data after spatiotemporal downscaling in combination with the geographic information.

[0059] Among them, the spatiotemporal downscaling model includes a multi-scale feature fusion network, a forecast time adaptation network, a non-integer rate upsampling layer and a terrain fusion network. The input of the multi-scale feature fusion network is the pre-processed forecast data to be processed, and the output is a feature map. The network is used to extract and fuse multi-scale spatiotemporal features of the forecast data to be processed; the input of the forecast time adaptation network is the forecast time information and the feature map output by the multi-scale feature fusion network, and the output is a target feature map. The network is used to update its own network parameters using the forecast time information, and use the updated network parameters to process the feature map to generate a target feature map with better time adaptability; the input of the non-integer rate upsampling layer is the forecast time information and the target feature map output by the forecast time adaptation network. The network is used to update its own network parameters using the forecast time information, and use the updated network parameters to perform non-integer rate upsampling on the target feature map to obtain the upsampled feature map; the input of the terrain fusion network is the terrain information and the upsampled feature map, and the output is the target forecast data after spatiotemporal downscaling. The network is used to fuse the terrain information and the upsampled feature map to generate the target forecast data after spatiotemporal downscaling.

[0060] The deep learning-based spatiotemporal downscaling method provided in the embodiment of the present invention can simultaneously achieve efficient downscaling in the time dimension and the space dimension by adopting a spatiotemporal downscaling model under a unified framework, and introduces forecast timeliness information to improve the timeliness adaptability of the feature map, so as to optimize the data processing process in different prediction time periods, thereby effectively reducing the deviation in the target forecast data, and thus significantly improving the spatiotemporal downscaling effect.

[0061] Before using the spatiotemporal downscaling model to perform spatiotemporal downscaling on the forecast data to be processed, the spatiotemporal downscaling model needs to be trained in advance. Figure 2 The figure shows a schematic diagram of the process of a spatiotemporal downscaling model, including data set preparation, quality control and normalization, model construction, model training, model preservation and evaluation.

[0062] For ease of understanding, the embodiment of the present invention provides a specific implementation of training a spatiotemporal downscaling model, including the following steps S202 to S208:

[0063] Step S202, obtain historical forecast data, historical forecast time information, geographic information and historical high-resolution land surface data. The historical forecast data can be C1D multiple forecast variables of the historical period provided by the European Center for Medium-Range Weather Forecasts (ECMWF), the historical forecast time information is the validity period of the historical forecast data, the geographic information includes digital elevation model (DEM), soil sand content ratio, land type, etc., and the historical high-resolution road surface data is also a variable provided by the High Resolution Land Surface Data Assimilation System (HRCLDAS), which is used as a label.

[0064] Step S204, preprocessing the historical forecast data and the historical high-resolution land surface data respectively. The preprocessing of the historical forecast data includes normalization processing, and the preprocessing of the historical high-resolution land surface data includes quality control processing and normalization processing.

[0065] Optionally, quality control processing uses a combination of limit value testing and row and column standard deviation analysis. Specifically, the limit value test aims to ensure that the values ​​of all grid points are within a reasonable range in a climatological sense; for data points outside the range, the average value is replaced. Row and column standard deviation analysis identifies potential anomalies by calculating the standard deviation of each row and column in the image. If the standard deviation of multiple consecutive rows or columns is found to be zero, the sample will be marked as suspicious and further confirmed by visualization to see if it is indeed an anomaly. Once confirmed as an anomaly, the relevant sample will be removed from the dataset.

[0066] Normalization uses the maximum and minimum value normalization method. The normalization formula is:

[0067]

[0068] Among them, X * is the normalized historical forecast data or historical high-resolution land surface data, X is the historical forecast data or historical high-resolution land surface data, max(X) is the maximum value of the historical forecast data or historical high-resolution land surface data, and min(X) is the minimum value of the historical forecast data or historical high-resolution land surface data.

[0069] Step S206, using the preprocessed historical forecast data, historical forecast timeliness information, and geographic information as model inputs, and using the preprocessed historical high-resolution land surface data as training labels to construct a training data set.

[0070] Step S208, using the training data set to train the spatiotemporal downscaling model. In the specific implementation, after the pre-processed historical forecast data, historical forecast timeliness information, and geographic information in the training data set are input into the spatiotemporal downscaling model, the feature map of the historical forecast data is extracted through the spatiotemporal downscaling model, and the historical forecast timeliness information is used to improve the timeliness adaptability of the feature map to obtain the target feature map, and after the target feature map is upsampled at a non-integer multiple rate, the target historical forecast data after spatiotemporal downscaling is generated in combination with the geographic information. Finally, the spatiotemporal downscaling model is trained based on the target historical forecast data and the historical high-resolution land surface data.

[0071] Preferably, the optimizer in the model training process uses Adam, the initial learning rate is 2e-4, and the loss function is SmoothL1. After the model training is completed, the model parameters need to be saved so that the spatiotemporal downscaling model with the parameters can be used to generate high spatiotemporal resolution forecast data (ie, target forecast data) in the future.

[0072] To facilitate understanding of the process of performing spatiotemporal downscaling on the output data by the spatiotemporal downscaling model in the training stage and the prediction stage (ie, the aforementioned step S208 and the aforementioned step S106), an embodiment of the present invention provides a specific implementation method.

[0073] First, the structure of the spatiotemporal downscaling model is explained. The spatiotemporal downscaling model can adapt to forecast data of any magnification. The embodiment of the present invention provides the following Figure 3 The specific structural diagram of a spatiotemporal downscaling model shown in FIG. 1 includes a multi-scale feature fusion network (Unet3d), a forecast time adaptation network, a non-integer upsampling layer (Upsample), and a terrain fusion network (Conv2d). The forecast time adaptation network includes multiple levels (such as 2 levels) of forecast time adaptation subnetworks, and each level of the forecast time adaptation subnetwork includes a spatial attention convolution unit (SA_Conv) and an activation function unit (ReLU). In addition, Figure 3 It also shows that the input of the spatiotemporal downscaling model includes the preprocessed forecast data (LR), forecast timeliness information (t) and geographic information (Static_data), and the output is the target forecast data (HR).

[0074] The embodiment of the present invention takes the prediction stage as an example and provides a specific implementation method of performing spatiotemporal downscaling on the output data by a spatiotemporal downscaling model. See steps 1 to 4 below:

[0075] Step 1: Through a multi-scale feature fusion network, multi-scale spatiotemporal features are extracted and fused on the forecast data to be processed to obtain a feature map of the forecast data to be processed.

[0076] In a specific implementation, the multi-scale feature fusion network is based on a 3D U-Net architecture and uses 3D convolution (Conv3d) technology to extract spatiotemporal features (i.e., feature maps) from the forecast data to be processed. In order to enhance the nonlinear fitting ability of the model, ReLU is used as the activation function, and a channel attention mechanism is introduced after each ReLU activation to enhance the expressiveness of important features.

[0077] Step 2: Process the feature map through the multi-level forecast time adaptation sub-network in the forecast time adaptation network to improve the time adaptability of the feature map and obtain the target feature map.

[0078] In one embodiment, in each level of forecast time adaptation subnetwork, the network parameters of the spatial attention convolution unit are adjusted based on the forecast time information, and the feature map is convolved using the adjusted network parameters. The feature map after the convolution operation is processed by the activation function unit and output. The feature map output by the last forecast time adaptation subnetwork is used as the target feature map. The adjusted network parameters include the bias and convolution kernel of the convolution unit.

[0079] The purpose of the forecast time adaptation network is to improve the adaptability of the model to different forecast times and reduce the prediction error between forecast periods. It combines the input feature map with the forecast time information and uses the spatial attention convolution (SA_Conv) operation, that is, first use the fully connected layer to process the forecast time information to generate a tensor, and then use this tensor to adjust the convolution kernel parameters to form a new convolution kernel. After each SA_Conv, the ReLU activation function is applied to further improve the model performance. Figure 3 Taking as an example, the feature map is processed by a spatial attention convolution unit, an activation function unit, another spatial attention convolution unit, and another activation function unit, and the target feature map is output.

[0080] The embodiments of the present invention provide Figure 4 A schematic diagram of the structure of a spatial attention convolution unit in a prediction time adaptation subnetwork is shown in FIG. Figure 4 The spatial attention convolution unit is shown to include a multi-level (such as 2-level) fully connected layer FC and a convolution unit Conv. Based on this structure, the embodiment of the present invention provides a specific implementation of the SA_Conv operation, including:

[0081] (I) The forecast timeliness information is processed through the multi-level fully connected layers in the spatial attention convolution unit to generate tensors. Please continue to refer to Figure 4 , the forecast time information t is generated into a tensor after passing through two fully connected layers.

[0082] (ii) Fuse the tensor with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit to obtain a new convolution kernel and a new bias, so as to adjust the network parameters of the spatial attention convolution unit. Figure 4 , there are two preset tensors Tensor, one of which represents the current bias of the convolution layer, and the other represents the current convolution kernel of the convolution layer. The tensor generated based on the forecast time information t is fused with the two tensors Tensor to obtain a new convolution kernel Kernel and a new bias Bias.

[0083] On this basis, the new convolution kernel Kernel and the new bias Bias are configured to the convolution layer Conv, so as to use the convolution layer Conv to process the feature map feature output by the multi-scale feature fusion network to obtain the target feature map feature.

[0084] Step 3: Through the non-integer rate upsampling network, the network parameters of the non-integer rate upsampling layer are adjusted based on the forecast data to be processed and the forecast timeliness information, and the target feature map is subjected to non-integer rate upsampling processing using the adjusted network parameters to obtain the upsampled feature map.

[0085] Considering that the super-resolution requirements of meteorological data often involve non-integer multiple expansion, this solution does not adopt the traditional PixelShuffle method, but uses an upsampling algorithm designed specifically for non-integer multiples. Figure 5 The schematic diagram of the structure of a non-integer upsampling network is shown, and the non-integer upsampling network includes a multi-level fully connected layer FC, a grid sampling layer grid_sample and a convolution layer Conv. Specifically, it includes 3 levels of fully connected layers FC, and the third level includes 3 fully connected layers FC with different parameters.

[0086] Based on this structure, an embodiment of the present invention provides a specific implementation of non-integer rate upsampling processing, including:

[0087] (i) Perform spatial projection on the forecast data to be processed, so as to project the pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtain the coordinate information of the pixels in the spatiotemporal downscaling space. Specifically, each pixel on the forecast data to be processed HR is projected to the LR space (also known as the spatiotemporal downscaling space) to calculate the coordinate information C(x) and C(y) on the LR space.

[0088] (ii) Based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space, determine the relative distance corresponding to the pixel. Specifically, calculate the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space C(x), C(y), and determine the relative distance R(x), R(y) corresponding to the pixel.

[0089] (III) The relative distance and prediction time information corresponding to the pixels are processed by the multi-level fully connected layers in the non-integer upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer upsampling layer. Please continue to refer to Figure 5After the relative distances R(x), R(y) and the forecast timeliness information t are processed by the three-level fully connected layer FC, the offset offset of the grid sampling layer grid_sample and the convolution kernel Kernel and bias Bias of the convolution layer Conv are obtained.

[0090] (IV) Through the grid sampling layer with adjusted parameters, the target feature map and the coordinate information of the pixel in the spatiotemporal downscaling space are upsampled to obtain the output result of the grid sampling layer. Figure 5 , configure the offset offset to the grid sampling layer grid_sample, so as to use the grid sampling layer grid_sample to upsample the target feature map feature output by the prediction time adaptation network.

[0091] (V) Through the convolution layer with adjusted parameters, the output result of the grid sampling layer is convolved to obtain the output result of the convolution layer. Figure 5 , configure the convolution kernel Kernel and bias Bias to the convolution layer Conv, so as to use the convolution layer Conv to perform a convolution operation on the output result of the grid sampling layer.

[0092] (VI) Add the output of the grid sampling layer to the output of the convolutional layer to obtain the upsampled feature map. Figure 5 , the grid sampling layer grid_sample is jump-connected to the convolution layer Conv, that is, the output result of the grid sampling layer grid_sample is added to the output result of the convolution layer Conv to obtain the up-sampled feature map feature.

[0093] Step 4, through the terrain fusion network, based on the fusion results of the upsampled feature map and the geographic information, the target forecast data after spatiotemporal downscaling is generated. In a specific embodiment, after completing the upsampling step, the terrain fusion network receives geographic information data such as digital elevation model (DEM), soil sand content, land type, and upsampled feature map. All of this information is spliced ​​together along the channel dimension, and then sent to a series of convolutional layers for deep processing, and finally outputs target forecast data with higher resolution.

[0094] Furthermore, the target forecast data can be used to evaluate the results of the spatiotemporal downscaling model, and the evaluation indicators are MAE and RMSE. Specifically, MAE and RMSE can be calculated according to the following formula:

[0095]

[0096] Among them, n is the amount of forecast data to be processed, Y iis the target forecast data corresponding to the i-th forecast data to be processed, is the true value corresponding to the i-th forecast data to be processed (such as high-resolution land surface data).

[0097] In summary, firstly, a data set is constructed and the data is quality controlled and normalized, then a spatiotemporal downscaling model is built and the model is trained using the data set, and finally the trained model is used for real-time downscaling. The present invention uses deep learning to perform spatiotemporal downscaling of arbitrary magnifications. By adopting a spatiotemporal downscaling model under a unified architecture, the present invention can simultaneously achieve efficient downscaling in time and space dimensions, and introduce a forecast time adaptation layer to optimize data processing in different forecast periods, thereby effectively reducing the deviation in the forecast results. Experimental verification shows that the ECMWF model spatiotemporal downscaling technology disclosed in the embodiment of the present invention can reduce the spatial resolution of the ECMWF model from 0.125° to any resolution of 0.125° to 0.01°, and the time resolution from 3h to 1h, while improving the spatiotemporal resolution and correcting the forecast with obvious deviations. The embodiment of the present invention can reduce the spatiotemporal resolution and reduce the forecast error through this objective method, provide a reference for forecasters, better serve people's production and life, and meet the current social needs for forecast accuracy.

[0098] Based on the above embodiments, the present invention provides a spatiotemporal downscaling device based on deep learning. Figure 6 The schematic diagram of the structure of a deep learning-based spatiotemporal downscaling device is shown in FIG. 1 , and the device mainly includes the following parts:

[0099] The data acquisition module 602 is used to acquire the forecast data to be processed, the forecast timeliness information and the geographic information;

[0100] The data preprocessing module 604 is used to preprocess the forecast data to be processed, and input the preprocessed forecast data to be processed, forecast timeliness information and geographic information into the pre-trained spatiotemporal downscaling model;

[0101] The spatiotemporal downscaling module 606 is used to extract the feature map of the forecast data to be processed through the spatiotemporal downscaling model, and use the forecast time information to improve the time adaptability of the feature map to obtain the target feature map, perform non-integer multiple upsampling on the target feature map, and generate the target forecast data after spatiotemporal downscaling in combination with the geographic information.

[0102] The deep learning-based spatiotemporal downscaling device provided in an embodiment of the present invention can simultaneously achieve efficient downscaling in the time dimension and the space dimension by adopting a spatiotemporal downscaling model under a unified framework, and introduces forecast timeliness information to improve the timeliness adaptability of the feature map, so as to optimize the data processing process in different prediction time periods, thereby effectively reducing the deviation in the target forecast data, and thus significantly improving the spatiotemporal downscaling effect.

[0103] In one embodiment, the spatiotemporal downscaling model includes a multi-scale feature fusion network, a forecast time adaptation network, a non-integer multiple upsampling layer and a terrain fusion network; the spatiotemporal downscaling module 606 is specifically used to:

[0104] Through the multi-scale feature fusion network, multi-scale spatiotemporal features are extracted and fused for the forecast data to be processed, so as to obtain the feature map of the forecast data to be processed;

[0105] The feature map is processed through a multi-level forecast time-effectiveness adaptation sub-network in the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature map and obtain a target feature map;

[0106] Through a non-integer rate upsampling network, the network parameters of the non-integer rate upsampling layer are adjusted based on the forecast data to be processed and the forecast timeliness information, and the target feature map is subjected to non-integer rate upsampling processing using the adjusted network parameters to obtain an upsampled feature map;

[0107] Through the terrain fusion network, the target forecast data after spatiotemporal downscaling is generated based on the fusion results of the upsampled feature map and geographic information.

[0108] In one embodiment, the forecast timeliness adaptation network includes multiple levels of forecast timeliness adaptation sub-networks, each level of forecast timeliness adaptation sub-network includes a spatial attention convolution unit and an activation function unit; the spatiotemporal downscaling module 606 is specifically used for:

[0109] In each level of forecast time adaptation sub-network, the network parameters of the spatial attention convolution unit are adjusted based on the forecast time information through the spatial attention convolution unit, and the feature map is convolved using the adjusted network parameters. The feature map after the convolution operation is processed by the activation function unit and then output;

[0110] The feature map output by the last prediction time adaptation sub-network is used as the target feature map.

[0111] In one implementation, the spatiotemporal downscaling module 606 is specifically configured to:

[0112] The forecast timeliness information is processed through the multi-level fully connected layers in the spatial attention convolution unit to generate tensors;

[0113] The tensor is fused with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit respectively to obtain a new convolution kernel and a new bias to adjust the network parameters of the spatial attention convolution unit.

[0114] In one implementation, the spatiotemporal downscaling module 606 is specifically configured to:

[0115] Performing spatial projection on the forecast data to be processed, so as to project pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtaining the coordinate information of the pixels in the spatiotemporal downscaling space;

[0116] Based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space, the relative distance corresponding to the pixel is determined;

[0117] The relative distance and prediction time information corresponding to the pixels are processed by the multi-level fully connected layers in the non-integer rate upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer rate upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer rate upsampling layer.

[0118] In one implementation, the spatiotemporal downscaling module 606 is specifically configured to:

[0119] Through the grid sampling layer with adjusted parameters, the coordinate information of the target feature map and pixels in the spatiotemporal downscaling space is upsampled to obtain the output result of the grid sampling layer;

[0120] Through the convolution layer with adjusted parameters, the output result of the grid sampling layer is convolved to obtain the output result of the convolution layer;

[0121] The output of the grid sampling layer is added to the output of the convolutional layer to obtain the upsampled feature map.

[0122] In one embodiment, a model training module is further included, which is used to:

[0123] Obtain historical forecast data, historical forecast time information, geographic information and historical high-resolution land surface data;

[0124] Preprocess the historical forecast data and historical high-resolution land surface data respectively;

[0125] The preprocessed historical forecast data, historical forecast timeliness information, and geographic information are used as model inputs, and the preprocessed historical high-resolution land surface data are used as training labels to construct a training data set.

[0126] The spatiotemporal downscaling model is trained using the training dataset.

[0127] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0128] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0129] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 70, a memory 71, a bus 72 and a communication interface 73, wherein the processor 70, the communication interface 73 and the memory 71 are connected via the bus 72; the processor 70 is used to execute an executable module stored in the memory 71, such as a computer program.

[0130] The memory 71 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 73 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0131] The bus 72 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0132] Among them, the memory 71 is used to store programs, and the processor 70 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 70 or implemented by the processor 70.

[0133] The processor 70 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 70. The above processor 70 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 71, and the processor 70 reads the information in the memory 71 and completes the steps of the above method in combination with its hardware.

[0134] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0136] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A spatiotemporal downscaling method based on deep learning, characterized in that: include: Obtain forecast data to be processed, forecast timeliness information and geographic information; Preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, the forecast timeliness information and the geographic information into a pre-trained spatiotemporal downscaling model; The feature map of the forecast data to be processed is extracted through the spatiotemporal downscaling model, and the forecast timeliness information is used to improve the timeliness adaptability of the feature map to obtain a target feature map. After the target feature map is upsampled at a non-integer rate, the target forecast data after spatiotemporal downscaling is generated in combination with the geographic information.

2. The deep learning-based spatiotemporal downscaling method according to claim 1, characterized in that: The spatiotemporal downscaling model includes a multi-scale feature fusion network, a forecast time adaptation network, a non-integer rate upsampling layer and a terrain fusion network; The feature map of the forecast data to be processed is extracted by the spatiotemporal downscaling model, and the time adaptability of the feature map is improved by using the forecast time information to obtain a target feature map. After performing non-integer multiple upsampling processing on the target feature map, the target forecast data after spatiotemporal downscaling is generated in combination with the geographic information, including: Through the multi-scale feature fusion network, multi-scale spatiotemporal feature extraction and fusion are performed on the forecast data to be processed to obtain a feature map of the forecast data to be processed; Processing the feature graph through a multi-level forecast time-effectiveness adaptation sub-network within the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature graph to obtain a target feature graph; By means of the non-integer rate upsampling network, the network parameters of the non-integer rate upsampling layer are adjusted based on the forecast data to be processed and the forecast timeliness information, and the target feature map is subjected to non-integer rate upsampling processing using the adjusted network parameters to obtain an upsampled feature map; Through the terrain fusion network, based on the fusion result of the upsampled feature map and the geographic information, the target forecast data after spatiotemporal downscaling is generated.

3. The deep learning-based spatiotemporal downscaling method according to claim 2, characterized in that: The forecast timeliness adaptation network includes multiple levels of forecast timeliness adaptation sub-networks, and each level of the forecast timeliness adaptation sub-network includes a spatial attention convolution unit and an activation function unit; The feature graph is processed by a multi-level forecast time-effectiveness adaptation sub-network in the forecast time-effectiveness adaptation network to improve the time-effectiveness adaptability of the feature graph to obtain a target feature graph, including: In each level of the forecast time adaptation subnetwork, the network parameters of the spatial attention convolution unit are adjusted based on the forecast time information through the spatial attention convolution unit, and the feature map is convolved using the adjusted network parameters, and the feature map after the convolution operation is processed by the activation function unit and then output; The feature map output by the last forecast time adaptation sub-network is used as the target feature map.

4. The deep learning-based spatiotemporal downscaling method according to claim 3, characterized in that: By means of the spatial attention convolution unit, adjusting the network parameters of the spatial attention convolution unit based on the forecast timeliness information includes: Processing the forecast timeliness information through a multi-level fully connected layer in the spatial attention convolution unit to generate a tensor; The tensor is fused with the convolution kernel and bias of the convolution layer in the spatial attention convolution unit respectively to obtain a new convolution kernel and a new bias, thereby adjusting the network parameters of the spatial attention convolution unit.

5. The deep learning-based spatiotemporal downscaling method according to claim 2, characterized in that: By means of the non-integer rate upsampling network, adjusting the network parameters of the non-integer rate upsampling layer based on the forecast data to be processed and the forecast timeliness information, comprising: Performing spatial projection on the forecast data to be processed, so as to project pixels in the forecast data to be processed from the current space to the spatiotemporal downscaling space, and obtaining coordinate information of the pixels in the spatiotemporal downscaling space; Determine the relative distance corresponding to the pixel based on the coordinate information of the pixel in the current space and the coordinate information of the pixel in the spatiotemporal downscaling space; The relative distance corresponding to the pixel and the forecast timeliness information are processed by the multi-level fully connected layers in the non-integer rate upsampling network, and the offset of the grid sampling layer and the convolution kernel and bias of the convolution layer in the non-integer rate upsampling network are obtained respectively, so as to adjust the network parameters of the non-integer rate upsampling layer.

6. The deep learning-based spatiotemporal downscaling method according to claim 5, characterized in that: Performing non-integer upsampling processing on the target feature map using the adjusted network parameters to obtain an upsampled feature map, including: By adjusting the parameters of the grid sampling layer, upsampling the target feature map and the coordinate information of the pixel in the spatiotemporal downscaling space is performed to obtain an output result of the grid sampling layer; By adjusting the parameters of the convolution layer, a convolution operation is performed on the output result of the grid sampling layer to obtain the output result of the convolution layer; The output result of the grid sampling layer is added to the output result of the convolution layer to obtain an upsampled feature map.

7. The deep learning-based spatiotemporal downscaling method according to claim 1, characterized in that: The method further comprises: Obtain historical forecast data, historical forecast time information, geographic information and historical high-resolution land surface data; Preprocessing the historical forecast data and the historical high-resolution land surface data respectively; The pre-processed historical forecast data, the historical forecast timeliness information, and the geographic information are used as model inputs, and the pre-processed historical high-resolution land surface data are used as training labels to construct a training data set; The spatiotemporal downscaling model is trained using the training data set.

8. A spatiotemporal downscaling device based on deep learning, characterized in that: include: A data acquisition module is used to obtain forecast data to be processed, forecast timeliness information and geographic information; A data preprocessing module, used for preprocessing the forecast data to be processed, and inputting the preprocessed forecast data to be processed, the forecast timeliness information and the geographic information into a pre-trained spatiotemporal downscaling model; The spatiotemporal downscaling module is used to extract the feature map of the forecast data to be processed through the spatiotemporal downscaling model, and use the forecast timeliness information to improve the timeliness adaptability of the feature map to obtain a target feature map, perform non-integer multiple upsampling processing on the target feature map, and generate the target forecast data after spatiotemporal downscaling in combination with the geographic information.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images